Package org.codelibs.fess.chat

Class DefaultChatContentFetcher

java.lang.Object
org.codelibs.fess.chat.DefaultChatContentFetcher
All Implemented Interfaces:
ChatContentFetcher
public class DefaultChatContentFetcher extends Object implements ChatContentFetcher
Default ChatContentFetcher. Dispatches per document by content_length: small documents are fetched in full (existing behavior), large documents use query-relevant highlighted passages so the relevant section reaches the LLM. Highlight snippets are normalized into the content field so the downstream context builder is unchanged. A large document with no extractable highlight passage falls back to its full content truncated to the threshold; its short digest is never used as a passage.

  • Constructor Details

    • DefaultChatContentFetcher

      public DefaultChatContentFetcher()
      Default constructor.

  • Method Details

    • fetchContent

      public List<Map<String,Object>> fetchContent(ChatContentRequest request)
      Description copied from interface: ChatContentFetcher
      Resolves the LLM-context content for the given relevant documents.
      Specified by:
      fetchContent in interface ChatContentFetcher
      Parameters:
      request - the fetch request (doc ids, search results, query)
      Returns:
      document maps in the same order as request.getDocIds(), each with the content field set to the text to send to the LLM

    • decideStrategy

      protected DefaultChatContentFetcher.Strategy decideStrategy(String docId, Long contentLength, ChatContentRequest request)
      Decides the fetch strategy for a document.
      Parameters:
      docId - the document id
      contentLength - the indexed content length (null if unknown)
      request - the fetch request
      Returns:
      DefaultChatContentFetcher.Strategy.HIGHLIGHT for large documents, otherwise DefaultChatContentFetcher.Strategy.FULL

    • getFulltextThreshold

      protected long getFulltextThreshold()
      Returns the content_length threshold above which highlighted passages are used.
      Returns:
      the threshold in characters

    • truncateContent

      protected void truncateContent(Map<String,Object> doc, long maxLength)
      Truncates the content field of a document to at most maxLength characters, keeping the leading portion. Used as the fallback for a large document that produced no highlight passage so it still contributes a leading excerpt without crowding out the answer context.
      Parameters:
      doc - the document map (modified in place)
      maxLength - the maximum content length in characters

    • buildSizeMap

      protected Map<String,Long> buildSizeMap(List<Map<String,Object>> searchResults)
      Builds a doc_id -> content_length map from the search-phase results.
      Parameters:
      searchResults - the search-phase result maps (may be null)
      Returns:
      the size map (empty when searchResults is null/empty)

    • toLong

      protected Long toLong(Object value)
      Converts a value to Long, tolerating Number and numeric String.
      Parameters:
      value - the value
      Returns:
      the Long value, or null if not convertible

    • fetchFullContent

      protected List<Map<String,Object>> fetchFullContent(List<String> docIds)
      Fetches full content for the given document ids via doc_id lookup (no highlighting).
      Parameters:
      docIds - the document ids
      Returns:
      document maps (unordered)

    • fetchHighlightedContent

      protected List<Map<String,Object>> fetchHighlightedContent(List<String> docIds, String query)
      Fetches query-relevant highlighted passages for large documents via a doc_id-restricted highlighted search, normalizing the snippet into the content field. Returns empty on failure (callers fall back to full).
      Parameters:
      docIds - the large-document ids
      query - the highlight query
      Returns:
      document maps with content set to the highlighted snippet

    • normalizeHighlightedDocs

      protected List<Map<String,Object>> normalizeHighlightedDocs(List<Map<String,Object>> docs)
      Normalizes highlighted search results: copies the highlight passage into the content field, and drops documents that produced no highlight passage so the caller can fall back to their truncated full content.

      The passage is read from the raw highlight field (hl_content), not from content_description. content_description is built by ViewHelper#getContentDescription from hl_content,digest and therefore falls back to the short crawl-time digest when there is no highlight match; relying on it would mask the "no passage" case and send the digest to the LLM instead of the truncated full-content fallback. Reading the highlight field directly keeps that case detectable.

      Parameters:
      docs - the highlighted search result maps
      Returns:
      maps that produced a highlight passage only, each with content set to it

    • getHighlightContentField

      protected String getHighlightContentField()
      Returns the document-map key that holds the highlight passage for the content field (the highlight prefix joined with the content field name, e.g. hl_content).
      Returns:
      the highlight content field key