Package org.codelibs.fess.helper

Class DocumentHelper

java.lang.Object
org.codelibs.fess.helper.DocumentHelper
public class DocumentHelper extends Object
Helper class for document processing and manipulation in the Fess search system. This class provides utilities for processing document content, titles, and digests, handling text normalization, content extraction, and similar document hash encoding/decoding. It also manages document processing requests and integrates with the crawler system.

  • Field Details

    • SIMILAR_DOC_HASH_PREFIX

      protected static final String SIMILAR_DOC_HASH_PREFIX
      Prefix used for encoded similar document hashes
      See Also:

  • Constructor Details

    • DocumentHelper

      public DocumentHelper()
      Default constructor for DocumentHelper. Creates a new document helper instance.

  • Method Details

    • init

      @PostConstruct public void init()
      Initializes the document helper after construction. Sets up the TikaExtractor with configuration parameters for text processing.

    • getTitle

      public String getTitle(org.codelibs.fess.crawler.entity.ResponseData responseData, String title, Map<String,Object> dataMap)
      Processes and normalizes a document title. Applies text normalization using configured space characters and returns a clean title suitable for indexing.
      Parameters:
      responseData - the response data from crawling (not currently used)
      title - the raw title text to process
      dataMap - additional data map (not currently used)
      Returns:
      the normalized title, or empty string if title is null

    • getContent

      public String getContent(org.codelibs.fess.opensearch.config.exentity.CrawlingConfig crawlingConfig, org.codelibs.fess.crawler.entity.ResponseData responseData, String content, Map<String,Object> dataMap)
      Processes and normalizes document content. Applies text normalization including duplicate term removal, size limits, and space character handling. May preserve original content based on configuration.
      Parameters:
      crawlingConfig - the crawling configuration containing processing parameters
      responseData - the response data from crawling
      content - the raw content text to process
      dataMap - additional data map
      Returns:
      the normalized content, or empty string if content is null

    • getMaxAlphanumTermSize

      protected int getMaxAlphanumTermSize()
      Gets the maximum size for alphanumeric terms from configuration.
      Returns:
      the maximum alphanumeric term size

    • getMaxSymbolTermSize

      protected int getMaxSymbolTermSize()
      Gets the maximum size for symbol terms from configuration.
      Returns:
      the maximum symbol term size

    • isDuplicateTermRemoved

      protected boolean isDuplicateTermRemoved()
      Checks if duplicate term removal is enabled in configuration.
      Returns:
      true if duplicate terms should be removed, false otherwise

    • getSpaceChars

      protected int[] getSpaceChars()
      Gets the array of space character codes from configuration.
      Returns:
      array of character codes to be treated as spaces

    • getDigest

      public String getDigest(org.codelibs.fess.crawler.entity.ResponseData responseData, String content, Map<String,Object> dataMap, int maxWidth)
      Creates a digest (abbreviated summary) of document content. Truncates and normalizes content to create a summary suitable for display.
      Parameters:
      responseData - the response data from crawling (not currently used)
      content - the content to create a digest from
      dataMap - additional data map (not currently used)
      maxWidth - the maximum width of the digest
      Returns:
      the abbreviated and normalized digest, or empty string if content is null

    • processRequest

      public Map<String,Object> processRequest(org.codelibs.fess.opensearch.config.exentity.CrawlingConfig crawlingConfig, String crawlingInfoId, String url)
      Processes a crawling request for a specific URL. Executes the full crawling pipeline including client execution, rule processing, transformation, and data extraction.
      Parameters:
      crawlingConfig - the crawling configuration to use
      crawlingInfoId - the crawling session ID
      url - the URL to process
      Returns:
      a map containing the processed document data
      Throws:
      org.codelibs.fess.crawler.exception.CrawlingAccessException - if crawling fails or configuration is invalid
      org.codelibs.fess.crawler.exception.ChildUrlsException - if the URL redirects to another location
      org.codelibs.fess.crawler.exception.CrawlerSystemException - if data deserialization fails

    • decodeSimilarDocHash

      public String decodeSimilarDocHash(String hash)
      Decodes a similar document hash from its compressed and encoded form. Reverses the encoding process applied by encodeSimilarDocHash.
      Parameters:
      hash - the encoded hash string to decode
      Returns:
      the decoded hash string, or the original hash if decoding fails

    • encodeSimilarDocHash

      public String encodeSimilarDocHash(String hash)
      Encodes a similar document hash using GZIP compression and Base64 encoding. This reduces storage space for hash values while maintaining uniqueness.
      Parameters:
      hash - the hash string to encode
      Returns:
      the encoded hash string with prefix, or the original hash if encoding fails

    • appendLineNumber

      public String appendLineNumber(String prefix, String content)
      Appends line numbers to each line of content with a given prefix. Useful for debugging and displaying content with line references.
      Parameters:
      prefix - the prefix to add before each line number
      content - the content to add line numbers to
      Returns:
      the content with line numbers prepended, or empty string if content is blank