Package org.codelibs.fess.helper

Class DataIndexHelper

java.lang.Object
org.codelibs.fess.helper.DataIndexHelper
public class DataIndexHelper extends Object
Helper class for managing data crawling operations in Fess. This class coordinates the execution of data store crawling processes, managing multiple concurrent crawling threads and handling the indexing of crawled documents into the search engine.

The DataIndexHelper supports:

  • Concurrent crawling of multiple data configurations
  • Thread pool management for crawler execution
  • Session-based crawling with cleanup operations
  • Old document deletion after successful crawling
  • Crawling execution monitoring and timing

  • Field Details

    • crawlingExecutionInterval

      protected long crawlingExecutionInterval
      Interval in milliseconds between crawler thread executions. Used to control the rate at which new crawler threads are started and the frequency of status checks.

    • crawlerPriority

      protected int crawlerPriority
      Thread priority for crawler threads. Defaults to normal thread priority.

    • dataCrawlingThreadList

      protected final List<DataIndexHelper.DataCrawlingThread> dataCrawlingThreadList
      Thread-safe list of active data crawling threads. Used to track and manage all currently running crawler threads.

  • Constructor Details

    • DataIndexHelper

      public DataIndexHelper()
      Creates a new instance of DataIndexHelper. This constructor initializes the helper for managing data crawling operations, including thread pool management and session-based crawling coordination.

  • Method Details

    • crawl

      public void crawl(String sessionId)
      Initiates crawling for all configured data stores. This method retrieves all available data configurations and starts the crawling process for each one.
      Parameters:
      sessionId - unique identifier for this crawling session

    • crawl

      public void crawl(String sessionId, List<String> configIdList)
      Initiates crawling for specific data configurations. This method starts crawling only for the data configurations specified in the configIdList parameter.
      Parameters:
      sessionId - unique identifier for this crawling session
      configIdList - list of data configuration IDs to crawl

    • doCrawl

      protected void doCrawl(String sessionId, List<org.codelibs.fess.opensearch.config.exentity.DataConfig> configList)
      Performs the actual crawling operation for the provided data configurations. This method manages the creation and execution of crawler threads, monitors their progress, and handles cleanup operations.

      The method:

      • Creates crawler threads for each data configuration
      • Manages concurrent execution based on thread count limits
      • Monitors thread completion and handles cleanup
      • Records execution timing and statistics
      Parameters:
      sessionId - unique identifier for this crawling session
      configList - list of data configurations to crawl

    • setCrawlingExecutionInterval

      public void setCrawlingExecutionInterval(long crawlingExecutionInterval)
      Sets the crawling execution interval. This interval controls the delay between starting new crawler threads and the frequency of status checks during crawling operations.
      Parameters:
      crawlingExecutionInterval - interval in milliseconds

    • setCrawlerPriority

      public void setCrawlerPriority(int crawlerPriority)
      Sets the thread priority for crawler threads. This priority will be applied to all newly created crawler threads.
      Parameters:
      crawlerPriority - thread priority (typically Thread.MIN_PRIORITY to Thread.MAX_PRIORITY)