Package org.codelibs.fess.indexer

Class IndexUpdater

java.lang.Object
java.lang.Thread
org.codelibs.fess.indexer.IndexUpdater
All Implemented Interfaces:
Runnable
public class IndexUpdater extends Thread
IndexUpdater is responsible for updating the search index with crawled document data. This class extends Thread and continuously processes access results from the crawler, transforms them into indexed documents, and updates the OpenSearch index.

The updater performs the following key operations:

  • Retrieves crawled documents from the data service
  • Transforms document data using appropriate transformers
  • Applies document boosting rules and click/favorite count enhancements
  • Sends processed documents to the search engine for indexing
  • Manages cleanup of processed crawler session data

The updater runs continuously until crawling is finished and all documents are processed. It includes error handling, retry logic, and performance monitoring capabilities.

  • Field Details

    • sessionIdList

      protected List<String> sessionIdList
      List of crawler session IDs to process

    • searchEngineClient

      protected SearchEngineClient searchEngineClient
      OpenSearch client for index operations

    • dataService

      protected org.codelibs.fess.crawler.service.DataService<org.codelibs.fess.crawler.entity.OpenSearchAccessResult> dataService
      Service for managing crawled document data

    • urlQueueService

      protected org.codelibs.fess.crawler.service.UrlQueueService<org.codelibs.fess.crawler.entity.OpenSearchUrlQueue> urlQueueService
      Service for managing URL crawling queue

    • urlFilterService

      protected org.codelibs.fess.crawler.service.UrlFilterService urlFilterService
      Service for URL filtering operations

    • clickLogBhv

      protected org.codelibs.fess.opensearch.log.exbhv.ClickLogBhv clickLogBhv
      Behavior class for click log operations

    • favoriteLogBhv

      protected org.codelibs.fess.opensearch.log.exbhv.FavoriteLogBhv favoriteLogBhv
      Behavior class for favorite log operations

    • systemHelper

      protected SystemHelper systemHelper
      Helper for system-level operations

    • indexingHelper

      protected IndexingHelper indexingHelper
      Helper for document indexing operations

    • finishCrawling

      protected boolean finishCrawling
      Flag indicating if crawling should be finished

    • executeTime

      protected long executeTime
      Total execution time in milliseconds

    • documentSize

      protected long documentSize
      Total number of processed documents

    • maxIndexerErrorCount

      protected int maxIndexerErrorCount
      Maximum number of indexer errors allowed

    • maxErrorCount

      protected int maxErrorCount
      Maximum number of general errors allowed before termination

    • finishedSessionIdList

      protected List<String> finishedSessionIdList
      List of finished crawler session IDs for cleanup

  • Constructor Details

    • IndexUpdater

      public IndexUpdater()
      Default constructor for IndexUpdater. Initializes a new instance with default settings.

  • Method Details

    • init

      @PostConstruct public void init()
      Initializes the IndexUpdater after dependency injection. Sets up the ingest factory if available in the component container.

    • destroy

      @PreDestroy public void destroy()
      Destroys the IndexUpdater when the container is shutting down. Stops all crawler instances if crawling is still in progress.

    • addFinishedSessionId

      public void addFinishedSessionId(String sessionId)
      Adds a finished session ID to the cleanup list. This method is thread-safe and adds the session ID to be cleaned up later.
      Parameters:
      sessionId - the crawler session ID that has finished processing

    • run

      public void run()
      Main execution method that runs the index updating process. Continuously processes crawled documents from the data service, transforms them, and updates the search index until crawling is finished and all documents are processed.

      The method performs the following operations in a loop:

      • Retrieves access results from the data service
      • Processes each document through transformers
      • Applies document boosting and metadata enhancements
      • Sends processed documents to the search engine
      • Cleans up processed data and manages crawler sessions

      The method includes error handling, retry logic, and will terminate if too many empty results are encountered or if a system shutdown is requested.

      Specified by:
      run in interface Runnable
      Overrides:
      run in class Thread

    • ingest

      protected Map<String,Object> ingest(org.codelibs.fess.crawler.entity.AccessResult<String> accessResult, Map<String,Object> map)
      Processes a document through the ingest pipeline if an ingest factory is available. Applies all configured ingesters to transform and enrich the document data.
      Parameters:
      accessResult - the access result containing document metadata
      map - the document data map to process
      Returns:
      the processed document map after applying all ingesters

    • updateDocument

      protected void updateDocument(Map<String,Object> map)
      Updates a document with additional metadata and enhancements. Adds click counts, favorite counts, document boosting, and generates document ID. Also applies language-specific updates through the language helper.
      Parameters:
      map - the document data map to update with additional metadata

    • addBoostValue

      protected void addBoostValue(Map<String,Object> map, float documentBoost)
      Adds a boost value to the document for search relevance scoring. The boost value affects how highly the document will rank in search results.
      Parameters:
      map - the document data map to add the boost value to
      documentBoost - the boost value to apply to the document

    • addClickCountField

      protected void addClickCountField(Map<String,Object> doc)
      Adds a click count field to the document based on search log data. The click count represents how many times users have clicked on this document in search results.
      Parameters:
      doc - the document data map to add the click count to

    • addFavoriteCountField

      protected void addFavoriteCountField(Map<String,Object> map)
      Adds a favorite count field to the document based on user favorite data. The favorite count represents how many users have marked this document as a favorite.
      Parameters:
      map - the document data map to add the favorite count to

    • getExecuteTime

      public long getExecuteTime()
      Gets the total execution time for index updates.
      Returns:
      the total execution time in milliseconds

    • getSessionIdList

      public List<String> getSessionIdList()
      Gets the list of crawler session IDs being processed.
      Returns:
      the list of session IDs

    • setSessionIdList

      public void setSessionIdList(List<String> sessionIdList)
      Sets the list of crawler session IDs to process.
      Parameters:
      sessionIdList - the list of session IDs to set

    • setFinishCrawling

      public void setFinishCrawling(boolean finishCrawling)
      Sets the flag indicating whether crawling should be finished.
      Parameters:
      finishCrawling - true if crawling should be finished, false otherwise

    • getDocumentSize

      public long getDocumentSize()
      Gets the total number of documents processed.
      Returns:
      the total document count

    • setUncaughtExceptionHandler

      public void setUncaughtExceptionHandler(Thread.UncaughtExceptionHandler eh)
      Sets the uncaught exception handler for this IndexUpdater thread.
      Overrides:
      setUncaughtExceptionHandler in class Thread
      Parameters:
      eh - the uncaught exception handler to set

    • setDefaultUncaughtExceptionHandler

      public static void setDefaultUncaughtExceptionHandler(Thread.UncaughtExceptionHandler eh)
      Sets the default uncaught exception handler for all threads.
      Parameters:
      eh - the default uncaught exception handler to set

    • setMaxIndexerErrorCount

      public void setMaxIndexerErrorCount(int maxIndexerErrorCount)
      Sets the maximum number of indexer errors allowed.
      Parameters:
      maxIndexerErrorCount - the maximum error count to set

    • addDocBoostMatcher

      public void addDocBoostMatcher(DocBoostMatcher rule)
      Adds a document boost matcher rule for enhancing document relevance scores.
      Parameters:
      rule - the document boost matcher rule to add

    • setCrawlerList

      public void setCrawlerList(List<org.codelibs.fess.crawler.Crawler> crawlerList)
      Sets the list of crawler instances that this updater will manage.
      Parameters:
      crawlerList - the list of crawlers to set