Package org.codelibs.fess.ds.callback

Class FileListIndexUpdateCallbackImpl

java.lang.Object
org.codelibs.fess.ds.callback.FileListIndexUpdateCallbackImpl
All Implemented Interfaces:
IndexUpdateCallback
public class FileListIndexUpdateCallbackImpl extends Object implements IndexUpdateCallback
Implementation of IndexUpdateCallback that handles file list index updates with concurrent processing. This callback processes file events (create, modify, delete) and manages document indexing and deletion operations in the search engine. It supports recursive crawling with configurable depth and access count limits.

The implementation uses an executor service for concurrent processing of file operations and maintains a cache of URLs to be deleted for batch processing. It handles redirect following and child URL discovery during the crawling process.

  • Field Details

    • URL_EXCLUDE_PATTERN

      protected static final String URL_EXCLUDE_PATTERN
      The key used to specify the pattern for excluding URLs. This constant can be used to retrieve or set the URL exclusion pattern in configuration or processing logic.
      See Also:

    • indexUpdateCallback

      protected IndexUpdateCallback indexUpdateCallback
      The underlying index update callback to delegate operations to.

    • crawlerClientFactory

      protected org.codelibs.fess.crawler.client.CrawlerClientFactory crawlerClientFactory
      Factory for creating crawler clients to handle different URL schemes.

    • deleteUrlList

      protected List<String> deleteUrlList
      List of URLs to be deleted, cached for batch processing. All access is synchronized via indexUpdateCallback lock.

    • maxDeleteDocumentCacheSize

      protected int maxDeleteDocumentCacheSize
      Maximum size of the delete URL cache before batch deletion is triggered.

    • maxRedirectCount

      protected int maxRedirectCount
      Maximum number of redirects to follow when processing URLs.

  • Constructor Details

    • FileListIndexUpdateCallbackImpl

      public FileListIndexUpdateCallbackImpl(IndexUpdateCallback indexUpdateCallback, org.codelibs.fess.crawler.client.CrawlerClientFactory crawlerClientFactory, int nThreads)
      Constructs a new FileListIndexUpdateCallbackImpl with the specified parameters.
      Parameters:
      indexUpdateCallback - the underlying index update callback to delegate to
      crawlerClientFactory - the factory for creating crawler clients
      nThreads - the number of threads for the executor service (minimum 1)

  • Method Details

    • newFixedThreadPool

      protected ExecutorService newFixedThreadPool(int nThreads)
      Creates a new fixed thread pool executor with the specified number of threads.
      Parameters:
      nThreads - the number of threads in the pool
      Returns:
      a new ThreadPoolExecutor configured for this callback

    • store

      public void store(DataStoreParams paramMap, Map<String,Object> dataMap)
      Description copied from interface: IndexUpdateCallback
      Stores a document in the search index with the specified parameters and data.
      Specified by:
      store in interface IndexUpdateCallback
      Parameters:
      paramMap - the data store parameters containing configuration and metadata
      dataMap - the document data to be indexed as key-value pairs

    • getParamValue

      protected String getParamValue(DataStoreParams paramMap, String key, String defaultValue)
      Retrieves a parameter value from the data store parameters map.
      Parameters:
      paramMap - the parameter map to search
      key - the parameter key to look up
      defaultValue - the default value to return if key is not found
      Returns:
      the parameter value as a string, or the default value if not found

    • addDocument

      protected void addDocument(DataStoreParams paramMap, Map<String,Object> dataMap)
      Adds a document to the search index by crawling the specified URL and processing the content. This method handles recursive crawling with depth and access count limits, follows redirects, and processes child URLs discovered during crawling.
      Parameters:
      paramMap - the data store parameters containing crawling configuration
      dataMap - the data map containing the document information including the URL

    • isUrlCrawlable

      protected boolean isUrlCrawlable(DataStoreParams paramMap, String url)
      Determines whether the specified URL is crawlable based on the exclusion pattern provided in the paramMap. If the URL_EXCLUDE_PATTERN key exists in the parameter map, its value is used as a regular expression pattern to match against the given URL. If the URL matches the exclusion pattern, the method returns false, indicating that the URL should not be crawled. Otherwise, it returns true.
      Parameters:
      paramMap - the parameter map containing potential exclusion patterns
      url - the URL to be checked for crawlability
      Returns:
      true if the URL is crawlable; false if it matches the exclusion pattern

    • getMaxAccessCount

      protected long getMaxAccessCount(DataStoreParams paramMap, Map<String,Object> dataMap)
      Determines the maximum number of URLs to access during crawling. This method checks for explicit max_access_count parameter or recursive flag.
      Parameters:
      paramMap - the data store parameters
      dataMap - the data map containing crawling configuration
      Returns:
      the maximum access count (-1 for unlimited, 1 for single access, or specified count)

    • getMaxDepth

      protected int getMaxDepth(DataStoreParams paramMap, Map<String,Object> dataMap)
      Determines the maximum crawling depth from the configuration parameters.
      Parameters:
      paramMap - the data store parameters
      dataMap - the data map containing crawling configuration
      Returns:
      the maximum crawling depth (-1 for unlimited depth)

    • processRequest

      protected String processRequest(DataStoreParams paramMap, Map<String,Object> dataMap, String url, org.codelibs.fess.crawler.client.CrawlerClient client)
      Processes a single crawl request by executing the HTTP request, handling redirects, transforming the response data, and indexing the document.
      Parameters:
      paramMap - the data store parameters
      dataMap - the data map to be updated with response data
      url - the URL to process
      client - the crawler client to use for the request
      Returns:
      the redirect URL if a redirect occurred, null otherwise
      Throws:
      org.codelibs.fess.crawler.exception.ChildUrlsException - if child URLs are discovered during processing
      DataStoreCrawlingException - if an error occurs during crawling

    • mergeResponseData

      protected void mergeResponseData(Map<String,Object> dataMap, Map<String,Object> responseDataMap)
      Merges response data from the crawler into the original data map. Handles special ".overwrite" suffix fields by removing the suffix and overwriting the base field.
      Parameters:
      dataMap - the original data map to merge into
      responseDataMap - the response data map from the crawler

    • deleteDocument

      protected boolean deleteDocument(DataStoreParams paramMap, Map<String,Object> dataMap)
      Deletes a document from the search index based on the URL in the data map. For recursive operations, performs immediate deletion. For single documents, adds to the delete cache for batch processing.
      Parameters:
      paramMap - the data store parameters
      dataMap - the data map containing the URL to delete
      Returns:
      true if the deletion was processed successfully, false otherwise

    • commit

      public void commit()
      Description copied from interface: IndexUpdateCallback
      Commits all pending index update operations to ensure data persistence. This method should be called after all documents have been stored.
      Specified by:
      commit in interface IndexUpdateCallback

    • deleteDocuments

      protected void deleteDocuments()
      Performs batch deletion of all URLs in the delete cache. Clears the delete URL list after processing.

    • getDocumentSize

      public long getDocumentSize()
      Description copied from interface: IndexUpdateCallback
      Returns the total number of documents processed by this callback.
      Specified by:
      getDocumentSize in interface IndexUpdateCallback
      Returns:
      the document count

    • getExecuteTime

      public long getExecuteTime()
      Description copied from interface: IndexUpdateCallback
      Returns the total execution time for index update operations.
      Specified by:
      getExecuteTime in interface IndexUpdateCallback
      Returns:
      the execution time in milliseconds

    • setMaxDeleteDocumentCacheSize

      public void setMaxDeleteDocumentCacheSize(int maxDeleteDocumentCacheSize)
      Sets the maximum size of the delete document cache.
      Parameters:
      maxDeleteDocumentCacheSize - the maximum cache size before batch deletion is triggered

    • setMaxRedirectCount

      public void setMaxRedirectCount(int maxRedirectCount)
      Sets the maximum number of redirects to follow when processing URLs.
      Parameters:
      maxRedirectCount - the maximum redirect count

    • setExecutorTerminationTimeout

      public void setExecutorTerminationTimeout(int executorTerminationTimeout)
      Sets the timeout for executor service termination during shutdown.
      Parameters:
      executorTerminationTimeout - the timeout in seconds