Package org.codelibs.fess.crawler

Class FessCrawlerThread

java.lang.Object
org.codelibs.fess.crawler.CrawlerThread
org.codelibs.fess.crawler.FessCrawlerThread
All Implemented Interfaces:
Runnable
public class FessCrawlerThread extends org.codelibs.fess.crawler.CrawlerThread
FessCrawlerThread is a specialized crawler thread implementation for the Fess search engine. This class extends the base CrawlerThread and provides Fess-specific functionality for crawling and indexing documents, including incremental crawling capabilities, content modification checking, and integration with the Fess search engine backend.

Key features include:

  • Incremental crawling support with last-modified timestamp checking
  • Document expiration handling
  • Child URL extraction and queueing
  • Integration with Fess configuration and permission systems
  • Client selection based on URL patterns
See Also:
  • CrawlerThread
  • CrawlerClient

  • Field Summary

    Fields
    Modifier and Type
    Field
    Description
    protected ConcurrentHashMap<String,org.codelibs.core.misc.Pair<String,Pattern>>
    clientRuleCache
    Cache for client rules mapping client names to their corresponding URL patterns.
    protected static final String
    CRAWLER_CLIENTS
    Configuration key for crawler clients used in parameter maps

    Fields inherited from class org.codelibs.fess.crawler.CrawlerThread

    clientFactory, crawlerContainer, crawlerContext, dataService, logHelper, noWaitOnFolder, urlQueueService

  • Constructor Summary

    Constructors
    Constructor
    Description
    FessCrawlerThread()
    Default constructor.

  • Method Summary

    Modifier and Type
    Method
    Description
    protected Set<org.codelibs.fess.crawler.entity.RequestData>
    getAnchorSet(Object obj)
    Extracts anchor URLs from the given object and converts them to RequestData objects.
    protected Set<org.codelibs.fess.crawler.entity.RequestData>
    getChildUrlSet(SearchEngineClient searchEngineClient, String id)
    Retrieves child URLs for a given document ID from the search engine index.
    protected org.codelibs.fess.crawler.client.CrawlerClient
    getClient(String url)
    Retrieves the appropriate crawler client for the given URL based on configured rules.
    protected List<org.codelibs.core.misc.Pair<String,Pattern>>
    getClientRuleList(String value)
    Parses client rule configuration string into a list of client name and pattern pairs.
    protected boolean
    isContentUpdated(org.codelibs.fess.crawler.client.CrawlerClient client, org.codelibs.fess.crawler.entity.UrlQueue<?> urlQueue)
    Determines whether the content at the given URL has been updated since the last crawl.
    protected void
    processResponse(org.codelibs.fess.crawler.entity.UrlQueue<?> urlQueue, org.codelibs.fess.crawler.entity.ResponseData responseData)
    Processes the response data from a crawled URL, including failure handling.
    protected void
    storeChildUrl(String childUrl, String parentUrl, float weight, int depth)
    Stores a child URL in the crawling queue with duplicate host handling.
    protected void
    storeChildUrlsToQueue(org.codelibs.fess.crawler.entity.UrlQueue<?> urlQueue, Set<org.codelibs.fess.crawler.entity.RequestData> childUrlSet)
    Stores child URLs from the given set into the crawling queue for future processing.

    Methods inherited from class org.codelibs.fess.crawler.CrawlerThread

    addSitemapsFromRobotsTxt, finishCrawling, isContinue, isNoWaitOnFolder, isValid, log, run, setClientFactory, setCrawlerContext, setNoWaitOnFolder, startCrawling, storeChildUrls

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Field Details

    • CRAWLER_CLIENTS

      protected static final String CRAWLER_CLIENTS
      Configuration key for crawler clients used in parameter maps
      See Also:

    • clientRuleCache

      protected ConcurrentHashMap<String,org.codelibs.core.misc.Pair<String,Pattern>> clientRuleCache
      Cache for client rules mapping client names to their corresponding URL patterns. This cache improves performance by avoiding repeated parsing of client configuration rules. The key is the rule string, and the value is a pair containing the client name and compiled pattern.

  • Constructor Details

    • FessCrawlerThread

      public FessCrawlerThread()
      Default constructor.

  • Method Details

    • isContentUpdated

      protected boolean isContentUpdated(org.codelibs.fess.crawler.client.CrawlerClient client, org.codelibs.fess.crawler.entity.UrlQueue<?> urlQueue)
      Determines whether the content at the given URL has been updated since the last crawl. This method implements incremental crawling by comparing timestamps and checking document expiration. It also handles special cases for different URL schemes (SMB, file, FTP).
      Overrides:
      isContentUpdated in class org.codelibs.fess.crawler.CrawlerThread
      Parameters:
      client - the crawler client to use for accessing the URL
      urlQueue - the URL queue item containing the URL to check
      Returns:
      true if the content has been updated and should be crawled, false otherwise

    • storeChildUrlsToQueue

      protected void storeChildUrlsToQueue(org.codelibs.fess.crawler.entity.UrlQueue<?> urlQueue, Set<org.codelibs.fess.crawler.entity.RequestData> childUrlSet)
      Stores child URLs from the given set into the crawling queue for future processing. The crawling depth is incremented by 1 from the parent URL's depth.
      Parameters:
      urlQueue - the parent URL queue item
      childUrlSet - the set of child URLs to be queued for crawling

    • getAnchorSet

      protected Set<org.codelibs.fess.crawler.entity.RequestData> getAnchorSet(Object obj)
      Extracts anchor URLs from the given object and converts them to RequestData objects. The input object can be either a single string or a list of strings representing URLs.
      Parameters:
      obj - the object containing anchor URLs (String or List of Strings)
      Returns:
      a set of RequestData objects for the anchor URLs, or null if no valid URLs found

    • getChildUrlSet

      protected Set<org.codelibs.fess.crawler.entity.RequestData> getChildUrlSet(SearchEngineClient searchEngineClient, String id)
      Retrieves child URLs for a given document ID from the search engine index. This method queries the search engine for child documents and extracts their URLs.
      Parameters:
      searchEngineClient - the search engine client to query
      id - the parent document ID to find children for
      Returns:
      a set of RequestData objects for the child URLs, or null if no children found

    • processResponse

      protected void processResponse(org.codelibs.fess.crawler.entity.UrlQueue<?> urlQueue, org.codelibs.fess.crawler.entity.ResponseData responseData)
      Processes the response data from a crawled URL, including failure handling. This method extends the base response processing to handle Fess-specific failure URL tracking when certain HTTP status codes are encountered.
      Overrides:
      processResponse in class org.codelibs.fess.crawler.CrawlerThread
      Parameters:
      urlQueue - the URL queue item that was processed
      responseData - the response data from the crawl operation

    • storeChildUrl

      protected void storeChildUrl(String childUrl, String parentUrl, float weight, int depth)
      Stores a child URL in the crawling queue with duplicate host handling. This method applies duplicate host conversion before storing the URL.
      Overrides:
      storeChildUrl in class org.codelibs.fess.crawler.CrawlerThread
      Parameters:
      childUrl - the child URL to store
      parentUrl - the parent URL that referenced this child URL
      weight - the weight/priority of the child URL
      depth - the crawling depth of the child URL

    • getClient

      protected org.codelibs.fess.crawler.client.CrawlerClient getClient(String url)
      Retrieves the appropriate crawler client for the given URL based on configured rules. This method uses client rules to determine which specific client implementation should be used for crawling the URL, falling back to the default client if no specific rule matches.
      Overrides:
      getClient in class org.codelibs.fess.crawler.CrawlerThread
      Parameters:
      url - the URL to get a client for
      Returns:
      the crawler client instance to use for the URL

    • getClientRuleList

      protected List<org.codelibs.core.misc.Pair<String,Pattern>> getClientRuleList(String value)
      Parses client rule configuration string into a list of client name and pattern pairs. The configuration string format is "clientName:pattern,clientName:pattern,..." Results are cached to improve performance on subsequent calls.
      Parameters:
      value - the client rule configuration string
      Returns:
      a list of pairs containing client names and their corresponding compiled patterns