Package org.codelibs.fess.exec

Class Crawler

java.lang.Object
org.codelibs.fess.exec.Crawler
public class Crawler extends Object
Main executable class for running crawling operations in the Fess search engine. This class serves as the entry point for crawling web content, file systems, and data stores. It manages the crawling lifecycle, including initialization, execution coordination, monitoring, and cleanup operations.

The crawler can operate in different modes based on command-line options:

  • Web crawling - crawls web sites and web content
  • File system crawling - crawls file systems and documents
  • Data store crawling - crawls databases and other data sources
  • Combined crawling - runs multiple crawling types simultaneously

Command line usage: 

 java org.codelibs.fess.exec.Crawler [options...]
   -s, --sessionId sessionId     : Session ID for the crawling session
   -n, --name name               : Name for the crawling session
   -w, --webConfigIds ids        : Comma-separated web config IDs
   -f, --fileConfigIds ids       : Comma-separated file config IDs
   -d, --dataConfigIds ids       : Comma-separated data config IDs
   -p, --properties path         : Properties file path
   -e, --expires days            : Expires for documents (in days)
   -h, --hotThread interval      : Interval for hot thread logging

  • Field Details

    • searchEngineClient

      protected SearchEngineClient searchEngineClient
      Injected search engine client for OpenSearch operations.

    • webFsIndexHelper

      protected WebFsIndexHelper webFsIndexHelper
      Injected helper for web and file system indexing operations.

    • dataIndexHelper

      protected DataIndexHelper dataIndexHelper
      Injected helper for data store indexing operations.

    • pathMappingService

      protected PathMappingService pathMappingService
      Injected service for managing path mappings during crawling.

    • crawlingInfoService

      protected CrawlingInfoService crawlingInfoService
      Injected service for managing crawling session information.

  • Constructor Details

    • Crawler

      public Crawler()
      Creates a new instance of Crawler.

  • Method Details

    • addError

      public static void addError(String msg)
      Adds an error message to the error queue for later processing. This method is thread-safe and can be called from multiple crawler threads.
      Parameters:
      msg - the error message to add; ignored if null or blank

    • main

      public static void main(String[] args)
      Main entry point for the crawler application. Parses command-line arguments, initializes the application container, sets up monitoring, and executes the crawling process.
      Parameters:
      args - command-line arguments as defined in the Options class

    • sendMail

      protected void sendMail(Map<String,String> infoMap)
      Sends email notification with crawling results and statistics. The email contains detailed information about the crawling session including execution times, index sizes, and status information.
      Parameters:
      infoMap - map containing crawling session information and statistics

    • doCrawl

      public int doCrawl(Crawler.Options options)
      Executes the actual crawling operations based on the provided options. This method coordinates web/file system crawling and data store crawling, running them in parallel threads when multiple types are requested.
      Parameters:
      options - crawling configuration options
      Returns:
      exit code (Constants.EXIT_OK for success, Constants.EXIT_FAIL for failure)

    • writeTimeToSessionInfo

      protected void writeTimeToSessionInfo(CrawlingInfoHelper crawlingInfoHelper, String key)
      Writes the current timestamp to the crawling session information. The timestamp is formatted in ISO 8601 extended format.
      Parameters:
      crawlingInfoHelper - helper for managing crawling session information
      key - the key under which to store the timestamp