Package org.codelibs.fess.job

Class CrawlJob

java.lang.Object
org.codelibs.fess.job.ExecJob
org.codelibs.fess.job.CrawlJob
public class CrawlJob extends ExecJob
CrawlJob is responsible for executing the crawling process in Fess. This job launches a separate crawler process that can crawl web sites, file systems, and data sources based on the configured crawling settings.

The job supports selective crawling by specifying configuration IDs for different types of crawlers (web, file, data). It manages the crawler process lifecycle, handles timeout scenarios, and ensures proper cleanup of resources.

Key features:

  • Concurrent crawler process management with configurable limits
  • Selective crawling based on configuration IDs
  • Document expiration handling
  • Hot thread monitoring for performance analysis
  • Process isolation with separate JVM

  • Field Details

    • namespace

      protected String namespace
      The namespace identifier for the crawling session. Used to organize and identify crawling activities in the system. Defaults to the system crawling info name.

    • webConfigIds

      protected String[] webConfigIds
      Array of web crawling configuration IDs to process. If null, all available web configurations will be crawled (when no other config IDs are specified).

    • fileConfigIds

      protected String[] fileConfigIds
      Array of file system crawling configuration IDs to process. If null, all available file configurations will be crawled (when no other config IDs are specified).

    • dataConfigIds

      protected String[] dataConfigIds
      Array of data source crawling configuration IDs to process. If null, all available data configurations will be crawled (when no other config IDs are specified).

    • documentExpires

      protected int documentExpires
      Document expiration setting in days. -2: use system default, -1: never expire, 0 or positive: expire after specified days.

    • hotThreadInterval

      protected int hotThreadInterval
      Hot thread monitoring interval in seconds. -1: disabled, positive value: enable hot thread monitoring with specified interval. Used for performance analysis and debugging of the crawler process.

  • Constructor Details

    • CrawlJob

      public CrawlJob()
      Default constructor for CrawlJob. Initializes the job with default settings.

  • Method Details

    • namespace

      public CrawlJob namespace(String namespace)
      Sets the namespace for the crawling session. The namespace is used to organize and identify crawling activities.
      Parameters:
      namespace - the namespace identifier for the crawling session
      Returns:
      this CrawlJob instance for method chaining

    • documentExpires

      public CrawlJob documentExpires(int documentExpires)
      Sets the document expiration period in days. Controls how long crawled documents remain in the search index.
      Parameters:
      documentExpires - the expiration period: -2 (system default), -1 (never expire), 0 or positive (expire after specified days)
      Returns:
      this CrawlJob instance for method chaining

    • webConfigIds

      public CrawlJob webConfigIds(String[] webConfigIds)
      Sets the web crawling configuration IDs to process. If not set, all available web configurations will be crawled.
      Parameters:
      webConfigIds - array of web crawling configuration IDs, or null for all
      Returns:
      this CrawlJob instance for method chaining

    • fileConfigIds

      public CrawlJob fileConfigIds(String[] fileConfigIds)
      Sets the file system crawling configuration IDs to process. If not set, all available file configurations will be crawled.
      Parameters:
      fileConfigIds - array of file crawling configuration IDs, or null for all
      Returns:
      this CrawlJob instance for method chaining

    • dataConfigIds

      public CrawlJob dataConfigIds(String[] dataConfigIds)
      Sets the data source crawling configuration IDs to process. If not set, all available data configurations will be crawled.
      Parameters:
      dataConfigIds - array of data crawling configuration IDs, or null for all
      Returns:
      this CrawlJob instance for method chaining

    • hotThread

      public CrawlJob hotThread(int hotThreadInterval)
      Sets the hot thread monitoring interval for performance analysis. Hot threads help identify performance bottlenecks in the crawler process.
      Parameters:
      hotThreadInterval - monitoring interval in seconds, -1 to disable
      Returns:
      this CrawlJob instance for method chaining

    • execute

      public String execute()
      Description copied from class: ExecJob
      Executes the job and returns the result as a string. This method contains the main business logic for the specific job implementation.
      Specified by:
      execute in class ExecJob
      Returns:
      the execution result message or summary

    • getRunningJobCount

      protected int getRunningJobCount()
      Gets the count of currently running crawler jobs. This method queries the scheduled jobs to count active crawler processes. Used to enforce maximum concurrent crawler limits.
      Returns:
      the number of currently running crawler jobs

    • executeCrawler

      protected void executeCrawler()
      Executes the crawler process in a separate JVM. This method constructs the command line arguments, sets up the classpath, and launches the crawler as an external process. It handles process lifecycle, monitors output, and ensures proper cleanup.
      Throws:
      JobProcessingException - if the crawler process fails or times out

    • getExecuteType

      protected String getExecuteType()
      Description copied from class: ExecJob
      Returns the execution type identifier for this job. This type is used for classpath construction, configuration, and logging purposes.
      Specified by:
      getExecuteType in class ExecJob
      Returns:
      the execution type (e.g., "crawler", "suggest", etc.)