Package org.codelibs.fess.helper

Class CrawlerStatsHelper

java.lang.Object
org.codelibs.fess.helper.CrawlerStatsHelper
public class CrawlerStatsHelper extends Object
Helper class for managing crawler statistics and performance metrics. This class provides functionality to track, record, and report statistics about crawler operations including timing data, performance metrics, and operational events. It uses an internal cache to maintain statistics objects and provides methods to begin tracking, record events, and finalize statistics collection.

  • Nested Class Summary

    Nested Classes
    Modifier and Type
    Class
    Description
    static enum 
    CrawlerStatsHelper.StatsAction
    Enumeration of predefined statistics actions that can be recorded during crawler operations.
    static class 
    CrawlerStatsHelper.StatsKeyObject
    Key object for statistics tracking that contains an identifier and optional URL.
    static class 
    CrawlerStatsHelper.StatsObject
    Statistics data object that stores timestamped events and maintains reference counting.

  • Field Summary

    Fields
    Modifier and Type
    Field
    Description
    protected long
    cacheExpireAfterWrite
    Time in milliseconds after which cache entries expire after write.
    protected String
    loggerName
    Name of the logger used for statistics output.
    protected long
    maxCacheSize
    Maximum number of statistics objects to cache.
    protected com.google.common.cache.LoadingCache<String,CrawlerStatsHelper.StatsObject>
    statsCache
    Cache for storing statistics objects keyed by crawler object identifiers.
    protected org.apache.logging.log4j.Logger
    statsLogger
    Logger instance specifically for outputting crawler statistics.

  • Constructor Summary

    Constructors
    Constructor
    Description
    CrawlerStatsHelper()
    Creates a new instance of CrawlerStatsHelper.

  • Method Summary

    Modifier and Type
    Method
    Description
    void
    begin(Object keyObj)
    Begins statistics tracking for the specified crawler object.
    void
    destroy()
    Cleanup method called when the helper is being destroyed.
    void
    discard(Object keyObj)
    Discards statistics tracking for the specified crawler object.
    void
    done(Object keyObj)
    Marks statistics tracking as complete for the specified crawler object.
    protected String
    escapeValue(String action)
    Escapes special characters in a value string for safe logging.
    protected org.dbflute.optional.OptionalThing<String>
    getCacheKey(Object keyObj)
    Generates a cache key from a crawler object.
    protected long
    getCurrentTimeMillis()
    Gets the current system time in milliseconds.
    protected String
    getUrl(Object keyObj)
    Extracts the URL string from a crawler object for logging purposes.
    void
    init()
    Initializes the crawler statistics helper.
    protected void
    log(StringBuilder buf)
    Outputs a log message using the statistics logger.
    protected void
    printStats(Object keyObj, CrawlerStatsHelper.StatsObject data, long begin, boolean done)
    Outputs statistics information for a crawler object.
    void
    record(Object keyObj, String action)
    Records a custom statistics action for the specified crawler object.
    void
    record(Object keyObj, CrawlerStatsHelper.StatsAction action)
    Records a statistics action for the specified crawler object.
    void
    runOnThread(Object keyObj)
    Increments the thread reference count for the specified crawler object.
    void
    setCacheExpireAfterWrite(long cacheExpireAfterWrite)
    Sets the cache expiration time after write in milliseconds.
    void
    setLoggerName(String loggerName)
    Sets the name of the logger used for statistics output.
    void
    setMaxCacheSize(long maxCacheSize)
    Sets the maximum number of statistics objects to cache.

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Field Details

    • statsLogger

      protected org.apache.logging.log4j.Logger statsLogger
      Logger instance specifically for outputting crawler statistics.

    • loggerName

      protected String loggerName
      Name of the logger used for statistics output.

    • maxCacheSize

      protected long maxCacheSize
      Maximum number of statistics objects to cache.

    • cacheExpireAfterWrite

      protected long cacheExpireAfterWrite
      Time in milliseconds after which cache entries expire after write.

    • statsCache

      protected com.google.common.cache.LoadingCache<String,CrawlerStatsHelper.StatsObject> statsCache
      Cache for storing statistics objects keyed by crawler object identifiers.

  • Constructor Details

    • CrawlerStatsHelper

      public CrawlerStatsHelper()
      Creates a new instance of CrawlerStatsHelper.

  • Method Details

    • init

      @PostConstruct public void init()
      Initializes the crawler statistics helper. Sets up the statistics logger and creates the cache for storing statistics objects with the configured size and expiration settings.

    • destroy

      @PreDestroy public void destroy()
      Cleanup method called when the helper is being destroyed. Logs cache statistics and processes any remaining statistics objects in the cache before shutdown.

    • begin

      public void begin(Object keyObj)
      Begins statistics tracking for the specified crawler object. Creates a new statistics object in the cache and starts timing.
      Parameters:
      keyObj - the crawler object to track (UrlQueue, StatsKeyObject, String, or Number)

    • record

      public void record(Object keyObj, CrawlerStatsHelper.StatsAction action)
      Records a statistics action for the specified crawler object.
      Parameters:
      keyObj - the crawler object being tracked
      action - the statistics action to record

    • record

      public void record(Object keyObj, String action)
      Records a custom statistics action for the specified crawler object.
      Parameters:
      keyObj - the crawler object being tracked
      action - the custom action name to record

    • done

      public void done(Object keyObj)
      Marks statistics tracking as complete for the specified crawler object. Decrements the reference count and if it reaches zero, removes the statistics object from cache and outputs the final statistics.
      Parameters:
      keyObj - the crawler object to complete tracking for

    • discard

      public void discard(Object keyObj)
      Discards statistics tracking for the specified crawler object. Removes the statistics object from cache without outputting statistics.
      Parameters:
      keyObj - the crawler object to discard tracking for

    • printStats

      protected void printStats(Object keyObj, CrawlerStatsHelper.StatsObject data, long begin, boolean done)
      Outputs statistics information for a crawler object.
      Parameters:
      keyObj - the crawler object the statistics relate to
      data - the statistics data to output
      begin - the timestamp when tracking began
      done - whether tracking was completed normally

    • runOnThread

      public void runOnThread(Object keyObj)
      Increments the thread reference count for the specified crawler object. Used when the same object is being processed on multiple threads.
      Parameters:
      keyObj - the crawler object running on an additional thread

    • getCurrentTimeMillis

      protected long getCurrentTimeMillis()
      Gets the current system time in milliseconds.
      Returns:
      current time in milliseconds

    • getUrl

      protected String getUrl(Object keyObj)
      Extracts the URL string from a crawler object for logging purposes.
      Parameters:
      keyObj - the crawler object to extract URL from
      Returns:
      the URL string or a default value if not extractable

    • getCacheKey

      protected org.dbflute.optional.OptionalThing<String> getCacheKey(Object keyObj)
      Generates a cache key from a crawler object.
      Parameters:
      keyObj - the crawler object to generate key for
      Returns:
      Optional cache key string, empty if object type not supported

    • escapeValue

      protected String escapeValue(String action)
      Escapes special characters in a value string for safe logging.
      Parameters:
      action - the string value to escape
      Returns:
      the escaped string with tabs replaced by spaces

    • log

      protected void log(StringBuilder buf)
      Outputs a log message using the statistics logger.
      Parameters:
      buf - the string buffer containing the log message

    • setLoggerName

      public void setLoggerName(String loggerName)
      Sets the name of the logger used for statistics output.
      Parameters:
      loggerName - the logger name to use

    • setMaxCacheSize

      public void setMaxCacheSize(long maxCacheSize)
      Sets the maximum number of statistics objects to cache.
      Parameters:
      maxCacheSize - the maximum cache size

    • setCacheExpireAfterWrite

      public void setCacheExpireAfterWrite(long cacheExpireAfterWrite)
      Sets the cache expiration time after write in milliseconds.
      Parameters:
      cacheExpireAfterWrite - the expiration time in milliseconds