Package org.codelibs.fess.helper

Class CrawlingConfigHelper

java.lang.Object
org.codelibs.fess.helper.CrawlingConfigHelper
public class CrawlingConfigHelper extends Object
Helper class for managing crawling configurations. Provides functionality to store, retrieve, and manage different types of crawling configurations including web, file, and data configurations. Supports caching and session-based configuration management.

  • Field Summary

    Fields
    Modifier and Type
    Field
    Description
    protected int
    count
    Counter for generating unique session identifiers.
    protected com.google.common.cache.Cache<String,org.codelibs.fess.opensearch.config.exentity.CrawlingConfig>
    crawlingConfigCache
    Cache for storing crawling configurations to improve performance.
    protected final Map<String,org.codelibs.fess.opensearch.config.exentity.CrawlingConfig>
    crawlingConfigMap
    Map storing crawling configurations by session ID.

  • Constructor Summary

    Constructors
    Constructor
    Description
    CrawlingConfigHelper()
    Creates a new instance of CrawlingConfigHelper.

  • Method Summary

    Modifier and Type
    Method
    Description
    org.codelibs.fess.opensearch.config.exentity.CrawlingConfig
    get(String sessionCountId)
    Retrieves a stored crawling configuration from the session-based storage.
    List<org.codelibs.fess.opensearch.config.exentity.DataConfig>
    getAllDataConfigList()
    Retrieves all available data crawling configurations.
    List<org.codelibs.fess.opensearch.config.exentity.DataConfig>
    getAllDataConfigList(boolean withLabelType, boolean withRoleType, boolean available, List<String> idList)
    Retrieves data crawling configurations with various filtering options.
    List<org.codelibs.fess.opensearch.config.exentity.FileConfig>
    getAllFileConfigList()
    Retrieves all available file crawling configurations.
    List<org.codelibs.fess.opensearch.config.exentity.FileConfig>
    getAllFileConfigList(boolean withLabelType, boolean withRoleType, boolean available, List<String> idList)
    Retrieves file crawling configurations with various filtering options.
    List<org.codelibs.fess.opensearch.config.exentity.WebConfig>
    getAllWebConfigList()
    Retrieves all available web crawling configurations.
    List<org.codelibs.fess.opensearch.config.exentity.WebConfig>
    getAllWebConfigList(boolean withLabelType, boolean withRoleType, boolean available, List<String> idList)
    Retrieves web crawling configurations with various filtering options.
    org.codelibs.fess.opensearch.config.exentity.CrawlingConfig.ConfigType
    getConfigType(String configId)
    Determines the configuration type from a given config ID.
    org.codelibs.fess.opensearch.config.exentity.CrawlingConfig
    getCrawlingConfig(String configId)
    Retrieves a crawling configuration by its config ID.
    List<org.codelibs.fess.opensearch.config.exentity.DataConfig>
    getDataConfigListByIds(List<String> idList)
    Retrieves data crawling configurations filtered by a list of IDs.
    org.dbflute.optional.OptionalEntity<org.codelibs.fess.opensearch.config.exentity.CrawlingConfig>
    getDefaultConfig(org.codelibs.fess.opensearch.config.exentity.CrawlingConfig.ConfigType configType)
    Retrieves the default crawling configuration template for a given configuration type.
    List<String>
    getExcludedUrlList(String configId)
    Retrieves a list of URLs that should be excluded from crawling based on failure counts.
    List<org.codelibs.fess.opensearch.config.exentity.FileConfig>
    getFileConfigListByIds(List<String> idList)
    Retrieves file crawling configurations filtered by a list of IDs.
    protected String
    getId(String configId)
    Extracts the actual ID from a config ID by removing the type prefix.
    org.dbflute.optional.OptionalThing<String>
    getPipeline(String configId)
    Retrieves the pipeline configuration parameter for a given config ID.
    List<org.codelibs.fess.opensearch.config.exentity.WebConfig>
    getWebConfigListByIds(List<String> idList)
    Retrieves web crawling configurations filtered by a list of IDs.
    void
    init()
    Initializes the CrawlingConfigHelper by setting up the crawling configuration cache.
    void
    refresh()
    Refreshes the crawling configuration cache by invalidating all cached entries.
    void
    remove(String sessionCountId)
    Removes a stored crawling configuration from the session-based storage.
    String
    store(String sessionId, org.codelibs.fess.opensearch.config.exentity.CrawlingConfig crawlingConfig)
    Stores a crawling configuration in the session-based storage with a unique identifier.

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Field Details

    • crawlingConfigMap

      protected final Map<String,org.codelibs.fess.opensearch.config.exentity.CrawlingConfig> crawlingConfigMap
      Map storing crawling configurations by session ID.

    • count

      protected int count
      Counter for generating unique session identifiers.

    • crawlingConfigCache

      protected com.google.common.cache.Cache<String,org.codelibs.fess.opensearch.config.exentity.CrawlingConfig> crawlingConfigCache
      Cache for storing crawling configurations to improve performance.

  • Constructor Details

    • CrawlingConfigHelper

      public CrawlingConfigHelper()
      Creates a new instance of CrawlingConfigHelper.

  • Method Details

    • init

      @PostConstruct public void init()
      Initializes the CrawlingConfigHelper by setting up the crawling configuration cache. This method is called automatically after the bean construction is complete. The cache is configured with a maximum size of 100 entries and expires after 10 minutes.

    • getConfigType

      public org.codelibs.fess.opensearch.config.exentity.CrawlingConfig.ConfigType getConfigType(String configId)
      Determines the configuration type from a given config ID. The config type is identified by the first character of the config ID.
      Parameters:
      configId - the configuration ID to analyze
      Returns:
      the ConfigType (WEB, FILE, or DATA) or null if the config ID is invalid or doesn't match any known type

    • getId

      protected String getId(String configId)
      Extracts the actual ID from a config ID by removing the type prefix. Config IDs are formatted as [type_prefix][actual_id], so this method returns everything after the first character.
      Parameters:
      configId - the configuration ID to process
      Returns:
      the actual ID without the type prefix, or null if the config ID is invalid

    • getCrawlingConfig

      public org.codelibs.fess.opensearch.config.exentity.CrawlingConfig getCrawlingConfig(String configId)
      Retrieves a crawling configuration by its config ID. This method uses caching to improve performance and automatically determines the configuration type and delegates to the appropriate service.
      Parameters:
      configId - the configuration ID to retrieve
      Returns:
      the CrawlingConfig object or null if not found or on error

    • getPipeline

      public org.dbflute.optional.OptionalThing<String> getPipeline(String configId)
      Retrieves the pipeline configuration parameter for a given config ID. The pipeline parameter is extracted from the crawling configuration's parameter map.
      Parameters:
      configId - the configuration ID to get the pipeline for
      Returns:
      an OptionalThing containing the pipeline string if found, or empty if not found or blank

    • refresh

      public void refresh()
      Refreshes the crawling configuration cache by invalidating all cached entries. This forces the next access to reload configurations from the underlying services.

    • store

      public String store(String sessionId, org.codelibs.fess.opensearch.config.exentity.CrawlingConfig crawlingConfig)
      Stores a crawling configuration in the session-based storage with a unique identifier. The generated session count ID combines the session ID with an incrementing counter.
      Parameters:
      sessionId - the session identifier
      crawlingConfig - the crawling configuration to store
      Returns:
      the unique session count ID that can be used to retrieve the stored configuration

    • remove

      public void remove(String sessionCountId)
      Removes a stored crawling configuration from the session-based storage.
      Parameters:
      sessionCountId - the session count ID of the configuration to remove

    • get

      public org.codelibs.fess.opensearch.config.exentity.CrawlingConfig get(String sessionCountId)
      Retrieves a stored crawling configuration from the session-based storage.
      Parameters:
      sessionCountId - the session count ID of the configuration to retrieve
      Returns:
      the stored CrawlingConfig or null if not found

    • getAllWebConfigList

      public List<org.codelibs.fess.opensearch.config.exentity.WebConfig> getAllWebConfigList()
      Retrieves all available web crawling configurations. This is a convenience method that calls the overloaded version with default parameters (withLabelType=true, withRoleType=true, available=true, idList=null).
      Returns:
      a list of all available WebConfig objects

    • getWebConfigListByIds

      public List<org.codelibs.fess.opensearch.config.exentity.WebConfig> getWebConfigListByIds(List<String> idList)
      Retrieves web crawling configurations filtered by a list of IDs. If the ID list is null, returns all available configurations.
      Parameters:
      idList - the list of configuration IDs to retrieve, or null for all configurations
      Returns:
      a list of WebConfig objects with the specified IDs

    • getAllWebConfigList

      public List<org.codelibs.fess.opensearch.config.exentity.WebConfig> getAllWebConfigList(boolean withLabelType, boolean withRoleType, boolean available, List<String> idList)
      Retrieves web crawling configurations with various filtering options.
      Parameters:
      withLabelType - whether to include label type information (currently not used in implementation)
      withRoleType - whether to include role type information (currently not used in implementation)
      available - whether to filter only available configurations
      idList - the list of configuration IDs to retrieve, or null for no ID filtering
      Returns:
      a list of WebConfig objects matching the criteria

    • getAllFileConfigList

      public List<org.codelibs.fess.opensearch.config.exentity.FileConfig> getAllFileConfigList()
      Retrieves all available file crawling configurations. This is a convenience method that calls the overloaded version with default parameters (withLabelType=true, withRoleType=true, available=true, idList=null).
      Returns:
      a list of all available FileConfig objects

    • getFileConfigListByIds

      public List<org.codelibs.fess.opensearch.config.exentity.FileConfig> getFileConfigListByIds(List<String> idList)
      Retrieves file crawling configurations filtered by a list of IDs. If the ID list is null, returns all available configurations.
      Parameters:
      idList - the list of configuration IDs to retrieve, or null for all configurations
      Returns:
      a list of FileConfig objects with the specified IDs

    • getAllFileConfigList

      public List<org.codelibs.fess.opensearch.config.exentity.FileConfig> getAllFileConfigList(boolean withLabelType, boolean withRoleType, boolean available, List<String> idList)
      Retrieves file crawling configurations with various filtering options.
      Parameters:
      withLabelType - whether to include label type information (currently not used in implementation)
      withRoleType - whether to include role type information (currently not used in implementation)
      available - whether to filter only available configurations
      idList - the list of configuration IDs to retrieve, or null for no ID filtering
      Returns:
      a list of FileConfig objects matching the criteria

    • getAllDataConfigList

      public List<org.codelibs.fess.opensearch.config.exentity.DataConfig> getAllDataConfigList()
      Retrieves all available data crawling configurations. This is a convenience method that calls the overloaded version with default parameters (withLabelType=true, withRoleType=true, available=true, idList=null).
      Returns:
      a list of all available DataConfig objects

    • getDataConfigListByIds

      public List<org.codelibs.fess.opensearch.config.exentity.DataConfig> getDataConfigListByIds(List<String> idList)
      Retrieves data crawling configurations filtered by a list of IDs. If the ID list is null, returns all available configurations.
      Parameters:
      idList - the list of configuration IDs to retrieve, or null for all configurations
      Returns:
      a list of DataConfig objects with the specified IDs

    • getAllDataConfigList

      public List<org.codelibs.fess.opensearch.config.exentity.DataConfig> getAllDataConfigList(boolean withLabelType, boolean withRoleType, boolean available, List<String> idList)
      Retrieves data crawling configurations with various filtering options.
      Parameters:
      withLabelType - whether to include label type information (currently not used in implementation)
      withRoleType - whether to include role type information (currently not used in implementation)
      available - whether to filter only available configurations
      idList - the list of configuration IDs to retrieve, or null for no ID filtering
      Returns:
      a list of DataConfig objects matching the criteria

    • getExcludedUrlList

      public List<String> getExcludedUrlList(String configId)
      Retrieves a list of URLs that should be excluded from crawling based on failure counts. URLs are excluded if they have failed more than the configured failure count threshold. URLs can also be filtered by failure type using a regular expression pattern.
      Parameters:
      configId - the configuration ID to get excluded URLs for
      Returns:
      a list of URLs that should be excluded from crawling, or an empty list if none

    • getDefaultConfig

      public org.dbflute.optional.OptionalEntity<org.codelibs.fess.opensearch.config.exentity.CrawlingConfig> getDefaultConfig(org.codelibs.fess.opensearch.config.exentity.CrawlingConfig.ConfigType configType)
      Retrieves the default crawling configuration template for a given configuration type. The default template is identified by the configured form admin default template name.
      Parameters:
      configType - the type of configuration (WEB, FILE, or DATA)
      Returns:
      an OptionalEntity containing the default CrawlingConfig if found, or empty if not found or configType is null