Package org.codelibs.fess.crawler.service

Class FessUrlQueueService

java.lang.Object
org.codelibs.fess.crawler.service.impl.AbstractCrawlerService
org.codelibs.fess.crawler.service.impl.OpenSearchUrlQueueService
org.codelibs.fess.crawler.service.FessUrlQueueService
All Implemented Interfaces:
org.codelibs.fess.crawler.service.UrlQueueService<org.codelibs.fess.crawler.entity.OpenSearchUrlQueue>
public class FessUrlQueueService extends org.codelibs.fess.crawler.service.impl.OpenSearchUrlQueueService
Fess-specific URL queue service that extends OpenSearch URL queue functionality. This service provides customized URL fetching strategies including sequential and random ordering based on crawling configuration parameters.

  • Nested Class Summary

    Nested classes/interfaces inherited from class org.codelibs.fess.crawler.service.impl.OpenSearchUrlQueueService

    org.codelibs.fess.crawler.service.impl.OpenSearchUrlQueueService.QueueHolder

    Nested classes/interfaces inherited from class org.codelibs.fess.crawler.service.impl.AbstractCrawlerService

    org.codelibs.fess.crawler.service.impl.AbstractCrawlerService.EsTimestampConverter

  • Field Summary

    Fields
    Modifier and Type
    Field
    Description
    protected static final String
    ORDER_RANDOM
    Configuration value for random URL processing order
    protected static final String
    ORDER_SEQUENTIAL
    Configuration value for sequential URL processing order

    Fields inherited from class org.codelibs.fess.crawler.service.impl.OpenSearchUrlQueueService

    dataService, maxCrawlingQueueSize, pollingFetchSize, sessionCache

    Fields inherited from class org.codelibs.fess.crawler.service.impl.AbstractCrawlerService

    _DOC, bulkBufferSize, CREATE_TIME, fesenClient, ID, idPrefixLength, index, LAST_MODIFIED, murmur3Hash, numberOfReplicas, numberOfShards, scrollSize, scrollTimeout, SESSION_ID, timestampFields, URL

  • Constructor Summary

    Constructors
    Constructor
    Description
    FessUrlQueueService(org.codelibs.fess.crawler.util.OpenSearchCrawlerConfig crawlerConfig)
    Constructs a new FessUrlQueueService with the specified crawler configuration.

  • Method Summary

    Modifier and Type
    Method
    Description
    protected List<org.codelibs.fess.crawler.entity.OpenSearchUrlQueue>
    fetchUrlQueueList(String sessionId)
    Fetches URL queue list for the specified session with configurable ordering strategy.

    Methods inherited from class org.codelibs.fess.crawler.service.impl.OpenSearchUrlQueueService

    add, clearCache, delete, destroy, exists, generateUrlQueues, getQueueHolder, init, insert, offerAll, poll, saveSession, setMaxCrawlingQueueSize, setPollingFetchSize, updateSessionId, visited

    Methods inherited from class org.codelibs.fess.crawler.service.impl.AbstractCrawlerService

    buildFailureMessage, createMapping, delete, delete, deleteAll, deleteBySessionId, doInsertAll, get, getBulkBufferSize, getClient, getCount, getDateFromSource, getIndex, getList, getList, getScrollSize, getScrollTimeout, getXContentBuilder, insert, insertAll, insertAll, refresh, setBulkBufferSize, setId, setIdPrefixLength, setIndex, setNumberOfReplicas, setNumberOfShards, setScrollSize, setScrollTimeout

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    Methods inherited from interface org.codelibs.fess.crawler.service.UrlQueueService

    deleteAll

  • Field Details

    • ORDER_SEQUENTIAL

      protected static final String ORDER_SEQUENTIAL
      Configuration value for sequential URL processing order
      See Also:

    • ORDER_RANDOM

      protected static final String ORDER_RANDOM
      Configuration value for random URL processing order
      See Also:

  • Constructor Details

    • FessUrlQueueService

      public FessUrlQueueService(org.codelibs.fess.crawler.util.OpenSearchCrawlerConfig crawlerConfig)
      Constructs a new FessUrlQueueService with the specified crawler configuration.
      Parameters:
      crawlerConfig - the OpenSearch crawler configuration

  • Method Details

    • fetchUrlQueueList

      protected List<org.codelibs.fess.crawler.entity.OpenSearchUrlQueue> fetchUrlQueueList(String sessionId)
      Fetches URL queue list for the specified session with configurable ordering strategy. Supports sequential (default) and random ordering based on crawling configuration.
      Overrides:
      fetchUrlQueueList in class org.codelibs.fess.crawler.service.impl.OpenSearchUrlQueueService
      Parameters:
      sessionId - the crawling session identifier
      Returns:
      list of URL queue entries for processing