Package org.codelibs.fess.crawler

Class FessCrawlerThread

java.lang.Object
org.codelibs.fess.crawler.CrawlerThread
org.codelibs.fess.crawler.FessCrawlerThread
All Implemented Interfaces:
Runnable
public class FessCrawlerThread extends org.codelibs.fess.crawler.CrawlerThread

  • Field Summary

    Fields
    Modifier and Type
    Field
    Description
    protected ConcurrentHashMap<String,org.codelibs.core.misc.Pair<String,Pattern>>
    clientRuleCache
     
    protected static final String
    CRAWLER_CLIENTS
     

    Fields inherited from class org.codelibs.fess.crawler.CrawlerThread

    clientFactory, crawlerContainer, crawlerContext, dataService, logHelper, noWaitOnFolder, urlQueueService

  • Constructor Summary

    Constructors
    Constructor
    Description
    FessCrawlerThread()
     

  • Method Summary

    Modifier and Type
    Method
    Description
    protected Set<org.codelibs.fess.crawler.entity.RequestData>
    getAnchorSet(Object obj)
     
    protected Set<org.codelibs.fess.crawler.entity.RequestData>
    getChildUrlSet(SearchEngineClient searchEngineClient, String id)
     
    protected org.codelibs.fess.crawler.client.CrawlerClient
    getClient(String url)
     
    protected List<org.codelibs.core.misc.Pair<String,Pattern>>
    getClientRuleList(String value)
     
    protected boolean
    isContentUpdated(org.codelibs.fess.crawler.client.CrawlerClient client, org.codelibs.fess.crawler.entity.UrlQueue<?> urlQueue)
     
    protected void
    processResponse(org.codelibs.fess.crawler.entity.UrlQueue<?> urlQueue, org.codelibs.fess.crawler.entity.ResponseData responseData)
     
    protected void
    storeChildUrl(String childUrl, String parentUrl, String metaData, int depth)
     
    protected void
    storeChildUrlsToQueue(org.codelibs.fess.crawler.entity.UrlQueue<?> urlQueue, Set<org.codelibs.fess.crawler.entity.RequestData> childUrlSet)
     

    Methods inherited from class org.codelibs.fess.crawler.CrawlerThread

    addSitemapsFromRobotsTxt, finishCrawling, isContinue, isNoWaitOnFolder, isValid, log, run, setClientFactory, setCrawlerContext, setNoWaitOnFolder, startCrawling, storeChildUrls

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Field Details

  • Constructor Details

    • FessCrawlerThread

      public FessCrawlerThread()

  • Method Details

    • isContentUpdated

      protected boolean isContentUpdated(org.codelibs.fess.crawler.client.CrawlerClient client, org.codelibs.fess.crawler.entity.UrlQueue<?> urlQueue)
      Overrides:
      isContentUpdated in class org.codelibs.fess.crawler.CrawlerThread

    • storeChildUrlsToQueue

      protected void storeChildUrlsToQueue(org.codelibs.fess.crawler.entity.UrlQueue<?> urlQueue, Set<org.codelibs.fess.crawler.entity.RequestData> childUrlSet)

    • getAnchorSet

      protected Set<org.codelibs.fess.crawler.entity.RequestData> getAnchorSet(Object obj)

    • getChildUrlSet

      protected Set<org.codelibs.fess.crawler.entity.RequestData> getChildUrlSet(SearchEngineClient searchEngineClient, String id)

    • processResponse

      protected void processResponse(org.codelibs.fess.crawler.entity.UrlQueue<?> urlQueue, org.codelibs.fess.crawler.entity.ResponseData responseData)
      Overrides:
      processResponse in class org.codelibs.fess.crawler.CrawlerThread

    • storeChildUrl

      protected void storeChildUrl(String childUrl, String parentUrl, String metaData, int depth)
      Overrides:
      storeChildUrl in class org.codelibs.fess.crawler.CrawlerThread

    • getClient

      protected org.codelibs.fess.crawler.client.CrawlerClient getClient(String url)
      Overrides:
      getClient in class org.codelibs.fess.crawler.CrawlerThread

    • getClientRuleList

      protected List<org.codelibs.core.misc.Pair<String,Pattern>> getClientRuleList(String value)