Rate Limiting Configuration

Overview

Fess includes rate limiting features to maintain system stability and performance. These features protect the system from excessive requests and enable fair resource allocation.

Rate limiting is applied in the following scenarios:

  • All HTTP requests including search API and AI mode API (via RateLimitFilter)

  • Crawler requests (via crawl configuration)

HTTP Request Rate Limiting

You can limit the number of HTTP requests to Fess per IP address. This applies to all HTTP requests including search API, AI mode API, admin pages, etc.

Configuration

app/WEB-INF/classes/fess_config.properties:

# Enable rate limiting (default: false)
rate.limit.enabled=true

# Maximum requests per window (default: 100)
rate.limit.requests.per.window=100

# Window size (milliseconds) (default: 60000)
rate.limit.window.ms=60000

Behavior

  • Requests exceeding the rate limit return HTTP 429 (Too Many Requests)

  • Requests from IPs blocked for exceeding the limit, and from IPs on the blocked IP list, return HTTP 403 (Forbidden)

  • The response body is JSON ({"error":"rate_limit_exceeded",...,"retry_after":60} for 429, {"error":"ip_blocked",...} for 403). A 429 response has a Retry-After header

  • Limits are applied per IP address

  • For each IP, the counting window starts from the first request and resets after the window period expires (fixed window algorithm)

  • When the limit is exceeded, the IP is blocked for the duration specified by rate.limit.block.duration.ms

AI Mode Rate Limiting

AI mode functionality has rate limiting to control LLM API costs and resource consumption. In addition to the HTTP request rate limiting above, AI mode has its own specific rate limit settings.

For AI mode specific rate limiting settings, see AI Mode Configuration.

Note

AI mode rate limiting is applied separately from the LLM provider’s rate limiting. Configure both limits accordingly.

Crawler Rate Limiting

You can configure request intervals to prevent the crawler from overloading target sites.

Web Crawl Configuration

Configure the following in Admin Panel under “Crawler” -> “Web”:

  • Request Interval: Wait time between requests (milliseconds)

  • Thread Count: Number of parallel crawling threads

Recommended settings:

# General sites
intervalTime=1000
numOfThread=1

# Large sites (with permission)
intervalTime=500
numOfThread=3

Respecting robots.txt

Fess respects the Crawl-delay directive in robots.txt by default.

# robots.txt example
User-agent: *
Crawl-delay: 10

robots.txt handling is controlled by crawler.ignore.robots.txt in app/WEB-INF/classes/fess_config.properties (default: false). Setting it to true disables robots.txt handling, including Crawl-delay.

# Ignore robots.txt (default: false)
crawler.ignore.robots.txt=false

Crawl-delay applies per origin and is capped at 60 seconds. It paces URLs, not requests, so in an incremental crawl the HEAD and the GET of one URL are sent back to back. The interval of a crawl configuration still works separately, as the wait before the next URL.

robots.txt is interpreted according to RFC 9309. Between Allow: and Disallow:, the longest matching rule wins, and the start URLs are checked against robots.txt too. A URL that robots.txt disallows is logged at INFO in fess-crawler.log and is not recorded as a failure URL.

Backoff after 429/503 responses

When a server answers 429 Too Many Requests or 503 Service Unavailable, Fess pauses requests to that origin and retries the URL up to three times. The wait is the Retry-After header when it is present, otherwise an exponential wait that starts at 10 seconds (up to 5 minutes). When the last retry fails too, a WARN is written to fess-crawler.log.

When robots.txt cannot be fetched

When fetching robots.txt fails with a 5xx, a 429 or a timeout, the URLs of that origin are put back into the queue until the backoff ends. If three retries after the first failure also fail, no URL of that origin is crawled for the rest of the crawl, and one WARN is written to fess-crawler.log. Up to 15.8, a robots.txt that could not be fetched meant “allow all”. To restore that behavior, set the following in the “Config Parameters” of the web crawl configuration:

client.robotsTxtAllowOnUnavailable=true

To turn robots.txt handling off entirely, use client.robotsTxtEnabled=false (per crawl configuration) or crawler.ignore.robots.txt=true.

All Rate Limiting Properties

All properties configurable in app/WEB-INF/classes/fess_config.properties.

Property Description Default
rate.limit.enabled Enable rate limiting false
rate.limit.requests.per.window Maximum requests per window 100
rate.limit.window.ms Window size (milliseconds) 60000
rate.limit.block.duration.ms IP block duration when limit is exceeded (milliseconds) 300000
rate.limit.retry.after.seconds Retry-After header value (seconds) 60
rate.limit.whitelist.ips IP addresses excluded from rate limiting (comma-separated) 127.0.0.1,::1
rate.limit.blocked.ips IP addresses to block (comma-separated) (empty)
rate.limit.trusted.proxies Trusted proxy IPs (for X-Forwarded-For/X-Real-IP resolution) 127.0.0.1,::1
rate.limit.cleanup.interval Cleanup interval (number of requests, reserved) 1000

Note

rate.limit.cleanup.interval is a setting reserved for future use. In the current implementation, the request counters and blocked IP information are cleaned up automatically based on the internal cache expiration (rate.limit.window.ms and rate.limit.block.duration.ms), so this value is not used.

Advanced Rate Limiting Configuration

Custom Rate Limiting

To apply different rate limiting logic based on specific conditions, custom component implementation is required.

// Custom RateLimitHelper example
public class CustomRateLimitHelper extends RateLimitHelper {
    @Override
    public boolean allowRequest(String ip) {
        // Custom logic
    }
}

Exclusion Settings

You can exclude specific IP addresses from rate limiting or block them.

# Whitelist IPs (excluded from rate limiting, comma-separated)
rate.limit.whitelist.ips=127.0.0.1,::1,192.168.1.100

# Blocked IP list (always blocked, comma-separated)
rate.limit.blocked.ips=203.0.113.50

# Trusted proxy IPs (comma-separated)
rate.limit.trusted.proxies=127.0.0.1,::1

Note

When using a reverse proxy, set the proxy’s IP address in rate.limit.trusted.proxies. Client IPs are resolved from X-Forwarded-For and X-Real-IP headers only for requests coming from trusted proxies.

Monitoring and Alerts

Configuration for monitoring rate limiting status:

Log Output

When rate limiting is applied, it is recorded in the logs:

<Logger name="org.codelibs.fess.helper.RateLimitHelper" level="INFO"/>

The message is logged at INFO, so it is not written at the default log level (warn). Add the Logger above to log4j2.xml (for its location, see Log Configuration) or set FESS_LOG_LEVEL to info.

Troubleshooting

Legitimate Requests Being Blocked

Cause: Limit values are too strict

Solutions:

  1. Increase rate.limit.requests.per.window

  2. Add specific IPs to the whitelist (rate.limit.whitelist.ips)

  3. Adjust the window size (rate.limit.window.ms)

Rate Limiting Not Working

Cause: Configuration is not properly applied

Check:

  1. Is rate.limit.enabled=true configured?

  2. Is the configuration file being read correctly?

  3. Has Fess been restarted?

Performance Impact

If rate limiting checks themselves impact performance:

  1. Use the whitelist to skip checks for trusted IPs

  2. Disable rate limiting (rate.limit.enabled=false)

Reference Information