Overview
Fess Data Store Connectors provide functionality to retrieve and index content from data sources other than websites and file systems.
Using Data Store Connectors, you can make data searchable from the following sources:
Cloud storage (Box, Dropbox, Google Drive, OneDrive)
Collaboration tools (Confluence, Jira, Slack)
Databases (MySQL, PostgreSQL, Oracle, etc.)
Other systems (Git, Salesforce, Elasticsearch, etc.)
Available Connectors
Fess provides connectors for various data sources. Many connectors are available as plugins and can be installed as needed.
Cloud Storage
| Connector | Plugin | Description |
|---|---|---|
| Box Connector | fess-ds-box | Crawl files and folders from Box.com |
| Dropbox Connector | fess-ds-dropbox | Crawl files and folders from Dropbox |
| Google Workspace Connector | fess-ds-gsuite | Crawl Google Drive, Gmail, etc. |
| Microsoft 365 Connector | fess-ds-office365 | Crawl OneDrive, SharePoint, etc. |
Collaboration Tools
| Connector | Plugin | Description |
|---|---|---|
| Atlassian Connector | fess-ds-atlassian | Crawl Confluence and Jira |
| Slack Connector | fess-ds-slack | Crawl Slack messages and files |
Development & Operations Tools
| Connector | Plugin | Description |
|---|---|---|
| Git Connector | fess-ds-git | Crawl source code from Git repositories |
| Elasticsearch/OpenSearch Connector | fess-ds-elasticsearch | Retrieve data from Elasticsearch/OpenSearch |
| Salesforce Connector | fess-ds-salesforce | Crawl Salesforce objects |
Databases & Files
| Connector | Plugin | Description |
|---|---|---|
| Database Connector | (Built-in) | Retrieve data from JDBC-compatible databases |
| CSV Connector | fess-ds-csv | Retrieve data from CSV files |
| JSON Connector | fess-ds-json | Retrieve data from JSON files |
Installing Connectors
Installing Plugins
Data Store Connector plugins can be installed from the admin console or using the plugin command.
From Admin Console
Log in to the admin console
Navigate to “System” -> “Plugins”
Search for the target plugin in the “Available” tab
Click “Install”
Restart Fess
Command Line
# Install a plugin
./bin/fess-plugin install fess-ds-box
# List installed plugins
./bin/fess-plugin list
Docker Environment
# Install plugins at startup
docker run -e FESS_PLUGINS="fess-ds-box,fess-ds-dropbox" codelibs/fess:15.5.0
Data Store Configuration Basics
Data Store Connector configuration is done in the admin console under “Crawler” -> “Data Store”.
Common Configuration Items
Configuration items common to all Data Store Connectors:
| Item | Description |
|---|---|
| Name | Identifier name for the configuration |
| Handler Name | Handler name for the connector (e.g., BoxDataStore) |
| Parameters | Connector-specific configuration parameters (key=value format) |
| Script | Index field mapping script |
| Boost | Search result priority |
| Enabled | Whether to enable this configuration |
Parameter Configuration
Parameters are specified in key=value format, separated by newlines:
api.key=xxxxxxxxxxxxx
folder.id=0
max.depth=3
Script Configuration
Scripts map retrieved data to Fess index fields:
url=data.url
title=data.name
content=data.content
mimetype=data.mimetype
filetype=data.filetype
filename=data.filename
created=data.created
lastModified=data.lastModified
contentLength=data.contentLength
Authentication Configuration
Many Data Store Connectors require OAuth 2.0 or API key authentication.
OAuth 2.0 Authentication
Common OAuth 2.0 configuration parameters:
client.id=Client ID
client.secret=Client Secret
refresh.token=Refresh Token
Or:
access.token=Access Token
API Key Authentication
api.key=API Key
api.secret=API Secret
Service Account Authentication
service.account.email=Service account email address
service.account.key=Private key (JSON format or key file path)
Performance Tuning
Configuration for processing large amounts of data:
# Batch size
batch.size=100
# Wait time between requests (milliseconds)
interval=1000
# Number of parallel threads
thread.size=1
# Timeout (milliseconds)
timeout=30000
Troubleshooting
Connector Not Displayed
Verify that the plugin is correctly installed
Restart Fess
Check logs for errors
Authentication Errors
Verify that authentication information is correct
Check token expiration
Verify that required permissions are granted
Verify that API access is allowed on the service side
Cannot Retrieve Data
Verify that parameter format is correct
Check access permissions for target folders/files
Check filter settings
Check logs for detailed error messages
Debug Configuration
When investigating issues, adjust the log level:
app/WEB-INF/classes/log4j2.xml:
<Logger name="org.codelibs.fess.ds" level="DEBUG"/>
Reference Information
Data Store Crawling - Data Store Configuration Guide
Plugin - Plugin Management Guide
../api/admin/api-admin-dataconfig - Data Store Configuration API