Menu

Data Store Configuration

Overview

Data Store Crawling Configuaration page manages configurations for crawling on Data Store, such as Database.

Management Operations

Display Configurations

Select Crawler > Data Store in the left menu to display a list page of Data Store Configuration, as below.

image0

Click a configuration name if you want to edit it.

Create Configuration

Click Create New button to display a form page for Data Store configuration.

image1

Configurations

Name

Configuration name.

Handler Name

Type of Data Store.

  • DatabaseDataStore: crawl data in Database.

  • CsvDataStore: crawl data in CSV file.

  • CsvListDataStore: crawl file paths in CSV file.

Parameter

Parameters for Data Store crawling.

Script

Field mapping on index. The format is key/value, such as [Field Name In Index]=[Value(Groovy Supported)].

Boost

Boost value is a weight for indexed documents of this configuration.

Permissions

Permissions for this configuration. This format is “{user/group/role}name”. For example, to display search results on users who belong to developer group, the permission is {group}developer.

Virtual Hosts

Virtual Host keys for this configuration. e.g. fess (if setting Host:fess.codelibs.org=fess in General)

Status

If enabled, the scheduled job of Default Crawler includes this configuration.

Description

Comments for this configuration.

Delete Configuration

Click a configuration on a list page, and click Delete button to display a confirmation dialog. Click Delete button to delete the configuration.

Example

DatabaseDataStore

This section describes Database crawling.

This example uses MySQL. There is the following table in testdb, and you can access data with username=foo and password=bar.

CREATE TABLE doc (
    id BIGINT NOT NULL AUTO_INCREMENT,
    title VARCHAR(100) NOT NULL,
    content VARCHAR(255) NOT NULL,
    latitude VARCHAR(20),
    longitude VARCHAR(20),
    versionNo INTEGER NOT NULL,
    PRIMARY KEY (id)
);

Insert the following data.

INSERT INTO doc (title, content, latitude, longitude, versionNo) VALUES ('Title 1', 'This is Content 1.', '37.77493', ' -122.419416', 1);
INSERT INTO doc (title, content, latitude, longitude, versionNo) VALUES ('Title 2', 'This is Content 2.', '34.701909', '135.494977', 1);
INSERT INTO doc (title, content, latitude, longitude, versionNo) VALUES ('Title 3', 'This is Content 3.', '-33.868901', '151.207091', 1);
INSERT INTO doc (title, content, latitude, longitude, versionNo) VALUES ('Title 4', 'This is Content 4.', '51.500152', '-0.13.636', 1);
INSERT INTO doc (title, content, latitude, longitude, versionNo) VALUES ('Title 5', 'This is Content 5.', '35.6813.6', '139.766084', 1);

Parameter

Set parameters of crawling config as below.

driver=com.mysql.jdbc.Driver
url=jdbc:mysql://localhost:3306/testdb?useUnicode=true&characterEncoding=UTF-8
username=foo
password=bar
sql=select * from doc

The value of parameters is key/value format. The description is below.

driver JDBC driver class
url URL
username Username to access to DB
password Password to access to DB
sql SQL statement to crawl data

Table: Parameter for DB

Script

Set script values of crawling config as below.

url="http://localhost/" + id
host="localhost"
site="localhost"
title=title
content=content
digest=content
anchor=
content_length=content.length()
last_modified=new java.util.Date()
location=latitude + "," + longitude
latitude=latitude
longitude=longitude

The format is key/value. For specified values, Groovy language is available. The description for Script is below.

url URL(Displayed link in search result page)
host Host name
site Site path
title Title
content Content
cache Content Cache(Not search target)
digest Displayed description in search result page
anchor Links contained in this document(optional)
content_length Content length
last_modified Last modified for this document

Table: Configuaration for Script

JDBC Driver

To crawl data in database, you need JDBC driver and put Jar file into app/WEB-INF/lib directory. In this example, copy mysql-connector-java-5.1.39.jar to app/WEB-INF/lib.

CsvDataStore

CsvDataStore is to import data from CSV files.

For example, create test.csv in /home/john/csv, and the content is below.

1,Title 1,This is Test1.
2,Title 2,This is Test2.
3,Title 3,This is Test3.
4,Title 4,This is Test4.
5,Title 5,This is Test5.
6,Title 6,This is Test6.
7,Title 7,This is Test7.
8,Title 8,This is Test8.
9,Title 9,This is Test9.

Parameter

Set parameters of crawling config as below.

directories=/home/john/csv
fileEncoding=UTF-8

The value of parameters is key/value format. For the description, see below.

directories Directories which contains csv file(.csv or .tsv)
files CSV files (if you want to specify them)
fileEncoding Encofing of CSV files
separatorCharacter Separator character in CSV content

Table: Parameters for CSV file

Script

Set script values of crawling config as below.

url="http://localhost/" + cell1
host="localhost"
site="localhost"
title=cell2
content=cell3
cache=cell3
digest=cell3
anchor=
content_length=cell3.length()
last_modified=new java.util.Date()

The format is key/value. For keys, they are the same as database crawling. You can use values in CSV file as cell[number] (cell1 is a first cell). If cell does not exist, it returns null.

CsvListDataStore

You can use CsvListDataStore to crawl a large number of target files, and also reduce the crawling time to use CSV file which contains only updated file paths.

The file format is as below.

[Action]<Separator character>[Path]

Specify one of the following value for Action.

create:Created file modify:Updated file delete:Deleted file

For Path, you can sepecify a path in the same notation as a setting for file crawling.

For example, create test.csv in /home/john/csv, and the content is below.

modify,smb://servername/data/testfile1.txt
modify,smb://servername/data/testfile2.txt
modify,smb://servername/data/testfile3.txt
modify,smb://servername/data/testfile4.txt
modify,smb://servername/data/testfile5.txt
modify,smb://servername/data/testfile6.txt
modify,smb://servername/data/testfile7.txt
modify,smb://servername/data/testfile8.txt
modify,smb://servername/data/testfile9.txt
modify,smb://servername/data/testfile10.txt

Parameter

Set parameters of crawling config as below.

directories=/home/taro/csv
fileEncoding=Shift_JIS

The value of parameters is key/value format. For the description, see below.

directories Directories which contains csv file(.csv or .tsv)
fileEncoding Encofing of CSV files
separatorCharacter Separator character in CSV content

Table: Parameter for CsvListDataStore

Script

Set script values of crawling config as below.

event_type=cell1
url=cell2

The format is key/value. For keys, they are the same as database crawling.

If authentication is required at the crawling destination, you must also add the following settings.

crawler.file.auth=example
crawler.file.auth.example.scheme=SAMBA
crawler.file.auth.example.username=username
crawler.file.auth.example.password=password