Search Indexing Process

Indexing in Livelink Search

The Extractor:

“Indexupdate” which is the Extractor process, makes call to index.update (function) request handler in llserver process. Index.update sends request to llserver process. It is the llserver performs that performs actual extraction.

Indexupdate.exe is run by Admin process every 1 minute.

Extractor receives metadata from database and content from file storage, packs it into IPool and sends it to Document Conversion process.

After enabling thread logging, index.update function shows the objects being extracted.

There are two modes of extraction:

Full extraction mode- Building index for the first time. Extractor traverses DTree table from highest dataid to lowest dataid creating IPool messages. After DTree, it uses DTreeNotify and WIndexNotify to get most recent data indexed first.

Incremental Extraction mode- Updates new items being added, runs every minute

Extraction can be captured by enabling thread log and connect log on Admin server hosting Extractor process. In this mode, DTreeNotify and WIndexNotify tables are used for new updates.

Fun fact:

If the following query returns “-1” then extractor is in Incremental Extraction mode:

SELECT IniValue FROM KIni WHERE IniSection=’OTIndex’;

Logging Extractor:

Extraction is captured in logs by enabling Livelink thread logs and Connect logs on Livelink Admin server hosting the Extractor process.

Enter the following in opentext.ini file:

Debug=2                                  /* for thread logs*/

wantLogs=TRUE                       /* for connect logs*/

wantTimings=TRUE

wantVerbose=TRUE

wantDebugSearch=TRUE

Restart the services.

(Note: Connect logs increase in size very rapidly. Use them only when necessary and disable them after use.)

Document Conversion Services:

DCS is a set of processes and services responsible for preparing data prior to indexing. DCS performs tasks such as managing the data flows and IPools during ingestion, extracting text and metadata from content. It is responsible for retrieving indexable content out of objects in an IPool message and passing them onto Update Distributor. It’s a scheduled process that starts every minute.

Otdoccnv process is run by Admin server every 1 minute. There are two instances of otdoccnv process running for every data source- master and slave.

Master- reads documents from IPool#1 and sends them to slave for conversion, reads the result from slave and writes to IPool#2.

Slave- converts the content to HTML format

Logging DCS:

Add the following lines [FilterEngine] section of opentext.ini file:

logfile=\livelinkInstall\logs\otdoccnv.log

This will generate two log files:

·         otdoccnv.log.master

·         otdoccnv.log.slave

Update Distributor:

The process involved is “otupdatedistributor”. As the name suggests, it distributes any new index update task to appropriate index engine. Update Distributor monitors IPool directory to check for new indexing requests. Then determines the index engine to send update request and sends it to that engine.

For selecting index engine, Update Distributor sends a message to all index engines, using key, to check for duplicate entry with them. The one that confirms having duplicate object is given that IPool for writing/updating or deleting. If data is not in any index engine then it gives IPool to one of them using Round-Robin method. During allocation, it also checks for the mode of partition such as Read/Write, Read-only and Update-only (we'll deal with partition modes in detail in next pages). It also considers parameters such as memory available with partition.

Update Distributor works in the following sequence:

1. Read from LLHome\config\search.ini file

2. Contact RMI Registry Server

3. Contact the index engine

4. Start to process IPool messages

Update Distributor is responsible for rolling back transaction if indexing of any IPool fails.

Logging Update Distributor:

System Object Volume -> Enterprise Data Source -> Enterprise Data Flow Manager -> Functions menu for Enterprise Update Distributor -> Properties -> Advanced Settings

Index Engine:

Index Engine (otindexengine) performs the task of adding, updating and deleting objects in the search index. It accepts the request from Update Distributor. Each partition has at least one index engine.

Ipupdate process is run by Admin server. This process separates content from metadata. While metadata is saved in *._d_ temp file, content is stored in *.urn temp file.  Eg 10._d_ is for metadata and 10.urn is created for content.

Logging Index Engine:

Index engines logs can be enabled from: Administration Page -> System Object Volume -> Enterprise Data Source -> Enterprise Data Flow Manager -> Functions menu for Enterprise Update Distributor -> Properties -> Index Engine

Ipupdate logs can be viewed from this location: Enterprise\index\ipupdate.log

IPool Structure:

IPool (Interchange Pool) is used to encapsulate data for indexing. IPool in simplest form consists of two directories. Enterprise Data Flow (EDF) process is responsible for writing to and reading from IPool directory.

It can be viewed from Enterprise Data Flow Manager page.