Content Server 16 is here!!!

My Dear Readers,

After a brief waiting period, Content Server 16 is now available in KC with some new features and a refreshingly new User Interface.

Grab your own copy of it. Very soon we will have something coming up on this front in my blog.

Till then Happy learning :)
(and also keep posting your comments here ;) )

OScript Data Types

Oscript has a list of data types which can be used for storing data. These can be classified into two categories:

Value Data Types - When passed as a parameter, value data types get copied. Hence any change to copied data does not effect the original. Following are the value data types:

1. Integer
2. Real
3. Boolean
4. String
5. Date
6. Undefined
7. Object
8. File
9. Error

Reference Data Types - When passed as a parameter, a reference of these data types are passed. Any change to passed variable changes the original one as well. Following are the reference data types:

1. Assoc - an array of key-value pair
2. Rec Array -a two-dimensional array or collection of multiple Records
3. List - a one dimensional array. An element of this variable can be of any data type

Processing Notification in OT Content Server

The important agents related to notification are:

9000	Processes events taking place in content server
8999	Processing events from Notify events to LLEventQueue
9999	Sends emails from NotifyMessages to SMTP
4000	Deletes messages from NotifyMessages

The important tables related to notification are:

AgentConfig	Stores notification configuration
AgentSchedule	stores agent schedule configuration
AgentStats	stores notification statistics
NotifyInterests2	stores every user’s notification interest
NotifyEvents	stores events which have occurred in content server
LLEventQueue	an event is assigned to user, gets stored in this table
NotifyMessages	stores notification messages which are sent as email

To setup Notification:

• Configure mail smtp settings
• Enable notification from administrator page
• Setup notification from Personal -> Notification section
• Enable notification on desired location

Notification Process

If we enable notification over a folder so that user receives an email if a new item is added inside it, the process which content server follows from item addition to sending user an email follows the following steps:

1. When an item is added, an event is generated which gets stored in NotifyEvents table.
2. Agent 8999 processes these events from NotifyEvents to LLEventQueue and then deletes entry from NotifyEvents table.
3. Agent 9000 processes requests from LLEventQueue and writes it down to NotifyMessages after checking user’s notification settings.
4. Agent 9999 sends email for every entry in NotifyMessages table to SMTP server. After 9999 has finished processing, agent 4000 deletes entries in NotifyMessages table.
5. Two threads enables agent to execute- Notify102, to handle all notification related agents and Agent101 to handle all other agents.

New Features in Content Server 10.5

The new features in Content Server 10.5 are:

1. Better user interface - Content Server provides better interface than previous Livelink versions.

2. Drag & Drop on Web Page using HTML 5 - Just like Google Drive, you can simply drag and drop documents from your system to required location in Content Server webpage and the document will get uploaded on its own. No need to follow the lengthy procedure of selecting Add Document, browsing for your doc and the adding it. This uses HTML 5 to implement this feature and probably that's why OpenText is calling it drag and drop using HTML 5. This will certainly make it more easy to use for users.

3. Return to last search result - This allows user to return to previous search result instead of searching the same thing again.

4. Duplicate Name Check- In the current version you cannot create two items with same name at a particular location. If you try to do so, it’ll show an error message in the “Add Item” page itself. Thus at the same time, you’ll have to enter a different name for your Item.

5. Login Credentials Verification- In older versions, entering incorrect login credentials showed you an error page specifying incorrect username/password. In 10.5, this is shown at login page itself as a message on top. So without going back you can re-enter your credentials.

6. Facets - Its a filtering feature to omit showing of unnecessary items. Its a very handy tool to view required items in a location especially when you have lots of items at one place. Earlier this was visible only after we search anything in Content Server/Livelink, but now its present even in normal folder view. A good addition indeed to enhance user experience.

7. More Threads- Content server 10.5 runs with 32 threads instead of 5 in 9.7.1. This makes performance faster than before.

8. Workflow GUI - The workflow GUI has been give a new look to make it more like industry standards.

9. Thumbnail view - CS 10.5 allows users to view items in thumbnail view. This provides more ease of viewing option to users.

10. Transport Warehouse- This feature allows moving objects from one instance to another.

11. Cluster Management - It handles everything related to applying patches. Particularly helpful in clustered environment.

Earlier we had to manually install new patches for every instance of Livelink installation. Cluster Management allows patch installation on all instances in a clustered environment in one go. Server details and other configuration needs to be entered and the corresponding patch has to be installed from Knowledge Center manually.

Content Server Search Architecture and Scaling

Back End Scaling – Admin server

Livelink admin server can be scaled to improve performance and task distribution for high volume requirements.

Search architecture can be broken down into three parts:

• Data Flow – It consists of Extractor, IPool, DCS. It also contains Classification Importer and Prospector Importer
• Search Grid – Update Distributor, Index Engine, Index, Search Engine, Search Federator. This is the scalable part of search architecture.
• Search manager – Search Federator, Search Manager

Search Architecture

Scalability can be achieved by the addition of more Admin server, search federators and Partitions. A diagram showing all three added to the architecture is shown below.

Scaled Search Architecture

Front End Scaling – Web server

Horizontal scaling

In horizontal scaling we have installations done of different physical servers. Thus if one server goes down we can still have others to share the load and keep the application up and running.

Horizontal Scaling

Vertical scaling

In vertical scaling, one physical server has two or more installations done no it. This allows optimal use of server’s resources and also reduces the cost of hardware since we can get two or more installations done at the cost of one.

Vertical Scaling

Ideally, a combination of horizontal and vertical scaling is the best solution.

Horizontal and Vertical scaling

Search IPC

This post deals with how the various components of Livelink search interact with each other.

Grid Registry

Grid Registry is a process which is used by other components to coordinate their use of RMI. On startup, each component reports its RMI port usage to Grid registry, and obtain RMI port of other processes from it. The processes then use RMI directly without further involving the Grid Registry.

Inter-Process Communication

Internal RMI (Remote Method Invocation) Communication

Search Federator uses RMI protocol to communicate with Search Engines and Update Distributor uses RMI protocol to communicate with the Index Engines.

Internal Socket Connections

Beginning with SE10 Update 4, sockets can be used as an alternative to RMI. This method is used by Content Server 10.

The direct socket approach uses fewer threads than the equivalent RMI configuration. In direct socket connection, RMI Grid registry is not required.

Search Process

Livelink Search Process

Search Federator:

Otsearchfederator is the process involved for this task. Search federator is responsible for:

• Accepting search query from content server and sending it to search engines
• Collecting and sorting result received from search engines
• Removing duplicate entries in search result

Search Engine:

Search Engine performs the task of searching queries in search index. The process involved is “otsearchengine”. Each search engine carries out the search in its partition. They are also involved in building facets, computing position information for highlighting search results.

OT does not reveal the algorithm followed by search engine to carry out its activity.

Search Index:

• Consists of several partitions, each having an index engine and one or more search engine
• Object’s Content data is stored in disk while it’s metadata is stored in RAM (for faster retrieval)
• Physically, index is stored in OTHOME\index\enterprise\index1 (index1 for first partition) and IPool is stored in OTHOME\index\enterprise\data_flow
• VerifyIndex tool can be used to check if the index is corrupt [enterprise data source-> maintenance -> verify contents of the index]
• Partitions can be Read/Write, Update-Only, Read-Only. Changing partition’s mode from Read/Write to Read only or Update only, also moves the metadata from memory to disk.

 Partitions- Search index can be divided horizontally into several pieces called “partitions”.

SE10 can often provide better indexing or searching performance by allowing operations to be distributed to multiple partitions. These partitions can be run on separate physical or virtual computers or CPUs to improve performance.

Partition Implementation Overview

Every partition is a self-contained subset of the search index. Every one of them has its own index files, a Search Engine, and an Index Engine. The partitions are tied together by the Update Distributor (for indexing) and by the Search Federator (for queries).

Each partition is relatively independent of the other partitions in the system during indexing. If one partition is given an object to index, the other partitions are idle. The Update Distributor can distribute the indexing load across multiple partitions. For systems with high indexing volumes, using multiple partitions this way can help achieve higher performance, since partitions can be indexing objects in parallel.

A search query normally is serviced by all partitions. Only partitions containing matches to the query will return results. The Search Federator will blend results from multiple partitions into a consolidated set of search results.

• Update-Only Partitions - In this mode, the partition will not accept new objects to index, but it will update existing objects or delete existing objects. If a partition is marked as Update-Only, then the Update Distributor will not send it new objects.
• Read-Only Partitions - In this mode, the partition will respond to search queries, but will not process any indexing requests. Objects in the partition cannot be added removed or modified.
• Read-Write Partitions - For completeness, the normal mode of operation for a partition is “Read-Write” mode. In this mode, the partition will accept new objects, can delete objects and update objects.

Index server:

The main process for index server is otdb. Child processes of otdb are:

1.    Otupbld-index building process, adding, updating and deleting

2.    Otmrg-merging smaller and many indexes into one large index

3.    Otcomp- compacting index when 30% of it gets deleted

4.    Mltcon- multiple connection manager

Otsumlog utility can be used to generate separate log files for each child process of index server (otdb)