<chapter id="quick-start">
-<title>Quick Start </title>
-
-<para>
-In this section, we will test the system by indexing a small set of sample
-GILS records that are included with the software distribution. Go to the
-<literal remap="tt">test/gils</literal> subdirectory of the distribution archive. There you will
-find a configuration
-file named <literal remap="tt">zebra.cfg</literal> with the following contents:
-
-<screen>
-# Where are the YAZ tables located.
-profilePath: ../../../yaz/tab ../../tab
-
-# Files that describe the attribute sets supported.
-attset: bib1.att
-attset: gils.att
-</screen>
-
-</para>
-
-<para>
-Now, edit the file and set <literal remap="tt">profilePath</literal> to the path of the
-YAZ profile tables (sub directory <literal remap="tt">tab</literal> of the YAZ distribution
-archive).
-</para>
-
-<para>
-The 48 test records are located in the sub directory <literal remap="tt">records</literal>.
-To index these, type:
-
-<screen>
-$ ../../index/zebraidx -t grs.sgml update records
-</screen>
-
-</para>
-
-<para>
-In the command above the option <literal remap="tt">-t</literal> specified the record
-type — in this case <literal remap="tt">grs.sgml</literal>. The word <literal remap="tt">update</literal> followed
-by a directory root updates all files below that directory node.
-</para>
-
-<para>
-If your indexing command was successful, you are now ready to
-fire up a server. To start a server on port 2100, type:
-
-<screen>
-$ ../../index/zebrasrv tcp:@:2100
-</screen>
-
-</para>
-
-<para>
-The Zebra index that you have just created has a single database
-named <literal remap="tt">Default</literal>. The database contains records structured according to
-the GILS profile, and the server will
-return records in either either USMARC, GRS-1, or SUTRS depending
-on what your client asks
-for.
-</para>
-
-<para>
-To test the server, you can use any Z39.50 client (1992 or later). For
-instance, you can use the demo client that comes with YAZ: Just cd to
-the <literal remap="tt">client</literal> subdirectory of the YAZ distribution and type:
-</para>
-
-<para>
-
-<screen>
-$ client tcp:localhost:2100
-</screen>
-
-</para>
-
-<para>
-When the client has connected, you can type:
-</para>
-
-<para>
-
-<screen>
-Z> find surficial
-Z> show 1
-</screen>
-
-</para>
-
-<para>
-The default retrieval syntax for the client is USMARC. To try other
-formats for the same record, try:
-</para>
-
-<para>
-
-<screen>
-Z>format sutrs
-Z>show 1
-Z>format grs-1
-Z>show 1
-Z>elements B
-Z>show 1
-</screen>
-
-</para>
-
-<para>
-<emphasis remap="it">NOTE: You may notice that more fields are returned when your
-client requests SUTRS or GRS-1 records. When retrieving GILS records,
-this is normal - not all of the GILS data elements have mappings in
-the USMARC record format.</emphasis>
-</para>
-
-<para>
-If you've made it this far, there's a good chance that
-you've got through the compilation OK.
-</para>
-
+ <title>Quick Start </title>
+
+ <para>
+ In this section, we will test the system by indexing a small set of sample
+ GILS records that are included with the software distribution. Go to the
+ <literal>test/gils</literal> subdirectory of the distribution archive.
+ There you will find a configuration
+ file named <literal>zebra.cfg</literal> with the following contents:
+
+ <screen>
+ # Where are the YAZ tables located.
+ profilePath: ../../../yaz/tab ../../tab
+
+ # Files that describe the attribute sets supported.
+ attset: bib1.att
+ attset: gils.att
+ </screen>
+ </para>
+
+ <para>
+ Now, edit the file and set <literal>profilePath</literal> to the path of the
+ YAZ profile tables (sub directory <literal>tab</literal> of the YAZ
+ distribution archive).
+ </para>
+
+ <para>
+ The 48 test records are located in the sub directory
+ <literal>records</literal>. To index these, type:
+
+ <screen>
+ $ ../../index/zebraidx -t grs.sgml update records
+ </screen>
+ </para>
+
+ <para>
+ In the command above the option <literal>-t</literal> specified the record
+ type — in this case <literal>grs.sgml</literal>.
+ The word <literal>update</literal> followed
+ by a directory root updates all files below that directory node.
+ </para>
+
+ <para>
+ If your indexing command was successful, you are now ready to
+ fire up a server. To start a server on port 2100, type:
+
+ <screen>
+ $ ../../index/zebrasrv tcp:@:2100
+ </screen>
+
+ </para>
+
+ <para>
+ The Zebra index that you have just created has a single database
+ named <literal>Default</literal>.
+ The database contains records structured according to
+ the GILS profile, and the server will
+ return records in either either USMARC, GRS-1, or SUTRS depending
+ on what your client asks for.
+ </para>
+
+ <para>
+ To test the server, you can use any Z39.50 client (1992 or later).
+ For instance, you can use the demo client that comes with YAZ: Just
+ cd to the <literal>client</literal> subdirectory of the YAZ distribution
+ and type:
+ </para>
+ <para>
+ <screen>
+ $ ./yaz-client tcp:localhost:2100
+ </screen>
+ </para>
+
+ <para>
+ When the client has connected, you can type:
+ </para>
+
+<para>
+
+ <screen>
+ Z> find surficial
+ Z> show 1
+ </screen>
+ </para>
+
+ <para>
+ The default retrieval syntax for the client is USMARC. To try other
+ formats for the same record, try:
+ </para>
+ <para>
+ <screen>
+ Z>format sutrs
+ Z>show 1
+ Z>format grs-1
+ Z>show 1
+ Z>elements B
+ Z>show 1
+ </screen>
+ </para>
+
+ <note>
+ <para>You may notice that more fields are returned when your
+ client requests SUTRS or GRS-1 records. When retrieving GILS records,
+ this is normal - not all of the GILS data elements have mappings in
+ the USMARC record format.
+ </para>
+ </note>
+ <para>
+ If you've made it this far, there's a good chance that
+ you've got through the compilation OK.
+ </para>
+
</chapter>
<chapter id="administration">
-<title>Administrating Zebra</title>
-
-<para>
-Unlike many simpler retrieval systems, Zebra supports safe, incremental
-updates to an existing index.
-</para>
-
-<para>
-Normally, when Zebra modifies the index it reads a number of records
-that you specify.
-Depending on your specifications and on the contents of each record
-one the following events take place for each record:
-<variablelist>
-
-<varlistentry>
-<term>Insert</term>
-<listitem>
-<para>
-The record is indexed as if it never occurred
-before. Either the Zebra system doesn't know how to identify the record or
-Zebra can identify the record but didn't find it to be already indexed.
-</para>
-</listitem>
-</varlistentry>
-<varlistentry>
-<term>Modify</term>
-<listitem>
-<para>
-The record has already been indexed. In this case
-either the contents of the record or the location (file) of the record
-indicates that it has been indexed before.
-</para>
-</listitem>
-</varlistentry>
-<varlistentry>
-<term>Delete</term>
-<listitem>
-<para>
-The record is deleted from the index. As in the
-update-case it must be able to identify the record.
-</para>
-</listitem>
-</varlistentry>
-</variablelist>
-</para>
-
-<para>
-Please note that in both the modify- and delete- case the Zebra
-indexer must be able to generate a unique key that identifies the record in
-question (more on this below).
-</para>
-
-<para>
-To administrate the Zebra retrieval system, you run the
-<literal remap="tt">zebraidx</literal> program. This program supports a number of options
-which are preceded by a minus, and a few commands (not preceded by
-minus).
-</para>
-
-<para>
-Both the Zebra administrative tool and the Z39.50 server share a
-set of index files and a global configuration file. The
-name of the configuration file defaults to <literal remap="tt">zebra.cfg</literal>.
-The configuration file includes specifications on how to index
-various kinds of records and where the other configuration files
-are located. <literal remap="tt">zebrasrv</literal> and <literal remap="tt">zebraidx</literal> <emphasis>must</emphasis>
-be run in the directory where the configuration file lives unless you
-indicate the location of the configuration file by option
-<literal remap="tt">-c</literal>.
-</para>
-
-<sect1 id="record-types">
-<title>Record Types</title>
-
-<para>
-Indexing is a per-record process, in which either insert/modify/delete
-will occur. Before a record is indexed search keys are extracted from
-whatever might be the layout the original record (sgml,html,text, etc..).
-The Zebra system currently supports two fundamantal types of records:
-structured and simple text.
-To specify a particular extraction process, use either the
-command line option <literal remap="tt">-t</literal> or specify a
-<literal remap="tt">recordType</literal> setting in the configuration file.
-</para>
-
-</sect1>
-
-<sect1 id="configuration-file">
-<title>The Zebra Configuration File</title>
-
-<para>
-The Zebra configuration file, read by <literal remap="tt">zebraidx</literal> and
-<literal remap="tt">zebrasrv</literal> defaults to <literal remap="tt">zebra.cfg</literal> unless specified
-by <literal remap="tt">-c</literal> option.
-</para>
-
-<para>
-You can edit the configuration file with a normal text editor.
-parameter names and values are seperated by colons in the file. Lines
-starting with a hash sign (<literal remap="tt">#</literal>) are treated as comments.
-</para>
-
-<para>
-If you manage different sets of records that share common
-characteristics, you can organize the configuration settings for each
-type into "groups".
-When <literal remap="tt">zebraidx</literal> is run and you wish to address a given group
-you specify the group name with the <literal remap="tt">-g</literal> option. In this case
-settings that have the group name as their prefix will be used
-by <literal remap="tt">zebraidx</literal>. If no <literal remap="tt">-g</literal> option is specified, the settings
-with no prefix are used.
-</para>
-
-<para>
-In the configuration file, the group name is placed before the option
-name itself, separated by a dot (.). For instance, to set the record type
-for group <literal remap="tt">public</literal> to <literal remap="tt">grs.sgml</literal> (the SGML-like format for structured
-records) you would write:
-</para>
-
-<para>
-
-<screen>
-public.recordType: grs.sgml
-</screen>
-
-</para>
-
-<para>
-To set the default value of the record type to <literal remap="tt">text</literal> write:
-</para>
-
-<para>
-
-<screen>
-recordType: text
-</screen>
-
-</para>
-
-<para>
-The available configuration settings are summarized below. They will be
-explained further in the following sections.
-</para>
-
-<para>
-<variablelist>
-
-<varlistentry>
-<term><emphasis remap="it">group</emphasis>.recordType[<emphasis remap="it">.name</emphasis>]</term>
-<listitem>
-<para>
-Specifies how records with the file extension <emphasis remap="it">name</emphasis> should
-be handled by the indexer. This option may also be specified
-as a command line option (<literal remap="tt">-t</literal>). Note that if you do not
-specify a <emphasis remap="it">name</emphasis>, the setting applies to all files. In general,
-the record type specifier consists of the elements (each
-element separated by dot), <emphasis remap="it">fundamental-type</emphasis>,
-<emphasis remap="it">file-read-type</emphasis> and arguments. Currently, two
-fundamental types exist, <literal remap="tt">text</literal> and <literal remap="tt">grs</literal>.
-</para>
-</listitem>
-</varlistentry>
-<varlistentry>
-<term><emphasis remap="it">group</emphasis>.recordId</term>
-<listitem>
-<para>
-Specifies how the records are to be identified when updated. See
-section <xref linkend="locating-records"/>.
-</para>
-</listitem>
-</varlistentry>
-<varlistentry>
-<term><emphasis remap="it">group</emphasis>.database</term>
-<listitem>
-<para>
-Specifies the Z39.50 database name.
-</para>
-</listitem>
-</varlistentry>
-<varlistentry>
-<term><emphasis remap="it">group</emphasis>.storeKeys</term>
-<listitem>
-<para>
-Specifies whether key information should be saved for a given
-group of records. If you plan to update/delete this type of
-records later this should be specified as 1; otherwise it
-should be 0 (default), to save register space. See section
-<xref linkend="file-ids"/>.
-</para>
-</listitem>
-</varlistentry>
-<varlistentry>
-<term><emphasis remap="it">group</emphasis>.storeData</term>
-<listitem>
-<para>
-Specifies whether the records should be stored internally
-in the Zebra system files. If you want to maintain the raw records yourself,
-this option should be false (0). If you want Zebra to take care of the records
-for you, it should be true(1).
-</para>
-</listitem>
-</varlistentry>
-<varlistentry>
-<term>register</term>
-<listitem>
-<para>
-Specifies the location of the various register files that Zebra uses
-to represent your databases. See section
-<xref linkend="register-location"/>.
-</para>
-</listitem>
-</varlistentry>
-<varlistentry>
-<term>shadow</term>
-<listitem>
-<para>
-Enables the <emphasis remap="it">safe update</emphasis> facility of Zebra, and tells the system
-where to place the required, temporary files. See section
-<xref linkend="shadow-registers"/>.
-</para>
-</listitem>
-</varlistentry>
-<varlistentry>
-<term>lockDir</term>
-<listitem>
-<para>
-Directory in which various lock files are stored.
-</para>
-</listitem>
-</varlistentry>
-<varlistentry>
-<term>keyTmpDir</term>
-<listitem>
-<para>
-Directory in which temporary files used during zebraidx' update
-phase are stored.
-</para>
-</listitem>
-</varlistentry>
-<varlistentry>
-<term>setTmpDir</term>
-<listitem>
-<para>
-Specifies the directory that the server uses for temporary result sets.
-If not specified <literal remap="tt">/tmp</literal> will be used.
-</para>
-</listitem>
-</varlistentry>
-<varlistentry>
-<term>profilePath</term>
-<listitem>
-<para>
-Specifies the location of profile specification files.
-</para>
-</listitem>
-</varlistentry>
-<varlistentry>
-<term>attset</term>
-<listitem>
-<para>
-Specifies the filename(s) of attribute set files for use in
-searching. At least the Bib-1 set should be loaded (<literal remap="tt">bib1.att</literal>).
-The <literal remap="tt">profilePath</literal> setting is used to look for the specified files.
-See section <xref linkend="attset-files"/>
-</para>
-</listitem>
-</varlistentry>
-<varlistentry>
-<term>memMax</term>
-<listitem>
-<para>
-Specifies size of internal memory to use for the zebraidx program. The
-amount is given in megabytes - default is 4 (4 MB).
-</para>
-</listitem>
-</varlistentry>
-</variablelist>
-</para>
-
-</sect1>
-
-<sect1 id="locating-records">
-<title>Locating Records</title>
-
-<para>
-The default behaviour of the Zebra system is to reference the
-records from their original location, i.e. where they were found when you
-ran <literal remap="tt">zebraidx</literal>. That is, when a client wishes to retrieve a record
-following a search operation, the files are accessed from the place
-where you originally put them - if you remove the files (without
-running <literal remap="tt">zebraidx</literal> again, the client will receive a diagnostic
-message.
-</para>
-
-<para>
-If your input files are not permanent - for example if you retrieve
-your records from an outside source, or if they were temporarily
-mounted on a CD-ROM drive,
-you may want Zebra to make an internal copy of them. To do this,
-you specify 1 (true) in the <literal remap="tt">storeData</literal> setting. When
-the Z39.50 server retrieves the records they will be read from the
-internal file structures of the system.
-</para>
-
-</sect1>
-
-<sect1 id="simple-indexing">
-<title>Indexing with no Record IDs (Simple Indexing)</title>
-
-<para>
-If you have a set of records that are not expected to change over time
-you may can build your database without record IDs.
-This indexing method uses less space than the other methods and
-is simple to use.
-</para>
-
-<para>
-To use this method, you simply omit the <literal remap="tt">recordId</literal> entry
-for the group of files that you index. To add a set of records you use
-<literal remap="tt">zebraidx</literal> with the <literal remap="tt">update</literal> command. The
-<literal remap="tt">update</literal> command will always add all of the records that it
-encounters to the index - whether they have already been indexed or
-not. If the set of indexed files change, you should delete all of the
-index files, and build a new index from scratch.
-</para>
-
-<para>
-Consider a system in which you have a group of text files called
-<literal remap="tt">simple</literal>. That group of records should belong to a Z39.50 database
-called <literal remap="tt">textbase</literal>. The following <literal remap="tt">zebra.cfg</literal> file will suffice:
-</para>
-
-<para>
-
-<screen>
-profilePath: /usr/local/yaz
-attset: bib1.att
-simple.recordType: text
-simple.database: textbase
-</screen>
-
-</para>
-
-<para>
-Since the existing records in an index can not be addressed by their
-IDs, it is impossible to delete or modify records when using this method.
-</para>
-
-</sect1>
-
-<sect1 id="file-ids">
-<title>Indexing with File Record IDs</title>
-
-<para>
-If you have a set of files that regularly change over time: Old files
-are deleted, new ones are added, or existing files are modified, you
-can benefit from using the <emphasis remap="it">file ID</emphasis> indexing methodology. Examples
-of this type of database might include an index of WWW resources, or a
-USENET news spool area. Briefly speaking, the file key methodology
-uses the directory paths of the individual records as a unique
-identifier for each record. To perform indexing of a directory with
-file keys, again, you specify the top-level directory after the
-<literal remap="tt">update</literal> command. The command will recursively traverse the
-directories and compare each one with whatever have been indexed before in
-that same directory. If a file is new (not in the previous version of
-the directory) it is inserted into the registers; if a file was
-already indexed and it has been modified since the last update,
-the index is also modified; if a file has been removed since the last
-visit, it is deleted from the index.
-</para>
-
-<para>
-The resulting system is easy to administrate. To delete a record you
-simply have to delete the corresponding file (say, with the <literal remap="tt">rm</literal>
-command). And to add records you create new files (or directories with
-files). For your changes to take effect in the register you must run
-<literal remap="tt">zebraidx update</literal> with the same directory root again. This mode
-of operation requires more disk space than simpler indexing methods,
-but it makes it easier for you to keep the index in sync with a
-frequently changing set of data. If you combine this system with the
-<emphasis remap="it">safe update</emphasis> facility (see below), you never have to take your
-server offline for maintenance or register updating purposes.
-</para>
-
-<para>
-To enable indexing with pathname IDs, you must specify <literal remap="tt">file</literal> as
-the value of <literal remap="tt">recordId</literal> in the configuration file. In addition,
-you should set <literal remap="tt">storeKeys</literal> to <literal remap="tt">1</literal>, since the Zebra
-indexer must save additional information about the contents of each record
-in order to modify the indices correctly at a later time.
-</para>
-
-<para>
-For example, to update records of group <literal remap="tt">esdd</literal> located below
-<literal remap="tt">/data1/records/</literal> you should type:
-
-<screen>
-$ zebraidx -g esdd update /data1/records
-</screen>
-
-</para>
-
-<para>
-The corresponding configuration file includes:
-
-<screen>
-esdd.recordId: file
-esdd.recordType: grs.sgml
-esdd.storeKeys: 1
-</screen>
-
-</para>
-
-<para>
-<emphasis>Important note: You cannot start out with a group of records with simple
-indexing (no record IDs as in the previous section) and then later
-enable file record Ids. Zebra must know from the first time that you
-index the group that
-the files should be indexed with file record IDs.</emphasis>
-</para>
-
-<para>
-You cannot explicitly delete records when using this method (using the
-<emphasis remap="bf">delete</emphasis> command to <literal remap="tt">zebraidx</literal>. Instead
-you have to delete the files from the file system (or move them to a
-different location)
-and then run <literal remap="tt">zebraidx</literal> with the <emphasis remap="bf">update</emphasis> command.
-</para>
-
-</sect1>
-
-<sect1 id="generic-ids">
-<title>Indexing with General Record IDs</title>
-
-<para>
-When using this method you construct an (almost) arbritrary, internal
-record key based on the contents of the record itself and other system
-information. If you have a group of records that explicitly associates
-an ID with each record, this method is convenient. For example, the
-record format may contain a title or a ID-number - unique within the group.
-In either case you specify the Z39.50 attribute set and use-attribute
-location in which this information is stored, and the system looks at
-that field to determine the identity of the record.
-</para>
-
-<para>
-As before, the record ID is defined by the <literal remap="tt">recordId</literal> setting
-in the configuration file. The value of the record ID specification
-consists of one or more tokens separated by whitespace. The resulting
-ID is
-represented in the index by concatenating the tokens and separating them by
-ASCII value (1).
-</para>
-
-<para>
-There are three kinds of tokens:
-<variablelist>
-
-<varlistentry>
-<term>Internal record info</term>
-<listitem>
-<para>
-The token refers to a key that is
-extracted from the record. The syntax of this token is
-<literal remap="tt">(</literal> <emphasis>set</emphasis> <literal remap="tt">,</literal> <emphasis>use</emphasis> <literal remap="tt">)</literal>, where <emphasis>set</emphasis> is the
-attribute set name <emphasis>use</emphasis> is the name or value of the attribute.
-</para>
-</listitem>
-</varlistentry>
-<varlistentry>
-<term>System variable</term>
-<listitem>
-<para>
-The system variables are preceded by
-
-<screen>
-$
-</screen>
- and immediately followed by the system variable name, which
-may one of
-<variablelist>
-
-<varlistentry>
-<term>group</term>
-<listitem>
-<para>
-Group name.
-</para>
-</listitem>
-</varlistentry>
-<varlistentry>
-<term>database</term>
-<listitem>
-<para>
-Current database specified.
-</para>
-</listitem>
-</varlistentry>
-<varlistentry>
-<term>type</term>
-<listitem>
-<para>
-Record type.
-</para>
-</listitem>
-</varlistentry>
-</variablelist>
-</para>
-</listitem>
-</varlistentry>
-<varlistentry>
-<term>Constant string</term>
-<listitem>
-<para>
-A string used as part of the ID — surrounded
-by single- or double quotes.
-</para>
-</listitem>
-</varlistentry>
-</variablelist>
-</para>
-
-<para>
-For instance, the sample GILS records that come with the Zebra
-distribution contain a unique ID in the data tagged Control-Identifier.
-The data is mapped to the Bib-1 use attribute Identifier-standard
-(code 1007). To use this field as a record id, specify
-<literal remap="tt">(bib1,Identifier-standard)</literal> as the value of the
-<literal remap="tt">recordId</literal> in the configuration file.
-If you have other record types that uses the same field for a
-different purpose, you might add the record type
-(or group or database name) to the record id of the gils
-records as well, to prevent matches with other types of records.
-In this case the recordId might be set like this:
-
-<screen>
-gils.recordId: $type (bib1,Identifier-standard)
-</screen>
-
-</para>
-
-<para>
-(see section <xref linkend="data-model"/>
-for details of how the mapping between elements of your records and
-searchable attributes is established).
-</para>
-
-<para>
-As for the file record ID case described in the previous section,
-updating your system is simply a matter of running <literal remap="tt">zebraidx</literal>
-with the <literal remap="tt">update</literal> command. However, the update with general
-keys is considerably slower than with file record IDs, since all files
-visited must be (re)read to discover their IDs.
-</para>
-
-<para>
-As you might expect, when using the general record IDs
-method, you can only add or modify existing records with the <literal remap="tt">update</literal>
-command. If you wish to delete records, you must use the,
-<literal remap="tt">delete</literal> command, with a directory as a parameter.
-This will remove all records that match the files below that root
-directory.
-</para>
-
+ <title>Administrating Zebra</title>
+
+ <para>
+ Unlike many simpler retrieval systems, Zebra supports safe, incremental
+ updates to an existing index.
+ </para>
+
+ <para>
+ Normally, when Zebra modifies the index it reads a number of records
+ that you specify.
+ Depending on your specifications and on the contents of each record
+ one the following events take place for each record:
+ <variablelist>
+
+ <varlistentry>
+ <term>Insert</term>
+ <listitem>
+ <para>
+ The record is indexed as if it never occurred before.
+ Either the Zebra system doesn't know how to identify the record or
+ Zebra can identify the record but didn't find it to be already indexed.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term>Modify</term>
+ <listitem>
+ <para>
+ The record has already been indexed. In this case
+ either the contents of the record or the location (file) of the record
+ indicates that it has been indexed before.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term>Delete</term>
+ <listitem>
+ <para>
+ The record is deleted from the index. As in the
+ update-case it must be able to identify the record.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
+
+ <para>
+ Please note that in both the modify- and delete- case the Zebra
+ indexer must be able to generate a unique key that identifies the record in
+ question (more on this below).
+ </para>
+
+ <para>
+ To administrate the Zebra retrieval system, you run the
+ <literal>zebraidx</literal> program.
+ This program supports a number of options which are preceded by a dash,
+ and a few commands (not preceded by dash).
+</para>
+
+ <para>
+ Both the Zebra administrative tool and the Z39.50 server share a
+ set of index files and a global configuration file. The
+ name of the configuration file defaults to <literal>zebra.cfg</literal>.
+ The configuration file includes specifications on how to index
+ various kinds of records and where the other configuration files
+ are located. <literal>zebrasrv</literal> and <literal>zebraidx</literal>
+ <emphasis>must</emphasis> be run in the directory where the
+ configuration file lives unless you indicate the location of the
+ configuration file by option <literal>-c</literal>.
+ </para>
+
+ <sect1 id="record-types">
+ <title>Record Types</title>
+
+ <para>
+ Indexing is a per-record process, in which either insert/modify/delete
+ will occur. Before a record is indexed search keys are extracted from
+ whatever might be the layout the original record (sgml,html,text, etc..).
+ The Zebra system currently supports two fundamantal types of records:
+ structured and simple text.
+ To specify a particular extraction process, use either the
+ command line option <literal>-t</literal> or specify a
+ <literal>recordType</literal> setting in the configuration file.
+ </para>
+
+ </sect1>
+
+ <sect1 id="configuration-file">
+ <title>The Zebra Configuration File</title>
+
+ <para>
+ The Zebra configuration file, read by <literal>zebraidx</literal> and
+ <literal>zebrasrv</literal> defaults to <literal>zebra.cfg</literal>
+ unless specified by <literal>-c</literal> option.
+ </para>
+
+ <para>
+ You can edit the configuration file with a normal text editor.
+ parameter names and values are seperated by colons in the file. Lines
+ starting with a hash sign (<literal>#</literal>) are
+ treated as comments.
+ </para>
+
+ <para>
+ If you manage different sets of records that share common
+ characteristics, you can organize the configuration settings for each
+ type into "groups".
+ When <literal>zebraidx</literal> is run and you wish to address a
+ given group you specify the group name with the <literal>-g</literal>
+ option.
+ In this case settings that have the group name as their prefix
+ will be used by <literal>zebraidx</literal>.
+ If no <literal>-g</literal> option is specified, the settings
+ without prefix are used.
+ </para>
+
+ <para>
+ In the configuration file, the group name is placed before the option
+ name itself, separated by a dot (.). For instance, to set the record type
+ for group <literal>public</literal> to <literal>grs.sgml</literal>
+ (the SGML-like format for structured records) you would write:
+ </para>
+
+ <para>
+ <screen>
+ public.recordType: grs.sgml
+ </screen>
+ </para>
+
+ <para>
+ To set the default value of the record type to <literal>text</literal>
+ write:
+ </para>
+
+ <para>
+ <screen>
+ recordType: text
+ </screen>
+ </para>
+
+ <para>
+ The available configuration settings are summarized below. They will be
+ explained further in the following sections.
+ </para>
+
+ <para>
+ <variablelist>
+
+ <varlistentry>
+ <term>
+ <emphasis>group</emphasis>
+ .recordType[<emphasis>.name</emphasis>]
+ </term>
+ <listitem>
+ <para>
+ Specifies how records with the file extension
+ <emphasis>name</emphasis> should be handled by the indexer.
+ This option may also be specified as a command line option
+ (<literal>-t</literal>). Note that if you do not specify a
+ <emphasis>name</emphasis>, the setting applies to all files.
+ In general, the record type specifier consists of the elements (each
+ element separated by dot), <emphasis>fundamental-type</emphasis>,
+ <emphasis>file-read-type</emphasis> and arguments. Currently, two
+ fundamental types exist, <literal>text</literal> and
+ <literal>grs</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term><emphasis>group</emphasis>.recordId</term>
+ <listitem>
+ <para>
+ Specifies how the records are to be identified when updated. See
+ section <xref linkend="locating-records"/>.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term><emphasis>group</emphasis>.database</term>
+ <listitem>
+ <para>
+ Specifies the Z39.50 database name.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term><emphasis>group</emphasis>.storeKeys</term>
+ <listitem>
+ <para>
+ Specifies whether key information should be saved for a given
+ group of records. If you plan to update/delete this type of
+ records later this should be specified as 1; otherwise it
+ should be 0 (default), to save register space. See section
+ <xref linkend="file-ids"/>.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term><emphasis>group</emphasis>.storeData</term>
+ <listitem>
+ <para>
+ Specifies whether the records should be stored internally
+ in the Zebra system files.
+ If you want to maintain the raw records yourself,
+ this option should be false (0).
+ If you want Zebra to take care of the records for you, it
+ should be true(1).
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term>register</term>
+ <listitem>
+ <para>
+ Specifies the location of the various register files that Zebra uses
+ to represent your databases. See section
+ <xref linkend="register-location"/>.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term>shadow</term>
+ <listitem>
+ <para>
+ Enables the <emphasis>safe update</emphasis> facility of Zebra, and
+ tells the system where to place the required, temporary files.
+ See section
+ <xref linkend="shadow-registers"/>.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term>lockDir</term>
+ <listitem>
+ <para>
+ Directory in which various lock files are stored.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term>keyTmpDir</term>
+ <listitem>
+ <para>
+ Directory in which temporary files used during zebraidx' update
+ phase are stored.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term>setTmpDir</term>
+ <listitem>
+ <para>
+ Specifies the directory that the server uses for temporary result sets.
+ If not specified <literal>/tmp</literal> will be used.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term>profilePath</term>
+ <listitem>
+ <para>
+ Specifies the location of profile specification files.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term>attset</term>
+ <listitem>
+ <para>
+ Specifies the filename(s) of attribute set files for use in
+ searching. At least the Bib-1 set should be loaded
+ (<literal>bib1.att</literal>).
+ The <literal>profilePath</literal> setting is used to look for
+ the specified files.
+ See section <xref linkend="attset-files"/>
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term>memMax</term>
+ <listitem>
+ <para>
+ Specifies size of internal memory to use for the zebraidx program. The
+ amount is given in megabytes - default is 4 (4 MB).
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
+
+ </sect1>
+
+ <sect1 id="locating-records">
+ <title>Locating Records</title>
+
+ <para>
+ The default behaviour of the Zebra system is to reference the
+ records from their original location, i.e. where they were found when you
+ ran <literal>zebraidx</literal>.
+ That is, when a client wishes to retrieve a record
+ following a search operation, the files are accessed from the place
+ where you originally put them - if you remove the files (without
+ running <literal>zebraidx</literal> again, the client
+ will receive a diagnostic message.
+ </para>
+
+ <para>
+ If your input files are not permanent - for example if you retrieve
+ your records from an outside source, or if they were temporarily
+ mounted on a CD-ROM drive,
+ you may want Zebra to make an internal copy of them. To do this,
+ you specify 1 (true) in the <literal>storeData</literal> setting. When
+ the Z39.50 server retrieves the records they will be read from the
+ internal file structures of the system.
+ </para>
+
+ </sect1>
+
+ <sect1 id="simple-indexing">
+ <title>Indexing with no Record IDs (Simple Indexing)</title>
+
+ <para>
+ If you have a set of records that are not expected to change over time
+ you may can build your database without record IDs.
+ This indexing method uses less space than the other methods and
+ is simple to use.
+ </para>
+
+ <para>
+ To use this method, you simply omit the <literal>recordId</literal> entry
+ for the group of files that you index. To add a set of records you use
+ <literal>zebraidx</literal> with the <literal>update</literal> command. The
+ <literal>update</literal> command will always add all of the records that it
+ encounters to the index - whether they have already been indexed or
+ not. If the set of indexed files change, you should delete all of the
+ index files, and build a new index from scratch.
+ </para>
+
+ <para>
+ Consider a system in which you have a group of text files called
+ <literal>simple</literal>.
+ That group of records should belong to a Z39.50 database called
+ <literal>textbase</literal>.
+ The following <literal>zebra.cfg</literal> file will suffice:
+ </para>
+ <para>
+
+ <screen>
+ profilePath: /usr/local/yaz
+ attset: bib1.att
+ simple.recordType: text
+ simple.database: textbase
+ </screen>
+
+ </para>
+
+ <para>
+ Since the existing records in an index can not be addressed by their
+ IDs, it is impossible to delete or modify records when using this method.
+ </para>
+
+ </sect1>
+
+ <sect1 id="file-ids">
+ <title>Indexing with File Record IDs</title>
+
+ <para>
+ If you have a set of files that regularly change over time: Old files
+ are deleted, new ones are added, or existing files are modified, you
+ can benefit from using the <emphasis>file ID</emphasis>
+ indexing methodology.
+ Examples of this type of database might include an index of WWW
+ resources, or a USENET news spool area.
+ Briefly speaking, the file key methodology uses the directory paths
+ of the individual records as a unique identifier for each record.
+ To perform indexing of a directory with file keys, again, you specify
+ the top-level directory after the <literal>update</literal> command.
+ The command will recursively traverse the directories and compare
+ each one with whatever have been indexed before in that same directory.
+ If a file is new (not in the previous version of the directory) it
+ is inserted into the registers; if a file was already indexed and
+ it has been modified since the last update, the index is also
+ modified; if a file has been removed since the last
+ visit, it is deleted from the index.
+ </para>
+
+ <para>
+ The resulting system is easy to administrate. To delete a record you
+ simply have to delete the corresponding file (say, with the
+ <literal>rm</literal> command). And to add records you create new
+ files (or directories with files). For your changes to take effect
+ in the register you must run <literal>zebraidx update</literal> with
+ the same directory root again. This mode of operation requires more
+ disk space than simpler indexing methods, but it makes it easier for
+ you to keep the index in sync with a frequently changing set of data.
+ If you combine this system with the <emphasis>safe update</emphasis>
+ facility (see below), you never have to take your server offline for
+ maintenance or register updating purposes.
+ </para>
+
+ <para>
+ To enable indexing with pathname IDs, you must specify
+ <literal>file</literal> as the value of <literal>recordId</literal>
+ in the configuration file. In addition, you should set
+ <literal>storeKeys</literal> to <literal>1</literal>, since the Zebra
+ indexer must save additional information about the contents of each record
+ in order to modify the indices correctly at a later time.
+ </para>
+
+ <para>
+ For example, to update records of group <literal>esdd</literal>
+ located below
+ <literal>/data1/records/</literal> you should type:
+ <screen>
+ $ zebraidx -g esdd update /data1/records
+ </screen>
+ </para>
+
+ <para>
+ The corresponding configuration file includes:
+ <screen>
+ esdd.recordId: file
+ esdd.recordType: grs.sgml
+ esdd.storeKeys: 1
+ </screen>
+ </para>
+
+ <note>
+ <para>You cannot start out with a group of records with simple
+ indexing (no record IDs as in the previous section) and then later
+ enable file record Ids. Zebra must know from the first time that you
+ index the group that
+ the files should be indexed with file record IDs.
+ </para>
+ </note>
+
+ <para>
+ You cannot explicitly delete records when using this method (using the
+ <literal>delete</literal> command to <literal>zebraidx</literal>. Instead
+ you have to delete the files from the file system (or move them to a
+ different location)
+ and then run <literal>zebraidx</literal> with the
+ <literal>update</literal> command.
+ </para>
</sect1>
+
+ <sect1 id="generic-ids">
+ <title>Indexing with General Record IDs</title>
+
+ <para>
+ When using this method you construct an (almost) arbritrary, internal
+ record key based on the contents of the record itself and other system
+ information. If you have a group of records that explicitly associates
+ an ID with each record, this method is convenient. For example, the
+ record format may contain a title or a ID-number - unique within the group.
+ In either case you specify the Z39.50 attribute set and use-attribute
+ location in which this information is stored, and the system looks at
+ that field to determine the identity of the record.
+ </para>
+
+ <para>
+ As before, the record ID is defined by the <literal>recordId</literal>
+ setting in the configuration file. The value of the record ID specification
+ consists of one or more tokens separated by whitespace. The resulting
+ ID is represented in the index by concatenating the tokens and
+ separating them by ASCII value (1).
+ </para>
+
+ <para>
+ There are three kinds of tokens:
+ <variablelist>
+
+ <varlistentry>
+ <term>Internal record info</term>
+ <listitem>
+ <para>
+ The token refers to a key that is
+ extracted from the record. The syntax of this token is
+ <literal>(</literal> <emphasis>set</emphasis> <literal>,</literal>
+ <emphasis>use</emphasis> <literal>)</literal>,
+ where <emphasis>set</emphasis> is the
+ attribute set name <emphasis>use</emphasis> is the
+ name or value of the attribute.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term>System variable</term>
+ <listitem>
+ <para>
+ The system variables are preceded by
+
+ <screen>
+ $
+ </screen>
+ and immediately followed by the system variable name, which
+ may one of
+ <variablelist>
+
+ <varlistentry>
+ <term>group</term>
+ <listitem>
+ <para>
+ Group name.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term>database</term>
+ <listitem>
+ <para>
+ Current database specified.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term>type</term>
+ <listitem>
+ <para>
+ Record type.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term>Constant string</term>
+ <listitem>
+ <para>
+ A string used as part of the ID — surrounded
+ by single- or double quotes.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
+
+ <para>
+ For instance, the sample GILS records that come with the Zebra
+ distribution contain a unique ID in the data tagged Control-Identifier.
+ The data is mapped to the Bib-1 use attribute Identifier-standard
+ (code 1007). To use this field as a record id, specify
+ <literal>(bib1,Identifier-standard)</literal> as the value of the
+ <literal>recordId</literal> in the configuration file.
+ If you have other record types that uses the same field for a
+ different purpose, you might add the record type
+ (or group or database name) to the record id of the gils
+ records as well, to prevent matches with other types of records.
+ In this case the recordId might be set like this:
+
+ <screen>
+ gils.recordId: $type (bib1,Identifier-standard)
+ </screen>
+
+ </para>
+
+ <para>
+ (see section <xref linkend="data-model"/>
+ for details of how the mapping between elements of your records and
+ searchable attributes is established).
+ </para>
+
+ <para>
+ As for the file record ID case described in the previous section,
+ updating your system is simply a matter of running
+ <literal>zebraidx</literal>
+ with the <literal>update</literal> command. However, the update with general
+ keys is considerably slower than with file record IDs, since all files
+ visited must be (re)read to discover their IDs.
+ </para>
+
+ <para>
+ As you might expect, when using the general record IDs
+ method, you can only add or modify existing records with the
+ <literal>update</literal> command.
+ If you wish to delete records, you must use the,
+ <literal>delete</literal> command, with a directory as a parameter.
+ This will remove all records that match the files below that root
+ directory.
+ </para>
+
+ </sect1>
+
+ <sect1 id="register-location">
+ <title>Register Location</title>
+
+ <para>
+ Normally, the index files that form dictionaries, inverted
+ files, record info, etc., are stored in the directory where you run
+ <literal>zebraidx</literal>. If you wish to store these, possibly large,
+ files somewhere else, you must add the <literal>register</literal>
+ entry to the <literal>zebra.cfg</literal> file.
+ Furthermore, the Zebra system allows its file
+ structures to span multiple file systems, which is useful for
+ managing very large databases.
+ </para>
+
+ <para>
+ The value of the <literal>register</literal> setting is a sequence
+ of tokens. Each token takes the form:
+
+ <screen>
+ <emphasis>dir</emphasis><literal>:</literal><emphasis>size</emphasis>.
+ </screen>
+
+ The <emphasis>dir</emphasis> specifies a directory in which index files
+ will be stored and the <emphasis>size</emphasis> specifies the maximum
+ size of all files in that directory. The Zebra indexer system fills
+ each directory in the order specified and use the next specified
+ directories as needed.
+ The <emphasis>size</emphasis> is an integer followed by a qualifier
+ code, <literal>M</literal> for megabytes,
+ <literal>k</literal> for kilobytes.
+ </para>
+
+ <para>
+ For instance, if you have allocated two disks for your register, and
+ the first disk is mounted
+ on <literal>/d1</literal> and has 200 Mb of free space and the
+ second, mounted on <literal>/d2</literal> has 300 Mb, you could
+ put this entry in your configuration file:
+
+ <screen>
+ register: /d1:200M /d2:300M
+ </screen>
+
+ </para>
+
+ <para>
+ Note that Zebra does not verify that the amount of space specified is
+ actually available on the directory (file system) specified - it is
+ your responsibility to ensure that enough space is available, and that
+ other applications do not attempt to use the free space. In a large
+ production system, it is recommended that you allocate one or more
+ filesystem exclusively to the Zebra register files.
+ </para>
+
+ </sect1>
+
+ <sect1 id="shadow-registers">
+ <title>Safe Updating - Using Shadow Registers</title>
+
+ <sect2>
+ <title>Description</title>
+
+ <para>
+ The Zebra server supports <emphasis>updating</emphasis> of the index
+ structures. That is, you can add, modify, or remove records from
+ databases managed by Zebra without rebuilding the entire index.
+ Since this process involves modifying structured files with various
+ references between blocks of data in the files, the update process
+ is inherently sensitive to system crashes, or to process interruptions:
+ Anything but a successfully completed update process will leave the
+ register files in an unknown state, and you will essentially have no
+ recourse but to re-index everything, or to restore the register files
+ from a backup medium.
+ Further, while the update process is active, users cannot be
+ allowed to access the system, as the contents of the register files
+ may change unpredictably.
+ </para>
+
+ <para>
+ You can solve these problems by enabling the shadow register system in
+ Zebra.
+ During the updating procedure, <literal>zebraidx</literal> will temporarily
+ write changes to the involved files in a set of "shadow
+ files", without modifying the files that are accessed by the
+ active server processes. If the update procedure is interrupted by a
+ system crash or a signal, you simply repeat the procedure - the
+ register files have not been changed or damaged, and the partially
+ written shadow files are automatically deleted before the new updating
+ procedure commences.
+ </para>
+
+ <para>
+ At the end of the updating procedure (or in a separate operation, if
+ you so desire), the system enters a "commit mode". First,
+ any active server processes are forced to access those blocks that
+ have been changed from the shadow files rather than from the main
+ register files; the unmodified blocks are still accessed at their
+ normal location (the shadow files are not a complete copy of the
+ register files - they only contain those parts that have actually been
+ modified). If the commit process is interrupted at any point during the
+ commit process, the server processes will continue to access the
+ shadow files until you can repeat the commit procedure and complete
+ the writing of data to the main register files. You can perform
+ multiple update operations to the registers before you commit the
+ changes to the system files, or you can execute the commit operation
+ at the end of each update operation. When the commit phase has
+ completed successfully, any running server processes are instructed to
+ switch their operations to the new, operational register, and the
+ temporary shadow files are deleted.
+ </para>
+
+ </sect2>
+
+ <sect2>
+ <title>How to Use Shadow Register Files</title>
+
+ <para>
+ The first step is to allocate space on your system for the shadow
+ files.
+ You do this by adding a <literal>shadow</literal> entry to the
+ <literal>zebra.cfg</literal> file.
+ The syntax of the <literal>shadow</literal> entry is exactly the
+ same as for the <literal>register</literal> entry
+ (see section <xref linkend="register-location"/>).
+ The location of the shadow area should be
+ <emphasis>different</emphasis> from the location of the main register
+ area (if you have specified one - remember that if you provide no
+ <literal>register</literal> setting, the default register area is the
+ working directory of the server and indexing processes).
+ </para>
+
+ <para>
+ The following excerpt from a <literal>zebra.cfg</literal> file shows
+ one example of a setup that configures both the main register
+ location and the shadow file area.
+ Note that two directories or partitions have been set aside
+ for the shadow file area. You can specify any number of directories
+ for each of the file areas, but remember that there should be no
+ overlaps between the directories used for the main registers and the
+ shadow files, respectively.
+ </para>
+ <para>
+
+ <screen>
+ register: /d1:500M
+
+ shadow: /scratch1:100M /scratch2:200M
+ </screen>
+
+ </para>
+
+ <para>
+ When shadow files are enabled, an extra command is available at the
+ <literal>zebraidx</literal> command line.
+ In order to make changes to the system take effect for the
+ users, you'll have to submit a "commit" command after a
+ (sequence of) update operation(s).
+ You can ask the indexer to commit the changes immediately
+ after the update operation:
+ </para>
+
+ <para>
+
+ <screen>
+ $ zebraidx update /d1/records update /d2/more-records commit
+ </screen>
+
+ </para>
+
+ <para>
+ Or you can execute multiple updates before committing the changes:
+ </para>
+
+ <para>
+
+ <screen>
+ $ zebraidx -g books update /d1/records update /d2/more-records
+ $ zebraidx -g fun update /d3/fun-records
+ $ zebraidx commit
+ </screen>
+
+ </para>
+
+ <para>
+ If one of the update operations above had been interrupted, the commit
+ operation on the last line would fail: <literal>zebraidx</literal>
+ will not let you commit changes that would destroy the running register.
+ You'll have to rerun all of the update operations since your last
+ commit operation, before you can commit the new changes.
+ </para>
+
+ <para>
+ Similarly, if the commit operation fails, <literal>zebraidx</literal>
+ will not let you start a new update operation before you have
+ successfully repeated the commit operation.
+ The server processes will keep accessing the shadow files rather
+ than the (possibly damaged) blocks of the main register files
+ until the commit operation has successfully completed.
+ </para>
+
+ <para>
+ You should be aware that update operations may take slightly longer
+ when the shadow register system is enabled, since more file access
+ operations are involved. Further, while the disk space required for
+ the shadow register data is modest for a small update operation, you
+ may prefer to disable the system if you are adding a very large number
+ of records to an already very large database (we use the terms
+ <emphasis>large</emphasis> and <emphasis>modest</emphasis>
+ very loosely here, since every application will have a
+ different perception of size).
+ To update the system without the use of the the shadow files,
+ simply run <literal>zebraidx</literal> with the <literal>-n</literal>
+ option (note that you do not have to execute the
+ <emphasis>commit</emphasis> command of <literal>zebraidx</literal>
+ when you temporarily disable the use of the shadow registers in
+ this fashion.
+ Note also that, just as when the shadow registers are not enabled,
+ server processes will be barred from accessing the main register
+ while the update procedure takes place.
+ </para>
+
+ </sect2>
+
+ </sect1>
+
+</chapter>
-<sect1 id="register-location">
-<title>Register Location</title>
-
-<para>
-Normally, the index files that form dictionaries, inverted
-files, record info, etc., are stored in the directory where you run
-<literal remap="tt">zebraidx</literal>. If you wish to store these, possibly large, files
-somewhere else, you must add the <literal remap="tt">register</literal> entry to the
-<literal remap="tt">zebra.cfg</literal> file. Furthermore, the Zebra system allows its file
-structures to
-span multiple file systems, which is useful for managing very large
-databases.
-</para>
-
-<para>
-The value of the <literal remap="tt">register</literal> setting is a sequence of tokens.
-Each token takes the form:
-
-<screen>
-<emphasis>dir</emphasis><literal remap="tt">:</literal><emphasis>size</emphasis>.
-</screen>
-
-The <emphasis>dir</emphasis> specifies a directory in which index files will be
-stored and the <emphasis>size</emphasis> specifies the maximum size of all
-files in that directory. The Zebra indexer system fills each directory
-in the order specified and use the next specified directories as needed.
-The <emphasis>size</emphasis> is an integer followed by a qualifier
-code, <literal remap="tt">M</literal> for megabytes, <literal remap="tt">k</literal> for kilobytes.
-</para>
+<chapter id="zebraidx">
+ <title>Running the Maintenance Interface (zebraidx)</title>
+
+ <para>
+ The following is a complete reference to the command line interface to
+ the <literal>zebraidx</literal> application.
+ </para>
+
+ <para>
+ Syntax
+
+ <screen>
+ $ zebraidx [options] command [directory] ...
+ </screen>
+
+ Options:
+ <variablelist>
+
+ <varlistentry>
+ <term>-t <replaceable>type</replaceable></term>
+ <listitem>
+ <para>
+ Update all files as <replaceable>type</replaceable>. Currently, the
+ types supported are <literal>text</literal> and
+ <literal>grs</literal><replaceable>.subtype</replaceable>.
+ If no <replaceable>subtype</replaceable> is provided for the GRS
+ (General Record Structure) type, the canonical input format
+ is assumed (see section <xref linkend="local-representation"/>).
+ Generally, it is probably advisable to specify the record types
+ in the <literal>zebra.cfg</literal> file (see section
+ <xref linkend="record-types"/>), to avoid confusion at
+ subsequent updates.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term>-c <replaceable>config-file</replaceable></term>
+ <listitem>
+ <para>
+ Read the configuration file
+ <replaceable>config-file</replaceable> instead of
+ <literal>zebra.cfg</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term>-g <replaceable>group</replaceable></term>
+ <listitem>
+ <para>
+ Update the files according to the group
+ settings for <replaceable>group</replaceable> (see section
+ <xref linkend="configuration-file"/>).
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term>-d <replaceable>database</replaceable></term>
+ <listitem>
+ <para>
+ The records located should be associated with the database name
+ <replaceable>database</replaceable> for access through the Z39.50 server.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term>-m <replaceable>mbytes</replaceable></term>
+ <listitem>
+ <para>
+ Use <replaceable>mbytes</replaceable> of megabytes before flushing
+ keys to background storage. This setting affects performance when
+ updating large databases.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term>-n</term>
+ <listitem>
+ <para>
+ Disable the use of shadow registers for this operation
+ (see section <xref linkend="shadow-registers"/>).
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term>-s</term>
+ <listitem>
+ <para>
+ Show analysis of the indexing process. The maintenance
+ program works in a read-only mode and doesn't change the state
+ of the index. This options is very useful when you wish to test a
+ new profile.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term>-V</term>
+ <listitem>
+ <para>
+ Show Zebra version.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term>-v <replaceable>level</replaceable></term>
+ <listitem>
+ <para>
+ Set the log level to <replaceable>level</replaceable>.
+ <replaceable>level</replaceable> should be one of
+ <literal>none</literal>, <literal>debug</literal>, and
+ <literal>all</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
+
+ <para>
+ Commands
+ <variablelist>
+
+ <varlistentry>
+ <term>update <replaceable>directory</replaceable></term>
+ <listitem>
+ <para>
+ Update the register with the files contained in
+ <replaceable>directory</replaceable>.
+ If no directory is provided, a list of files is read from
+ <literal>stdin</literal>.
+ See section <xref linkend="administration"/>.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term>delete <replaceable>directory</replaceable></term>
+ <listitem>
+ <para>
+ Remove the records corresponding to the files found under
+ <replaceable>directory</replaceable> from the register.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term>commit</term>
+ <listitem>
+ <para>
+ Write the changes resulting from the last <literal>update</literal>
+ commands to the register. This command is only available if the use of
+ shadow register files is enabled (see section
+ <xref linkend="shadow-registers"/>).
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
-<para>
-For instance, if you have allocated two disks for your register, and
-the first disk is mounted
-on <literal remap="tt">/d1</literal> and has 200 Mb of free space and the
-second, mounted on <literal remap="tt">/d2</literal> has 300 Mb, you could
-put this entry in your configuration file:
+</chapter>
-<screen>
-register: /d1:200M /d2:300M
-</screen>
+<chapter id="server">
+ <title>The Z39.50 Server</title>
+
+ <sect1 id="zebrasrv">
+ <title>Running the Z39.50 Server (zebrasrv)</title>
+
+ <para>
+ <emphasis remap="bf">Syntax</emphasis>
+
+ <screen>
+ zebrasrv [options] [listener-address ...]
+ </screen>
+
+ </para>
+
+ <para>
+ <emphasis remap="bf">Options</emphasis>
+ <variablelist>
+
+ <varlistentry>
+ <term>-a <replaceable>APDU file</replaceable></term>
+ <listitem>
+ <para>
+ Specify a file for dumping PDUs (for diagnostic purposes).
+ The special name "-" sends output to <literal>stderr</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term>-c <replaceable>config-file</replaceable></term>
+ <listitem>
+ <para>
+ Read configuration information from
+ <replaceable>config-file</replaceable>.
+ The default configuration is <literal>./zebra.cfg</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term>-S</term>
+ <listitem>
+ <para>
+ Don't fork on connection requests. This can be useful for
+ symbolic-level debugging. The server can only accept a single
+ connection in this mode.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term>-s</term>
+ <listitem>
+ <para>
+ Use the SR protocol.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term>-z</term>
+ <listitem>
+ <para>
+ Use the Z39.50 protocol (default). These two options complement
+ eachother. You can use both multiple times on the same command
+ line, between listener-specifications (see below). This way, you
+ can set up the server to listen for connections in both protocols
+ concurrently, on different local ports.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term>-l <replaceable>logfile</replaceable></term>
+ <listitem>
+ <para>
+ Specify an output file for the diagnostic messages.
+ The default is to write this information to <literal>stderr</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term>-v <replaceable>log-level</replaceable></term>
+ <listitem>
+ <para>
+ The log level. Use a comma-separated list of members of the set
+ {fatal,debug,warn,log,all,none}.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term>-u <replaceable>username</replaceable></term>
+ <listitem>
+ <para>
+ Set user ID. Sets the real UID of the server process to that of the
+ given <replaceable>username</replaceable>.
+ It's useful if you aren't comfortable with having the
+ server run as root, but you need to start it as such to bind a
+ privileged port.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term>-w <replaceable>working-directory</replaceable></term>
+ <listitem>
+ <para>
+ Change working directory.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term>-i</term>
+ <listitem>
+ <para>
+ Run under the Internet superserver, <literal>inetd</literal>.
+ Make sure you use the logfile option <literal>-l</literal> in
+ conjunction with this mode and specify the <literal>-l</literal>
+ option before any other options.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term>-t <replaceable>timeout</replaceable></term>
+ <listitem>
+ <para>
+ Set the idle session timeout (default 60 minutes).
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term>-k <replaceable>kilobytes</replaceable></term>
+ <listitem>
+ <para>
+ Set the (approximate) maximum size of
+ present response messages. Default is 1024 Kb (1 Mb).
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
+
+ <para>
+ A <replaceable>listener-address</replaceable> consists of a transport
+ mode followed by a colon (:) followed by a listener address.
+ The transport mode is either <literal>ssl</literal> or
+ <literal>tcp</literal>.
+ </para>
+
+ <para>
+ For TCP, an address has the form
+ </para>
+
+ <para>
+
+ <screen>
+ hostname | IP-number [: portnumber]
+ </screen>
+
+ </para>
+
+ <para>
+ The port number defaults to 210 (standard Z39.50 port).
+ </para>
+
+ <para>
+ Examples
+ </para>
+
+ <para>
+
+ <screen>
+ tcp:dranet.dra.com
+
+ ssl:secure.lib.com:3000
+ </screen>
+
+ </para>
+
+ <para>
+ In both cases, the special hostname "@" is mapped to
+ the address INADDR_ANY, which causes the server to listen on any local
+ interface. To start the server listening on the registered port for
+ Z39.50, and to drop root privileges once the ports are bound, execute
+ the server like this (from a root shell):
+ </para>
+
+ <para>
+
+ <screen>
+ zebrasrv -u daemon tcp:@
+ </screen>
+
+ </para>
+
+ <para>
+ You can replace <literal>daemon</literal> with another user, eg.
+ your own account, or a dedicated IR server account.
+ </para>
+
+ <para>
+ The default behavior for <literal>zebrasrv</literal> is to establish
+ a single TCP/IP listener, for the Z39.50 protocol, on port 9999.
+ </para>
+
+ </sect1>
+
+ <sect1 id="protocol-support">
+ <title>Z39.50 Protocol Support and Behavior</title>
+
+ <sect2>
+ <title>Initialization</title>
+
+ <para>
+ During initialization, the server will negotiate to version 3 of the
+ Z39.50 protocol, and the option bits for Search, Present, Scan,
+ NamedResultSets, and concurrentOperations will be set, if requested by
+ the client. The maximum PDU size is negotiated down to a maximum of
+ 1Mb by default.
+ </para>
+
+ </sect2>
+
+ <sect2 id="search">
+ <title>Search</title>
+
+ <para>
+ The supported query type are 1 and 101. All operators are currently
+ supported with the restriction that only proximity units of type "word"
+ are supported for the proximity operator.
+ Queries can be arbitrarily complex.
+ Named result sets are supported, and result sets can be used as operands
+ without limitations.
+ Searches may span multiple databases.
+ </para>
+
+ <para>
+ The server has full support for piggy-backed present requests (see
+ also the following section).
+ </para>
+
+ <para>
+ <emphasis>Use</emphasis> attributes are interpreted according to the
+ attribute sets which have been loaded in the
+ <literal>zebra.cfg</literal> file, and are matched against specific
+ fields as specified in the <literal>.abs</literal> file which
+ describes the profile of the records which have been loaded.
+ If no Use attribute is provided, a default of Bib-1 Any is assumed.
+ </para>
+
+ <para>
+ If a <emphasis>Structure</emphasis> attribute of
+ <emphasis>Phrase</emphasis> is used in conjunction with a
+ <emphasis>Completeness</emphasis> attribute of
+ <emphasis>Complete (Sub)field</emphasis>, the term is matched
+ against the contents of the phrase (long word) register, if one
+ exists for the given <emphasis>Use</emphasis> attribute.
+ A phrase register is created for those fields in the
+ <literal>.abs</literal> file that contains a
+ <literal>p</literal>-specifier.
+ </para>
+
+ <para>
+ If <emphasis>Structure</emphasis>=<emphasis>Phrase</emphasis> is
+ used in conjunction with <emphasis>Incomplete Field</emphasis> - the
+ default value for <emphasis>Completeness</emphasis>, the
+ search is directed against the normal word registers, but if the term
+ contains multiple words, the term will only match if all of the words
+ are found immediately adjacent, and in the given order.
+ The word search is performed on those fields that are indexed as
+ type <literal>w</literal> in the <literal>.abs</literal> file.
+ </para>
+
+ <para>
+ If the <emphasis>Structure</emphasis> attribute is
+ <emphasis>Word List</emphasis>,
+ <emphasis>Free-form Text</emphasis>, or
+ <emphasis>Document Text</emphasis>, the term is treated as a
+ natural-language, relevance-ranked query.
+ This search type uses the word register, i.e. those fields
+ that are indexed as type <literal>w</literal> in the
+ <literal>.abs</literal> file.
+ </para>
+
+ <para>
+ If the <emphasis>Structure</emphasis> attribute is
+ <emphasis>Numeric String</emphasis> the term is treated as an integer.
+ The search is performed on those fields that are indexed
+ as type <literal>n</literal> in the <literal>.abs</literal> file.
+ </para>
+
+ <para>
+ If the <emphasis>Structure</emphasis> attribute is
+ <emphasis>URx</emphasis> the term is treated as a URX (URL) entity.
+ The search is performed on those fields that are indexed as type
+ <literal>u</literal> in the <literal>.abs</literal> file.
+ </para>
+
+ <para>
+ If the <emphasis>Structure</emphasis> attribute is
+ <emphasis>Local Number</emphasis> the term is treated as
+ native Zebra Record Identifier.
+ </para>
+
+ <para>
+ If the <emphasis>Relation</emphasis> attribute is
+ <emphasis>Equals</emphasis> (default), the term is matched
+ in a normal fashion (modulo truncation and processing of
+ individual words, if required).
+ If <emphasis>Relation</emphasis> is <emphasis>Less Than</emphasis>,
+ <emphasis>Less Than or Equal</emphasis>,
+ <emphasis>Greater than</emphasis>, or <emphasis>Greater than or
+ Equal</emphasis>, the term is assumed to be numerical, and a
+ standard regular expression is constructed to match the given
+ expression.
+ If <emphasis>Relation</emphasis> is <emphasis>Relevance</emphasis>,
+ the standard natural-language query processor is invoked.
+ </para>
+
+ <para>
+ For the <emphasis>Truncation</emphasis> attribute,
+ <emphasis>No Truncation</emphasis> is the default.
+ <emphasis>Left Truncation</emphasis> is not supported.
+ <emphasis>Process #</emphasis> is supported, as is
+ <emphasis>Regxp-1</emphasis>.
+ <emphasis>Regxp-2</emphasis> enables the fault-tolerant (fuzzy)
+ search. As a default, a single error (deletion, insertion,
+ replacement) is accepted when terms are matched against the register
+ contents.
+ </para>
+
+ <sect3>
+ <title>Regular expressions</title>
+
+ <para>
+ Each term in a query is interpreted as a regular expression if
+ the truncation value is either <emphasis>Regxp-1</emphasis> (102)
+ or <emphasis>Regxp-2</emphasis> (103).
+ Both query types follow the same syntax with the operands:
+ <variablelist>
+
+ <varlistentry>
+ <term>x</term>
+ <listitem>
+ <para>
+ Matches the character <emphasis>x</emphasis>.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term>.</term>
+ <listitem>
+ <para>
+ Matches any character.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term><literal>[</literal>..<literal>]</literal></term>
+ <listitem>
+ <para>
+ Matches the set of characters specified;
+ such as <literal>[abc]</literal> or <literal>[a-c]</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ and the operators:
+ <variablelist>
+
+ <varlistentry>
+ <term>x*</term>
+ <listitem>
+ <para>
+ Matches <emphasis>x</emphasis> zero or more times. Priority: high.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term>x+</term>
+ <listitem>
+ <para>
+ Matches <emphasis>x</emphasis> one or more times. Priority: high.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term>x?</term>
+ <listitem>
+ <para>
+ Matches <emphasis>x</emphasis> once or twice. Priority: high.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term>xy</term>
+ <listitem>
+ <para>
+ Matches <emphasis>x</emphasis>, then <emphasis>y</emphasis>.
+ Priority: medium.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term>x|y</term>
+ <listitem>
+ <para>
+ Matches either <emphasis>x</emphasis> or <emphasis>y</emphasis>.
+ Priority: low.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ The order of evaluation may be changed by using parentheses.
+ </para>
+
+ <para>
+ If the first character of the <emphasis>Regxp-2</emphasis> query
+ is a plus character (<literal>+</literal>) it marks the
+ beginning of a section with non-standard specifiers.
+ The next plus character marks the end of the section.
+ Currently Zebra only supports one specifier, the error tolerance,
+ which consists one digit.
+ </para>
+
+ <para>
+ Since the plus operator is normally a suffix operator the addition to
+ the query syntax doesn't violate the syntax for standard regular
+ expressions.
+ </para>
+
+ </sect3>
+
+ <sect3>
+ <title>Query examples</title>
+
+ <para>
+ Phrase search for <emphasis>information retrieval</emphasis> in
+ the title-register:
+ <screen>
+ @attr 1=4 "information retrieval"
+ </screen>
+ </para>
+
+ <para>
+ Ranked search for the same thing:
+ <screen>
+ @attr 1=4 @attr 2=102 "Information retrieval"
+ </screen>
+ </para>
+
+ <para>
+ Phrase search with a regular expression:
+ <screen>
+ @attr 1=4 @attr 5=102 "informat.* retrieval"
+ </screen>
+ </para>
+
+ <para>
+ Ranked search with a regular expression:
+ <screen>
+ @attr 1=4 @attr 5=102 @attr 2=102 "informat.* retrieval"
+ </screen>
+ </para>
+
+ <para>
+ In the GILS schema (<literal>gils.abs</literal>), the
+ west-bounding-coordinate is indexed as type <literal>n</literal>,
+ and is therefore searched by specifying
+ <emphasis>structure</emphasis>=<emphasis>Numeric String</emphasis>.
+ To match all those records with west-bounding-coordinate greater
+ than -114 we use the following query:
+ <screen>
+ @attr 4=109 @attr 2=5 @attr gils 1=2038 -114
+ </screen>
+ </para>
+ </sect3>
+ </sect2>
+
+ <sect2>
+ <title>Present</title>
+ <para>
+ The present facility is supported in a standard fashion. The requested
+ record syntax is matched against the ones supported by the profile of
+ each record retrieved. If no record syntax is given, SUTRS is the
+ default. The requested element set name, again, is matched against any
+ provided by the relevant record profiles.
+ </para>
+ </sect2>
+ <sect2>
+ <title>Scan</title>
+ <para>
+ The attribute combinations provided with the termListAndStartPoint are
+ processed in the same way as operands in a query (see above).
+ Currently, only the term and the globalOccurrences are returned with
+ the termInfo structure.
+ </para>
+ </sect2>
+ <sect2>
+ <title>Sort</title>
+
+ <para>
+ Z39.50 specifies three diffent types of sort criterias.
+ Of these Zebra supports the attribute specification type in which
+ case the use attribute specifies the "Sort register".
+ Sort registers are created for those fields that are of type "sort" in
+ the default.idx file.
+ The corresponding character mapping file in default.idx specifies the
+ ordinal of each character used in the actual sort.
+ </para>
+
+ <para>
+ Z39.50 allows the client to specify sorting on one or more input
+ result sets and one output result set.
+ Zebra supports sorting on one result set only which may or may not
+ be the same as the output result set.
+ </para>
+ </sect2>
+ <sect2>
+ <title>Close</title>
+ <para>
+ If a Close PDU is received, the server will respond with a Close PDU
+ with reason=FINISHED, no matter which protocol version was negotiated
+ during initialization. If the protocol version is 3 or more, the
+ server will generate a Close PDU under certain circumstances,
+ including a session timeout (60 minutes by default), and certain kinds of
+ protocol errors. Once a Close PDU has been sent, the protocol
+ association is considered broken, and the transport connection will be
+ closed immediately upon receipt of further data, or following a short
+ timeout.
+ </para>
+ </sect2>
+ </sect1>
+</chapter>
-</para>
+<chapter id="record-model">
+ <title>The Record Model</title>
+
+ <para>
+ The Zebra system is designed to support a wide range of data management
+ applications. The system can be configured to handle virtually any
+ kind of structured data. Each record in the system is associated with
+ a <emphasis>record schema</emphasis> which lends context to the data
+ elements of the record.
+ Any number of record schema can coexist in the system.
+ Although it may be wise to use only a single schema within
+ one database, the system poses no such restrictions.
+ </para>
+
+ <para>
+ The record model described in this chapter applies to the fundamental,
+ structured
+ record type <literal>grs</literal> as introduced in
+ section <xref linkend="record-types"/>.
+ </para>
+
+ <para>
+ Records pass through three different states during processing in the
+ system.
+ </para>
+
+ <para>
+
+ <itemizedlist>
+ <listitem>
+
+ <para>
+ When records are accessed by the system, they are represented
+ in their local, or native format. This might be SGML or HTML files,
+ News or Mail archives, MARC records. If the system doesn't already
+ know how to read the type of data you need to store, you can set up an
+ input filter by preparing conversion rules based on regular
+ expressions and possibly augmented by a flexible scripting language
+ (Tcl).
+ The input filter produces as output an internal representation:
+
+ </para>
+ </listitem>
+ <listitem>
+
+ <para>
+ When records are processed by the system, they are represented
+ in a tree-structure, constructed by tagged data elements hanging off a
+ root node. The tagged elements may contain data or yet more tagged
+ elements in a recursive structure. The system performs various
+ actions on this tree structure (indexing, element selection, schema
+ mapping, etc.),
+
+ </para>
+ </listitem>
+ <listitem>
+
+ <para>
+ Before transmitting records to the client, they are first
+ converted from the internal structure to a form suitable for exchange
+ over the network - according to the Z39.50 standard.
+ </para>
+ </listitem>
+
+ </itemizedlist>
+
+ </para>
+
+ <sect1 id="local-representation">
+ <title>Local Representation</title>
+
+ <para>
+ As mentioned earlier, Zebra places few restrictions on the type of
+ data that you can index and manage. Generally, whatever the form of
+ the data, it is parsed by an input filter specific to that format, and
+ turned into an internal structure that Zebra knows how to handle. This
+ process takes place whenever the record is accessed - for indexing and
+ retrieval.
+ </para>
+
+ <para>
+ The RecordType parameter in the <literal>zebra.cfg</literal> file, or
+ the <literal>-t</literal> option to the indexer tells Zebra how to
+ process input records.
+ Two basic types of processing are available - raw text and structured
+ data. Raw text is just that, and it is selected by providing the
+ argument <emphasis>text</emphasis> to Zebra. Structured records are
+ all handled internally using the basic mechanisms described in the
+ subsequent sections.
+ Zebra can read structured records in many different formats.
+ How this is done is governed by additional parameters after the
+ "grs" keyboard, separated by "." characters.
+ </para>
+
+ <para>
+ Three basic subtypes to the <emphasis>grs</emphasis> type are
+ currently available:
+ </para>
+
+ <para>
+ <variablelist>
+ <varlistentry>
+ <term>grs.sgml</term>
+ <listitem>
+ <para>
+ This is the canonical input format —
+ described below. It is a simple SGML-like syntax.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term>grs.regx.<emphasis>filter</emphasis></term>
+ <listitem>
+ <para>
+ This enables a user-supplied input
+ filter. The mechanisms of these filters are described below.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term>grs.marc.<emphasis>abstract syntax</emphasis></term>
+ <listitem>
+ <para>
+ This allows Zebra to read
+ records in the ISO2709 (MARC) encoding standard. In this case, the
+ last paramemeter <emphasis>abstract syntax</emphasis> names the
+ <literal>.abs</literal> file (see below)
+ which describes the specific MARC structure of the input record as
+ well as the indexing rules.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
+
+ <sect2>
+ <title>Canonical Input Format</title>
+
+ <para>
+ Although input data can take any form, it is sometimes useful to
+ describe the record processing capabilities of the system in terms of
+ a single, canonical input format that gives access to the full
+ spectrum of structure and flexibility in the system. In Zebra, this
+ canonical format is an "SGML-like" syntax.
+ </para>
+
+ <para>
+ To use the canonical format specify <literal>grs.sgml</literal> as
+ the record type.
+ </para>
+
+ <para>
+ Consider a record describing an information resource (such a record is
+ sometimes known as a <emphasis>locator record</emphasis>).
+ It might contain a field describing the distributor of the
+ information resource, which might in turn be partitioned into
+ various fields providing details about the distributor, like this:
+ </para>
+
+ <para>
+
+ <screen>
+ <Distributor>
+ <Name> USGS/WRD </Name>
+ <Organization> USGS/WRD </Organization>
+ <Street-Address>
+ U.S. GEOLOGICAL SURVEY, 505 MARQUETTE, NW
+ </Street-Address>
+ <City> ALBUQUERQUE </City>
+ <State> NM </State>
+ <Zip-Code> 87102 </Zip-Code>
+ <Country> USA </Country>
+ <Telephone> (505) 766-5560 </Telephone>
+ </Distributor>
+ </screen>
+
+ </para>
+
+ <note>
+ <para>
+ The indentation used above is used to illustrate how Zebra
+ interprets the markup. The indentation, in itself, has no
+ significance to the parser for the canonical input format, which
+ discards superfluous whitespace.
+ </para>
+ </note>
+ <para>
+ The keywords surrounded by <...> are
+ <emphasis>tags</emphasis>, while the sections of text
+ in between are the <emphasis>data elements</emphasis>.
+ A data element is characterized by its location in the tree
+ that is made up by the nested elements.
+ Each element is terminated by a closing tag - beginning
+ with <literal><</literal>/, and containing the same symbolic
+ tag-name as the corresponding opening tag.
+ The general closing tag - <literal><</literal>>/ -
+ terminates the element started by the last opening tag. The
+ structuring of elements is significant.
+ The element <emphasis>Telephone</emphasis>,
+ for instance, may be indexed and presented to the client differently,
+ depending on whether it appears inside the
+ <emphasis>Distributor</emphasis> element, or some other,
+ structured data element such a <emphasis>Supplier</emphasis> element.
+ </para>
+
+ <sect3>
+ <title>Record Root</title>
+
+ <para>
+ The first tag in a record describes the root node of the tree that
+ makes up the total record. In the canonical input format, the root tag
+ should contain the name of the schema that lends context to the
+ elements of the record (see section
+ <xref linkend="internal-representation"/>).
+ The following is a GILS record that
+ contains only a single element (strictly speaking, that makes it an
+ illegal GILS record, since the GILS profile includes several mandatory
+ elements - Zebra does not validate the contents of a record against
+ the Z39.50 profile, however - it merely attempts to match up elements
+ of a local representation with the given schema):
+ </para>
+
+ <para>
+
+ <screen>
+ <gils>
+ <title>Zen and the Art of Motorcycle Maintenance</title>
+ </gils>
+ </screen>
+
+ </para>
+
+ </sect3>
+
+ <sect3>
+ <title>Variants</title>
+
+ <para>
+ Zebra allows you to provide individual data elements in a number of
+ <emphasis>variant forms</emphasis>. Examples of variant forms are
+ textual data elements which might appear in different languages, and
+ images which may appear in different formats or layouts.
+ The variant system in Zebra is essentially a representation of
+ the variant mechanism of Z39.50-1995.
+ </para>
+
+ <para>
+ The following is an example of a title element which occurs in two
+ different languages.
+ </para>
+
+ <para>
+
+ <screen>
+ <title>
+ <var lang lang "eng">
+ Zen and the Art of Motorcycle Maintenance</>
+ <var lang lang "dan">
+ Zen og Kunsten at Vedligeholde en Motorcykel</>
+ </title>
+ </screen>
+
+ </para>
+
+ <para>
+ The syntax of the <emphasis>variant element</emphasis> is
+ <literal><var class type value></literal>.
+ The available values for the <emphasis>class</emphasis> and
+ <emphasis>type</emphasis> fields are given by the variant set
+ that is associated with the current schema
+ (see section <xref linkend="variant-set"/>).
+ </para>
+
+ <para>
+ Variant elements are terminated by the general end-tag </>, by
+ the variant end-tag </var>, by the appearance of another variant
+ tag with the same <emphasis>class</emphasis> and
+ <emphasis>value</emphasis> settings, or by the
+ appearance of another, normal tag. In other words, the end-tags for
+ the variants used in the example above could have been saved.
+ </para>
+
+ <para>
+ Variant elements can be nested. The element
+ </para>
+
+ <para>
+
+ <screen>
+ <title>
+ <var lang lang "eng"><var body iana "text/plain">
+ Zen and the Art of Motorcycle Maintenance
+ </title>
+ </screen>
-<para>
-Note that Zebra does not verify that the amount of space specified is
-actually available on the directory (file system) specified - it is
-your responsibility to ensure that enough space is available, and that
-other applications do not attempt to use the free space. In a large production system,
-it is recommended that you allocate one or more filesystem exclusively
-to the Zebra register files.
-</para>
+ </para>
-</sect1>
+ <para>
+ Associates two variant components to the variant list for the title
+ element.
+ </para>
-<sect1 id="shadow-registers">
-<title>Safe Updating - Using Shadow Registers</title>
+ <para>
+ Given the nesting rules described above, we could write
+ </para>
-<sect2>
-<title>Description</title>
+ <para>
-<para>
-The Zebra server supports <emphasis remap="it">updating</emphasis> of the index structures. That is,
-you can add, modify, or remove records from databases managed by Zebra
-without rebuilding the entire index. Since this process involves
-modifying structured files with various references between blocks of
-data in the files, the update process is inherently sensitive to
-system crashes, or to process interruptions: Anything but a
-successfully completed update process will leave the register files in
-an unknown state, and you will essentially have no recourse but to
-re-index everything, or to restore the register files from a backup
-medium. Further, while the update process is active, users cannot be
-allowed to access the system, as the contents of the register files
-may change unpredictably.
-</para>
-
-<para>
-You can solve these problems by enabling the shadow register system in
-Zebra. During the updating procedure, <literal remap="tt">zebraidx</literal> will temporarily
-write changes to the involved files in a set of "shadow
-files", without modifying the files that are accessed by the
-active server processes. If the update procedure is interrupted by a
-system crash or a signal, you simply repeat the procedure - the
-register files have not been changed or damaged, and the partially
-written shadow files are automatically deleted before the new updating
-procedure commences.
-</para>
-
-<para>
-At the end of the updating procedure (or in a separate operation, if
-you so desire), the system enters a "commit mode". First,
-any active server processes are forced to access those blocks that
-have been changed from the shadow files rather than from the main
-register files; the unmodified blocks are still accessed at their
-normal location (the shadow files are not a complete copy of the
-register files - they only contain those parts that have actually been
-modified). If the commit process is interrupted at any point during the
-commit process, the server processes will continue to access the
-shadow files until you can repeat the commit procedure and complete
-the writing of data to the main register files. You can perform
-multiple update operations to the registers before you commit the
-changes to the system files, or you can execute the commit operation
-at the end of each update operation. When the commit phase has
-completed successfully, any running server processes are instructed to
-switch their operations to the new, operational register, and the
-temporary shadow files are deleted.
-</para>
-
-</sect2>
-
-<sect2>
-<title>How to Use Shadow Register Files</title>
-
-<para>
-The first step is to allocate space on your system for the shadow
-files. You do this by adding a <literal remap="tt">shadow</literal> entry to the <literal remap="tt">zebra.cfg</literal>
-file. The syntax of the <literal remap="tt">shadow</literal> entry is exactly the same as for
-the <literal remap="tt">register</literal> entry (see section <xref linkend="register-location"/>). The location of the shadow area should be
-<emphasis remap="it">different</emphasis> from the location of the main register area (if you
-have specified one - remember that if you provide no <literal remap="tt">register</literal>
-setting, the default register area is the
-working directory of the server and indexing processes).
-</para>
-
-<para>
-The following excerpt from a <literal remap="tt">zebra.cfg</literal> file shows one example of
-a setup that configures both the main register location and the shadow
-file area. Note that two directories or partitions have been set aside
-for the shadow file area. You can specify any number of directories
-for each of the file areas, but remember that there should be no
-overlaps between the directories used for the main registers and the
-shadow files, respectively.
-</para>
-
-<para>
-
-<screen>
-register: /d1:500M
-
-shadow: /scratch1:100M /scratch2:200M
-</screen>
-
-</para>
-
-<para>
-When shadow files are enabled, an extra command is available at the
-<literal remap="tt">zebraidx</literal> command line. In order to make changes to the system
-take effect for the users, you'll have to submit a
-"commit" command after a (sequence of) update
-operation(s). You can ask the indexer to commit the changes
-immediately after the update operation:
-</para>
-
-<para>
-
-<screen>
-$ zebraidx update /d1/records update /d2/more-records commit
-</screen>
-
-</para>
-
-<para>
-Or you can execute multiple updates before committing the changes:
-</para>
-
-<para>
-
-<screen>
-$ zebraidx -g books update /d1/records update /d2/more-records
-$ zebraidx -g fun update /d3/fun-records
-$ zebraidx commit
-</screen>
-
-</para>
-
-<para>
-If one of the update operations above had been interrupted, the commit
-operation on the last line would fail: <literal remap="tt">zebraidx</literal> will not let you
-commit changes that would destroy the running register. You'll have to
-rerun all of the update operations since your last commit operation,
-before you can commit the new changes.
-</para>
-
-<para>
-Similarly, if the commit operation fails, <literal remap="tt">zebraidx</literal> will not let
-you start a new update operation before you have successfully repeated
-the commit operation. The server processes will keep accessing the
-shadow files rather than the (possibly damaged) blocks of the main
-register files until the commit operation has successfully completed.
-</para>
-
-<para>
-You should be aware that update operations may take slightly longer
-when the shadow register system is enabled, since more file access
-operations are involved. Further, while the disk space required for
-the shadow register data is modest for a small update operation, you
-may prefer to disable the system if you are adding a very large number
-of records to an already very large database (we use the terms
-<emphasis remap="it">large</emphasis> and <emphasis remap="it">modest</emphasis> very loosely here, since every
-application will have a different perception of size). To update the system
-without the use of the the shadow files, simply run <literal remap="tt">zebraidx</literal> with
-the <literal remap="tt">-n</literal> option (note that you do not have to execute the
-<emphasis remap="bf">commit</emphasis> command of <literal remap="tt">zebraidx</literal> when you temporarily disable the
-use of the shadow registers in this fashion. Note also that, just as
-when the shadow registers are not enabled, server processes will be
-barred from accessing the main register while the update procedure
-takes place.
-</para>
-
-</sect2>
-
-</sect1>
-
-</chapter>
-
-<chapter id="zebraidx">
-<title>Running the Maintenance Interface (zebraidx)</title>
-
-<para>
-The following is a complete reference to the command line interface to
-the <literal remap="tt">zebraidx</literal> application.
-</para>
-
-<para>
-<emphasis remap="bf">Syntax</emphasis>
-
-<screen>
-$ zebraidx [options] command [directory] ...
-</screen>
-
-<emphasis remap="bf">Options</emphasis>
-<variablelist>
-
-<varlistentry>
-<term>-t <emphasis remap="it">type</emphasis></term>
-<listitem>
-<para>
-Update all files as <emphasis remap="it">type</emphasis>. Currently, the
-types supported are <literal remap="tt">text</literal> and <literal remap="tt">grs</literal><emphasis remap="it">.subtype</emphasis>. If no
-<emphasis remap="it">subtype</emphasis> is provided for the GRS (General Record Structure) type,
-the canonical input format is assumed (see section <xref linkend="local-representation"/>). Generally, it
-is probably advisable to specify the record types in the
-<literal remap="tt">zebra.cfg</literal> file
-(see section <xref linkend="record-types"/>), to avoid
-confusion at subsequent updates.
-</para>
-</listitem>
-</varlistentry>
-<varlistentry>
-<term>-c <emphasis remap="it">config-file</emphasis></term>
-<listitem>
-<para>
-Read the configuration file
-<emphasis remap="it">config-file</emphasis> instead of <literal remap="tt">zebra.cfg</literal>.
-</para>
-</listitem>
-</varlistentry>
-<varlistentry>
-<term>-g <emphasis remap="it">group</emphasis></term>
-<listitem>
-<para>
-Update the files according to the group
-settings for <emphasis remap="it">group</emphasis> (see section
-<xref linkend="configuration-file"/>).
-</para>
-</listitem>
-</varlistentry>
-<varlistentry>
-<term>-d <emphasis remap="it">database</emphasis></term>
-<listitem>
-<para>
-The records located should be associated
-with the database name <emphasis remap="it">database</emphasis> for access through the Z39.50
-server.
-</para>
-</listitem>
-</varlistentry>
-<varlistentry>
-<term>-m <emphasis remap="it">mbytes</emphasis></term>
-<listitem>
-<para>
-Use <emphasis remap="it">mbytes</emphasis> of megabytes before flushing
-keys to background storage. This setting affects performance when
-updating large databases.
-</para>
-</listitem>
-</varlistentry>
-<varlistentry>
-<term>-n</term>
-<listitem>
-<para>
-Disable the use of shadow registers for this operation
-(see section <xref linkend="shadow-registers"/>).
-</para>
-</listitem>
-</varlistentry>
-<varlistentry>
-<term>-s</term>
-<listitem>
-<para>
-Show analysis of the indexing process. The maintenance
-program works in a read-only mode and doesn't change the state
-of the index. This options is very useful when you wish to test a
-new profile.
-</para>
-</listitem>
-</varlistentry>
-<varlistentry>
-<term>-V</term>
-<listitem>
-<para>
-Show Zebra version.
-</para>
-</listitem>
-</varlistentry>
-<varlistentry>
-<term>-v <emphasis remap="it">level</emphasis></term>
-<listitem>
-<para>
-Set the log level to <emphasis remap="it">level</emphasis>. <emphasis remap="it">level</emphasis>
-should be one of <literal remap="tt">none</literal>, <literal remap="tt">debug</literal>, and <literal remap="tt">all</literal>.
-</para>
-</listitem>
-</varlistentry>
-</variablelist>
-</para>
-
-<para>
-<emphasis remap="bf">Commands</emphasis>
-<variablelist>
-
-<varlistentry>
-<term>Update <emphasis remap="it">directory</emphasis></term>
-<listitem>
-<para>
-Update the register with the files
-contained in <emphasis remap="it">directory</emphasis>. If no directory is provided, a list of
-files is read from <literal remap="tt">stdin</literal>.
-See section <xref linkend="administration"/>.
-</para>
-</listitem>
-</varlistentry>
-<varlistentry>
-<term>Delete <emphasis remap="it">directory</emphasis></term>
-<listitem>
-<para>
-Remove the records corresponding to
-the files found under <emphasis remap="it">directory</emphasis> from the register.
-</para>
-</listitem>
-</varlistentry>
-<varlistentry>
-<term>Commit</term>
-<listitem>
-<para>
-Write the changes resulting from the last <emphasis remap="bf">update</emphasis>
-commands to the register. This command is only available if the use of
-shadow register files is enabled (see section
-<xref linkend="shadow-registers"/>).
-</para>
-</listitem>
-</varlistentry>
-</variablelist>
-</para>
-
-</chapter>
-
-<chapter id="server">
-<title>The Z39.50 Server</title>
-
-<sect1 id="zebrasrv">
-<title>Running the Z39.50 Server (zebrasrv)</title>
-
-<para>
-<emphasis remap="bf">Syntax</emphasis>
-
-<screen>
-zebrasrv [options] [listener-address ...]
-</screen>
-
-</para>
-
-<para>
-<emphasis remap="bf">Options</emphasis>
-<variablelist>
-
-<varlistentry>
-<term>-a <emphasis remap="it">APDU file</emphasis></term>
-<listitem>
-<para>
-Specify a file for dumping PDUs (for diagnostic purposes).
-The special name "-" sends output to <literal>stderr</literal>.
-</para>
-</listitem>
-</varlistentry>
-<varlistentry>
-<term>-c <emphasis remap="it">config-file</emphasis></term>
-<listitem>
-<para>
-Read configuration information from <emphasis remap="it">config-file</emphasis>. The default configuration is <literal remap="tt">./zebra.cfg</literal>.
-</para>
-</listitem>
-</varlistentry>
-<varlistentry>
-<term>-S</term>
-<listitem>
-<para>
-Don't fork on connection requests. This can be useful for
-symbolic-level debugging. The server can only accept a single
-connection in this mode.
-</para>
-</listitem>
-</varlistentry>
-<varlistentry>
-<term>-s</term>
-<listitem>
-<para>
-Use the SR protocol.
-</para>
-</listitem>
-</varlistentry>
-<varlistentry>
-<term>-z</term>
-<listitem>
-<para>
-Use the Z39.50 protocol (default). These two options complement
-eachother. You can use both multiple times on the same command
-line, between listener-specifications (see below). This way, you
-can set up the server to listen for connections in both protocols
-concurrently, on different local ports.
-</para>
-</listitem>
-</varlistentry>
-<varlistentry>
-<term>-l <emphasis remap="it">logfile</emphasis></term>
-<listitem>
-<para>
-Specify an output file for the diagnostic
-messages. The default is to write this information to <literal remap="tt">stderr</literal>.
-</para>
-</listitem>
-</varlistentry>
-<varlistentry>
-<term>-v <emphasis remap="it">log-level</emphasis></term>
-<listitem>
-<para>
-The log level. Use a comma-separated list of members of the set
-{fatal,debug,warn,log,all,none}.
-</para>
-</listitem>
-</varlistentry>
-<varlistentry>
-<term>-u <emphasis remap="it">username</emphasis></term>
-<listitem>
-<para>
-Set user ID. Sets the real UID of the server process to that of the
-given <emphasis remap="it">username</emphasis>. It's useful if you aren't comfortable with having the
-server run as root, but you need to start it as such to bind a
-privileged port.
-</para>
-</listitem>
-</varlistentry>
-<varlistentry>
-<term>-w <emphasis remap="it">working-directory</emphasis></term>
-<listitem>
-<para>
-Change working directory.
-</para>
-</listitem>
-</varlistentry>
-<varlistentry>
-<term>-i</term>
-<listitem>
-<para>
-Run under the Internet superserver, <literal remap="tt">inetd</literal>. Make
-sure you use the logfile option <literal remap="tt">-l</literal> in conjunction with this
-mode and specify the <literal remap="tt">-l</literal> option before any other options.
-</para>
-</listitem>
-</varlistentry>
-<varlistentry>
-<term>-t <emphasis remap="it">timeout</emphasis></term>
-<listitem>
-<para>
-Set the idle session timeout (default 60 minutes).
-</para>
-</listitem>
-</varlistentry>
-<varlistentry>
-<term>-k <emphasis remap="it">kilobytes</emphasis></term>
-<listitem>
-<para>
-Set the (approximate) maximum size of
-present response messages. Default is 1024 Kb (1 Mb).
-</para>
-</listitem>
-</varlistentry>
-</variablelist>
-</para>
-
-<para>
-A <emphasis remap="it">listener-address</emphasis> consists of a transport mode followed by a
-colon (:) followed by a listener address. The transport mode is
-either <literal remap="tt">osi</literal> or <literal remap="tt">tcp</literal>.
-</para>
-
-<para>
-For TCP, an address has the form
-</para>
-
-<para>
-
-<screen>
-hostname | IP-number [: portnumber]
-</screen>
-
-</para>
-
-<para>
-The port number defaults to 210 (standard Z39.50 port).
-</para>
-
-<para>
-For OSI (only available if the server is compiled with XTI/mOSI
-support enabled), the address form is
-</para>
-
-<para>
-
-<screen>
-[t-selector /] hostname | IP-number [: portnumber]
-</screen>
-
-</para>
-
-<para>
-The transport selector is given as a string of hex digits (with an even
-number of digits). The default port number is 102 (RFC1006 port).
-</para>
-
-<para>
-Examples
-</para>
-
-<para>
-
-<screen>
-tcp:dranet.dra.com
-
-osi:0402/dbserver.osiworld.com:3000
-</screen>
-
-</para>
-
-<para>
-In both cases, the special hostname "@" is mapped to
-the address INADDR_ANY, which causes the server to listen on any local
-interface. To start the server listening on the registered ports for
-Z39.50 and SR over OSI/RFC1006, and to drop root privileges once the
-ports are bound, execute the server like this (from a root shell):
-</para>
-
-<para>
-
-<screen>
-zebrasrv -u daemon tcp:@ -s osi:@
-</screen>
-
-</para>
-
-<para>
-You can replace <literal remap="tt">daemon</literal> with another user, eg. your own account, or
-a dedicated IR server account.
-</para>
-
-<para>
-The default behavior for <literal remap="tt">zebrasrv</literal> is to establish a single TCP/IP
-listener, for the Z39.50 protocol, on port 9999.
-</para>
-
-</sect1>
-
-<sect1 id="protocol-support">
-<title>Z39.50 Protocol Support and Behavior</title>
-
-<sect2>
-<title>Initialization</title>
-
-<para>
-During initialization, the server will negotiate to version 3 of the
-Z39.50 protocol, and the option bits for Search, Present, Scan,
-NamedResultSets, and concurrentOperations will be set, if requested by
-the client. The maximum PDU size is negotiated down to a maximum of
-1Mb by default.
-</para>
-
-</sect2>
-
-<sect2 id="search">
-<title>Search</title>
-
-<para>
-The supported query type are 1 and 101. All operators are currently
-supported with the restriction that only proximity units of type "word" are
-supported for the proximity operator.
-Queries can be arbitrarily complex.
-Named result sets are supported, and result sets can be used as operands
-without limitations.
-Searches may span multiple databases.
-</para>
-
-<para>
-The server has full support for piggy-backed present requests (see
-also the following section).
-</para>
-
-<para>
-<emphasis remap="bf">Use</emphasis> attributes are interpreted according to the attribute sets which
-have been loaded in the <literal remap="tt">zebra.cfg</literal> file, and are matched against
-specific fields as specified in the <literal remap="tt">.abs</literal> file which describes the
-profile of the records which have been loaded. If no <emphasis remap="bf">Use</emphasis>
-attribute is provided, a default of Bib-1 <emphasis remap="bf">Any</emphasis> is assumed.
-</para>
-
-<para>
-If a <emphasis remap="bf">Structure</emphasis> attribute of <emphasis remap="bf">Phrase</emphasis> is used in conjunction with a
-<emphasis remap="bf">Completeness</emphasis> attribute of <emphasis remap="bf">Complete (Sub)field</emphasis>, the term is
-matched against the contents of the phrase (long word) register, if one
-exists for the given <emphasis remap="bf">Use</emphasis> attribute.
-A phrase register is created for those fields in the <literal remap="tt">.abs</literal>
-file that contains a <literal remap="tt">p</literal>-specifier.
-</para>
-
-<para>
-If <emphasis remap="bf">Structure</emphasis>=<emphasis remap="bf">Phrase</emphasis> is used in conjunction with
-<emphasis remap="bf">Incomplete Field</emphasis> - the default value for <emphasis remap="bf">Completeness</emphasis>, the
-search is directed against the normal word registers, but if the term
-contains multiple words, the term will only match if all of the words
-are found immediately adjacent, and in the given order.
-The word search is performed on those fields that are indexed as
-type <literal remap="tt">w</literal> in the <literal remap="tt">.abs</literal> file.
-</para>
-
-<para>
-If the <emphasis remap="bf">Structure</emphasis> attribute is <emphasis remap="bf">Word List</emphasis>,
-<emphasis remap="bf">Free-form Text</emphasis>, or <emphasis remap="bf">Document Text</emphasis>, the term is treated as a
-natural-language, relevance-ranked query.
-This search type uses the word register, i.e. those fields
-that are indexed as type <literal remap="tt">w</literal> in the <literal remap="tt">.abs</literal> file.
-</para>
-
-<para>
-If the <emphasis remap="bf">Structure</emphasis> attribute is <emphasis remap="bf">Numeric String</emphasis> the
-term is treated as an integer. The search is performed on those
-fields that are indexed as type <literal remap="tt">n</literal> in the <literal remap="tt">.abs</literal> file.
-</para>
-
-<para>
-If the <emphasis remap="bf">Structure</emphasis> attribute is <emphasis remap="bf">URx</emphasis> the
-term is treated as a URX (URL) entity. The search is performed on those
-fields that are indexed as type <literal remap="tt">u</literal> in the <literal remap="tt">.abs</literal> file.
-</para>
-
-<para>
-If the <emphasis remap="bf">Structure</emphasis> attribute is <emphasis remap="bf">Local Number</emphasis> the
-term is treated as native Zebra Record Identifier.
-</para>
-
-<para>
-If the <emphasis remap="bf">Relation</emphasis> attribute is <emphasis remap="bf">Equals</emphasis> (default), the term is
-matched in a normal fashion (modulo truncation and processing of
-individual words, if required). If <emphasis remap="bf">Relation</emphasis> is <emphasis remap="bf">Less Than</emphasis>,
-<emphasis remap="bf">Less Than or Equal</emphasis>, <emphasis remap="bf">Greater than</emphasis>, or <emphasis remap="bf">Greater than or
-Equal</emphasis>, the term is assumed to be numerical, and a standard regular
-expression is constructed to match the given expression. If
-<emphasis remap="bf">Relation</emphasis> is <emphasis remap="bf">Relevance</emphasis>, the standard natural-language query
-processor is invoked.
-</para>
-
-<para>
-For the <emphasis remap="bf">Truncation</emphasis> attribute, <emphasis remap="bf">No Truncation</emphasis> is the default.
-<emphasis remap="bf">Left Truncation</emphasis> is not supported. <emphasis remap="bf">Process #</emphasis> is supported, as
-is <emphasis remap="bf">Regxp-1</emphasis>. <emphasis remap="bf">Regxp-2</emphasis> enables the fault-tolerant (fuzzy)
-search. As a default, a single error (deletion, insertion,
-replacement) is accepted when terms are matched against the register
-contents.
-</para>
-
-<sect3>
-<title>Regular expressions</title>
-
-<para>
-Each term in a query is interpreted as a regular expression if
-the truncation value is either <emphasis remap="bf">Regxp-1</emphasis> (102) or <emphasis remap="bf">Regxp-2</emphasis> (103).
-Both query types follow the same syntax with the operands:
-<variablelist>
-
-<varlistentry>
-<term>x</term>
-<listitem>
-<para>
-Matches the character <emphasis remap="it">x</emphasis>.
-</para>
-</listitem>
-</varlistentry>
-<varlistentry>
-<term>.</term>
-<listitem>
-<para>
-Matches any character.
-</para>
-</listitem>
-</varlistentry>
-<varlistentry>
-<term><literal remap="tt">[</literal>..<literal remap="tt">]</literal></term>
-<listitem>
-<para>
-Matches the set of characters specified;
-such as <literal remap="tt">[abc]</literal> or <literal remap="tt">[a-c]</literal>.
-</para>
-</listitem>
-</varlistentry>
-</variablelist>
-and the operators:
-<variablelist>
-
-<varlistentry>
-<term>x*</term>
-<listitem>
-<para>
-Matches <emphasis remap="it">x</emphasis> zero or more times. Priority: high.
-</para>
-</listitem>
-</varlistentry>
-<varlistentry>
-<term>x+</term>
-<listitem>
-<para>
-Matches <emphasis remap="it">x</emphasis> one or more times. Priority: high.
-</para>
-</listitem>
-</varlistentry>
-<varlistentry>
-<term>x?</term>
-<listitem>
-<para>
-Matches <emphasis remap="it">x</emphasis> once or twice. Priority: high.
-</para>
-</listitem>
-</varlistentry>
-<varlistentry>
-<term>xy</term>
-<listitem>
-<para>
-Matches <emphasis remap="it">x</emphasis>, then <emphasis remap="it">y</emphasis>. Priority: medium.
-</para>
-</listitem>
-</varlistentry>
-<varlistentry>
-<term>x|y</term>
-<listitem>
-<para>
-Matches either <emphasis remap="it">x</emphasis> or <emphasis remap="it">y</emphasis>. Priority: low.
-</para>
-</listitem>
-</varlistentry>
-</variablelist>
-The order of evaluation may be changed by using parentheses.
-</para>
-
-<para>
-If the first character of the <emphasis remap="bf">Regxp-2</emphasis> query is a plus character
-(<literal remap="tt">+</literal>) it marks the beginning of a section with non-standard
-specifiers. The next plus character marks the end of the section.
-Currently Zebra only supports one specifier, the error tolerance,
-which consists one digit.
-</para>
-
-<para>
-Since the plus operator is normally a suffix operator the addition to
-the query syntax doesn't violate the syntax for standard regular
-expressions.
-</para>
-
-</sect3>
-
-<sect3>
-<title>Query examples</title>
-
-<para>
-Phrase search for <emphasis remap="bf">information retrieval</emphasis> in the title-register:
-
-<screen>
- @attr 1=4 "information retrieval"
-</screen>
-
-</para>
-
-<para>
-Ranked search for the same thing:
-
-<screen>
- @attr 1=4 @attr 2=102 "Information retrieval"
-</screen>
-
-</para>
-
-<para>
-Phrase search with a regular expression:
-
-<screen>
- @attr 1=4 @attr 5=102 "informat.* retrieval"
-</screen>
-
-</para>
-
-<para>
-Ranked search with a regular expression:
-
-<screen>
- @attr 1=4 @attr 5=102 @attr 2=102 "informat.* retrieval"
-</screen>
-
-</para>
-
-<para>
-In the GILS schema (<literal remap="tt">gils.abs</literal>), the west-bounding-coordinate is
-indexed as type <literal remap="tt">n</literal>, and is therefore searched by specifying
-<emphasis remap="bf">structure</emphasis>=<emphasis remap="bf">Numeric String</emphasis>.
-To match all those records with west-bounding-coordinate greater
-than -114 we use the following query:
-
-<screen>
- @attr 4=109 @attr 2=5 @attr gils 1=2038 -114
-</screen>
-
-</para>
-
-</sect3>
-
-</sect2>
-
-<sect2>
-<title>Present</title>
-
-<para>
-The present facility is supported in a standard fashion. The requested
-record syntax is matched against the ones supported by the profile of
-each record retrieved. If no record syntax is given, SUTRS is the
-default. The requested element set name, again, is matched against any
-provided by the relevant record profiles.
-</para>
-
-</sect2>
-
-<sect2>
-<title>Scan</title>
-
-<para>
-The attribute combinations provided with the termListAndStartPoint are
-processed in the same way as operands in a query (see above).
-Currently, only the term and the globalOccurrences are returned with
-the termInfo structure.
-</para>
-
-</sect2>
-
-<sect2>
-<title>Sort</title>
-
-<para>
-Z39.50 specifies three diffent types of sort criterias.
-Of these Zebra supports the attribute specification type in which
-case the use attribute specifies the "Sort register".
-Sort registers are created for those fields that are of type "sort" in
-the default.idx file.
-The corresponding character mapping file in default.idx specifies the
-ordinal of each character used in the actual sort.
-</para>
-
-<para>
-Z39.50 allows the client to specify sorting on one or more input
-result sets and one output result set.
-Zebra supports sorting on one result set only which may or may not
-be the same as the output result set.
-</para>
-
-</sect2>
-
-<sect2>
-<title>Close</title>
-
-<para>
-If a Close PDU is received, the server will respond with a Close PDU
-with reason=FINISHED, no matter which protocol version was negotiated
-during initialization. If the protocol version is 3 or more, the
-server will generate a Close PDU under certain circumstances,
-including a session timeout (60 minutes by default), and certain kinds of
-protocol errors. Once a Close PDU has been sent, the protocol
-association is considered broken, and the transport connection will be
-closed immediately upon receipt of further data, or following a short
-timeout.
-</para>
-
-</sect2>
-
-</sect1>
-
-</chapter>
-
-<chapter id="record-model">
-<title>The Record Model</title>
-
-<para>
-The Zebra system is designed to support a wide range of data management
-applications. The system can be configured to handle virtually any
-kind of structured data. Each record in the system is associated with
-a <emphasis remap="it">record schema</emphasis> which lends context to the data elements of the
-record. Any number of record schema can coexist in the system.
-Although it may be wise to use only a single schema within
-one database, the system poses no such restrictions.
-</para>
-
-<para>
-The record model described in this chapter applies to the fundamental,
-structured
-record type <literal remap="tt">grs</literal> as introduced in
-section <xref linkend="record-types"/>.
-</para>
-
-<para>
-Records pass through three different states during processing in the
-system.
-</para>
-
-<para>
-
-<itemizedlist>
-<listitem>
-
-<para>
-When records are accessed by the system, they are represented
-in their local, or native format. This might be SGML or HTML files,
-News or Mail archives, MARC records. If the system doesn't already
-know how to read the type of data you need to store, you can set up an
-input filter by preparing conversion rules based on regular
-expressions and possibly augmented by a flexible scripting language (Tcl). The input filter
-produces as output an internal representation:
-
-</para>
-</listitem>
-<listitem>
-
-<para>
-When records are processed by the system, they are represented
-in a tree-structure, constructed by tagged data elements hanging off a
-root node. The tagged elements may contain data or yet more tagged
-elements in a recursive structure. The system performs various
-actions on this tree structure (indexing, element selection, schema
-mapping, etc.),
-
-</para>
-</listitem>
-<listitem>
-
-<para>
-Before transmitting records to the client, they are first
-converted from the internal structure to a form suitable for exchange
-over the network - according to the Z39.50 standard.
-</para>
-</listitem>
-
-</itemizedlist>
-
-</para>
-
-<sect1 id="local-representation">
-<title>Local Representation</title>
-
-<para>
-As mentioned earlier, Zebra places few restrictions on the type of
-data that you can index and manage. Generally, whatever the form of
-the data, it is parsed by an input filter specific to that format, and
-turned into an internal structure that Zebra knows how to handle. This
-process takes place whenever the record is accessed - for indexing and
-retrieval.
-</para>
-
-<para>
-The RecordType parameter in the <literal remap="tt">zebra.cfg</literal> file, or the <literal remap="tt">-t</literal>
-option to the indexer tells Zebra how to process input records. Two
-basic types of processing are available - raw text and structured
-data. Raw text is just that, and it is selected by providing the
-argument <emphasis remap="bf">text</emphasis> to Zebra. Structured records are all handled
-internally using the basic mechanisms described in the subsequent
-sections. Zebra can read structured records in many different formats.
-How this is done is governed by additional parameters after the
-"grs" keyboard, separated by "." characters.
-</para>
-
-<para>
-Three basic subtypes to the <emphasis remap="bf">grs</emphasis> type are currently available:
-</para>
-
-<para>
-<variablelist>
-
-<varlistentry>
-<term>grs.sgml</term>
-<listitem>
-<para>
-This is the canonical input format —
-described below. It is a simple SGML-like syntax.
-</para>
-</listitem>
-</varlistentry>
-<varlistentry>
-<term>grs.regx.<emphasis remap="it">filter</emphasis></term>
-<listitem>
-<para>
-This enables a user-supplied input
-filter. The mechanisms of these filters are described below.
-</para>
-</listitem>
-</varlistentry>
-<varlistentry>
-<term>grs.marc.<emphasis remap="it">abstract syntax</emphasis></term>
-<listitem>
-<para>
-This allows Zebra to read
-records in the ISO2709 (MARC) encoding standard. In this case, the
-last paramemeter <emphasis remap="it">abstract syntax</emphasis> names the .abs file (see below)
-which describes the specific MARC structure of the input record as
-well as the indexing rules.
-</para>
-</listitem>
-</varlistentry>
-</variablelist>
-</para>
-
-<sect2>
-<title>Canonical Input Format</title>
-
-<para>
-Although input data can take any form, it is sometimes useful to
-describe the record processing capabilities of the system in terms of
-a single, canonical input format that gives access to the full
-spectrum of structure and flexibility in the system. In Zebra, this
-canonical format is an "SGML-like" syntax.
-</para>
-
-<para>
-To use the canonical format specify <literal remap="tt">grs.sgml</literal> as the record
-type,
-</para>
-
-<para>
-Consider a record describing an information resource (such a record is
-sometimes known as a <emphasis remap="it">locator record</emphasis>). It might contain a field
-describing the distributor of the information resource, which might in
-turn be partitioned into various fields providing details about the
-distributor, like this:
-</para>
-
-<para>
-
-<screen>
-<Distributor>
- <Name> USGS/WRD </Name>
- <Organization> USGS/WRD </Organization>
- <Street-Address>
- U.S. GEOLOGICAL SURVEY, 505 MARQUETTE, NW
- </Street-Address>
- <City> ALBUQUERQUE </City>
- <State> NM </State>
- <Zip-Code> 87102 </Zip-Code>
- <Country> USA </Country>
- <Telephone> (505) 766-5560 </Telephone>
-</Distributor>
-</screen>
-
-</para>
-
-<para>
-<emphasis remap="it">NOTE: The indentation used above is used to illustrate how Zebra
-interprets the markup. The indentation, in itself, has no
-significance to the parser for the canonical input format, which
-discards superfluous whitespace.</emphasis>
-</para>
-
-<para>
-The keywords surrounded by <...> are <emphasis remap="it">tags</emphasis>, while the
-sections of text in between are the <emphasis remap="it">data elements</emphasis>. A data element
-is characterized by its location in the tree that is made up by the
-nested elements. Each element is terminated by a closing tag -
-beginning with <literal remap="tt"><</literal>/, and containing the same symbolic tag-name as
-the corresponding opening tag. The general closing tag - <literal remap="tt"><</literal>>/ -
-terminates the element started by the last opening tag. The
-structuring of elements is significant. The element <emphasis remap="bf">Telephone</emphasis>,
-for instance, may be indexed and presented to the client differently,
-depending on whether it appears inside the <emphasis remap="bf">Distributor</emphasis> element,
-or some other, structured data element such a <emphasis remap="bf">Supplier</emphasis> element.
-</para>
-
-<sect3>
-<title>Record Root</title>
-
-<para>
-The first tag in a record describes the root node of the tree that
-makes up the total record. In the canonical input format, the root tag
-should contain the name of the schema that lends context to the
-elements of the record (see section
-<xref linkend="internal-representation"/>).
-The following is a GILS record that
-contains only a single element (strictly speaking, that makes it an
-illegal GILS record, since the GILS profile includes several mandatory
-elements - Zebra does not validate the contents of a record against
-the Z39.50 profile, however - it merely attempts to match up elements
-of a local representation with the given schema):
-</para>
-
-<para>
-
-<screen>
-<gils>
- <title>Zen and the Art of Motorcycle Maintenance</title>
-</gils>
-</screen>
-
-</para>
-
-</sect3>
-
-<sect3>
-<title>Variants</title>
-
-<para>
-Zebra allows you to provide individual data elements in a number of
-<emphasis remap="it">variant forms</emphasis>. Examples of variant forms are textual data
-elements which might appear in different languages, and images which
-may appear in different formats or layouts. The variant system in
-Zebra is
-essentially a representation of the variant mechanism of
-Z39.50-1995.
-</para>
-
-<para>
-The following is an example of a title element which occurs in two
-different languages.
-</para>
-
-<para>
-
-<screen>
-<title>
- <var lang lang "eng">
- Zen and the Art of Motorcycle Maintenance</>
- <var lang lang "dan">
- Zen og Kunsten at Vedligeholde en Motorcykel</>
-</title>
-</screen>
-
-</para>
-
-<para>
-The syntax of the <emphasis remap="it">variant element</emphasis> is <literal remap="tt"><var class
-type value></literal>. The available values for the <emphasis remap="it">class</emphasis> and
-<emphasis remap="it">type</emphasis> fields are given by the variant set that is associated with the
-current schema (see section <xref linkend="variant-set"/>).
-</para>
-
-<para>
-Variant elements are terminated by the general end-tag </>, by
-the variant end-tag </var>, by the appearance of another variant
-tag with the same <emphasis remap="it">class</emphasis> and <emphasis remap="it">value</emphasis> settings, or by the
-appearance of another, normal tag. In other words, the end-tags for
-the variants used in the example above could have been saved.
-</para>
-
-<para>
-Variant elements can be nested. The element
-</para>
-
-<para>
-
-<screen>
-<title>
- <var lang lang "eng"><var body iana "text/plain">
- Zen and the Art of Motorcycle Maintenance
-</title>
-</screen>
-
-</para>
-
-<para>
-Associates two variant components to the variant list for the title
-element.
-</para>
-
-<para>
-Given the nesting rules described above, we could write
-</para>
-
-<para>
-
-<screen>
-<title>
- <var body iana "text/plain>
- <var lang lang "eng">
- Zen and the Art of Motorcycle Maintenance
- <var lang lang "dan">
- Zen og Kunsten at Vedligeholde en Motorcykel
-</title>
-</screen>
-
-</para>
-
-<para>
-The title element above comes in two variants. Both have the IANA body
-type "text/plain", but one is in English, and the other in
-Danish. The client, using the element selection mechanism of Z39.50,
-can retrieve information about the available variant forms of data
-elements, or it can select specific variants based on the requirements
-of the end-user.
-</para>
-
-</sect3>
-
-</sect2>
-
-<sect2>
-<title>Input Filters</title>
-
-<para>
-In order to handle general input formats, Zebra allows the
-operator to define filters which read individual records in their native format
-and produce an internal representation that the system can
-work with.
-</para>
-
-<para>
-Input filters are ASCII files, generally with the suffix <literal remap="tt">.flt</literal>.
-The system looks for the files in the directories given in the
-<emphasis remap="bf">profilePath</emphasis> setting in the <literal remap="tt">zebra.cfg</literal> files. The record type
-for the filter is <literal remap="tt">grs.regx.</literal><emphasis remap="it">filter-filename</emphasis>
-(fundamental type <literal remap="tt">grs</literal>, file read type <literal remap="tt">regx</literal>, argument
-<emphasis remap="it">filter-filename</emphasis>).
-</para>
-
-<para>
-Generally, an input filter consists of a sequence of rules, where each
-rule consists of a sequence of expressions, followed by an action. The
-expressions are evaluated against the contents of the input record,
-and the actions normally contribute to the generation of an internal
-representation of the record.
-</para>
-
-<para>
-An expression can be either of the following:
-</para>
-
-<para>
-<variablelist>
-
-<varlistentry>
-<term>INIT</term>
-<listitem>
-<para>
-The action associated with this expression is evaluated
-exactly once in the lifetime of the application, before any records
-are read. It can be used in conjunction with an action that
-initializes tables or other resources that are used in the processing
-of input records.
-</para>
-</listitem>
-</varlistentry>
-<varlistentry>
-<term>BEGIN</term>
-<listitem>
-<para>
-Matches the beginning of the record. It can be used to
-initialize variables, etc. Typically, the <emphasis remap="bf">BEGIN</emphasis> rule is also used
-to establish the root node of the record.
-</para>
-</listitem>
-</varlistentry>
-<varlistentry>
-<term>END</term>
-<listitem>
-<para>
-Matches the end of the record - when all of the contents
-of the record has been processed.
-</para>
-</listitem>
-</varlistentry>
-<varlistentry>
-<term>/pattern/</term>
-<listitem>
-<para>
-Matches a string of characters from the input
-record.
-</para>
-</listitem>
-</varlistentry>
-<varlistentry>
-<term>BODY</term>
-<listitem>
-<para>
-This keyword may only be used between two patterns. It
-matches everything between (not including) those patterns.
-</para>
-</listitem>
-</varlistentry>
-<varlistentry>
-<term>FINISH</term>
-<listitem>
-<para>
-The expression asssociated with this pattern is evaluated
-once, before the application terminates. It can be used to release
-system resources - typically ones allocated in the <emphasis remap="bf">INIT</emphasis> step.
-</para>
-</listitem>
-</varlistentry>
-</variablelist>
-</para>
-
-<para>
-An action is surrounded by curly braces ({...}), and consists of a
-sequence of statements. Statements may be separated by newlines or
-semicolons (;). Within actions, the strings that matched the
-expressions immediately preceding the action can be referred to as
-$0, $1, $2, etc.
-</para>
-
-<para>
-The available statements are:
-</para>
-
-<para>
-<variablelist>
-
-<varlistentry>
-<term>begin <emphasis remap="it">type [parameter ... ]</emphasis></term>
-<listitem>
-<para>
-Begin a new
-data element. The type is one of the following:
-<variablelist>
-
-<varlistentry>
-<term>record</term>
-<listitem>
-<para>
-Begin a new record. The followingparameter should be the
-name of the schema that describes the structure of the record, eg.
-<literal remap="tt">gils</literal> or <literal remap="tt">wais</literal> (see below). The <literal remap="tt">begin record</literal> call should
-precede
-any other use of the <emphasis remap="bf">begin</emphasis> statement.
-</para>
-</listitem>
-</varlistentry>
-<varlistentry>
-<term>element</term>
-<listitem>
-<para>
-Begin a new tagged element. The parameter is the
-name of the tag. If the tag is not matched anywhere in the tagsets
-referenced by the current schema, it is treated as a local string
-tag.
-</para>
-</listitem>
-</varlistentry>
-<varlistentry>
-<term>variant</term>
-<listitem>
-<para>
-Begin a new node in a variant tree. The parameters are
-<emphasis remap="it">class type value</emphasis>.
-</para>
-</listitem>
-</varlistentry>
-</variablelist>
-</para>
-</listitem>
-</varlistentry>
-<varlistentry>
-<term>data</term>
-<listitem>
-<para>
-Create a data element. The concatenated arguments make
-up the value of the data element. The option <literal remap="tt">-text</literal> signals that
-the layout (whitespace) of the data should be retained for
-transmission. The option <literal remap="tt">-element</literal> <emphasis remap="it">tag</emphasis> wraps the data up in
-the <emphasis remap="it">tag</emphasis>. The use of the <literal remap="tt">-element</literal> option is equivalent to
-preceding the command with a <emphasis remap="bf">begin element</emphasis> command, and following
-it with the <emphasis remap="bf">end</emphasis> command.
-</para>
-</listitem>
-</varlistentry>
-<varlistentry>
-<term>end <emphasis remap="it">[type]</emphasis></term>
-<listitem>
-<para>
-Close a tagged element. If no parameter is given,
-the last element on the stack is terminated. The first parameter, if
-any, is a type name, similar to the <emphasis remap="bf">begin</emphasis> statement. For the
-<emphasis remap="bf">element</emphasis> type, a tag name can be provided to terminate a specific tag.
-</para>
-</listitem>
-</varlistentry>
-</variablelist>
-</para>
-
-<para>
-The following input filter reads a Usenet news file, producing a
-record in the WAIS schema. Note that the body of a news posting is
-separated from the list of headers by a blank line (or rather a
-sequence of two newline characters.
-</para>
-
-<para>
-
-<screen>
-BEGIN { begin record wais }
-
-/^From:/ BODY /$/ { data -element name $1 }
-/^Subject:/ BODY /$/ { data -element title $1 }
-/^Date:/ BODY /$/ { data -element lastModified $1 }
-/\n\n/ BODY END {
- begin element bodyOfDisplay
- begin variant body iana "text/plain"
- data -text $1
- end record
- }
-</screen>
-
-</para>
-
-<para>
-If Zebra is compiled with support for Tcl (Tool Command Language)
-enabled, the statements described above are supplemented with a complete
-scripting environment, including control structures (conditional
-expressions and loop constructs), and powerful string manipulation
-mechanisms for modifying the elements of a record. Tcl is a popular
-scripting environment, with several tutorials available both online
-and in hardcopy.
-</para>
-
-<para>
-<emphasis remap="it">NOTE: Tcl support is not currently available, but will be
-included with one of the next alpha or beta releases.</emphasis>
-</para>
-
-<para>
-<emphasis remap="it">NOTE: Variant support is not currently available in the input
-filter, but will be included with one of the next alpha or beta
-releases.</emphasis>
-</para>
-
-</sect2>
-
-</sect1>
-
-<sect1 id="internal-representation">
-<title>Internal Representation</title>
-
-<para>
-When records are manipulated by the system, they're represented in a
-tree-structure, with data elements at the leaf nodes, and tags or
-variant components at the non-leaf nodes. The root-node identifies the
-schema that lends context to the tagging and structuring of the
-record. Imagine a simple record, consisting of a 'title' element and
-an 'author' element:
-</para>
-
-<para>
-
-<screen>
- TITLE "Zen and the Art of Motorcycle Maintenance"
-ROOT
- AUTHOR "Robert Pirsig"
-</screen>
-
-</para>
-
-<para>
-A slightly more complex record would have the author element consist
-of two elements, a surname and a first name:
-</para>
-
-<para>
-
-<screen>
- TITLE "Zen and the Art of Motorcycle Maintenance"
-ROOT
- FIRST-NAME "Robert"
- AUTHOR
- SURNAME "Pirsig"
-</screen>
-
-</para>
-
-<para>
-The root of the record will refer to the record schema that describes
-the structuring of this particular record. The schema defines the
-element tags (TITLE, FIRST-NAME, etc.) that may occur in the record, as
-well as the structuring (SURNAME should appear below AUTHOR, etc.). In
-addition, the schema establishes element set names that are used by
-the client to request a subset of the elements of a given record. The
-schema may also establish rules for converting the record to a
-different schema, by stating, for each element, a mapping to a
-different tag path.
-</para>
-
-<sect2>
-<title>Tagged Elements</title>
-
-<para>
-A data element is characterized by its tag, and its position in the
-structure of the record. For instance, while the tag "telephone
-number" may be used different places in a record, we may need to
-distinguish between these occurrences, both for searching and
-presentation purposes. For instance, while the phone numbers for the
-"customer" and the "service provider" are both
-representatives for the same type of resource (a telephone number), it
-is essential that they be kept separate. The record schema provides
-the structure of the record, and names each data element (defined by
-the sequence of tags - the tag path - by which the element can be
-reached from the root of the record).
-</para>
-
-</sect2>
-
-<sect2>
-<title>Variants</title>
-
-<para>
-The children of a tag node may be either more tag nodes, a data node
-(possibly accompanied by tag nodes),
-or a tree of variant nodes. The children of variant nodes are either
-more variant nodes or a data node (possibly accompanied by more
-variant nodes). Each leaf node, which is normally a
-data node, corresponds to a <emphasis remap="it">variant form</emphasis> of the tagged element
-identified by the tag which parents the variant tree. The following
-title element occurs in two different languages:
-</para>
-
-<para>
-
-<screen>
- VARIANT LANG=ENG "War and Peace"
-TITLE
- VARIANT LANG=DAN "Krig og Fred"
-</screen>
-
-</para>
-
-<para>
-Which of the two elements are transmitted to the client by the server
-depends on the specifications provided by the client, if any.
-</para>
-
-<para>
-In practice, each variant node is associated with a triple of class,
-type, value, corresponding to the variant mechanism of Z39.50.
-</para>
-
-</sect2>
-
-<sect2>
-<title>Data Elements</title>
-
-<para>
-Data nodes have no children (they are always leaf nodes in the record
-tree).
-</para>
-
-<para>
-<emphasis remap="it">NOTE: Documentation needs extension here about types of nodes - numerical,
-textual, etc., plus the various types of inclusion notes.</emphasis>
-</para>
-
-</sect2>
-
-</sect1>
-
-<sect1 id="data-model">
-<title>Configuring Your Data Model</title>
-
-<para>
-The following sections describe the configuration files that govern
-the internal management of data records. The system searches for the files
-in the directories specified by the <emphasis remap="bf">profilePath</emphasis> setting in the
-<literal remap="tt">zebra.cfg</literal> file.
-</para>
-
-<sect2>
-<title>The Abstract Syntax</title>
-
-<para>
-The abstract syntax definition (also known as an Abstract Record
-Structure, or ARS) is the focal point of the
-record schema description. For a given schema, the ABS file may state any
-or all of the following:
-</para>
-
-<para>
-
-<itemizedlist>
-<listitem>
-
-<para>
-The object identifier of the Z39.50 schema associated
-with the ARS, so that it can be referred to by the client.
-
-</para>
-</listitem>
-<listitem>
-
-<para>
-The attribute set (which can possibly be a compound of multiple
-sets) which applies in the profile. This is used when indexing and
-searching the records belonging to the given profile.
-
-</para>
-</listitem>
-<listitem>
-
-<para>
-The Tag set (again, this can consist of several different sets).
-This is used when reading the records from a file, to recognize the
-different tags, and when transmitting the record to the client -
-mapping the tags to their numerical representation, if they are
-known.
-
-</para>
-</listitem>
-<listitem>
-
-<para>
-The variant set which is used in the profile. This provides a
-vocabulary for specifying the <emphasis remap="it">forms</emphasis> of data that appear inside
-the records.
-
-</para>
-</listitem>
-<listitem>
-
-<para>
-Element set names, which are a shorthand way for the client to
-ask for a subset of the data elements contained in a record. Element
-set names, in the retrieval module, are mapped to <emphasis remap="it">element
-specifications</emphasis>, which contain information equivalent to the
-<emphasis remap="it">Espec-1</emphasis> syntax of Z39.50.
-
-</para>
-</listitem>
-<listitem>
-
-<para>
-Map tables, which may specify mappings to <emphasis remap="it">other</emphasis> database
-profiles, if desired.
-
-</para>
-</listitem>
-<listitem>
-
-<para>
-Possibly, a set of rules describing the mapping of elements to a
-MARC representation.
-
-</para>
-</listitem>
-<listitem>
-
-<para>
-A list of element descriptions (this is the actual ARS of the
-schema, in Z39.50 terms), which lists the ways in which the various
-tags can be used and organized hierarchically.
-</para>
-</listitem>
-
-</itemizedlist>
-
-</para>
-
-<para>
-Several of the entries above simply refer to other files, which
-describe the given objects.
-</para>
-
-</sect2>
-
-<sect2>
-<title>The Configuration Files</title>
-
-<para>
-This section describes the syntax and use of the various tables which
-are used by the retrieval module.
-</para>
-
-<para>
-The number of different file types may appear daunting at first, but
-each type corresponds fairly clearly to a single aspect of the Z39.50
-retrieval facilities. Further, the average database administrator,
-who is simply reusing an existing profile for which tables already
-exist, shouldn't have to worry too much about the contents of these tables.
-</para>
-
-<para>
-Generally, the files are simple ASCII files, which can be maintained
-using any text editor. Blank lines, and lines beginning with a (#) are
-ignored. Any characters on a line followed by a (#) are also ignored.
-All other
-lines contain <emphasis remap="it">directives</emphasis>, which provide some setting or value
-to the system. Generally, settings are characterized by a single
-keyword, identifying the setting, followed by a number of parameters.
-Some settings are repeatable (r), while others may occur only once in a
-file. Some settings are optional (o), whicle others again are
-mandatory (m).
-</para>
-
-</sect2>
-
-<sect2>
-<title>The Abstract Syntax (.abs) Files</title>
-
-<para>
-The name of this file type is slightly misleading in Z39.50 terms,
-since, apart from the actual abstract syntax of the profile, it also
-includes most of the other definitions that go into a database
-profile.
-</para>
-
-<para>
-When a record in the canonical, SGML-like format is read from a file
-or from the database, the first tag of the file should reference the
-profile that governs the layout of the record. If the first tag of the
-record is, say, <literal remap="tt"><gils></literal>, the system will look for the profile
-definition in the file <literal remap="tt">gils.abs</literal>. Profile definitions are cached,
-so they only have to be read once during the lifespan of the current
-process.
-</para>
-
-<para>
-When writing your own input filters, the <emphasis remap="bf">record-begin</emphasis> command
-introduces the profile, and should always be called first thing when
-introducing a new record.
-</para>
-
-<para>
-The file may contain the following directives:
-</para>
-
-<para>
-<variablelist>
-
-<varlistentry>
-<term>name <emphasis remap="it">symbolic-name</emphasis></term>
-<listitem>
-<para>
-(m) This provides a shorthand name or
-description for the profile. Mostly useful for diagnostic purposes.
-</para>
-</listitem>
-</varlistentry>
-<varlistentry>
-<term>reference <emphasis remap="it">OID-name</emphasis></term>
-<listitem>
-<para>
-(m) The reference name of the OID for
-the profile. The reference names can be found in the <emphasis remap="bf">util</emphasis>
-module of <emphasis remap="bf">YAZ</emphasis>.
-</para>
-</listitem>
-</varlistentry>
-<varlistentry>
-<term>attset <emphasis remap="it">filename</emphasis></term>
-<listitem>
-<para>
-(m) The attribute set that is used for
-indexing and searching records belonging to this profile.
-</para>
-</listitem>
-</varlistentry>
-<varlistentry>
-<term>tagset <emphasis remap="it">filename</emphasis></term>
-<listitem>
-<para>
-(o) The tag set (if any) that describe
-that fields of the records.
-</para>
-</listitem>
-</varlistentry>
-<varlistentry>
-<term>varset <emphasis remap="it">filename</emphasis></term>
-<listitem>
-<para>
-(o) The variant set used in the profile.
-</para>
-</listitem>
-</varlistentry>
-<varlistentry>
-<term>maptab <emphasis remap="it">filename</emphasis></term>
-<listitem>
-<para>
-(o,r) This points to a
-conversion table that might be used if the client asks for the record
-in a different schema from the native one.
-</para>
-</listitem></varlistentry>
-<varlistentry>
-<term>marc <emphasis remap="it">filename</emphasis></term>
-<listitem>
-<para>
-(o) Points to a file containing parameters
-for representing the record contents in the ISO2709 syntax. Read the
-description of the MARC representation facility below.
-</para>
-</listitem></varlistentry>
-<varlistentry>
-<term>esetname <emphasis remap="it">name filename</emphasis></term>
-<listitem>
-<para>
-(o,r) Associates the
-given element set name with an element selection file. If an (@) is
-given in place of the filename, this corresponds to a null mapping for
-the given element set name.
-</para>
-</listitem></varlistentry>
-<varlistentry>
-<term>any <emphasis remap="it">tags</emphasis></term>
-<listitem>
-<para>
-(o) This directive specifies a list of
-attributes which should be appended to the attribute list given for each
-element. The effect is to make every single element in the abstract
-syntax searchable by way of the given attributes. This directive
-provides an efficient way of supporting free-text searching across all
-elements. However, it does increase the size of the index
-significantly. The attributes can be qualified with a structure, as in
-the <emphasis remap="bf">elm</emphasis> directive below.
-</para>
-</listitem></varlistentry>
-<varlistentry>
-<term>elm <emphasis remap="it">path name attributes</emphasis></term>
-<listitem>
-<para>
-(o,r) Adds an element
-to the abstract record syntax of the schema. The <emphasis remap="it">path</emphasis> follows the
-syntax which is suggested by the Z39.50 document - that is, a sequence
-of tags separated by slashes (/). Each tag is given as a
-comma-separated pair of tag type and -value surrounded by parenthesis.
-The <emphasis remap="it">name</emphasis> is the name of the element, and the <emphasis remap="it">attributes</emphasis>
-specifies which attributes to use when indexing the element in a
-comma-separated list. A ! in
-place of the attribute name is equivalent to specifying an attribute
-name identical to the element name. A - in place of the attribute name
-specifies that no indexing is to take place for the given element. The
-attributes can be qualified with <emphasis remap="it">field types</emphasis> to specify which
-character set should govern the indexing procedure for that field. The
-same data element may be indexed into several different fields, using
-different character set definitions. See the section
-<xref linkend="field-structure-and-character-sets"/>.
-The default field type is "w" for
-<emphasis remap="it">word</emphasis>.
-</para>
-</listitem></varlistentry>
-</variablelist>
-</para>
-
-<para>
-<emphasis remap="it">NOTE: The mechanism for controlling indexing is not adequate for
-complex databases, and will probably be moved into a separate
-configuration table eventually.</emphasis>
-</para>
-
-<para>
-The following is an excerpt from the abstract syntax file for the GILS
-profile.
-</para>
-
-<para>
-
-<screen>
-name gils
-reference GILS-schema
-attset gils.att
-tagset gils.tag
-varset var1.var
-
-maptab gils-usmarc.map
-
-# Element set names
-
-esetname VARIANT gils-variant.est # for WAIS-compliance
-esetname B gils-b.est
-esetname G gils-g.est
-esetname F @
-
-elm (1,10) rank -
-elm (1,12) url -
-elm (1,14) localControlNumber Local-number
-elm (1,16) dateOfLastModification Date/time-last-modified
-elm (2,1) title w:!,p:!
-elm (4,1) controlIdentifier Identifier-standard
-elm (2,6) abstract Abstract
-elm (4,51) purpose !
-elm (4,52) originator -
-elm (4,53) accessConstraints !
-elm (4,54) useConstraints !
-elm (4,70) availability -
-elm (4,70)/(4,90) distributor -
-elm (4,70)/(4,90)/(2,7) distributorName !
-elm (4,70)/(4,90)/(2,10 distributorOrganization !
-elm (4,70)/(4,90)/(4,2) distributorStreetAddress !
-elm (4,70)/(4,90)/(4,3) distributorCity !
-</screen>
-
-</para>
-
-</sect2>
-
-<sect2 id="attset-files">
-<title>The Attribute Set (.att) Files</title>
-
-<para>
-This file type describes the <emphasis remap="bf">Use</emphasis> elements of an attribute set.
-It contains the following directives.
-</para>
-
-<para>
-<variablelist>
-
-<varlistentry>
-<term>name <emphasis remap="it">symbolic-name</emphasis></term>
-<listitem>
-<para>
-(m) This provides a shorthand name or
-description for the attribute set. Mostly useful for diagnostic purposes.
-</para>
-</listitem></varlistentry>
-<varlistentry>
-<term>reference <emphasis remap="it">OID-name</emphasis></term>
-<listitem>
-<para>
-(m) The reference name of the OID for
-the attribute set. The reference names can be found in the <emphasis remap="bf">util</emphasis>
-module of <emphasis remap="bf">YAZ</emphasis>.
-</para>
-</listitem></varlistentry>
-<varlistentry>
-<term>ordinal <emphasis remap="it">integer</emphasis></term>
-<listitem>
-<para>
-(m) This value will be used to represent the
-attribute set in the index. Care should be taken that each attribute
-set has a unique ordinal value.
-</para>
-</listitem></varlistentry>
-<varlistentry>
-<term>include <emphasis remap="it">filename</emphasis></term>
-<listitem>
-<para>
-(o,r) This directive is used to
-include another attribute set as a part of the current one. This is
-used when a new attribute set is defined as an extension to another
-set. For instance, many new attribute sets are defined as extensions
-to the <emphasis remap="bf">bib-1</emphasis> set. This is an important feature of the retrieval
-system of Z39.50, as it ensures the highest possible level of
-interoperability, as those access points of your database which are
-derived from the external set (say, bib-1) can be used even by clients
-who are unaware of the new set.
-</para>
-</listitem></varlistentry>
-<varlistentry>
-<term>att <emphasis remap="it">att-value att-name [local-value]</emphasis></term>
-<listitem>
-<para>
-(o,r) This
-repeatable directive introduces a new attribute to the set. The
-attribute value is stored in the index (unless a <emphasis remap="it">local-value</emphasis> is
-given, in which case this is stored). The name is used to refer to the
-attribute from the <emphasis remap="it">abstract syntax</emphasis>.
-</para>
-</listitem></varlistentry>
-</variablelist>
-</para>
-
-<para>
-This is an excerpt from the GILS attribute set definition. Notice how
-the file describing the <emphasis remap="it">bib-1</emphasis> attribute set is referenced.
-</para>
-
-<para>
-
-<screen>
-name gils
-reference GILS-attset
-include bib1.att
-ordinal 2
-
-att 2001 distributorName
-att 2002 indextermsControlled
-att 2003 purpose
-att 2004 accessConstraints
-att 2005 useConstraints
-</screen>
-
-</para>
-
-</sect2>
-
-<sect2>
-<title>The Tag Set (.tag) Files</title>
-
-<para>
-This file type defines the tagset of the profile, possibly by
-referencing other tag sets (most tag sets, for instance, will include
-tagsetG and tagsetM from the Z39.50 specification. The file may
-contain the following directives.
-</para>
-
-<para>
-<variablelist>
-
-<varlistentry>
-<term>name <emphasis remap="it">symbolic-name</emphasis></term>
-<listitem>
-<para>
-(m) This provides a shorthand name or
-description for the tag set. Mostly useful for diagnostic purposes.
-</para>
-</listitem></varlistentry>
-<varlistentry>
-<term>reference <emphasis remap="it">OID-name</emphasis></term>
-<listitem>
-<para>
-(o) The reference name of the OID for
-the tag set. The reference names can be found in the <emphasis remap="bf">util</emphasis>
-module of <emphasis remap="bf">YAZ</emphasis>. The directive is optional, since not all tag sets
-are registered outside of their schema.
-</para>
-</listitem></varlistentry>
-<varlistentry>
-<term>type <emphasis remap="it">integer</emphasis></term>
-<listitem>
-<para>
-(m) The type number of the tagset within the schema
-profile (note: this specification really should belong to the .abs
-file. This will be fixed in a future release).
-</para>
-</listitem></varlistentry>
-<varlistentry>
-<term>include <emphasis remap="it">filename</emphasis></term>
-<listitem>
-<para>
-(o,r) This directive is used
-to include the definitions of other tag sets into the current one.
-</para>
-</listitem></varlistentry>
-<varlistentry>
-<term>tag <emphasis remap="it">number names type</emphasis></term>
-<listitem>
-<para>
-(o,r) Introduces a new
-tag to the set. The <emphasis remap="it">number</emphasis> is the tag number as used in the protocol
-(there is currently no mechanism for specifying string tags at this
-point, but this would be quick work to add). The <emphasis remap="it">names</emphasis> parameter
-is a list of names by which the tag should be recognized in the input
-file format. The names should be separated by slashes (/). The
-<emphasis remap="it">type</emphasis> is th recommended datatype of the tag. It should be one of
-the following:
-
-<itemizedlist>
-<listitem>
-
-<para>
-structured
-</para>
-</listitem>
-<listitem>
-
-<para>
-string
-</para>
-</listitem>
-<listitem>
-
-<para>
-numeric
-</para>
-</listitem>
-<listitem>
-
-<para>
-bool
-</para>
-</listitem>
-<listitem>
-
-<para>
-oid
-</para>
-</listitem>
-<listitem>
-
-<para>
-generalizedtime
-</para>
-</listitem>
-<listitem>
-
-<para>
-intunit
-</para>
-</listitem>
-<listitem>
-
-<para>
-int
-</para>
-</listitem>
-<listitem>
-
-<para>
-octetstring
-</para>
-</listitem>
-<listitem>
-
-<para>
-null
-</para>
-</listitem>
-
-</itemizedlist>
-
-</para>
-</listitem></varlistentry>
-</variablelist>
-</para>
-
-<para>
-The following is an excerpt from the TagsetG definition file.
-</para>
-
-<para>
-
-<screen>
-name tagsetg
-reference TagsetG
-type 2
-
-tag 1 title string
-tag 2 author string
-tag 3 publicationPlace string
-tag 4 publicationDate string
-tag 5 documentId string
-tag 6 abstract string
-tag 7 name string
-tag 8 date generalizedtime
-tag 9 bodyOfDisplay string
-tag 10 organization string
-</screen>
-
-</para>
-
-</sect2>
-
-<sect2 id="variant-set">
-<title>The Variant Set (.var) Files</title>
-
-<para>
-The variant set file is a straightforward representation of the
-variant set definitions associated with the protocol. At present, only
-the <emphasis remap="it">Variant-1</emphasis> set is known.
-</para>
-
-<para>
-These are the directives allowed in the file.
-</para>
-
-<para>
-<variablelist>
-
-<varlistentry>
-<term>name <emphasis remap="it">symbolic-name</emphasis></term>
-<listitem>
-<para>
-(m) This provides a shorthand name or
-description for the variant set. Mostly useful for diagnostic purposes.
-</para>
-</listitem></varlistentry>
-<varlistentry>
-<term>reference <emphasis remap="it">OID-name</emphasis></term>
-<listitem>
-<para>
-(o) The reference name of the OID for
-the variant set, if one is required. The reference names can be found
-in the <emphasis remap="bf">util</emphasis> module of <emphasis remap="bf">YAZ</emphasis>.
-</para>
-</listitem></varlistentry>
-<varlistentry>
-<term>class <emphasis remap="it">integer class-name</emphasis></term>
-<listitem>
-<para>
-(m,r) Introduces a new
-class to the variant set.
-</para>
-</listitem></varlistentry>
-<varlistentry>
-<term>type <emphasis remap="it">integer type-name datatype</emphasis></term>
-<listitem>
-<para>
-(m,r) Addes a
-new type to the current class (the one introduced by the most recent
-<emphasis remap="bf">class</emphasis> directive). The type names belong to the same name space as
-the one used in the tag set definition file.
-</para>
-</listitem></varlistentry>
-</variablelist>
-</para>
-
-<para>
-The following is an excerpt from the file describing the variant set
-<emphasis remap="it">Variant-1</emphasis>.
-</para>
-
-<para>
-
-<screen>
-name variant-1
-reference Variant-1
-
-class 1 variantId
-
- type 1 variantId octetstring
-
-class 2 body
-
- type 1 iana string
- type 2 z39.50 string
- type 3 other string
-</screen>
-
-</para>
-
-</sect2>
-
-<sect2>
-<title>The Element Set (.est) Files</title>
-
-<para>
-The element set specification files describe a selection of a subset
-of the elements of a database record. The element selection mechanism
-is equivalent to the one supplied by the <emphasis remap="it">Espec-1</emphasis> syntax of the
-Z39.50 specification. In fact, the internal representation of an
-element set specification is identical to the <emphasis remap="it">Espec-1</emphasis> structure,
-and we'll refer you to the description of that structure for most of
-the detailed semantics of the directives below.
-</para>
-
-<para>
-<emphasis remap="it">NOTE: Not all of the Espec-1 functionality has been implemented yet.
-The fields that are mentioned below all work as expected, unless
-otherwise is noted.</emphasis>
-</para>
-
-<para>
-The directives available in the element set file are as follows:
-</para>
-
-<para>
-<variablelist>
-
-<varlistentry>
-<term>defaultVariantSetId <emphasis remap="it">OID-name</emphasis></term>
-<listitem>
-<para>
-(o) If variants are used in
-the following, this should provide the name of the variantset used
-(it's not currently possible to specify a different set in the
-individual variant request). In almost all cases (certainly all
-profiles known to us), the name <literal remap="tt">Variant-1</literal> should be given here.
-</para>
-</listitem></varlistentry>
-<varlistentry>
-<term>defaultVariantRequest <emphasis remap="it">variant-request</emphasis></term>
-<listitem>
-<para>
-(o) This directive
-provides a default variant request for
-use when the individual element requests (see below) do not contain a
-variant request. Variant requests consist of a blank-separated list of
-variant components. A variant compont is a comma-separated,
-parenthesized triple of variant class, type, and value (the two former
-values being represented as integers). The value can currently only be
-entered as a string (this will change to depend on the definition of
-the variant in question). The special value (@) is interpreted as a
-null value, however.
-</para>
-</listitem></varlistentry>
-<varlistentry>
-<term>simpleElement <emphasis remap="it">path ['variant' variant-request]</emphasis></term>
-<listitem>
-<para>
-(o,r) This corresponds to a simple element request in <emphasis remap="it">Espec-1</emphasis>. The
-path consists of a sequence of tag-selectors, where each of these can
-consist of either:
-</para>
-
-<para>
-
-<itemizedlist>
-<listitem>
-
-<para>
-A simple tag, consisting of a comma-separated type-value pair in
-parenthesis, possibly followed by a colon (:) followed by an
-occurrences-specification (see below). The tag-value can be a number
-or a string. If the first character is an apostrophe ('), this forces
-the value to be interpreted as a string, even if it appears to be numerical.
-
-</para>
-</listitem>
-<listitem>
-
-<para>
-A WildThing, represented as a question mark (?), possibly
-followed by a colon (:) followed by an occurrences specification (see
-below).
-
-</para>
-</listitem>
-<listitem>
-
-<para>
-A WildPath, represented as an asterisk (*). Note that the last
-element of the path should not be a wildPath (wildpaths don't work in
-this version).
-</para>
-</listitem>
-
-</itemizedlist>
-
-</para>
-
-<para>
-The occurrences-specification can be either the string <literal remap="tt">all</literal>, the
-string <literal remap="tt">last</literal>, or an explicit value-range. The value-range is
-represented as an integer (the starting point), possibly followed by a
-plus (+) and a second integer (the number of elements, default being
-one).
-</para>
-
-<para>
-The variant-request has the same syntax as the defaultVariantRequest
-above. Note that it may sometimes be useful to give an empty variant
-request, simply to disable the default for a specific set of fields
-(we aren't certain if this is proper <emphasis remap="it">Espec-1</emphasis>, but it works in
-this implementation).
-</para>
-</listitem></varlistentry>
-</variablelist>
-</para>
-
-<para>
-The following is an example of an element specification belonging to
-the GILS profile.
-</para>
-
-<para>
-
-<screen>
-simpleelement (1,10)
-simpleelement (1,12)
-simpleelement (2,1)
-simpleelement (1,14)
-simpleelement (4,1)
-simpleelement (4,52)
-</screen>
-
-</para>
-
-</sect2>
-
-<sect2 id="schema-mapping">
-<title>The Schema Mapping (.map) Files</title>
-
-<para>
-Sometimes, the client might want to receive a database record in
-a schema that differs from the native schema of the record. For
-instance, a client might only know how to process WAIS records, while
-the database record is represented in a more specific schema, such as
-GILS. In this module, a mapping of data to one of the MARC formats is
-also thought of as a schema mapping (mapping the elements of the
-record into fields consistent with the given MARC specification, prior
-to actually converting the data to the ISO2709). This use of the
-object identifier for USMARC as a schema identifier represents an
-overloading of the OID which might not be entirely proper. However,
-it represents the dual role of schema and record syntax which
-is assumed by the MARC family in Z39.50.
-</para>
-
-<para>
-<emphasis remap="it">NOTE: The schema-mapping functions are so far limited to a
-straightforward mapping of elements. This should be extended with
-mechanisms for conversions of the element contents, and conditional
-mappings of elements based on the record contents.</emphasis>
-</para>
-
-<para>
-These are the directives of the schema mapping file format:
-</para>
-
-<para>
-<variablelist>
-
-<varlistentry>
-<term>targetName <emphasis remap="it">name</emphasis></term>
-<listitem>
-<para>
-(m) A symbolic name for the target schema
-of the table. Useful mostly for diagnostic purposes.
-</para>
-</listitem></varlistentry>
-<varlistentry>
-<term>targetRef <emphasis remap="it">OID-name</emphasis></term>
-<listitem>
-<para>
-(m) An OID name for the target schema.
-This is used, for instance, by a server receiving a request to present
-a record in a different schema from the native one. The name, again,
-is found in the <emphasis remap="bf">oid</emphasis> module of <emphasis remap="bf">YAZ</emphasis>.
-</para>
-</listitem></varlistentry>
-<varlistentry>
-<term>map <emphasis remap="it">element-name target-path</emphasis></term>
-<listitem>
-<para>
-(o,r) Adds
-an element mapping rule to the table.
-</para>
-</listitem></varlistentry>
-</variablelist>
-</para>
-
-</sect2>
-
-<sect2>
-<title>The MARC (ISO2709) Representation (.mar) Files</title>
-
-<para>
-This file provides rules for representing a record in the ISO2709
-format. The rules pertain mostly to the values of the constant-length
-header of the record.
-</para>
-
-<para>
-<emphasis remap="it">NOTE: This will be described better. We're in the process of
-re-evaluating and most likely changing the way that MARC records are
-handled by the system.</emphasis>
-</para>
-
-</sect2>
-
-<sect2 id="field-structure-and-character-sets">
-<title>Field Structure and Character Sets
-</title>
-
-<para>
-In order to provide a flexible approach to national character set
-handling, Zebra allows the administrator to configure the set up the
-system to handle any 8-bit character set — including sets that
-require multi-octet diacritics or other multi-octet characters. The
-definition of a character set includes a specification of the
-permissible values, their sort order (this affects the display in the
-SCAN function), and relationships between upper- and lowercase
-characters. Finally, the definition includes the specification of
-space characters for the set.
-</para>
-
-<para>
-The operator can define different character sets for different fields,
-typical examples being standard text fields, numerical fields, and
-special-purpose fields such as WWW-style linkages (URx).
-</para>
-
-<para>
-The field types, and hence character sets, are associated with data
-elements by the .abs files (see above). The file <literal remap="tt">default.idx</literal>
-provides the association between field type codes (as used in the .abs
-files) and the character map files (with the .chr suffix). The format
-of the .idx file is as follows
-</para>
-
-<para>
-<variablelist>
-
-<varlistentry>
-<term>index <emphasis remap="it">field type code</emphasis></term>
-<listitem>
-<para>
-This directive introduces a new
-search index code. The argument is a one-character code to be used in the
-.abs files to select this particular index type. An index, roughly,
-corresponds to a particular structure attribute during search. Refer
-to section <xref linkend="search"/>.
-</para>
-</listitem></varlistentry>
-<varlistentry>
-<term>sort <emphasis remap="it">field code type</emphasis></term>
-<listitem>
-<para>
-This directive introduces a
-sort index. The argument is a one-character code to be used in the
-.abs fie to select this particular index type. The corresponding
-use attribute must be used in the sort request to refer to this
-particular sort index. The corresponding character map (see below)
-is used in the sort process.
-</para>
-</listitem></varlistentry>
-<varlistentry>
-<term>completeness <emphasis remap="it">boolean</emphasis></term>
-<listitem>
-<para>
-This directive enables or disables
-complete field indexing. The value of the <emphasis remap="it">boolean</emphasis> should be 0
-(disable) or 1. If completeness is enabled, the index entry will
-contain the complete contents of the field (up to a limit), with words
-(non-space characters) separated by single space characters
-(normalized to " " on display). When completeness is
-disabled, each word is indexed as a separate entry. Complete subfield
-indexing is most useful for fields which are typically browsed (eg.
-titles, authors, or subjects), or instances where a match on a
-complete subfield is essential (eg. exact title searching). For fields
-where completeness is disabled, the search engine will interpret a
-search containing space characters as a word proximity search.
-</para>
-</listitem></varlistentry>
-<varlistentry>
-<term>charmap <emphasis remap="it">filename</emphasis></term>
-<listitem>
-<para>
-This is the filename of the character
-map to be used for this index for field type.
-</para>
-</listitem></varlistentry>
-</variablelist>
-</para>
-
-<para>
-The contents of the character map files are structured as follows:
-</para>
-
-<para>
-<variablelist>
-
-<varlistentry>
-<term>lowercase <emphasis remap="it">value-set</emphasis></term>
-<listitem>
-<para>
-This directive introduces the basic
-value set of the field type. The format is an ordered list (without
-spaces) of the characters which may occur in "words" of
-the given type. The order of the entries in the list determines the
-sort order of the index. In addition to single characters, the
-following combinations are legal:
-</para>
-
-<para>
-
-<itemizedlist>
-<listitem>
-
-<para>
-Backslashes may be used to introduce three-digit octal, or
-two-digit hex representations of single characters (preceded by <literal remap="tt">x</literal>).
-In addition, the combinations
-\\, \\r, \\n, \\t, \\s (space — remember that real space-characters
-may ot occur in the value definition), and \\ are recognised,
-with their usual interpretation.
-
-</para>
-</listitem>
-<listitem>
-
-<para>
-Curly braces {} may be used to enclose ranges of single
-characters (possibly using the escape convention described in the
-preceding point), eg. {a-z} to entroduce the standard range of ASCII
-characters. Note that the interpretation of such a range depends on
-the concrete representation in your local, physical character set.
-
-</para>
-</listitem>
-<listitem>
-
-<para>
-paranthesises () may be used to enclose multi-byte characters -
-eg. diacritics or special national combinations (eg. Spanish
-"ll"). When found in the input stream (or a search term),
-these characters are viewed and sorted as a single character, with a
-sorting value depending on the position of the group in the value
-statement.
-</para>
-</listitem>
-
-</itemizedlist>
-
-</para>
-</listitem></varlistentry>
-<varlistentry>
-<term>uppercase <emphasis remap="it">value-set</emphasis></term>
-<listitem>
-<para>
-This directive introduces the
-upper-case equivalencis to the value set (if any). The number and
-order of the entries in the list should be the same as in the
-<literal remap="tt">lowercase</literal> directive.
-</para>
-</listitem></varlistentry>
-<varlistentry>
-<term>space <emphasis remap="it">value-set</emphasis></term>
-<listitem>
-<para>
-This directive introduces the character
-which separate words in the input stream. Depending on the
-completeness mode of the field in question, these characters either
-terminate an index entry, or delimit individual "words" in
-the input stream. The order of the elements is not significant —
-otherwise the representation is the same as for the <literal remap="tt">upercase</literal> and
-<literal remap="tt">lowercase</literal> directives.
-</para>
-</listitem></varlistentry>
-<varlistentry>
-<term>map <emphasis remap="it">value-set</emphasis> <emphasis remap="it">target</emphasis></term>
-<listitem>
-<para>
-This directive introduces a
-mapping between each of the members of the value-set on the left to
-the character on the right. The character on the right must occur in
-the value set (the <literal remap="tt">lowercase</literal> directive) of the character set, but
-it may be a paranthesis-enclosed multi-octet character. This directive
-may be used to map diacritics to their base characters, or to map
-HTML-style character-representations to their natural form, etc.
-</para>
-</listitem></varlistentry>
-</variablelist>
-</para>
-
-</sect2>
-
-</sect1>
-
-<sect1 id="formats">
-<title>Exchange Formats</title>
-
-<para>
-Converting records from the internal structure to en exchange format
-is largely an automatic process. Currently, the following exchange
-formats are supported:
-</para>
-
-<para>
-
-<itemizedlist>
-<listitem>
-
-<para>
-GRS-1. The internal representation is based on GRS-1, so the
-conversion here is straightforward. The system will create
-applied variant and supported variant lists as required, if a record
-contains variant information.
-
-</para>
-</listitem>
-<listitem>
-
-<para>
-SUTRS. Again, the mapping is fairly straighforward. Indentation
-is used to show the hierarchical structure of the record. All
-"GRS" type records support both the GRS-1 and SUTRS
-representations.
-
-</para>
-</listitem>
-<listitem>
-
-<para>
-ISO2709-based formats (USMARC, etc.). Only records with a
-two-level structure (corresponding to fields and subfields) can be
-directly mapped to ISO2709. For records with a different structuring
-(eg., GILS), the representation in a structure like USMARC involves a
-schema-mapping (see section <xref linkend="schema-mapping"/>), to an
-"implied" USMARC schema (implied,
-because there is no formal schema which specifies the use of the
-USMARC fields outside of ISO2709). The resultant, two-level record is
-then mapped directly from the internal representation to ISO2709. See
-the GILS schema definition files for a detailed example of this
-approach.
-
-</para>
-</listitem>
-<listitem>
-
-<para>
-Explain. This representation is only available for records
-belonging to the Explain schema.
-
-</para>
-</listitem>
-<listitem>
-
-<para>
-Summary. This ASN-1 based structure is only available for records
-belonging to the Summary schema - or schema which provide a mapping
-to this schema (see the description of the schema mapping facility
-above).
-
-</para>
-</listitem>
-<listitem>
-
-<para>
-SOIF. Support for this syntax is experimental, and is currently
-keyed to a private Index Data OID (1.2.840.10003.5.1000.81.2). All
-abstract syntaxes can be mapped to the SOIF format, although nested
-elements are represented by concatenation of the tag names at each
-level.
-
-</para>
-</listitem>
-
-</itemizedlist>
-
-</para>
-
-</sect1>
+ <screen>
+ <title>
+ <var body iana "text/plain>
+ <var lang lang "eng">
+ Zen and the Art of Motorcycle Maintenance
+ <var lang lang "dan">
+ Zen og Kunsten at Vedligeholde en Motorcykel
+ </title>
+ </screen>
+
+ </para>
+
+ <para>
+ The title element above comes in two variants. Both have the IANA body
+ type "text/plain", but one is in English, and the other in
+ Danish. The client, using the element selection mechanism of Z39.50,
+ can retrieve information about the available variant forms of data
+ elements, or it can select specific variants based on the requirements
+ of the end-user.
+ </para>
+
+ </sect3>
+
+ </sect2>
+
+ <sect2>
+ <title>Input Filters</title>
+
+ <para>
+ In order to handle general input formats, Zebra allows the
+ operator to define filters which read individual records in their
+ native format and produce an internal representation that the system
+ can work with.
+ </para>
+
+ <para>
+ Input filters are ASCII files, generally with the suffix
+ <literal>.flt</literal>.
+ The system looks for the files in the directories given in the
+ <emphasis>profilePath</emphasis> setting in the
+ <literal>zebra.cfg</literal> files.
+ The record type for the filter is
+ <literal>grs.regx.</literal><emphasis>filter-filename</emphasis>
+ (fundamental type <literal>grs</literal>, file read
+ type <literal>regx</literal>, argument
+ <emphasis>filter-filename</emphasis>).
+ </para>
+
+ <para>
+ Generally, an input filter consists of a sequence of rules, where each
+ rule consists of a sequence of expressions, followed by an action. The
+ expressions are evaluated against the contents of the input record,
+ and the actions normally contribute to the generation of an internal
+ representation of the record.
+ </para>
+
+ <para>
+ An expression can be either of the following:
+ </para>
+
+ <para>
+ <variablelist>
+
+ <varlistentry>
+ <term>INIT</term>
+ <listitem>
+ <para>
+ The action associated with this expression is evaluated
+ exactly once in the lifetime of the application, before any records
+ are read. It can be used in conjunction with an action that
+ initializes tables or other resources that are used in the processing
+ of input records.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term>BEGIN</term>
+ <listitem>
+ <para>
+ Matches the beginning of the record. It can be used to
+ initialize variables, etc. Typically, the
+ <emphasis>BEGIN</emphasis> rule is also used
+ to establish the root node of the record.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term>END</term>
+ <listitem>
+ <para>
+ Matches the end of the record - when all of the contents
+ of the record has been processed.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term>/pattern/</term>
+ <listitem>
+ <para>
+ Matches a string of characters from the input record.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term>BODY</term>
+ <listitem>
+ <para>
+ This keyword may only be used between two patterns.
+ It matches everything between (not including) those patterns.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term>FINISH</term>
+ <listitem>
+ <para>
+ The expression asssociated with this pattern is evaluated
+ once, before the application terminates. It can be used to release
+ system resources - typically ones allocated in the
+ <emphasis>INIT</emphasis> step.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
+
+ <para>
+ An action is surrounded by curly braces ({...}), and
+ consists of a sequence of statements. Statements may be separated
+ by newlines or semicolons (;).
+ Within actions, the strings that matched the expressions
+ immediately preceding the action can be referred to as
+ $0, $1, $2, etc.
+ </para>
+
+ <para>
+ The available statements are:
+ </para>
+
+ <para>
+ <variablelist>
+
+ <varlistentry>
+ <term>begin <emphasis>type [parameter ... ]</emphasis></term>
+ <listitem>
+ <para>
+ Begin a new
+ data element. The type is one of the following:
+ <variablelist>
+
+ <varlistentry>
+ <term>record</term>
+ <listitem>
+ <para>
+ Begin a new record. The followingparameter should be the
+ name of the schema that describes the structure of the record, eg.
+ <literal>gils</literal> or <literal>wais</literal> (see below).
+ The <literal>begin record</literal> call should precede
+ any other use of the <emphasis>begin</emphasis> statement.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term>element</term>
+ <listitem>
+ <para>
+ Begin a new tagged element. The parameter is the
+ name of the tag. If the tag is not matched anywhere in the tagsets
+ referenced by the current schema, it is treated as a local string
+ tag.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term>variant</term>
+ <listitem>
+ <para>
+ Begin a new node in a variant tree. The parameters are
+ <emphasis>class type value</emphasis>.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term>data</term>
+ <listitem>
+ <para>
+ Create a data element. The concatenated arguments make
+ up the value of the data element.
+ The option <literal>-text</literal> signals that
+ the layout (whitespace) of the data should be retained for
+ transmission.
+ The option <literal>-element</literal>
+ <emphasis>tag</emphasis> wraps the data up in
+ the <emphasis>tag</emphasis>.
+ The use of the <literal>-element</literal> option is equivalent to
+ preceding the command with a <emphasis>begin
+ element</emphasis> command, and following
+ it with the <emphasis>end</emphasis> command.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term>end <emphasis>[type]</emphasis></term>
+ <listitem>
+ <para>
+ Close a tagged element. If no parameter is given,
+ the last element on the stack is terminated.
+ The first parameter, if any, is a type name, similar
+ to the <emphasis>begin</emphasis> statement.
+ For the <emphasis>element</emphasis> type, a tag
+ name can be provided to terminate a specific tag.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
+
+ <para>
+ The following input filter reads a Usenet news file, producing a
+ record in the WAIS schema. Note that the body of a news posting is
+ separated from the list of headers by a blank line (or rather a
+ sequence of two newline characters.
+ </para>
+
+ <para>
+
+ <screen>
+ BEGIN { begin record wais }
+
+ /^From:/ BODY /$/ { data -element name $1 }
+ /^Subject:/ BODY /$/ { data -element title $1 }
+ /^Date:/ BODY /$/ { data -element lastModified $1 }
+ /\n\n/ BODY END {
+ begin element bodyOfDisplay
+ begin variant body iana "text/plain"
+ data -text $1
+ end record
+ }
+ </screen>
+
+ </para>
+
+ <para>
+ If Zebra is compiled with support for Tcl (Tool Command Language)
+ enabled, the statements described above are supplemented with a complete
+ scripting environment, including control structures (conditional
+ expressions and loop constructs), and powerful string manipulation
+ mechanisms for modifying the elements of a record. Tcl is a popular
+ scripting environment, with several tutorials available both online
+ and in hardcopy.
+ </para>
+
+ <para>
+ <emphasis>NOTE: Tcl support is not currently available, but will be
+ included with one of the next alpha or beta releases.</emphasis>
+ </para>
+
+ <para>
+ <emphasis>NOTE: Variant support is not currently available in the input
+ filter, but will be included with one of the next alpha or beta
+ releases.</emphasis>
+ </para>
+
+ </sect2>
+
+ </sect1>
+
+ <sect1 id="internal-representation">
+ <title>Internal Representation</title>
+
+ <para>
+ When records are manipulated by the system, they're represented in a
+ tree-structure, with data elements at the leaf nodes, and tags or
+ variant components at the non-leaf nodes. The root-node identifies the
+ schema that lends context to the tagging and structuring of the
+ record. Imagine a simple record, consisting of a 'title' element and
+ an 'author' element:
+ </para>
+
+ <para>
+
+ <screen>
+ TITLE "Zen and the Art of Motorcycle Maintenance"
+ ROOT
+ AUTHOR "Robert Pirsig"
+ </screen>
+
+ </para>
+
+ <para>
+ A slightly more complex record would have the author element consist
+ of two elements, a surname and a first name:
+ </para>
+
+ <para>
+
+ <screen>
+ TITLE "Zen and the Art of Motorcycle Maintenance"
+ ROOT
+ FIRST-NAME "Robert"
+ AUTHOR
+ SURNAME "Pirsig"
+ </screen>
+
+ </para>
+
+ <para>
+ The root of the record will refer to the record schema that describes
+ the structuring of this particular record. The schema defines the
+ element tags (TITLE, FIRST-NAME, etc.) that may occur in the record, as
+ well as the structuring (SURNAME should appear below AUTHOR, etc.). In
+ addition, the schema establishes element set names that are used by
+ the client to request a subset of the elements of a given record. The
+ schema may also establish rules for converting the record to a
+ different schema, by stating, for each element, a mapping to a
+ different tag path.
+ </para>
+
+ <sect2>
+ <title>Tagged Elements</title>
+
+ <para>
+ A data element is characterized by its tag, and its position in the
+ structure of the record. For instance, while the tag "telephone
+ number" may be used different places in a record, we may need to
+ distinguish between these occurrences, both for searching and
+ presentation purposes. For instance, while the phone numbers for the
+ "customer" and the "service provider" are both
+ representatives for the same type of resource (a telephone number), it
+ is essential that they be kept separate. The record schema provides
+ the structure of the record, and names each data element (defined by
+ the sequence of tags - the tag path - by which the element can be
+ reached from the root of the record).
+ </para>
+
+ </sect2>
+
+ <sect2>
+ <title>Variants</title>
+
+ <para>
+ The children of a tag node may be either more tag nodes, a data node
+ (possibly accompanied by tag nodes),
+ or a tree of variant nodes. The children of variant nodes are either
+ more variant nodes or a data node (possibly accompanied by more
+ variant nodes). Each leaf node, which is normally a
+ data node, corresponds to a <emphasis>variant form</emphasis> of the
+ tagged element identified by the tag which parents the variant tree.
+ The following title element occurs in two different languages:
+ </para>
+
+ <para>
+
+ <screen>
+ VARIANT LANG=ENG "War and Peace"
+ TITLE
+ VARIANT LANG=DAN "Krig og Fred"
+ </screen>
+
+ </para>
+
+ <para>
+ Which of the two elements are transmitted to the client by the server
+ depends on the specifications provided by the client, if any.
+ </para>
+
+ <para>
+ In practice, each variant node is associated with a triple of class,
+ type, value, corresponding to the variant mechanism of Z39.50.
+ </para>
+
+ </sect2>
+
+ <sect2>
+ <title>Data Elements</title>
+
+ <para>
+ Data nodes have no children (they are always leaf nodes in the record
+ tree).
+ </para>
+
+ <note>
+ <para>
+ Documentation needs extension here about types of nodes - numerical,
+ textual, etc., plus the various types of inclusion notes.
+ </para>
+ </note>
+
+ </sect2>
+
+ </sect1>
+
+ <sect1 id="data-model">
+ <title>Configuring Your Data Model</title>
+
+ <para>
+ The following sections describe the configuration files that govern
+ the internal management of data records. The system searches for the files
+ in the directories specified by the <emphasis>profilePath</emphasis>
+ setting in the <literal>zebra.cfg</literal> file.
+ </para>
+
+ <sect2>
+ <title>The Abstract Syntax</title>
+
+ <para>
+ The abstract syntax definition (also known as an Abstract Record
+ Structure, or ARS) is the focal point of the
+ record schema description. For a given schema, the ABS file may state any
+ or all of the following:
+ </para>
+
+ <para>
+
+ <itemizedlist>
+ <listitem>
+
+ <para>
+ The object identifier of the Z39.50 schema associated
+ with the ARS, so that it can be referred to by the client.
+ </para>
+ </listitem>
+
+ <listitem>
+ <para>
+ The attribute set (which can possibly be a compound of multiple
+ sets) which applies in the profile. This is used when indexing and
+ searching the records belonging to the given profile.
+ </para>
+ </listitem>
+
+ <listitem>
+ <para>
+ The Tag set (again, this can consist of several different sets).
+ This is used when reading the records from a file, to recognize the
+ different tags, and when transmitting the record to the client -
+ mapping the tags to their numerical representation, if they are
+ known.
+ </para>
+ </listitem>
+
+ <listitem>
+ <para>
+ The variant set which is used in the profile. This provides a
+ vocabulary for specifying the <emphasis>forms</emphasis> of data that appear inside
+ the records.
+ </para>
+ </listitem>
+
+ <listitem>
+ <para>
+ Element set names, which are a shorthand way for the client to
+ ask for a subset of the data elements contained in a record. Element
+ set names, in the retrieval module, are mapped to <emphasis>element
+ specifications</emphasis>, which contain information equivalent to the
+ <emphasis>Espec-1</emphasis> syntax of Z39.50.
+ </para>
+ </listitem>
+
+ <listitem>
+ <para>
+ Map tables, which may specify mappings to
+ <emphasis>other</emphasis> database profiles, if desired.
+ </para>
+ </listitem>
+
+ <listitem>
+ <para>
+ Possibly, a set of rules describing the mapping of elements to a
+ MARC representation.
+
+ </para>
+ </listitem>
+
+ <listitem>
+ <para>
+ A list of element descriptions (this is the actual ARS of the
+ schema, in Z39.50 terms), which lists the ways in which the various
+ tags can be used and organized hierarchically.
+ </para>
+ </listitem>
+
+ </itemizedlist>
+
+ </para>
+
+ <para>
+ Several of the entries above simply refer to other files, which
+ describe the given objects.
+ </para>
+
+ </sect2>
+
+ <sect2>
+ <title>The Configuration Files</title>
+
+ <para>
+ This section describes the syntax and use of the various tables which
+ are used by the retrieval module.
+ </para>
+
+ <para>
+ The number of different file types may appear daunting at first, but
+ each type corresponds fairly clearly to a single aspect of the Z39.50
+ retrieval facilities. Further, the average database administrator,
+ who is simply reusing an existing profile for which tables already
+ exist, shouldn't have to worry too much about the contents of these tables.
+ </para>
+
+ <para>
+ Generally, the files are simple ASCII files, which can be maintained
+ using any text editor. Blank lines, and lines beginning with a (#) are
+ ignored. Any characters on a line followed by a (#) are also ignored.
+ All other lines contain <emphasis>directives</emphasis>, which provide
+ some setting or value to the system.
+ Generally, settings are characterized by a single
+ keyword, identifying the setting, followed by a number of parameters.
+ Some settings are repeatable (r), while others may occur only once in a
+ file. Some settings are optional (o), whicle others again are
+ mandatory (m).
+ </para>
+
+ </sect2>
+
+ <sect2>
+ <title>The Abstract Syntax (.abs) Files</title>
+
+ <para>
+ The name of this file type is slightly misleading in Z39.50 terms,
+ since, apart from the actual abstract syntax of the profile, it also
+ includes most of the other definitions that go into a database
+ profile.
+ </para>
+
+ <para>
+ When a record in the canonical, SGML-like format is read from a file
+ or from the database, the first tag of the file should reference the
+ profile that governs the layout of the record. If the first tag of the
+ record is, say, <literal><gils></literal>, the system will look
+ for the profile definition in the file <literal>gils.abs</literal>.
+ Profile definitions are cached, so they only have to be read once
+ during the lifespan of the current process.
+ </para>
+
+ <para>
+ When writing your own input filters, the
+ <emphasis>record-begin</emphasis> command
+ introduces the profile, and should always be called first thing when
+ introducing a new record.
+ </para>
+
+ <para>
+ The file may contain the following directives:
+ </para>
+
+ <para>
+ <variablelist>
+
+ <varlistentry>
+ <term>name <emphasis>symbolic-name</emphasis></term>
+ <listitem>
+ <para>
+ (m) This provides a shorthand name or
+ description for the profile. Mostly useful for diagnostic purposes.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term>reference <emphasis>OID-name</emphasis></term>
+ <listitem>
+ <para>
+ (m) The reference name of the OID for the profile.
+ The reference names can be found in the <emphasis>util</emphasis>
+ module of <emphasis>YAZ</emphasis>.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term>attset <emphasis>filename</emphasis></term>
+ <listitem>
+ <para>
+ (m) The attribute set that is used for
+ indexing and searching records belonging to this profile.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term>tagset <emphasis>filename</emphasis></term>
+ <listitem>
+ <para>
+ (o) The tag set (if any) that describe
+ that fields of the records.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term>varset <emphasis>filename</emphasis></term>
+ <listitem>
+ <para>
+ (o) The variant set used in the profile.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term>maptab <emphasis>filename</emphasis></term>
+ <listitem>
+ <para>
+ (o,r) This points to a
+ conversion table that might be used if the client asks for the record
+ in a different schema from the native one.
+ </para>
+ </listitem></varlistentry>
+ <varlistentry>
+ <term>marc <emphasis>filename</emphasis></term>
+ <listitem>
+ <para>
+ (o) Points to a file containing parameters
+ for representing the record contents in the ISO2709 syntax. Read the
+ description of the MARC representation facility below.
+ </para>
+ </listitem></varlistentry>
+ <varlistentry>
+ <term>esetname <emphasis>name filename</emphasis></term>
+ <listitem>
+ <para>
+ (o,r) Associates the
+ given element set name with an element selection file. If an (@) is
+ given in place of the filename, this corresponds to a null mapping for
+ the given element set name.
+ </para>
+ </listitem></varlistentry>
+ <varlistentry>
+ <term>any <emphasis>tags</emphasis></term>
+ <listitem>
+ <para>
+ (o) This directive specifies a list of attributes
+ which should be appended to the attribute list given for each
+ element. The effect is to make every single element in the abstract
+ syntax searchable by way of the given attributes. This directive
+ provides an efficient way of supporting free-text searching across all
+ elements. However, it does increase the size of the index
+ significantly. The attributes can be qualified with a structure, as in
+ the <emphasis>elm</emphasis> directive below.
+ </para>
+ </listitem></varlistentry>
+ <varlistentry>
+ <term>elm <emphasis>path name attributes</emphasis></term>
+ <listitem>
+ <para>
+ (o,r) Adds an element to the abstract record syntax of the schema.
+ The <emphasis>path</emphasis> follows the
+ syntax which is suggested by the Z39.50 document - that is, a sequence
+ of tags separated by slashes (/). Each tag is given as a
+ comma-separated pair of tag type and -value surrounded by parenthesis.
+ The <emphasis>name</emphasis> is the name of the element, and
+ the <emphasis>attributes</emphasis>
+ specifies which attributes to use when indexing the element in a
+ comma-separated list.
+ A ! in place of the attribute name is equivalent to
+ specifying an attribute name identical to the element name.
+ A - in place of the attribute name
+ specifies that no indexing is to take place for the given element.
+ The attributes can be qualified with <emphasis>field
+ types</emphasis> to specify which
+ character set should govern the indexing procedure for that field.
+ The same data element may be indexed into several different
+ fields, using different character set definitions.
+ See the section <xref linkend="field-structure-and-character-sets"/>.
+ The default field type is "w" for <emphasis>word</emphasis>.
+ </para>
+ </listitem></varlistentry>
+ </variablelist>
+ </para>
+
+ <note>
+ <para>
+ The mechanism for controlling indexing is not adequate for
+ complex databases, and will probably be moved into a separate
+ configuration table eventually.
+ </para>
+ </note>
+
+ <para>
+ The following is an excerpt from the abstract syntax file for the GILS
+ profile.
+ </para>
+
+ <para>
+
+ <screen>
+ name gils
+ reference GILS-schema
+ attset gils.att
+ tagset gils.tag
+ varset var1.var
+
+ maptab gils-usmarc.map
+
+ # Element set names
+
+ esetname VARIANT gils-variant.est # for WAIS-compliance
+ esetname B gils-b.est
+ esetname G gils-g.est
+ esetname F @
+
+ elm (1,10) rank -
+ elm (1,12) url -
+ elm (1,14) localControlNumber Local-number
+ elm (1,16) dateOfLastModification Date/time-last-modified
+ elm (2,1) title w:!,p:!
+ elm (4,1) controlIdentifier Identifier-standard
+ elm (2,6) abstract Abstract
+ elm (4,51) purpose !
+ elm (4,52) originator -
+ elm (4,53) accessConstraints !
+ elm (4,54) useConstraints !
+ elm (4,70) availability -
+ elm (4,70)/(4,90) distributor -
+ elm (4,70)/(4,90)/(2,7) distributorName !
+ elm (4,70)/(4,90)/(2,10 distributorOrganization !
+ elm (4,70)/(4,90)/(4,2) distributorStreetAddress !
+ elm (4,70)/(4,90)/(4,3) distributorCity !
+ </screen>
+
+ </para>
+
+ </sect2>
+
+ <sect2 id="attset-files">
+ <title>The Attribute Set (.att) Files</title>
+
+ <para>
+ This file type describes the <emphasis>Use</emphasis> elements of
+ an attribute set.
+ It contains the following directives.
+ </para>
+
+ <para>
+ <variablelist>
+ <varlistentry>
+ <term>name <emphasis>symbolic-name</emphasis></term>
+ <listitem>
+ <para>
+ (m) This provides a shorthand name or
+ description for the attribute set.
+ Mostly useful for diagnostic purposes.
+ </para>
+ </listitem></varlistentry>
+ <varlistentry>
+ <term>reference <emphasis>OID-name</emphasis></term>
+ <listitem>
+ <para>
+ (m) The reference name of the OID for
+ the attribute set.
+ The reference names can be found in the <emphasis>util</emphasis>
+ module of <emphasis>YAZ</emphasis>.
+ </para>
+ </listitem></varlistentry>
+ <varlistentry>
+ <term>include <emphasis>filename</emphasis></term>
+ <listitem>
+ <para>
+ (o,r) This directive is used to
+ include another attribute set as a part of the current one. This is
+ used when a new attribute set is defined as an extension to another
+ set. For instance, many new attribute sets are defined as extensions
+ to the <emphasis>bib-1</emphasis> set.
+ This is an important feature of the retrieval
+ system of Z39.50, as it ensures the highest possible level of
+ interoperability, as those access points of your database which are
+ derived from the external set (say, bib-1) can be used even by clients
+ who are unaware of the new set.
+ </para>
+ </listitem></varlistentry>
+ <varlistentry>
+ <term>att
+ <emphasis>att-value att-name [local-value]</emphasis></term>
+ <listitem>
+ <para>
+ (o,r) This
+ repeatable directive introduces a new attribute to the set. The
+ attribute value is stored in the index (unless a
+ <emphasis>local-value</emphasis> is
+ given, in which case this is stored). The name is used to refer to the
+ attribute from the <emphasis>abstract syntax</emphasis>.
+ </para>
+ </listitem></varlistentry>
+ </variablelist>
+ </para>
+
+ <para>
+ This is an excerpt from the GILS attribute set definition.
+ Notice how the file describing the <emphasis>bib-1</emphasis>
+ attribute set is referenced.
+ </para>
+
+ <para>
+
+ <screen>
+ name gils
+ reference GILS-attset
+ include bib1.att
+
+ att 2001 distributorName
+ att 2002 indextermsControlled
+ att 2003 purpose
+ att 2004 accessConstraints
+ att 2005 useConstraints
+ </screen>
+
+ </para>
+
+ </sect2>
+
+ <sect2>
+ <title>The Tag Set (.tag) Files</title>
+
+ <para>
+ This file type defines the tagset of the profile, possibly by
+ referencing other tag sets (most tag sets, for instance, will include
+ tagsetG and tagsetM from the Z39.50 specification. The file may
+ contain the following directives.
+ </para>
+
+ <para>
+ <variablelist>
+
+ <varlistentry>
+ <term>name <emphasis>symbolic-name</emphasis></term>
+ <listitem>
+ <para>
+ (m) This provides a shorthand name or
+ description for the tag set. Mostly useful for diagnostic purposes.
+ </para>
+ </listitem></varlistentry>
+ <varlistentry>
+ <term>reference <emphasis>OID-name</emphasis></term>
+ <listitem>
+ <para>
+ (o) The reference name of the OID for the tag set.
+ The reference names can be found in the <emphasis>util</emphasis>
+ module of <emphasis>YAZ</emphasis>.
+ The directive is optional, since not all tag sets
+ are registered outside of their schema.
+ </para>
+ </listitem></varlistentry>
+ <varlistentry>
+ <term>type <emphasis>integer</emphasis></term>
+ <listitem>
+ <para>
+ (m) The type number of the tagset within the schema
+ profile (note: this specification really should belong to the .abs
+ file. This will be fixed in a future release).
+ </para>
+ </listitem></varlistentry>
+ <varlistentry>
+ <term>include <emphasis>filename</emphasis></term>
+ <listitem>
+ <para>
+ (o,r) This directive is used
+ to include the definitions of other tag sets into the current one.
+ </para>
+ </listitem></varlistentry>
+ <varlistentry>
+ <term>tag <emphasis>number names type</emphasis></term>
+ <listitem>
+ <para>
+ (o,r) Introduces a new tag to the set.
+ The <emphasis>number</emphasis> is the tag number as used
+ in the protocol (there is currently no mechanism for
+ specifying string tags at this point, but this would be quick
+ work to add).
+ The <emphasis>names</emphasis> parameter is a list of names
+ by which the tag should be recognized in the input file format.
+ The names should be separated by slashes (/).
+ The <emphasis>type</emphasis> is th recommended datatype of
+ the tag.
+ It should be one of the following:
+
+ <itemizedlist>
+ <listitem>
+ <para>
+ structured
+ </para>
+ </listitem>
+
+ <listitem>
+ <para>
+ string
+ </para>
+ </listitem>
+
+ <listitem>
+ <para>
+ numeric
+ </para>
+ </listitem>
+
+ <listitem>
+ <para>
+ bool
+ </para>
+ </listitem>
+
+ <listitem>
+ <para>
+ oid
+ </para>
+ </listitem>
+
+ <listitem>
+ <para>
+ generalizedtime
+ </para>
+ </listitem>
+
+ <listitem>
+ <para>
+ intunit
+ </para>
+ </listitem>
+
+ <listitem>
+ <para>
+ int
+ </para>
+ </listitem>
+
+ <listitem>
+ <para>
+ octetstring
+ </para>
+ </listitem>
+
+ <listitem>
+ <para>
+ null
+ </para>
+ </listitem>
+
+ </itemizedlist>
+
+ </para>
+ </listitem></varlistentry>
+ </variablelist>
+ </para>
+
+ <para>
+ The following is an excerpt from the TagsetG definition file.
+ </para>
+
+ <para>
+ <screen>
+ name tagsetg
+ reference TagsetG
+ type 2
+
+ tag 1 title string
+ tag 2 author string
+ tag 3 publicationPlace string
+ tag 4 publicationDate string
+ tag 5 documentId string
+ tag 6 abstract string
+ tag 7 name string
+ tag 8 date generalizedtime
+ tag 9 bodyOfDisplay string
+ tag 10 organization string
+ </screen>
+ </para>
+
+ </sect2>
+
+ <sect2 id="variant-set">
+ <title>The Variant Set (.var) Files</title>
+
+ <para>
+ The variant set file is a straightforward representation of the
+ variant set definitions associated with the protocol. At present, only
+ the <emphasis>Variant-1</emphasis> set is known.
+ </para>
+
+ <para>
+ These are the directives allowed in the file.
+ </para>
+
+ <para>
+ <variablelist>
+
+ <varlistentry>
+ <term>name <emphasis>symbolic-name</emphasis></term>
+ <listitem>
+ <para>
+ (m) This provides a shorthand name or
+ description for the variant set. Mostly useful for diagnostic purposes.
+ </para>
+ </listitem></varlistentry>
+ <varlistentry>
+ <term>reference <emphasis>OID-name</emphasis></term>
+ <listitem>
+ <para>
+ (o) The reference name of the OID for
+ the variant set, if one is required. The reference names can be found
+ in the <emphasis>util</emphasis> module of <emphasis>YAZ</emphasis>.
+ </para>
+ </listitem></varlistentry>
+ <varlistentry>
+ <term>class <emphasis>integer class-name</emphasis></term>
+ <listitem>
+ <para>
+ (m,r) Introduces a new
+ class to the variant set.
+ </para>
+ </listitem></varlistentry>
+ <varlistentry>
+ <term>type <emphasis>integer type-name datatype</emphasis></term>
+ <listitem>
+ <para>
+ (m,r) Addes a
+ new type to the current class (the one introduced by the most recent
+ <emphasis>class</emphasis> directive).
+ The type names belong to the same name space as the one used
+ in the tag set definition file.
+ </para>
+ </listitem></varlistentry>
+ </variablelist>
+ </para>
+
+ <para>
+ The following is an excerpt from the file describing the variant set
+ <emphasis>Variant-1</emphasis>.
+ </para>
+
+ <para>
+
+ <screen>
+ name variant-1
+ reference Variant-1
+
+ class 1 variantId
+
+ type 1 variantId octetstring
+
+ class 2 body
+
+ type 1 iana string
+ type 2 z39.50 string
+ type 3 other string
+ </screen>
+
+ </para>
+
+ </sect2>
+
+ <sect2>
+ <title>The Element Set (.est) Files</title>
+
+ <para>
+ The element set specification files describe a selection of a subset
+ of the elements of a database record. The element selection mechanism
+ is equivalent to the one supplied by the <emphasis>Espec-1</emphasis>
+ syntax of the Z39.50 specification.
+ In fact, the internal representation of an element set
+ specification is identical to the <emphasis>Espec-1</emphasis> structure,
+ and we'll refer you to the description of that structure for most of
+ the detailed semantics of the directives below.
+ </para>
+
+ <note>
+ <para>
+ Not all of the Espec-1 functionality has been implemented yet.
+ The fields that are mentioned below all work as expected, unless
+ otherwise is noted.
+ </para>
+ </note>
+
+ <para>
+ The directives available in the element set file are as follows:
+ </para>
+
+ <para>
+ <variablelist>
+ <varlistentry>
+ <term>defaultVariantSetId <emphasis>OID-name</emphasis></term>
+ <listitem>
+ <para>
+ (o) If variants are used in
+ the following, this should provide the name of the variantset used
+ (it's not currently possible to specify a different set in the
+ individual variant request). In almost all cases (certainly all
+ profiles known to us), the name
+ <literal>Variant-1</literal> should be given here.
+ </para>
+ </listitem></varlistentry>
+ <varlistentry>
+ <term>defaultVariantRequest <emphasis>variant-request</emphasis></term>
+ <listitem>
+ <para>
+ (o) This directive
+ provides a default variant request for
+ use when the individual element requests (see below) do not contain a
+ variant request. Variant requests consist of a blank-separated list of
+ variant components. A variant compont is a comma-separated,
+ parenthesized triple of variant class, type, and value (the two former
+ values being represented as integers). The value can currently only be
+ entered as a string (this will change to depend on the definition of
+ the variant in question). The special value (@) is interpreted as a
+ null value, however.
+ </para>
+ </listitem></varlistentry>
+ <varlistentry>
+ <term>simpleElement
+ <emphasis>path ['variant' variant-request]</emphasis></term>
+ <listitem>
+ <para>
+ (o,r) This corresponds to a simple element request
+ in <emphasis>Espec-1</emphasis>.
+ The path consists of a sequence of tag-selectors, where each of
+ these can consist of either:
+ </para>
+
+ <para>
+ <itemizedlist>
+ <listitem>
+ <para>
+ A simple tag, consisting of a comma-separated type-value pair in
+ parenthesis, possibly followed by a colon (:) followed by an
+ occurrences-specification (see below). The tag-value can be a number
+ or a string. If the first character is an apostrophe ('), this
+ forces the value to be interpreted as a string, even if it
+ appears to be numerical.
+ </para>
+ </listitem>
+
+ <listitem>
+ <para>
+ A WildThing, represented as a question mark (?), possibly
+ followed by a colon (:) followed by an occurrences
+ specification (see below).
+ </para>
+ </listitem>
+
+ <listitem>
+ <para>
+ A WildPath, represented as an asterisk (*). Note that the last
+ element of the path should not be a wildPath (wildpaths don't
+ work in this version).
+ </para>
+ </listitem>
+
+ </itemizedlist>
+
+ </para>
+
+ <para>
+ The occurrences-specification can be either the string
+ <literal>all</literal>, the string <literal>last</literal>, or
+ an explicit value-range. The value-range is represented as
+ an integer (the starting point), possibly followed by a
+ plus (+) and a second integer (the number of elements, default
+ being one).
+ </para>
+
+ <para>
+ The variant-request has the same syntax as the defaultVariantRequest
+ above. Note that it may sometimes be useful to give an empty variant
+ request, simply to disable the default for a specific set of fields
+ (we aren't certain if this is proper <emphasis>Espec-1</emphasis>,
+ but it works in this implementation).
+ </para>
+ </listitem></varlistentry>
+ </variablelist>
+ </para>
+
+ <para>
+ The following is an example of an element specification belonging to
+ the GILS profile.
+ </para>
+
+ <para>
+
+ <screen>
+ simpleelement (1,10)
+ simpleelement (1,12)
+ simpleelement (2,1)
+ simpleelement (1,14)
+ simpleelement (4,1)
+ simpleelement (4,52)
+ </screen>
+
+ </para>
+
+ </sect2>
+
+ <sect2 id="schema-mapping">
+ <title>The Schema Mapping (.map) Files</title>
+
+ <para>
+ Sometimes, the client might want to receive a database record in
+ a schema that differs from the native schema of the record. For
+ instance, a client might only know how to process WAIS records, while
+ the database record is represented in a more specific schema, such as
+ GILS. In this module, a mapping of data to one of the MARC formats is
+ also thought of as a schema mapping (mapping the elements of the
+ record into fields consistent with the given MARC specification, prior
+ to actually converting the data to the ISO2709). This use of the
+ object identifier for USMARC as a schema identifier represents an
+ overloading of the OID which might not be entirely proper. However,
+ it represents the dual role of schema and record syntax which
+ is assumed by the MARC family in Z39.50.
+ </para>
+
+ <para>
+ <emphasis>NOTE: The schema-mapping functions are so far limited to a
+ straightforward mapping of elements. This should be extended with
+ mechanisms for conversions of the element contents, and conditional
+ mappings of elements based on the record contents.</emphasis>
+ </para>
+
+ <para>
+ These are the directives of the schema mapping file format:
+ </para>
+
+ <para>
+ <variablelist>
+
+ <varlistentry>
+ <term>targetName <emphasis>name</emphasis></term>
+ <listitem>
+ <para>
+ (m) A symbolic name for the target schema
+ of the table. Useful mostly for diagnostic purposes.
+ </para>
+ </listitem></varlistentry>
+ <varlistentry>
+ <term>targetRef <emphasis>OID-name</emphasis></term>
+ <listitem>
+ <para>
+ (m) An OID name for the target schema.
+ This is used, for instance, by a server receiving a request to present
+ a record in a different schema from the native one.
+ The name, again, is found in the <emphasis>oid</emphasis>
+ module of <emphasis>YAZ</emphasis>.
+ </para>
+ </listitem></varlistentry>
+ <varlistentry>
+ <term>map <emphasis>element-name target-path</emphasis></term>
+ <listitem>
+ <para>
+ (o,r) Adds
+ an element mapping rule to the table.
+ </para>
+ </listitem></varlistentry>
+ </variablelist>
+ </para>
+
+ </sect2>
+
+ <sect2>
+ <title>The MARC (ISO2709) Representation (.mar) Files</title>
+
+ <para>
+ This file provides rules for representing a record in the ISO2709
+ format. The rules pertain mostly to the values of the constant-length
+ header of the record.
+ </para>
+
+ <para>
+ <emphasis>NOTE: This will be described better. We're in the process of
+ re-evaluating and most likely changing the way that MARC records are
+ handled by the system.</emphasis>
+ </para>
+
+ </sect2>
+
+ <sect2 id="field-structure-and-character-sets">
+ <title>Field Structure and Character Sets
+ </title>
+
+ <para>
+ In order to provide a flexible approach to national character set
+ handling, Zebra allows the administrator to configure the set up the
+ system to handle any 8-bit character set — including sets that
+ require multi-octet diacritics or other multi-octet characters. The
+ definition of a character set includes a specification of the
+ permissible values, their sort order (this affects the display in the
+ SCAN function), and relationships between upper- and lowercase
+ characters. Finally, the definition includes the specification of
+ space characters for the set.
+ </para>
+
+ <para>
+ The operator can define different character sets for different fields,
+ typical examples being standard text fields, numerical fields, and
+ special-purpose fields such as WWW-style linkages (URx).
+ </para>
+
+ <para>
+ The field types, and hence character sets, are associated with data
+ elements by the .abs files (see above).
+ The file <literal>default.idx</literal>
+ provides the association between field type codes (as used in the .abs
+ files) and the character map files (with the .chr suffix). The format
+ of the .idx file is as follows
+ </para>
+
+ <para>
+ <variablelist>
+
+ <varlistentry>
+ <term>index <emphasis>field type code</emphasis></term>
+ <listitem>
+ <para>
+ This directive introduces a new search index code.
+ The argument is a one-character code to be used in the
+ .abs files to select this particular index type. An index, roughly,
+ corresponds to a particular structure attribute during search. Refer
+ to section <xref linkend="search"/>.
+ </para>
+ </listitem></varlistentry>
+ <varlistentry>
+ <term>sort <emphasis>field code type</emphasis></term>
+ <listitem>
+ <para>
+ This directive introduces a
+ sort index. The argument is a one-character code to be used in the
+ .abs fie to select this particular index type. The corresponding
+ use attribute must be used in the sort request to refer to this
+ particular sort index. The corresponding character map (see below)
+ is used in the sort process.
+ </para>
+ </listitem></varlistentry>
+ <varlistentry>
+ <term>completeness <emphasis>boolean</emphasis></term>
+ <listitem>
+ <para>
+ This directive enables or disables complete field indexing.
+ The value of the <emphasis>boolean</emphasis> should be 0
+ (disable) or 1. If completeness is enabled, the index entry will
+ contain the complete contents of the field (up to a limit), with words
+ (non-space characters) separated by single space characters
+ (normalized to " " on display). When completeness is
+ disabled, each word is indexed as a separate entry. Complete subfield
+ indexing is most useful for fields which are typically browsed (eg.
+ titles, authors, or subjects), or instances where a match on a
+ complete subfield is essential (eg. exact title searching). For fields
+ where completeness is disabled, the search engine will interpret a
+ search containing space characters as a word proximity search.
+ </para>
+ </listitem></varlistentry>
+ <varlistentry>
+ <term>charmap <emphasis>filename</emphasis></term>
+ <listitem>
+ <para>
+ This is the filename of the character
+ map to be used for this index for field type.
+ </para>
+ </listitem></varlistentry>
+ </variablelist>
+ </para>
+
+ <para>
+ The contents of the character map files are structured as follows:
+ </para>
+
+ <para>
+ <variablelist>
+
+ <varlistentry>
+ <term>lowercase <emphasis>value-set</emphasis></term>
+ <listitem>
+ <para>
+ This directive introduces the basic value set of the field type.
+ The format is an ordered list (without spaces) of the
+ characters which may occur in "words" of the given type.
+ The order of the entries in the list determines the
+ sort order of the index. In addition to single characters, the
+ following combinations are legal:
+ </para>
+
+ <para>
+
+ <itemizedlist>
+ <listitem>
+ <para>
+ Backslashes may be used to introduce three-digit octal, or
+ two-digit hex representations of single characters
+ (preceded by <literal>x</literal>).
+ In addition, the combinations
+ \\, \\r, \\n, \\t, \\s (space — remember that real
+ space-characters may ot occur in the value definition), and
+ \\ are recognised, with their usual interpretation.
+ </para>
+ </listitem>
+
+ <listitem>
+ <para>
+ Curly braces {} may be used to enclose ranges of single
+ characters (possibly using the escape convention described in the
+ preceding point), eg. {a-z} to entroduce the
+ standard range of ASCII characters.
+ Note that the interpretation of such a range depends on
+ the concrete representation in your local, physical character set.
+ </para>
+ </listitem>
+
+ <listitem>
+ <para>
+ paranthesises () may be used to enclose multi-byte characters -
+ eg. diacritics or special national combinations (eg. Spanish
+ "ll"). When found in the input stream (or a search term),
+ these characters are viewed and sorted as a single character, with a
+ sorting value depending on the position of the group in the value
+ statement.
+ </para>
+ </listitem>
+
+ </itemizedlist>
+
+ </para>
+ </listitem></varlistentry>
+ <varlistentry>
+ <term>uppercase <emphasis>value-set</emphasis></term>
+ <listitem>
+ <para>
+ This directive introduces the
+ upper-case equivalencis to the value set (if any). The number and
+ order of the entries in the list should be the same as in the
+ <literal>lowercase</literal> directive.
+ </para>
+ </listitem></varlistentry>
+ <varlistentry>
+ <term>space <emphasis>value-set</emphasis></term>
+ <listitem>
+ <para>
+ This directive introduces the character
+ which separate words in the input stream. Depending on the
+ completeness mode of the field in question, these characters either
+ terminate an index entry, or delimit individual "words" in
+ the input stream. The order of the elements is not significant —
+ otherwise the representation is the same as for the
+ <literal>uppercase</literal> and <literal>lowercase</literal>
+ directives.
+ </para>
+ </listitem></varlistentry>
+ <varlistentry>
+ <term>map <emphasis>value-set</emphasis>
+ <emphasis>target</emphasis></term>
+ <listitem>
+ <para>
+ This directive introduces a
+ mapping between each of the members of the value-set on the left to
+ the character on the right. The character on the right must occur in
+ the value set (the <literal>lowercase</literal> directive) of
+ the character set, but
+ it may be a paranthesis-enclosed multi-octet character. This directive
+ may be used to map diacritics to their base characters, or to map
+ HTML-style character-representations to their natural form, etc.
+ </para>
+ </listitem></varlistentry>
+ </variablelist>
+ </para>
+
+ </sect2>
+
+ </sect1>
+
+ <sect1 id="formats">
+ <title>Exchange Formats</title>
+
+ <para>
+ Converting records from the internal structure to en exchange format
+ is largely an automatic process. Currently, the following exchange
+ formats are supported:
+ </para>
+
+ <para>
+ <itemizedlist>
+ <listitem>
+ <para>
+ GRS-1. The internal representation is based on GRS-1, so the
+ conversion here is straightforward. The system will create
+ applied variant and supported variant lists as required, if a record
+ contains variant information.
+ </para>
+ </listitem>
+
+ <listitem>
+ <para>
+ SUTRS. Again, the mapping is fairly straighforward. Indentation
+ is used to show the hierarchical structure of the record. All
+ "GRS" type records support both the GRS-1 and SUTRS
+ representations.
+ </para>
+ </listitem>
+
+ <listitem>
+ <para>
+ ISO2709-based formats (USMARC, etc.). Only records with a
+ two-level structure (corresponding to fields and subfields) can be
+ directly mapped to ISO2709. For records with a different structuring
+ (eg., GILS), the representation in a structure like USMARC involves a
+ schema-mapping (see section <xref linkend="schema-mapping"/>), to an
+ "implied" USMARC schema (implied,
+ because there is no formal schema which specifies the use of the
+ USMARC fields outside of ISO2709). The resultant, two-level record is
+ then mapped directly from the internal representation to ISO2709. See
+ the GILS schema definition files for a detailed example of this
+ approach.
+ </para>
+ </listitem>
+
+ <listitem>
+ <para>
+ Explain. This representation is only available for records
+ belonging to the Explain schema.
+ </para>
+ </listitem>
+
+ <listitem>
+ <para>
+ Summary. This ASN-1 based structure is only available for records
+ belonging to the Summary schema - or schema which provide a mapping
+ to this schema (see the description of the schema mapping facility
+ above).
+ </para>
+ </listitem>
+
+ <listitem>
+ <para>
+ SOIF. Support for this syntax is experimental, and is currently
+ keyed to a private Index Data OID (1.2.840.10003.5.1000.81.2). All
+ abstract syntaxes can be mapped to the SOIF format, although nested
+ elements are represented by concatenation of the tag names at each
+ level.
+ </para>
+ </listitem>
+
+ </itemizedlist>
+
+ </para>
+
+ </sect1>
</chapter>
+ <!-- Keep this comment at the end of the file
+ Local variables:
+ mode: sgml
+ sgml-omittag:t
+ sgml-shorttag:t
+ sgml-minimize-attributes:nil
+ sgml-always-quote-attributes:t
+ sgml-indent-step:1
+ sgml-indent-data:t
+ sgml-parent-document: "zebra.xml"
+ sgml-local-catalogs: nil
+ sgml-namecase-general:t
+ End:
+ -->