1 <chapter id="administration">
2 <!-- $Id: administration.xml,v 1.54 2007-12-19 09:30:29 adam Exp $ -->
3 <title>Administrating &zebra;</title>
4 <!-- ### It's a bit daft that this chapter (which describes half of
5 the configuration-file formats) is separated from
6 "recordmodel-grs.xml" (which describes the other half) by the
7 instructions on running zebraidx and zebrasrv. Some careful
8 re-ordering is required here.
12 Unlike many simpler retrieval systems, &zebra; supports safe, incremental
13 updates to an existing index.
17 Normally, when &zebra; modifies the index it reads a number of records
19 Depending on your specifications and on the contents of each record
20 one the following events take place for each record:
27 The record is indexed as if it never occurred before.
28 Either the &zebra; system doesn't know how to identify the record or
29 &zebra; can identify the record but didn't find it to be already indexed.
37 The record has already been indexed.
38 In this case either the contents of the record or the location
39 (file) of the record indicates that it has been indexed before.
47 The record is deleted from the index. As in the
48 update-case it must be able to identify the record.
56 Please note that in both the modify- and delete- case the &zebra;
57 indexer must be able to generate a unique key that identifies the record
58 in question (more on this below).
62 To administrate the &zebra; retrieval system, you run the
63 <literal>zebraidx</literal> program.
64 This program supports a number of options which are preceded by a dash,
65 and a few commands (not preceded by dash).
69 Both the &zebra; administrative tool and the &acro.z3950; server share a
70 set of index files and a global configuration file.
71 The name of the configuration file defaults to
72 <literal>zebra.cfg</literal>.
73 The configuration file includes specifications on how to index
74 various kinds of records and where the other configuration files
75 are located. <literal>zebrasrv</literal> and <literal>zebraidx</literal>
76 <emphasis>must</emphasis> be run in the directory where the
77 configuration file lives unless you indicate the location of the
78 configuration file by option <literal>-c</literal>.
81 <sect1 id="record-types">
82 <title>Record Types</title>
85 Indexing is a per-record process, in which either insert/modify/delete
86 will occur. Before a record is indexed search keys are extracted from
87 whatever might be the layout the original record (sgml,html,text, etc..).
88 The &zebra; system currently supports two fundamental types of records:
89 structured and simple text.
90 To specify a particular extraction process, use either the
91 command line option <literal>-t</literal> or specify a
92 <literal>recordType</literal> setting in the configuration file.
97 <sect1 id="zebra-cfg">
98 <title>The &zebra; Configuration File</title>
101 The &zebra; configuration file, read by <literal>zebraidx</literal> and
102 <literal>zebrasrv</literal> defaults to <literal>zebra.cfg</literal>
103 unless specified by <literal>-c</literal> option.
107 You can edit the configuration file with a normal text editor.
108 parameter names and values are separated by colons in the file. Lines
109 starting with a hash sign (<literal>#</literal>) are
114 If you manage different sets of records that share common
115 characteristics, you can organize the configuration settings for each
117 When <literal>zebraidx</literal> is run and you wish to address a
118 given group you specify the group name with the <literal>-g</literal>
120 In this case settings that have the group name as their prefix
121 will be used by <literal>zebraidx</literal>.
122 If no <literal>-g</literal> option is specified, the settings
123 without prefix are used.
127 In the configuration file, the group name is placed before the option
128 name itself, separated by a dot (.). For instance, to set the record type
129 for group <literal>public</literal> to <literal>grs.sgml</literal>
130 (the &acro.sgml;-like format for structured records) you would write:
135 public.recordType: grs.sgml
140 To set the default value of the record type to <literal>text</literal>
151 The available configuration settings are summarized below. They will be
152 explained further in the following sections.
156 FIXME - Didn't Adam make something to have multiple databases in multiple dirs...
164 <emphasis>group</emphasis>
165 .recordType[<emphasis>.name</emphasis>]:
166 <replaceable>type</replaceable>
170 Specifies how records with the file extension
171 <emphasis>name</emphasis> should be handled by the indexer.
172 This option may also be specified as a command line option
173 (<literal>-t</literal>). Note that if you do not specify a
174 <emphasis>name</emphasis>, the setting applies to all files.
175 In general, the record type specifier consists of the elements (each
176 element separated by dot), <emphasis>fundamental-type</emphasis>,
177 <emphasis>file-read-type</emphasis> and arguments. Currently, two
178 fundamental types exist, <literal>text</literal> and
179 <literal>grs</literal>.
184 <term><emphasis>group</emphasis>.recordId:
185 <replaceable>record-id-spec</replaceable></term>
188 Specifies how the records are to be identified when updated. See
189 <xref linkend="locating-records"/>.
194 <term><emphasis>group</emphasis>.database:
195 <replaceable>database</replaceable></term>
198 Specifies the &acro.z3950; database name.
199 <!-- FIXME - now we can have multiple databases in one server. -H -->
204 <term><emphasis>group</emphasis>.storeKeys:
205 <replaceable>boolean</replaceable></term>
208 Specifies whether key information should be saved for a given
209 group of records. If you plan to update/delete this type of
210 records later this should be specified as 1; otherwise it
211 should be 0 (default), to save register space.
212 <!-- ### this is the first mention of "register" -->
213 See <xref linkend="file-ids"/>.
218 <term><emphasis>group</emphasis>.storeData:
219 <replaceable>boolean</replaceable></term>
222 Specifies whether the records should be stored internally
223 in the &zebra; system files.
224 If you want to maintain the raw records yourself,
225 this option should be false (0).
226 If you want &zebra; to take care of the records for you, it
232 <!-- ### probably a better place to define "register" -->
233 <term>register: <replaceable>register-location</replaceable></term>
236 Specifies the location of the various register files that &zebra; uses
237 to represent your databases.
238 See <xref linkend="register-location"/>.
243 <term>shadow: <replaceable>register-location</replaceable></term>
246 Enables the <emphasis>safe update</emphasis> facility of &zebra;, and
247 tells the system where to place the required, temporary files.
248 See <xref linkend="shadow-registers"/>.
253 <term>lockDir: <replaceable>directory</replaceable></term>
256 Directory in which various lock files are stored.
261 <term>keyTmpDir: <replaceable>directory</replaceable></term>
264 Directory in which temporary files used during zebraidx's update
270 <term>setTmpDir: <replaceable>directory</replaceable></term>
273 Specifies the directory that the server uses for temporary result sets.
274 If not specified <literal>/tmp</literal> will be used.
279 <term>profilePath: <replaceable>path</replaceable></term>
282 Specifies a path of profile specification files.
283 The path is composed of one or more directories separated by
284 colon. Similar to <literal>PATH</literal> for UNIX systems.
290 <term>modulePath: <replaceable>path</replaceable></term>
293 Specifies a path of record filter modules.
294 The path is composed of one or more directories separated by
295 colon. Similar to <literal>PATH</literal> for UNIX systems.
296 The 'make install' procedure typically puts modules in
297 <filename>/usr/local/lib/idzebra-2.0/modules</filename>.
303 <term>index: <replaceable>filename</replaceable></term>
306 Defines the filename which holds fields structure
307 definitions. If omitted, the file <filename>default.idx</filename>
309 Refer to <xref linkend="default-idx-file"/> for
316 <term>staticrank: <replaceable>integer</replaceable></term>
319 Enables whether static ranking is to be enabled (1) or
320 disabled (0). If omitted, it is disabled - corresponding
322 Refer to <xref linkend="administration-ranking-static"/> .
329 <term>estimatehits:: <replaceable>integer</replaceable></term>
332 Controls whether &zebra; should calculate approximite hit counts and
333 at which hit count it is to be enabled.
334 A value of 0 disables approximiate hit counts.
335 For a positive value approximaite hit count is enabled
336 if it is known to be larger than <replaceable>integer</replaceable>.
339 Approximate hit counts can also be triggered by a particular
340 attribute in a query.
341 Refer to <xref linkend="querymodel-zebra-global-attr-limit"/>.
347 <term>attset: <replaceable>filename</replaceable></term>
350 Specifies the filename(s) of attribute set files for use in
351 searching. In many configurations <filename>bib1.att</filename>
352 is used, but that is not required. If Classic Explain
353 attributes is to be used for searching,
354 <filename>explain.att</filename> must be given.
355 The path to att-files in general can be given using
356 <literal>profilePath</literal> setting.
357 See also <xref linkend="attset-files"/>.
362 <term>memMax: <replaceable>size</replaceable></term>
365 Specifies <replaceable>size</replaceable> of internal memory
366 to use for the zebraidx program.
367 The amount is given in megabytes - default is 4 (4 MB).
368 The more memory, the faster large updates happen, up to about
369 half the free memory available on the computer.
374 <term>tempfiles: <replaceable>Yes/Auto/No</replaceable></term>
377 Tells zebra if it should use temporary files when indexing. The
378 default is Auto, in which case zebra uses temporary files only
379 if it would need more that <replaceable>memMax</replaceable>
380 megabytes of memory. This should be good for most uses.
386 <term>root: <replaceable>dir</replaceable></term>
389 Specifies a directory base for &zebra;. All relative paths
390 given (in profilePath, register, shadow) are based on this
391 directory. This setting is useful if your &zebra; server
392 is running in a different directory from where
393 <literal>zebra.cfg</literal> is located.
399 <term>passwd: <replaceable>file</replaceable></term>
402 Specifies a file with description of user accounts for &zebra;.
403 The format is similar to that known to Apache's htpasswd files
404 and UNIX' passwd files. Non-empty lines not beginning with
405 # are considered account lines. There is one account per-line.
406 A line consists of fields separate by a single colon character.
407 First field is username, second is password.
413 <term>passwd.c: <replaceable>file</replaceable></term>
416 Specifies a file with description of user accounts for &zebra;.
417 File format is similar to that used by the passwd directive except
418 that the password are encrypted. Use Apache's htpasswd or similar
425 <term>perm.<replaceable>user</replaceable>:
426 <replaceable>permstring</replaceable></term>
429 Specifies permissions (priviledge) for a user that are allowed
430 to access &zebra; via the passwd system. There are two kinds
431 of permissions currently: read (r) and write(w). By default
432 users not listed in a permission directive are given the read
433 privilege. To specify permissions for a user with no
434 username, or &acro.z3950; anonymous style use
435 <literal>anonymous</literal>. The permstring consists of
436 a sequence of characters. Include character <literal>w</literal>
437 for write/update access, <literal>r</literal> for read access and
438 <literal>a</literal> to allow anonymous access through this account.
444 <term>dbaccess <replaceable>accessfile</replaceable></term>
447 Names a file which lists database subscriptions for individual users.
448 The access file should consists of lines of the form <literal>username:
449 dbnames</literal>, where dbnames is a list of database names, seprated by
450 '+'. No whitespace is allowed in the database list.
460 <sect1 id="locating-records">
461 <title>Locating Records</title>
464 The default behavior of the &zebra; system is to reference the
465 records from their original location, i.e. where they were found when you
466 run <literal>zebraidx</literal>.
467 That is, when a client wishes to retrieve a record
468 following a search operation, the files are accessed from the place
469 where you originally put them - if you remove the files (without
470 running <literal>zebraidx</literal> again, the server will return
471 diagnostic number 14 (``System error in presenting records'') to
476 If your input files are not permanent - for example if you retrieve
477 your records from an outside source, or if they were temporarily
478 mounted on a CD-ROM drive,
479 you may want &zebra; to make an internal copy of them. To do this,
480 you specify 1 (true) in the <literal>storeData</literal> setting. When
481 the &acro.z3950; server retrieves the records they will be read from the
482 internal file structures of the system.
487 <sect1 id="simple-indexing">
488 <title>Indexing with no Record IDs (Simple Indexing)</title>
491 If you have a set of records that are not expected to change over time
492 you may can build your database without record IDs.
493 This indexing method uses less space than the other methods and
498 To use this method, you simply omit the <literal>recordId</literal> entry
499 for the group of files that you index. To add a set of records you use
500 <literal>zebraidx</literal> with the <literal>update</literal> command. The
501 <literal>update</literal> command will always add all of the records that it
502 encounters to the index - whether they have already been indexed or
503 not. If the set of indexed files change, you should delete all of the
504 index files, and build a new index from scratch.
508 Consider a system in which you have a group of text files called
509 <literal>simple</literal>.
510 That group of records should belong to a &acro.z3950; database called
511 <literal>textbase</literal>.
512 The following <literal>zebra.cfg</literal> file will suffice:
517 profilePath: /usr/local/idzebra/tab
519 simple.recordType: text
520 simple.database: textbase
526 Since the existing records in an index can not be addressed by their
527 IDs, it is impossible to delete or modify records when using this method.
532 <sect1 id="file-ids">
533 <title>Indexing with File Record IDs</title>
536 If you have a set of files that regularly change over time: Old files
537 are deleted, new ones are added, or existing files are modified, you
538 can benefit from using the <emphasis>file ID</emphasis>
539 indexing methodology.
540 Examples of this type of database might include an index of WWW
541 resources, or a USENET news spool area.
542 Briefly speaking, the file key methodology uses the directory paths
543 of the individual records as a unique identifier for each record.
544 To perform indexing of a directory with file keys, again, you specify
545 the top-level directory after the <literal>update</literal> command.
546 The command will recursively traverse the directories and compare
547 each one with whatever have been indexed before in that same directory.
548 If a file is new (not in the previous version of the directory) it
549 is inserted into the registers; if a file was already indexed and
550 it has been modified since the last update, the index is also
551 modified; if a file has been removed since the last
552 visit, it is deleted from the index.
556 The resulting system is easy to administrate. To delete a record you
557 simply have to delete the corresponding file (say, with the
558 <literal>rm</literal> command). And to add records you create new
559 files (or directories with files). For your changes to take effect
560 in the register you must run <literal>zebraidx update</literal> with
561 the same directory root again. This mode of operation requires more
562 disk space than simpler indexing methods, but it makes it easier for
563 you to keep the index in sync with a frequently changing set of data.
564 If you combine this system with the <emphasis>safe update</emphasis>
565 facility (see below), you never have to take your server off-line for
566 maintenance or register updating purposes.
570 To enable indexing with pathname IDs, you must specify
571 <literal>file</literal> as the value of <literal>recordId</literal>
572 in the configuration file. In addition, you should set
573 <literal>storeKeys</literal> to <literal>1</literal>, since the &zebra;
574 indexer must save additional information about the contents of each record
575 in order to modify the indexes correctly at a later time.
579 FIXME - There must be a simpler way to do this with Adams string tags -H
583 For example, to update records of group <literal>esdd</literal>
585 <literal>/data1/records/</literal> you should type:
587 $ zebraidx -g esdd update /data1/records
592 The corresponding configuration file includes:
595 esdd.recordType: grs.sgml
601 <para>You cannot start out with a group of records with simple
602 indexing (no record IDs as in the previous section) and then later
603 enable file record Ids. &zebra; must know from the first time that you
605 the files should be indexed with file record IDs.
610 You cannot explicitly delete records when using this method (using the
611 <literal>delete</literal> command to <literal>zebraidx</literal>. Instead
612 you have to delete the files from the file system (or move them to a
614 and then run <literal>zebraidx</literal> with the
615 <literal>update</literal> command.
617 <!-- ### what happens if a file contains multiple records? -->
620 <sect1 id="generic-ids">
621 <title>Indexing with General Record IDs</title>
624 When using this method you construct an (almost) arbitrary, internal
625 record key based on the contents of the record itself and other system
626 information. If you have a group of records that explicitly associates
627 an ID with each record, this method is convenient. For example, the
628 record format may contain a title or a ID-number - unique within the group.
629 In either case you specify the &acro.z3950; attribute set and use-attribute
630 location in which this information is stored, and the system looks at
631 that field to determine the identity of the record.
635 As before, the record ID is defined by the <literal>recordId</literal>
636 setting in the configuration file. The value of the record ID specification
637 consists of one or more tokens separated by whitespace. The resulting
638 ID is represented in the index by concatenating the tokens and
639 separating them by ASCII value (1).
643 There are three kinds of tokens:
647 <term>Internal record info</term>
650 The token refers to a key that is
651 extracted from the record. The syntax of this token is
652 <literal>(</literal> <emphasis>set</emphasis> <literal>,</literal>
653 <emphasis>use</emphasis> <literal>)</literal>,
654 where <emphasis>set</emphasis> is the
655 attribute set name <emphasis>use</emphasis> is the
656 name or value of the attribute.
661 <term>System variable</term>
664 The system variables are preceded by
669 and immediately followed by the system variable name, which
682 <term>database</term>
685 Current database specified.
702 <term>Constant string</term>
705 A string used as part of the ID — surrounded
706 by single- or double quotes.
714 For instance, the sample GILS records that come with the &zebra;
715 distribution contain a unique ID in the data tagged Control-Identifier.
716 The data is mapped to the &acro.bib1; use attribute Identifier-standard
717 (code 1007). To use this field as a record id, specify
718 <literal>(bib1,Identifier-standard)</literal> as the value of the
719 <literal>recordId</literal> in the configuration file.
720 If you have other record types that uses the same field for a
721 different purpose, you might add the record type
722 (or group or database name) to the record id of the gils
723 records as well, to prevent matches with other types of records.
724 In this case the recordId might be set like this:
727 gils.recordId: $type (bib1,Identifier-standard)
733 (see <xref linkend="grs"/>
734 for details of how the mapping between elements of your records and
735 searchable attributes is established).
739 As for the file record ID case described in the previous section,
740 updating your system is simply a matter of running
741 <literal>zebraidx</literal>
742 with the <literal>update</literal> command. However, the update with general
743 keys is considerably slower than with file record IDs, since all files
744 visited must be (re)read to discover their IDs.
748 As you might expect, when using the general record IDs
749 method, you can only add or modify existing records with the
750 <literal>update</literal> command.
751 If you wish to delete records, you must use the,
752 <literal>delete</literal> command, with a directory as a parameter.
753 This will remove all records that match the files below that root
759 <sect1 id="register-location">
760 <title>Register Location</title>
763 Normally, the index files that form dictionaries, inverted
764 files, record info, etc., are stored in the directory where you run
765 <literal>zebraidx</literal>. If you wish to store these, possibly large,
766 files somewhere else, you must add the <literal>register</literal>
767 entry to the <literal>zebra.cfg</literal> file.
768 Furthermore, the &zebra; system allows its file
769 structures to span multiple file systems, which is useful for
770 managing very large databases.
774 The value of the <literal>register</literal> setting is a sequence
775 of tokens. Each token takes the form:
778 <emphasis>dir</emphasis><literal>:</literal><emphasis>size</emphasis>
781 The <emphasis>dir</emphasis> specifies a directory in which index files
782 will be stored and the <emphasis>size</emphasis> specifies the maximum
783 size of all files in that directory. The &zebra; indexer system fills
784 each directory in the order specified and use the next specified
785 directories as needed.
786 The <emphasis>size</emphasis> is an integer followed by a qualifier
788 <literal>b</literal> for bytes,
789 <literal>k</literal> for kilobytes.
790 <literal>M</literal> for megabytes,
791 <literal>G</literal> for gigabytes.
792 Specifying a negative value disables the checking (it still needs the unit,
793 use <literal>-1b</literal>).
797 For instance, if you have allocated three disks for your register, and
798 the first disk is mounted
799 on <literal>/d1</literal> and has 2GB of free space, the
800 second, mounted on <literal>/d2</literal> has 3.6 GB, and the third,
801 on which you have more space than you bother to worry about, mounted on
802 <literal>/d3</literal> you could put this entry in your configuration file:
805 register: /d1:2G /d2:3600M /d3:-1b
810 Note that &zebra; does not verify that the amount of space specified is
811 actually available on the directory (file system) specified - it is
812 your responsibility to ensure that enough space is available, and that
813 other applications do not attempt to use the free space. In a large
814 production system, it is recommended that you allocate one or more
815 file system exclusively to the &zebra; register files.
820 <sect1 id="shadow-registers">
821 <title>Safe Updating - Using Shadow Registers</title>
823 <sect2 id="shadow-registers-description">
824 <title>Description</title>
827 The &zebra; server supports <emphasis>updating</emphasis> of the index
828 structures. That is, you can add, modify, or remove records from
829 databases managed by &zebra; without rebuilding the entire index.
830 Since this process involves modifying structured files with various
831 references between blocks of data in the files, the update process
832 is inherently sensitive to system crashes, or to process interruptions:
833 Anything but a successfully completed update process will leave the
834 register files in an unknown state, and you will essentially have no
835 recourse but to re-index everything, or to restore the register files
836 from a backup medium.
837 Further, while the update process is active, users cannot be
838 allowed to access the system, as the contents of the register files
839 may change unpredictably.
843 You can solve these problems by enabling the shadow register system in
845 During the updating procedure, <literal>zebraidx</literal> will temporarily
846 write changes to the involved files in a set of "shadow
847 files", without modifying the files that are accessed by the
848 active server processes. If the update procedure is interrupted by a
849 system crash or a signal, you simply repeat the procedure - the
850 register files have not been changed or damaged, and the partially
851 written shadow files are automatically deleted before the new updating
856 At the end of the updating procedure (or in a separate operation, if
857 you so desire), the system enters a "commit mode". First,
858 any active server processes are forced to access those blocks that
859 have been changed from the shadow files rather than from the main
860 register files; the unmodified blocks are still accessed at their
861 normal location (the shadow files are not a complete copy of the
862 register files - they only contain those parts that have actually been
863 modified). If the commit process is interrupted at any point during the
864 commit process, the server processes will continue to access the
865 shadow files until you can repeat the commit procedure and complete
866 the writing of data to the main register files. You can perform
867 multiple update operations to the registers before you commit the
868 changes to the system files, or you can execute the commit operation
869 at the end of each update operation. When the commit phase has
870 completed successfully, any running server processes are instructed to
871 switch their operations to the new, operational register, and the
872 temporary shadow files are deleted.
877 <sect2 id="shadow-registers-how-to-use">
878 <title>How to Use Shadow Register Files</title>
881 The first step is to allocate space on your system for the shadow
883 You do this by adding a <literal>shadow</literal> entry to the
884 <literal>zebra.cfg</literal> file.
885 The syntax of the <literal>shadow</literal> entry is exactly the
886 same as for the <literal>register</literal> entry
887 (see <xref linkend="register-location"/>).
888 The location of the shadow area should be
889 <emphasis>different</emphasis> from the location of the main register
890 area (if you have specified one - remember that if you provide no
891 <literal>register</literal> setting, the default register area is the
892 working directory of the server and indexing processes).
896 The following excerpt from a <literal>zebra.cfg</literal> file shows
897 one example of a setup that configures both the main register
898 location and the shadow file area.
899 Note that two directories or partitions have been set aside
900 for the shadow file area. You can specify any number of directories
901 for each of the file areas, but remember that there should be no
902 overlaps between the directories used for the main registers and the
903 shadow files, respectively.
909 shadow: /scratch1:100M /scratch2:200M
915 When shadow files are enabled, an extra command is available at the
916 <literal>zebraidx</literal> command line.
917 In order to make changes to the system take effect for the
918 users, you'll have to submit a "commit" command after a
919 (sequence of) update operation(s).
925 $ zebraidx update /d1/records
932 Or you can execute multiple updates before committing the changes:
938 $ zebraidx -g books update /d1/records /d2/more-records
939 $ zebraidx -g fun update /d3/fun-records
946 If one of the update operations above had been interrupted, the commit
947 operation on the last line would fail: <literal>zebraidx</literal>
948 will not let you commit changes that would destroy the running register.
949 You'll have to rerun all of the update operations since your last
950 commit operation, before you can commit the new changes.
954 Similarly, if the commit operation fails, <literal>zebraidx</literal>
955 will not let you start a new update operation before you have
956 successfully repeated the commit operation.
957 The server processes will keep accessing the shadow files rather
958 than the (possibly damaged) blocks of the main register files
959 until the commit operation has successfully completed.
963 You should be aware that update operations may take slightly longer
964 when the shadow register system is enabled, since more file access
965 operations are involved. Further, while the disk space required for
966 the shadow register data is modest for a small update operation, you
967 may prefer to disable the system if you are adding a very large number
968 of records to an already very large database (we use the terms
969 <emphasis>large</emphasis> and <emphasis>modest</emphasis>
970 very loosely here, since every application will have a
971 different perception of size).
972 To update the system without the use of the the shadow files,
973 simply run <literal>zebraidx</literal> with the <literal>-n</literal>
974 option (note that you do not have to execute the
975 <emphasis>commit</emphasis> command of <literal>zebraidx</literal>
976 when you temporarily disable the use of the shadow registers in
978 Note also that, just as when the shadow registers are not enabled,
979 server processes will be barred from accessing the main register
980 while the update procedure takes place.
988 <sect1 id="administration-ranking">
989 <title>Relevance Ranking and Sorting of Result Sets</title>
991 <sect2 id="administration-overview">
992 <title>Overview</title>
994 The default ordering of a result set is left up to the server,
995 which inside &zebra; means sorting in ascending document ID order.
996 This is not always the order humans want to browse the sometimes
997 quite large hit sets. Ranking and sorting comes to the rescue.
1001 In cases where a good presentation ordering can be computed at
1002 indexing time, we can use a fixed <literal>static ranking</literal>
1003 scheme, which is provided for the <literal>alvis</literal>
1004 indexing filter. This defines a fixed ordering of hit lists,
1005 independently of the query issued.
1009 There are cases, however, where relevance of hit set documents is
1010 highly dependent on the query processed.
1011 Simply put, <literal>dynamic relevance ranking</literal>
1012 sorts a set of retrieved records such that those most likely to be
1013 relevant to your request are retrieved first.
1014 Internally, &zebra; retrieves all documents that satisfy your
1015 query, and re-orders the hit list to arrange them based on
1016 a measurement of similarity between your query and the content of
1021 Finally, there are situations where hit sets of documents should be
1022 <literal>sorted</literal> during query time according to the
1023 lexicographical ordering of certain sort indexes created at
1029 <sect2 id="administration-ranking-static">
1030 <title>Static Ranking</title>
1033 &zebra; uses internally inverted indexes to look up term occurencies
1034 in documents. Multiple queries from different indexes can be
1035 combined by the binary boolean operations <literal>AND</literal>,
1036 <literal>OR</literal> and/or <literal>NOT</literal> (which
1037 is in fact a binary <literal>AND NOT</literal> operation).
1038 To ensure fast query execution
1039 speed, all indexes have to be sorted in the same order.
1042 The indexes are normally sorted according to document
1043 <literal>ID</literal> in
1044 ascending order, and any query which does not invoke a special
1045 re-ranking function will therefore retrieve the result set in
1047 <literal>ID</literal>
1055 directive in the main core &zebra; configuration file, the internal document
1056 keys used for ordering are augmented by a preceding integer, which
1057 contains the static rank of a given document, and the index lists
1059 first by ascending static rank,
1060 then by ascending document <literal>ID</literal>.
1062 is the ``best'' rank, as it occurs at the
1063 beginning of the list; higher numbers represent worse scores.
1066 The experimental <literal>alvis</literal> filter provides a
1067 directive to fetch static rank information out of the indexed &acro.xml;
1068 records, thus making <emphasis>all</emphasis> hit sets ordered
1069 after <emphasis>ascending</emphasis> static
1070 rank, and for those doc's which have the same static rank, ordered
1071 after <emphasis>ascending</emphasis> doc <literal>ID</literal>.
1072 See <xref linkend="record-model-alvisxslt"/> for the gory details.
1077 <sect2 id="administration-ranking-dynamic">
1078 <title>Dynamic Ranking</title>
1080 In order to fiddle with the static rank order, it is necessary to
1081 invoke additional re-ranking/re-ordering using dynamic
1082 ranking or score functions. These functions return positive
1083 integer scores, where <emphasis>highest</emphasis> score is
1085 hit sets are sorted according to <emphasis>descending</emphasis>
1087 to the index lists which are sorted according to
1088 ascending rank number and document ID).
1091 Dynamic ranking is enabled by a directive like one of the
1092 following in the zebra configuration file (use only one of these a time!):
1094 rank: rank-1 # default TDF-IDF like
1095 rank: rank-static # dummy do-nothing
1100 Dynamic ranking is done at query time rather than
1101 indexing time (this is why we
1102 call it ``dynamic ranking'' in the first place ...)
1103 It is invoked by adding
1104 the &acro.bib1; relation attribute with
1105 value ``relevance'' to the &acro.pqf; query (that is,
1106 <literal>@attr 2=102</literal>, see also
1107 <ulink url="&url.z39.50;bib1.html">
1108 The &acro.bib1; Attribute Set Semantics</ulink>, also in
1109 <ulink url="&url.z39.50.attset.bib1;">HTML</ulink>).
1110 To find all articles with the word <literal>Eoraptor</literal> in
1111 the title, and present them relevance ranked, issue the &acro.pqf; query:
1113 @attr 2=102 @attr 1=4 Eoraptor
1117 <sect3 id="administration-ranking-dynamic-rank1">
1118 <title>Dynamically ranking using &acro.pqf; queries with the 'rank-1'
1122 The default <literal>rank-1</literal> ranking module implements a
1123 TF/IDF (Term Frequecy over Inverse Document Frequency) like
1124 algorithm. In contrast to the usual defintion of TF/IDF
1125 algorithms, which only considers searching in one full-text
1126 index, this one works on multiple indexes at the same time.
1128 &zebra; does boolean queries and searches in specific addressed
1129 indexes (there are inverted indexes pointing from terms in the
1130 dictionary to documents and term positions inside documents).
1134 <term>Query Components</term>
1137 First, the boolean query is dismantled into its principal components,
1138 i.e. atomic queries where one term is looked up in one index.
1139 For example, the query
1141 @attr 2=102 @and @attr 1=1010 Utah @attr 1=1018 Springer
1143 is a boolean AND between the atomic parts
1145 @attr 2=102 @attr 1=1010 Utah
1149 @attr 2=102 @attr 1=1018 Springer
1151 which gets processed each for itself.
1157 <term>Atomic hit lists</term>
1160 Second, for each atomic query, the hit list of documents is
1164 In this example, two hit lists for each index
1165 <literal>@attr 1=1010</literal> and
1166 <literal>@attr 1=1018</literal> are computed.
1172 <term>Atomic scores</term>
1175 Third, each document in the hit list is assigned a score (_if_ ranking
1176 is enabled and requested in the query) using a TF/IDF scheme.
1179 In this example, both atomic parts of the query assign the magic
1180 <literal>@attr 2=102</literal> relevance attribute, and are
1181 to be used in the relevance ranking functions.
1184 It is possible to apply dynamic ranking on only parts of the
1187 @and @attr 2=102 @attr 1=1010 Utah @attr 1=1018 Springer
1189 searches for all documents which have the term 'Utah' on the
1190 body of text, and which have the term 'Springer' in the publisher
1191 field, and sort them in the order of the relevance ranking made on
1192 the body-of-text index only.
1198 <term>Hit list merging</term>
1201 Fourth, the atomic hit lists are merged according to the boolean
1202 conditions to a final hit list of documents to be returned.
1205 This step is always performed, independently of the fact that
1206 dynamic ranking is enabled or not.
1212 <term>Document score computation</term>
1215 Fifth, the total score of a document is computed as a linear
1216 combination of the atomic scores of the atomic hit lists
1219 Ranking weights may be used to pass a value to a ranking
1220 algorithm, using the non-standard &acro.bib1; attribute type 9.
1221 This allows one branch of a query to use one value while
1222 another branch uses a different one. For example, we can search
1223 for <literal>utah</literal> in the
1224 <literal>@attr 1=4</literal> index with weight 30, as
1225 well as in the <literal>@attr 1=1010</literal> index with weight 20:
1227 @attr 2=102 @or @attr 9=30 @attr 1=4 utah @attr 9=20 @attr 1=1010 city
1231 The default weight is
1232 sqrt(1000) ~ 34 , as the &acro.z3950; standard prescribes that the top score
1233 is 1000 and the bottom score is 0, encoded in integers.
1237 The ranking-weight feature is experimental. It may change in future
1245 <term>Re-sorting of hit list</term>
1248 Finally, the final hit list is re-ordered according to scores.
1256 Still need to describe the exact TF/IDF formula. Here's the info, need -->
1257 <!--to extract it in human readable form .. MC
1259 static int calc (void *set_handle, zint sysno, zint staticrank,
1262 int i, lo, divisor, score = 0;
1263 struct rank_set_info *si = (struct rank_set_info *) set_handle;
1265 if (!si->no_rank_entries)
1266 return -1; /* ranking not enabled for any terms */
1268 for (i = 0; i < si->no_entries; i++)
1270 yaz_log(log_level, "calc: i=%d rank_flag=%d lo=%d",
1271 i, si->entries[i].rank_flag, si->entries[i].local_occur);
1272 if (si->entries[i].rank_flag && (lo = si->entries[i].local_occur))
1273 score += (8+log2_int (lo)) * si->entries[i].global_inv *
1274 si->entries[i].rank_weight;
1276 divisor = si->no_rank_entries * (8+log2_int (si->last_pos/si->no_entries));
1277 score = score / divisor;
1278 yaz_log(log_level, "calc sysno=" ZINT_FORMAT " score=%d", sysno, score);
1281 /* reset the counts for the next term */
1282 for (i = 0; i < si->no_entries; i++)
1283 si->entries[i].local_occur = 0;
1288 where lo = si->entries[i].local_occur is the local documents term-within-index frequency, si->entries[i].global_inv represents the IDF part (computed in static void *begin()), and
1289 si->entries[i].rank_weight is the weight assigner per index (default 34, or set in the @attr 9=xyz magic)
1291 Finally, the IDF part is computed as:
1293 static void *begin (struct zebra_register *reg,
1294 void *class_handle, RSET rset, NMEM nmem,
1295 TERMID *terms, int numterms)
1297 struct rank_set_info *si =
1298 (struct rank_set_info *) nmem_malloc (nmem,sizeof(*si));
1301 yaz_log(log_level, "rank-1 begin");
1302 si->no_entries = numterms;
1303 si->no_rank_entries = 0;
1305 si->entries = (struct rank_term_info *)
1306 nmem_malloc (si->nmem, sizeof(*si->entries)*numterms);
1307 for (i = 0; i < numterms; i++)
1309 zint g = rset_count(terms[i]->rset);
1310 yaz_log(log_level, "i=%d flags=%s '%s'", i,
1311 terms[i]->flags, terms[i]->name );
1312 if (!strncmp (terms[i]->flags, "rank,", 5))
1314 const char *cp = strstr(terms[i]->flags+4, ",w=");
1315 si->entries[i].rank_flag = 1;
1317 si->entries[i].rank_weight = atoi (cp+3);
1319 si->entries[i].rank_weight = 34; /* sqrroot of 1000 */
1320 yaz_log(log_level, " i=%d weight=%d g="ZINT_FORMAT, i,
1321 si->entries[i].rank_weight, g);
1322 (si->no_rank_entries)++;
1325 si->entries[i].rank_flag = 0;
1326 si->entries[i].local_occur = 0; /* FIXME */
1327 si->entries[i].global_occur = g;
1328 si->entries[i].global_inv = 32 - log2_int (g);
1329 yaz_log(log_level, " global_inv = %d g = " ZINT_FORMAT,
1330 (int) (32-log2_int (g)), g);
1331 si->entries[i].term = terms[i];
1332 si->entries[i].term_index=i;
1333 terms[i]->rankpriv = &(si->entries[i]);
1339 where g = rset_count(terms[i]->rset) is the count of all documents in this specific index hit list, and the IDF part then is
1341 si->entries[i].global_inv = 32 - log2_int (g);
1348 The <literal>rank-1</literal> algorithm
1349 does not use the static rank
1350 information in the list keys, and will produce the same ordering
1351 with or without static ranking enabled.
1356 <sect3 id="administration-ranking-dynamic-rank1">
1357 <title>Dynamically ranking &acro.pqf; queries with the 'rank-static'
1360 The dummy <literal>rank-static</literal> reranking/scoring
1361 function returns just
1362 <literal>score = max int - staticrank</literal>
1363 in order to preserve the static ordering of hit sets that would
1364 have been produced had it not been invoked.
1365 Obviously, to combine static and dynamic ranking usefully,
1367 to make a new ranking
1368 function; this is left
1369 as an exercise for the reader.
1376 <literal>Dynamic ranking</literal> is not compatible
1377 with <literal>estimated hit sizes</literal>, as all documents in
1378 a hit set must be accessed to compute the correct placing in a
1379 ranking sorted list. Therefore the use attribute setting
1380 <literal>@attr 2=102</literal> clashes with
1381 <literal>@attr 9=integer</literal>.
1386 we might want to add ranking like this:
1388 Simple BM25 Extension to Multiple Weighted Fields
1389 Stephen Robertson, Hugo Zaragoza and Michael Taylor
1393 mitaylor2microsoft.com
1398 <sect3 id="administration-ranking-dynamic-cql">
1399 <title>Dynamically ranking &acro.cql; queries</title>
1401 Dynamic ranking can be enabled during sever side &acro.cql;
1402 query expansion by adding <literal>@attr 2=102</literal>
1403 chunks to the &acro.cql; config file. For example
1405 relationModifier.relevant = 2=102
1407 invokes dynamic ranking each time a &acro.cql; query of the form
1410 Z> f alvis.text =/relevant house
1412 is issued. Dynamic ranking can also be automatically used on
1413 specific &acro.cql; indexes by (for example) setting
1415 index.alvis.text = 1=text 2=102
1417 which then invokes dynamic ranking each time a &acro.cql; query of the form
1420 Z> f alvis.text = house
1430 <sect2 id="administration-ranking-sorting">
1431 <title>Sorting</title>
1433 &zebra; sorts efficiently using special sorting indexes
1434 (type=<literal>s</literal>; so each sortable index must be known
1435 at indexing time, specified in the configuration of record
1436 indexing. For example, to enable sorting according to the &acro.bib1;
1437 <literal>Date/time-added-to-db</literal> field, one could add the line
1439 xelm /*/@created Date/time-added-to-db:s
1441 to any <literal>.abs</literal> record-indexing configuration file.
1442 Similarly, one could add an indexing element of the form
1444 <z:index name="date-modified" type="s">
1445 <xsl:value-of select="some/xpath"/>
1448 to any <literal>alvis</literal>-filter indexing stylesheet.
1451 Indexing can be specified at searching time using a query term
1452 carrying the non-standard
1453 &acro.bib1; attribute-type <literal>7</literal>. This removes the
1454 need to send a &acro.z3950; <literal>Sort Request</literal>
1455 separately, and can dramatically improve latency when the client
1456 and server are on separate networks.
1457 The sorting part of the query is separate from the rest of the
1458 query - the actual search specification - and must be combined
1462 A sorting subquery needs two attributes: an index (such as a
1463 &acro.bib1; type-1 attribute) specifying which index to sort on, and a
1464 type-7 attribute whose value is be <literal>1</literal> for
1465 ascending sorting, or <literal>2</literal> for descending. The
1466 term associated with the sorting attribute is the priority of
1467 the sort key, where <literal>0</literal> specifies the primary
1468 sort key, <literal>1</literal> the secondary sort key, and so
1471 <para>For example, a search for water, sort by title (ascending),
1472 is expressed by the &acro.pqf; query
1474 @or @attr 1=1016 water @attr 7=1 @attr 1=4 0
1476 whereas a search for water, sort by title ascending,
1477 then date descending would be
1479 @or @or @attr 1=1016 water @attr 7=1 @attr 1=4 0 @attr 7=2 @attr 1=30 1
1483 Notice the fundamental differences between <literal>dynamic
1484 ranking</literal> and <literal>sorting</literal>: there can be
1485 only one ranking function defined and configured; but multiple
1486 sorting indexes can be specified dynamically at search
1487 time. Ranking does not need to use specific indexes, so
1488 dynamic ranking can be enabled and disabled without
1489 re-indexing; whereas, sorting indexes need to be
1490 defined before indexing.
1498 <sect1 id="administration-extended-services">
1499 <title>Extended Services: Remote Insert, Update and Delete</title>
1503 Extended services are only supported when accessing the &zebra;
1504 server using the <ulink url="&url.z39.50;">&acro.z3950;</ulink>
1505 protocol. The <ulink url="&url.sru;">&acro.sru;</ulink> protocol does
1506 not support extended services.
1511 The extended services are not enabled by default in zebra - due to the
1512 fact that they modify the system. &zebra; can be configured
1514 search, and to allow only updates for a particular admin user
1515 in the main zebra configuration file <filename>zebra.cfg</filename>.
1516 For user <literal>admin</literal>, you could use:
1520 passwd: passwordfile
1522 And in the password file
1523 <filename>passwordfile</filename>, you have to specify users and
1524 encrypted passwords as colon separated strings.
1525 Use a tool like <filename>htpasswd</filename>
1526 to maintain the encrypted passwords.
1530 It is essential to configure &zebra; to store records internally,
1532 modifications and deletion of records:
1537 The general record type should be set to any record filter which
1538 is able to parse &acro.xml; records, you may use any of the two
1539 declarations (but not both simultaneously!)
1542 # recordType: alvis.filter_alvis_config.xml
1544 To enable transaction safe shadow indexing,
1545 which is extra important for this kind of operation, set
1547 shadow: directoryname: size (e.g. 1000M)
1549 See <xref linkend="zebra-cfg"/> for additional information on
1550 these configuration options.
1554 It is not possible to carry information about record types or
1555 similar to &zebra; when using extended services, due to
1556 limitations of the <ulink url="&url.z39.50;">&acro.z3950;</ulink>
1557 protocol. Therefore, indexing filters can not be chosen on a
1558 per-record basis. One and only one general &acro.xml; indexing filter
1560 <!-- but because it is represented as an OID, we would need some
1561 form of proprietary mapping scheme between record type strings and
1564 However, as a minimum, it would be extremely useful to enable
1565 people to use &acro.marc21;, assuming grs.marcxml.marc21 as a record
1572 <sect2 id="administration-extended-services-z3950">
1573 <title>Extended services in the &acro.z3950; protocol</title>
1576 The <ulink url="&url.z39.50;">&acro.z3950;</ulink> standard allows
1577 servers to accept special binary <emphasis>extended services</emphasis>
1578 protocol packages, which may be used to insert, update and delete
1579 records into servers. These carry control and update
1580 information to the servers, which are encoded in seven package fields:
1583 <table id="administration-extended-services-z3950-table" frame="top">
1584 <title>Extended services &acro.z3950; Package Fields</title>
1588 <entry>Parameter</entry>
1589 <entry>Value</entry>
1590 <entry>Notes</entry>
1595 <entry><literal>type</literal></entry>
1596 <entry><literal>'update'</literal></entry>
1597 <entry>Must be set to trigger extended services</entry>
1600 <entry><literal>action</literal></entry>
1601 <entry><literal>string</literal></entry>
1603 Extended service action type with
1604 one of four possible values: <literal>recordInsert</literal>,
1605 <literal>recordReplace</literal>,
1606 <literal>recordDelete</literal>,
1607 and <literal>specialUpdate</literal>
1611 <entry><literal>record</literal></entry>
1612 <entry><literal>&acro.xml; string</literal></entry>
1613 <entry>An &acro.xml; formatted string containing the record</entry>
1616 <entry><literal>syntax</literal></entry>
1617 <entry><literal>'xml'</literal></entry>
1618 <entry>XML/SUTRS/MARC. GRS-1 not supported.
1619 The default filter (record type) as given by recordType in
1620 zebra.cfg is used to parse the record.</entry>
1623 <entry><literal>recordIdOpaque</literal></entry>
1624 <entry><literal>string</literal></entry>
1626 Optional client-supplied, opaque record
1627 identifier used under insert operations.
1631 <entry><literal>recordIdNumber </literal></entry>
1632 <entry><literal>positive number</literal></entry>
1633 <entry>&zebra;'s internal system number,
1634 not allowed for <literal>recordInsert</literal> or
1635 <literal>specialUpdate</literal> actions which result in fresh
1640 <entry><literal>databaseName</literal></entry>
1641 <entry><literal>database identifier</literal></entry>
1643 The name of the database to which the extended services should be
1653 The <literal>action</literal> parameter can be any of
1654 <literal>recordInsert</literal> (will fail if the record already exists),
1655 <literal>recordReplace</literal> (will fail if the record does not exist),
1656 <literal>recordDelete</literal> (will fail if the record does not
1658 <literal>specialUpdate</literal> (will insert or update the record
1659 as needed, record deletion is not possible).
1663 During all actions, the
1664 usual rules for internal record ID generation apply, unless an
1665 optional <literal>recordIdNumber</literal> &zebra; internal ID or a
1666 <literal>recordIdOpaque</literal> string identifier is assigned.
1667 The default ID generation is
1668 configured using the <literal>recordId:</literal> from
1669 <filename>zebra.cfg</filename>.
1670 See <xref linkend="zebra-cfg"/>.
1674 Setting of the <literal>recordIdNumber</literal> parameter,
1675 which must be an existing &zebra; internal system ID number, is not
1676 allowed during any <literal>recordInsert</literal> or
1677 <literal>specialUpdate</literal> action resulting in fresh record
1682 When retrieving existing
1683 records indexed with &acro.grs1; indexing filters, the &zebra; internal
1684 ID number is returned in the field
1685 <literal>/*/id:idzebra/localnumber</literal> in the namespace
1686 <literal>xmlns:id="http://www.indexdata.dk/zebra/"</literal>,
1687 where it can be picked up for later record updates or deletes.
1691 A new element set for retrieval of internal record
1692 data has been added, which can be used to access minimal records
1693 containing only the <literal>recordIdNumber</literal> &zebra;
1694 internal ID, or the <literal>recordIdOpaque</literal> string
1695 identifier. This works for any indexing filter used.
1696 See <xref linkend="special-retrieval"/>.
1700 The <literal>recordIdOpaque</literal> string parameter
1701 is an client-supplied, opaque record
1702 identifier, which may be used under
1703 insert, update and delete operations. The
1704 client software is responsible for assigning these to
1705 records. This identifier will
1706 replace zebra's own automagic identifier generation with a unique
1707 mapping from <literal>recordIdOpaque</literal> to the
1708 &zebra; internal <literal>recordIdNumber</literal>.
1709 <emphasis>The opaque <literal>recordIdOpaque</literal> string
1711 are not visible in retrieval records, nor are
1712 searchable, so the value of this parameter is
1713 questionable. It serves mostly as a convenient mapping from
1714 application domain string identifiers to &zebra; internal ID's.
1720 <sect2 id="administration-extended-services-yaz-client">
1721 <title>Extended services from yaz-client</title>
1724 We can now start a yaz-client admin session and create a database:
1727 $ yaz-client localhost:9999 -u admin/secret
1731 Now the <literal>Default</literal> database was created,
1732 we can insert an &acro.xml; file (esdd0006.grs
1733 from example/gils/records) and index it:
1736 Z> update insert id1234 esdd0006.grs
1739 The 3rd parameter - <literal>id1234</literal> here -
1740 is the <literal>recordIdOpaque</literal> package field.
1743 Actually, we should have a way to specify "no opaque record id" for
1744 yaz-client's update command.. We'll fix that.
1747 The newly inserted record can be searched as usual:
1752 Received SearchResponse.
1753 Search was a success.
1754 Number of hits: 1, setno 1
1755 SearchResult-1: term=utah cnt=1
1762 Let's delete the beast, using the same
1763 <literal>recordIdOpaque</literal> string parameter:
1766 Z> update delete id1234
1767 No last record (update ignored)
1768 Z> update delete 1 esdd0006.grs
1769 Got extended services response
1774 Received SearchResponse.
1775 Search was a success.
1776 Number of hits: 0, setno 2
1777 SearchResult-1: term=utah cnt=0
1784 If shadow register is enabled in your
1785 <filename>zebra.cfg</filename>,
1786 you must run the adm-commit command
1792 after each update session in order write your changes from the
1793 shadow to the life register space.
1798 <sect2 id="administration-extended-services-yaz-php">
1799 <title>Extended services from yaz-php</title>
1802 Extended services are also available from the &yaz; &acro.php; client layer. An
1803 example of an &yaz;-&acro.php; extended service transaction is given here:
1806 $record = '<record><title>A fine specimen of a record</title></record>';
1808 $options = array('action' => 'recordInsert',
1810 'record' => $record,
1811 'databaseName' => 'mydatabase'
1814 yaz_es($yaz, 'update', $options);
1815 yaz_es($yaz, 'commit', array());
1818 if ($error = yaz_error($yaz))
1828 <!-- Keep this comment at the end of the file
1833 sgml-minimize-attributes:nil
1834 sgml-always-quote-attributes:t
1837 sgml-parent-document: "zebra.xml"
1838 sgml-local-catalogs: nil
1839 sgml-namecase-general:t