1 <chapter id="querymodel">
2 <!-- $Id: querymodel.xml,v 1.8 2006-06-16 12:54:55 marc Exp $ -->
3 <title>Query Model</title>
5 <sect1 id="querymodel-overview">
6 <title>Query Model Overview</title>
9 <sect2 id="querymodel-query-languages">
10 <title>Query Languages</title>
13 Zebra is born as a networking Information Retrieval engine adhering
14 to the international standards
15 <ulink url="&url.z39.50;">Z39.50</ulink> and
16 <ulink url="&url.sru;">SRU</ulink>,
18 <literal>type-1 Reverse Polish Notation (RPN)</literal> query
20 Unfortunately, this model has only defined a binary
21 encoded representation, which is used as transport packaging in
22 the Z39.50 protocol layer. This representation is not human
23 readable, nor defines any convenient way to specify queries.
26 Since the <literal>type-1 (RPN)</literal>
27 query structure has no direct, useful string
28 representation, every origin application needs to provide some
29 form of mapping from a local query notation or representation to it.
33 <sect3 id="querymodel-query-languages-pqf">
34 <title>Prefix Query Format (PQF)</title>
37 Index Data has defined a textual representaion in the
38 <literal>Prefix Query Format</literal>, short
39 <literal>PQF</literal>, which mappes
40 <literal>one-to-one</literal> to binary encoded
41 <literal>type-1 RPN</literal> query packages.
42 It has been adopted by other
43 parties developing Z39.50 software, and is often referred to as
44 <literal>Prefix Query Notation</literal>, or in short
45 <literal>PQN</literal>. See
46 <xref linkend="querymodel-pqf"/> for further explanaitions and
47 descriptions of Zebra's capabilities.
51 <sect3 id="querymodel-query-languages-cql">
52 <title>Common Query Language (CQL)</title>
54 The query model of the <literal>type-1 RPN</literal>,
55 expressed in <literal>PQF/PQN</literal> is natively supported.
56 On the other hand, the default <literal>SRU</literal>
57 webservices <literal>Common Query Language</literal>
58 <ulink url="&url.cql;">CQL</ulink> is not natively supported.
61 Zebra can be configured to understand and map CQL to PQF. See
62 <xref linkend="querymodel-cql-to-pqf"/>.
68 <sect2 id="querymodel-operation-types">
69 <title>Operation types</title>
71 Zebra supports all of the three different
72 <literal>Z39.50/SRU</literal> operations defined in the
73 standards: <literal>explain</literal>, <literal>search</literal>,
74 and <literal>scan</literal>. A short description of the
75 functionality and purpose of each is quite in order here.
78 <sect3 id="querymodel-operation-type-explain">
79 <title>Explain Operation</title>
81 The <emphasis>syntax</emphasis> of Z39.50/SRU queries is
82 well known to any client, but the specific
83 <emphasis>semantics</emphasis> - taking into account a
84 particular servers functionalities and abilities - must be
85 discovered from case to case. Enters the
86 <literal>explain</literal> operation, which provides the means
88 <emphasis>fields</emphasis> (also called
89 <emphasis>indexes</emphasis> or <emphasis>access points</emphasis>
90 are provided, which default parameter the server uses, which
91 retrieve document formats are defined, and which specific parts
92 of the general query model are supported.
95 The Z39.50 embeddes the <literal>explain</literal> operation
97 <literal>search</literal> in the magic
98 <literal>IR-Explain-1</literal> database;
99 see <xref linkend="querymodel-exp1"/>.
102 In SRU, <literal>explain</literal> is an entirely seperate
103 operation, which returns an <literal>Zeerex
104 XML</literal> record according to the
105 structure defined by the protocol.
108 In both cases, the information gathered through
109 <literal>explain</literal> operations can be used to
110 auto-configure a client user interface to the servers
115 <sect3 id="querymodel-operation-type-search">
116 <title>Search Operation</title>
118 Search and retrieve interactions are the raison d'ĂȘtre.
119 They are used to query the remote database and
120 return search result documents. Search queries span from
121 simple free text searches to nested complex boolean queries,
122 targeting specific indexes, and possibly enhanced with many
123 query semantic specifications. Search interactions are the heart
124 and soul of Z39.50/SRU servers.
128 <sect3 id="querymodel-operation-type-scan">
129 <title>Scan Operation</title>
131 The <literal>scan</literal> operation is a helper functionality,
132 which operates on one index or access point a time.
136 the means to investigate the content of specific indexes.
137 Scanning an index returns a handfull of terms actually fond in
138 the indexes, and in addition the <literal>scan</literal>
139 operation returns th enumber of documents indexed by each term.
140 A search client can use this information to propose proper
141 spelling of search terms, to auto-fill search boxes, or to
142 display controlled vocabularies.
151 <sect1 id="querymodel-pqf">
152 <title>Prefix Query Format structure and syntax</title>
154 The <ulink url="&url.yaz.pqf;">PQF grammer</ulink>
155 is documented in the YAZ manual, and shall not be
156 repeated here. This textual PQF representation
157 is always during search mapped to the equivalent Zebra internal
161 <sect2 id="querymodel-pqf-tree">
162 <title>PQF tree structure</title>
164 The PQF parse tree - or the equivalent textual representation -
165 may start with one specification of the
166 <emphasis>attribute set</emphasis> used. Following is a query
168 consists of <emphasis>atomic query parts (APT)</emphasis>, eventually
169 paired by <emphasis>boolean binary operators</emphasis>, and
170 finally <emphasis>recursively combined </emphasis> into
174 <sect3 id="querymodel-attribute-sets">
175 <title>Attribute sets</title>
177 Attribute sets define the exact meaning and semantics of queries
178 issued. Zebra comes with some predefined attribute set
179 definitions, others can easily be defined and added to the
182 The Zebra internal query procesing is modeled after
183 the <literal>Bib1</literal> attribute set, and the non-use
184 attributes type 2-6 are hard-wired in. It is therefore essential
185 to be familiar with <xref linkend="querymodel-bib1"/>.
189 <table id="querymodel-attribute-sets-table"
190 frame="all" rowsep="1" colsep="1" align="center">
192 <caption>Attribute sets predefined in Zebra</caption>
195 <tr><td>one</td><td>two</td></tr>
200 <td><literal>exp-1</literal></td>
201 <td><literal>Explain</literal> attribute set</td>
202 <td>Special attribute set used on the special automagic
203 <literal>IR-Explain-1</literal> database to gain information on
204 server capabilities, database names, and database
208 <td><literal>bib-1</literal></td>
209 <td><literal>Bib1</literal> attribute set</td>
210 <td>Standard PQF query language attribute set which defines the
211 semantics of Z39.50 searching. In addition, all of the
212 non-use attributes (type 2-9) define the Zebra internal query
216 <td><literal>gils</literal></td>
217 <td><literal>GILS</literal> attribute set</td>
218 <td>Extention to the <literal>Bib1</literal> attribute set.</td>
224 <sect3 id="querymodel-boolean-operators">
225 <title>Boolean operators</title>
227 A pair of subquery trees, or of atomic queries, is combined
228 using the standard boolean operators into new query trees.
231 <table id="querymodel-boolean-operators-table"
232 frame="all" rowsep="1" colsep="1" align="center">
234 <caption>Boolean operators</caption>
237 <tr><td>one</td><td>two</td></tr>
241 <tr><td><literal>@and</literal></td>
242 <td>binary <literal>AND</literal> operator</td>
243 <td>Set intersection of two atomic queries hit sets</td>
245 <tr><td><literal>@or</literal></td>
246 <td>binary <literal>OR</literal> operator</td>
247 <td>Set union of two atomic queries hit sets</td>
249 <tr><td><literal>@not</literal></td>
250 <td>binary <literal>AND NOT</literal> operator</td>
251 <td>Set complement of two atomic queries hit sets</td>
253 <tr><td><literal>@prox</literal></td>
254 <td>binary <literal>PROXIMY</literal> operator</td>
255 <td>Set intersection of two atomic queries hit sets. In
256 addition, the intersection set is purged for all
257 documents which do not satisfy the requested query
258 term proximity. Usually a proper subset of the AND
265 For example, we can combine the terms
266 <emphasis>information</emphasis> and <emphasis>retrieval</emphasis>
267 into different searches in the default index of the default
268 attribute set as follows.
269 Querying for the union of all documents containing the
270 terms <emphasis>information</emphasis> OR
271 <emphasis>retrieval</emphasis>:
273 Z> find @or information retrieval
277 Querying for the intersection of all documents containing the
278 terms <emphasis>information</emphasis> AND
279 <emphasis>retrieval</emphasis>:
280 The hit set is a subset of the coresponding
283 Z> find @and information retrieval
287 Querying for the intersection of all documents containing the
288 terms <emphasis>information</emphasis> AND
289 <emphasis>retrieval</emphasis>, taking proximity into account:
290 The hit set is a subset of the coresponding
293 Z> find @prox information retrieval
297 Querying for the intersection of all documents containing the
298 terms <emphasis>information</emphasis> AND
299 <emphasis>retrieval</emphasis>, in the same order and near each
300 other as described in the term list
301 The hit set is a subset of the coresponding
304 Z> find "information retrieval"
310 <sect3 id="querymodel-atomic-queries">
311 <title>Atomic queries (APT)</title>
313 Atomic queries are the query parts which work on one acess point
314 only. These consist of <literal>an attribute list</literal>
315 followed by a <literal>single term</literal> or a
316 <literal>quoted term list</literal>, and are often called
317 <emphasis>Attributes-Plus-Terms (APT)</emphasis> queries.
320 Unsupplied non-use attributes type 2-9 are either inherited from
321 higher nodes in the query tree, or are set to Zebra's default values.
322 See <xref linkend="querymodel-bib1"/> for details.
325 <table id="querymodel-atomic-queries-table"
326 frame="all" rowsep="1" colsep="1" align="center">
328 <caption>Atomic queries</caption>
331 <tr><td>one</td><td>two</td></tr>
335 <tr><td><emphasis>attribute list</emphasis></td>
336 <td>List of <literal>orthogonal</literal> attributes</td>
337 <td>Any of the orthogonal attribute types may be omitted,
338 these are inherited from higher query tree nodes, or if not
339 inherited, are set to the default Zebra configuration values.
342 <tr><td><emphasis>term</emphasis></td>
343 <td>single <literal>term</literal>
344 or <literal>quoted term list</literal> </td>
345 <td>Here the search terms or list of search terms is added
351 Querying for the term <emphasis>information</emphasis> in the
352 default index using the default attribite set, the server choice
353 of access point/index, and the default non-use attributes.
355 Z> find "information"
359 Equivalent query fully specified including all default values:
361 Z> find @attrset bib-1 @attr 1=1017 @attr 2=3 @attr 3=3 @attr 4=1 @attr 5=100 @attr 6=1 "information"
366 Finding all documents which have empty titles. Notice that the
367 empty term must be quoted, but is otherwise legal.
375 <sect3 id="querymodel-use-string">
376 <title>Zebra's special use attribute type 1 of form 'string'</title>
378 The numeric <literal>use (type 1)</literal> attribute is usually
379 refered to from a given
380 attribute set. In addition, Zebra let you use
381 <emphasis>any internal index
382 name defined in your configuration</emphasis>
383 as use atribute value. This is a great feature for
384 debugging, and when you do
385 not need the complecity of defined use attribute values. It is
386 the preferred way of accessing Zebra indexes directly.
389 Finding all documents which have the term list "information
390 retrieval" in an Zebra index, using it's internal full string name.
392 Z> find @attr 1=sometext "information retrieval"
396 Searching the bib-1 use attribute 54 using it's string name:
398 Z> find @attr 1=Code-language eng
402 Searching in any silly string index - if it's defined in your
403 indexation rules and can be parsed by the PQF parser.
404 This is definitely not the recommended use of
405 this facility, as it might confuse your users with some very
408 Z> find @attr 1=silly/xpath/alike[@index]/name "information retrieval"
412 See <xref linkend="querymodel-bib1-mapping"/> for details, and
413 <xref linkend="server-sru"/>
414 for the SRU PQF query extention using string names as a fast
419 <sect3 id="querymodel-use-xpath">
420 <title>Zebra's special use attribute type 1 of form 'XPath'
421 for GRS filters</title>
423 As we have seen above, it is possible (albeit seldom a great
425 <ulink url="http://www.w3.org/TR/xpath">XPath 1.0</ulink> based
426 search by defining <literal>use (type 1)</literal>
427 <emphasis>string</emphasis> attributes which in appearence
428 <emphasis>resemble XPath queries</emphasis>. There are two
429 problems with this approach: first, the XPath-look-alike has to
430 be defined at indexation time, no new undefined
431 XPath queries can entered at search time, and second, it might
432 confuse users very much that an XPath-alike index name in fact
433 gets populated from a possible entirely different XML element
434 than it pretends to acess.
437 When using the <literal>GRS Record Model</literal>
438 (see <xref linkend="record-model-grs"/>), we have the
439 possibility to embed <emphasis>life</emphasis>
441 in the PQF queries, which are here called
442 <literal>use (type 1)</literal> <emphasis>xpath</emphasis>
443 attributes. You must enable the
444 <literal>xpath enable</literal> directive in your
445 <literal>.abs</literal> config files.
448 Only a <emphasis>very</emphasis> restricted subset of the
449 <ulink url="http://www.w3.org/TR/xpath">XPath 1.0</ulink>
450 standard is supported as the GRS record model is simpler than
451 a full XML DOM structure. See the following examples for
455 Finding all documents which have the term "content"
456 inside a text node found in a specific XML DOM
457 <emphasis>subtree</emphasis>, whose starting element is
460 Z> find @attr 1=/root content
461 Z> find @attr 1=/root/first content
463 <emphasis>Notice that the
464 XPath must be absolute, i.e., must start with '/', and that the
465 XPath <literal>decendant-or-self</literal> axis followed by a
466 text node selection <literal>text()</literal> is implicitly
467 appended to the stated XPath.
469 It follows that the above searches are interpreted as:
471 Z> find @attr 1=/root//text() content
472 Z> find @attr 1=/root/first//text() content
477 Filter the adressing XPath by a predicate working on exact
479 attributes (in the XML sense) can be done: return all those docs which
480 have the term "english" contained in one of all text subnodes of
481 the subtree defined by the XPath
482 <literal>/record/title[@lang='en']</literal>
484 Z> find @attr 1=/record/title[@lang='en'] english
489 Combining numeric indexes, boolean expressions,
490 and xpath based searches is possible:
492 Z> find @attr 1=/record/title @and foo bar
493 Z> find @and @attr 1=/record/title foo @attr 1=4 bar
497 Escaping PQF keywords and other non-parseable XPath constructs
498 with <literal>'{ }'</literal> to prevent syntax errors:
500 Z> find @attr {1=/root/first[@attr='danish']} content
501 Z> find @attr {1=/root/second[@attr='danish lake']}
502 Z> find @attr {1=/root/third[@attr='dansk s\xc3\xb8']}
506 It is worth mentioning that these dynamic performed XPath
507 queries are a performance bottelneck, as no optimized
508 specialized indexes can be used. Therefore, avoid the use of
509 this facility when speed is essential, and the database content
510 size is medium to large.
516 <sect2 id="querymodel-exp1">
517 <title>Explain Attribute Set</title>
519 The Z39.50 standard defines the
520 <ulink url="&url.z39.50.explain;">Explain</ulink>attribute set
521 <literal>exp-1</literal>, which is used to discover information
522 about a server's search semantics and functional capabilities
523 Zebra exposes a "classic"
524 Explain database by base name <literal>IR-Explain-1</literal>, which
525 is populated with system internal information.
528 The attribute-set <literal>exp-1</literal> consists of a single
529 <literal>Use (type 1)</literal> attribute.
532 In addition, the non-Use
533 <literal>bib-1</literal> attributes, that is, the types
534 <literal>Relation</literal>, <literal>Position</literal>,
535 <literal>Structure</literal>, <literal>Truncation</literal>,
536 and <literal>Completeness</literal> are imported from
537 the <literal>bib-1</literal> attribute set, and may be used
538 within any explain query.
541 <sect3 id="querymodel-exp1-use">
542 <title>Use Attributes (type = 1)</title>
544 The following Explain search atributes are supported:
545 <literal>ExplainCategory</literal> (@attr 1=1),
546 <literal>DatabaseName</literal> (@attr 1=3),
547 <literal>DateAdded</literal> (@attr 1=9),
548 <literal>DateChanged</literal>(@attr 1=10).
551 A search in the use attribute <literal>ExplainCategory</literal>
552 supports only these predefined values:
553 <literal>CategoryList</literal>, <literal>TargetInfo</literal>,
554 <literal>DatabaseInfo</literal>, <literal>AttributeDetails</literal>.
557 See <filename>tab/explain.att</filename> and the
558 <ulink url="&url.z39.50;">Z39.50</ulink> standard
559 for more information.
564 <title>Explain searches with yaz-client</title>
566 Classic Explain only defines retrieval of Explain information
567 via ASN.1. Pratically no Z39.50 clients supports this. Fortunately
568 they don't have to - Zebra allows retrieval of this information
570 <literal>SUTRS</literal>, <literal>XML</literal>,
571 <literal>GRS-1</literal> and <literal>ASN.1</literal> Explain.
575 List supported categories to find out which explain commands are
579 Z> find @attr exp1 1=1 categorylist
586 Get target info, that is, investigate which databases exist at
587 this server endpoint:
590 Z> find @attr exp1 1=1 targetinfo
601 List all supported databases, the number of hits
602 is the number of databases found, which most commonly are the
604 the <literal>Default</literal> and the
605 <literal>IR-Explain-1</literal> databases.
608 Z> find @attr exp1 1=1 databaseinfo
615 Get database info record for database <literal>Default</literal>.
618 Z> find @and @attr exp1 1=1 databaseinfo @attr exp1 1=3 Default
620 Identical query with explicitly specified attribute set:
623 Z> find @attrset exp1 @and @attr 1=1 databaseinfo @attr 1=3 Default
628 Get attribute details record for database
629 <literal>Default</literal>.
630 This query is very useful to study the internal Zebra indexes.
631 If records have been indexed using the <literal>alvis</literal>
632 XSLT filter, the string representation names of the known indexes can be
636 Z> find @and @attr exp1 1=1 attributedetails @attr exp1 1=3 Default
638 Identical query with explicitly specified attribute set:
641 Z> find @attrset exp1 @and @attr 1=1 attributedetails @attr 1=3 Default
648 <sect2 id="querymodel-bib1">
649 <title>Bib1 Attribute Set</title>
651 Something about querying to be written ..
654 Most of the information contained in this section is an excerpt of
655 the <literal>ATTRIBUTE SET BIB-1 (Z39.50-1995)
657 found at <ulink url="&url.z39.50.attset.bib1.1995;">. The BIB-1
658 Attribute Set Semantics</ulink> from 1995, also in an updated
659 <ulink url="&url.z39.50.attset.bib1;">Bib-1
660 Attribute Set</ulink>
661 version from 2003. Index Data is not the copyright holder of this
666 <sect3 id="querymodel-bib1-use">
667 <title>Use Attributes (type 1)</title>
671 A use attribute specifies an access point for any atomic query.
672 These acess points are highly dependent on the attribute set used
673 in the query, and are user configurable using the following
674 default configuration files:
675 <filename>tab/bib1.att</filename>,
676 <filename>tab/dan1.att</filename>,
677 <filename>tab/explain.att</filename>, and
678 <filename>tab/gils.att</filename>.
679 New attribute sets can be added by adding new
680 <filename>tab/*.att</filename> configuration files, which need to
681 be sourced in the main configuration <filename>zebra.cfg</filename>.
685 In addition, Zebra allows the acess of
686 <emphasis>internal index names</emphasis> and <emphasis>dynamic
687 XPath</emphasis> as use attributes.
688 See <xref linkend="querymodel-use-string"/> and
689 <xref linkend="querymodel-use-xpath"/> for
690 alternative acess to the Zebra internal index names and XPath queries.
694 Phrase search for <emphasis>information retrieval</emphasis> in
697 Z> find @attr 1=4 "information retrieval"
702 <sect3 id="querymodel-bib1-relation">
703 <title>Relation Attributes (type 2)</title>
706 Relation attributes describe the relationship of the access
708 of the relation) to the search term as qualified by the attributes (right
709 side of the relation), e.g., Date-publication <= 1975.
712 <table id="querymodel-bib1-relation-table"
713 frame="all" rowsep="1" colsep="1" align="center">
715 <caption>Relation Attributes (type 2)</caption>
730 <td>Less than or equal</td>
740 <td>Greater or equal</td>
745 <td>Greater than</td>
770 <td>AlwaysMatches</td>
778 The relation attribute
779 <literal>relevance (102)</literal> is supported, see
780 <xref linkend="administration-ranking"/> for full information.
781 <!-- always-matches (103) not supported for all indexes -->
785 All ordering operations are based on a lexicographical ordering,
786 <emphasis>expect</emphasis> when the
787 <literal>structure attribute numeric (109)</literal> is used. In
788 this case, ordering is numerical. See
789 <xref linkend="querymodel-bib1-structure"/>.
793 Ranked search for <emphasis>information retrieval</emphasis> in
796 Z> find @attr 1=4 @attr 2=102 "information retrieval"
801 <sect3 id="querymodel-bib1-position">
802 <title>Position Attributes (type 3)</title>
805 The position attribute specifies the location of the search term
806 within the field or subfield in which it appears.
809 <table id="querymodel-bib1-position-table"
810 frame="all" rowsep="1" colsep="1" align="center">
812 <caption>Position Attributes (type 3)</caption>
822 <td>First in field </td>
827 <td>First in subfield</td>
832 <td>Any position in field</td>
840 The position attribute values <literal>first in field (1)</literal>,
841 and <literal>first in subfield(2)</literal> are unsupported.
842 Using them does not trigger an error, but silent defaults to
843 <literal>any position in field (3)</literal>.
848 <sect3 id="querymodel-bib1-structure">
849 <title>Structure Attributes (type 4)</title>
852 The structure attribute specifies the type of search
853 term. This causes the search to be mapped on
854 different Zebra internal indexes, which must have been defined
859 The possible values of the
860 <literal>structure attribute (type 4)</literal> can be defined
861 using the configuraiton file <filename>
862 tab/default.idx</filename>.
863 The default configuration is summerized in this table.
866 <table id="querymodel-bib1-structure-table"
867 frame="all" rowsep="1" colsep="1" align="center">
869 <caption>Structure Attributes (type 4)</caption>
899 <td>Date (normalized)</td>
909 <td>Date (un-normalized)</td>
914 <td>Name (normalized) </td>
919 <td>Name (un-normalized) </td>
934 <td>Free-form-text</td>
939 <td>Document-text</td>
944 <td>Local-number</td>
954 <td>Numeric string</td>
963 The structure attribute value <literal>local-number
965 is supported, and maps always to the Zebra internal document ID.
970 the GILS schema (<literal>gils.abs</literal>), the
971 west-bounding-coordinate is indexed as type <literal>n</literal>,
972 and is therefore searched by specifying
973 <emphasis>structure</emphasis>=<emphasis>Numeric String</emphasis>.
974 To match all those records with west-bounding-coordinate greater
975 than -114 we use the following query:
977 Z> find @attr 4=109 @attr 2=5 @attr gils 1=2038 -114
981 <sect3 id="querymodel-bib1-truncation">
982 <title>Truncation Attributes (type = 5)</title>
985 The truncation attribute specifies whether variations of one or
986 more characters are allowed between serch term and hit terms, or
987 not. Using non-default truncation attributes will broaden the
988 document hit set of a search query.
991 <table id="querymodel-bib1-truncation-table"
992 frame="all" rowsep="1" colsep="1" align="center">
994 <caption>Truncation Attributes (type 5)</caption>
1004 <td>Right truncation </td>
1009 <td>Left truncation</td>
1014 <td>Left and right truncation</td>
1019 <td>Do not truncate</td>
1024 <td>Process # in search term</td>
1042 Truncation attribute value
1043 <literal>Process # in search term (100)</literal> is a
1044 poor-man's regular expression search. It maps
1045 each <literal>#</literal> to <literal>.*</literal>, and
1046 performes then a <literal>Regexp-1 (102)</literal> regular
1050 Truncation attribute value
1051 <literal>Regexp-1 (102)</literal> is a normal regular search,
1055 Truncation attribute value
1056 <literal>Regexp-2 (103) </literal> is a Zebra specific extention
1057 which allows <emphasis>fuzzy</emphasis> matches. One single
1058 error in spelling of search terms is allowed, i.e., a document
1059 is hit if it includes a term which can be mapped to the used
1060 search term by one character substitution, addition, deletion or
1064 Special 104, 105, 106 are deprecated and will be removed! -->
1067 <sect3 id="querymodel-bib1-completeness">
1068 <title>Completeness Attributes (type = 6)</title>
1070 This attribute is ONLY used if structure w, p is to be
1071 chosen. completeness is ignorned if not w, p is to be
1073 Incomplete field(1) is the default and makes Zebra use
1075 complete subfield(2) and complete field(3) both triggers
1076 search field type p.
1082 <sect2 id="querymodel-zebra-attr-search">
1083 <title>Zebra specific Search Extentions to all Attribute Sets</title>
1085 Zebra extends the Bib1 attribute types, and these extentions are
1086 recognized regardless of attribute
1087 set used in a <literal>search</literal> operation query.
1090 <table id="querymodel-zebra-attr-search-table"
1091 frame="all" rowsep="1" colsep="1" align="center">
1093 <caption>Zebra Search Attribute Extentions</caption>
1099 <td>Zebra version</td>
1104 <td>Embedded Sort</td>
1116 <td>Rank Weight</td>
1122 <td>Approx Limit</td>
1128 <td>Term Reference</td>
1136 <sect3 id="querymodel-zebra-attr-sorting">
1137 <title>Zebra Extention Embedded Sort Attribute (type 7)</title>
1140 The embedded sort is a way to specify sort within a query - thus
1141 removing the need to send a Sort Request separately. It is both
1142 faster and does not require clients to deal with the Sort
1146 The possible values after attribute <literal>type 7</literal> are
1147 <literal>1</literal> ascending and
1148 <literal>2</literal> descending.
1149 The attributes+term (APT) node is separate from the
1150 rest and must be <literal>@or</literal>'ed.
1151 The term associated with APT is the sorting level in integers,
1152 where <literal>0</literal> means primary sort,
1153 <literal>1</literal> means secondary sort, and so forth.
1154 See also <xref linkend="administration-ranking"/>.
1157 For example, searching for water, sort by title (ascending)
1159 Z> find @or @attr 1=1016 water @attr 7=1 @attr 1=4 0
1163 Or, searching for water, sort by title ascending, then date descending
1165 Z> find @or @or @attr 1=1016 water @attr 7=1 @attr 1=4 0 @attr 7=2 @attr 1=30 1
1169 <sect3 id="querymodel-zebra-attr-estimation">
1170 <title>Zebra Extention Term Set Attribute (type 8)</title>
1173 The Term Set feature is a facility that allows a search to store
1174 hitting terms in a "pseudo" resultset; thus a search (as usual) +
1175 a scan-like facility. Requires a client that can do named result
1176 sets since the search generates two result sets. The value for
1177 attribute 8 is the name of a result set (string). The terms in
1178 the named term set are returned as SUTRS records.
1181 For example, searching for u in title, right truncated, and
1182 storing the result in term set named 'aset'
1184 Z> find @attr 5=1 @attr 1=4 @attr 8=aset u
1188 The model has one serious flaw: we don't know the size of term
1189 set. Experimental. Do not use in production code.
1192 <sect3 id="querymodel-zebra-attr-weight">
1193 <title>Zebra Extention Rank Weight Attribute (type 9)</title>
1196 Rank weight is a way to pass a value to a ranking algorithm - so
1197 that one APT has one value - while another as a different one.
1198 See also <xref linkend="administration-ranking"/>.
1201 For example, searching for utah in title with weight 30 as well
1202 as any with weight 20:
1204 Z> find @attr 2=102 @or @attr 9=30 @attr 1=4 utah @attr 9=20 utah
1208 <sect3 id="querymodel-zebra-attr-limit">
1209 <title>Zebra Extention Approximative Limit Attribute (type 9)</title>
1212 Newer Zebra versions normally estemiates hit count for every APT
1213 (leaf) in the query tree. These hit counts are returned as part of
1214 the searchResult-1 facility in the binary encoded Z39.50 search
1218 By setting a limit for the APT we can make Zebra turn into
1219 approximate hit count when a certain hit count limit is
1220 reached. A value of zero means exact hit count.
1223 For example, we might be intersted in exact hit count for a, but
1224 for b we allow hit count estimates for 1000 and higher.
1226 Z> find @and a @attr 9=1000 b
1230 The estimated hit count fascility makes searches faster, as one
1231 only needs to process large hit lists partially.
1234 This facility clashes with rank weight, because there all
1235 documents in the hit lists need to be examined for scoring and
1237 It is an experimental
1238 extention. Do not use in production code.
1241 <sect3 id="querymodel-zebra-attr-termref">
1242 <title>Zebra Extention Term Reference Attribute (type 10)</title>
1245 Zebra supports the <literal>searchResult-1</literal> facility.
1246 If the <literal>Term Reference Attribute (type 10)</literal> is
1247 given, that specifies a subqueryId value returned as part of the
1248 search result. It is a way for a client to name an APT part of a
1258 Experimental. Do not use in production code.
1265 <sect2 id="querymodel-zebra-attr-scan">
1266 <title>Zebra specific Scan Extentions to all Attribute Sets</title>
1268 Zebra extends the Bib1 attribute types, and these extentions are
1269 recognized regardless of attribute
1270 set used in a <literal>scan</literal> operation query.
1272 <table id="querymodel-zebra-attr-scan-table"
1273 frame="all" rowsep="1" colsep="1" align="center">
1275 <caption>Zebra Scan Attribute Extentions</caption>
1281 <td>Zebra version</td>
1286 <td>Result Set Narrow</td>
1292 <td>Approximative Limit</td>
1300 <sect3 id="querymodel-zebra-attr-narrow">
1301 <title>Zebra Extention Result Set Narrow (type 8)</title>
1304 If attribute <literal>Result Set Narrow (type 8)</literal>
1305 is given for <literal>scan</literal>, the value is the name of a
1306 result set. Each hit count in <literal>scan</literal> is
1307 <literal>@and</literal>'ed with the result set given.
1316 Experimental and buggy. Definitely not to be used in production code.
1319 <sect3 id="querymodel-zebra-attr-approx">
1320 <title>Zebra Extention Approximative Limit (type 9)</title>
1323 The <literal>Zebra Extention Approximative Limit (type
1324 9)</literal> is a way to enable approx
1325 hit counts for <literal>scan</literal> hit counts, in the same
1326 way as for <literal>search</literal> hit counts.
1335 Experimental. Do not use in production code.
1342 <sect2 id="querymodel-bib1-mapping">
1343 <title>Mapping from Bib1 Attributes to Zebra internal
1344 register indexes</title>
1350 <!-- see in util/zebramap.c
1353 if (completeness_value == 2 || completeness_value == 3)
1359 *sort_flag =(sort_relation_value > 0) ? 1 : 0;
1360 *search_type = "phrase";
1361 strcpy(rank_type, "void");
1362 if (relation_value == 102)
1364 if (weight_value == -1)
1366 sprintf(rank_type, "rank,w=%d,u=%d", weight_value, use_value);
1368 if (relation_value == 103)
1370 *search_type = "always";
1378 switch (structure_value)
1380 case 6: /* word list */
1381 *search_type = "and-list";
1383 case 105: /* free-form-text */
1384 *search_type = "or-list";
1386 case 106: /* document-text */
1387 *search_type = "or-list";
1390 case 1: /* phrase */
1392 case 108: /* string */
1393 *search_type = "phrase";
1395 case 107: /* local-number */
1396 *search_type = "local";
1399 case 109: /* numeric string */
1401 *search_type = "numeric";
1405 *search_type = "phrase";
1409 *search_type = "phrase";
1413 *search_type = "phrase";
1417 *search_type = "phrase";
1428 <emphasis>Use</emphasis> attributes are interpreted according to the
1429 attribute sets which have been loaded in the
1430 <literal>zebra.cfg</literal> file, and are matched against specific
1431 fields as specified in the <literal>.abs</literal> file which
1432 describes the profile of the records which have been loaded.
1433 If no Use attribute is provided, a default of Bib-1 Any is assumed.
1437 If a <emphasis>Structure</emphasis> attribute of
1438 <emphasis>Phrase</emphasis> is used in conjunction with a
1439 <emphasis>Completeness</emphasis> attribute of
1440 <emphasis>Complete (Sub)field</emphasis>, the term is matched
1441 against the contents of the phrase (long word) register, if one
1442 exists for the given <emphasis>Use</emphasis> attribute.
1443 A phrase register is created for those fields in the
1444 <literal>.abs</literal> file that contains a
1445 <literal>p</literal>-specifier.
1446 <!-- ### whatever the hell _that_ is -->
1450 If <emphasis>Structure</emphasis>=<emphasis>Phrase</emphasis> is
1451 used in conjunction with <emphasis>Incomplete Field</emphasis> - the
1452 default value for <emphasis>Completeness</emphasis>, the
1453 search is directed against the normal word registers, but if the term
1454 contains multiple words, the term will only match if all of the words
1455 are found immediately adjacent, and in the given order.
1456 The word search is performed on those fields that are indexed as
1457 type <literal>w</literal> in the <literal>.abs</literal> file.
1461 If the <emphasis>Structure</emphasis> attribute is
1462 <emphasis>Word List</emphasis>,
1463 <emphasis>Free-form Text</emphasis>, or
1464 <emphasis>Document Text</emphasis>, the term is treated as a
1465 natural-language, relevance-ranked query.
1466 This search type uses the word register, i.e. those fields
1467 that are indexed as type <literal>w</literal> in the
1468 <literal>.abs</literal> file.
1472 If the <emphasis>Structure</emphasis> attribute is
1473 <emphasis>Numeric String</emphasis> the term is treated as an integer.
1474 The search is performed on those fields that are indexed
1475 as type <literal>n</literal> in the <literal>.abs</literal> file.
1479 If the <emphasis>Structure</emphasis> attribute is
1480 <emphasis>URx</emphasis> the term is treated as a URX (URL) entity.
1481 The search is performed on those fields that are indexed as type
1482 <literal>u</literal> in the <literal>.abs</literal> file.
1486 If the <emphasis>Structure</emphasis> attribute is
1487 <emphasis>Local Number</emphasis> the term is treated as
1488 native Zebra Record Identifier.
1492 If the <emphasis>Relation</emphasis> attribute is
1493 <emphasis>Equals</emphasis> (default), the term is matched
1494 in a normal fashion (modulo truncation and processing of
1495 individual words, if required).
1496 If <emphasis>Relation</emphasis> is <emphasis>Less Than</emphasis>,
1497 <emphasis>Less Than or Equal</emphasis>,
1498 <emphasis>Greater than</emphasis>, or <emphasis>Greater than or
1499 Equal</emphasis>, the term is assumed to be numerical, and a
1500 standard regular expression is constructed to match the given
1502 If <emphasis>Relation</emphasis> is <emphasis>Relevance</emphasis>,
1503 the standard natural-language query processor is invoked.
1507 For the <emphasis>Truncation</emphasis> attribute,
1508 <emphasis>No Truncation</emphasis> is the default.
1509 <emphasis>Left Truncation</emphasis> is not supported.
1510 <emphasis>Process # in search term</emphasis> is supported, as is
1511 <emphasis>Regxp-1</emphasis>.
1512 <emphasis>Regxp-2</emphasis> enables the fault-tolerant (fuzzy)
1513 search. As a default, a single error (deletion, insertion,
1514 replacement) is accepted when terms are matched against the register
1519 <sect2 id="querymodel-regular">
1520 <title>Zebra Regular Expressions in Truncation Attribute (type = 5)</title>
1523 Each term in a query is interpreted as a regular expression if
1524 the truncation value is either <emphasis>Regxp-1 (@attr 5=102)</emphasis>
1525 or <emphasis>Regxp-2 (@attr 5=103)</emphasis>.
1526 Both query types follow the same syntax with the operands:
1529 <table id="querymodel-regular-operands-table"
1530 frame="all" rowsep="1" colsep="1" align="center">
1532 <caption>Regular Expression Operands</caption>
1535 <tr><td>one</td><td>two</td></tr>
1540 <td><literal>x</literal></td>
1541 <td>Matches the character <literal>x</literal>.</td>
1544 <td><literal>.</literal></td>
1545 <td>Matches any character.</td>
1548 <td><literal>[ .. ]</literal></td>
1549 <td>Matches the set of characters specified;
1550 such as <literal>[abc]</literal> or <literal>[a-c]</literal>.</td>
1556 The above operands can be combined with the following operators:
1559 <table id="querymodel-regular-operators-table"
1560 frame="all" rowsep="1" colsep="1" align="center">
1561 <caption>Regular Expression Operators</caption>
1564 <tr><td>one</td><td>two</td></tr>
1569 <td><literal>x*</literal></td>
1570 <td>Matches <literal>x</literal> zero or more times.
1571 Priority: high.</td>
1574 <td><literal>x+</literal></td>
1575 <td>Matches <literal>x</literal> one or more times.
1576 Priority: high.</td>
1579 <td><literal>x?</literal></td>
1580 <td> Matches <literal>x</literal> zero or once.
1581 Priority: high.</td>
1584 <td><literal>xy</literal></td>
1585 <td> Matches <literal>x</literal>, then <literal>y</literal>.
1586 Priority: medium.</td>
1589 <td><literal>x|y</literal></td>
1590 <td> Matches either <literal>x</literal> or <literal>y</literal>.
1594 <td><literal>( )</literal></td>
1595 <td>The order of evaluation may be changed by using parentheses.</td>
1601 If the first character of the <literal>Regxp-2</literal> query
1602 is a plus character (<literal>+</literal>) it marks the
1603 beginning of a section with non-standard specifiers.
1604 The next plus character marks the end of the section.
1605 Currently Zebra only supports one specifier, the error tolerance,
1606 which consists one digit.
1610 Since the plus operator is normally a suffix operator the addition to
1611 the query syntax doesn't violate the syntax for standard regular
1616 For example, a phrase search with regular expressions in
1617 the title-register is performed like this:
1619 Z> find @attr 1=4 @attr 5=102 "informat.* retrieval"
1624 Combinations with other attributes are possible. For example, a
1625 ranked search with a regular expression:
1627 Z> find @attr 1=4 @attr 5=102 @attr 2=102 "informat.* retrieval"
1635 The RecordType parameter in the <literal>zebra.cfg</literal> file, or
1636 the <literal>-t</literal> option to the indexer tells Zebra how to
1637 process input records.
1638 Two basic types of processing are available - raw text and structured
1639 data. Raw text is just that, and it is selected by providing the
1640 argument <literal>text</literal> to Zebra. Structured records are
1641 all handled internally using the basic mechanisms described in the
1642 subsequent sections.
1643 Zebra can read structured records in many different formats.
1649 <sect1 id="querymodel-cql-to-pqf">
1650 <title>Server Side CQL to PQF Query Translation</title>
1653 <literal><cql2rpn>l2rpn.txt</cql2rpn></literal>
1654 YAZ Frontend Virtual
1655 Hosts option, one can configure
1656 the YAZ Frontend CQL-to-PQF
1657 converter, specifying the interpretation of various
1658 <ulink url="&url.cql;">CQL</ulink>
1659 indexes, relations, etc. in terms of Type-1 query attributes.
1660 <!-- The yaz-client config file -->
1663 For example, using server-side CQL-to-PQF conversion, one might
1664 query a zebra server like this:
1667 yaz-client localhost:9999
1669 Z> find text=(plant and soil)
1672 and - if properly configured - even static relevance ranking can
1673 be performed using CQL query syntax:
1676 Z> find text = /relevant (plant and soil)
1682 By the way, the same configuration can be used to
1683 search using client-side CQL-to-PQF conversion:
1684 (the only difference is <literal>querytype cql2rpn</literal>
1686 <literal>querytype cql</literal>, and the call specifying a local
1690 yaz-client -q local/cql2pqf.txt localhost:9999
1691 Z> querytype cql2rpn
1692 Z> find text=(plant and soil)
1698 Exhaustive information can be found in the
1699 Section "Specification of CQL to RPN mappings" in the YAZ manual.
1700 <ulink url="http://www.indexdata.dk/yaz/doc/tools.tkl#tools.cql.map">
1701 http://www.indexdata.dk/yaz/doc/tools.tkl#tools.cql.map</ulink>,
1702 and shall therefore not be repeated here.
1707 <ulink url="http://www.loc.gov/z3950/agency/zing/cql/dc-indexes.html">
1708 http://www.loc.gov/z3950/agency/zing/cql/dc-indexes.html</ulink>
1709 for the Maintenance Agency's work-in-progress mapping of Dublin Core
1710 indexes to Attribute Architecture (util, XD and BIB-2)
1720 <!-- Keep this comment at the end of the file
1725 sgml-minimize-attributes:nil
1726 sgml-always-quote-attributes:t
1729 sgml-parent-document: "zebra.xml"
1730 sgml-local-catalogs: nil
1731 sgml-namecase-general:t