1 <chapter id="querymodel">
2 <!-- $Id: querymodel.xml,v 1.34 2007-06-22 12:59:23 marc Exp $ -->
3 <title>Query Model</title>
5 <section id="querymodel-overview">
6 <title>Query Model Overview</title>
8 <section id="querymodel-query-languages">
9 <title>Query Languages</title>
12 &zebra; is born as a networking Information Retrieval engine adhering
13 to the international standards
14 <ulink url="&url.z39.50;">&acro.z3950;</ulink> and
15 <ulink url="&url.sru;">&acro.sru;</ulink>,
17 type-1 Reverse Polish Notation (&acro.rpn;) query
19 Unfortunately, this model has only defined a binary
20 encoded representation, which is used as transport packaging in
21 the &acro.z3950; protocol layer. This representation is not human
22 readable, nor defines any convenient way to specify queries.
25 Since the type-1 (&acro.rpn;)
26 query structure has no direct, useful string
27 representation, every client application needs to provide some
28 form of mapping from a local query notation or representation to it.
32 <section id="querymodel-query-languages-pqf">
33 <title>Prefix Query Format (&acro.pqf;)</title>
35 Index Data has defined a textual representation in the
36 <ulink url="&url.yaz.pqf;">Prefix Query Format</ulink>, short
37 <emphasis>&acro.pqf;</emphasis>, which maps
38 one-to-one to binary encoded
39 <emphasis>type-1 &acro.rpn;</emphasis> queries.
40 &acro.pqf; has been adopted by other
41 parties developing &acro.z3950; software, and is often referred to as
42 <emphasis>Prefix Query Notation</emphasis>, or in short
44 <xref linkend="querymodel-rpn"/> for further explanations and
45 descriptions of &zebra;'s capabilities.
49 <section id="querymodel-query-languages-cql">
50 <title>Common Query Language (&acro.cql;)</title>
52 The query model of the type-1 &acro.rpn;,
53 expressed in &acro.pqf;/&acro.pqn; is natively supported.
54 On the other hand, the default &acro.sru;
55 web services <emphasis>Common Query Language</emphasis>
56 <ulink url="&url.cql;">&acro.cql;</ulink> is not natively supported.
59 &zebra; can be configured to understand and map &acro.cql; to &acro.pqf;. See
60 <xref linkend="querymodel-cql-to-pqf"/>.
66 <section id="querymodel-operation-types">
67 <title>Operation types</title>
69 &zebra; supports all of the three different
70 &acro.z3950;/&acro.sru; operations defined in the
71 standards: explain, search,
72 and scan. A short description of the
73 functionality and purpose of each is quite in order here.
76 <section id="querymodel-operation-type-explain">
77 <title>Explain Operation</title>
79 The <emphasis>syntax</emphasis> of &acro.z3950;/&acro.sru; queries is
80 well known to any client, but the specific
81 <emphasis>semantics</emphasis> - taking into account a
82 particular servers functionalities and abilities - must be
83 discovered from case to case. Enters the
84 explain operation, which provides the means for learning which
85 <emphasis>fields</emphasis> (also called
86 <emphasis>indexes</emphasis> or <emphasis>access points</emphasis>)
87 are provided, which default parameter the server uses, which
88 retrieve document formats are defined, and which specific parts
89 of the general query model are supported.
92 The &acro.z3950; embeds the explain operation
95 <literal>IR-Explain-1</literal> database;
96 see <xref linkend="querymodel-exp1"/>.
99 In &acro.sru;, explain is an entirely separate
100 operation, which returns an ZeeRex &acro.xml; record according to the
101 structure defined by the protocol.
104 In both cases, the information gathered through
105 explain operations can be used to
106 auto-configure a client user interface to the servers
111 <section id="querymodel-operation-type-search">
112 <title>Search Operation</title>
114 Search and retrieve interactions are the raison d'ĂȘtre.
115 They are used to query the remote database and
116 return search result documents. Search queries span from
117 simple free text searches to nested complex boolean queries,
118 targeting specific indexes, and possibly enhanced with many
119 query semantic specifications. Search interactions are the heart
120 and soul of &acro.z3950;/&acro.sru; servers.
124 <section id="querymodel-operation-type-scan">
125 <title>Scan Operation</title>
127 The scan operation is a helper functionality,
128 which operates on one index or access point a time.
132 the means to investigate the content of specific indexes.
133 Scanning an index returns a handful of terms actually found in
134 the indexes, and in addition the scan
135 operation returns the number of documents indexed by each term.
136 A search client can use this information to propose proper
137 spelling of search terms, to auto-fill search boxes, or to
138 display controlled vocabularies.
147 <section id="querymodel-rpn">
148 <title>&acro.rpn; queries and semantics</title>
150 The <ulink url="&url.yaz.pqf;">&acro.pqf; grammar</ulink>
151 is documented in the &yaz; manual, and shall not be
152 repeated here. This textual &acro.pqf; representation
153 is not transmistted to &zebra; during search, but it is in the
154 client mapped to the equivalent &acro.z3950; binary
158 <section id="querymodel-rpn-tree">
159 <title>&acro.rpn; tree structure</title>
161 The &acro.rpn; parse tree - or the equivalent textual representation in &acro.pqf; -
162 may start with one specification of the
163 <emphasis>attribute set</emphasis> used. Following is a query
165 consists of <emphasis>atomic query parts (&acro.apt;)</emphasis> or
166 <emphasis>named result sets</emphasis>, eventually
167 paired by <emphasis>boolean binary operators</emphasis>, and
168 finally <emphasis>recursively combined </emphasis> into
172 <section id="querymodel-attribute-sets">
173 <title>Attribute sets</title>
175 Attribute sets define the exact meaning and semantics of queries
176 issued. &zebra; comes with some predefined attribute set
177 definitions, others can easily be defined and added to the
181 <table id="querymodel-attribute-sets-table" frame="top">
182 <title>Attribute sets predefined in &zebra;</title>
186 <entry>Attribute set</entry>
187 <entry>&acro.pqf; notation (Short hand)</entry>
188 <entry>Status</entry>
195 <entry>Explain</entry>
196 <entry><literal>exp-1</literal></entry>
197 <entry>Special attribute set used on the special automagic
198 <literal>IR-Explain-1</literal> database to gain information on
199 server capabilities, database names, and database
200 and semantics.</entry>
201 <entry>predefined</entry>
204 <entry>&acro.bib1;</entry>
205 <entry><literal>bib-1</literal></entry>
206 <entry>Standard &acro.pqf; query language attribute set which defines the
207 semantics of &acro.z3950; searching. In addition, all of the
208 non-use attributes (types 2-14) define the hard-wired
209 &zebra; internal query
211 <entry>default</entry>
215 <entry><literal>gils</literal></entry>
216 <entry>Extension to the &acro.bib1; attribute set.</entry>
217 <entry>predefined</entry>
221 <entry>&acro.idxpath;</entry>
222 <entry><literal>idxpath</literal></entry>
223 <entry>Hardwired &acro.xpath; like attribute set, only available for
224 indexing with the &acro.grs1; record model</entry>
225 <entry>deprecated</entry>
233 The use attributes (type 1) mappings the
234 predefined attribute sets are found in the
235 attribute set configuration files <filename>tab/*.att</filename>.
240 The &zebra; internal query processing is modeled after
241 the &acro.bib1; attribute set, and the non-use
242 attributes type 2-6 are hard-wired in. It is therefore essential
243 to be familiar with <xref linkend="querymodel-bib1-nonuse"/>.
249 <section id="querymodel-boolean-operators">
250 <title>Boolean operators</title>
252 A pair of sub query trees, or of atomic queries, is combined
253 using the standard boolean operators into new query trees.
254 Thus, boolean operators are always internal nodes in the query tree.
257 <table id="querymodel-boolean-operators-table" frame="top">
258 <title>Boolean operators</title>
262 <entry>Keyword</entry>
263 <entry>Operator</entry>
264 <entry>Description</entry>
268 <row><entry><literal>@and</literal></entry>
269 <entry>binary AND operator</entry>
270 <entry>Set intersection of two atomic queries hit sets</entry>
272 <row><entry><literal>@or</literal></entry>
273 <entry>binary OR operator</entry>
274 <entry>Set union of two atomic queries hit sets</entry>
276 <row><entry><literal>@not</literal></entry>
277 <entry>binary AND NOT operator</entry>
278 <entry>Set complement of two atomic queries hit sets</entry>
280 <row><entry><literal>@prox</literal></entry>
281 <entry>binary PROXIMITY operator</entry>
282 <entry>Set intersection of two atomic queries hit sets. In
283 addition, the intersection set is purged for all
284 documents which do not satisfy the requested query
285 term proximity. Usually a proper subset of the AND
293 For example, we can combine the terms
294 <emphasis>information</emphasis> and <emphasis>retrieval</emphasis>
295 into different searches in the default index of the default
296 attribute set as follows.
297 Querying for the union of all documents containing the
298 terms <emphasis>information</emphasis> OR
299 <emphasis>retrieval</emphasis>:
301 Z> find @or information retrieval
305 Querying for the intersection of all documents containing the
306 terms <emphasis>information</emphasis> AND
307 <emphasis>retrieval</emphasis>:
308 The hit set is a subset of the corresponding
311 Z> find @and information retrieval
315 Querying for the intersection of all documents containing the
316 terms <emphasis>information</emphasis> AND
317 <emphasis>retrieval</emphasis>, taking proximity into account:
318 The hit set is a subset of the corresponding
320 (see the <ulink url="&url.yaz.pqf;">&acro.pqf; grammar</ulink> for
321 details on the proximity operator):
323 Z> find @prox 0 3 0 2 k 2 information retrieval
327 Querying for the intersection of all documents containing the
328 terms <emphasis>information</emphasis> AND
329 <emphasis>retrieval</emphasis>, in the same order and near each
330 other as described in the term list.
331 The hit set is a subset of the corresponding
334 Z> find "information retrieval"
340 <section id="querymodel-atomic-queries">
341 <title>Atomic queries (&acro.apt;)</title>
343 Atomic queries are the query parts which work on one access point
344 only. These consist of <emphasis>an attribute list</emphasis>
345 followed by a <emphasis>single term</emphasis> or a
346 <emphasis>quoted term list</emphasis>, and are often called
347 <emphasis>Attributes-Plus-Terms (&acro.apt;)</emphasis> queries.
350 Atomic (&acro.apt;) queries are always leaf nodes in the &acro.pqf; query tree.
351 UN-supplied non-use attributes types 2-12 are either inherited from
352 higher nodes in the query tree, or are set to &zebra;'s default values.
353 See <xref linkend="querymodel-bib1"/> for details.
356 <table id="querymodel-atomic-queries-table" frame="top">
357 <title>Atomic queries (&acro.apt;)</title>
368 <entry><emphasis>attribute list</emphasis></entry>
369 <entry>List of <emphasis>orthogonal</emphasis> attributes</entry>
370 <entry>Any of the orthogonal attribute types may be omitted,
371 these are inherited from higher query tree nodes, or if not
372 inherited, are set to the default &zebra; configuration values.
376 <entry><emphasis>term</emphasis></entry>
377 <entry>single <emphasis>term</emphasis>
378 or <emphasis>quoted term list</emphasis> </entry>
379 <entry>Here the search terms or list of search terms is added
386 Querying for the term <emphasis>information</emphasis> in the
387 default index using the default attribute set, the server choice
388 of access point/index, and the default non-use attributes.
394 Equivalent query fully specified including all default values:
396 Z> find @attrset bib-1 @attr 1=1017 @attr 2=3 @attr 3=3 @attr 4=1 @attr 5=100 @attr 6=1 information
401 Finding all documents which have the term
402 <emphasis>debussy</emphasis> in the title field.
404 Z> find @attr 1=4 debussy
409 The <emphasis>scan</emphasis> operation is only supported with
410 atomic &acro.apt; queries, as it is bound to one access point at a
411 time. Boolean query trees are not allowed during
412 <emphasis>scan</emphasis>.
416 For example, we might want to scan the title index, starting with
418 <emphasis>debussy</emphasis>, and displaying this and the
419 following terms in lexicographic order:
421 Z> scan @attr 1=4 debussy
427 <section id="querymodel-resultset">
428 <title>Named Result Sets</title>
430 Named result sets are supported in &zebra;, and result sets can be
431 used as operands without limitations. It follows that named
432 result sets are leaf nodes in the &acro.pqf; query tree, exactly as
433 atomic &acro.apt; queries are.
436 After the execution of a search, the result set is available at
437 the server, such that the client can use it for subsequent
438 searches or retrieval requests. The Z30.50 standard actually
439 stresses the fact that result sets are volatile. It may cease
440 to exist at any time point after search, and the server will
441 send a diagnostic to the effect that the requested
442 result set does not exist any more.
446 Defining a named result set and re-using it in the next query,
447 using <application>yaz-client</application>. Notice that the client, not
448 the server, assigns the string '1' to the
451 Z> f @attr 1=4 mozart
453 Number of hits: 43, setno 1
455 Z> f @and @set 1 @attr 1=4 amadeus
457 Number of hits: 14, setno 2
463 Named result sets are only supported by the &acro.z3950; protocol.
464 The &acro.sru; web service is stateless, and therefore the notion of
465 named result sets does not exist when accessing a &zebra; server by
466 the &acro.sru; protocol.
471 <section id="querymodel-use-string">
472 <title>&zebra;'s special access point of type 'string'</title>
474 The numeric <emphasis>use (type 1)</emphasis> attribute is usually
475 referred to from a given
476 attribute set. In addition, &zebra; let you use
477 <emphasis>any internal index
478 name defined in your configuration</emphasis>
479 as use attribute value. This is a great feature for
480 debugging, and when you do
481 not need the complexity of defined use attribute values. It is
482 the preferred way of accessing &zebra; indexes directly.
485 Finding all documents which have the term list "information
486 retrieval" in an &zebra; index, using its internal full string
487 name. Scanning the same index.
489 Z> find @attr 1=sometext "information retrieval"
490 Z> scan @attr 1=sometext aterm
494 Searching or scanning
495 the bib-1 use attribute 54 using its string name:
497 Z> find @attr 1=Code-language eng
498 Z> scan @attr 1=Code-language ""
502 It is possible to search
503 in any silly string index - if it's defined in your
504 indexation rules and can be parsed by the &acro.pqf; parser.
505 This is definitely not the recommended use of
506 this facility, as it might confuse your users with some very
509 Z> find @attr 1=silly/xpath/alike[@index]/name "information retrieval"
513 See also <xref linkend="querymodel-pqf-apt-mapping"/> for details, and
514 <xref linkend="zebrasrv-sru"/>
515 for the &acro.sru; &acro.pqf; query extension using string names as a fast
520 <section id="querymodel-use-xpath">
521 <title>&zebra;'s special access point of type 'XPath'
522 for &acro.grs1; filters</title>
524 As we have seen above, it is possible (albeit seldom a great
526 <ulink url="http://www.w3.org/TR/xpath">XPath 1.0</ulink> based
527 search by defining <emphasis>use (type 1)</emphasis>
528 <emphasis>string</emphasis> attributes which in appearance
529 <emphasis>resemble XPath queries</emphasis>. There are two
530 problems with this approach: first, the XPath-look-alike has to
531 be defined at indexation time, no new undefined
532 XPath queries can entered at search time, and second, it might
533 confuse users very much that an XPath-alike index name in fact
534 gets populated from a possible entirely different &acro.xml; element
535 than it pretends to access.
538 When using the &acro.grs1; Record Model
539 (see <xref linkend="grs"/>), we have the
540 possibility to embed <emphasis>life</emphasis>
542 in the &acro.pqf; queries, which are here called
543 <emphasis>use (type 1)</emphasis> <emphasis>xpath</emphasis>
544 attributes. You must enable the
545 <literal>xpath enable</literal> directive in your
546 <literal>.abs</literal> configuration files.
550 Only a <emphasis>very</emphasis> restricted subset of the
551 <ulink url="http://www.w3.org/TR/xpath">XPath 1.0</ulink>
552 standard is supported as the &acro.grs1; record model is simpler than
553 a full &acro.xml; &acro.dom; structure. See the following examples for
558 Finding all documents which have the term "content"
559 inside a text node found in a specific &acro.xml; &acro.dom;
560 <emphasis>subtree</emphasis>, whose starting element is
563 Z> find @attr 1=/root content
564 Z> find @attr 1=/root/first content
566 <emphasis>Notice that the
567 XPath must be absolute, i.e., must start with '/', and that the
568 XPath <literal>descendant-or-self</literal> axis followed by a
569 text node selection <literal>text()</literal> is implicitly
570 appended to the stated XPath.
572 It follows that the above searches are interpreted as:
574 Z> find @attr 1=/root//text() content
575 Z> find @attr 1=/root/first//text() content
580 Searching inside attribute strings is possible:
582 Z> find @attr 1=/link/@creator morten
587 Filter the addressing XPath by a predicate working on exact
589 attributes (in the &acro.xml; sense) can be done: return all those docs which
590 have the term "english" contained in one of all text sub nodes of
591 the subtree defined by the XPath
592 <literal>/record/title[@lang='en']</literal>. And similar
595 Z> find @attr 1=/record/title[@lang='en'] english
596 Z> find @attr 1=/link[@creator='sisse'] sibelius
597 Z> find @attr 1=/link[@creator='sisse']/description[@xml:lang='da'] sibelius
602 Combining numeric indexes, boolean expressions,
603 and xpath based searches is possible:
605 Z> find @attr 1=/record/title @and foo bar
606 Z> find @and @attr 1=/record/title foo @attr 1=4 bar
610 Escaping &acro.pqf; keywords and other non-parseable XPath constructs
611 with <literal>'{ }'</literal> to prevent client-side &acro.pqf; parsing
614 Z> find @attr {1=/root/first[@attr='danish']} content
615 Z> find @attr {1=/record/@set} oai
620 It is worth mentioning that these dynamic performed XPath
621 queries are a performance bottleneck, as no optimized
622 specialized indexes can be used. Therefore, avoid the use of
623 this facility when speed is essential, and the database content
624 size is medium to large.
630 <section id="querymodel-exp1">
631 <title>Explain Attribute Set</title>
633 The &acro.z3950; standard defines the
634 <ulink url="&url.z39.50.explain;">Explain</ulink> attribute set
635 Exp-1, which is used to discover information
636 about a server's search semantics and functional capabilities
637 &zebra; exposes a "classic"
638 Explain database by base name <literal>IR-Explain-1</literal>, which
639 is populated with system internal information.
642 The attribute-set <literal>exp-1</literal> consists of a single
643 use attribute (type 1).
646 In addition, the non-Use
647 &acro.bib1; attributes, that is, the types
648 <emphasis>Relation</emphasis>, <emphasis>Position</emphasis>,
649 <emphasis>Structure</emphasis>, <emphasis>Truncation</emphasis>,
650 and <emphasis>Completeness</emphasis> are imported from
651 the &acro.bib1; attribute set, and may be used
652 within any explain query.
655 <section id="querymodel-exp1-use">
656 <title>Use Attributes (type = 1)</title>
658 The following Explain search attributes are supported:
659 <literal>ExplainCategory</literal> (@attr 1=1),
660 <literal>DatabaseName</literal> (@attr 1=3),
661 <literal>DateAdded</literal> (@attr 1=9),
662 <literal>DateChanged</literal>(@attr 1=10).
665 A search in the use attribute <literal>ExplainCategory</literal>
666 supports only these predefined values:
667 <literal>CategoryList</literal>, <literal>TargetInfo</literal>,
668 <literal>DatabaseInfo</literal>, <literal>AttributeDetails</literal>.
671 See <filename>tab/explain.att</filename> and the
672 <ulink url="&url.z39.50;">&acro.z3950;</ulink> standard
673 for more information.
677 <section id="querymodel-examples">
678 <title>Explain searches with yaz-client</title>
680 Classic Explain only defines retrieval of Explain information
681 via ASN.1. Practically no &acro.z3950; clients supports this. Fortunately
682 they don't have to - &zebra; allows retrieval of this information
684 <literal>&acro.sutrs;</literal>, <literal>&acro.xml;</literal>,
685 <literal>&acro.grs1;</literal> and <literal>ASN.1</literal> Explain.
689 List supported categories to find out which explain commands are
693 Z> find @attr exp1 1=1 categorylist
700 Get target info, that is, investigate which databases exist at
701 this server endpoint:
704 Z> find @attr exp1 1=1 targetinfo
715 List all supported databases, the number of hits
716 is the number of databases found, which most commonly are the
718 the <literal>Default</literal> and the
719 <literal>IR-Explain-1</literal> databases.
722 Z> find @attr exp1 1=1 databaseinfo
729 Get database info record for database <literal>Default</literal>.
732 Z> find @and @attr exp1 1=1 databaseinfo @attr exp1 1=3 Default
734 Identical query with explicitly specified attribute set:
737 Z> find @attrset exp1 @and @attr 1=1 databaseinfo @attr 1=3 Default
742 Get attribute details record for database
743 <literal>Default</literal>.
744 This query is very useful to study the internal &zebra; indexes.
745 If records have been indexed using the <literal>alvis</literal>
746 &acro.xslt; filter, the string representation names of the known indexes can be
750 Z> find @and @attr exp1 1=1 attributedetails @attr exp1 1=3 Default
752 Identical query with explicitly specified attribute set:
755 Z> find @attrset exp1 @and @attr 1=1 attributedetails @attr 1=3 Default
762 <section id="querymodel-bib1">
763 <title>&acro.bib1; Attribute Set</title>
765 Most of the information contained in this section is an excerpt of
766 the ATTRIBUTE SET &acro.bib1; (&acro.z3950;-1995) SEMANTICS
767 found at <ulink url="&url.z39.50.attset.bib1.1995;">. The &acro.bib1;
768 Attribute Set Semantics</ulink> from 1995, also in an updated
769 <ulink url="&url.z39.50.attset.bib1;">&acro.bib1;
770 Attribute Set</ulink>
771 version from 2003. Index Data is not the copyright holder of this
772 information, except for the configuration details, the listing of
773 &zebra;'s capabilities, and the example queries.
777 <section id="querymodel-bib1-use">
778 <title>Use Attributes (type 1)</title>
781 A use attribute specifies an access point for any atomic query.
782 These access points are highly dependent on the attribute set used
783 in the query, and are user configurable using the following
784 default configuration files:
785 <filename>tab/bib1.att</filename>,
786 <filename>tab/dan1.att</filename>,
787 <filename>tab/explain.att</filename>, and
788 <filename>tab/gils.att</filename>.
791 For example, some few &acro.bib1; use
792 attributes from the <filename>tab/bib1.att</filename> are:
796 att 3 Conference-name
799 att 1009 Subject-name-personal
800 att 1010 Body-of-text
801 att 1011 Date/time-added-to-db
804 att 1017 Server-choice
808 att 1036 Author-Title-Subject
812 New attribute sets can be added by adding new
813 <filename>tab/*.att</filename> configuration files, which need to
814 be sourced in the main configuration <filename>zebra.cfg</filename>.
817 In addition, &zebra; allows the access of
818 <emphasis>internal index names</emphasis> and <emphasis>dynamic
819 XPath</emphasis> as use attributes; see
820 <xref linkend="querymodel-use-string"/> and
821 <xref linkend="querymodel-use-xpath"/>.
825 Phrase search for <emphasis>information retrieval</emphasis> in
826 the title-register, scanning the same register afterwards:
828 Z> find @attr 1=4 "information retrieval"
829 Z> scan @attr 1=4 information
837 <section id="querymodel-bib1-nonuse">
838 <title>&zebra; general Bib1 Non-Use Attributes (type 2-6)</title>
840 <section id="querymodel-bib1-relation">
841 <title>Relation Attributes (type 2)</title>
844 Relation attributes describe the relationship of the access
846 of the relation) to the search term as qualified by the attributes (right
847 side of the relation), e.g., Date-publication <= 1975.
850 <table id="querymodel-bib1-relation-table" frame="top">
851 <title>Relation Attributes (type 2)</title>
855 <entry>Relation</entry>
862 <entry>Less than</entry>
864 <entry>supported</entry>
867 <entry>Less than or equal</entry>
869 <entry>supported</entry>
874 <entry>default</entry>
877 <entry>Greater or equal</entry>
879 <entry>supported</entry>
882 <entry>Greater than</entry>
884 <entry>supported</entry>
887 <entry>Not equal</entry>
889 <entry>unsupported</entry>
892 <entry>Phonetic</entry>
894 <entry>unsupported</entry>
899 <entry>unsupported</entry>
902 <entry>Relevance</entry>
904 <entry>supported</entry>
907 <entry>AlwaysMatches</entry>
909 <entry>supported *</entry>
916 AlwaysMatches searches are only supported if alwaysmatches indexing
917 has been enabled. See <xref linkend="default-idx-file"/>
922 The relation attributes 1-5 are supported and work exactly as
924 All ordering operations are based on a lexicographical ordering,
925 <emphasis>except</emphasis> when the
926 structure attribute numeric (109) is used. In
927 this case, ordering is numerical. See
928 <xref linkend="querymodel-bib1-structure"/>.
930 Z> find @attr 1=Title @attr 2=1 music
932 Number of hits: 11745, setno 1
934 Z> find @attr 1=Title @attr 2=2 music
936 Number of hits: 11771, setno 2
938 Z> find @attr 1=Title @attr 2=3 music
940 Number of hits: 532, setno 3
942 Z> find @attr 1=Title @attr 2=4 music
944 Number of hits: 11463, setno 4
946 Z> find @attr 1=Title @attr 2=5 music
948 Number of hits: 11419, setno 5
953 The relation attribute
954 <emphasis>Relevance (102)</emphasis> is supported, see
955 <xref linkend="administration-ranking"/> for full information.
959 Ranked search for <emphasis>information retrieval</emphasis> in
962 Z> find @attr 1=4 @attr 2=102 "information retrieval"
967 The relation attribute
968 <emphasis>AlwaysMatches (103)</emphasis> is in the default
970 supported in conjecture with structure attribute
971 <emphasis>Phrase (1)</emphasis> (which may be omitted by
973 It can be configured to work with other structure attributes,
974 see the configuration file
975 <filename>tab/default.idx</filename> and
976 <xref linkend="querymodel-pqf-apt-mapping"/>.
979 <emphasis>AlwaysMatches (103)</emphasis> is a
980 great way to discover how many documents have been indexed in a
981 given field. The search term is ignored, but needed for correct
982 &acro.pqf; syntax. An empty search term may be supplied.
984 Z> find @attr 1=Title @attr 2=103 ""
985 Z> find @attr 1=Title @attr 2=103 @attr 4=1 ""
992 <section id="querymodel-bib1-position">
993 <title>Position Attributes (type 3)</title>
996 The position attribute specifies the location of the search term
997 within the field or subfield in which it appears.
1000 <table id="querymodel-bib1-position-table" frame="top">
1001 <title>Position Attributes (type 3)</title>
1005 <entry>Position</entry>
1006 <entry>Value</entry>
1007 <entry>Notes</entry>
1012 <entry>First in field </entry>
1014 <entry>supported *</entry>
1017 <entry>First in subfield</entry>
1019 <entry>supported *</entry>
1022 <entry>Any position in field</entry>
1024 <entry>default</entry>
1032 &zebra; only supports first-in-field seaches if the
1033 <literal>firstinfield</literal> is enabled for the index
1034 Refer to <xref linkend="default-idx-file"/>.
1035 &zebra; does not distinguish between first in field and
1036 first in subfield. They result in the same hit count.
1037 Searching for first position in (sub)field in only supported in &zebra;
1043 <section id="querymodel-bib1-structure">
1044 <title>Structure Attributes (type 4)</title>
1047 The structure attribute specifies the type of search
1048 term. This causes the search to be mapped on
1049 different &zebra; internal indexes, which must have been defined
1054 The possible values of the
1055 <literal>structure attribute (type 4)</literal> can be defined
1056 using the configuration file <filename>
1057 tab/default.idx</filename>.
1058 The default configuration is summarized in this table.
1061 <table id="querymodel-bib1-structure-table" frame="top">
1062 <title>Structure Attributes (type 4)</title>
1066 <entry>Structure</entry>
1067 <entry>Value</entry>
1068 <entry>Notes</entry>
1073 <entry>Phrase </entry>
1075 <entry>default</entry>
1080 <entry>supported</entry>
1085 <entry>supported</entry>
1090 <entry>supported</entry>
1093 <entry>Date (normalized)</entry>
1095 <entry>supported</entry>
1098 <entry>Word list</entry>
1100 <entry>supported</entry>
1103 <entry>Date (un-normalized)</entry>
1105 <entry>unsupported</entry>
1108 <entry>Name (normalized) </entry>
1110 <entry>unsupported</entry>
1113 <entry>Name (un-normalized) </entry>
1115 <entry>unsupported</entry>
1118 <entry>Structure</entry>
1120 <entry>unsupported</entry>
1125 <entry>supported</entry>
1128 <entry>Free-form-text</entry>
1130 <entry>supported</entry>
1133 <entry>Document-text</entry>
1135 <entry>supported</entry>
1138 <entry>Local-number</entry>
1140 <entry>supported</entry>
1143 <entry>String</entry>
1145 <entry>unsupported</entry>
1148 <entry>Numeric string</entry>
1150 <entry>supported</entry>
1157 The structure attribute values
1158 <literal>Word list (6)</literal>
1159 is supported, and maps to the boolean <literal>AND</literal>
1160 combination of words supplied. The word list is useful when
1161 google-like bag-of-word queries need to be translated from a GUI
1162 query language to &acro.pqf;. For example, the following queries
1165 Z> find @attr 1=Title @attr 4=6 "mozart amadeus"
1166 Z> find @attr 1=Title @and mozart amadeus
1171 The structure attribute value
1172 <literal>Free-form-text (105)</literal> and
1173 <literal>Document-text (106)</literal>
1174 are supported, and map both to the boolean <literal>OR</literal>
1175 combination of words supplied. The following queries
1178 Z> find @attr 1=Body-of-text @attr 4=105 "bach salieri teleman"
1179 Z> find @attr 1=Body-of-text @attr 4=106 "bach salieri teleman"
1180 Z> find @attr 1=Body-of-text @or bach @or salieri teleman
1182 This <literal>OR</literal> list of terms is very useful in
1183 combination with relevance ranking:
1185 Z> find @attr 1=Body-of-text @attr 2=102 @attr 4=105 "bach salieri teleman"
1190 The structure attribute value
1191 <literal>Local number (107)</literal>
1192 is supported, and maps always to the &zebra; internal document ID,
1193 irrespectively which use attribute is specified. The following queries
1194 have exactly the same unique record in the hit set:
1196 Z> find @attr 4=107 10
1197 Z> find @attr 1=4 @attr 4=107 10
1198 Z> find @attr 1=1010 @attr 4=107 10
1204 the GILS schema (<literal>gils.abs</literal>), the
1205 west-bounding-coordinate is indexed as type <literal>n</literal>,
1206 and is therefore searched by specifying
1207 <emphasis>structure</emphasis>=<emphasis>Numeric String</emphasis>.
1208 To match all those records with west-bounding-coordinate greater
1209 than -114 we use the following query:
1211 Z> find @attr 4=109 @attr 2=5 @attr gils 1=2038 -114
1216 The exact mapping between &acro.pqf; queries and &zebra; internal indexes
1217 and index types is explained in
1218 <xref linkend="querymodel-pqf-apt-mapping"/>.
1223 <section id="querymodel-bib1-truncation">
1224 <title>Truncation Attributes (type = 5)</title>
1227 The truncation attribute specifies whether variations of one or
1228 more characters are allowed between search term and hit terms, or
1229 not. Using non-default truncation attributes will broaden the
1230 document hit set of a search query.
1233 <table id="querymodel-bib1-truncation-table" frame="top">
1234 <title>Truncation Attributes (type 5)</title>
1238 <entry>Truncation</entry>
1239 <entry>Value</entry>
1240 <entry>Notes</entry>
1245 <entry>Right truncation </entry>
1247 <entry>supported</entry>
1250 <entry>Left truncation</entry>
1252 <entry>supported</entry>
1255 <entry>Left and right truncation</entry>
1257 <entry>supported</entry>
1260 <entry>Do not truncate</entry>
1262 <entry>default</entry>
1265 <entry>Process # in search term</entry>
1267 <entry>supported</entry>
1270 <entry>RegExpr-1 </entry>
1272 <entry>supported</entry>
1275 <entry>RegExpr-2</entry>
1277 <entry>supported</entry>
1284 The truncation attribute values 1-3 perform the obvious way:
1286 Z> scan @attr 1=Body-of-text schnittke
1292 Z> find @attr 1=Body-of-text @attr 5=1 schnittke
1294 Number of hits: 95, setno 7
1296 Z> find @attr 1=Body-of-text @attr 5=2 schnittke
1298 Number of hits: 81, setno 6
1300 Z> find @attr 1=Body-of-text @attr 5=3 schnittke
1302 Number of hits: 95, setno 8
1307 The truncation attribute value
1308 <literal>Process # in search term (101)</literal> is a
1309 poor-man's regular expression search. It maps
1310 each <literal>#</literal> to <literal>.*</literal>, and
1311 performs then a <literal>Regexp-1 (102)</literal> regular
1312 expression search. The following two queries are equivalent:
1314 Z> find @attr 1=Body-of-text @attr 5=101 schnit#ke
1315 Z> find @attr 1=Body-of-text @attr 5=102 schnit.*ke
1317 Number of hits: 89, setno 10
1322 The truncation attribute value
1323 <literal>Regexp-1 (102)</literal> is a normal regular search,
1324 see <xref linkend="querymodel-regular"/> for details.
1326 Z> find @attr 1=Body-of-text @attr 5=102 schnit+ke
1327 Z> find @attr 1=Body-of-text @attr 5=102 schni[a-t]+ke
1332 The truncation attribute value
1333 <literal>Regexp-2 (103) </literal> is a &zebra; specific extension
1334 which allows <emphasis>fuzzy</emphasis> matches. One single
1335 error in spelling of search terms is allowed, i.e., a document
1336 is hit if it includes a term which can be mapped to the used
1337 search term by one character substitution, addition, deletion or
1340 Z> find @attr 1=Body-of-text @attr 5=100 schnittke
1342 Number of hits: 81, setno 14
1344 Z> find @attr 1=Body-of-text @attr 5=103 schnittke
1346 Number of hits: 103, setno 15
1352 <section id="querymodel-bib1-completeness">
1353 <title>Completeness Attributes (type = 6)</title>
1357 The <literal>Completeness Attributes (type = 6)</literal>
1358 is used to specify that a given search term or term list is either
1359 part of the terms of a given index/field
1360 (<literal>Incomplete subfield (1)</literal>), or is
1361 what literally is found in the entire field's index
1362 (<literal>Complete field (3)</literal>).
1365 <table id="querymodel-bib1-completeness-table" frame="top">
1366 <title>Completeness Attributes (type = 6)</title>
1370 <entry>Completeness</entry>
1371 <entry>Value</entry>
1372 <entry>Notes</entry>
1377 <entry>Incomplete subfield</entry>
1379 <entry>default</entry>
1382 <entry>Complete subfield</entry>
1384 <entry>deprecated</entry>
1387 <entry>Complete field</entry>
1389 <entry>supported</entry>
1396 The <literal>Completeness Attributes (type = 6)</literal>
1397 is only partially and conditionally
1398 supported in the sense that it is ignored if the hit index is
1399 not of structure <literal>type="w"</literal> or
1400 <literal>type="p"</literal>.
1403 <literal>Incomplete subfield (1)</literal> is the default, and
1405 register <literal>type="w"</literal>, whereas
1406 <literal>Complete field (3)</literal> triggers
1407 search and scan in index <literal>type="p"</literal>.
1410 The <literal>Complete subfield (2)</literal> is a reminiscens
1411 from the happy <literal>&acro.marc;</literal>
1412 binary format days. &zebra; does not support it, but maps silently
1413 to <literal>Complete field (3)</literal>.
1418 The exact mapping between &acro.pqf; queries and &zebra; internal indexes
1419 and index types is explained in
1420 <xref linkend="querymodel-pqf-apt-mapping"/>.
1429 <section id="querymodel-zebra">
1430 <title>Extended &zebra; &acro.rpn; Features</title>
1432 The &zebra; internal query engine has been extended to specific needs
1433 not covered by the <literal>bib-1</literal> attribute set query
1434 model. These extensions are <emphasis>non-standard</emphasis>
1435 and <emphasis>non-portable</emphasis>: most functional extensions
1436 are modeled over the <literal>bib-1</literal> attribute set,
1437 defining type 7 and higher values.
1438 There are also the special
1439 <literal>string</literal> type index names for the
1440 <literal>idxpath</literal> attribute set.
1443 <section id="querymodel-zebra-attr-allrecords">
1444 <title>&zebra; specific retrieval of all records</title>
1446 &zebra; defines a hardwired <literal>string</literal> index name
1447 called <literal>_ALLRECORDS</literal>. It matches any record
1448 contained in the database, if used in conjunction with
1449 the relation attribute
1450 <literal>AlwaysMatches (103)</literal>.
1453 The <literal>_ALLRECORDS</literal> index name is used for total database
1454 export. The search term is ignored, it may be empty.
1456 Z> find @attr 1=_ALLRECORDS @attr 2=103 ""
1460 Combination with other index types can be made. For example, to
1461 find all records which are <emphasis>not</emphasis> indexed in
1462 the <literal>Title</literal> register, issue one of the two
1465 Z> find @not @attr 1=_ALLRECORDS @attr 2=103 "" @attr 1=Title @attr 2=103 ""
1466 Z> find @not @attr 1=_ALLRECORDS @attr 2=103 "" @attr 1=4 @attr 2=103 ""
1471 The special string index <literal>_ALLRECORDS</literal> is
1472 experimental, and the provided functionality and syntax may very
1473 well change in future releases of &zebra;.
1478 <section id="querymodel-zebra-attr-search">
1479 <title>&zebra; specific Search Extensions to all Attribute Sets</title>
1481 &zebra; extends the &acro.bib1; attribute types, and these extensions are
1482 recognized regardless of attribute
1483 set used in a <literal>search</literal> operation query.
1486 <table id="querymodel-zebra-attr-search-table" frame="top">
1487 <title>&zebra; Search Attribute Extensions</title>
1492 <entry>Value</entry>
1493 <entry>Operation</entry>
1494 <entry>&zebra; version</entry>
1499 <entry>Embedded Sort</entry>
1501 <entry>search</entry>
1505 <entry>Term Set</entry>
1507 <entry>search</entry>
1511 <entry>Rank Weight</entry>
1513 <entry>search</entry>
1517 <entry>Term Reference</entry>
1519 <entry>search</entry>
1523 <entry>Local Approx Limit</entry>
1525 <entry>search</entry>
1529 <entry>Global Approx Limit</entry>
1531 <entry>search</entry>
1532 <entry>2.0.8</entry>
1535 <entry>Maximum number of truncated terms (truncmax)</entry>
1537 <entry>search</entry>
1538 <entry>2.0.10</entry>
1542 Specifies whether un-indexed fields should be ignored.
1543 A zero value (default) throws a diagnostic when an un-indexed
1544 field is specified. A non-zero value makes it return 0 hits.
1547 <entry>search</entry>
1548 <entry>2.0.16</entry>
1554 <section id="querymodel-zebra-attr-sorting">
1555 <title>&zebra; Extension Embedded Sort Attribute (type 7)</title>
1557 The embedded sort is a way to specify sort within a query - thus
1558 removing the need to send a Sort Request separately. It is both
1559 faster and does not require clients to deal with the Sort
1564 All ordering operations are based on a lexicographical ordering,
1565 <emphasis>except</emphasis> when the
1566 <literal>structure attribute numeric (109)</literal> is used. In
1567 this case, ordering is numerical. See
1568 <xref linkend="querymodel-bib1-structure"/>.
1572 The possible values after attribute <literal>type 7</literal> are
1573 <literal>1</literal> ascending and
1574 <literal>2</literal> descending.
1575 The attributes+term (&acro.apt;) node is separate from the
1576 rest and must be <literal>@or</literal>'ed.
1577 The term associated with &acro.apt; is the sorting level in integers,
1578 where <literal>0</literal> means primary sort,
1579 <literal>1</literal> means secondary sort, and so forth.
1580 See also <xref linkend="administration-ranking"/>.
1583 For example, searching for water, sort by title (ascending)
1585 Z> find @or @attr 1=1016 water @attr 7=1 @attr 1=4 0
1589 Or, searching for water, sort by title ascending, then date descending
1591 Z> find @or @or @attr 1=1016 water @attr 7=1 @attr 1=4 0 @attr 7=2 @attr 1=30 1
1597 &zebra; Extension Term Set Attribute
1598 From the manual text, I can not see what is the point with this feature.
1599 I think it makes more sense when there are multiple terms in a query, or
1602 We decided 2006-06-03 to disable this feature, as it is covered by
1603 scan within a resultset. Better use ressources to upgrade this
1604 feature for good performance.
1608 <section id="querymodel-zebra-attr-estimation">
1609 <title>&zebra; Extension Term Set Attribute (type 8)</title>
1611 The Term Set feature is a facility that allows a search to store
1612 hitting terms in a "pseudo" resultset; thus a search (as usual) +
1613 a scan-like facility. Requires a client that can do named result
1614 sets since the search generates two result sets. The value for
1615 attribute 8 is the name of a result set (string). The terms in
1616 the named term set are returned as &acro.sutrs; records.
1619 For example, searching for u in title, right truncated, and
1620 storing the result in term set named 'aset'
1622 Z> find @attr 5=1 @attr 1=4 @attr 8=aset u
1626 The model has one serious flaw: we don't know the size of term
1627 set. Experimental. Do not use in production code.
1633 <section id="querymodel-zebra-attr-weight">
1634 <title>&zebra; Extension Rank Weight Attribute (type 9)</title>
1636 Rank weight is a way to pass a value to a ranking algorithm - so
1637 that one &acro.apt; has one value - while another as a different one.
1638 See also <xref linkend="administration-ranking"/>.
1641 For example, searching for utah in title with weight 30 as well
1642 as any with weight 20:
1644 Z> find @attr 2=102 @or @attr 9=30 @attr 1=4 utah @attr 9=20 utah
1649 <section id="querymodel-zebra-attr-termref">
1650 <title>&zebra; Extension Term Reference Attribute (type 10)</title>
1652 &zebra; supports the searchResult-1 facility.
1653 If the Term Reference Attribute (type 10) is
1654 given, that specifies a subqueryId value returned as part of the
1655 search result. It is a way for a client to name an &acro.apt; part of a
1666 Experimental. Do not use in production code.
1674 <section id="querymodel-zebra-local-attr-limit">
1675 <title>Local Approximative Limit Attribute (type 11)</title>
1677 &zebra; computes - unless otherwise configured -
1678 the exact hit count for every &acro.apt;
1679 (leaf) in the query tree. These hit counts are returned as part of
1680 the searchResult-1 facility in the binary encoded &acro.z3950; search
1684 By setting an estimation limit size of the resultset of the &acro.apt;
1685 leaves, &zebra; stoppes processing the result set when the limit
1687 Hit counts under this limit are still precise, but hit counts over it
1688 are estimated using the statistics gathered from the chopped
1692 Specifying a limit of <literal>0</literal> resuts in exact hit counts.
1695 For example, we might be interested in exact hit count for a, but
1696 for b we allow hit count estimates for 1000 and higher.
1698 Z> find @and a @attr 11=1000 b
1703 The estimated hit count facility makes searches faster, as one
1704 only needs to process large hit lists partially.
1705 It is mostly used in huge databases, where you you want trade
1706 exactness of hit counts against speed of execution.
1711 Do not use approximative hit count limits
1712 in conjunction with relevance ranking, as re-sorting of the
1713 result set only works when the entire result set has
1719 <section id="querymodel-zebra-global-attr-limit">
1720 <title>Global Approximative Limit Attribute (type 12)</title>
1722 By default &zebra; computes precise hit counts for a query as
1723 a whole. Setting attribute 12 makes it perform approximative
1724 hit counts instead. It has the same semantics as
1725 <literal>estimatehits</literal> for the <xref linkend="zebra-cfg"/>.
1728 The attribute (12) can occur anywhere in the query tree.
1729 Unlike regular attributes it does not relate to the leaf (&acro.apt;)
1730 - but to the whole query.
1734 Do not use approximative hit count limits
1735 in conjunction with relevance ranking, as re-sorting of the
1736 result set only works when the entire result set has
1744 <section id="querymodel-zebra-attr-scan">
1745 <title>&zebra; specific Scan Extensions to all Attribute Sets</title>
1747 &zebra; extends the Bib1 attribute types, and these extensions are
1748 recognized regardless of attribute
1749 set used in a scan operation query.
1751 <table id="querymodel-zebra-attr-scan-table" frame="top">
1752 <title>&zebra; Scan Attribute Extensions</title>
1758 <entry>Operation</entry>
1759 <entry>&zebra; version</entry>
1764 <entry>Result Set Narrow</entry>
1770 <entry>Approximative Limit</entry>
1779 <section id="querymodel-zebra-attr-narrow">
1780 <title>&zebra; Extension Result Set Narrow (type 8)</title>
1782 If attribute Result Set Narrow (type 8)
1783 is given for scan, the value is the name of a
1784 result set. Each hit count in scan is
1785 <literal>@and</literal>'ed with the result set given.
1788 Consider for example
1789 the case of scanning all title fields around the
1790 scanterm <emphasis>mozart</emphasis>, then refining the scan by
1791 issuing a filtering query for <emphasis>amadeus</emphasis> to
1792 restrict the scan to the result set of the query:
1794 Z> scan @attr 1=4 mozart
1797 mozartforskningen (1)
1801 Z> f @attr 1=4 amadeus
1803 Number of hits: 15, setno 2
1805 Z> scan @attr 1=4 @attr 8=2 mozart
1808 mozartforskningen (0)
1816 &zebra; 2.0.2 and later is able to skip 0 hit counts. This, however,
1817 is known not to scale if the number of terms to skip is high.
1818 This most likely will happen if the result set is small (and
1819 result in many 0 hits).
1823 <section id="querymodel-zebra-attr-approx">
1824 <title>&zebra; Extension Approximative Limit (type 11)</title>
1826 The &zebra; Extension Approximative Limit (type 11) is a way to
1827 enable approximate hit counts for scan hit counts, in the same
1828 way as for search hit counts.
1833 <section id="querymodel-idxpath">
1834 <title>&zebra; special &acro.idxpath; Attribute Set for &acro.grs1; indexing</title>
1836 The attribute-set <literal>idxpath</literal> consists of a single
1837 Use (type 1) attribute. All non-use attributes behave as normal.
1840 This feature is enabled when defining the
1841 <literal>xpath enable</literal> option in the &acro.grs1; filter
1842 <filename>*.abs</filename> configuration files. If one wants to use
1843 the special <literal>idxpath</literal> numeric attribute set, the
1844 main &zebra; configuration file <filename>zebra.cfg</filename>
1845 directive <literal>attset: idxpath.att</literal> must be enabled.
1849 The <literal>idxpath</literal> is deprecated, may not be
1850 supported in future &zebra; versions, and should definitely
1851 not be used in production code.
1855 <section id="querymodel-idxpath-use">
1856 <title>&acro.idxpath; Use Attributes (type = 1)</title>
1858 This attribute set allows one to search &acro.grs1; filter indexed
1859 records by &acro.xpath; like structured index names.
1864 The <literal>idxpath</literal> option defines hard-coded
1865 index names, which might clash with your own index names.
1869 <table id="querymodel-idxpath-use-table" frame="top">
1870 <title>&zebra; specific &acro.idxpath; Use Attributes (type 1)</title>
1874 <entry>&acro.idxpath;</entry>
1875 <entry>Value</entry>
1876 <entry>String Index</entry>
1877 <entry>Notes</entry>
1882 <entry>&acro.xpath; Begin</entry>
1884 <entry>_XPATH_BEGIN</entry>
1885 <entry>deprecated</entry>
1888 <entry>&acro.xpath; End</entry>
1890 <entry>_XPATH_END</entry>
1891 <entry>deprecated</entry>
1894 <entry>&acro.xpath; CData</entry>
1896 <entry>_XPATH_CDATA</entry>
1897 <entry>deprecated</entry>
1900 <entry>&acro.xpath; Attribute Name</entry>
1902 <entry>_XPATH_ATTR_NAME</entry>
1903 <entry>deprecated</entry>
1906 <entry>&acro.xpath; Attribute CData</entry>
1908 <entry>_XPATH_ATTR_CDATA</entry>
1909 <entry>deprecated</entry>
1916 See <filename>tab/idxpath.att</filename> for more information.
1919 Search for all documents starting with root element
1920 <literal>/root</literal> (either using the numeric or the string
1923 Z> find @attrset idxpath @attr 1=1 @attr 4=3 root/
1924 Z> find @attr idxpath 1=1 @attr 4=3 root/
1925 Z> find @attr 1=_XPATH_BEGIN @attr 4=3 root/
1929 Search for all documents where specific nested &acro.xpath;
1930 <literal>/c1/c2/../cn</literal> exists. Notice the very
1931 counter-intuitive <emphasis>reverse</emphasis> notation!
1933 Z> find @attrset idxpath @attr 1=1 @attr 4=3 cn/cn-1/../c1/
1934 Z> find @attr 1=_XPATH_BEGIN @attr 4=3 cn/cn-1/../c1/
1938 Search for CDATA string <emphasis>text</emphasis> in any element
1940 Z> find @attrset idxpath @attr 1=1016 text
1941 Z> find @attr 1=_XPATH_CDATA text
1945 Search for CDATA string <emphasis>anothertext</emphasis> in any
1948 Z> find @attrset idxpath @attr 1=1015 anothertext
1949 Z> find @attr 1=_XPATH_ATTR_CDATA anothertext
1953 Search for all documents with have an &acro.xml; element node
1954 including an &acro.xml; attribute named <emphasis>creator</emphasis>
1956 Z> find @attrset idxpath @attr 1=3 @attr 4=3 creator
1957 Z> find @attr 1=_XPATH_ATTR_NAME @attr 4=3 creator
1961 Combining usual <literal>bib-1</literal> attribute set searches
1962 with <literal>idxpath</literal> attribute set searches:
1964 Z> find @and @attr idxpath 1=1 @attr 4=3 link/ @attr 1=4 mozart
1965 Z> find @and @attr 1=_XPATH_BEGIN @attr 4=3 link/ @attr 1=_XPATH_CDATA mozart
1969 Scanning is supported on all <literal>idxpath</literal>
1970 indexes, both specified as numeric use attributes, or as string
1973 Z> scan @attrset idxpath @attr 1=1016 text
1974 Z> scan @attr 1=_XPATH_ATTR_CDATA anothertext
1975 Z> scan @attrset idxpath @attr 1=3 @attr 4=3 ''
1983 <section id="querymodel-pqf-apt-mapping">
1984 <title>Mapping from &acro.pqf; atomic &acro.apt; queries to &zebra; internal
1985 register indexes</title>
1987 The rules for &acro.pqf; &acro.apt; mapping are rather tricky to grasp in the
1988 first place. We deal first with the rules for deciding which
1989 internal register or string index to use, according to the use
1990 attribute or access point specified in the query. Thereafter we
1991 deal with the rules for determining the correct structure type of
1995 <section id="querymodel-pqf-apt-mapping-accesspoint">
1996 <title>Mapping of &acro.pqf; &acro.apt; access points</title>
1998 &zebra; understands four fundamental different types of access
1999 points, of which only the
2000 <emphasis>numeric use attribute</emphasis> type access points
2001 are defined by the <ulink url="&url.z39.50;">&acro.z3950;</ulink>
2003 All other access point types are &zebra; specific, and non-portable.
2006 <table id="querymodel-zebra-mapping-accesspoint-types" frame="top">
2007 <title>Access point name mapping</title>
2011 <entry>Access Point</entry>
2013 <entry>Grammar</entry>
2014 <entry>Notes</entry>
2019 <entry>Use attribute</entry>
2020 <entry>numeric</entry>
2021 <entry>[1-9][1-9]*</entry>
2022 <entry>directly mapped to string index name</entry>
2025 <entry>String index name</entry>
2026 <entry>string</entry>
2027 <entry>[a-zA-Z](\-?[a-zA-Z0-9])*</entry>
2028 <entry>normalized name is used as internal string index name</entry>
2031 <entry>&zebra; internal index name</entry>
2032 <entry>zebra</entry>
2033 <entry>_[a-zA-Z](_?[a-zA-Z0-9])*</entry>
2034 <entry>hardwired internal string index name</entry>
2037 <entry>&acro.xpath; special index</entry>
2038 <entry>XPath</entry>
2040 <entry>special xpath search for &acro.grs1; indexed records</entry>
2047 <literal>Attribute set names</literal> and
2048 <literal>string index names</literal> are normalizes
2049 according to the following rules: all <emphasis>single</emphasis>
2050 hyphens <literal>'-'</literal> are stripped, and all upper case
2051 letters are folded to lower case.
2055 <emphasis>Numeric use attributes</emphasis> are mapped
2056 to the &zebra; internal
2057 string index according to the attribute set definition in use.
2058 The default attribute set is <literal>&acro.bib1;</literal>, and may be
2059 omitted in the &acro.pqf; query.
2063 According to normalization and numeric
2064 use attribute mapping, it follows that the following
2065 &acro.pqf; queries are considered equivalent (assuming the default
2066 configuration has not been altered):
2068 Z> find @attr 1=Body-of-text serenade
2069 Z> find @attr 1=bodyoftext serenade
2070 Z> find @attr 1=BodyOfText serenade
2071 Z> find @attr 1=bO-d-Y-of-tE-x-t serenade
2072 Z> find @attr 1=1010 serenade
2073 Z> find @attrset &acro.bib1; @attr 1=1010 serenade
2074 Z> find @attrset bib1 @attr 1=1010 serenade
2075 Z> find @attrset Bib1 @attr 1=1010 serenade
2076 Z> find @attrset b-I-b-1 @attr 1=1010 serenade
2081 The <emphasis>numerical</emphasis>
2082 <literal>use attributes (type 1)</literal>
2083 are interpreted according to the
2084 attribute sets which have been loaded in the
2085 <literal>zebra.cfg</literal> file, and are matched against specific
2086 fields as specified in the <literal>.abs</literal> file which
2087 describes the profile of the records which have been loaded.
2088 If no use attribute is provided, a default of
2089 &acro.bib1; Use Any (1016) is assumed.
2090 The predefined use attribute sets
2091 can be reconfigured by tweaking the configuration files
2092 <filename>tab/*.att</filename>, and
2093 new attribute sets can be defined by adding similar files in the
2094 configuration path <literal>profilePath</literal> of the server.
2098 String indexes can be accessed directly,
2099 independently which attribute set is in use. These are just
2100 ignored. The above mentioned name normalization applies.
2101 String index names are defined in the
2102 used indexing filter configuration files, for example in the
2103 <literal>&acro.grs1;</literal>
2104 <filename>*.abs</filename> configuration files, or in the
2105 <literal>alvis</literal> filter &acro.xslt; indexing stylesheets.
2109 &zebra; internal indexes can be accessed directly,
2110 according to the same rules as the user defined
2111 string indexes. The only difference is that
2112 &zebra; internal index names are hardwired,
2114 must start with the character <literal>'_'</literal>.
2118 Finally, <literal>&acro.xpath;</literal> access points are only
2119 available using the <literal>&acro.grs1;</literal> filter for indexing.
2120 These access point names must start with the character
2121 <literal>'/'</literal>, they are <emphasis>not
2122 normalized</emphasis>, but passed unaltered to the &zebra; internal
2123 &acro.xpath; engine. See <xref linkend="querymodel-use-xpath"/>.
2131 <section id="querymodel-pqf-apt-mapping-structuretype">
2132 <title>Mapping of &acro.pqf; &acro.apt; structure and completeness to
2133 register type</title>
2135 Internally &zebra; has in its default configuration several
2136 different types of registers or indexes, whose tokenization and
2137 character normalization rules differ. This reflects the fact that
2138 searching fundamental different tokens like dates, numbers,
2139 bitfields and string based text needs different rule sets.
2142 <table id="querymodel-zebra-mapping-structure-types" frame="top">
2143 <title>Structure and completeness mapping to register types</title>
2147 <entry>Structure</entry>
2148 <entry>Completeness</entry>
2149 <entry>Register type</entry>
2150 <entry>Notes</entry>
2156 phrase (@attr 4=1), word (@attr 4=2),
2157 word-list (@attr 4=6),
2158 free-form-text (@attr 4=105), or document-text (@attr 4=106)
2160 <entry>Incomplete field (@attr 6=1)</entry>
2161 <entry>Word ('w')</entry>
2162 <entry>Traditional tokenized and character normalized word index</entry>
2166 phrase (@attr 4=1), word (@attr 4=2),
2167 word-list (@attr 4=6),
2168 free-form-text (@attr 4=105), or document-text (@attr 4=106)
2170 <entry>complete field' (@attr 6=3)</entry>
2171 <entry>Phrase ('p')</entry>
2172 <entry>Character normalized, but not tokenized index for phrase
2177 <entry>urx (@attr 4=104)</entry>
2178 <entry>ignored</entry>
2179 <entry>URX/URL ('u')</entry>
2180 <entry>Special index for URL web addresses</entry>
2183 <entry>numeric (@attr 4=109)</entry>
2184 <entry>ignored</entry>
2185 <entry>Numeric ('n')</entry>
2186 <entry>Special index for digital numbers</entry>
2189 <entry>key (@attr 4=3)</entry>
2190 <entry>ignored</entry>
2191 <entry>Null bitmap ('0')</entry>
2192 <entry>Used for non-tokenizated and non-normalized bit sequences</entry>
2195 <entry>year (@attr 4=4)</entry>
2196 <entry>ignored</entry>
2197 <entry>Year ('y')</entry>
2198 <entry>Non-tokenizated and non-normalized 4 digit numbers</entry>
2201 <entry>date (@attr 4=5)</entry>
2202 <entry>ignored</entry>
2203 <entry>Date ('d')</entry>
2204 <entry>Non-tokenizated and non-normalized ISO date strings</entry>
2207 <entry>ignored</entry>
2208 <entry>ignored</entry>
2209 <entry>Sort ('s')</entry>
2210 <entry>Used with special sort attribute set (@attr 7=1, @attr 7=2)</entry>
2213 <entry>overruled</entry>
2214 <entry>overruled</entry>
2215 <entry>special</entry>
2216 <entry>Internal record ID register, used whenever
2217 Relation Always Matches (@attr 2=103) is specified</entry>
2223 <!-- see in util/zebramap.c -->
2226 If a <emphasis>Structure</emphasis> attribute of
2227 <emphasis>Phrase</emphasis> is used in conjunction with a
2228 <emphasis>Completeness</emphasis> attribute of
2229 <emphasis>Complete (Sub)field</emphasis>, the term is matched
2230 against the contents of the phrase (long word) register, if one
2231 exists for the given <emphasis>Use</emphasis> attribute.
2232 A phrase register is created for those fields in the
2233 &acro.grs1; <filename>*.abs</filename> file that contains a
2234 <literal>p</literal>-specifier.
2236 Z> scan @attr 1=Title @attr 4=1 @attr 6=3 beethoven
2238 bayreuther festspiele (1)
2239 * beethoven bibliography database (1)
2242 Z> find @attr 1=Title @attr 4=1 @attr 6=3 "beethoven bibliography"
2244 Number of hits: 0, setno 5
2246 Z> find @attr 1=Title @attr 4=1 @attr 6=3 "beethoven bibliography database"
2248 Number of hits: 1, setno 6
2253 If <emphasis>Structure</emphasis>=<emphasis>Phrase</emphasis> is
2254 used in conjunction with <emphasis>Incomplete Field</emphasis> - the
2255 default value for <emphasis>Completeness</emphasis>, the
2256 search is directed against the normal word registers, but if the term
2257 contains multiple words, the term will only match if all of the words
2258 are found immediately adjacent, and in the given order.
2259 The word search is performed on those fields that are indexed as
2260 type <literal>w</literal> in the &acro.grs1; <filename>*.abs</filename> file.
2262 Z> scan @attr 1=Title @attr 4=1 @attr 6=1 beethoven
2268 Z> find @attr 1=Title @attr 4=1 @attr 6=1 beethoven
2270 Number of hits: 18, setno 1
2272 Z> find @attr 1=Title @attr 4=1 @attr 6=1 "beethoven bibliography"
2274 Number of hits: 2, setno 2
2280 If the <emphasis>Structure</emphasis> attribute is
2281 <emphasis>Word List</emphasis>,
2282 <emphasis>Free-form Text</emphasis>, or
2283 <emphasis>Document Text</emphasis>, the term is treated as a
2284 natural-language, relevance-ranked query.
2285 This search type uses the word register, i.e. those fields
2286 that are indexed as type <literal>w</literal> in the
2287 &acro.grs1; <filename>*.abs</filename> file.
2291 If the <emphasis>Structure</emphasis> attribute is
2292 <emphasis>Numeric String</emphasis> the term is treated as an integer.
2293 The search is performed on those fields that are indexed
2294 as type <literal>n</literal> in the &acro.grs1;
2295 <filename>*.abs</filename> file.
2299 If the <emphasis>Structure</emphasis> attribute is
2300 <emphasis>URX</emphasis> the term is treated as a URX (URL) entity.
2301 The search is performed on those fields that are indexed as type
2302 <literal>u</literal> in the <filename>*.abs</filename> file.
2306 If the <emphasis>Structure</emphasis> attribute is
2307 <emphasis>Local Number</emphasis> the term is treated as
2308 native &zebra; Record Identifier.
2312 If the <emphasis>Relation</emphasis> attribute is
2313 <emphasis>Equals</emphasis> (default), the term is matched
2314 in a normal fashion (modulo truncation and processing of
2315 individual words, if required).
2316 If <emphasis>Relation</emphasis> is <emphasis>Less Than</emphasis>,
2317 <emphasis>Less Than or Equal</emphasis>,
2318 <emphasis>Greater than</emphasis>, or <emphasis>Greater than or
2319 Equal</emphasis>, the term is assumed to be numerical, and a
2320 standard regular expression is constructed to match the given
2322 If <emphasis>Relation</emphasis> is <emphasis>Relevance</emphasis>,
2323 the standard natural-language query processor is invoked.
2327 For the <emphasis>Truncation</emphasis> attribute,
2328 <emphasis>No Truncation</emphasis> is the default.
2329 <emphasis>Left Truncation</emphasis> is not supported.
2330 <emphasis>Process # in search term</emphasis> is supported, as is
2331 <emphasis>Regxp-1</emphasis>.
2332 <emphasis>Regxp-2</emphasis> enables the fault-tolerant (fuzzy)
2333 search. As a default, a single error (deletion, insertion,
2334 replacement) is accepted when terms are matched against the register
2341 <section id="querymodel-regular">
2342 <title>&zebra; Regular Expressions in Truncation Attribute (type = 5)</title>
2345 Each term in a query is interpreted as a regular expression if
2346 the truncation value is either <emphasis>Regxp-1 (@attr 5=102)</emphasis>
2347 or <emphasis>Regxp-2 (@attr 5=103)</emphasis>.
2348 Both query types follow the same syntax with the operands:
2351 <table id="querymodel-regular-operands-table" frame="top">
2352 <title>Regular Expression Operands</title>
2356 <entry><literal>x</literal></entry>
2357 <entry>Matches the character <literal>x</literal>.</entry>
2360 <entry><literal>.</literal></entry>
2361 <entry>Matches any character.</entry>
2364 <entry><literal>[ .. ]</literal></entry>
2365 <entry>Matches the set of characters specified;
2366 such as <literal>[abc]</literal> or <literal>[a-c]</literal>.</entry>
2373 The above operands can be combined with the following operators:
2376 <table id="querymodel-regular-operators-table" frame="top">
2377 <title>Regular Expression Operators</title>
2381 <entry><literal>x*</literal></entry>
2382 <entry>Matches <literal>x</literal> zero or more times.
2383 Priority: high.</entry>
2386 <entry><literal>x+</literal></entry>
2387 <entry>Matches <literal>x</literal> one or more times.
2388 Priority: high.</entry>
2391 <entry><literal>x?</literal></entry>
2392 <entry> Matches <literal>x</literal> zero or once.
2393 Priority: high.</entry>
2396 <entry><literal>xy</literal></entry>
2397 <entry> Matches <literal>x</literal>, then <literal>y</literal>.
2398 Priority: medium.</entry>
2401 <entry><literal>x|y</literal></entry>
2402 <entry> Matches either <literal>x</literal> or <literal>y</literal>.
2403 Priority: low.</entry>
2406 <entry><literal>( )</literal></entry>
2407 <entry>The order of evaluation may be changed by using parentheses.</entry>
2414 If the first character of the <literal>Regxp-2</literal> query
2415 is a plus character (<literal>+</literal>) it marks the
2416 beginning of a section with non-standard specifiers.
2417 The next plus character marks the end of the section.
2418 Currently &zebra; only supports one specifier, the error tolerance,
2419 which consists one digit.
2420 <!-- TODO Nice thing, but what does
2421 that error tolerance digit *mean*? Maybe an example would be nice? -->
2425 Since the plus operator is normally a suffix operator the addition to
2426 the query syntax doesn't violate the syntax for standard regular
2431 For example, a phrase search with regular expressions in
2432 the title-register is performed like this:
2434 Z> find @attr 1=4 @attr 5=102 "informat.* retrieval"
2439 Combinations with other attributes are possible. For example, a
2440 ranked search with a regular expression:
2442 Z> find @attr 1=4 @attr 5=102 @attr 2=102 "informat.* retrieval"
2450 The RecordType parameter in the <literal>zebra.cfg</literal> file, or
2451 the <literal>-t</literal> option to the indexer tells &zebra; how to
2452 process input records.
2453 Two basic types of processing are available - raw text and structured
2454 data. Raw text is just that, and it is selected by providing the
2455 argument <literal>text</literal> to &zebra;. Structured records are
2456 all handled internally using the basic mechanisms described in the
2457 subsequent sections.
2458 &zebra; can read structured records in many different formats.
2464 <section id="querymodel-cql-to-pqf">
2465 <title>Server Side &acro.cql; to &acro.pqf; Query Translation</title>
2468 <literal><cql2rpn>l2rpn.txt</cql2rpn></literal>
2469 &yaz; Frontend Virtual
2470 Hosts option, one can configure
2471 the &yaz; Frontend &acro.cql;-to-&acro.pqf;
2472 converter, specifying the interpretation of various
2473 <ulink url="&url.cql;">&acro.cql;</ulink>
2474 indexes, relations, etc. in terms of Type-1 query attributes.
2475 <!-- The yaz-client config file -->
2478 For example, using server-side &acro.cql;-to-&acro.pqf; conversion, one might
2479 query a zebra server like this:
2482 yaz-client localhost:9999
2484 Z> find text=(plant and soil)
2487 and - if properly configured - even static relevance ranking can
2488 be performed using &acro.cql; query syntax:
2491 Z> find text = /relevant (plant and soil)
2497 By the way, the same configuration can be used to
2498 search using client-side &acro.cql;-to-&acro.pqf; conversion:
2499 (the only difference is <literal>querytype cql2rpn</literal>
2501 <literal>querytype cql</literal>, and the call specifying a local
2505 yaz-client -q local/cql2pqf.txt localhost:9999
2506 Z> querytype cql2rpn
2507 Z> find text=(plant and soil)
2513 Exhaustive information can be found in the
2514 Section <ulink url="&url.yaz.cql2pqf;">&acro.cql; to &acro.rpn; conversion"</ulink>
2515 in the &yaz; manual.
2520 <ulink url="http://www.loc.gov/z3950/agency/zing/cql/dc-indexes.html"/>
2521 for the Maintenance Agency's work-in-progress mapping of Dublin Core
2522 indexes to Attribute Architecture (util, XD and BIB-2)
2530 <!-- Keep this comment at the end of the file
2535 sgml-minimize-attributes:nil
2536 sgml-always-quote-attributes:t
2539 sgml-parent-document: "zebra.xml"
2540 sgml-local-catalogs: nil
2541 sgml-namecase-general:t