1 <chapter id="record-model-domxml">
2 <!-- $Id: recordmodel-domxml.xml,v 1.8 2007-02-21 15:03:30 marc Exp $ -->
3 <title>&dom; &xml; Record Model and Filter Module</title>
6 The record model described in this chapter applies to the fundamental,
8 record type <literal>dom</literal>, introduced in
9 <xref linkend="componentmodulesdom"/>. The &dom; &xml; record model
10 is experimental, and it's inner workings might change in future
11 releases of the &zebra; Information Server.
16 <section id="record-model-domxml-filter">
17 <title>&dom; Record Filter Architecture</title>
20 The &dom; &xml; filter uses a standard &dom; &xml; structure as
21 internal data model, and can therefore parse, index, and display
22 any &xml; document type. It is wellsuited to work on
23 standardized &xml;-based formats such as Dublin Core, MODS, METS,
24 MARCXML, OAI-PMH, RSS, and performs equally well on any other
25 non-standard &xml; format.
28 A parser for binary &marc; records based on the ISO2709 library
29 standard is provided, it transforms these to the internal
30 &marcxml; &dom; representation. Other binary document parsers
31 are planned to follow.
35 The &dom; filter architecture consists of four
36 different pipelines, each being a chain of arbitraily many sucessive
37 &xslt; transformations of the internal &dom; &xml;
38 representations of documents.
41 <figure id="record-model-domxml-architecture-fig">
42 <title>&dom; &xml; filter architecture</title>
45 <imagedata fileref="domfilter.pdf" format="PDF" scale="50"/>
48 <imagedata fileref="domfilter.png" format="PNG"/>
51 <!-- Fall back if none of the images can be used -->
53 [Here there should be a diagram showing the &dom; &xml;
54 filter architecture, but is seems that your
55 tool chain has not been able to include the diagram in this
63 <table id="record-model-domxml-architecture-table" frame="top">
64 <title>&dom; &xml; filter pipelines overview</title>
70 <entry>Description</entry>
78 <entry><literal>input</literal></entry>
80 <entry>input parsing and initial
81 transformations to common &xml; format</entry>
82 <entry>Input raw &xml; record buffers, &xml; streams and
83 binary &marc; buffers</entry>
84 <entry>Common &xml; &dom;</entry>
87 <entry><literal>extract</literal></entry>
89 <entry>indexing term extraction
90 transformations</entry>
91 <entry>Common &xml; &dom;</entry>
92 <entry>Indexing &xml; &dom;</entry>
95 <entry><literal>store</literal></entry>
97 <entry> transformations before internal document
99 <entry>Common &xml; &dom;</entry>
100 <entry>Storage &xml; &dom;</entry>
103 <entry><literal>retrieve</literal></entry>
105 <entry>multiple document retrieve transformations from
106 storage to different output
107 formats are possible</entry>
108 <entry>Storage &xml; &dom;</entry>
109 <entry>Output &xml; syntax in requested formats</entry>
116 The &dom; &xml; filter pipelines use &xslt; (and if supported on
117 your platform, even &exslt;), it brings thus full &xpath;
118 support to the indexing, storage and display rules of not only
119 &xml; documents, but also binary &marc; records.
124 <section id="record-model-domxml-pipeline">
125 <title>&dom; &xml; filter pipeline configuration</title>
128 The experimental, loadable &dom; &xml;/&xslt; filter module
129 <literal>mod-dom.so</literal>
130 is invoked by the <filename>zebra.cfg</filename> configuration statement
132 recordtype.xml: dom.db/filter_dom_conf.xml
134 In this example the &dom; &xml; filter is configured to work
135 on all data files with suffix
136 <filename>*.xml</filename>, where the configuration file is found in the
137 path <filename>db/filter_dom_conf.xml</filename>.
140 <para>The &dom; &xslt; filter configuration file must be
141 valid &xml;. It might look like this:
144 <?xml version="1.0" encoding="UTF8"?>
145 <dom xmlns="http://indexdata.com/zebra-2.0">
147 <xmlreader level="1"/>
148 <!-- <marc inputcharset="marc-8"/> -->
151 <xslt stylesheet="common2index.xsl"/>
154 <xslt stylesheet="common2store.xsl"/>
157 <xslt stylesheet="store2dc.xsl"/>
159 <retrieve name="mods">
160 <xslt stylesheet="store2mods.xsl"/>
167 The root &xml; element <literal><dom></literal> and all other &dom;
168 &xml; filter elements are residing in the namespace
169 <literal>http://indexdata.com/zebra-2.0</literal>.
172 All pipeline definition elements - i.e. the
173 <literal><input></literal>,
174 <literal><extact></literal>,
175 <literal><store></literal>, and
176 <literal><retrieve></literal> elements - are optional.
177 Missing pipeline definitions are just interpreted
178 do-nothing identity pipelines.
181 All pipeine definition elements may contain zero or more
182 <literal><![CDATA[<xslt stylesheet="path/file.xsl"/>]]></literal>
183 &xslt; transformation instructions, which are performed
184 sequentially from top to bottom.
185 The paths in the <literal>stylesheet</literal> attributes
186 are relative to zebras working directory, or absolute to the file
191 <section id="record-model-domxml-pipeline-input">
192 <title>Input pipeline</title>
194 The <literal><input></literal> pipeline definition element
195 may contain either one &xml; Reader definition
196 <literal><![CDATA[<xmlreader level="1"/>]]></literal>, used to split
197 an &xml; collection input stream into individual &xml; &dom;
198 documents at the prescribed element level,
201 <literal><![CDATA[<marc inputcharset="marc-8"/>]]></literal>, which defines
202 a conversion to &marcxml; format &dom; trees. The allowed values
203 of the <literal>inputcharset</literal> attribute depend on your
204 local <productname>iconv</productname> set-up.
207 Both input parsers deliver individual &dom; &xml; documents to the
208 following chain of zero or more
209 <literal><![CDATA[<xslt stylesheet="path/file.xsl"/>]]></literal>
210 &xslt; transformations. At the end of this pipeline, the documents
211 are in the common format, used to feed both the
212 <literal><extact></literal> and
213 <literal><store></literal> pipelines.
217 <section id="record-model-domxml-pipeline-extract">
218 <title>Extract pipeline</title>
220 The <literal><extact></literal> pipeline takes documents
221 from any common &dom; &xml; format to the &zebra; specific
222 indexing &dom; &xml; format.
223 It may consist of zero ore more
224 <literal><![CDATA[<xslt stylesheet="path/file.xsl"/>]]></literal>
225 &xslt; transformations, and the outcome is handled to the
226 &zebra; core to drive the proces of building the inverted
228 <xref linkend="record-model-domxml-canonical-index"/> for
233 <section id="record-model-domxml-pipeline-store">
234 <title>Store pipeline</title>
235 The <literal><store></literal> pipeline takes documents
236 from any common &dom; &xml; format to the &zebra; specific
237 storage &dom; &xml; format.
238 It may consist of zero ore more
239 <literal><![CDATA[<xslt stylesheet="path/file.xsl"/>]]></literal>
240 &xslt; transformations, and the outcome is handled to the
241 &zebra; core for deposition into the internal storage system.
244 <section id="record-model-domxml-pipeline-retrieve">
245 <title>Retrieve pipeline</title>
247 Finally, there may be one or more
248 <literal><retrieve></literal> pipeline definitions, each
249 of them again consisting of zero or more
250 <literal><![CDATA[<xslt stylesheet="path/file.xsl"/>]]></literal>
251 &xslt; transformations. These are used for document
252 presentation after search, and take the internal storage &dom;
253 &xml; to the requested output formats during record present
257 The possible multiple
258 <literal><retrieve></literal> pipeline definitions
259 are distinguished by their unique <literal>name</literal>
260 attributes, these are the literal <literal>schema</literal> or
261 <literal>element set</literal> names used in
262 <ulink url="http://www.loc.gov/standards/sru/srw/">&srw;</ulink>,
263 <ulink url="&url.sru;">&sru;</ulink> and
264 &z3950; protocol queries.
269 <section id="record-model-domxml-canonical-index">
270 <title>Canonical Indexing Format</title>
273 &dom; &xml; indexing comes in two flavors: pure
274 processing-instruction governed plain &xml; documents, and - very
275 similar to the Alvis filter indexing format - &xml; documents
276 containing &xml; <literal><record></literal> and
277 <literal><index></literal> instructions from the magic
278 namespace <literal>xmlns:z="http://indexdata.dk/zebra-2.0"</literal>.
281 <section id="record-model-domxml-canonical-index-pi">
282 <title>Processing-instruction governed indexing format</title>
284 <para>The output of the processing instruction driven
285 indexing &xslt; stylesheets must contain
286 processing instructions named
287 <literal>zebra-2.0</literal>.
288 The output of the &xslt; indexing transformation is then
289 parsed using &dom; methods, and the contained instructions are
290 performed on the <emphasis>elements and their
291 subtrees directly following the processing instructions</emphasis>.
294 For example, the output of the command
296 xsltproc dom-index-pi.xsl marc-one.xml
298 might look like this:
301 <?xml version="1.0" encoding="UTF-8"?>
302 <?zebra-2.0 record id=11224466 rank=42?>
304 <?zebra-2.0 index control:w?>
305 <control>11224466</control>
306 <?zebra-2.0 index title:w title:p title:s any:w?>
307 <title>How to program a computer</title>
314 <section id="record-model-domxml-canonical-index-element">
315 <title>Magic element governed indexing format</title>
317 <para>The output of the indexing &xslt; stylesheets must contain
318 certain elements in the magic
319 <literal>xmlns:z="http://indexdata.dk/zebra-2.0"</literal>
320 namespace. The output of the &xslt; indexing transformation is then
321 parsed using &dom; methods, and the contained instructions are
322 performed on the <emphasis>magic elements and their
326 For example, the output of the command
328 xsltproc dom-index-element.xsl marc-one.xml
330 might look like this:
333 <?xml version="1.0" encoding="UTF-8"?>
334 <z:record xmlns:z="http://indexdata.com/zebra-2.0"
335 z:id="11224466" z:rank="42">
336 <z:index name="control">11224466</z:index>
337 <z:index name="title:w title:p title:s any:w">
338 How to program a computer</z:index>
346 <section id="record-model-domxml-canonical-index-semantics">
347 <title>Semantics of the indexing formats</title>
350 Both indexing formats are defined with equal semantics and
355 <para>This means the following: From the original &xml; file
356 <literal>one-record.xml</literal> (or from the &xml; record &dom; of the
357 same form coming from a splitted input file), the indexing
358 stylesheet produces an indexing &xml; record, which is defined by
359 the <literal>record</literal> element in the magic namespace
360 <literal>xmlns:z="http://indexdata.dk/zebra/xslt/1"</literal>.
361 &zebra; uses the content of
362 <literal>z:id="oai:JTRS:CP-3290---Volume-I"</literal> as internal
363 record ID, and - in case static ranking is set - the content of
364 <literal>z:rank="47896"</literal> as static rank. Following the
365 discussion in <xref linkend="administration-ranking"/>
366 we see that this records is internally ordered
367 lexicographically according to the value of the string
368 <literal>oai:JTRS:CP-3290---Volume-I47896</literal>.
369 The type of action performed during indexing is defined by
370 <literal>z:type="update"></literal>, with recognized values
371 <literal>insert</literal>, <literal>update</literal>, and
372 <literal>delete</literal>.
376 <para>In these examples, the following literal indexes are constructed:
384 where the indexing type is defined after the
385 literal <literal>':'</literal> charaacter.
386 Any value from the standard configuration
387 file <filename>default.idx</filename> will do.
389 <literal>text()</literal> node content recursively contained
390 inside the <literal><z:index></literal> element, or any
391 element following a <literal>index</literal> processing instruction,
392 will be filtered through the
393 appropriate charmap for character normalization, and will be
394 inserted in the named indexes.
399 Specific to this example, we see that the single word
400 <literal>oai:JTRS:CP-3290---Volume-I</literal> will be literal,
401 byte for byte without any form of character normalization,
402 inserted into the index named <literal>oai:identifier</literal>,
404 <literal>Kumar Krishen and *Calvin Burnham, Editors</literal>
405 will be inserted using the <literal>w</literal> character
406 normalization defined in <filename>default.idx</filename> into
407 the index <literal>dc:creator</literal> (that is, after character
408 normalization the index will keep the inidividual words
409 <literal>kumar</literal>, <literal>krishen</literal>,
410 <literal>and</literal>, <literal>calvin</literal>,
411 <literal>burnham</literal>, and <literal>editors</literal>), and
412 finally both the texts
413 <literal>Proceedings of the 4th International Conference and Exhibition:
414 World Congress on Superconductivity - Volume I</literal>
416 <literal>Kumar Krishen and *Calvin Burnham, Editors</literal>
417 will be inserted into the index <literal>dc:all</literal> using
418 the same character normalization map <literal>w</literal>.
421 Finally, this example configuration can be queried using &pqf;
422 queries, either transported by &z3950;, (here using a yaz-client)
425 Z> open localhost:9999
429 Z> f @attr 1=dc_creator Kumar
430 Z> scan @attr 1=dc_creator adam
432 Z> f @attr 1=dc_title @attr 4=2 "proceeding congress superconductivity"
433 Z> scan @attr 1=dc_title abc
437 extentions <literal>x-pquery</literal> and
438 <literal>x-pScanClause</literal> to
442 http://localhost:9999/?version=1.1&operation=searchRetrieve&x-pquery=%40attr+1%3Ddc_creator+%40attr+4%3D6+%22the
443 http://localhost:9999/?version=1.1&operation=scan&x-pScanClause=@attr+1=dc_date+@attr+4=2+a
446 See <xref linkend="zebrasrv-sru"/> for more information on &sru;/&srw;
447 configuration, and <xref linkend="gfs-config"/> or the &yaz;
448 <ulink url="&url.yaz.cql;">&cql; section</ulink>
449 for the details or the &yaz; frontend server.
452 Notice that there are no <filename>*.abs</filename>,
453 <filename>*.est</filename>, <filename>*.map</filename>, or other &grs1;
454 filter configuration files involves in this process, and that the
455 literal index names are used during search and retrieval.
464 <section id="record-model-domxml-conf">
465 <title>&dom; Record Model Configuration</title>
468 <section id="record-model-domxml-index">
469 <title>&dom; Indexing Configuration</title>
471 As mentioned above, there can be only one indexing
472 stylesheet, and configuration of the indexing process is a synonym
473 of writing an &xslt; stylesheet which produces &xml; output containing the
474 magic elements discussed in
475 <xref linkend="record-model-domxml-internal"/>.
476 Obviously, there are million of different ways to accomplish this
477 task, and some comments and code snippets are in order to lead
478 our paduans on the right track to the good side of the force.
481 Stylesheets can be written in the <emphasis>pull</emphasis> or
482 the <emphasis>push</emphasis> style: <emphasis>pull</emphasis>
483 means that the output &xml; structure is taken as starting point of
484 the internal structure of the &xslt; stylesheet, and portions of
485 the input &xml; are <emphasis>pulled</emphasis> out and inserted
486 into the right spots of the output &xml; structure. On the other
487 side, <emphasis>push</emphasis> &xslt; stylesheets are recursavly
488 calling their template definitions, a process which is commanded
489 by the input &xml; structure, and avake to produce some output &xml;
490 whenever some special conditions in the input styelsheets are
491 met. The <emphasis>pull</emphasis> type is well-suited for input
492 &xml; with strong and well-defined structure and semantcs, like the
493 following &oai; indexing example, whereas the
494 <emphasis>push</emphasis> type might be the only possible way to
495 sort out deeply recursive input &xml; formats.
498 A <emphasis>pull</emphasis> stylesheet example used to index
499 &oai; harvested records could use some of the following template
503 <xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
504 xmlns:z="http://indexdata.dk/zebra/xslt/1"
505 xmlns:oai="http://www.openarchives.org/&oai;/2.0/"
506 xmlns:oai_dc="http://www.openarchives.org/&oai;/2.0/oai_dc/"
507 xmlns:dc="http://purl.org/dc/elements/1.1/"
510 <xsl:output indent="yes" method="xml" version="1.0" encoding="UTF-8"/>
512 <!-- disable all default text node output -->
513 <xsl:template match="text()"/>
515 <!-- match on oai xml record root -->
516 <xsl:template match="/">
517 <z:record z:id="{normalize-space(oai:record/oai:header/oai:identifier)}"
519 <!-- you might want to use z:rank="{some &xslt; function here}" -->
520 <xsl:apply-templates/>
524 <!-- &oai; indexing templates -->
525 <xsl:template match="oai:record/oai:header/oai:identifier">
526 <z:index name="oai_identifier" type="0">
527 <xsl:value-of select="."/>
533 <!-- DC specific indexing templates -->
534 <xsl:template match="oai:record/oai:metadata/oai_dc:dc/dc:title">
535 <z:index name="dc_title" type="w">
536 <xsl:value-of select="."/>
548 that the names and types of the indexes can be defined in the
549 indexing &xslt; stylesheet <emphasis>dynamically according to
550 content in the original &xml; records</emphasis>, which has
551 opportunities for great power and wizardery as well as grande
555 The following excerpt of a <emphasis>push</emphasis> stylesheet
556 <emphasis>might</emphasis>
557 be a good idea according to your strict control of the &xml;
558 input format (due to rigerours checking against well-defined and
559 tight RelaxNG or &xml; Schema's, for example):
562 <xsl:template name="element-name-indexes">
563 <z:index name="{name()}" type="w">
564 <xsl:value-of select="'1'"/>
569 This template creates indexes which have the name of the working
570 node of any input &xml; file, and assigns a '1' to the index.
572 <literal>find @attr 1=xyz 1</literal>
573 finds all files which contain at least one
574 <literal>xyz</literal> &xml; element. In case you can not control
575 which element names the input files contain, you might ask for
576 disaster and bad karma using this technique.
579 One variation over the theme <emphasis>dynamically created
580 indexes</emphasis> will definitely be unwise:
583 <!-- match on oai xml record root -->
584 <xsl:template match="/">
585 <z:record z:type="update">
587 <!-- create dynamic index name from input content -->
588 <xsl:variable name="dynamic_content">
589 <xsl:value-of select="oai:record/oai:header/oai:identifier"/>
592 <!-- create zillions of indexes with unknown names -->
593 <z:index name="{$dynamic_content}" type="w">
594 <xsl:value-of select="oai:record/oai:metadata/oai_dc:dc"/>
601 Don't be tempted to cross
602 the line to the dark side of the force, paduan; this leads
603 to suffering and pain, and universal
604 disentigration of your project schedule.
608 <section id="record-model-domxml-elementset">
609 <title>&dom; Exchange Formats</title>
611 An exchange format can be anything which can be the outcome of an
612 &xslt; transformation, as far as the stylesheet is registered in
613 the main &dom; &xslt; filter configuration file, see
614 <xref linkend="record-model-domxml-filter"/>.
615 In principle anything that can be expressed in &xml;, HTML, and
616 TEXT can be the output of a <literal>schema</literal> or
617 <literal>element set</literal> directive during search, as long as
618 the information comes from the
619 <emphasis>original input record &xml; &dom; tree</emphasis>
620 (and not the transformed and <emphasis>indexed</emphasis> &xml;!!).
623 In addition, internal administrative information from the &zebra;
624 indexer can be accessed during record retrieval. The following
625 example is a summary of the possibilities:
628 <xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
629 xmlns:z="http://indexdata.dk/zebra/xslt/1"
632 <!-- register internal zebra parameters -->
633 <xsl:param name="id" select="''"/>
634 <xsl:param name="filename" select="''"/>
635 <xsl:param name="score" select="''"/>
636 <xsl:param name="schema" select="''"/>
638 <xsl:output indent="yes" method="xml" version="1.0" encoding="UTF-8"/>
640 <!-- use then for display of internal information -->
641 <xsl:template match="/">
643 <id><xsl:value-of select="$id"/></id>
644 <filename><xsl:value-of select="$filename"/></filename>
645 <score><xsl:value-of select="$score"/></score>
646 <schema><xsl:value-of select="$schema"/></schema>
657 <section id="record-model-domxml-example">
658 <title>&dom; Filter &oai; Indexing Example</title>
660 The sourcecode tarball contains a working &dom; filter example in
661 the directory <filename>examples/dom-oai/</filename>, which
662 should get you started.
665 More example data can be harvested from any &oai; complient server,
666 see details at the &oai;
667 <ulink url="http://www.openarchives.org/">
668 http://www.openarchives.org/</ulink> web site, and the community
670 <ulink url="http://www.openarchives.org/community/index.html">
671 http://www.openarchives.org/community/index.html</ulink>.
674 <ulink url="http://www.oaforum.org/tutorial/">
675 http://www.oaforum.org/tutorial/</ulink>.
687 c) Main "dom" &xslt; filter config file:
688 cat db/filter_dom_conf.xml
690 <?xml version="1.0" encoding="UTF8"?>
692 <schema name="dom" stylesheet="db/dom2dom.xsl" />
693 <schema name="index" identifier="http://indexdata.dk/zebra/xslt/1"
694 stylesheet="db/dom2index.xsl" />
695 <schema name="dc" stylesheet="db/dom2dc.xsl" />
696 <schema name="dc-short" stylesheet="db/dom2dc_short.xsl" />
697 <schema name="snippet" snippet="25" stylesheet="db/dom2snippet.xsl" />
698 <schema name="help" stylesheet="db/dom2help.xsl" />
702 the paths are relative to the directory where zebra.init is placed
705 The split level decides where the SAX parser shall split the
706 collections of records into individual records, which then are
707 loaded into &dom;, and have the indexing &xslt; stylesheet applied.
709 The indexing stylesheet is found by it's identifier.
711 All the other stylesheets are for presentation after search.
713 - in data/ a short sample of harvested carnivorous plants
714 ZEBRA_INDEX_DIRS=data/carnivor_20050118_2200_short-346.xml
716 - in root also one single data record - nice for testing the xslt
719 xsltproc db/dom2index.xsl carni*.xml
723 - in db/ a cql2pqf.txt yaz-client config file
724 which is also used in the yaz-server <ulink url="&url.cql;">&cql;</ulink>-to-&pqf; process
726 see: http://www.indexdata.com/yaz/doc/tools.tkl#tools.cql.map
728 - in db/ an indexing &xslt; stylesheet. This is a PULL-type XSLT thing,
729 as it constructs the new &xml; structure by pulling data out of the
730 respective elements/attributes of the old structure.
732 Notice the special zebra namespace, and the special elements in this
733 namespace which indicate to the zebra indexer what to do.
735 <z:record id="67ht7" rank="675" type="update">
736 indicates that a new record with given id and static rank has to be updated.
738 <z:index name="title" type="w">
739 encloses all the text/&xml; which shall be indexed in the index named
740 "title" and of index type "w" (see file default.idx in your zebra
752 <!-- Keep this comment at the end of the file
757 sgml-minimize-attributes:nil
758 sgml-always-quote-attributes:t
761 sgml-parent-document: "zebra.xml"
762 sgml-local-catalogs: nil
763 sgml-namecase-general:t