1 <?xml version="1.0" standalone="no"?>
2 <!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.4//EN"
3 "http://www.oasis-open.org/docbook/xml/4.4/docbookx.dtd"
5 <!ENTITY % local SYSTEM "local.ent">
7 <!ENTITY % entities SYSTEM "entities.ent">
9 <!ENTITY % idcommon SYSTEM "common/common.ent">
14 <title>YAZ User's Guide and Reference</title>
16 <author><firstname>Sebastian</firstname><surname>Hammer</surname></author>
17 <author><firstname>Adam</firstname><surname>Dickmeiss</surname></author>
18 <author><firstname>Mike</firstname><surname>Taylor</surname></author>
19 <author><firstname>Heikki</firstname><surname>Levanto</surname></author>
20 <author><firstname>Dennis</firstname><surname>Schafroth</surname></author>
22 <releaseinfo>&version;</releaseinfo>
24 <year>©right-year;</year>
25 <holder>Index Data</holder>
29 This document is the programmer's guide and reference to the &yaz;
30 package version &version;. &yaz; is a compact toolkit that provides
31 access to the Z39.50 and SRU/Solr protocols, as well as a set of
32 higher-level tools for implementing the server and client
34 The documentation can be used on its own, or as a reference when
35 looking at the example applications provided with the package.
40 <imagedata fileref="common/id.png" format="PNG"/>
43 <imagedata fileref="common/id.eps" format="EPS"/>
48 <chapter id="introduction">
49 <title>Introduction</title>
51 &yaz; is a C/C++ library for information retrieval applications
52 using the Z39.50/SRU/Solr protocols for information retrieval.
60 <ulink url="&url.z39.50;">Z39.50</ulink> version 3 support.
61 Amendments and Z39.50-2002 revision is supported.
67 <ulink url="&url.sru;">SRU GET/POST/SOAP</ulink>
68 version 1.1, 1.2 and 2.0 (over HTTP and HTTPS).
73 Includes BER encoders/decoders for the
74 <ulink url="&url.ill;">ISO ILL</ulink>
81 <ulink url="&url.solr;">Apache Solr</ulink> Web Service version 1.4.x
87 Supports the following transports: BER over TCP/IP
88 (<ulink url="&url.ber.over.tcpip;">RFC1729</ulink>),
89 BER over unix local socket, and
90 <ulink url="&url.http.1.1;">HTTP 1.1</ulink>.
95 Secure Socket Layer support using
96 <ulink url="&url.gnutls;">GnuTLS</ulink>.
97 If enabled, &yaz; uses HTTPS transport (for SOAP) or
98 "Secure BER" (for Z39.50).
104 <ulink url="&url.zoom;">ZOOM</ulink> C API implementing
105 Z39.50, SRU and Solr Web Service.
110 The &yaz; library offers a set of useful utilities
111 related to the protocols, such as MARC (ISO2709) parser,
112 CCL (ISO8777) parser,
113 <ulink url="&url.cql;">CQL</ulink>
114 parser, memory management routines, character set conversion.
119 Portable code. &yaz; compiles out-of-the box on most Unixes and
120 on Windows using Microsoft Visual C++.
125 Fast operation. The C based BER encoders/decoders as well
126 as the server component of &yaz; is very fast.
131 Liberal license that allows for commercial use of &yaz;.
137 <sect1 id="introduction.reading">
138 <title>Reading this Manual</title>
140 Most implementors only need to read a fraction of the
141 material in this manual, so a quick walkthrough of the chapters
147 <xref linkend="installation"/> contains installation
148 instructions for &yaz;. You don't need to read this
149 if you expect to download &yaz; binaries.
150 However, the chapter contains information about how
151 to make <emphasis>your</emphasis> application link
157 <xref linkend="zoom"/> describes the ZOOM API of &yaz;.
158 This is definitely worth reading if you wish to develop a Z39.50/SRU
164 <xref linkend="server"/> describes the generic frontend server
165 and explains how to develop server Z39.50/SRU applications for &yaz;.
166 Obviously worth reading if you're to develop a server.
171 <xref linkend="yaz-client"/> describes how to use the &yaz; Z39.50
172 client. If you're a developer and wish to test your server
173 or a server from another party, you might find this chapter
179 <xref linkend="asn"/> documents the most commonly used Z39.50
180 C data structures offered by the &yaz; API. Client
181 developers using ZOOM and non-Z39.50 implementors may skip this.
186 <xref linkend="soap"/> describes how SRU and SOAP is used
187 in &yaz;. Only if you're developing SRU applications
188 this section is a must.
193 <xref linkend="tools"/> contains sections for the various
194 tools offered by &yaz;. Scan through the material quickly
195 and see what's relevant to you! SRU implementors
196 might find the <link linkend="cql">CQL</link> section
202 <xref linkend="odr"/> goes through the details of the
203 ODR module which is the work horse that encodes and decodes
204 BER packages. Implementors using ZOOM only, do <emphasis>not</emphasis>
206 Most other Z39.50 implementors only need to read the first two
207 sections (<xref linkend="odr.introduction"/> and
208 <xref linkend="odr.use"/>).
213 <xref linkend="comstack"/> describes the network layer module
214 COMSTACK. Implementors using ZOOM or the generic frontend server
215 may skip this. Others, presumably, handling client/server
216 communication on their own should read this.
221 <sect1 id="introduction.api">
222 <title>The API</title>
224 The <ulink url="&url.yaz;">&yaz;</ulink>
225 toolkit offers several different levels of access to the
226 <ulink url="&url.z39.50;">ISO23950/Z39.50</ulink>,
227 <ulink url="&url.ill;">ILL</ulink> and
228 <ulink url="&url.sru;">SRU</ulink>
230 The level that you need to use depends on your requirements, and
231 the role (server or client) that you want to implement.
232 If you're developing a client application you should consider the
233 <link linkend="zoom">ZOOM</link> API.
234 It is, by far, the easiest way to develop clients in C.
235 Server implementers should consider the
236 <link linkend="server">generic frontend server</link>.
237 None of those high-level APIs support the whole protocol, but
238 they do include most facilities used in existing Z39.50 applications.
241 If you're using 'exotic' functionality (meaning anything not included in
242 the high-level APIs), developing non-standard extensions to Z39.50 or
243 you're going to develop an ILL application you'll have to learn the lower
247 The YAZ toolkit modules are shown in figure <xref linkend="yaz.layer"/>.
249 <figure id="yaz.layer">
250 <title>YAZ layers</title>
253 <imagedata fileref="apilayer.png" format="PNG"/>
256 <imagedata fileref="apilayer.eps" format="EPS"/>
261 There are four layers.
264 <para>A client or server application (or both).
265 This layer includes ZOOM and the generic frontend server.
270 The second layer provides a C represenation of the
271 protocol units (packages) for Z39.50 ASN.1, ILL ASN.1,
277 The third layer encodes and decodes protocol data units to
278 simple packages (buffer with certain length). The &odr; module
279 encodes and decodes BER whereas the HTTP modules encodes and
280 decodes HTTP ruquests/responses.
285 The lowest layer is &comstack; which exchanges the encoded packages
286 with a peer process over a network.
292 The &asn; module represents the ASN.1 definition of
293 the Z39.50 protocol. It establishes a set of type and
294 structure definitions, with one structure for each of the top-level
295 PDUs, and one structure or type for each of the contained ASN.1 types.
296 For primitive types, or other types that are defined by the ASN.1
297 standard itself (such as the EXTERNAL type), the C representation is
298 provided by the &odr; (Open Data Representation) subsystem.
301 &odr; is a basic mechanism for representing an
302 ASN.1 type in the C programming language, and for implementing BER
303 encoders and decoders for values of that type. The types defined in
304 the &asn; module generally have the prefix <literal>Z_</literal>, and
305 a suffix corresponding to the name of the type in the ASN.1
306 specification of the protocol (generally Z39.50-1995). In the case of
307 base types (those originating in the ASN.1 standard itself), the prefix
308 <literal>Odr_</literal> is sometimes seen. Either way, look for
309 the actual definition in either <filename>z-core.h</filename> (for the types
310 from the protocol), <filename>odr.h</filename> (for the primitive ASN.1
312 The &asn; library also provides functions (which are, in turn,
313 defined using &odr; primitives) for encoding and decoding data values.
314 Their general form is
316 <funcprototype><funcdef>int <function>z_<replaceable>xxx</replaceable></function></funcdef>
317 <paramdef>ODR <parameter>o</parameter></paramdef>
318 <paramdef>Z_<replaceable>xxx</replaceable> **<parameter>p</parameter></paramdef>
319 <paramdef>int <parameter>optional</parameter></paramdef>
320 <paramdef>const char *<parameter>name</parameter></paramdef>
323 (note the lower-case "z" in the function name)
327 If you are using the premade definitions of the &asn; module, and you
328 are not adding a new protocol of your own, the only parts of &odr; that you
329 need to worry about are documented in
330 <xref linkend="odr.use"/>.
334 When you have created a BER-encoded buffer, you can use the &comstack;
335 subsystem to transmit (or receive) data over the network. The &comstack;
336 module provides simple functions for establishing a connection
337 (passively or actively, depending on the role of your application),
338 and for exchanging BER-encoded PDUs over that connection. When you
339 create a connection endpoint, you need to specify what transport to
340 use (TCP/IP, SSL or UNIX sockets).
341 For the remainder of the connection's lifetime, you don't have
342 to worry about the underlying transport protocol at all - the &comstack;
343 will ensure that the correct mechanism is used.
346 We call the combined interfaces to &odr;, &asn;, and &comstack; the service
347 level API. It's the API that most closely models the Z39.50
348 service/protocol definition, and it provides unlimited access to all
349 fields and facilities of the protocol definitions.
352 The reason that the &yaz; service-level API is a conglomerate of the
353 APIs from three different submodules is twofold. First, we wanted to allow
354 the user a choice of different options for each major task. For instance,
355 if you don't like the protocol API provided by &odr;/&asn;, you
356 can use SNACC or BERUtils instead, and still have the benefits of the
357 transparent transport approach of the &comstack; module. Secondly,
358 we realize that you may have to fit the toolkit into an existing
359 event-processing structure, in a way that is incompatible with
360 the &comstack; interface or some other part of &yaz;.
364 <chapter id="installation">
365 <title>Compilation and Installation</title>
366 <sect1 id="installation-introduction">
367 <title>Introduction</title>
369 The latest version of the software will generally be found at:
372 <ulink url="&url.yaz.download;"/>
375 We have tried our best to keep the software portable, and on many
376 platforms, you should be able to compile everything with little or
380 The software is regularly tested on
381 <ulink url="&url.debian;">Debian GNU/Linux</ulink>,
382 <ulink url="&url.centos;">CentOS</ulink>,
383 <ulink url="&url.ubuntu;">Ubuntu Linux</ulink>,
384 <ulink url="&url.freebsd;">FreeBSD (i386)</ulink>,
385 <ulink url="&url.macosx;">MAC OSX</ulink>,
389 Some versions have be known to work on Windows XP, Solaris, HP/UX,
390 DEC Unix, <ulink url="&url.netbsd;">NetBSD</ulink>,
391 <ulink url="&url.openbsd;">OpenBSD</ulink>,
393 Data General DG/UX (with some CFLAGS tinkering),
394 SGI/IRIX, DDE Supermax, Apple Macintosh (using the Codewarrior programming
395 environment and the GUSI socket libraries),
399 If you move the software to other platforms, we'd be grateful if you'd
400 let us know about it. If you run into difficulties, we will try to help
401 if we can, and if you solve the problems, we would be happy to include
402 your fixes in the next release. So far, we have mostly avoided
403 <literal>#ifdefs</literal> for individual platforms, and we'd
404 like to keep it that way as far as it makes sense.
407 We maintain a mailing-list for the purpose of announcing new releases and
408 bug-fixes, as well as general discussion. Subscribe by
410 <ulink url="&url.yaz.mailinglist;">here</ulink>.
411 General questions and problems can be directed at
412 <ulink url="&url.yaz.mail;"/>, or the address given at the top of
416 <sect1 id="installation.unix"><title>UNIX</title>
419 <ulink url="&url.debian;">Debian GNU/Linux</ulink> (i386 and amd64),
420 <ulink url="&url.ubuntu;">Ubuntu</ulink> (i386 and amd64)
422 <ulink url="&url.centos;">CentOS</ulink> (amd64 only) packages for &yaz;.
423 You should be able to create packages for other CPUs by building
424 them from the source package.
427 YAZ is also part of several packages repositories. Some of them are
432 Solaris CSW: <ulink url="http://www.opencsw.org/packages/yaz/"/>
437 Solaris: <ulink url="http://unixpackages.com"/>
442 FreeBSD: <ulink url="http://www.freshports.org/net/yaz"/>
447 Debian: <ulink url="http://packages.debian.org/search?keywords=yaz"/>
452 Ubuntu: <ulink url="https://launchpad.net/ubuntu/+source/yaz"/>
458 <ulink url="http://ftp.netbsd.org/pub/pkgsrc/current/pkgsrc/net/yaz/README.html"/>
462 <sect2 id="installation.source.unix">
463 <title>Compiling from source on Unix</title>
465 Note that if your system doesn't have a native ANSI C compiler, you may
466 have to acquire one separately. We recommend
467 <ulink url="&url.gcc;">GCC</ulink>.
470 If you wish to use character set conversion facilities in &yaz; or if you
471 are compiling &yaz; for use with Zebra it is a good idea to ensure that
472 the iconv library is installed. Some Unixes today already have it
474 <ulink url="&url.libiconv;">GNU libiconv</ulink>.
477 YAZ 3.0.16 and later includes a wrapper for the
478 <ulink url="&url.icu;">ICU</ulink>
479 (International Components for Unicode).
480 In order to use this, the developer version of the ICU library
481 must be available. ICU support is recommended for applications
482 such as Pazpar2 and Zebra.
485 The <ulink url="&url.libxslt;">libxslt</ulink>,
486 <ulink url="&url.libxml2;">libxml2</ulink> librararies are required
487 if &yaz; is to support SRU/Solr.
488 These libraries are very portable and should compile out-of-the
489 box on virtually all Unix platforms. It is available in binary
490 forms for Linux and others.
494 <ulink url="&url.autoconf;">Autoconf</ulink>,
495 <ulink url="&url.automake;">Automake</ulink> and
496 <ulink url="&url.libtool;">Libtool</ulink>
497 are used to generate Makefiles and configure &yaz; for the system.
498 You do <emphasis>not</emphasis> need these tools unless you're using the
499 Git version of &yaz;.
502 The CQL parser for &yaz; is built using
503 GNU <ulink url="&url.bison;">Bison</ulink>.
504 This tool is only needed if you're using the Git version of &yaz;.
507 &yaz; includes a tiny ASN.1 compiler. This compiler is
508 written in <ulink url="&url.tcl;">Tcl</ulink>.
509 But as for Bison you do not need it unless you're using Git
510 version of &yaz; or you're using the compiler to build your own codecs
514 Generally it should be sufficient to run configure without options,
521 The configure script attempts to use use the C compiler specified by
522 the <literal>CC</literal> environment variable. If not set, GNU C will be
523 used if it is available. The <literal>CFLAGS</literal> environment
524 variable holds options to be passed to the C compiler. If you're using
525 Bourne-compatible shell you may pass something like this to use a
526 particular C compiler with optimization enabled:
529 CC=/opt/ccs/bin/cc CFLAGS=-O ./configure
532 To customize &yaz;, the configure script also accepts a set of options.
533 The most important are:
537 <literal>--prefix</literal>=<replaceable>prefix</replaceable>
540 <para>Specifies installation prefix for &yaz;. This is
541 only needed if you run <literal>make install</literal> later to
542 perform a "system" installation. The prefix is
543 <literal>/usr/local</literal> if not specified.
549 <literal>--enable-tcpd</literal>
552 <para>The front end server will be built using Wietse's
553 <ulink url="&url.tcpwrapper;">TCP wrapper library</ulink>.
554 It allows you to allow/deny clients depending on IP number.
555 The TCP wrapper library is often used in GNU/Linux and
559 <refentrytitle>hosts_access</refentrytitle>
560 <manvolnum>5</manvolnum>
564 <refentrytitle>tcpd</refentrytitle>
565 <manvolnum>8</manvolnum>
572 <literal>--enable-threads</literal>
575 <para>&yaz; will be built using POSIX threads.
576 Specifically, <constant>_REENTRANT</constant> will be defined during
583 <literal>--disable-shared</literal>
586 <para>The make process will not create shared
587 libraries (also known as shared objects <filename>.so</filename>).
588 By default, shared libraries are created -
589 equivalent to <literal>--enable-shared</literal>.
595 <literal>--disable-shared</literal>
598 <para>The make process will not create
599 static libraries (<filename>.a</filename>).
600 By default, static libraries are created -
601 equivalent to <literal>--enable-static</literal>.
607 <literal>--with-iconv</literal>[=<replaceable>prefix</replaceable>]
610 <para>Compile &yaz; with iconv library in directory
611 <replaceable>prefix</replaceable>. By default configure will
612 search for iconv on the system. Use this option if it
613 doesn't find iconv. Alternatively,
614 <literal>--without-iconv</literal>, can be used to force &yaz;
621 <literal>--with-xslt</literal>[=<replaceable>prefix</replaceable>]
624 <para>Compile &yaz; with
625 <ulink url="&url.libxslt;">libxslt</ulink> in directory
626 <replaceable>prefix</replaceable>.
627 Use this option if you want XSLT and XML support.
628 By default, configure will
629 search for libxslt on the system. Use this option if
630 libxslt is not found automatically. Alternatively,
631 <literal>--without-xslt</literal>, can be used to force &yaz;
638 <literal>--with-xml2</literal>[=<replaceable>prefix</replaceable>]
641 <para>Compile &yaz; with
642 <ulink url="&url.libxml2;">libxml2</ulink> in directory
643 <replaceable>prefix</replaceable>.
644 Use this option if you want &yaz; to use XML and support SRU/Solr.
645 By default, configure will
646 search for libxml2 on the system. Use this option if
647 libxml2 is not found automatically. Alternatively,
648 <literal>--without-xml2</literal>, can be used to force &yaz;
652 Note that option <literal>--with-xslt</literal>
653 also enables libxml2.
659 <literal>--with-gnutls</literal>[=<replaceable>prefix</replaceable>]
662 <para>&yaz; will be linked with the GNU TLS libraries and
663 an SSL COMSTACK will be provided. By default configure enables
664 SSL support for YAZ if the GNU TLS development libraries are found
671 <literal>--with-icu</literal>[=<replaceable>prefix</replaceable>]
674 <para>&yaz; will be linked the
675 <ulink url="&url.icu;">ICU</ulink> library in the prefix if given.
676 If prefix is not given, the libraries exposed by the script
677 <application>icu-config</application> will be used if found.
684 <literal>--with-memcached</literal>
687 <para>&yaz; will be linked with
688 <ulink url="&url.libmemcached;">libMemcached</ulink> to allow
689 for result-set caching for ZOOM.
690 The prefix can not be given.
691 Note that 0.40 of libmemcached is required.
697 <literal>--with-redis</literal>
700 <para>&yaz; will be linked with the hiredis C library
701 to allow for result-set caching for ZOOM on a
702 <ulink url="&url.redis;">redis</ulink> server.
703 The prefix can not be given.
711 When configured, build the software by typing:
717 The following files are generated by the make process:
720 <term><filename>src/libyaz.la</filename></term>
722 Main &yaz; library. This is no ordinary library. It's
724 By default, &yaz; creates a static library in
725 <filename>lib/.libs/libyaz.a</filename>.
729 <term><filename>src/libyaz_server.la</filename></term>
731 Generic Frontend server. This is an add-on for libyaz.la.
732 Code in this library uses POSIX threads functions - if POSIX
733 threads are available on the platform.
737 <term><filename>src/libyaz_icu.la</filename></term>
739 Functions that wrap the ICU library.
743 <term><filename>ztest/yaz-ztest</filename></term>
744 <listitem><para>Test Z39.50 server.
748 <term><filename>client/yaz-client</filename></term>
749 <listitem><para>Z39.50 client for testing the protocol.
750 See chapter <link linkend="yaz-client">
751 YAZ client</link> for more information.
755 <term><filename>util/yaz-config</filename></term>
756 <listitem><para>A Bourne-shell script, generated by configure, that
757 specifies how external applications should compile - and link with
762 <term><filename>util/yaz-asncomp</filename></term>
763 <listitem><para>The ASN.1 compiler for &yaz;. Requires the
764 Tcl Shell, <application>tclsh</application>, in
765 <literal>PATH</literal> to operate.
769 <term><filename>util/yaz-iconv</filename></term>
770 <listitem><para>This program converts data in one character set to
771 another. This command exercises the YAZ character set
776 <term><filename>util/yaz-marcdump</filename></term>
777 <listitem><para>This program parses ISO2709 encoded MARC records
778 and prints them in line-format or XML.
782 <term><filename>util/yaz-icu</filename></term>
783 <listitem><para>This program exposes the ICU wrapper library if that
784 is enabled for YAZ. Only if ICU is available this program is
789 <term><filename>util/yaz-url</filename></term>
790 <listitem><para>This program is a simple HTTP page fetcher ala
795 <term><filename>zoom/zoomsh</filename></term>
797 A simple shell implemented on top of the
798 <link linkend="zoom">ZOOM</link> functions.
799 The shell is a command line application that allows you to enter
800 simple commands to perform ZOOM operations.
804 <term><filename>zoom/zoomtst1</filename>,
805 <filename>zoom/zoomtst2</filename>, ..</term>
807 Several small applications that demonstrates the ZOOM API.
813 If you wish to install &yaz; in system directories
814 <filename>/usr/local/bin</filename>,
815 <filename>/usr/local/lib</filename> .. etc, you can type:
821 You probably need to have root access in order to perform this.
822 You must specify the <literal>--prefix</literal> option for configure if
823 you wish to install &yaz; in other directories than the default
824 <filename>/usr/local/</filename>.
827 If you wish to perform an un-installation of &yaz;, use:
833 This will only work if you haven't reconfigured &yaz; (and therefore
834 changed installation prefix). Note that uninstall will not
835 remove directories created by make install, e.g.
836 <filename>/usr/local/include/yaz</filename>.
839 <sect2 id="installation-linking-yaz-unix">
840 <title>How to make apps using YAZ on UNIX</title>
842 This section describes how to compile - and link your own
843 applications using the &yaz; toolkit.
844 If you're used to Makefiles this shouldn't be hard. As for
845 other libraries you have used before, you need to set a proper include
846 path for your C/C++ compiler and specify the location of
847 &yaz; libraries. You can do it by hand, but generally we suggest
848 you use the <filename>yaz-config</filename> that is generated
849 by <filename>configure</filename>. This is especially
850 important if you're using the threaded version of &yaz; which
851 require you to pass more options to your linker/compiler.
854 The <filename>yaz-config</filename> script accepts command line
855 options that makes the <filename>yaz-config</filename> script print
856 options that you should use in your make process.
857 The most important ones are:
858 <literal>--cflags</literal>, <literal>--libs</literal>
859 which prints C compiler flags, and linker flags respectively.
862 A small and complete <literal>Makefile</literal> for a C
863 application consisting of one source file,
864 <filename>myprog.c</filename>, may look like this:
866 YAZCONFIG=/usr/local/bin/yaz-config
867 CFLAGS=`$(YAZCONFIG) --cflags`
868 LIBS=`$(YAZCONFIG) --libs`
870 $(CC) $(CFLAGS) -o myprog myprog.o $(LIBS)
874 The CFLAGS variable consists of a C compiler directive that will set
875 the include path to the <emphasis>parent</emphasis> directory
876 of <filename>yaz</filename>. That is, if &yaz; header files were
877 installed in <filename>/usr/local/include/yaz</filename>,
878 then include path is set to <filename>/usr/local/include</filename>.
879 Therefore, in your applications you should use
881 #include <yaz/proto.h>
883 and <emphasis>not</emphasis>
885 #include <proto.h>
889 For Libtool users, the <filename>yaz-config</filename> script provides
890 a different variant of option <literal>--libs</literal>, called
891 <literal>--lalibs</literal> that returns the name of the
892 Libtool archive(s) for &yaz; rather than the ordinary ones.
895 For applications using the threaded version of &yaz;,
896 specify <literal>threads</literal> after the
897 other options. When <literal>threads</literal> is given,
898 more flags and linker flags will be printed by
899 <filename>yaz-config</filename>. If our previous example was
900 using threads, you'd have to modify the lines that set
901 <literal>CFLAGS</literal> and <literal>LIBS</literal> as
904 CFLAGS=`$(YAZCONFIG) --cflags threads`
905 LIBS=`$(YAZCONFIG) --libs threads`
907 There is no need specify POSIX thread libraries in your Makefile.
908 The <literal>LIBS</literal> variable includes that as well.
912 <sect1 id="installation.win32">
913 <title>Windows</title>
914 <para>The easiest way to install YAZ on Windows is by downloading
916 <ulink url="&url.yaz.download.win32;">
917 Index Data's Windows support area
919 The installer comes with source too - in case you wish to
920 compile YAZ with different compiler options, etc.
923 <sect2 id="installation.win32.source">
924 <title>Compiling from Source on Windows</title>
926 &yaz; is shipped with "makefiles" for the NMAKE tool that comes
927 with <ulink url="&url.vstudio;">
928 Microsoft Visual Studio</ulink>. It has been tested with
929 Microsoft Visual Studio 2015.
932 Start a command prompt and switch the sub directory
933 <filename>WIN</filename> where the file <filename>makefile</filename>
934 is located. Customize the installation by editing the
935 <filename>makefile</filename> file (for example by using notepad).
936 The following summarizes the most important settings in that file:
939 <term><literal>DEBUG</literal></term>
941 If set to 1, the software is
942 compiled with debugging libraries (code generation is
943 multi-threaded debug DLL).
944 If set to 0, the software is compiled with release libraries
945 (code generation is multi-threaded DLL).
949 <term><literal>HAVE_TCL</literal>, <literal>TCL</literal></term>
951 If <literal>HAVE_TCL</literal> is set to 1, nmake will
952 use the ASN.1 compiler (<ulink url="&url.tcl;">Tcl</ulink> based).
953 You must set <literal>TCL</literal> to the full path of the Tcl
954 interpreter. A Windows version of Tcl is part of
955 <ulink url="&url.gitwindows;">Git for Windows</ulink>.
958 If you do not have Tcl installed, set
959 <literal>HAVE_TCL</literal> to 0.
963 <term><literal>HAVE_BISON</literal>,
964 <literal>BISON</literal></term>
966 If GNU Bison is present, you might set <literal>HAVE_BISON</literal>
967 to 1 and specify the Bison executable in <literal>BISON</literal>.
968 Bison is only required if you use the Git version of
969 YAZ or if you modify the grammar for CQL
970 (<filename>cql.y</filename>).
973 A Windows version of GNU Bison can be fetched from here:
975 url="&url.indexdata.support.windows;bison-2.4.1-setup.exe">
976 Index Data's Windows support area
981 <term><literal>HAVE_ICONV</literal>,
982 <literal>ICONV_DIR</literal></term>
984 If <literal>HAVE_ICONV</literal> is set to 1, YAZ is compiled
985 with iconv support. In this configuration, set
986 <literal>ICONV_DIR</literal> to the iconv source directory.
990 <term><literal>HAVE_LIBXML2</literal>,
991 <literal>LIBXML2_DIR</literal></term>
994 If <literal>HAVE_LIBXML2</literal> is set to 1, YAZ is compiled
995 with SRU support. In this configuration, set
996 <literal>LIBXML2_DIR</literal> to the
997 <ulink url="&url.libxml2;">libxml2</ulink> source directory.
1000 You can get pre-compiled Libxml2+Libxslt DLLs and headers from
1001 <ulink url="&url.libxml2.download.windows;">here</ulink>.
1002 Should you with to compile those libraries yourself, refer to
1003 to <xref linkend="installation.windows.libxml2"/>
1008 <term><literal>HAVE_LIBXSLT</literal>,
1009 <literal>LIBXSLT_DIR</literal></term>
1012 If <literal>HAVE_LIBXSLT</literal> is set to 1, YAZ is compiled
1013 with XSLT support. In this configuration, set
1014 <literal>LIBXSLT_DIR</literal> to the
1015 <ulink url="&url.libxslt;">libxslt</ulink> source directory.
1019 libxslt depends on libxml2.
1025 <term><literal>HAVE_ICU</literal>,
1026 <literal>ICU_DIR</literal></term>
1029 If <literal>HAVE_ICU</literal> is set to 1, YAZ is compiled
1030 with <ulink url="&url.icu;">ICU</ulink> support.
1031 In this configuration, set
1032 <literal>ICU_DIR</literal> to the
1033 <ulink url="&url.icu;">ICU</ulink> source directory.
1036 Pre-compiled ICU libraries for various versions of Visual Studio
1038 <ulink url="http://www.npcglib.org/~stathis/blog/precompiled-icu/">
1040 </ulink> or from Index Data's <ulink
1041 url="&url.indexdata.support.windows;">Windows support site</ulink>.
1048 When satisfied with the settings in the makefile, type
1055 If the <filename>nmake</filename> command is not found on your system
1056 you probably haven't defined the environment variables required to
1057 use that tool. To fix that, find and run the batch file
1058 <filename>vcvars32.bat</filename>. You need to run it from within
1059 the command prompt or set the environment variables "globally";
1060 otherwise it doesn't work.
1064 If you wish to recompile &yaz; - for example if you modify
1065 settings in the <filename>makefile</filename> you can delete
1066 object files, etc by running.
1072 The following files are generated upon successful compilation:
1075 <term><filename>bin/yaz&soversion;.dll</filename> /
1076 <filename>bin/yaz&soversion;d.dll</filename></term>
1078 &yaz; Release/Debug DLL.
1082 <term><filename>lib/yaz&soversion;.lib</filename> /
1083 <filename>lib/yaz&soversion;d.lib</filename></term>
1085 Import library for <filename>yaz&soversion;.dll</filename> /
1086 <filename>yaz&soversion;d.dll</filename>.
1090 <term><filename>bin/yaz_cond&soversion;.dll</filename> /
1091 <filename>bin/yaz_cond&soversion;d.dll</filename></term>
1093 Release/Debug DLL for condition variable utilities (condvar.c).
1097 <term><filename>lib/yaz_cond&soversion;.lib</filename> /
1098 <filename>lib/yaz_cond&soversion;d.lib</filename></term>
1100 Import library for <filename>yaz_cond&soversion;.dll</filename> /
1101 <filename>yaz_cond&soversion;d.dll</filename>.
1105 <term><filename>bin/yaz_icu&soversion;.dll</filename> /
1106 <filename>bin/yaz_icu&soversion;d.dll</filename></term>
1108 Release/Debug DLL for the ICU wrapper utility.
1109 Only build if HAVE_ICU is 1.
1113 <term><filename>lib/yaz_icu&soversion;.lib</filename> /
1114 <filename>lib/yaz_icu&soversion;d.lib</filename></term>
1116 Import library for <filename>yaz_icu&soversion;.dll</filename> /
1117 <filename>yaz_icu&soversion;d.dll</filename>.
1121 <term><filename>bin/yaz-ztest.exe</filename></term>
1123 Z39.50 multi-threaded test/example server. It's a WIN32
1124 console application.
1128 <term><filename>bin/yaz-client.exe</filename></term>
1130 &yaz; Z39.50 client application. It's a WIN32 console application.
1131 See chapter <link linkend="yaz-client">YAZ client</link> for more
1136 <term><filename>bin/yaz-icu.exe</filename></term>
1137 <listitem><para>This program exposes the ICU wrapper library if that
1138 is enabled for YAZ. Only if ICU is available this program is
1143 <term><filename>bin/zoomsh.exe</filename></term>
1145 Simple console application implemented on top of the
1146 <link linkend="zoom">ZOOM</link> functions.
1147 The application is a command line shell that allows you to enter
1148 simple commands to perform ZOOM operations.
1152 <term><filename>bin/zoomtst1.exe</filename>,
1153 <filename>bin/zoomtst2.exe</filename>, ..</term>
1155 Several small applications that demonstrate the ZOOM API.
1162 <sect2 id="installation-linking-yaz-win32">
1163 <title>How to make apps using YAZ on Windows</title>
1165 This section will go though the process of linking your Windows
1166 applications with &yaz;.
1169 Some people are confused by the fact that we use the nmake
1170 tool to build &yaz;. They think they have to do that too - in order
1171 to make their Windows applications work with &yaz;. The good news is that
1172 you don't have to. You can use the integrated environment of
1173 Visual Studio if desired for your own application.
1176 When setting up a project or Makefile you have to set the following:
1179 <term>include path</term>
1181 Set it to the <filename>include</filename> directory of &yaz;.
1185 <term>import library <filename>yaz&soversion;.lib</filename></term>
1187 You must link with this library. It's located in the
1188 sub directory <filename>lib</filename> of &yaz;.
1189 If you want to link with the debug version of &yaz;, you must
1190 link against <filename>yaz&soversion;d.lib</filename> instead.
1194 <term>dynamic link library
1195 <filename>yaz&soversion;.dll</filename>
1198 This DLL must be in your execution path when you invoke
1199 your application. Specifically, you should distribute this
1200 DLL with your application.
1207 <sect2 id="installation.windows.libxml2">
1208 <title>Compiling Libxml2 and Libxslt on windows</title>
1210 Download libxml2 and Libxslt source and unpack it.
1211 In the example below we install Libxml2 2.9.2 and Libxslt 1.1.28
1212 for 32-bit, so we use the destination directories
1213 libxml2.2.9.2.win32 and libxslt-1.1.28.win32 to reflect both
1214 version and architecture.
1217 cscript configure.js prefix=c:\libxml2-2.9.2.win32 iconv=no
1224 There's an error in <filename>configure.js</filename> for Libxml2 2.9.2.
1225 Line 17 should be assigned to <filename>configure.ac</filename>
1226 rather than <filename>configure.in</filename>.
1230 For Libxslt it is similar. We must ensure that compilation of
1231 Libxslt links against the already installed libxml2.
1234 cscript configure.js prefix=c:\libxslt-1.1.28.win32 iconv=no \
1235 lib=c:\libxml2-2.9.2.win32\lib \
1236 include=c:\libxml2-2.9.2.win32\include\libxml2
1246 ### Still to document:
1247 ZOOM_connection_errcode(c)
1248 ZOOM_connection_errmsg(c)
1249 ZOOM_connection_addinfo(c)
1250 ZOOM_connection_addinfo(c)
1251 ZOOM_connection_diagset(c);
1252 ZOOM_connection_save_apdu_wrbuf
1253 ZOOM_diag_str(error)
1254 ZOOM_resultset_record_immediate(s, pos)
1255 ZOOM_resultset_cache_reset(r)
1256 ZOOM_options_set_callback(opt, function, handle)
1257 ZOOM_options_create_with_parent2(parent1, parent2)
1258 ZOOM_options_getl(opt, name, len)
1259 ZOOM_options_setl(opt, name, value, len)
1260 ZOOM_options_get_bool(opt, name, defa)
1261 ZOOM_options_get_int(opt, name, defa)
1262 ZOOM_options_set_int(opt, name, value)
1267 &zoom; is an acronym for 'Z39.50 Object-Orientation Model' and is
1268 an initiative started by Mike Taylor (Mike is from the UK, which
1269 explains the peculiar name of the model). The goal of &zoom; is to
1270 provide a common Z39.50 client API not bound to a particular
1271 programming language or toolkit.
1274 From YAZ version 2.1.12, <ulink url="&url.sru;">SRU</ulink> is supported.
1275 You can make SRU ZOOM connections by specifying scheme
1276 <literal>http://</literal> for the hostname for a connection.
1277 The dialect of SRU used is specified by the value of the
1278 connection's <literal>sru</literal> option, which may be SRU over
1279 HTTP GET (<literal>get</literal>),
1280 SRU over HTTP POST (<literal>post</literal>), (SRU over
1281 SOAP) (<literal>soap</literal>) or <literal>solr</literal>
1282 (<ulink url="&url.solr;">Solr</ulink> Web Service).
1283 Using the facility for embedding options in target strings, a
1284 connection can be forced to use SRU rather the SRW (the default) by
1285 prefixing the target string with <literal>sru=get,</literal>, like this:
1286 <literal>sru=get,http://sru.miketaylor.org.uk:80/sru.pl</literal>
1289 <ulink url="&url.solr;">Solr</ulink> protocol support was added to
1290 YAZ in version 4.1.0, as a dialect of a SRU protocol, since both are
1291 HTTP based protocols.
1294 The lack of a simple Z39.50 client API for &yaz; has become more
1295 and more apparent over time. So when the first &zoom; specification
1297 an implementation for &yaz; was quickly developed. For the first time, it is
1298 now as easy (or easier!) to develop clients as it is to develop
1299 servers with &yaz;. This
1300 chapter describes the &zoom; C binding. Before going further, please
1301 reconsider whether C is the right programming language for the job.
1302 There are other language bindings available for &yaz;, and still
1304 are in active development. See the
1305 <ulink url="&url.zoom;">ZOOM web-site</ulink> for
1309 In order to fully understand this chapter you should read and
1310 try the example programs <literal>zoomtst1.c</literal>,
1311 <literal>zoomtst2.c</literal>, .. in the <literal>zoom</literal>
1315 The C language misses features found in object oriented languages
1316 such as C++, Java, etc. For example, you'll have to manually,
1317 destroy all objects you create, even though you may think of them as
1318 temporary. Most objects have a <literal>_create</literal> - and a
1319 <literal>_destroy</literal> variant.
1320 All objects are in fact pointers to internal stuff, but you don't see
1321 that because of typedefs. All destroy methods should gracefully ignore a
1322 <literal>NULL</literal> pointer.
1325 In each of the sections below you'll find a sub section called
1326 protocol behavior, that describes how the API maps to the Z39.50
1329 <sect1 id="zoom-connections">
1330 <title>Connections</title>
1331 <para>The Connection object is a session with a target.
1334 #include <yaz/zoom.h>
1336 ZOOM_connection ZOOM_connection_new(const char *host, int portnum);
1338 ZOOM_connection ZOOM_connection_create(ZOOM_options options);
1340 void ZOOM_connection_connect(ZOOM_connection c, const char *host,
1342 void ZOOM_connection_destroy(ZOOM_connection c);
1345 Connection objects are created with either function
1346 <function>ZOOM_connection_new</function> or
1347 <function>ZOOM_connection_create</function>.
1348 The former creates and automatically attempts to establish a network
1349 connection with the target. The latter doesn't establish
1350 a connection immediately, thus allowing you to specify options
1351 before establishing network connection using the function
1352 <function>ZOOM_connection_connect</function>.
1353 If the port number, <literal>portnum</literal>, is zero, the
1354 <literal>host</literal> is consulted for a port specification.
1355 If no port is given, 210 is used. A colon denotes the beginning of
1356 a port number in the host string. If the host string includes a
1357 slash, the following part specifies a database for the connection.
1360 You can prefix the host with a scheme followed by colon. The
1361 default scheme is <literal>tcp</literal> (Z39.50 protocol).
1362 The scheme <literal>http</literal> selects SRU/get over HTTP by default,
1363 but can overridden to use SRU/post, SRW, and the Solr protocol.
1366 You can prefix the scheme-qualified host-string with one or more
1368 <literal><parameter>key</parameter>=<parameter>value</parameter></literal>
1369 sequences, each of which represents an option to be set into the
1370 connection structure <emphasis>before</emphasis> the
1371 protocol-level connection is forged and the initialization
1372 handshake takes place. This facility can be used to provide
1373 authentication credentials, as in host-strings such as:
1374 <literal>user=admin,password=halfAm4n,tcp:localhost:8017/db</literal>
1377 Connection objects should be destroyed using the function
1378 <function>ZOOM_connection_destroy</function>.
1381 void ZOOM_connection_option_set(ZOOM_connection c,
1382 const char *key, const char *val);
1384 void ZOOM_connection_option_setl(ZOOM_connection c,
1386 const char *val, int len);
1388 const char *ZOOM_connection_option_get(ZOOM_connection c,
1390 const char *ZOOM_connection_option_getl(ZOOM_connection c,
1395 The functions <function>ZOOM_connection_option_set</function> and
1396 <function>ZOOM_connection_option_setl</function> allows you to
1397 set an option given by <parameter>key</parameter> to the value
1398 <parameter>value</parameter> for the connection.
1399 For <function>ZOOM_connection_option_set</function>, the
1400 value is assumed to be a 0-terminated string. Function
1401 <function>ZOOM_connection_option_setl</function> specifies a
1402 value of a certain size (len).
1405 Functions <function>ZOOM_connection_option_get</function> and
1406 <function>ZOOM_connection_option_getl</function> returns
1407 the value for an option given by <parameter>key</parameter>.
1409 <table id="zoom-connection-options" frame="top">
1410 <title>ZOOM Connection Options</title>
1412 <colspec colwidth="4*" colname="name"></colspec>
1413 <colspec colwidth="7*" colname="description"></colspec>
1414 <colspec colwidth="3*" colname="default"></colspec>
1417 <entry>Option</entry>
1418 <entry>Description</entry>
1419 <entry>Default</entry>
1424 implementationName</entry><entry>Name of Your client
1425 </entry><entry>none</entry></row>
1427 user</entry><entry>Authentication user name
1428 </entry><entry>none</entry></row>
1430 group</entry><entry>Authentication group name
1431 </entry><entry>none</entry></row>
1433 password</entry><entry>Authentication password.
1434 </entry><entry>none</entry></row>
1436 authenticationMode</entry><entry>How authentication is encoded.
1437 </entry><entry>basic</entry></row>
1439 host</entry><entry>Target host. This setting is "read-only".
1440 It's automatically set internally when connecting to a target.
1441 </entry><entry>none</entry></row>
1443 proxy</entry><entry>Proxy host. If set, the logical host
1444 is encoded in the otherInfo area of the Z39.50 Init PDU
1445 with OID 1.2.840.10003.10.1000.81.1.
1446 </entry><entry>none</entry></row>
1448 clientIP</entry><entry>Client IP. If set, is
1449 encoded in the otherInfo area of a Z39.50 PDU with OID
1450 1.2.840.10003.10.1000.81.3. Holds the original IP addreses
1451 of a client. Is used if ZOOM is used in a gateway of some sort.
1452 </entry><entry>none</entry></row>
1454 async</entry><entry>If true (1) the connection operates in
1455 asynchronous operation which means that all calls are non-blocking
1457 <link linkend="zoom.events"><function>ZOOM_event</function></link>.
1458 </entry><entry>0</entry></row>
1460 maximumRecordSize</entry><entry> Maximum size of single record.
1461 </entry><entry>1 MB</entry></row>
1463 preferredMessageSize</entry><entry> Maximum size of multiple records.
1464 </entry><entry>1 MB</entry></row>
1466 lang</entry><entry> Language for negotiation.
1467 </entry><entry>none</entry></row>
1469 charset</entry><entry> Character set for negotiation.
1470 </entry><entry>none</entry></row>
1472 serverImplementationId</entry><entry>
1473 Implementation ID of server. (The old targetImplementationId
1474 option is also supported for the benefit of old applications.)
1475 </entry><entry>none</entry></row>
1477 targetImplementationName</entry><entry>
1478 Implementation Name of server. (The old
1479 targetImplementationName option is also supported for the
1480 benefit of old applications.)
1481 </entry><entry>none</entry></row>
1483 serverImplementationVersion</entry><entry>
1484 Implementation Version of server. (the old
1485 targetImplementationVersion option is also supported for the
1486 benefit of old applications.)
1487 </entry><entry>none</entry></row>
1489 databaseName</entry><entry>One or more database names
1490 separated by character plus (<literal>+</literal>), which is to
1491 be used by subsequent search requests on this Connection.
1492 </entry><entry>Default</entry></row>
1494 piggyback</entry><entry>True (1) if piggyback should be
1495 used in searches; false (0) if not.
1496 </entry><entry>1</entry></row>
1498 smallSetUpperBound</entry><entry>If hits is less than or equal to this
1499 value, then target will return all records using small element set name
1500 </entry><entry>0</entry></row>
1502 largeSetLowerBound</entry><entry>If hits is greater than this
1503 value, the target will return no records.
1504 </entry><entry>1</entry></row>
1506 mediumSetPresentNumber</entry><entry>This value represents
1507 the number of records to be returned as part of a search when
1508 hits is less than or equal to large set lower bound and if hits
1509 is greater than small set upper bound.
1510 </entry><entry>0</entry></row>
1512 smallSetElementSetName</entry><entry>
1513 The element set name to be used for small result sets.
1514 </entry><entry>none</entry></row>
1516 mediumSetElementSetName</entry><entry>
1517 The element set name to be used for medium-sized result sets.
1518 </entry><entry>none</entry></row>
1520 init_opt_search, init_opt_present, init_opt_delSet, etc.</entry><entry>
1521 After a successful Init, these options may be interrogated to
1522 discover whether the server claims to support the specified
1524 </entry><entry>none</entry></row>
1526 <entry>sru</entry><entry>
1527 SRU/Solr transport type. Must be either <literal>soap</literal>,
1528 <literal>get</literal>, <literal>post</literal>, or
1529 <literal>solr</literal>.
1530 </entry><entry>soap</entry></row>
1532 sru_version</entry><entry>
1533 SRU/SRW version. Should be <literal>1.1</literal>, or
1534 <literal>1.2</literal>. This is, prior to connect, the version
1535 to offer (highest version). And following connect (in fact
1536 first operation), holds the negotiated version with the server
1537 (same or lower version).
1538 </entry><entry>1.2</entry></row>
1539 <row id="zoom.extraArgs.option"><entry>
1540 extraArgs</entry><entry>
1541 Extra arguments for SRU/Solr URLs. The value must be
1542 URL encoded already.
1543 </entry><entry></entry></row>
1544 <row id="zoom.facets.option"><entry>
1545 facets</entry><entry>
1546 Requested or recommended facets may be given before a search is sent.
1547 The value of this setting is described in <xref linkend="facets"/>
1548 For inspection of the facets returned, refer to the functions
1549 described in <xref linkend="zoom.facets"/>.
1550 </entry><entry>none</entry></row>
1552 apdulog</entry><entry>
1553 If set to a true value such as "1", a log of low-level
1554 protocol packets is emitted on standard error stream. This
1555 can be very useful for debugging.
1556 </entry><entry>0</entry></row>
1558 saveAPDU</entry><entry>
1559 If set to a true value such as "1", a log of low-level
1560 protocol packets is saved. The log can be retrieved by reading
1561 option APDU. Setting saveAPDU always has the side effect of
1562 resetting the currently saved log. This setting is
1563 <emphasis>write-only</emphasis>. If read, NULL will be returned.
1564 It is only recognized in
1565 <function>ZOOM_connection_option_set</function>.
1566 </entry><entry>0</entry></row>
1569 Returns the log of protocol packets. Will be empty if logging
1570 is not enabled (see saveAPDU above). This setting is
1571 <emphasis>read-only</emphasis>. It is only recognized if used
1572 in call to <function>ZOOM_connection_option_get</function> or
1573 <function>ZOOM_connection_option_getl</function>.
1574 </entry><entry></entry></row>
1576 memcached</entry><entry>
1577 If given and non-empty,
1578 <ulink url="&url.libmemcached;">libMemcached</ulink>
1579 will be configured for the connection.
1580 This option is inspected by ZOOM when a connection is established.
1581 If the <literal>memcached</literal> option is given
1582 and YAZ is compiled without libMemcached support, an internal
1583 diagnostic (10018) will be thrown.
1584 libMemcached support is available for YAZ 5.0.13 or later. If this
1585 option is supplied for an earlier version of YAZ, it is
1586 <emphasis>ignored</emphasis>.
1587 The value of this option is a list options - each is of the
1588 form <literal>--name=value</literal>.
1589 Option <literal>--server=</literal>host[:port] specifies a memcached
1590 server. It may be repeated for multiple memcached servers.
1591 Option <literal>--expire=</literal>seconds sets expiry time in seconds
1592 for how long result sets are to be cached.
1593 </entry><entry>none</entry></row>
1595 redis</entry><entry>
1596 If given and non-empty,
1597 a <ulink url="&url.redis;">redis</ulink> context will be created
1599 This option is inspected by ZOOM when a connection is established.
1600 If the <literal>redis</literal> option is given
1601 and YAZ is compiled without redis support, an internal
1602 diagnostic (10018) will be thrown.
1603 redis support is available for YAZ 5.2.0 or later. If this
1604 option is supplied for an earlier version of YAZ, it is
1605 <emphasis>ignored</emphasis>.
1606 The value of this option is a set of options, similar to that
1607 of the memcached setting. At this stage only --server=host[:port]
1608 and --expire=seconds are supported.
1609 </entry><entry>none</entry></row>
1614 If either option <literal>lang</literal> or <literal>charset</literal>
1616 <ulink url="&url.z39.50.charneg;">
1617 Character Set and Language Negotiation</ulink> is in effect.
1620 int ZOOM_connection_error(ZOOM_connection c, const char **cp,
1621 const char **addinfo);
1622 int ZOOM_connection_error_x(ZOOM_connection c, const char **cp,
1623 const char **addinfo, const char **dset);
1626 Function <function>ZOOM_connection_error</function> checks for
1627 errors for the last operation(s) performed. The function returns
1628 zero if no errors occurred; non-zero otherwise indicating the error.
1629 Pointers <parameter>cp</parameter> and <parameter>addinfo</parameter>
1630 holds messages for the error and additional-info if passed as
1631 non-<literal>NULL</literal>. Function
1632 <function>ZOOM_connection_error_x</function> is an extended version
1633 of <function>ZOOM_connection_error</function> that is capable of
1634 returning name of diagnostic set in <parameter>dset</parameter>.
1636 <sect2 id="zoom-connection-z39.50">
1637 <title>Z39.50 Protocol behavior</title>
1639 The calls <function>ZOOM_connection_new</function> and
1640 <function>ZOOM_connection_connect</function> establishes a TCP/IP
1641 connection and sends an Initialize Request to the target if
1642 possible. In addition, the calls wait for an Initialize Response
1643 from the target and the result is inspected (OK or rejected).
1646 If <literal>proxy</literal> is set then the client will establish
1647 a TCP/IP connection with the peer as specified by the
1648 <literal>proxy</literal> host and the hostname as part of the
1649 connect calls will be set as part of the Initialize Request.
1650 The proxy server will then "forward" the PDU's transparently
1651 to the target behind the proxy.
1654 For the authentication parameters, if option <literal>user</literal>
1655 is set and both options <literal>group</literal> and
1656 <literal>pass</literal> are unset, then Open style
1657 authentication is used (Version 2/3) in which case the username
1658 is usually followed by a slash, then by a password.
1659 If either <literal>group</literal>
1660 or <literal>pass</literal> is set then idPass authentication
1661 (Version 3 only) is used. If none of the options are set, no
1662 authentication parameters are set as part of the Initialize Request
1666 When option <literal>async</literal> is 1, it really means that
1667 all network operations are postponed (and queued) until the
1668 function <literal>ZOOM_event</literal> is invoked. When doing so
1669 it doesn't make sense to check for errors after
1670 <literal>ZOOM_connection_new</literal> is called since that
1671 operation "connecting - and init" is still incomplete and the
1672 API cannot tell the outcome (yet).
1675 <sect2 id="zoom.sru.init.behavior">
1676 <title>SRU/Solr Protocol behavior</title>
1678 The HTTP based protocols (SRU, SRW, Solr) do not feature an
1679 Inititialize Request, so the connection phase merely establishes a
1680 TCP/IP connection with the HTTP server.
1682 <para>Most of the ZOOM connection options do not
1683 affect SRU/Solr and they are ignored. However, future versions
1684 of &yaz; might honor <literal>implementationName</literal> and
1685 put that as part of User-Agent header for HTTP requests.
1688 The <literal>charset</literal> is used in the Content-Type header
1692 Setting <literal>authentcationMode</literal> specifies how
1693 authentication parameters are encoded for HTTP. The default is
1694 "<literal>basic</literal>" where <literal>user</literal> and
1695 <literal>password</literal> are encoded by using HTTP basic
1699 If <literal>authentcationMode</literal> is "<literal>url</literal>", then
1700 user and password are encoded in the URL by parameters
1701 <literal>x-username</literal> and <literal>x-password</literal> as
1702 given by the SRU standard.
1706 <sect1 id="zoom.query">
1707 <title>Queries</title>
1709 Query objects represents queries.
1712 ZOOM_query ZOOM_query_create(void);
1714 void ZOOM_query_destroy(ZOOM_query q);
1716 int ZOOM_query_prefix(ZOOM_query q, const char *str);
1718 int ZOOM_query_cql(ZOOM_query s, const char *str);
1720 int ZOOM_query_sortby(ZOOM_query q, const char *criteria);
1722 int ZOOM_query_sortby2(ZOOM_query q, const char *strategy,
1723 const char *criteria);
1726 Create query objects using <function>ZOOM_query_create</function>
1727 and destroy them by calling <function>ZOOM_query_destroy</function>.
1728 RPN-queries can be specified in <link linkend="PQF">PQF</link>
1729 notation by using the
1730 function <function>ZOOM_query_prefix</function>.
1731 The <function>ZOOM_query_cql</function> specifies a CQL
1732 query to be sent to the server/target.
1733 More query types will be added in future versions of &yaz;, such as
1734 <link linkend="CCL">CCL</link> to RPN-mapping, native CCL query,
1735 etc. In addition to a search, a sort criteria may be set. Function
1736 <function>ZOOM_query_sortby</function> enables Z39.50 sorting and
1737 it takes sort criteria using the same string notation as
1738 yaz-client's <link linkend="sortspec">sort command</link>.
1740 <para id="zoom.query.sortby2">
1741 <function>ZOOM_query_sortby2</function> is similar to
1742 <function>ZOOM_query_sortby</function> but allows a strategy for
1743 sorting. The reason for the strategy parameter is that some
1744 protocols offer multiple ways of performing sorting.
1745 For example, Z39.50 has the standard sort, which is performed after
1746 search on an existing result set.
1747 It's also possible to use CQL in Z39.50 as the query type and use
1748 CQL's SORTBY keyword. Finally, Index Data's
1749 Zebra server also allows sorting to be specified as part of RPN (Type 7).
1751 <table id="zoom-sort-strategy" frame="top">
1752 <title>ZOOM sort strategy</title>
1754 <colspec colwidth="2*" colname="name"/>
1755 <colspec colwidth="5*" colname="description"/>
1759 <entry>Description</entry>
1764 <entry>z39.50</entry><entry>Z39.50 resultset sort</entry>
1767 <entry>type7</entry><entry>Sorting embedded in RPN(Type-7)</entry>
1770 <entry>cql</entry><entry>CQL SORTBY</entry>
1773 <entry>sru11</entry><entry>SRU sortKeys parameter</entry>
1776 <entry>solr</entry><entry>Solr sort</entry>
1779 <entry>embed</entry><entry>type7 for Z39.50, cql for SRU,
1780 solr for Solr protocol</entry>
1786 <sect1 id="zoom.resultsets"><title>Result sets</title>
1788 The result set object is a container for records returned from
1792 ZOOM_resultset ZOOM_connection_search(ZOOM_connection, ZOOM_query q);
1794 ZOOM_resultset ZOOM_connection_search_pqf(ZOOM_connection c,
1796 void ZOOM_resultset_destroy(ZOOM_resultset r);
1799 Function <function>ZOOM_connection_search</function> creates
1800 a result set, given a connection and query.
1801 Destroy a result set by calling
1802 <function>ZOOM_resultset_destroy</function>.
1803 Simple clients using PQF only, may use the function
1804 <function>ZOOM_connection_search_pqf</function> in which case
1805 creating query objects is not necessary.
1808 void ZOOM_resultset_option_set(ZOOM_resultset r,
1809 const char *key, const char *val);
1811 const char *ZOOM_resultset_option_get(ZOOM_resultset r, const char *key);
1813 size_t ZOOM_resultset_size(ZOOM_resultset r);
1816 Functions <function>ZOOM_resultset_options_set</function> and
1817 <function>ZOOM_resultset_get</function> sets and gets an option
1818 for a result set similar to <function>ZOOM_connection_option_get</function>
1819 and <function>ZOOM_connection_option_set</function>.
1822 The number of hits, also called result-count, is returned by
1823 function <function>ZOOM_resultset_size</function>.
1825 <table id="zoom.resultset.options"
1826 frame="top"><title>ZOOM Result set Options</title>
1828 <colspec colwidth="4*" colname="name"></colspec>
1829 <colspec colwidth="7*" colname="description"></colspec>
1830 <colspec colwidth="2*" colname="default"></colspec>
1833 <entry>Option</entry>
1834 <entry>Description</entry>
1835 <entry>Default</entry>
1840 start</entry><entry>Offset of first record to be
1841 retrieved from target. First record has offset 0 unlike the
1842 protocol specifications where first record has position 1.
1843 This option affects ZOOM_resultset_search and
1844 ZOOM_resultset_search_pqf and must be set before any of
1845 these functions are invoked. If a range of
1846 records must be fetched manually after search,
1847 function ZOOM_resultset_records should be used.
1848 </entry><entry>0</entry></row>
1850 count</entry><entry>Number of records to be retrieved.
1851 This option affects ZOOM_resultset_search and
1852 ZOOM_resultset_search_pqf and must be set before any of
1853 these functions are invoked.
1854 </entry><entry>0</entry></row>
1856 presentChunk</entry><entry>The number of records to be
1857 requested from the server in each chunk (present request). The
1858 value 0 means to request all the records in a single chunk.
1859 (The old <literal>step</literal>
1860 option is also supported for the benefit of old applications.)
1861 </entry><entry>0</entry></row>
1863 elementSetName</entry><entry>Element-Set name of records.
1864 Most targets should honor element set name <literal>B</literal>
1865 and <literal>F</literal> for brief and full respectively.
1866 </entry><entry>none</entry></row>
1868 preferredRecordSyntax</entry><entry>Preferred Syntax, such as
1869 <literal>USMARC</literal>, <literal>SUTRS</literal>, etc.
1870 </entry><entry>none</entry></row>
1872 schema</entry><entry>Schema for retrieval, such as
1873 <literal>Gils-schema</literal>, <literal>Geo-schema</literal>, etc.
1874 </entry><entry>none</entry></row>
1876 setname</entry><entry>Name of Result Set (Result Set ID).
1877 If this option isn't set, the ZOOM module will automatically
1878 allocate a result set name.
1879 </entry><entry>default</entry></row>
1881 rpnCharset</entry><entry>Character set for RPN terms.
1882 If this is set, ZOOM C will assume that the ZOOM application is
1883 running UTF-8. Terms in RPN queries are then converted to the
1884 rpnCharset. If this is unset, ZOOM C will not assume any encoding
1885 of RPN terms and no conversion is performed.
1886 </entry><entry>none</entry></row>
1891 For servers that support Search Info report, the following
1892 options may be read using <function>ZOOM_resultset_get</function>.
1893 This detailed information is read after a successful search has
1897 This information is a list of of items, where each item is
1898 information about a term or subquery. All items in the list
1900 <literal>SearchResult.</literal><replaceable>no</replaceable>
1901 where no presents the item number (0=first, 1=second).
1902 Read <literal>searchresult.size</literal> to determine the
1905 <table id="zoom.search.info.report.options"
1906 frame="top"><title>Search Info Report Options</title>
1908 <colspec colwidth="4*" colname="name"></colspec>
1909 <colspec colwidth="7*" colname="description"></colspec>
1912 <entry>Option</entry>
1913 <entry>Description</entry>
1918 <entry>searchresult.size</entry>
1920 number of search result entries. This option is non-existent
1921 if no entries are returned by the server.
1925 <entry>searchresult.<replaceable>no</replaceable>.id</entry>
1926 <entry>sub query ID</entry>
1929 <entry>searchresult.<replaceable>no</replaceable>.count</entry>
1930 <entry>result count for item (number of hits)</entry>
1933 <entry>searchresult.<replaceable>no</replaceable>.subquery.term</entry>
1934 <entry>subquery term</entry>
1938 searchresult.<replaceable>no</replaceable>.interpretation.term
1940 <entry>interpretation term</entry>
1944 searchresult.<replaceable>no</replaceable>.recommendation.term
1946 <entry>recommendation term</entry>
1951 <sect2 id="zoom.z3950.resultset.sort">
1952 <title>Z39.50 Result-set Sort</title>
1954 void ZOOM_resultset_sort(ZOOM_resultset r,
1955 const char *sort_type, const char *sort_spec);
1957 int ZOOM_resultset_sort1(ZOOM_resultset r,
1958 const char *sort_type, const char *sort_spec);
1961 <function>ZOOM_resultset_sort</function> and
1962 <function>ZOOM_resultset_sort1</function> both sort an existing
1963 result-set. The sort_type parameter is not used. Set it to "yaz".
1964 The sort_spec is same notation as ZOOM_query_sortby and identical
1965 to that offered by yaz-client's
1966 <link linkend="sortspec">sort command</link>.
1969 These functions only work for Z39.50. Use the more generic utility
1970 <link linkend="zoom.query.sortby2">
1971 <function>ZOOM_query_sortby2</function></link>
1972 for other protocols (and even Z39.50).
1975 <sect2 id="zoom.z3950.resultset.behavior">
1976 <title>Z39.50 Protocol behavior</title>
1978 The creation of a result set involves at least a SearchRequest
1979 - SearchResponse protocol handshake. Following that, if a sort
1980 criteria was specified as part of the query, a SortRequest -
1981 SortResponse handshake takes place. Note that it is necessary to
1982 perform sorting before any retrieval takes place, so no records will
1983 be returned from the target as part of the SearchResponse because these
1984 would be unsorted. Hence, piggyback is disabled when sort criteria
1985 are set. Following Search - and a possible sort - Retrieval takes
1986 place - as one or more Present Requests/Response pairs being
1990 The API allows for two different modes for retrieval. A high level
1991 mode which is somewhat more powerful and a low level one.
1992 The low level is enabled when searching on a Connection object
1993 for which the settings
1994 <literal>smallSetUpperBound</literal>,
1995 <literal>mediumSetPresentNumber</literal> and
1996 <literal>largeSetLowerBound</literal> are set. The low level mode
1997 thus allows you to precisely set how records are returned as part
1998 of a search response as offered by the Z39.50 protocol.
1999 Since the client may be retrieving records as part of the
2000 search response, this mode doesn't work well if sorting is used.
2003 The high-level mode allows you to fetch a range of records from
2004 the result set with a given start offset. When you use this mode
2005 the client will automatically use piggyback if that is possible
2006 with the target, and perform one or more present requests as needed.
2007 Even if the target returns fewer records as part of a present response
2008 because of a record size limit, etc. the client will repeat sending
2009 present requests. As an example, if option <literal>start</literal>
2010 is 0 (default) and <literal>count</literal> is 4, and
2011 <literal>piggyback</literal> is 1 (default) and no sorting criteria
2012 is specified, then the client will attempt to retrieve the 4
2013 records as part the search response (using piggyback). On the other
2014 hand, if either <literal>start</literal> is positive or if
2015 a sorting criteria is set, or if <literal>piggyback</literal>
2016 is 0, then the client will not perform piggyback but send Present
2020 If either of the options <literal>mediumSetElementSetName</literal> and
2021 <literal>smallSetElementSetName</literal> are unset, the value
2022 of option <literal>elementSetName</literal> is used for piggyback
2023 searches. This means that for the high-level mode you only have
2024 to specify one elementSetName option rather than three.
2027 <sect2 id="zoom.sru.resultset.behavior">
2028 <title>SRU Protocol behavior</title>
2030 Current version of &yaz; does not take advantage of a result set id
2031 returned by the SRU server. Future versions might do, however.
2032 Since the ZOOM driver does not save result set IDs, any
2033 present (retrieval) is transformed to a SRU SearchRetrieveRequest
2034 with same query but, possibly, different offsets.
2037 Option <literal>schema</literal> specifies SRU schema
2038 for retrieval. However, options <literal>elementSetName</literal> and
2039 <literal>preferredRecordSyntax</literal> are ignored.
2042 Options <literal>start</literal> and <literal>count</literal>
2043 are supported by SRU.
2044 The remaining options
2045 <literal>piggyback</literal>,
2046 <literal>smallSetUpperBound</literal>,
2047 <literal>largeSetLowerBound</literal>,
2048 <literal>mediumSetPresentNumber</literal>,
2049 <literal>mediumSetElementSetName</literal>,
2050 <literal>smallSetElementSetName</literal> are
2054 SRU supports CQL queries, <emphasis>not</emphasis> PQF.
2055 If PQF is used, however, the PQF query is transferred anyway
2056 using non-standard element <literal>pQuery</literal> in
2057 SRU SearchRetrieveRequest.
2060 Solr queries need to be done in Solr query format.
2063 Unfortunately, SRU and Solr do not define a database setting. Hence,
2064 <literal>databaseName</literal> is unsupported and ignored.
2065 However, the path part in host parameter for functions
2066 <function>ZOOM_connecton_new</function> and
2067 <function>ZOOM_connection_connect</function> acts as a
2068 database (at least for the &yaz; SRU server).
2072 <sect1 id="zoom.records">
2073 <title>Records</title>
2075 A record object is a retrieval record on the client side -
2076 created from result sets.
2079 void ZOOM_resultset_records(ZOOM_resultset r,
2081 size_t start, size_t count);
2082 ZOOM_record ZOOM_resultset_record(ZOOM_resultset s, size_t pos);
2084 const char *ZOOM_record_get(ZOOM_record rec, const char *type,
2087 int ZOOM_record_error(ZOOM_record rec, const char **msg,
2088 const char **addinfo, const char **diagset);
2090 ZOOM_record ZOOM_record_clone(ZOOM_record rec);
2092 void ZOOM_record_destroy(ZOOM_record rec);
2095 References to temporary records are returned by functions
2096 <function>ZOOM_resultset_records</function> or
2097 <function>ZOOM_resultset_record</function>.
2100 If a persistent reference to a record is desired
2101 <function>ZOOM_record_clone</function> should be used.
2102 It returns a record reference that should be destroyed
2103 by a call to <function>ZOOM_record_destroy</function>.
2106 A single record is returned by function
2107 <function>ZOOM_resultset_record</function> that takes a
2108 position as argument. First record has position zero.
2109 If no record could be obtained <literal>NULL</literal> is returned.
2112 Error information for a record can be checked with
2113 <function>ZOOM_record_error</function> which returns non-zero
2114 (error code) if record is in error, called <emphasis>Surrogate
2115 Diagnostics</emphasis> in Z39.50.
2118 Function <function>ZOOM_resultset_records</function> retrieves
2119 a number of records from a result set. Parameter <literal>start</literal>
2120 and <literal>count</literal> specifies the range of records to
2121 be returned. Upon completion, the array
2122 <literal>recs[0], ..recs[count-1]</literal>
2123 holds record objects for the records. The array of records
2124 <literal>recs</literal> should be allocated prior the call
2125 <function>ZOOM_resultset_records</function>. Note that for those
2126 records that couldn't be retrieved from the target,
2127 <literal>recs[ ..]</literal> is set to <literal>NULL</literal>.
2129 <para id="zoom.record.get">
2130 In order to extract information about a single record,
2131 <function>ZOOM_record_get</function> is provided. The
2132 function returns a pointer to certain record information. The
2133 nature (type) of the pointer depends on the parameter,
2134 <parameter>type</parameter>.
2137 The <parameter>type</parameter> is a string of the format:
2140 <replaceable>format</replaceable>[;charset=<replaceable>from</replaceable>[/<replaceable>opacfrom</replaceable>][,<replaceable>to</replaceable>]][;format=<replaceable>v</replaceable>][;base64=<replaceable>xpath</replaceable>]
2143 If <literal>charset</literal> is given, then <replaceable>from</replaceable>
2144 specifies the character set of the record in its original form
2145 (as returned by the server), <replaceable>to</replaceable> specifies
2146 the output (returned) character set encoding.
2147 If <replaceable>to</replaceable> is omitted, then UTF-8 is assumed.
2148 If charset is not given, then no character set conversion takes place.
2149 OPAC records may be returned in a different
2150 set from the bibliographic MARC record. If this is this the case,
2151 <replaceable>opacfrom</replaceable> should be set to the character set
2152 of the OPAC record part.
2156 The <literal>format</literal> is generic but can only be used to
2157 specify XML indentation when the value <replaceable>v</replaceable>
2158 is 1 (<literal>format=1</literal>).
2161 The <literal>base64</literal> allows a full record to be extracted
2162 from base64-encoded string in an XML document.
2166 Specifying the OPAC record character set requires YAZ 4.1.5 or later.
2169 Specifying the base64 parameter requires YAZ 4.2.35 or later.
2173 The format argument controls whether record data should be XML
2174 pretty-printed (post process operation).
2175 It is enabled only if format value <replaceable>v</replaceable> is
2176 <literal>1</literal> and the record content is XML well-formed.
2179 In addition, for certain types, the length
2180 <literal>len</literal> passed will be set to the size in bytes of
2181 the returned information.
2184 The following are the supported values for <replaceable>form</replaceable>.
2186 <varlistentry><term><literal>database</literal></term>
2187 <listitem><para>The Database of the record is returned
2188 as a C null-terminated string. Return type
2189 <literal>const char *</literal>.
2192 <varlistentry><term><literal>syntax</literal></term>
2193 <listitem><para>The transfer syntax of the record is returned
2194 as a C null-terminated string containing the symbolic name of
2195 the record syntax, e.g. <literal>Usmarc</literal>. Return type
2197 <literal>const char *</literal>.
2200 <varlistentry><term><literal>schema</literal></term>
2201 <listitem><para>The schema of the record is returned
2202 as a C null-terminated string. Return type is
2203 <literal>const char *</literal>.
2206 <varlistentry><term><literal>render</literal></term>
2207 <listitem><para>The record is returned in a display friendly
2208 format. Upon completion, buffer is returned
2209 (type <literal>const char *</literal>) and length is stored in
2210 <literal>*len</literal>.
2213 <varlistentry><term><literal>raw</literal></term>
2214 <listitem><para>The record is returned in the internal
2215 YAZ specific format. For GRS-1, Explain, and others, the
2216 raw data is returned as type
2217 <literal>Z_External *</literal> which is just the type for
2218 the member <literal>retrievalRecord</literal> in
2219 type <literal>NamePlusRecord</literal>.
2220 For SUTRS and octet aligned record (including all MARCs) the
2221 octet buffer is returned and the length of the buffer.
2224 <varlistentry><term><literal>xml</literal></term>
2225 <listitem><para>The record is returned in XML if possible.
2226 SRU, Solr and Z39.50 records with transfer syntax XML are
2227 returned verbatim. MARC records are returned in
2228 <ulink url="&url.marcxml;">
2231 (converted from ISO2709 to MARCXML by YAZ).
2232 OPAC records are also converted to XML and the
2233 bibliographic record is converted to MARCXML (when possible).
2234 GRS-1 records are not supported for this form.
2235 Upon completion, the XML buffer is returned
2236 (type <literal>const char *</literal>) and length is stored in
2237 <literal>*len</literal>.
2240 <varlistentry><term><literal>opac</literal></term>
2241 <listitem><para>OPAC information for record is returned in XML
2242 if an OPAC record is present at the position given. If no
2243 OPAC record is present, a NULL pointer is returned.
2246 <varlistentry><term><literal>txml</literal></term>
2247 <listitem><para>The record is returned in TurboMARC if possible.
2248 SRU and Z39.50 records with transfer syntax XML are
2249 returned verbatim. MARC records are returned in
2250 <link linkend="tools.turbomarc">
2253 (converted from ISO2709 to TurboMARC by YAZ).
2254 Upon completion, the XML buffer is returned
2255 (type <literal>const char *</literal>) and length is stored in
2256 <literal>*len</literal>.
2259 <varlistentry><term><literal>json</literal></term>
2260 <listitem><para>Like xml, but MARC records are converted to
2261 <ulink url="&url.marc_in_json;">MARC-in-JSON</ulink>.
2269 <ulink url="&url.marc21;">MARC21</ulink>
2271 <ulink url="&url.marc8;">MARC-8</ulink>
2272 character set encoding.
2273 An application that wishes to display in Latin-1 would use
2275 render; charset=marc8,iso-8859-1
2278 <sect2 id="zoom.z3950.record.behavior">
2279 <title>Z39.50 Protocol behavior</title>
2281 The functions <function>ZOOM_resultset_record</function> and
2282 <function>ZOOM_resultset_records</function> inspects the client-side
2283 record cache. Records not found in cache are fetched using
2285 The functions may block (and perform network I/O) - even though option
2286 <literal>async</literal> is 1, because they return records objects.
2287 (And there's no way to return records objects without retrieving them!)
2290 There is a trick, however, in the usage of function
2291 <function>ZOOM_resultset_records</function> that allows for
2292 delayed retrieval (and makes it non-blocking). By using
2293 a null pointer for <parameter>recs</parameter> you're indicating
2294 you're not interested in getting records objects
2295 <emphasis>now</emphasis>.
2298 <sect2 id="zoom.sru.record.behavior">
2299 <title>SRU/Solr Protocol behavior</title>
2301 The ZOOM driver for SRU/Solr treats records returned by a SRU/Solr server
2302 as if they where Z39.50 records with transfer syntax XML and
2303 no element set name or database name.
2307 <sect1 id="zoom.facets"><title>Facets</title>
2309 Facet operations is not part of the official ZOOM specification, but
2310 is an Index Data extension for YAZ-based Z39.50 targets,
2311 <ulink url="&url.solr;">Solr</ulink> and SRU 2.0 targets.
2313 Facets may be requestd by the
2314 <link linkend="zoom.facets.option">facets</link> option before a
2316 For inspection of the returned facets, the following functions are
2320 ZOOM_facet_field *ZOOM_resultset_facets(ZOOM_resultset r);
2322 ZOOM_facet_field ZOOM_resultset_get_facet_field(ZOOM_resultset r,
2323 const char *facet_name);
2325 ZOOM_facet_field ZOOM_resultset_get_facet_field_by_index(ZOOM_resultset r,
2328 size_t ZOOM_resultset_facets_size(ZOOM_resultset r);
2330 const char *ZOOM_facet_field_name(ZOOM_facet_field facet_field);
2332 size_t ZOOM_facet_field_term_count(ZOOM_facet_field facet_field);
2334 const char *ZOOM_facet_field_get_term(ZOOM_facet_field facet_field,
2335 size_t idx, int *freq);
2338 References to temporary structures are returned by all functions.
2339 They are only valid as long the Result set is valid.
2340 <function>ZOOM_resultset_get_facet_field</function> or
2341 <function>ZOOM_resultset_get_facet_field_by_index</function>.
2342 <function>ZOOM_resultset_facets</function>.
2343 <function>ZOOM_facet_field_name</function>.
2344 <function>ZOOM_facet_field_get_term</function>.
2346 <para id="zoom.resultset.get_facet_field">
2347 A single Facet field is returned by function
2348 <function>ZOOM_resultset_get_facet_field</function> or
2349 <function>ZOOM_resultset_get_facet_field_by_index</function> that takes
2350 a result set and facet name or positive index respectively. First
2351 facet has position zero. If no facet could be obtained (invalid name
2352 or index out of bounds) <literal>NULL</literal> is returned.
2354 <para id="zoom.resultset.facets">
2355 An array of facets field can be returned by
2356 <function>ZOOM_resultset_facets</function>. The length of the array is
2357 given by <function>ZOOM_resultset_facets_size</function>. The array is
2358 zero-based and the last entry will be at
2359 <function>ZOOM_resultset_facets_size(result_set)</function>-1.
2361 <para id="zoom.resultset.facets_names">
2362 It is possible to interate over facets by name, by calling
2363 <function>ZOOM_resultset_facets_names</function>.
2364 This will return a const array of char * where each string can be used
2365 as parameter for <function>ZOOM_resultset_get_facet_field</function>.
2368 Function <function>ZOOM_facet_field_name</function> gets the request
2369 facet name from a returned facet field.
2372 Function <function>ZOOM_facet_field_get_term</function> returns the
2373 idx'th term and term count for a facet field.
2374 Idx must between 0 and
2375 <function>ZOOM_facet_field_term_count</function>-1, otherwise the
2376 returned reference will be <literal>NULL</literal>. On a valid idx, the
2377 value of the freq reference will be the term count.
2378 The <literal>freq</literal> parameter must be valid pointer to integer.
2381 <sect1 id="zoom.scan"><title>Scan</title>
2383 This section describes an interface for Scan. Scan is not an
2384 official part of the ZOOM model yet. The result of a scan operation
2385 is the <literal>ZOOM_scanset</literal> which is a set of terms
2386 returned by a target.
2390 The Scan interface is supported for both Z39.50, SRU and Solr.
2394 ZOOM_scanset ZOOM_connection_scan(ZOOM_connection c,
2395 const char *startpqf);
2397 ZOOM_scanset ZOOM_connection_scan1(ZOOM_connection c,
2400 size_t ZOOM_scanset_size(ZOOM_scanset scan);
2402 const char *ZOOM_scanset_term(ZOOM_scanset scan, size_t pos,
2403 size_t *occ, size_t *len);
2405 const char *ZOOM_scanset_display_term(ZOOM_scanset scan, size_t pos,
2406 size_t *occ, size_t *len);
2408 void ZOOM_scanset_destroy(ZOOM_scanset scan);
2410 const char *ZOOM_scanset_option_get(ZOOM_scanset scan,
2413 void ZOOM_scanset_option_set(ZOOM_scanset scan, const char *key,
2417 The scan set is created by function
2418 <function>ZOOM_connection_scan</function> which performs a scan
2419 operation on the connection using the specified
2420 <parameter>startpqf</parameter>.
2421 If the operation was successful, the size of the scan set can be
2422 retrieved by a call to <function>ZOOM_scanset_size</function>.
2423 Like result sets, the items are numbered 0..size-1.
2424 To obtain information about a particular scan term, call function
2425 <function>ZOOM_scanset_term</function>. This function takes
2426 a scan set offset <literal>pos</literal> and returns a pointer
2427 to a <emphasis>raw term</emphasis> or <literal>NULL</literal> if
2429 If present, the <literal>occ</literal> and <literal>len</literal>
2430 are set to the number of occurrences and the length
2431 of the actual term respectively.
2432 <function>ZOOM_scanset_display_term</function> is similar to
2433 <function>ZOOM_scanset_term</function> except that it returns
2434 the <emphasis>display term</emphasis> rather than the raw term.
2435 In a few cases, the term is different from display term. Always
2436 use the display term for display and the raw term for subsequent
2437 scan operations (to get more terms, next scan result, etc).
2440 A scan set may be freed by a call to function
2441 <function>ZOOM_scanset_destroy</function>.
2442 Functions <function>ZOOM_scanset_option_get</function> and
2443 <function>ZOOM_scanset_option_set</function> retrieves and sets
2444 an option respectively.
2447 The <parameter>startpqf</parameter> is a subset of PQF, namely
2448 the Attributes+Term part. Multiple <literal>@attr</literal> can
2449 be used. For example to scan in title (complete) phrases:
2451 @attr 1=4 @attr 6=2 "science o"
2455 The <function>ZOOM_connecton_scan1</function> is a newer and
2456 more generic alternative to <function>ZOOM_connection_scan</function>
2457 which allows to use both CQL and PQF for Scan.
2459 <table frame="top" id="zoom.scanset.options">
2460 <title>ZOOM Scan Set Options</title>
2462 <colspec colwidth="4*" colname="name"></colspec>
2463 <colspec colwidth="7*" colname="description"></colspec>
2464 <colspec colwidth="2*" colname="default"></colspec>
2467 <entry>Option</entry>
2468 <entry>Description</entry>
2469 <entry>Default</entry>
2474 number</entry><entry>Number of Scan Terms requested in next scan.
2475 After scan it holds the actual number of terms returned.
2476 </entry><entry>20</entry></row>
2478 position</entry><entry>Preferred Position of term in response
2479 in next scan; actual position after completion of scan.
2480 </entry><entry>1</entry></row>
2482 stepSize</entry><entry>Step Size
2483 </entry><entry>0</entry></row>
2485 scanStatus</entry><entry>An integer indicating the Scan Status
2487 </entry><entry>0</entry></row>
2489 rpnCharset</entry><entry>Character set for RPN terms.
2490 If this is set, ZOOM C will assume that the ZOOM application is
2491 running UTF-8. Terms in RPN queries are then converted to the
2492 rpnCharset. If this is unset, ZOOM C will not assume any encoding
2493 of RPN terms and no conversion is performed.
2494 </entry><entry>none</entry></row>
2499 <sect1 id="zoom.extendedservices">
2500 <title>Extended Services</title>
2502 ZOOM offers an interface to a subset of the Z39.50 extended services
2503 as well as a few privately defined ones:
2508 Z39.50 Item Order (ILL).
2509 See <xref linkend="zoom.item.order"/>.
2514 Record Update. This allows a client to insert, modify or delete
2516 See <xref linkend="zoom.record.update"/>.
2521 Database Create. This a non-standard feature. Allows a client
2522 to create a database.
2523 See <xref linkend="zoom.database.create"/>.
2528 Database Drop. This a non-standard feature. Allows a client
2529 to delete/drop a database.
2530 See <xref linkend="zoom.database.drop"/>.
2535 Commit operation. This a non-standard feature. Allows a client
2536 to commit operations.
2537 See <xref linkend="zoom.commit"/>.
2540 <!-- all the ILL PDU options should go here too -->
2543 To create an extended service operation, a <literal>ZOOM_package</literal>
2544 must be created. The operation is a five step operation. The
2545 package is created, package is configured by means of options,
2546 the package is sent, result is inspected (by means of options),
2547 the package is destroyed.
2550 ZOOM_package ZOOM_connection_package(ZOOM_connection c,
2551 ZOOM_options options);
2553 const char *ZOOM_package_option_get(ZOOM_package p,
2555 void ZOOM_package_option_set(ZOOM_package p, const char *key,
2557 void ZOOM_package_send(ZOOM_package p, const char *type);
2559 void ZOOM_package_destroy(ZOOM_package p);
2562 The <function>ZOOM_connection_package</function> creates a
2563 package for the connection given using the options specified.
2566 Functions <function>ZOOM_package_option_get</function> and
2567 <function>ZOOM_package_option_set</function> gets and sets
2571 <function>ZOOM_package_send</function> sends
2572 the package the via connection specified in
2573 <function>ZOOM_connection_package</function>.
2574 The <parameter>type</parameter> specifies the actual extended service
2575 package type to be sent.
2577 <table frame="top" id="zoom.extendedservices.type">
2578 <title>Extended Service Type</title>
2580 <colspec colwidth="3*" colname="value"></colspec>
2581 <colspec colwidth="7*" colname="description"></colspec>
2585 <entry>Description</entry>
2590 <entry>itemorder</entry><entry>Item Order</entry>
2593 <entry>update</entry><entry>Record Update</entry>
2596 <entry>create</entry><entry>Database Create</entry>
2599 <entry>drop</entry><entry>Database Drop</entry>
2602 <entry>commit</entry><entry>Commit Operation</entry>
2608 <table frame="top" id="zoom.extendedservices.options">
2609 <title>Extended Service Common Options</title>
2611 <colspec colwidth="4*" colname="name"></colspec>
2612 <colspec colwidth="7*" colname="description"></colspec>
2613 <colspec colwidth="3*" colname="default"></colspec>
2616 <entry>Option</entry>
2617 <entry>Description</entry>
2618 <entry>Default</entry>
2623 <entry>package-name</entry>
2624 <entry>Extended Service Request package name. Must be specified
2625 as part of a request.</entry>
2629 <entry>user-id</entry>
2630 <entry>User ID of Extended Service Package. Is a request option.</entry>
2634 <entry>function</entry>
2636 Function of package - one of <literal>create</literal>,
2637 <literal>delete</literal>, <literal>modify</literal>. Is
2640 <entry><literal>create</literal></entry>
2643 <entry>waitAction</entry>
2645 Wait action for package. Possible values:
2646 <literal>wait</literal>, <literal>waitIfPossible</literal>,
2647 <literal>dontWait</literal> or <literal>dontReturnPackage</literal>.
2649 <entry><literal>waitIfPossible</literal></entry>
2652 <entry>targetReference</entry>
2654 Target Reference. This is part of the response as returned
2655 by the server. Read it after a successful operation.
2657 <entry><literal>none</literal></entry>
2662 <sect2 id="zoom.item.order">
2663 <title>Item Order</title>
2665 For Item Order, <literal>type</literal> must be set to
2666 <literal>itemorder</literal> in
2667 <function>ZOOM_package_send</function>.
2670 <table frame="top" id="zoom.item.order.options">
2671 <title>Item Order Options</title>
2673 <colspec colwidth="4*" colname="name"></colspec>
2674 <colspec colwidth="7*" colname="description"></colspec>
2675 <colspec colwidth="3*" colname="default"></colspec>
2678 <entry>Option</entry>
2679 <entry>Description</entry>
2680 <entry>Default</entry>
2685 <entry>contact-name</entry>
2686 <entry>ILL contact name</entry>
2690 <entry>contact-phone</entry>
2691 <entry>ILL contact phone</entry>
2695 <entry>contact-email</entry>
2696 <entry>ILL contact email</entry>
2700 <entry>itemorder-setname</entry>
2701 <entry>Name of result set for record</entry>
2702 <entry>default</entry>
2705 <entry>itemorder-item</entry>
2706 <entry>Position for item (record) requested. An integer</entry>
2713 There are two variants of item order: ILL-variant and
2714 XML document variant. In order to use the XML variant the setting
2715 <literal>doc</literal> must hold the XML item order document. If that
2716 setting is unset, the ILL-variant is used.
2719 <table frame="top" id="zoom.illrequest.options">
2720 <title>ILL Request Options</title>
2722 <colspec colwidth="4*" colname="name"></colspec>
2725 <entry>Option</entry>
2729 <row><entry>protocol-version-num</entry></row>
2730 <row><entry>transaction-id,initial-requester-id,person-or-institution-symbol,person</entry></row>
2731 <row><entry>transaction-id,initial-requester-id,person-or-institution-symbol,institution</entry></row>
2732 <row><entry>transaction-id,initial-requester-id,name-of-person-or-institution,name-of-person</entry></row>
2733 <row><entry>transaction-id,initial-requester-id,name-of-person-or-institution,name-of-institution</entry></row>
2734 <row><entry>transaction-id,transaction-group-qualifier</entry></row>
2735 <row><entry>transaction-id,transaction-qualifier</entry></row>
2736 <row><entry>transaction-id,sub-transaction-qualifier</entry></row>
2737 <row><entry>service-date-time,this,date</entry></row>
2738 <row><entry>service-date-time,this,time</entry></row>
2739 <row><entry>service-date-time,original,date</entry></row>
2740 <row><entry>service-date-time,original,time</entry></row>
2741 <row><entry>requester-id,person-or-institution-symbol,person</entry></row>
2742 <row><entry>requester-id,person-or-institution-symbol,institution</entry></row>
2743 <row><entry>requester-id,name-of-person-or-institution,name-of-person</entry></row>
2744 <row><entry>requester-id,name-of-person-or-institution,name-of-institution</entry></row>
2745 <row><entry>responder-id,person-or-institution-symbol,person</entry></row>
2746 <row><entry>responder-id,person-or-institution-symbol,institution</entry></row>
2747 <row><entry>responder-id,name-of-person-or-institution,name-of-person</entry></row>
2748 <row><entry>responder-id,name-of-person-or-institution,name-of-institution</entry></row>
2749 <row><entry>transaction-type</entry></row>
2750 <row><entry>delivery-address,postal-address,name-of-person-or-institution,name-of-person</entry></row>
2751 <row><entry>delivery-address,postal-address,name-of-person-or-institution,name-of-institution</entry></row>
2752 <row><entry>delivery-address,postal-address,extended-postal-delivery-address</entry></row>
2753 <row><entry>delivery-address,postal-address,street-and-number</entry></row>
2754 <row><entry>delivery-address,postal-address,post-office-box</entry></row>
2755 <row><entry>delivery-address,postal-address,city</entry></row>
2756 <row><entry>delivery-address,postal-address,region</entry></row>
2757 <row><entry>delivery-address,postal-address,country</entry></row>
2758 <row><entry>delivery-address,postal-address,postal-code</entry></row>
2759 <row><entry>delivery-address,electronic-address,telecom-service-identifier</entry></row>
2760 <row><entry>delivery-address,electronic-address,telecom-service-addreess</entry></row>
2761 <row><entry>billing-address,postal-address,name-of-person-or-institution,name-of-person</entry></row>
2762 <row><entry>billing-address,postal-address,name-of-person-or-institution,name-of-institution</entry></row>
2763 <row><entry>billing-address,postal-address,extended-postal-delivery-address</entry></row>
2764 <row><entry>billing-address,postal-address,street-and-number</entry></row>
2765 <row><entry>billing-address,postal-address,post-office-box</entry></row>
2766 <row><entry>billing-address,postal-address,city</entry></row>
2767 <row><entry>billing-address,postal-address,region</entry></row>
2768 <row><entry>billing-address,postal-address,country</entry></row>
2769 <row><entry>billing-address,postal-address,postal-code</entry></row>
2770 <row><entry>billing-address,electronic-address,telecom-service-identifier</entry></row>
2771 <row><entry>billing-address,electronic-address,telecom-service-addreess</entry></row>
2772 <row><entry>ill-service-type</entry></row>
2773 <row><entry>requester-optional-messages,can-send-RECEIVED</entry></row>
2774 <row><entry>requester-optional-messages,can-send-RETURNED</entry></row>
2775 <row><entry>requester-optional-messages,requester-SHIPPED</entry></row>
2776 <row><entry>requester-optional-messages,requester-CHECKED-IN</entry></row>
2777 <row><entry>search-type,level-of-service</entry></row>
2778 <row><entry>search-type,need-before-date</entry></row>
2779 <row><entry>search-type,expiry-date</entry></row>
2780 <row><entry>search-type,expiry-flag</entry></row>
2781 <row><entry>place-on-hold</entry></row>
2782 <row><entry>client-id,client-name</entry></row>
2783 <row><entry>client-id,client-status</entry></row>
2784 <row><entry>client-id,client-identifier</entry></row>
2785 <row><entry>item-id,item-type</entry></row>
2786 <row><entry>item-id,call-number</entry></row>
2787 <row><entry>item-id,author</entry></row>
2788 <row><entry>item-id,title</entry></row>
2789 <row><entry>item-id,sub-title</entry></row>
2790 <row><entry>item-id,sponsoring-body</entry></row>
2791 <row><entry>item-id,place-of-publication</entry></row>
2792 <row><entry>item-id,publisher</entry></row>
2793 <row><entry>item-id,series-title-number</entry></row>
2794 <row><entry>item-id,volume-issue</entry></row>
2795 <row><entry>item-id,edition</entry></row>
2796 <row><entry>item-id,publication-date</entry></row>
2797 <row><entry>item-id,publication-date-of-component</entry></row>
2798 <row><entry>item-id,author-of-article</entry></row>
2799 <row><entry>item-id,title-of-article</entry></row>
2800 <row><entry>item-id,pagination</entry></row>
2801 <row><entry>item-id,ISBN</entry></row>
2802 <row><entry>item-id,ISSN</entry></row>
2803 <row><entry>item-id,additional-no-letters</entry></row>
2804 <row><entry>item-id,verification-reference-source</entry></row>
2805 <row><entry>copyright-complicance</entry></row>
2806 <row><entry>retry-flag</entry></row>
2807 <row><entry>forward-flag</entry></row>
2808 <row><entry>requester-note</entry></row>
2809 <row><entry>forward-note</entry></row>
2814 <sect2 id="zoom.record.update">
2815 <title>Record Update</title>
2817 For Record Update, <literal>type</literal> must be set to
2818 <literal>update</literal> in
2819 <function>ZOOM_package_send</function>.
2821 <table frame="top" id="zoom.record.update.options">
2822 <title>Record Update Options</title>
2824 <colspec colwidth="4*" colname="name"></colspec>
2825 <colspec colwidth="7*" colname="description"></colspec>
2826 <colspec colwidth="3*" colname="default"></colspec>
2829 <entry>Option</entry>
2830 <entry>Description</entry>
2831 <entry>Default</entry>
2836 <entry>action</entry>
2838 The update action. One of
2839 <literal>specialUpdate</literal>,
2840 <literal>recordInsert</literal>,
2841 <literal>recordReplace</literal>,
2842 <literal>recordDelete</literal>,
2843 <literal>elementUpdate</literal>.
2845 <entry><literal>specialUpdate (recordInsert for updateVersion=1 which does not support specialUpdate)</literal></entry>
2848 <entry>recordIdOpaque</entry>
2849 <entry>Opaque Record ID</entry>
2853 <entry>recordIdNumber</entry>
2854 <entry>Record ID number</entry>
2858 <entry>record</entry>
2859 <entry>The record itself</entry>
2863 <entry>recordOpaque</entry>
2864 <entry>Specifies an opaque record which is
2865 encoded as an ASN.1 ANY type with the OID as tiven by option
2866 <literal>syntax</literal> (see below).
2867 Option <literal>recordOpaque</literal> is an alternative
2868 to record - and <literal>record</literal> option (above) is
2869 ignored if recordOpaque is set. This option is only available in
2870 YAZ 3.0.35 and later, and is meant to facilitate Updates with
2876 <entry>syntax</entry>
2877 <entry>The record syntax (transfer syntax). Is a string that
2878 is a known record syntax.
2880 <entry>no syntax</entry>
2883 <entry>databaseName</entry>
2884 <entry>Database from connection object</entry>
2885 <entry>Default</entry>
2888 <entry>correlationInfo.note</entry>
2889 <entry>Correlation Info Note (string)</entry>
2893 <entry>correlationInfo.id</entry>
2894 <entry>Correlation Info ID (integer)</entry>
2898 <entry>elementSetName</entry>
2899 <entry>Element Set for Record</entry>
2903 <entry>updateVersion</entry>
2904 <entry>Record Update version which holds one of the values
2905 1, 2 or 3. Each version has a distinct OID:
2907 (<ulink url="&url.z39.50.extupdate1;">first version</ulink>) ,
2909 (second version) and
2910 1.2.840.10003.9.5.1.1
2911 (<ulink url="&url.z39.50.extupdate3;">third and
2912 newest version</ulink>).
2922 <sect2 id="zoom.database.create"><title>Database Create</title>
2924 For Database Create, <literal>type</literal> must be set to
2925 <literal>create</literal> in
2926 <function>ZOOM_package_send</function>.
2929 <table frame="top" id="zoom.database.create.options">
2930 <title>Database Create Options</title>
2932 <colspec colwidth="4*" colname="name"></colspec>
2933 <colspec colwidth="7*" colname="description"></colspec>
2934 <colspec colwidth="3*" colname="default"></colspec>
2937 <entry>Option</entry>
2938 <entry>Description</entry>
2939 <entry>Default</entry>
2944 <entry>databaseName</entry>
2945 <entry>Database from connection object</entry>
2946 <entry>Default</entry>
2952 <sect2 id="zoom.database.drop">
2953 <title>Database Drop</title>
2955 For Database Drop, <literal>type</literal> must be set to
2956 <literal>drop</literal> in
2957 <function>ZOOM_package_send</function>.
2959 <table frame="top" id="zoom.database.drop.options">
2960 <title>Database Drop Options</title>
2962 <colspec colwidth="4*" colname="name"></colspec>
2963 <colspec colwidth="7*" colname="description"></colspec>
2964 <colspec colwidth="3*" colname="default"></colspec>
2967 <entry>Option</entry>
2968 <entry>Description</entry>
2969 <entry>Default</entry>
2974 <entry>databaseName</entry>
2975 <entry>Database from connection object</entry>
2976 <entry>Default</entry>
2982 <sect2 id="zoom.commit">
2983 <title>Commit Operation</title>
2985 For Commit, <literal>type</literal> must be set to
2986 <literal>commit</literal> in
2987 <function>ZOOM_package_send</function>.
2990 <sect2 id="zoom.extended.services.behavior">
2991 <title>Protocol behavior</title>
2993 All the extended services are Z39.50-only.
2997 The database create, drop, and commit services are privately defined
2999 Refer to <filename>esadmin.asn</filename> in YAZ for the ASN.1
3005 <sect1 id="zoom.options">
3006 <title>Options</title>
3008 Most &zoom; objects provide a way to specify options to change behavior.
3009 From an implementation point of view, a set of options is just like
3010 an associative array / hash.
3013 ZOOM_options ZOOM_options_create(void);
3015 ZOOM_options ZOOM_options_create_with_parent(ZOOM_options parent);
3017 void ZOOM_options_destroy(ZOOM_options opt);
3020 const char *ZOOM_options_get(ZOOM_options opt, const char *name);
3022 void ZOOM_options_set(ZOOM_options opt, const char *name,
3026 typedef const char *(*ZOOM_options_callback)
3027 (void *handle, const char *name);
3029 ZOOM_options_callback
3030 ZOOM_options_set_callback(ZOOM_options opt,
3031 ZOOM_options_callback c,
3035 <sect1 id="zoom.queryconversions">
3036 <title>Query conversions</title>
3038 int ZOOM_query_cql2rpn(ZOOM_query s, const char *cql_str,
3039 ZOOM_connection conn);
3041 int ZOOM_query_ccl2rpn(ZOOM_query s, const char *ccl_str,
3043 int *ccl_error, const char **error_string,
3047 <function>ZOOM_query_cql2rpn</function> translates the CQL string,
3048 client-side, into RPN which may be passed to the server.
3049 This is useful for servers that don't themselves
3050 support CQL, for which <function>ZOOM_query_cql</function> is useless.
3051 'conn' is used only as a place to stash diagnostics if compilation
3052 fails; if this information is not needed, a null pointer may be used.
3053 The CQL conversion is driven by option <literal>cqlfile</literal> from
3054 connection conn. This specifies a conversion file (e.g. pqf.properties)
3055 which <emphasis>must</emphasis> be present.
3058 <function>ZOOM_query_ccl2rpn</function> translates the CCL string,
3059 client-side, into RPN which may be passed to the server.
3060 The conversion is driven by the specification given by
3061 <literal>config</literal>. Upon completion 0 is returned on success; -1
3062 is returned on failure. On failure <literal>error_string</literal> and
3063 <literal>error_pos</literal> hold the error message and position of
3064 first error in original CCL string.
3067 <sect1 id="zoom.events"><title>Events</title>
3069 If you're developing non-blocking applications, you have to deal
3073 int ZOOM_event(int no, ZOOM_connection *cs);
3076 The <function>ZOOM_event</function> executes pending events for
3077 a number of connections. Supply the number of connections in
3078 <literal>no</literal> and an array of connections in
3079 <literal>cs</literal> (<literal>cs[0] ... cs[no-1]</literal>).
3080 A pending event could be sending a search, receiving a response,
3082 When an event has occurred for one of the connections, this function
3083 returns a positive integer <literal>n</literal> denoting that an event
3084 occurred for connection <literal>cs[n-1]</literal>.
3085 When no events are pending for the connections, a value of zero is
3087 To ensure that all outstanding requests are performed, call this function
3088 repeatedly until zero is returned.
3091 If <function>ZOOM_event</function> returns, and returns non-zero, the
3092 last event that occurred can be expected.
3095 int ZOOM_connection_last_event(ZOOM_connection cs);
3098 <function>ZOOM_connection_last_event</function> returns an event type
3099 (integer) for the last event.
3102 <table frame="top" id="zoom.event.ids">
3103 <title>ZOOM Event IDs</title>
3105 <colspec colwidth="4*" colname="name"></colspec>
3106 <colspec colwidth="7*" colname="description"></colspec>
3109 <entry>Event</entry>
3110 <entry>Description</entry>
3115 <entry>ZOOM_EVENT_NONE</entry>
3116 <entry>No event has occurred</entry>
3119 <entry>ZOOM_EVENT_CONNECT</entry>
3120 <entry>TCP/IP connect has initiated</entry>
3123 <entry>ZOOM_EVENT_SEND_DATA</entry>
3124 <entry>Data has been transmitted (sending)</entry>
3127 <entry>ZOOM_EVENT_RECV_DATA</entry>
3128 <entry>Data has been received</entry>
3131 <entry>ZOOM_EVENT_TIMEOUT</entry>
3132 <entry>Timeout</entry>
3135 <entry>ZOOM_EVENT_UNKNOWN</entry>
3136 <entry>Unknown event</entry>
3139 <entry>ZOOM_EVENT_SEND_APDU</entry>
3140 <entry>An APDU has been transmitted (sending)</entry>
3143 <entry>ZOOM_EVENT_RECV_APDU</entry>
3144 <entry>An APDU has been received</entry>
3147 <entry>ZOOM_EVENT_RECV_RECORD</entry>
3148 <entry>A result-set record has been received</entry>
3151 <entry>ZOOM_EVENT_RECV_SEARCH</entry>
3152 <entry>A search result has been received</entry>
3159 <chapter id="server">
3160 <title>Generic server</title>
3161 <sect1 id="server.introduction"><title>Introduction</title>
3163 If you aren't into documentation, a good way to learn how the
3164 back end interface works is to look at the <filename>backend.h</filename>
3165 file. Then, look at the small dummy-server in
3166 <filename>ztest/ztest.c</filename>. The <filename>backend.h</filename>
3167 file also makes a good reference, once you've chewed your way through
3168 the prose of this file.
3171 If you have a database system that you would like to make available by
3172 means of Z39.50 or SRU, &yaz; basically offers two options. You
3173 can use the APIs provided by the &asn;, &odr;, and &comstack;
3175 create and decode PDUs, and exchange them with a client.
3176 Using this low-level interface gives you access to all fields and
3177 options of the protocol, and you can construct your server as close
3178 to your existing database as you like.
3179 It is also a fairly involved process, requiring
3180 you to set up an event-handling mechanism, protocol state machine,
3181 etc. To simplify server implementation, we have implemented a compact
3182 and simple, but reasonably full-functioned server-frontend that will
3183 handle most of the protocol mechanics, while leaving you to
3184 concentrate on your database interface.
3188 The backend interface was designed in anticipation of a specific
3189 integration task, while still attempting to achieve some degree of
3190 generality. We realize fully that there are points where the
3191 interface can be improved significantly. If you have specific
3192 functions or parameters that you think could be useful, send us a
3193 mail (or better, sign on to the mailing list referred to in the
3194 top-level README file). We will try to fit good suggestions into future
3195 releases, to the extent that it can be done without requiring
3196 too many structural changes in existing applications.
3201 The &yaz; server does not support XCQL.
3205 <sect1 id="server.frontend">
3206 <title>The Database Frontend</title>
3208 We refer to this software as a generic database frontend. Your
3209 database system is the <emphasis>backend database</emphasis>, and the
3210 interface between the two is called the <emphasis>backend API</emphasis>.
3211 The backend API consists of a small number of function handlers and
3212 structure definitions. You are required to provide the
3213 <function>main()</function> routine for the server (which can be
3214 quite simple), as well as a set of handlers to match each of the
3216 The interface functions that you write can use any mechanism you like
3217 to communicate with your database system: You might link the whole
3218 thing together with your database application and access it by
3219 function calls; you might use IPC to talk to a database server
3220 somewhere; or you might link with third-party software that handles
3221 the communication for you (like a commercial database client library).
3222 At any rate, the handlers will perform the tasks of:
3235 Scanning the database index (optional - if you wish to implement SCAN).
3238 Extended Services (optional).
3241 Result-Set Delete (optional).
3244 Result-Set Sort (optional).
3247 Return Explain for SRU (optional).
3251 (more functions will be added in time to support as much of
3252 Z39.50-1995 as possible).
3255 <sect1 id="server.backend">
3256 <title>The Backend API</title>
3258 The header file that you need to use the interface are in the
3259 <filename>include/yaz</filename> directory. It's called
3260 <filename>backend.h</filename>. It will include other files from
3261 the <filename>include/yaz</filename> directory, so you'll
3262 probably want to use the -I option of your compiler to tell it
3263 where to find the files. When you run
3264 <literal>make</literal> in the top-level &yaz; directory,
3265 everything you need to create your server is to link with the
3266 <filename>lib/libyaz.la</filename> library.
3269 <sect1 id="server.main">
3270 <title>Your main() Routine</title>
3272 As mentioned, your <function>main()</function> routine can be quite brief.
3273 If you want to initialize global parameters, or read global configuration
3274 tables, this is the place to do it. At the end of the routine, you should
3278 int statserv_main(int argc, char **argv,
3279 bend_initresult *(*bend_init)(bend_initrequest *r),
3280 void (*bend_close)(void *handle));
3283 The third and fourth arguments are pointers to handlers. Handler
3284 <function>bend_init</function> is called whenever the server receives
3285 an Initialize Request, so it serves as a Z39.50 session initializer. The
3286 <function>bend_close</function> handler is called when the session is
3290 <function>statserv_main</function> will establish listening sockets
3291 according to the parameters given. When connection requests are received,
3292 the event handler will typically <function>fork()</function> and
3293 create a sub-process to handle a new connection.
3294 Alternatively the server may be setup to create threads for each
3296 If you do use global variables and forking, you should be aware, then,
3297 that these cannot be shared between associations, unless you explicitly
3298 disable forking by command line parameters.
3301 The server provides a mechanism for controlling some of its behavior
3302 without using command-line options. The function
3305 statserv_options_block *statserv_getcontrol(void);
3308 will return a pointer to a <literal>struct statserv_options_block</literal>
3309 describing the current default settings of the server. The structure
3310 contains these elements:
3313 <term><literal>int dynamic</literal></term>
3315 A boolean value, which determines whether the server
3316 will fork on each incoming request (TRUE), or not (FALSE). Default is
3317 TRUE. This flag is only read by UNIX-based servers (WIN32-based servers
3322 <term><literal>int threads</literal></term>
3324 A boolean value, which determines whether the server
3325 will create a thread on each incoming request (TRUE), or not (FALSE).
3326 Default is FALSE. This flag is only read by UNIX-based servers
3327 that offer POSIX Threads support.
3328 WIN32-based servers always operate in threaded mode.
3332 <term><literal>int inetd</literal></term>
3334 A boolean value, which determines whether the server
3335 will operate under a UNIX INET daemon (inetd). Default is FALSE.
3339 <term><literal>char logfile[ODR_MAXNAME+1]</literal></term>
3340 <listitem><para>File for diagnostic output ("": stderr).
3344 <term><literal>char apdufile[ODR_MAXNAME+1]</literal></term>
3346 Name of file for logging incoming and outgoing APDUs
3347 ("": don't log APDUs, "-":
3348 <literal>stderr</literal>).
3352 <term><literal>char default_listen[1024]</literal></term>
3353 <listitem><para>Same form as the command-line specification of
3354 listener address. "": no default listener address.
3355 Default is to listen at "tcp:@:9999". You can only
3356 specify one default listener address in this fashion.
3360 <term><literal>enum oid_proto default_proto;</literal></term>
3361 <listitem><para>Either <literal>PROTO_Z3950</literal> or
3362 <literal>PROTO_SR</literal>.
3363 Default is <literal>PROTO_Z39_50</literal>.
3367 <term><literal>int idle_timeout;</literal></term>
3368 <listitem><para>Maximum session idle-time, in minutes. Zero indicates
3369 no (infinite) timeout. Default is 15 minutes.
3373 <term><literal>int maxrecordsize;</literal></term>
3374 <listitem><para>Maximum permissible record (message) size. Default
3375 is 64 MB. This amount of memory will only be allocated if a
3376 client requests a very large amount of records in one operation
3378 Set it to a lower number if you are worried about resource
3379 consumption on your host system.
3383 <term><literal>char configname[ODR_MAXNAME+1]</literal></term>
3384 <listitem><para>Passed to the backend when a new connection is received.
3388 <term><literal>char setuid[ODR_MAXNAME+1]</literal></term>
3389 <listitem><para>Set user id to the user specified, after binding
3390 the listener addresses.
3395 <literal>void (*bend_start)(struct statserv_options_block *p)</literal>
3397 <listitem><para>Pointer to function which is called after the
3398 command line options have been parsed - but before the server
3400 For forked UNIX servers, this handler is called in the mother
3401 process; for threaded servers, this handler is called in the
3403 The default value of this pointer is NULL in which case it
3404 isn't invoked by the frontend server.
3405 When the server operates as an NT service, this handler is called
3406 whenever the service is started.
3411 <literal>void (*bend_stop)(struct statserv_options_block *p)</literal>
3413 <listitem><para>Pointer to function which is called whenever the server
3414 has stopped listening for incoming connections. This function pointer
3415 has a default value of NULL in which case it isn't called.
3416 When the server operates as an NT service, this handler is called
3417 whenever the service is stopped.
3421 <term><literal>void *handle</literal></term>
3422 <listitem><para>User defined pointer (default value NULL).
3423 This is a per-server handle that can be used to specify "user-data".
3424 Do not confuse this with the session-handle as returned by bend_init.
3430 The pointer returned by <literal>statserv_getcontrol</literal> points to
3431 a static area. You are allowed to change the contents of the structure,
3432 but the changes will not take effect until you call
3435 void statserv_setcontrol(statserv_options_block *block);
3439 You should generally update this structure before calling
3440 <function>statserv_main()</function>.
3444 <sect1 id="server.backendfunctions">
3445 <title>The Backend Functions</title>
3447 For each service of the protocol, the backend interface declares one or
3448 two functions. You are required to provide implementations of the
3449 functions representing the services that you wish to implement.
3451 <sect2 id="server.init">
3454 bend_initresult (*bend_init)(bend_initrequest *r);
3457 This handler is called once for each new connection request, after
3458 a new process/thread has been created, and an Initialize Request has
3459 been received from the client. The pointer to the
3460 <function>bend_init</function> handler is passed in the call to
3461 <function>statserv_start</function>.
3464 This handler is also called when operating in SRU mode - when
3465 a connection has been made (even though SRU does not offer
3469 Unlike previous versions of YAZ, the <function>bend_init</function> also
3470 serves as a handler that defines the Z39.50 services that the backend
3471 intends to support. Pointers to <emphasis>all</emphasis> service handlers,
3472 including search - and fetch must be specified here in this handler.
3475 The request - and result structures are defined as
3478 typedef struct bend_initrequest
3480 /** \brief user/name/password to be read */
3481 Z_IdAuthentication *auth;
3482 /** \brief encoding stream (for results) */
3484 /** \brief printing stream */
3486 /** \brief decoding stream (use stream for results) */
3488 /** \brief reference ID */
3489 Z_ReferenceId *referenceId;
3490 /** \brief peer address of client */
3493 /** \brief character set and language negotiation
3495 see include/yaz/z-charneg.h
3497 Z_CharSetandLanguageNegotiation *charneg_request;
3499 /** \brief character negotiation response */
3500 Z_External *charneg_response;
3502 /** \brief character set (encoding) for query terms
3504 This is NULL by default. It should be set to the native character
3505 set that the backend assumes for query terms */
3506 char *query_charset;
3508 /** \brief whehter query_charset also applies to recors
3510 Is 0 (No) by default. Set to 1 (yes) if records is in the same
3511 character set as queries. If in doubt, use 0 (No).
3513 int records_in_same_charset;
3515 char *implementation_id;
3516 char *implementation_name;
3517 char *implementation_version;
3519 /** \brief Z39.50 sort handler */
3520 int (*bend_sort)(void *handle, bend_sort_rr *rr);
3521 /** \brief SRU/Z39.50 search handler */
3522 int (*bend_search)(void *handle, bend_search_rr *rr);
3523 /** \brief SRU/Z39.50 fetch handler */
3524 int (*bend_fetch)(void *handle, bend_fetch_rr *rr);
3525 /** \brief SRU/Z39.50 present handler */
3526 int (*bend_present)(void *handle, bend_present_rr *rr);
3527 /** \brief Z39.50 extended services handler */
3528 int (*bend_esrequest) (void *handle, bend_esrequest_rr *rr);
3529 /** \brief Z39.50 delete result set handler */
3530 int (*bend_delete)(void *handle, bend_delete_rr *rr);
3531 /** \brief Z39.50 scan handler */
3532 int (*bend_scan)(void *handle, bend_scan_rr *rr);
3533 /** \brief Z39.50 segment facility handler */
3534 int (*bend_segment)(void *handle, bend_segment_rr *rr);
3535 /** \brief SRU explain handler */
3536 int (*bend_explain)(void *handle, bend_explain_rr *rr);
3537 /** \brief SRU scan handler */
3538 int (*bend_srw_scan)(void *handle, bend_scan_rr *rr);
3539 /** \brief SRU record update handler */
3540 int (*bend_srw_update)(void *handle, bend_update_rr *rr);
3542 /** \brief whether named result sets are supported (0=disable, 1=enable) */
3543 int named_result_sets;
3546 typedef struct bend_initresult
3548 int errcode; /* 0==OK */
3549 char *errstring; /* system error string or NULL */
3550 void *handle; /* private handle to the backend module */
3554 In general, the server frontend expects that the
3555 <literal>bend_*result</literal> pointer that you return is valid at
3556 least until the next call to a <literal>bend_* function</literal>.
3557 This applies to all of the functions described herein. The parameter
3558 structure passed to you in the call belongs to the server frontend, and
3559 you should not make assumptions about its contents after the current
3560 function call has completed. In other words, if you want to retain any
3561 of the contents of a request structure, you should copy them.
3564 The <literal>errcode</literal> should be zero if the initialization of
3565 the backend went well. Any other value will be interpreted as an error.
3566 The <literal>errstring</literal> isn't used in the current version, but
3567 one option would be to stick it in the initResponse as a VisibleString.
3568 The <literal>handle</literal> is the most important parameter. It should
3569 be set to some value that uniquely identifies the current session to
3570 the backend implementation. It is used by the frontend server in any
3571 future calls to a backend function.
3572 The typical use is to set it to point to a dynamically allocated state
3573 structure that is private to your backend module.
3576 The <literal>auth</literal> member holds the authentication information
3577 part of the Z39.50 Initialize Request. Interpret this if your server
3578 requires authentication.
3581 The members <literal>peer_name</literal>,
3582 <literal>implementation_id</literal>,
3583 <literal>implementation_name</literal> and
3584 <literal>implementation_version</literal> holds
3585 DNS of client, ID of implementor, name
3586 of client (Z39.50) implementation - and version.
3589 The <literal>bend_</literal> - members are set to NULL when
3590 <function>bend_init</function> is called. Modify the pointers by
3591 setting them to point to backend functions.
3594 <sect2 id="server.search.retrieve">
3595 <title>Search and Retrieve</title>
3597 We now describe the handlers that are required to support search -
3598 and retrieve. You must support two functions - one for search - and one
3599 for fetch (retrieval of one record). If desirable you can provide a
3600 third handler which is called when a present request is received which
3601 allows you to optimize retrieval of multiple-records.
3604 int (*bend_search) (void *handle, bend_search_rr *rr);
3607 char *setname; /* name to give to this set */
3608 int replace_set; /* replace set, if it already exists */
3609 int num_bases; /* number of databases in list */
3610 char **basenames; /* databases to search */
3611 Z_ReferenceId *referenceId;/* reference ID */
3612 Z_Query *query; /* query structure */
3613 ODR stream; /* encode stream */
3614 ODR decode; /* decode stream */
3615 ODR print; /* print stream */
3617 bend_request request;
3618 bend_association association;
3620 int hits; /* number of hits */
3621 int errcode; /* 0==OK */
3622 char *errstring; /* system error string or NULL */
3623 Z_OtherInformation *search_info; /* additional search info */
3624 char *srw_sortKeys; /* holds SRU/SRW sortKeys info */
3625 char *srw_setname; /* holds SRU/SRW generated resultsetID */
3626 int *srw_setnameIdleTime; /* holds SRU/SRW life-time */
3627 int estimated_hit_count; /* if hit count is estimated */
3628 int partial_resultset; /* if result set is partial */
3632 The <function>bend_search</function> handler is a fairly close
3633 approximation of a protocol Z39.50 Search Request - and Response PDUs.
3634 The <literal>setname</literal> is the resultSetName from the protocol.
3635 You are required to establish a mapping between the set name and whatever
3636 your backend database likes to use.
3637 Similarly, the <literal>replace_set</literal> is a boolean value
3638 corresponding to the resultSetIndicator field in the protocol.
3639 <literal>num_bases/basenames</literal> is a length of/array of character
3640 pointers to the database names provided by the client.
3641 The <literal>query</literal> is the full query structure as defined in
3642 the protocol ASN.1 specification.
3643 It can be either of the possible query types, and it's up to you to
3644 determine if you can handle the provided query type.
3645 Rather than reproduce the C interface here, we'll refer you to the
3646 structure definitions in the file
3647 <filename>include/yaz/z-core.h</filename>. If you want to look at the
3648 attributeSetId OID of the RPN query, you can either match it against
3649 your own internal tables, or you can use the <link linkend="tools.oid">
3653 The structure contains a number of hits, and an
3654 <literal>errcode/errstring</literal> pair. If an error occurs
3655 during the search, or if you're unhappy with the request, you should
3656 set the errcode to a value from the BIB-1 diagnostic set. The value
3657 will then be returned to the user in a nonsurrogate diagnostic record
3658 in the response. The <literal>errstring</literal>, if provided, will
3659 go in the addinfo field. Look at the protocol definition for the
3660 defined error codes, and the suggested uses of the addinfo field.
3663 The <function>bend_search</function> handler is also called when
3664 the frontend server receives a SRU SearchRetrieveRequest.
3665 For SRU, a CQL query is usually provided by the client.
3666 The CQL query is available as part of <literal>Z_Query</literal>
3667 structure (note that CQL is now part of Z39.50 via an external).
3668 To support CQL in existing implementations that only do Type-1,
3669 we refer to the CQL-to-PQF tool described
3670 <link linkend="cql.to.pqf">here</link>.
3673 To maintain backwards compatibility, the frontend server
3674 of yaz always assume that error codes are BIB-1 diagnostics.
3675 For SRU operation, a Bib-1 diagnostic code is mapped to
3679 int (*bend_fetch) (void *handle, bend_fetch_rr *rr);
3681 typedef struct bend_fetch_rr {
3682 char *setname; /* set name */
3683 int number; /* record number */
3684 Z_ReferenceId *referenceId;/* reference ID */
3685 Odr_oid *request_format; /* format, transfer syntax (OID) */
3686 Z_RecordComposition *comp; /* Formatting instructions */
3687 ODR stream; /* encoding stream - memory source if req */
3688 ODR print; /* printing stream */
3690 char *basename; /* name of database that provided record */
3691 int len; /* length of record or -1 if structured */
3692 char *record; /* record */
3693 int last_in_set; /* is it? */
3694 Odr_oid *output_format; /* response format/syntax (OID) */
3695 int errcode; /* 0==success */
3696 char *errstring; /* system error string or NULL */
3697 int surrogate_flag; /* surrogate diagnostic */
3698 char *schema; /* string record schema input/output */
3702 The frontend server calls the <function>bend_fetch</function> handler
3703 when it needs database records to fulfill a Z39.50 Search Request, a
3704 Z39.50 Present Request or a SRU SearchRetrieveRequest.
3705 The <literal>setname</literal> is simply the name of the result set
3706 that holds the reference to the desired record.
3707 The <literal>number</literal> is the offset into the set (with 1
3708 being the first record in the set). The <literal>format</literal> field
3709 is the record format requested by the client (See
3710 <xref linkend="tools.oid"/>).
3711 A value of NULL for <literal>format</literal> indicates that the
3712 client did not request a specific format.
3713 The <literal>stream</literal> argument is an &odr; stream which
3714 should be used for allocating space for structured data records.
3715 The stream will be reset when all records have been assembled, and
3716 the response package has been transmitted.
3717 For unstructured data, the backend is responsible for maintaining a
3718 static or dynamic buffer for the record between calls.
3721 If a SRU SearchRetrieveRequest is received by the frontend server,
3722 the <literal>referenceId</literal> is NULL and the
3723 <literal>format</literal> (transfer syntax) is the OID for XML.
3724 The schema for SRU is stored in both the
3725 <literal>Z_RecordComposition</literal>
3726 structure and <literal>schema</literal> (simple string).
3729 In the structure, the <literal>basename</literal> is the name of the
3730 database that holds the
3731 record. <literal>len</literal> is the length of the record returned, in
3732 bytes, and <literal>record</literal> is a pointer to the record.
3733 <literal>last_in_set</literal> should be nonzero only if the record
3734 returned is the last one in the given result set.
3735 <literal>errcode</literal> and <literal>errstring</literal>, if
3736 given, will be interpreted as a global error pertaining to the
3737 set, and will be returned in a non-surrogate-diagnostic.
3738 If you wish to return the error as a surrogate-diagnostic
3739 (local error) you can do this by setting
3740 <literal>surrogate_flag</literal> to 1 also.
3743 If the <literal>len</literal> field has the value -1, then
3744 <literal>record</literal> is assumed to point to a constructed data
3745 type. The <literal>format</literal> field will be used to determine
3746 which encoder should be used to serialize the data.
3750 If your backend generates structured records, it should use
3751 <function>odr_malloc()</function> on the provided stream for allocating
3752 data: This allows the frontend server to keep track of the record sizes.
3756 The <literal>format</literal> field is mapped to an object identifier
3757 in the direct reference of the resulting EXTERNAL representation
3762 The current version of &yaz; only supports the direct reference mode.
3766 int (*bend_present) (void *handle, bend_present_rr *rr);
3769 char *setname; /* set name */
3771 int number; /* record number */
3772 Odr_oid *format; /* format, transfer syntax (OID) */
3773 Z_ReferenceId *referenceId;/* reference ID */
3774 Z_RecordComposition *comp; /* Formatting instructions */
3775 ODR stream; /* encoding stream - memory source if required */
3776 ODR print; /* printing stream */
3777 bend_request request;
3778 bend_association association;
3780 int hits; /* number of hits */
3781 int errcode; /* 0==OK */
3782 char *errstring; /* system error string or NULL */
3786 The <function>bend_present</function> handler is called when
3787 the server receives a Z39.50 Present Request.
3788 The <literal>setname</literal>,
3789 <literal>start</literal> and <literal>number</literal> is the
3790 name of the result set - start position - and number of records to
3791 be retrieved respectively. <literal>format</literal> and
3792 <literal>comp</literal> is the preferred transfer syntax and element
3793 specifications of the present request.
3796 Note that this is handler serves as a supplement for
3797 <function>bend_fetch</function> and need not to be defined in order to
3798 support search - and retrieve.
3801 <sect2 id="server.delete">
3802 <title>Delete</title>
3804 For back-ends that supports delete of a result set, only one handler
3808 int (*bend_delete)(void *handle, bend_delete_rr *rr);
3810 typedef struct bend_delete_rr {
3814 Z_ReferenceId *referenceId;
3815 int delete_status; /* status for the whole operation */
3816 int *statuses; /* status each set - indexed as setnames */
3823 The delete set function definition is rather primitive, mostly because
3824 we have had no practical need for it as of yet. If someone wants
3825 to provide a full delete service, we'd be happy to add the
3826 extra parameters that are required. Are there clients out there
3827 that will actually delete sets they no longer need?
3831 <sect2 id="server.scan">
3834 For servers that wish to offer the scan service one handler
3838 int (*bend_scan)(void *handle, bend_scan_rr *rr);
3841 BEND_SCAN_SUCCESS, /* ok */
3842 BEND_SCAN_PARTIAL /* not all entries could be found */
3845 typedef struct bend_scan_rr {
3846 int num_bases; /* number of elements in databaselist */
3847 char **basenames; /* databases to search */
3848 Odr_oid *attributeset;
3849 Z_ReferenceId *referenceId; /* reference ID */
3850 Z_AttributesPlusTerm *term;
3851 ODR stream; /* encoding stream - memory source if required */
3852 ODR print; /* printing stream */
3854 int *step_size; /* step size */
3855 int term_position; /* desired index of term in result list/returned */
3856 int num_entries; /* number of entries requested/returned */
3858 /* scan term entries. The called handler does not have
3859 to allocate this. Size of entries is num_entries (see above) */
3860 struct scan_entry *entries;
3861 bend_scan_status status;
3864 char *scanClause; /* CQL scan clause */
3865 char *setname; /* Scan in result set (NULL if omitted) */
3869 This backend server handles both Z39.50 scan
3870 and SRU scan. In order for a handler to distinguish between SRU (CQL) scan
3871 Z39.50 Scan, it must check for a non-NULL value of
3872 <literal>scanClause</literal>.
3876 If designed today, it would be a choice using a union or similar,
3877 but that would break binary compatibility with existing servers.
3882 <sect1 id="server.invocation">
3883 <title>Application Invocation</title>
3885 The finished application has the following
3886 invocation syntax (by way of <function>statserv_main()</function>):
3894 A listener specification consists of a transport mode followed by a
3895 colon (:) followed by a listener address. The transport mode is
3896 either <literal>tcp</literal>, <literal>unix:</literal> or
3897 <literal>ssl</literal>.
3900 For TCP and SSL, an address has the form
3903 hostname | IP-number [: portnumber]
3906 The port number defaults to 210 (standard Z39.50 port).
3909 For UNIX, the address is the filename of socket.
3912 For TCP/IP and SSL, the special hostnames <literal>@</literal>,
3913 maps to <literal>IN6ADDR_ANY_INIT</literal> with
3914 IPV4 binding as well (bindv6only=0),
3915 The special hostname <literal>@4</literal> binds to
3916 <literal>INADDR_ANY</literal> (IPV4 only listener).
3917 The special hostname <literal>@6</literal> binds to
3918 <literal>IN6ADDR_ANY_INIT</literal> with bindv6only=1 (IPV6 only listener).
3920 <example id="server.example.running.unix">
3921 <title>Running the GFS on Unix</title>
3923 Assuming the server application <replaceable>appname</replaceable> is
3924 started as root, the following will make it listen on port 210.
3925 The server will change identity to <literal>nobody</literal>
3926 and write its log to <filename>/var/log/app.log</filename>.
3928 application -l /var/log/app.log -u nobody tcp:@:210
3932 The server will accept Z39.50 requests and offer SRU service on port 210.
3935 <example id="server.example.apache.sru">
3936 <title>Setting up Apache as SRU Frontend</title>
3938 If you use <ulink url="&url.apache;">Apache</ulink>
3939 as your public web server and want to offer HTTP port 80
3940 access to the YAZ server on 210, you can use the
3941 <ulink url="&url.apache.directive.proxypass;">
3942 <literal>ProxyPass</literal></ulink>
3944 If you have virtual host
3945 <literal>srw.mydomain</literal> you can use the following directives
3946 in Apache's httpd.conf:
3949 ErrorLog /home/srw/logs/error_log
3950 TransferLog /home/srw/logs/access_log
3951 ProxyPass / http://srw.mydomain:210/
3956 The above is for the Apache 1.3 series.
3959 <example id="server.example.local.access">
3960 <title>Running a server with local access only</title>
3962 A server that is only being accessed from the local host should listen
3963 on UNIX file socket rather than an Internet socket. To listen on
3964 <filename>/tmp/mysocket</filename> start the server as follows:
3966 application unix:/tmp/mysocket
3971 <sect1 id="server.vhosts">
3972 <title>GFS Configuration and Virtual Hosts</title>
3977 <title>The Z39.50 ASN.1 Module</title>
3978 <sect1 id="asn.introduction">
3979 <title>Introduction</title>
3981 The &asn; module provides you with a set of C struct definitions for the
3982 various PDUs of the Z39.50 protocol, as well as for the complex types
3983 appearing within the PDUs. For the primitive data types, the C
3984 representation often takes the form of an ordinary C language type,
3985 such as <literal>Odr_int</literal> which is equivalent to an integral
3986 C integer. For ASN.1 constructs that have no direct
3987 representation in C, such as general octet strings and bit strings,
3988 the &odr; module (see section <link linkend="odr">The ODR Module</link>)
3989 provides auxiliary definitions.
3992 The &asn; module is located in sub directory <filename>z39.50</filename>.
3993 There you'll find C files that implement encoders and decoders for the
3994 Z39.50 types. You'll also find the protocol definitions:
3995 <filename>z3950v3.asn</filename>, <filename>esupdate.asn</filename>,
3999 <sect1 id="asn.preparing">
4000 <title>Preparing PDUs</title>
4002 A structure representing a complex ASN.1 type doesn't in itself contain the
4003 members of that type. Instead, the structure contains
4004 <emphasis>pointers</emphasis> to the members of the type.
4005 This is necessary, in part, to allow a mechanism for specifying which
4006 of the optional structure (SEQUENCE) members are present, and which
4007 are not. It follows that you will need to somehow provide space for
4008 the individual members of the structure, and set the pointers to
4009 refer to the members.
4012 The conversion routines don't care how you allocate and maintain your
4013 C structures - they just follow the pointers that you provide.
4014 Depending on the complexity of your application, and your personal
4015 taste, there are at least three different approaches that you may take
4016 when you allocate the structures.
4019 You can use static or automatic local variables in the function that
4020 prepares the PDU. This is a simple approach, and it provides the most
4021 efficient form of memory management. While it works well for flat
4022 PDUs like the InitReqest, it will generally not be sufficient for say,
4023 the generation of an arbitrarily complex RPN query structure.
4026 You can individually create the structure and its members using the
4027 <function>malloc(2)</function> function. If you want to ensure that
4028 the data is freed when it is no longer needed, you will have to
4029 define a function that individually releases each member of a
4030 structure before freeing the structure itself.
4033 You can use the <function>odr_malloc()</function> function (see
4034 <xref linkend="odr.use"/> for details). When you use
4035 <function>odr_malloc()</function>, you can release all of the
4036 allocated data in a single operation, independent of any pointers and
4037 relations between the data. The <function>odr_malloc()</function> function
4038 is based on a "nibble-memory"
4039 scheme, in which large portions of memory are allocated, and then
4040 gradually handed out with each call to <function>odr_malloc()</function>.
4041 The next time you call <function>odr_reset()</function>, all of the
4042 memory allocated since the last call is recycled for future use (actually,
4043 it is placed on a free-list).
4046 You can combine all of the methods described here. This will often be
4047 the most practical approach. For instance, you might use
4048 <function>odr_malloc()</function> to allocate an entire structure and
4049 some of its elements, while you leave other elements pointing to global
4050 or per-session default variables.
4053 The &asn; module provides an important aid in creating new PDUs. For
4054 each of the PDU types (say, <function>Z_InitRequest</function>), a
4055 function is provided that allocates and initializes an instance of
4056 that PDU type for you. In the case of the InitRequest, the function is
4057 simply named <function>zget_InitRequest()</function>, and it sets up
4058 reasonable default value for all of the mandatory members. The optional
4059 members are generally initialized to null pointers. This last aspect
4060 is very important: it ensures that if the PDU definitions are
4061 extended after you finish your implementation (to accommodate
4062 new versions of the protocol, say), you won't get into trouble with
4063 uninitialized pointers in your structures. The functions use
4064 <function>odr_malloc()</function> to
4065 allocate the PDUs and its members, so you can free everything again with a
4066 single call to <function>odr_reset()</function>. We strongly recommend
4067 that you use the <literal>zget_*</literal>
4068 functions whenever you are preparing a PDU (in a C++ API, the
4069 <literal>zget_</literal>
4070 functions would probably be promoted to constructors for the
4074 The prototype for the individual PDU types generally look like this:
4077 Z_<type> *zget_<type>(ODR o);
4083 Z_InitRequest *zget_InitRequest(ODR o);
4086 The &odr; handle should generally be your encoding stream, but it
4090 As well as the individual PDU functions, a function
4091 <function>zget_APDU()</function> is provided, which allocates
4092 a top-level Z-APDU of the type requested:
4095 Z_APDU *zget_APDU(ODR o, int which);
4098 The <varname>which</varname> parameter is (of course) the discriminator
4099 belonging to the <varname>Z_APDU</varname> <literal>CHOICE</literal> type.
4100 All of the interface described here is provided by the &asn; module, and
4101 you access it through the <filename>proto.h</filename> header file.
4104 <sect1 id="asn.external">
4105 <title>EXTERNAL Data</title>
4107 In order to achieve extensibility and adaptability to different
4108 application domains, the new version of the protocol defines many
4109 structures outside of the main ASN.1 specification, referencing them
4110 through ASN.1 EXTERNAL constructs. To simplify the construction and
4111 access to the externally referenced data, the &asn; module defines a
4112 specialized version of the EXTERNAL construct, called
4113 <literal>Z_External</literal>.It is defined thus:
4116 typedef struct Z_External
4118 Odr_oid *direct_reference;
4119 int *indirect_reference;
4124 Z_External_single = 0,
4126 Z_External_arbitrary,
4128 /* Specific types */
4130 Z_External_explainRecord,
4131 Z_External_resourceReport1,
4132 Z_External_resourceReport2
4140 Odr_any *single_ASN1_type;
4141 Odr_oct *octet_aligned;
4142 Odr_bitmask *arbitrary;
4144 /* Specific types */
4146 Z_ExplainRecord *explainRecord;
4147 Z_ResourceReport1 *resourceReport1;
4148 Z_ResourceReport2 *resourceReport2;
4156 When decoding, the &asn; module will attempt to determine which
4157 syntax describes the data by looking at the reference fields
4158 (currently only the direct-reference). For ASN.1 structured data, you
4159 need only consult the <literal>which</literal> field to determine the
4160 type of data. You can the access the data directly through the union.
4161 When constructing data for encoding, you set the union pointer to point
4162 to the data, and set the <literal>which</literal> field accordingly.
4163 Remember also to set the direct (or indirect) reference to the correct
4164 OID for the data type.
4165 For non-ASN.1 data such as MARC records, use the
4166 <literal>octet_aligned</literal> arm of the union.
4169 Some servers return ASN.1 structured data values (e.g. database
4170 records) as BER-encoded records placed in the
4171 <literal>octet-aligned</literal> branch of the EXTERNAL CHOICE.
4172 The ASN-module will <emphasis>not</emphasis> automatically decode
4173 these records. To help you decode the records in the application, the
4177 Z_ext_typeent *z_ext_gettypebyref(const oid *oid);
4180 can be used to retrieve information about the known, external data
4181 types. The function returns a pointer to a static area, or NULL, if no
4182 match for the given direct reference is found. The
4183 <literal>Z_ext_typeent</literal>
4187 typedef struct Z_ext_typeent
4189 int oid[OID_SIZE]; /* the direct-reference OID. */
4190 int what; /* discriminator value for the external CHOICE */
4191 Odr_fun fun; /* decoder function */
4195 The <literal>what</literal> member contains the
4196 <literal>Z_External</literal> union discriminator value for the
4197 given type: For the SUTRS record syntax, the value would be
4198 <literal>Z_External_sutrs</literal>.
4199 The <literal>fun</literal> member contains a pointer to the
4200 function which encodes/decodes the given type. Again, for the SUTRS
4201 record syntax, the value of <literal>fun</literal> would be
4202 <literal>z_SUTRS</literal> (a function pointer).
4205 If you receive an EXTERNAL which contains an octet-string value that
4206 you suspect of being an ASN.1-structured data value, you can use
4207 <literal>z_ext_gettypebyref</literal> to look for the provided
4209 If the return value is different from NULL, you can use the provided
4210 function to decode the BER string (see <xref linkend="odr.use"/>
4214 If you want to <emphasis>send</emphasis> EXTERNALs containing
4215 ASN.1-structured values in the octet-aligned branch of the CHOICE, this
4216 is possible too. However, on the encoding phase, it requires a somewhat
4217 involved juggling around of the various buffers involved.
4220 If you need to add new, externally defined data types, you must update
4221 the struct above, in the source file <filename>prt-ext.h</filename>, as
4222 well as the encoder/decoder in the file <filename>prt-ext.c</filename>.
4223 When changing the latter, remember to update both the
4224 <literal>arm</literal> arrary and the list
4225 <literal>type_table</literal>, which drives the CHOICE biasing that
4226 is necessary to tell the different, structured types apart
4231 Eventually, the EXTERNAL processing will most likely
4232 automatically insert the correct OIDs or indirect-refs. First,
4233 however, we need to determine how application-context management
4234 (specifically the presentation-context-list) should fit into the
4239 <sect1 id="asn.pdu">
4240 <title>PDU Contents Table</title>
4242 We include, for reference, a listing of the fields of each top-level
4243 PDU, as well as their default settings.
4245 <table frame="top" id="asn.default.initialize.request">
4246 <title>Default settings for PDU Initialize Request</title>
4248 <colspec colwidth="7*" colname="field"></colspec>
4249 <colspec colwidth="5*" colname="type"></colspec>
4250 <colspec colwidth="7*" colname="value"></colspec>
4253 <entry>Field</entry>
4255 <entry>Default Value</entry>
4260 referenceId</entry><entry>Z_ReferenceId</entry><entry>NULL
4263 protocolVersion</entry><entry>Odr_bitmask</entry><entry>Empty bitmask
4266 options</entry><entry>Odr_bitmask</entry><entry>Empty bitmask
4269 preferredMessageSize</entry><entry>Odr_int</entry><entry>30*1024
4272 maximumRecordSize</entry><entry>Odr_int</entry><entry>30*1024
4275 idAuthentication</entry><entry>Z_IdAuthentication</entry><entry>NULL
4278 implementationId</entry><entry>char*</entry><entry>"81"
4281 implementationName</entry><entry>char*</entry><entry>"YAZ"
4284 implementationVersion</entry><entry>char*</entry><entry>YAZ_VERSION
4287 userInformationField</entry><entry>Z_UserInformation</entry><entry>NULL
4290 otherInfo</entry><entry>Z_OtherInformation</entry><entry>NULL
4295 <table frame="top" id="asn.default.initialize.response">
4296 <title>Default settings for PDU Initialize Response</title>
4298 <colspec colwidth="7*" colname="field"></colspec>
4299 <colspec colwidth="5*" colname="type"></colspec>
4300 <colspec colwidth="7*" colname="value"></colspec>
4303 <entry>Field</entry>
4305 <entry>Default Value</entry>
4310 referenceId</entry><entry>Z_ReferenceId</entry><entry>NULL
4313 protocolVersion</entry><entry>Odr_bitmask</entry><entry>Empty bitmask
4316 options</entry><entry>Odr_bitmask</entry><entry>Empty bitmask
4319 preferredMessageSize</entry><entry>Odr_int</entry><entry>30*1024
4322 maximumRecordSize</entry><entry>Odr_int</entry><entry>30*1024
4325 result</entry><entry>Odr_bool</entry><entry>TRUE
4328 implementationId</entry><entry>char*</entry><entry>"id)"
4331 implementationName</entry><entry>char*</entry><entry>"YAZ"
4334 implementationVersion</entry><entry>char*</entry><entry>YAZ_VERSION
4337 userInformationField</entry><entry>Z_UserInformation</entry><entry>NULL
4340 otherInfo</entry><entry>Z_OtherInformation</entry><entry>NULL
4345 <table frame="top" id="asn.default.search.request">
4346 <title>Default settings for PDU Search Request</title>
4348 <colspec colwidth="7*" colname="field"></colspec>
4349 <colspec colwidth="5*" colname="type"></colspec>
4350 <colspec colwidth="7*" colname="value"></colspec>
4353 <entry>Field</entry>
4355 <entry>Default Value</entry>
4360 referenceId</entry><entry>Z_ReferenceId</entry><entry>NULL
4363 smallSetUpperBound</entry><entry>Odr_int</entry><entry>0
4366 largeSetLowerBound</entry><entry>Odr_int</entry><entry>1
4369 mediumSetPresentNumber</entry><entry>Odr_int</entry><entry>0
4372 replaceIndicator</entry><entry>Odr_bool</entry><entry>TRUE
4375 resultSetName</entry><entry>char *</entry><entry>"default"
4378 num_databaseNames</entry><entry>Odr_int</entry><entry>0
4381 databaseNames</entry><entry>char **</entry><entry>NULL
4384 smallSetElementSetNames</entry><entry>Z_ElementSetNames
4388 mediumSetElementSetNames</entry><entry>Z_ElementSetNames
4392 preferredRecordSyntax</entry><entry>Odr_oid</entry><entry>NULL
4395 query</entry><entry>Z_Query</entry><entry>NULL
4398 additionalSearchInfo</entry><entry>Z_OtherInformation
4402 otherInfo</entry><entry>Z_OtherInformation</entry><entry>NULL
4407 <table frame="top" id="asn.default.search.response">
4408 <title>Default settings for PDU Search Response</title>
4410 <colspec colwidth="7*" colname="field"></colspec>
4411 <colspec colwidth="5*" colname="type"></colspec>
4412 <colspec colwidth="7*" colname="value"></colspec>
4415 <entry>Field</entry>
4417 <entry>Default Value</entry>
4422 referenceId</entry><entry>Z_ReferenceId</entry><entry>NULL
4425 resultCount</entry><entry>Odr_int</entry><entry>0
4428 numberOfRecordsReturned</entry><entry>Odr_int</entry><entry>0
4431 nextResultSetPosition</entry><entry>Odr_int</entry><entry>0
4434 searchStatus</entry><entry>Odr_bool</entry><entry>TRUE
4437 resultSetStatus</entry><entry>Odr_int</entry><entry>NULL
4440 presentStatus</entry><entry>Odr_int</entry><entry>NULL
4443 records</entry><entry>Z_Records</entry><entry>NULL
4446 additionalSearchInfo</entry>
4447 <entry>Z_OtherInformation</entry><entry>NULL
4450 otherInfo</entry><entry>Z_OtherInformation</entry><entry>NULL
4455 <table frame="top" id="asn.default.present.request">
4456 <title>Default settings for PDU Present Request</title>
4458 <colspec colwidth="7*" colname="field"></colspec>
4459 <colspec colwidth="5*" colname="type"></colspec>
4460 <colspec colwidth="7*" colname="value"></colspec>
4463 <entry>Field</entry>
4465 <entry>Default Value</entry>
4470 referenceId</entry><entry>Z_ReferenceId</entry><entry>NULL
4473 resultSetId</entry><entry>char*</entry><entry>"default"
4476 resultSetStartPoint</entry><entry>Odr_int</entry><entry>1
4479 numberOfRecordsRequested</entry><entry>Odr_int</entry><entry>10
4482 num_ranges</entry><entry>Odr_int</entry><entry>0
4485 additionalRanges</entry><entry>Z_Range</entry><entry>NULL
4488 recordComposition</entry><entry>Z_RecordComposition</entry><entry>NULL
4491 preferredRecordSyntax</entry><entry>Odr_oid</entry><entry>NULL
4494 maxSegmentCount</entry><entry>Odr_int</entry><entry>NULL
4497 maxRecordSize</entry><entry>Odr_int</entry><entry>NULL
4500 maxSegmentSize</entry><entry>Odr_int</entry><entry>NULL
4503 otherInfo</entry><entry>Z_OtherInformation</entry><entry>NULL
4508 <table frame="top" id="asn.default.present.response">
4509 <title>Default settings for PDU Present Response</title>
4511 <colspec colwidth="7*" colname="field"></colspec>
4512 <colspec colwidth="5*" colname="type"></colspec>
4513 <colspec colwidth="7*" colname="value"></colspec>
4516 <entry>Field</entry>
4518 <entry>Default Value</entry>
4523 referenceId</entry><entry>Z_ReferenceId</entry><entry>NULL
4526 numberOfRecordsReturned</entry><entry>Odr_int</entry><entry>0
4529 nextResultSetPosition</entry><entry>Odr_int</entry><entry>0
4532 presentStatus</entry><entry>Odr_int</entry><entry>Z_PresentStatus_success
4535 records</entry><entry>Z_Records</entry><entry>NULL
4538 otherInfo</entry><entry>Z_OtherInformation</entry><entry>NULL
4543 <table frame="top" id="asn.default.delete.result.set.request">
4544 <title>Default settings for Delete Result Set Request</title>
4546 <colspec colwidth="7*" colname="field"></colspec>
4547 <colspec colwidth="5*" colname="type"></colspec>
4548 <colspec colwidth="7*" colname="value"></colspec>
4551 <entry>Field</entry>
4553 <entry>Default Value</entry>
4557 <row><entry>referenceId
4558 </entry><entry>Z_ReferenceId</entry><entry>NULL
4561 deleteFunction</entry><entry>Odr_int</entry><entry>Z_DeleteResultSetRequest_list
4564 num_ids</entry><entry>Odr_int</entry><entry>0
4567 resultSetList</entry><entry>char**</entry><entry>NULL
4570 otherInfo</entry><entry>Z_OtherInformation</entry><entry>NULL
4575 <table frame="top" id="asn.default.delete.result.set.response">
4576 <title>Default settings for Delete Result Set Response</title>
4578 <colspec colwidth="7*" colname="field"></colspec>
4579 <colspec colwidth="5*" colname="type"></colspec>
4580 <colspec colwidth="7*" colname="value"></colspec>
4583 <entry>Field</entry>
4585 <entry>Default Value</entry>
4590 referenceId</entry><entry>Z_ReferenceId</entry><entry>NULL
4593 deleteOperationStatus</entry><entry>Odr_int</entry>
4594 <entry>Z_DeleteStatus_success</entry></row>
4596 num_statuses</entry><entry>Odr_int</entry><entry>0
4599 deleteListStatuses</entry><entry>Z_ListStatus**</entry><entry>NULL
4602 numberNotDeleted</entry><entry>Odr_int</entry><entry>NULL
4605 num_bulkStatuses</entry><entry>Odr_int</entry><entry>0
4608 bulkStatuses</entry><entry>Z_ListStatus</entry><entry>NUL
4611 deleteMessage</entry><entry>char*</entry><entry>NULL
4614 otherInfo</entry><entry>Z_OtherInformation</entry><entry>NULL
4619 <table frame="top" id="asn.default.scan.request">
4620 <title>Default settings for Scan Request</title>
4622 <colspec colwidth="7*" colname="field"></colspec>
4623 <colspec colwidth="5*" colname="type"></colspec>
4624 <colspec colwidth="7*" colname="value"></colspec>
4627 <entry>Field</entry>
4629 <entry>Default Value</entry>
4634 referenceId</entry><entry>Z_ReferenceId</entry><entry>NULL
4637 num_databaseNames</entry><entry>Odr_int</entry><entry>0
4640 databaseNames</entry><entry>char**</entry><entry>NULL
4643 attributeSet</entry><entry>Odr_oid</entry><entry>NULL
4646 termListAndStartPoint</entry><entry>Z_AttributesPlus...
4647 </entry><entry>NULL</entry></row>
4649 stepSize</entry><entry>Odr_int</entry><entry>NULL
4652 numberOfTermsRequested</entry><entry>Odr_int</entry><entry>20
4655 preferredPositionInResponse</entry><entry>Odr_int</entry><entry>NULL
4658 otherInfo</entry><entry>Z_OtherInformation</entry><entry>NULL
4663 <table frame="top" id="asn.default.scan.response">
4664 <title>Default settings for Scan Response</title>
4666 <colspec colwidth="7*" colname="field"></colspec>
4667 <colspec colwidth="5*" colname="type"></colspec>
4668 <colspec colwidth="7*" colname="value"></colspec>
4671 <entry>Field</entry>
4673 <entry>Default Value</entry>
4678 referenceId</entry><entry>Z_ReferenceId</entry><entry>NULL
4681 stepSize</entry><entry>Odr_int</entry><entry>NULL
4684 scanStatus</entry><entry>Odr_int</entry><entry>Z_Scan_success
4687 numberOfEntriesReturned</entry><entry>Odr_int</entry><entry>0
4690 positionOfTerm</entry><entry>Odr_int</entry><entry>NULL
4693 entries</entry><entry>Z_ListEntris</entry><entry>NULL
4696 attributeSet</entry><entry>Odr_oid</entry><entry>NULL
4699 otherInfo</entry><entry>Z_OtherInformation</entry><entry>NULL
4704 <table frame="top" id="asn.default.trigger.resource.control.request">
4705 <title>Default settings for Trigger Resource Control Request</title>
4707 <colspec colwidth="7*" colname="field"></colspec>
4708 <colspec colwidth="5*" colname="type"></colspec>
4709 <colspec colwidth="7*" colname="value"></colspec>
4712 <entry>Field</entry>
4714 <entry>Default Value</entry>
4719 referenceId</entry><entry>Z_ReferenceId</entry><entry>NULL
4722 requestedAction</entry><entry>Odr_int</entry><entry>
4723 Z_TriggerResourceCtrl_resou..
4726 prefResourceReportFormat</entry><entry>Odr_oid</entry><entry>NULL
4729 resultSetWanted</entry><entry>Odr_bool</entry><entry>NULL
4732 otherInfo</entry><entry>Z_OtherInformation</entry><entry>NULL
4737 <table frame="top" id="asn.default.resource.control.request">
4738 <title>Default settings for Resource Control Request</title>
4740 <colspec colwidth="7*" colname="field"></colspec>
4741 <colspec colwidth="5*" colname="type"></colspec>
4742 <colspec colwidth="7*" colname="value"></colspec>
4745 <entry>Field</entry>
4747 <entry>Default Value</entry>
4752 referenceId</entry><entry>Z_ReferenceId</entry><entry>NULL
4755 suspendedFlag</entry><entry>Odr_bool</entry><entry>NULL
4758 resourceReport</entry><entry>Z_External</entry><entry>NULL
4761 partialResultsAvailable</entry><entry>Odr_int</entry><entry>NULL
4764 responseRequired</entry><entry>Odr_bool</entry><entry>FALSE
4767 triggeredRequestFlag</entry><entry>Odr_bool</entry><entry>NULL
4770 otherInfo</entry><entry>Z_OtherInformation</entry><entry>NULL
4775 <table frame="top" id="asn.default.resource.control.response">
4776 <title>Default settings for Resource Control Response</title>
4778 <colspec colwidth="7*" colname="field"></colspec>
4779 <colspec colwidth="5*" colname="type"></colspec>
4780 <colspec colwidth="7*" colname="value"></colspec>
4783 <entry>Field</entry>
4785 <entry>Default Value</entry>
4790 referenceId</entry><entry>Z_ReferenceId</entry><entry>NULL
4793 continueFlag</entry><entry>bool_t</entry><entry>TRUE
4796 resultSetWanted</entry><entry>bool_t</entry><entry>NULL
4799 otherInfo</entry><entry>Z_OtherInformation</entry><entry>NULL
4804 <table frame="top" id="asn.default.access.control.request">
4805 <title>Default settings for Access Control Request</title>
4807 <colspec colwidth="7*" colname="field"></colspec>
4808 <colspec colwidth="5*" colname="type"></colspec>
4809 <colspec colwidth="7*" colname="value"></colspec>
4812 <entry>Field</entry>
4814 <entry>Default Value</entry>
4819 referenceId</entry><entry>Z_ReferenceId</entry><entry>NULL
4822 which</entry><entry>enum</entry><entry>Z_AccessRequest_simpleForm;
4825 u</entry><entry>union</entry><entry>NULL
4828 otherInfo</entry><entry>Z_OtherInformation</entry><entry>NULL
4833 <table frame="top" id="asn.default.access.control.response">
4834 <title>Default settings for Access Control Response</title>
4836 <colspec colwidth="7*" colname="field"></colspec>
4837 <colspec colwidth="5*" colname="type"></colspec>
4838 <colspec colwidth="7*" colname="value"></colspec>
4841 <entry>Field</entry>
4843 <entry>Default Value</entry>
4848 referenceId</entry><entry>Z_ReferenceId</entry><entry>NULL
4851 which</entry><entry>enum</entry><entry>Z_AccessResponse_simpleForm
4854 u</entry><entry>union</entry><entry>NULL
4857 diagnostic</entry><entry>Z_DiagRec</entry><entry>NULL
4860 otherInfo</entry><entry>Z_OtherInformation</entry><entry>NULL
4865 <table frame="top" id="asn.default.segment">
4866 <title>Default settings for Segment</title>
4868 <colspec colwidth="7*" colname="field"></colspec>
4869 <colspec colwidth="5*" colname="type"></colspec>
4870 <colspec colwidth="7*" colname="value"></colspec>
4873 <entry>Field</entry>
4875 <entry>Default Value</entry>
4880 referenceId</entry><entry>Z_ReferenceId</entry><entry>NULL
4883 numberOfRecordsReturned</entry><entry>Odr_int</entry><entry>value=0
4886 num_segmentRecords</entry><entry>Odr_int</entry><entry>0
4889 segmentRecords</entry><entry>Z_NamePlusRecord</entry><entry>NULL
4891 <row><entry>otherInfo</entry><entry>Z_OtherInformation</entry><entry>NULL
4896 <table frame="top" id="asn.default.close">
4897 <title>Default settings for Close</title>
4899 <colspec colwidth="7*" colname="field"></colspec>
4900 <colspec colwidth="5*" colname="type"></colspec>
4901 <colspec colwidth="7*" colname="value"></colspec>
4904 <entry>Field</entry>
4906 <entry>Default Value</entry>
4911 referenceId</entry><entry>Z_ReferenceId</entry><entry>NULL
4914 closeReason</entry><entry>Odr_int</entry><entry>Z_Close_finished
4917 diagnosticInformation</entry><entry>char*</entry><entry>NULL
4920 resourceReportFormat</entry><entry>Odr_oid</entry><entry>NULL
4923 resourceFormat</entry><entry>Z_External</entry><entry>NULL
4926 otherInfo</entry><entry>Z_OtherInformation</entry><entry>NULL
4934 <title>SOAP and SRU</title>
4935 <sect1 id="soap.introduction">
4936 <title>Introduction</title>
4938 &yaz; uses a very simple implementation of
4939 <ulink url="&url.soap;">SOAP</ulink> that only
4940 (currently) supports what is sufficient to offer SRU SOAP functionality.
4941 The implementation uses the
4942 <ulink url="&url.libxml2.api.tree;">tree API</ulink> of
4943 libxml2 to encode and decode SOAP packages.
4946 Like the Z39.50 ASN.1 module, the &yaz; SRU implementation uses
4947 simple C structs to represent SOAP packages as well as
4951 <sect1 id="soap.http">
4954 &yaz; only offers HTTP as transport carrier for SOAP, but it is
4955 relatively easy to change that.
4958 The following definition of <literal>Z_GDU</literal> (Generic Data
4959 Unit) allows for both HTTP and Z39.50 in one packet.
4962 #include <yaz/zgdu.h>
4964 #define Z_GDU_Z3950 1
4965 #define Z_GDU_HTTP_Request 2
4966 #define Z_GDU_HTTP_Response 3
4971 Z_HTTP_Request *HTTP_Request;
4972 Z_HTTP_Response *HTTP_Response;
4977 The corresponding Z_GDU encoder/decoder is <function>z_GDU</function>.
4978 The <literal>z3950</literal> is any of the known BER encoded Z39.50
4980 <literal>HTTP_Request</literal> and <literal>HTTP_Response</literal>
4981 is the HTTP Request and Response respectively.
4984 <sect1 id="soap.xml">
4985 <title>SOAP Packages</title>
4987 Every SOAP package in &yaz; is represented as follows:
4989 #include <yaz/soap.h>
5003 #define Z_SOAP_fault 1
5004 #define Z_SOAP_generic 2
5005 #define Z_SOAP_error 3
5009 Z_SOAP_Fault *fault;
5010 Z_SOAP_Generic *generic;
5011 Z_SOAP_Fault *soap_error;
5018 The <literal>fault</literal> and <literal>soap_error</literal>
5019 arms both represent a SOAP fault - struct
5020 <literal>Z_SOAP_Fault</literal>. Any other generic
5021 (valid) package is represented by <literal>Z_SOAP_Generic</literal>.
5024 The <literal>ns</literal> as part of <literal>Z_SOAP</literal>
5025 is the namespace for SOAP itself and reflects the SOAP
5026 version. For version 1.1 it is
5027 <literal>http://schemas.xmlsoap.org/soap/envelope/</literal>,
5028 for version 1.2 it is
5029 <literal>http://www.w3.org/2001/06/soap-envelope</literal>.
5032 int z_soap_codec(ODR o, Z_SOAP **pp,
5033 char **content_buf, int *content_len,
5034 Z_SOAP_Handler *handlers);
5037 The <literal>content_buf</literal> and <literal>content_len</literal>
5038 is XML buffer and length of buffer respectively.
5041 The <literal>handlers</literal> is a list of SOAP codec
5042 handlers - one handler for each service namespace. For SRU SOAP, the
5043 namespace would be <literal>http://www.loc.gov/zing/srw/v1.0/</literal>.
5046 When decoding, the <function>z_soap_codec</function>
5047 inspects the XML content
5048 and tries to match one of the services namespaces of the
5049 supplied handlers. If there is a match. a handler function
5050 is invoked which decodes that particular SOAP package.
5051 If successful, the returned <literal>Z_SOAP</literal> package will be
5052 of type <literal>Z_SOAP_Generic</literal>.
5053 Member <literal>no</literal> is
5054 set the offset of the handler that matched; <literal>ns</literal>
5055 is set to namespace of the matching handler; the void pointer
5056 <literal>p</literal> is set to the C data structure assocatiated
5060 When a NULL namespace is met (member <literal>ns</literal> below),
5061 that specifies end-of-list.
5064 Each handler is defined as follows:
5072 The <literal>ns</literal> is the namespace of the service associated with
5073 handler <literal>f</literal>. The <literal>client_data</literal>
5074 is user-defined data which is passed to the handler.
5077 The prototype for a SOAP service handler is:
5079 int handler(ODR o, void * ptr, void **handler_data,
5080 void *client_data, const char *ns);
5082 The <parameter>o</parameter> specifies the mode (decode/encode)
5083 as usual. The second argument, <parameter>ptr</parameter>,
5084 is a libxml2 tree node pointer (<literal>xmlNodePtr</literal>)
5085 and is a pointer to the <literal>Body</literal> element
5086 of the SOAP package. The <parameter>handler_data</parameter>
5087 is an opaque pointer to C definitions associated with the
5088 SOAP service. The <parameter>client_data</parameter> is the pointer
5089 which was set as part of the <literal>Z_SOAP_handler</literal>.
5090 Finally, <parameter>ns</parameter> is the service namespace.
5093 <sect1 id="soap.srw">
5096 SRU SOAP is just one implementation of a SOAP handler as described
5097 in the previous section.
5098 The encoder/decoder handler for SRU is defined as
5101 #include <yaz/srw.h>
5103 int yaz_srw_codec(ODR o, void * pptr,
5104 Z_SRW_GDU **handler_data,
5105 void *client_data, const char *ns);
5107 Here, <literal>Z_SRW_GDU</literal> is either
5108 searchRetrieveRequest or a searchRetrieveResponse.
5112 The xQuery and xSortKeys are not handled yet by
5113 the SRW implementation of &yaz;. Explain is also missing.
5114 Future versions of &yaz; will include these features.
5118 The definition of searchRetrieveRequest is:
5122 #define Z_SRW_query_type_cql 1
5123 #define Z_SRW_query_type_xcql 2
5124 #define Z_SRW_query_type_pqf 3
5132 #define Z_SRW_sort_type_none 1
5133 #define Z_SRW_sort_type_sort 2
5134 #define Z_SRW_sort_type_xSort 3
5142 int *maximumRecords;
5144 char *recordPacking;
5146 } Z_SRW_searchRetrieveRequest;
5148 Please observe that data of type xsd:string is represented
5149 as a char pointer (<literal>char *</literal>). A null pointer
5150 means that the element is absent.
5151 Data of type xsd:integer is represented as a pointer to
5152 an int (<literal>int *</literal>). Again, a null pointer
5153 is used for absent elements.
5156 The SearchRetrieveResponse has the following definition.
5159 int * numberOfRecords;
5161 int * resultSetIdleTime;
5163 Z_SRW_record *records;
5166 Z_SRW_diagnostic *diagnostics;
5167 int num_diagnostics;
5168 int *nextRecordPosition;
5169 } Z_SRW_searchRetrieveResponse;
5171 The <literal>num_records</literal> and <literal>num_diagnostics</literal>
5172 is number of returned records and diagnostics respectively, and also
5173 correspond to the "size of" arrays <literal>records</literal>
5174 and <literal>diagnostics</literal>.
5177 A retrieval record is defined as follows:
5181 char *recordData_buf;
5183 int *recordPosition;
5186 The record data is defined as a buffer of some length so that
5187 data can be of any type. SRW 1.0 currenly doesn't allow for this
5188 (only XML), but future versions might do.
5191 And, a diagnostic as:
5201 <chapter id="tools">
5202 <title>Supporting Tools</title>
5204 In support of the service API - primarily the ASN module, which
5205 provides the programmatic interface to the Z39.50 APDUs, &yaz; contains
5206 a collection of tools that support the development of applications.
5208 <sect1 id="tools.query">
5209 <title>Query Syntax Parsers</title>
5211 Since the type-1 (RPN) query structure has no direct, useful string
5212 representation, every origin application needs to provide some form of
5213 mapping from a local query notation or representation to a
5214 <token>Z_RPNQuery</token> structure. Some programmers will prefer to
5215 construct the query manually, perhaps using
5216 <function>odr_malloc()</function> to simplify memory management.
5217 The &yaz; distribution includes three separate, query-generating tools
5218 that may be of use to you.
5221 <title>Prefix Query Format</title>
5223 Since RPN or reverse polish notation is really just a fancy way of
5224 describing a suffix notation format (operator follows operands), it
5225 would seem that the confusion is total when we now introduce a prefix
5226 notation for RPN. The reason is one of simple laziness - it's somewhat
5227 simpler to interpret a prefix format, and this utility was designed
5228 for maximum simplicity, to provide a baseline representation for use
5229 in simple test applications and scripting environments (like Tcl). The
5230 demonstration client included with YAZ uses the PQF.
5234 The PQF has been adopted by other parties developing Z39.50
5235 software. It is often referred to as Prefix Query Notation
5240 The PQF is defined by the pquery module in the YAZ library.
5241 There are two sets of functions that have similar behavior. First
5242 set operates on a PQF parser handle, second set doesn't. First set
5243 of functions are more flexible than the second set. Second set
5244 is obsolete and is only provided to ensure backwards compatibility.
5247 First set of functions all operate on a PQF parser handle:
5250 #include <yaz/pquery.h>
5252 YAZ_PQF_Parser yaz_pqf_create(void);
5254 void yaz_pqf_destroy(YAZ_PQF_Parser p);
5256 Z_RPNQuery *yaz_pqf_parse(YAZ_PQF_Parser p, ODR o, const char *qbuf);
5258 Z_AttributesPlusTerm *yaz_pqf_scan(YAZ_PQF_Parser p, ODR o,
5259 Odr_oid **attributeSetId, const char *qbuf);
5261 int yaz_pqf_error(YAZ_PQF_Parser p, const char **msg, size_t *off);
5264 A PQF parser is created and destructed by functions
5265 <function>yaz_pqf_create</function> and
5266 <function>yaz_pqf_destroy</function> respectively.
5267 Function <function>yaz_pqf_parse</function> parses the query given
5268 by string <literal>qbuf</literal>. If parsing was successful,
5269 a Z39.50 RPN Query is returned which is created using ODR stream
5270 <literal>o</literal>. If parsing failed, a NULL pointer is
5272 Function <function>yaz_pqf_scan</function> takes a scan query in
5273 <literal>qbuf</literal>. If parsing was successful, the function
5274 returns attributes plus term pointer and modifies
5275 <literal>attributeSetId</literal> to hold attribute set for the
5276 scan request - both allocated using ODR stream <literal>o</literal>.
5277 If parsing failed, yaz_pqf_scan returns a NULL pointer.
5278 Error information for bad queries can be obtained by a call to
5279 <function>yaz_pqf_error</function> which returns an error code and
5280 modifies <literal>*msg</literal> to point to an error description,
5281 and modifies <literal>*off</literal> to the offset within the last
5282 query where parsing failed.
5285 The second set of functions are declared as follows:
5288 #include <yaz/pquery.h>
5290 Z_RPNQuery *p_query_rpn(ODR o, oid_proto proto, const char *qbuf);
5292 Z_AttributesPlusTerm *p_query_scan(ODR o, oid_proto proto,
5293 Odr_oid **attributeSetP, const char *qbuf);
5295 int p_query_attset(const char *arg);
5298 The function <function>p_query_rpn()</function> takes as arguments an
5299 &odr; stream (see section <link linkend="odr">The ODR Module</link>)
5300 to provide a memory source (the structure created is released on
5301 the next call to <function>odr_reset()</function> on the stream), a
5302 protocol identifier (one of the constants <token>PROTO_Z3950</token> and
5303 <token>PROTO_SR</token>), an attribute set reference, and
5304 finally a null-terminated string holding the query string.
5307 If the parse went well, <function>p_query_rpn()</function> returns a
5308 pointer to a <literal>Z_RPNQuery</literal> structure which can be
5309 placed directly into a <literal>Z_SearchRequest</literal>.
5310 If parsing failed, due to syntax error, a NULL pointer is returned.
5313 The <literal>p_query_attset</literal> specifies which attribute set
5314 to use if the query doesn't specify one by the
5315 <literal>@attrset</literal> operator.
5316 The <literal>p_query_attset</literal> returns 0 if the argument is a
5317 valid attribute set specifier; otherwise the function returns -1.
5320 The grammar of the PQF is as follows:
5323 query ::= top-set query-struct.
5325 top-set ::= [ '@attrset' string ]
5327 query-struct ::= attr-spec | simple | complex | '@term' term-type query
5329 attr-spec ::= '@attr' [ string ] string query-struct
5331 complex ::= operator query-struct query-struct.
5333 operator ::= '@and' | '@or' | '@not' | '@prox' proximity.
5335 simple ::= result-set | term.
5337 result-set ::= '@set' string.
5341 proximity ::= exclusion distance ordered relation which-code unit-code.
5343 exclusion ::= '1' | '0' | 'void'.
5345 distance ::= integer.
5347 ordered ::= '1' | '0'.
5349 relation ::= integer.
5351 which-code ::= 'known' | 'private' | integer.
5353 unit-code ::= integer.
5355 term-type ::= 'general' | 'numeric' | 'string' | 'oid' | 'datetime' | 'null'.
5358 You will note that the syntax above is a fairly faithful
5359 representation of RPN, except for the Attribute, which has been
5360 moved a step away from the term, allowing you to associate one or more
5361 attributes with an entire query structure. The parser will
5362 automatically apply the given attributes to each term as required.
5365 The @attr operator is followed by an attribute specification
5366 (<literal>attr-spec</literal> above). The specification consists
5367 of an optional attribute set, an attribute type-value pair and
5368 a sub-query. The attribute type-value pair is packed in one string:
5369 an attribute type, an equals sign, and an attribute value, like this:
5370 <literal>@attr 1=1003</literal>.
5371 The type is always an integer, but the value may be either an
5372 integer or a string (if it doesn't start with a digit character).
5373 A string attribute-value is encoded as a Type-1 "complex"
5374 attribute with the list of values containing the single string
5375 specified, and including no semantic indicators.
5378 Version 3 of the Z39.50 specification defines various encoding of terms.
5379 Use <literal>@term </literal> <replaceable>type</replaceable>
5380 <replaceable>string</replaceable>,
5381 where type is one of: <literal>general</literal>,
5382 <literal>numeric</literal> or <literal>string</literal>
5383 (for InternationalString).
5384 If no term type has been given, the <literal>general</literal> form
5385 is used. This is the only encoding allowed in both versions 2 and 3
5386 of the Z39.50 standard.
5388 <sect3 id="PQF-prox">
5389 <title>Using Proximity Operators with PQF</title>
5392 This is an advanced topic, describing how to construct
5393 queries that make very specific requirements on the
5394 relative location of their operands.
5395 You may wish to skip this section and go straight to
5396 <link linkend="pqf-examples">the example PQF queries</link>.
5401 Most Z39.50 servers do not support proximity searching, or
5402 support only a small subset of the full functionality that
5403 can be expressed using the PQF proximity operator. Be
5404 aware that the ability to <emphasis>express</emphasis> a
5405 query in PQF is no guarantee that any given server will
5406 be able to <emphasis>execute</emphasis> it.
5412 The proximity operator <literal>@prox</literal> is a special
5413 and more restrictive version of the conjunction operator
5414 <literal>@and</literal>. Its semantics are described in
5415 section 3.7.2 (Proximity) of Z39.50 the standard itself, which
5416 can be read on-line at
5417 <ulink url="&url.z39.50.proximity;"/>
5420 In PQF, the proximity operation is represented by a sequence
5423 @prox <replaceable>exclusion</replaceable> <replaceable>distance</replaceable> <replaceable>ordered</replaceable> <replaceable>relation</replaceable> <replaceable>which-code</replaceable> <replaceable>unit-code</replaceable>
5425 in which the meanings of the parameters are as described in
5426 the standard, and they can take the following values:
5429 <formalpara><title>exclusion</title>
5431 0 = false (i.e. the proximity condition specified by the
5432 remaining parameters must be satisfied) or
5433 1 = true (the proximity condition specified by the
5434 remaining parameters must <emphasis>not</emphasis> be
5440 <formalpara><title>distance</title><para>
5441 An integer specifying the difference between the locations
5442 of the operands: e.g. two adjacent words would have
5443 distance=1 since their locations differ by one unit.
5445 </formalpara></listitem>
5447 <formalpara><title>ordered</title><para>
5448 1 = ordered (the operands must occur in the order the
5449 query specifies them) or
5450 0 = unordered (they may appear in either order).
5455 <formalpara><title>relation</title><para>
5456 Recognised values are
5458 2 (lessThanOrEqual),
5460 4 (greaterThanOrEqual),
5467 <formalpara><title>which-code</title><para>
5468 <literal>known</literal>
5470 <literal>k</literal>
5471 (the unit-code parameter is taken from the well-known list
5472 of alternatives described below) or
5473 <literal>private</literal>
5475 <literal>p</literal>
5476 (the unit-code parameter has semantics specific to an
5477 out-of-band agreement such as a profile).
5482 <formalpara><title>unit-code</title><para>
5483 If the which-code parameter is <literal>known</literal>
5484 then the recognised values are
5494 10 (elementType) and
5496 If which-code is <literal>private</literal> then the
5497 acceptable values are determined by the profile.
5502 (The numeric values of the relation and well-known unit-code
5503 parameters are taken straight from
5504 <ulink url="&url.z39.50.proximity.asn1;"
5505 >the ASN.1</ulink> of the proximity structure in the standard.)
5508 <sect3 id="pqf-examples">
5509 <title>PQF queries</title>
5510 <example id="example.pqf.simple.terms">
5511 <title>PQF queries using simple terms</title>
5520 <example id="pqf.example.pqf.boolean.operators">
5521 <title>PQF boolean operators</title>
5524 @or "dylan" "zimmerman"
5526 @and @or dylan zimmerman when
5528 @and when @or dylan zimmerman
5532 <example id="example.pqf.result.sets">
5533 <title>PQF references to result sets</title>
5538 @and @set seta @set setb
5542 <example id="example.pqf.attributes">
5543 <title>Attributes for terms</title>
5548 @attr 1=4 @attr 4=1 "self portrait"
5550 @attrset exp1 @attr 1=1 CategoryList
5552 @attr gils 1=2008 Copenhagen
5554 @attr 1=/book/title computer
5558 <example id="example.pqf.proximity">
5559 <title>PQF Proximity queries</title>
5562 @prox 0 3 1 2 k 2 dylan zimmerman
5564 Here the parameters 0, 3, 1, 2, k and 2 represent exclusion,
5565 distance, ordered, relation, which-code and unit-code, in that
5569 <para>exclusion = 0: the proximity condition must hold</para>
5572 <para>distance = 3: the terms must be three units apart</para>
5576 ordered = 1: they must occur in the order they are specified
5581 relation = 2: lessThanOrEqual (to the distance of 3 units)
5586 which-code is "known", so the standard unit-codes are used
5590 <para>unit-code = 2: word.</para>
5593 So the whole proximity query means that the words
5594 <literal>dylan</literal> and <literal>zimmerman</literal> must
5595 both occur in the record, in that order, differing in position
5596 by three or fewer words (i.e. with two or fewer words between
5597 them.) The query would find "Bob Dylan, aka. Robert
5598 Zimmerman", but not "Bob Dylan, born as Robert Zimmerman"
5599 since the distance in this case is four.
5602 <example id="example.pqf.search.term.type">
5603 <title>PQF specification of search term type</title>
5606 @term string "a UTF-8 string, maybe?"
5610 <example id="example.pqf.mixed.queries">
5611 <title>PQF mixed queries</title>
5614 @or @and bob dylan @set Result-1
5616 @attr 4=1 @and @attr 1=1 "bob dylan" @attr 1=4 "slow train coming"
5618 @and @attr 2=4 @attr gils 1=2038 -114 @attr 2=2 @attr gils 1=2039 -109
5620 The last of these examples is a spatial search: in
5621 <ulink url="http://www.gils.net/prof_v2.html#sec_7_4"
5622 >the GILS attribute set</ulink>,
5624 2038 indicates West Bounding Coordinate and
5625 2030 indicates East Bounding Coordinate,
5626 so the query is for areas extending from -114 degrees longitude
5627 to no more than -109 degrees longitude.
5632 <sect2 id="CCL"><title>CCL</title>
5634 Not all users enjoy typing in prefix query structures and numerical
5635 attribute values, even in a minimalistic test client. In the library
5636 world, the more intuitive Common Command Language - CCL (ISO 8777)
5637 has enjoyed some popularity - especially before the widespread
5638 availability of graphical interfaces. It is still useful in
5639 applications where you for some reason or other need to provide a
5640 symbolic language for expressing boolean query structures.
5642 <sect3 id="ccl.syntax">
5643 <title>CCL Syntax</title>
5645 The CCL parser obeys the following grammar for the FIND argument.
5646 The syntax is annotated using lines prefixed by
5647 <literal>--</literal>.
5650 CCL-Find ::= CCL-Find Op Elements
5653 Op ::= "and" | "or" | "not"
5654 -- The above means that Elements are separated by boolean operators.
5656 Elements ::= '(' CCL-Find ')'
5659 | Qualifiers Relation Terms
5660 | Qualifiers Relation '(' CCL-Find ')'
5661 | Qualifiers '=' string '-' string
5662 -- Elements is either a recursive definition, a result set reference, a
5663 -- list of terms, qualifiers followed by terms, qualifiers followed
5664 -- by a recursive definition or qualifiers in a range (lower - upper).
5666 Set ::= 'set' = string
5667 -- Reference to a result set
5669 Terms ::= Terms Prox Term
5671 -- Proximity of terms.
5673 Term ::= Term string
5675 -- This basically means that a term may include a blank
5677 Qualifiers ::= Qualifiers ',' string
5679 -- Qualifiers is a list of strings separated by comma
5681 Relation ::= '=' | '>=' | '<=' | '<>' | '>' | '<'
5682 -- Relational operators. This really doesn't follow the ISO8777
5686 -- Proximity operator
5689 <example id="example.ccl.queries">
5690 <title>CCL queries</title>
5692 The following queries are all valid:
5703 (dylan and bob) or set=1
5712 Assuming that the qualifiers <literal>ti</literal>
5713 and <literal>au</literal>
5714 and <literal>date</literal> are defined, we may use:
5719 au=(bob dylan and slow train coming)
5721 date>1980 and (ti=((self portrait)))
5725 <sect3 id="ccl.qualifiers">
5726 <title>CCL Qualifiers</title>
5728 Qualifiers are used to direct the search to a particular searchable
5729 index, such as title (ti) and author indexes (au). The CCL standard
5730 itself doesn't specify a particular set of qualifiers, but it does
5731 suggest a few short-hand notations. You can customize the CCL parser
5732 to support a particular set of qualifiers to reflect the current target
5733 profile. Traditionally, a qualifier would map to a particular
5734 use-attribute within the BIB-1 attribute set. It is also
5735 possible to set other attributes, such as the structure
5739 A CCL profile is a set of predefined CCL qualifiers that may be
5740 read from a file or set in the CCL API.
5741 The YAZ client reads its CCL qualifiers from a file named
5742 <filename>default.bib</filename>. There are four types of
5743 lines in a CCL profile: qualifier specification,
5744 qualifier alias, comments and directives.
5746 <sect4 id="ccl.qualifier.specification">
5747 <title>Qualifier specification</title>
5749 A qualifier specification is of the form:
5752 <replaceable>qualifier-name</replaceable>
5753 [<replaceable>attributeset</replaceable><literal>,</literal>]<replaceable>type</replaceable><literal>=</literal><replaceable>val</replaceable>
5754 [<replaceable>attributeset</replaceable><literal>,</literal>]<replaceable>type</replaceable><literal>=</literal><replaceable>val</replaceable> ...
5757 where <replaceable>qualifier-name</replaceable> is the name of the
5758 qualifier to be used (e.g. <literal>ti</literal>),
5759 <replaceable>type</replaceable> is attribute type in the attribute
5760 set (Bib-1 is used if no attribute set is given) and
5761 <replaceable>val</replaceable> is attribute value.
5762 The <replaceable>type</replaceable> can be specified as an
5763 integer, or as a single-letter:
5764 <literal>u</literal> for use,
5765 <literal>r</literal> for relation, <literal>p</literal> for position,
5766 <literal>s</literal> for structure,<literal>t</literal> for truncation,
5767 or <literal>c</literal> for completeness.
5768 The attributes for the special qualifier name <literal>term</literal>
5769 are used when no CCL qualifier is given in a query.
5770 <table id="ccl.common.bib1.attributes">
5771 <title>Common Bib-1 attributes</title>
5773 <colspec colwidth="2*" colname="type"></colspec>
5774 <colspec colwidth="9*" colname="description"></colspec>
5778 <entry>Description</entry>
5783 <entry><literal>u=</literal><replaceable>value</replaceable></entry>
5785 Use attribute (1). Common use attributes are
5786 1 Personal-name, 4 Title, 7 ISBN, 8 ISSN, 30 Date,
5787 62 Subject, 1003 Author, 1016 Any. Specify value
5792 <entry><literal>r=</literal><replaceable>value</replaceable></entry>
5794 Relation attribute (2). Common values are
5795 1 <, 2 <=, 3 =, 4 >=, 5 >, 6 <>,
5796 100 phonetic, 101 stem, 102 relevance, 103 always matches.
5800 <entry><literal>p=</literal><replaceable>value</replaceable></entry>
5802 Position attribute (3). Values: 1 first in field, 2
5803 first in any subfield, 3 any position in field.
5807 <entry><literal>s=</literal><replaceable>value</replaceable></entry>
5809 Structure attribute (4). Values: 1 phrase, 2 word,
5810 3 key, 4 year, 5 date, 6 word list, 100 date (un),
5811 101 name (norm), 102 name (un), 103 structure, 104 urx,
5812 105 free-form-text, 106 document-text, 107 local-number,
5813 108 string, 109 numeric string.
5817 <entry><literal>t=</literal><replaceable>value</replaceable></entry>
5819 Truncation attribute (5). Values: 1 right, 2 left,
5820 3 left and right, 100 none, 101 process #, 102 regular-1,
5821 103 regular-2, 104 CCL.
5825 <entry><literal>c=</literal><replaceable>value</replaceable></entry>
5827 Completeness attribute (6). Values: 1 incomplete subfield,
5828 2 complete subfield, 3 complete field.
5836 Refer to <xref linkend="bib1"/> or the complete
5837 <ulink url="&url.z39.50.attset.bib1;">list of Bib-1 attributes</ulink>
5840 It is also possible to specify non-numeric attribute values,
5841 which are used in combination with certain types.
5842 The special combinations are:
5843 <table id="ccl.special.attribute.combos">
5844 <title>Special attribute combos</title>
5846 <colspec colwidth="2*" colname="name"></colspec>
5847 <colspec colwidth="9*" colname="description"></colspec>
5851 <entry>Description</entry>
5856 <entry><literal>s=pw</literal></entry>
5858 The structure is set to either word or phrase depending
5859 on the number of tokens in a term (phrase-word).
5863 <entry><literal>s=al</literal></entry>
5865 Each token in the term is ANDed (and-list).
5866 This does not set the structure at all.
5869 <row><entry><literal>s=ol</literal></entry>
5871 Each token in the term is ORed (or-list).
5872 This does not set the structure at all.
5875 <row><entry><literal>s=ag</literal></entry>
5877 Tokens that appears as phrases (with blank in them) gets
5878 structure phrase attached (4=1). Tokens that appear to be words
5879 gets structure word attached (4=2). Phrases and words are
5880 ANDed. This is a variant of s=al and s=pw, with the main
5881 difference that words are not split (with operator AND)
5882 but instead kept in one RPN token. This facility appeared
5886 <row><entry><literal>s=sl</literal></entry>
5888 Tokens are split into sub-phrases of all combinations - in order.
5889 This facility appeared in YAZ 5.14.0.
5892 <row><entry><literal>r=o</literal></entry>
5894 Allows ranges and the operators greater-than, less-than, ...
5896 This sets Bib-1 relation attribute accordingly (relation
5897 ordered). A query construct is only treated as a range if
5898 dash is used and that is surrounded by white-space. So
5899 <literal>-1980</literal> is treated as term
5900 <literal>"-1980"</literal> not <literal><= 1980</literal>.
5901 If <literal>- 1980</literal> is used, however, that is
5905 <row><entry><literal>r=r</literal></entry>
5907 Similar to <literal>r=o</literal> but assumes that terms
5908 are non-negative (not prefixed with <literal>-</literal>).
5909 Thus, a dash will always be treated as a range.
5910 The construct <literal>1980-1990</literal> is
5911 treated as a range with <literal>r=r</literal> but as a
5912 single term <literal>"1980-1990"</literal> with
5913 <literal>r=o</literal>. The special attribute
5914 <literal>r=r</literal> is available in YAZ 2.0.24 or later.
5917 <row><entry><literal>r=omiteq</literal></entry>
5919 This will omit relation=equals (@attr 2=3) when r=o / r=r
5920 is used. This is useful for servers that somehow break
5921 when an explicit relation=equals is used. Omitting the
5922 relation is usually safe because "equals" is the default
5923 behavior. This tweak was added in YAZ version 5.1.2.
5926 <row><entry><literal>t=l</literal></entry>
5928 Allows term to be left-truncated.
5929 If term is of the form <literal>?x</literal>, the resulting
5930 Type-1 term is <literal>x</literal> and truncation is left.
5933 <row><entry><literal>t=r</literal></entry>
5935 Allows term to be right-truncated.
5936 If term is of the form <literal>x?</literal>, the resulting
5937 Type-1 term is <literal>x</literal> and truncation is right.
5940 <row><entry><literal>t=n</literal></entry>
5942 If term is does not include <literal>?</literal>, the
5943 truncation attribute is set to none (100).
5946 <row><entry><literal>t=b</literal></entry>
5948 Allows term to be both left-and-right truncated.
5949 If term is of the form <literal>?x?</literal>, the
5950 resulting term is <literal>x</literal> and trunctation is
5951 set to both left and right.
5954 <row><entry><literal>t=x</literal></entry>
5956 Allows masking anywhere in a term, thus fully supporting
5957 # (mask one character) and ? (zero or more of any).
5958 If masking is used, truncation is set to 102 (regexp-1 in term)
5959 and the term is converted accordingly to a regular expression.
5962 <row><entry><literal>t=z</literal></entry>
5964 Allows masking anywhere in a term, thus fully supporting
5965 # (mask one character) and ? (zero or more of any).
5966 If masking is used, truncation is set to 104 (Z39.58 in term)
5967 and the term is converted accordingly to Z39.58 masking term -
5968 actually the same truncation as CCL itself.
5975 <example id="example.ccl.profile">
5976 <title>CCL profile</title>
5978 Consider the following definition:
5988 <literal>ti</literal> and <literal>au</literal> both set
5989 structure attribute to phrase (s=1).
5990 <literal>ti</literal>
5991 sets the use-attribute to 4. <literal>au</literal> sets the
5993 When no qualifiers are used in the query, the structure-attribute is
5994 set to free-form-text (105) (rule for <literal>term</literal>).
5995 The <literal>date</literal> sets the relation attribute to
5996 the relation used in the CCL query and sets the use attribute
6000 You can combine attributes. To Search for "ranked title" you
6003 ti,ranked=knuth computer
6005 which will set relation=ranked, use=title, structure=phrase.
6012 is a valid query. But
6020 <sect4 id="ccl.qualifier.alias">
6021 <title>Qualifier alias</title>
6023 A qualifier alias is of the form:
6026 <replaceable>q</replaceable>
6027 <replaceable>q1</replaceable> <replaceable>q2</replaceable> ..
6030 which declares <replaceable>q</replaceable> to
6031 be an alias for <replaceable>q1</replaceable>,
6032 <replaceable>q2</replaceable>... such that the CCL
6033 query <replaceable>q=x</replaceable> is equivalent to
6034 <replaceable>q1=x or q2=x or ...</replaceable>.
6037 <sect4 id="ccl.comments">
6038 <title>Comments</title>
6040 Lines with white space or lines that begin with
6041 character <literal>#</literal> are treated as comments.
6044 <sect4 id="ccl.directives">
6045 <title>Directives</title>
6047 Directive specifications takes the form
6049 <para><literal>@</literal><replaceable>directive</replaceable> <replaceable>value</replaceable>
6051 <table id="ccl.directives.table">
6052 <title>CCL directives</title>
6054 <colspec colwidth="2*" colname="name"></colspec>
6055 <colspec colwidth="8*" colname="description"></colspec>
6056 <colspec colwidth="1*" colname="default"></colspec>
6060 <entry>Description</entry>
6061 <entry>Default</entry>
6066 <entry>truncation</entry>
6067 <entry>Truncation character</entry>
6068 <entry><literal>?</literal></entry>
6072 <entry>Masking character. Requires YAZ 4.2.58 or later</entry>
6073 <entry><literal>#</literal></entry>
6076 <entry>field</entry>
6077 <entry>Specifies how multiple fields are to be
6078 combined. There are two modes: <literal>or</literal>:
6079 multiple qualifier fields are ORed,
6080 <literal>merge</literal>: attributes for the qualifier
6081 fields are merged and assigned to one term.
6083 <entry><literal>merge</literal></entry>
6087 <entry>Specifies if CCL operators and qualifiers should be
6088 compared with case sensitivity or not. Specify 1 for
6089 case sensitive; 0 for case insensitive.</entry>
6090 <entry><literal>1</literal></entry>
6094 <entry>Specifies token for CCL operator AND.</entry>
6095 <entry><literal>and</literal></entry>
6099 <entry>Specifies token for CCL operator OR.</entry>
6100 <entry><literal>or</literal></entry>
6104 <entry>Specifies token for CCL operator NOT.</entry>
6105 <entry><literal>not</literal></entry>
6109 <entry>Specifies token for CCL operator SET.</entry>
6110 <entry><literal>set</literal></entry>
6117 <sect3 id="ccl.api">
6118 <title>CCL API</title>
6120 All public definitions can be found in the header file
6121 <filename>ccl.h</filename>. A profile identifier is of type
6122 <literal>CCL_bibset</literal>. A profile must be created with the call
6123 to the function <function>ccl_qual_mk</function> which returns a profile
6124 handle of type <literal>CCL_bibset</literal>.
6127 To read a file containing qualifier definitions the function
6128 <function>ccl_qual_file</function> may be convenient. This function
6129 takes an already opened <literal>FILE</literal> handle pointer as
6130 argument along with a <literal>CCL_bibset</literal> handle.
6133 To parse a simple string with a FIND query use the function
6136 struct ccl_rpn_node *ccl_find_str(CCL_bibset bibset, const char *str,
6137 int *error, int *pos);
6140 which takes the CCL profile (<literal>bibset</literal>) and query
6141 (<literal>str</literal>) as input. Upon successful completion the RPN
6142 tree is returned. If an error occurs, such as a syntax error, the integer
6143 pointed to by <literal>error</literal> holds the error code and
6144 <literal>pos</literal> holds the offset inside query string in which
6148 An English representation of the error may be obtained by calling
6149 the <literal>ccl_err_msg</literal> function. The error codes are
6150 listed in <filename>ccl.h</filename>.
6153 To convert the CCL RPN tree (type
6154 <literal>struct ccl_rpn_node *</literal>)
6155 to the Z_RPNQuery of YAZ the function <function>ccl_rpn_query</function>
6156 must be used. This function which is part of YAZ is implemented in
6157 <filename>yaz-ccl.c</filename>.
6158 After calling this function the CCL RPN tree is probably no longer
6159 needed. The <literal>ccl_rpn_delete</literal> destroys the CCL RPN tree.
6162 A CCL profile may be destroyed by calling the
6163 <function>ccl_qual_rm</function> function.
6166 The token names for the CCL operators may be changed by setting the
6167 globals (all type <literal>char *</literal>)
6168 <literal>ccl_token_and</literal>, <literal>ccl_token_or</literal>,
6169 <literal>ccl_token_not</literal> and <literal>ccl_token_set</literal>.
6170 An operator may have aliases, i.e. there may be more than one name for
6171 the operator. To do this, separate each alias with a space character.
6178 <ulink url="&url.cql;">CQL</ulink>
6179 - Common Query Language - was defined for the
6180 <ulink url="&url.sru;">SRU</ulink> protocol.
6181 In many ways CQL has a similar syntax to CCL.
6182 The objective of CQL is different. Where CCL aims to be
6183 an end-user language, CQL is <emphasis>the</emphasis> protocol
6184 query language for SRU.
6188 If you are new to CQL, read the
6189 <ulink url="&url.cql.intro;">Gentle Introduction</ulink>.
6193 The CQL parser in &yaz; provides the following:
6197 It parses and validates a CQL query.
6202 It generates a C structure that allows you to convert
6203 a CQL query to some other query language, such as SQL.
6208 The parser converts a valid CQL query to PQF, thus providing a
6209 way to use CQL for both SRU servers and Z39.50 targets at the
6215 The parser converts CQL to XCQL.
6216 XCQL is an XML representation of CQL.
6217 XCQL is part of the SRU specification. However, since SRU
6218 supports CQL only, we don't expect XCQL to be widely used.
6219 Furthermore, CQL has the advantage over XCQL that it is
6225 <sect3 id="cql.parsing">
6226 <title>CQL parsing</title>
6228 A CQL parser is represented by the <literal>CQL_parser</literal>
6229 handle. Its contents should be considered &yaz; internal (private).
6231 #include <yaz/cql.h>
6233 typedef struct cql_parser *CQL_parser;
6235 CQL_parser cql_parser_create(void);
6236 void cql_parser_destroy(CQL_parser cp);
6238 A parser is created by <function>cql_parser_create</function> and
6239 is destroyed by <function>cql_parser_destroy</function>.
6242 To parse a CQL query string, the following function
6245 int cql_parser_string(CQL_parser cp, const char *str);
6247 A CQL query is parsed by the <function>cql_parser_string</function>
6248 which takes a query <parameter>str</parameter>.
6249 If the query was valid (no syntax errors), then zero is returned;
6250 otherwise -1 is returned to indicate a syntax error.
6254 int cql_parser_stream(CQL_parser cp,
6255 int (*getbyte)(void *client_data),
6256 void (*ungetbyte)(int b, void *client_data),
6259 int cql_parser_stdio(CQL_parser cp, FILE *f);
6261 The functions <function>cql_parser_stream</function> and
6262 <function>cql_parser_stdio</function> parse a CQL query
6263 - just like <function>cql_parser_string</function>.
6264 The only difference is that the CQL query can be
6265 fed to the parser in different ways.
6266 The <function>cql_parser_stream</function> uses a generic
6267 byte stream as input. The <function>cql_parser_stdio</function>
6268 uses a <literal>FILE</literal> handle which is opened for reading.
6271 <sect3 id="cql.tree">
6272 <title>CQL tree</title>
6274 If the query string is valid, the CQL parser
6275 generates a tree representing the structure of the
6280 struct cql_node *cql_parser_result(CQL_parser cp);
6282 <function>cql_parser_result</function> returns
6283 a pointer to the root node of the resulting tree.
6286 Each node in a CQL tree is represented by a
6287 <literal>struct cql_node</literal>.
6288 It is defined as follows:
6290 #define CQL_NODE_ST 1
6291 #define CQL_NODE_BOOL 2
6292 #define CQL_NODE_SORT 3
6302 struct cql_node *modifiers;
6306 struct cql_node *left;
6307 struct cql_node *right;
6308 struct cql_node *modifiers;
6312 struct cql_node *next;
6313 struct cql_node *modifiers;
6314 struct cql_node *search;
6319 There are three node types: search term (ST), boolean (BOOL)
6321 A modifier is treated as a search term too.
6324 The search term node has five members:
6328 <literal>index</literal>: index for search term.
6329 If an index is unspecified for a search term,
6330 <literal>index</literal> will be NULL.
6335 <literal>index_uri</literal>: index URi for search term
6336 or NULL if none could be resolved for the index.
6341 <literal>term</literal>: the search term itself.
6346 <literal>relation</literal>: relation for search term.
6351 <literal>relation_uri</literal>: relation URI for search term.
6356 <literal>modifiers</literal>: relation modifiers for search
6357 term. The <literal>modifiers</literal> list itself of cql_nodes
6358 each of type <literal>ST</literal>.
6364 The boolean node represents <literal>and</literal>,
6365 <literal>or</literal>, <literal>not</literal> +
6370 <literal>left</literal> and <literal>right</literal>: left
6371 - and right operand respectively.
6376 <literal>modifiers</literal>: proximity arguments.
6382 The sort node represents both the SORTBY clause.
6385 <sect3 id="cql.to.pqf">
6386 <title>CQL to PQF conversion</title>
6388 Conversion to PQF (and Z39.50 RPN) is tricky by the fact
6389 that the resulting RPN depends on the Z39.50 target
6390 capabilities (combinations of supported attributes).
6391 In addition, the CQL and SRU operates on index prefixes
6392 (URI or strings), whereas the RPN uses Object Identifiers
6396 The CQL library of &yaz; defines a <literal>cql_transform_t</literal>
6397 type. It represents a particular mapping between CQL and RPN.
6398 This handle is created and destroyed by the functions:
6400 cql_transform_t cql_transform_open_FILE (FILE *f);
6401 cql_transform_t cql_transform_open_fname(const char *fname);
6402 void cql_transform_close(cql_transform_t ct);
6404 The first two functions create a tranformation handle from
6405 either an already open FILE or from a filename respectively.
6408 The handle is destroyed by <function>cql_transform_close</function>
6409 in which case no further reference of the handle is allowed.
6412 When a <literal>cql_transform_t</literal> handle has been created
6413 you can convert to RPN.
6415 int cql_transform_buf(cql_transform_t ct,
6416 struct cql_node *cn, char *out, int max);
6418 This function converts the CQL tree <literal>cn</literal>
6419 using handle <literal>ct</literal>.
6420 For the resulting PQF, you supply a buffer <literal>out</literal>
6421 which must be able to hold at at least <literal>max</literal>
6425 If conversion failed, <function>cql_transform_buf</function>
6426 returns a non-zero SRU error code; otherwise zero is returned
6427 (conversion successful). The meanings of the numeric error
6428 codes are listed in the SRU specification somewhere (no
6429 direct link anymore).
6432 If conversion fails, more information can be obtained by calling
6434 int cql_transform_error(cql_transform_t ct, char **addinfop);
6436 This function returns the most recently returned numeric
6437 error-code and sets the string-pointer at
6438 <literal>*addinfop</literal> to point to a string containing
6439 additional information about the error that occurred: for
6440 example, if the error code is 15 ("Illegal or unsupported context
6441 set"), the additional information is the name of the requested
6442 context set that was not recognised.
6445 The SRU error-codes may be translated into brief human-readable
6446 error messages using
6448 const char *cql_strerror(int code);
6452 If you wish to be able to produce a PQF result in a different
6453 way, there are two alternatives.
6455 void cql_transform_pr(cql_transform_t ct,
6456 struct cql_node *cn,
6457 void (*pr)(const char *buf, void *client_data),
6460 int cql_transform_FILE(cql_transform_t ct,
6461 struct cql_node *cn, FILE *f);
6463 The former function produces output to a user-defined
6464 output stream. The latter writes the result to an already
6465 open <literal>FILE</literal>.
6468 <sect3 id="cql.to.rpn">
6469 <title>Specification of CQL to RPN mappings</title>
6471 The file supplied to functions
6472 <function>cql_transform_open_FILE</function>,
6473 <function>cql_transform_open_fname</function> follows
6474 a structure found in many Unix utilities.
6475 It consists of mapping specifications - one per line.
6476 Lines starting with <literal>#</literal> are ignored (comments).
6479 Each line is of the form
6481 <replaceable>CQL pattern</replaceable><literal> = </literal> <replaceable> RPN equivalent</replaceable>
6485 An RPN pattern is a simple attribute list. Each attribute pair
6488 [<replaceable>set</replaceable>] <replaceable>type</replaceable><literal>=</literal><replaceable>value</replaceable>
6490 The attribute <replaceable>set</replaceable> is optional.
6491 The <replaceable>type</replaceable> is the attribute type,
6492 <replaceable>value</replaceable> the attribute value.
6495 The character <literal>*</literal> (asterisk) has special meaning
6496 when used in the RPN pattern.
6497 Each occurrence of <literal>*</literal> is substituted with the
6498 CQL matching name (index, relation, qualifier etc).
6499 This facility can be used to copy a CQL name verbatim to the RPN result.
6502 The following CQL patterns are recognized:
6506 <literal>index.</literal><replaceable>set</replaceable><literal>.</literal><replaceable>name</replaceable>
6510 This pattern is invoked when a CQL index, such as
6511 dc.title is converted. <replaceable>set</replaceable>
6512 and <replaceable>name</replaceable> are the context set and index
6514 Typically, the RPN specifies an equivalent use attribute.
6517 For terms not bound by an index, the pattern
6518 <literal>index.cql.serverChoice</literal> is used.
6519 Here, the prefix <literal>cql</literal> is defined as
6520 <literal>http://www.loc.gov/zing/cql/cql-indexes/v1.0/</literal>.
6521 If this pattern is not defined, the mapping will fail.
6525 <literal>index.</literal><replaceable>set</replaceable><literal>.*</literal>
6526 is used when no other index pattern is matched.
6532 <literal>qualifier.</literal><replaceable>set</replaceable><literal>.</literal><replaceable>name</replaceable>
6537 For backwards compatibility, this is recognised as a synonym of
6538 <literal>index.</literal><replaceable>set</replaceable><literal>.</literal><replaceable>name</replaceable>
6544 <literal>relation.</literal><replaceable>relation</replaceable>
6548 This pattern specifies how a CQL relation is mapped to RPN.
6549 The <replaceable>pattern</replaceable> is name of relation
6550 operator. Since <literal>=</literal> is used as
6551 separator between CQL pattern and RPN, CQL relations
6552 including <literal>=</literal> cannot be
6553 used directly. To avoid a conflict, the names
6554 <literal>ge</literal>,
6555 <literal>eq</literal>,
6556 <literal>le</literal>,
6557 must be used for CQL operators, greater-than-or-equal,
6558 equal, less-than-or-equal respectively.
6559 The RPN pattern is supposed to include a relation attribute.
6562 For terms not bound by a relation, the pattern
6563 <literal>relation.scr</literal> is used. If the pattern
6564 is not defined, the mapping will fail.
6567 The special pattern, <literal>relation.*</literal> is used
6568 when no other relation pattern is matched.
6574 <literal>relationModifier.</literal><replaceable>mod</replaceable>
6578 This pattern specifies how a CQL relation modifier is mapped to RPN.
6579 The RPN pattern is usually a relation attribute.
6585 <literal>structure.</literal><replaceable>type</replaceable>
6589 This pattern specifies how a CQL structure is mapped to RPN.
6590 Note that this CQL pattern is somewhat similar to
6591 CQL pattern <literal>relation</literal>.
6592 The <replaceable>type</replaceable> is a CQL relation.
6595 The pattern, <literal>structure.*</literal> is used
6596 when no other structure pattern is matched.
6597 Usually, the RPN equivalent specifies a structure attribute.
6603 <literal>position.</literal><replaceable>type</replaceable>
6607 This pattern specifies how the anchor (position) of
6608 CQL is mapped to RPN.
6609 The <replaceable>type</replaceable> is one
6610 of <literal>first</literal>, <literal>any</literal>,
6611 <literal>last</literal>, <literal>firstAndLast</literal>.
6614 The pattern, <literal>position.*</literal> is used
6615 when no other position pattern is matched.
6621 <literal>set.</literal><replaceable>prefix</replaceable>
6625 This specification defines a CQL context set for a given prefix.
6626 The value on the right hand side is the URI for the set -
6627 <emphasis>not</emphasis> RPN. All prefixes used in
6628 index patterns must be defined this way.
6634 <literal>set</literal>
6638 This specification defines a default CQL context set for index names.
6639 The value on the right hand side is the URI for the set.
6645 <example id="example.cql.to.rpn.mapping">
6646 <title>CQL to RPN mapping file</title>
6648 This simple file defines two context sets, three indexes and three
6649 relations, a position pattern and a default structure.
6651 <programlisting><![CDATA[
6652 set.cql = http://www.loc.gov/zing/cql/context-sets/cql/v1.1/
6653 set.dc = http://www.loc.gov/zing/cql/dc-indexes/v1.0/
6655 index.cql.serverChoice = 1=1016
6656 index.dc.title = 1=4
6657 index.dc.subject = 1=21
6663 position.any = 3=3 6=1
6669 With the mappings above, the CQL query
6673 is converted to the PQF:
6675 @attr 1=1016 @attr 2=3 @attr 4=1 @attr 3=3 @attr 6=1 "computer"
6677 by rules <literal>index.cql.serverChoice</literal>,
6678 <literal>relation.scr</literal>, <literal>structure.*</literal>,
6679 <literal>position.any</literal>.
6686 is rejected, since <literal>position.right</literal> is
6692 >my = "http://www.loc.gov/zing/cql/dc-indexes/v1.0/" my.title = x
6696 @attr 1=4 @attr 2=3 @attr 4=1 @attr 3=3 @attr 6=1 "x"
6700 <example id="example.cql.to.rpn.string">
6701 <title>CQL to RPN string attributes</title>
6703 In this example we allow any index to be passed to RPN as
6706 <programlisting><![CDATA[
6707 # Identifiers for prefixes used in this file. (index.*)
6708 set.cql = info:srw/cql-context-set/1/cql-v1.1
6709 set.rpn = http://bogus/rpn
6710 set = http://bogus/rpn
6712 # The default index when none is specified by the query
6713 index.cql.serverChoice = 1=any
6722 The <literal>http://bogus/rpn</literal> context set is also the default
6723 so we can make queries such as
6727 which is converted to
6729 @attr 2=3 @attr 4=1 @attr 3=3 @attr 1=title "a"
6733 <example id="example.cql.to.rpn.bathprofile">
6734 <title>CQL to RPN using Bath Profile</title>
6736 The file <filename>etc/pqf.properties</filename> has mappings from
6737 the Bath Profile and Dublin Core to RPN.
6738 If YAZ is installed as a package it's usually located
6739 in <filename>/usr/share/yaz/etc</filename> and part of the
6740 development package, such as <literal>libyaz-dev</literal>.
6744 <sect3 id="cql.xcql">
6745 <title>CQL to XCQL conversion</title>
6747 Conversion from CQL to XCQL is trivial and does not
6748 require a mapping to be defined.
6749 There are three functions to choose from depending on the
6750 way you wish to store the resulting output (XML buffer
6753 int cql_to_xml_buf(struct cql_node *cn, char *out, int max);
6754 void cql_to_xml(struct cql_node *cn,
6755 void (*pr)(const char *buf, void *client_data),
6757 void cql_to_xml_stdio(struct cql_node *cn, FILE *f);
6759 Function <function>cql_to_xml_buf</function> converts
6760 to XCQL and stores the result in a user-supplied buffer of a given
6764 <function>cql_to_xml</function> writes the result in
6765 a user-defined output stream.
6766 <function>cql_to_xml_stdio</function> writes to a
6770 <sect3 id="rpn.to.cql">
6771 <title>PQF to CQL conversion</title>
6773 Conversion from PQF to CQL is offered by the two functions shown
6774 below. The former uses a generic stream for result. The latter
6775 puts result in a WRBUF (string container).
6777 #include <yaz/rpn2cql.h>
6779 int cql_transform_rpn2cql_stream(cql_transform_t ct,
6780 void (*pr)(const char *buf, void *client_data),
6784 int cql_transform_rpn2cql_wrbuf(cql_transform_t ct,
6788 The configuration is the same as used in CQL to PQF conversions.
6793 <sect1 id="tools.oid">
6794 <title>Object Identifiers</title>
6796 The basic YAZ representation of an OID is an array of integers,
6797 terminated with the value -1. This integer is of type
6798 <literal>Odr_oid</literal>.
6801 Fundamental OID operations and the type <literal>Odr_oid</literal>
6802 are defined in <filename>yaz/oid_util.h</filename>.
6805 An OID can either be declared as a automatic variable or it can
6806 be allocated using the memory utilities or ODR/NMEM. It's
6807 guaranteed that an OID can fit in <literal>OID_SIZE</literal> integers.
6809 <example id="tools.oid.bib1.1"><title>Create OID on stack</title>
6811 We can create an OID for the Bib-1 attribute set with:
6813 Odr_oid bib1[OID_SIZE];
6825 And OID may also be filled from a string-based representation using
6826 dots (.). This is achieved by the function
6828 int oid_dotstring_to_oid(const char *name, Odr_oid *oid);
6830 This functions returns 0 if name could be converted; -1 otherwise.
6832 <example id="tools.oid.bib1.2"><title>Using oid_oiddotstring_to_oid</title>
6834 We can fill the Bib-1 attribute set OID more easily with:
6836 Odr_oid bib1[OID_SIZE];
6837 oid_oiddotstring_to_oid("1.2.840.10003.3.1", bib1);
6842 We can also allocate an OID dynamically on an ODR stream with:
6844 Odr_oid *odr_getoidbystr(ODR o, const char *str);
6846 This creates an OID from a string-based representation using dots.
6847 This function take an &odr; stream as parameter. This stream is used to
6848 allocate memory for the data elements, which is released on a
6849 subsequent call to <function>odr_reset()</function> on that stream.
6851 <example id="tools.oid.bib1.3">
6852 <title>Using odr_getoidbystr</title>
6854 We can create an OID for the Bib-1 attribute set with:
6856 Odr_oid *bib1 = odr_getoidbystr(odr, "1.2.840.10003.3.1");
6863 char *oid_oid_to_dotstring(const Odr_oid *oid, char *oidbuf)
6865 does the reverse of <function>oid_oiddotstring_to_oid</function>. It
6866 converts an OID to the string-based representation using dots.
6867 The supplied char buffer <literal>oidbuf</literal> holds the resulting
6868 string and must be at least <literal>OID_STR_MAX</literal> in size.
6871 OIDs can be copied with <function>oid_oidcpy</function> which takes
6872 two OID lists as arguments. Alternatively, an OID copy can be allocated
6873 on an ODR stream with:
6875 Odr_oid *odr_oiddup(ODR odr, const Odr_oid *o);
6879 OIDs can be compared with <function>oid_oidcmp</function> which returns
6880 zero if the two OIDs provided are identical; non-zero otherwise.
6882 <sect2 id="tools.oid.database">
6883 <title>OID database</title>
6885 From YAZ version 3 and later, the oident system has been replaced
6886 by an OID database. OID database is a misnomer .. the old odient
6887 system was also a database.
6890 The OID database is really just a map between named Object Identifiers
6891 (string) and their OID raw equivalents. Most operations either
6892 convert from string to OID or other way around.
6895 Unfortunately, whenever we supply a string we must also specify the
6896 <emphasis>OID class</emphasis>. The class is necessary because some
6897 strings correspond to multiple OIDs. An example of such a string is
6898 <literal>Bib-1</literal> which may either be an attribute-set
6899 or a diagnostic-set.
6902 Applications using the YAZ database should include
6903 <filename>yaz/oid_db.h</filename>.
6906 A YAZ database handle is of type <literal>yaz_oid_db_t</literal>.
6907 Actually that's a pointer. You need not deal with that.
6908 YAZ has a built-in database which can be considered "constant" for
6910 We can get hold of that by using function <function>yaz_oid_std</function>.
6913 All functions with prefix <function>yaz_string_to_oid</function>
6914 converts from class + string to OID. We have variants of this
6915 operation due to different memory allocation strategies.
6918 All functions with prefix
6919 <function>yaz_oid_to_string</function> converts from OID to string
6922 <example id="tools.oid.bib1.4">
6923 <title>Create OID with YAZ DB</title>
6925 We can create an OID for the Bib-1 attribute set on the ODR stream
6929 yaz_string_to_oid_odr(yaz_oid_std(), CLASS_ATTSET, "Bib-1", odr);
6931 This is more complex than using <function>odr_getoidbystr</function>.
6932 You would only use <function>yaz_string_to_oid_odr</function> when the
6933 string (here Bib-1) is supplied by a user or configuration.
6937 <sect2 id="tools.oid.std">
6938 <title>Standard OIDs</title>
6940 All the object identifers in the standard OID database as returned
6941 by <function>yaz_oid_std</function> can be referenced directly in a
6942 program as a constant OID.
6943 Each constant OID is prefixed with <literal>yaz_oid_</literal> -
6944 followed by OID class (lowercase) - then by OID name (normalized and
6948 See <xref linkend="list-oids"/> for list of all object identifiers
6950 These are declared in <filename>yaz/oid_std.h</filename> but are
6951 included by <filename>yaz/oid_db.h</filename> as well.
6953 <example id="tools.oid.bib1.5">
6954 <title>Use a built-in OID</title>
6956 We can allocate our own OID filled with the constant OID for
6959 Odr_oid *bib1 = odr_oiddup(o, yaz_oid_attset_bib1);
6965 <sect1 id="tools.nmem">
6966 <title>Nibble Memory</title>
6968 Sometimes when you need to allocate and construct a large,
6969 interconnected complex of structures, it can be a bit of a pain to
6970 release the associated memory again. For the structures describing the
6971 Z39.50 PDUs and related structures, it is convenient to use the
6972 memory-management system of the &odr; subsystem (see
6973 <xref linkend="odr.use"/>). However, in some circumstances
6974 where you might otherwise benefit from using a simple nibble-memory
6975 management system, it may be impractical to use
6976 <function>odr_malloc()</function> and <function>odr_reset()</function>.
6977 For this purpose, the memory manager which also supports the &odr;
6978 streams is made available in the NMEM module. The external interface
6979 to this module is given in the <filename>nmem.h</filename> file.
6982 The following prototypes are given:
6985 NMEM nmem_create(void);
6986 void nmem_destroy(NMEM n);
6987 void *nmem_malloc(NMEM n, size_t size);
6988 void nmem_reset(NMEM n);
6989 size_t nmem_total(NMEM n);
6990 void nmem_init(void);
6991 void nmem_exit(void);
6994 The <function>nmem_create()</function> function returns a pointer to a
6995 memory control handle, which can be released again by
6996 <function>nmem_destroy()</function> when no longer needed.
6997 The function <function>nmem_malloc()</function> allocates a block of
6998 memory of the requested size. A call to <function>nmem_reset()</function>
6999 or <function>nmem_destroy()</function> will release all memory allocated
7000 on the handle since it was created (or since the last call to
7001 <function>nmem_reset()</function>. The function
7002 <function>nmem_total()</function> returns the number of bytes currently
7003 allocated on the handle.
7006 The nibble-memory pool is shared amongst threads. POSIX
7007 mutex'es and WIN32 Critical sections are introduced to keep the
7008 module thread safe. Function <function>nmem_init()</function>
7009 initializes the nibble-memory library and it is called automatically
7010 the first time the <literal>YAZ.DLL</literal> is loaded. &yaz; uses
7011 function <function>DllMain</function> to achieve this. You should
7012 <emphasis>not</emphasis> call <function>nmem_init</function> or
7013 <function>nmem_exit</function> unless you're absolute sure what
7014 you're doing. Note that in previous &yaz; versions you'd have to call
7015 <function>nmem_init</function> yourself.
7018 <sect1 id="tools.log">
7021 &yaz; has evolved a fairly complex log system which should be useful both
7022 for debugging &yaz; itself, debugging applications that use &yaz;, and for
7023 production use of those applications.
7026 The log functions are declared in header <filename>yaz/log.h</filename>
7027 and implemented in <filename>src/log.c</filename>.
7028 Due to name clash with syslog and some math utilities the logging
7029 interface has been modified as of YAZ 2.0.29. The obsolete interface
7030 is still available in header file <filename>yaz/log.h</filename>.
7031 The key points of the interface are:
7034 void yaz_log(int level, const char *fmt, ...)
7035 void yaz_log_init(int level, const char *prefix, const char *name);
7036 void yaz_log_init_file(const char *fname);
7037 void yaz_log_init_level(int level);
7038 void yaz_log_init_prefix(const char *prefix);
7039 void yaz_log_time_format(const char *fmt);
7040 void yaz_log_init_max_size(int mx);
7042 int yaz_log_mask_str(const char *str);
7043 int yaz_log_module_level(const char *name);
7046 The reason for the whole log module is the <function>yaz_log</function>
7047 function. It takes a bitmask indicating the log levels, a
7048 <literal>printf</literal>-like format string, and a variable number of
7052 The <literal>log level</literal> is a bit mask, that says on which level(s)
7053 the log entry should be made, and optionally set some behaviour of the
7054 logging. In the most simple cases, it can be one of <literal>YLOG_FATAL,
7055 YLOG_DEBUG, YLOG_WARN, YLOG_LOG</literal>. Those can be combined with bits
7056 that modify the way the log entry is written:<literal>YLOG_ERRNO,
7057 YLOG_NOTIME, YLOG_FLUSH</literal>.
7058 Most of the rest of the bits are deprecated, and should not be used. Use
7059 the dynamic log levels instead.
7062 Applications that use &yaz;, should not use the LOG_LOG for ordinary
7063 messages, but should make use of the dynamic loglevel system. This consists
7064 of two parts, defining the loglevel and checking it.
7067 To define the log levels, the (main) program should pass a string to
7068 <function>yaz_log_mask_str</function> to define which log levels are to be
7069 logged. This string should be a comma-separated list of log level names,
7070 and can contain both hard-coded names and dynamic ones. The log level
7071 calculation starts with <literal>YLOG_DEFAULT_LEVEL</literal> and adds a bit
7072 for each word it meets, unless the word starts with a '-', in which case it
7073 clears the bit. If the string <literal>'none'</literal> is found,
7074 all bits are cleared. Typically this string comes from the command-line,
7075 often identified by <literal>-v</literal>. The
7076 <function>yaz_log_mask_str</function> returns a log level that should be
7077 passed to <function>yaz_log_init_level</function> for it to take effect.
7080 Each module should check what log bits should be used, by calling
7081 <function>yaz_log_module_level</function> with a suitable name for the
7082 module. The name is cleared of a preceding path and an extension, if any,
7083 so it is quite possible to use <literal>__FILE__</literal> for it. If the
7084 name has been passed to <function>yaz_log_mask_str</function>, the routine
7085 returns a non-zero bitmask, which should then be used in consequent calls
7086 to yaz_log. (It can also be tested, so as to avoid unnecessary calls to
7087 yaz_log, in time-critical places, or when the log entry would take time
7091 Yaz uses the following dynamic log levels:
7092 <literal>server, session, request, requestdetail</literal> for the server
7094 <literal>zoom</literal> for the zoom client api.
7095 <literal>ztest</literal> for the simple test server.
7096 <literal>malloc, nmem, odr, eventl</literal> for internal
7097 debugging of yaz itself.
7098 Of course, any program using yaz is welcome to define as many new
7102 By default the log is written to stderr, but this can be changed by a call
7103 to <function>yaz_log_init_file</function> or
7104 <function>yaz_log_init</function>. If the log is directed to a file, the
7105 file size is checked at every write, and if it exceeds the limit given in
7106 <function>yaz_log_init_max_size</function>, the log is rotated. The
7107 rotation keeps one old version (with a <literal>.1</literal> appended to
7108 the name). The size defaults to 1GB. Setting it to zero will disable the
7112 A typical yaz-log looks like this
7113 13:23:14-23/11 yaz-ztest(1) [session] Starting session from tcp:127.0.0.1 (pid=30968)
7114 13:23:14-23/11 yaz-ztest(1) [request] Init from 'YAZ' (81) (ver 2.0.28) OK
7115 13:23:17-23/11 yaz-ztest(1) [request] Search Z: @attrset Bib-1 foo OK:7 hits
7116 13:23:22-23/11 yaz-ztest(1) [request] Present: [1] 2+2 OK 2 records returned
7117 13:24:13-23/11 yaz-ztest(1) [request] Close OK
7120 The log entries start with a time stamp. This can be omitted by setting the
7121 <literal>YLOG_NOTIME</literal> bit in the loglevel. This way automatic tests
7122 can be hoped to produce identical log files, that are easy to diff. The
7123 format of the time stamp can be set with
7124 <function>yaz_log_time_format</function>, which takes a format string just
7125 like <function>strftime</function>.
7128 Next in a log line comes the prefix, often the name of the program. For
7129 yaz-based servers, it can also contain the session number. Then
7130 comes one or more logbits in square brackets, depending on the logging
7131 level set by <function>yaz_log_init_level</function> and the loglevel
7132 passed to <function>yaz_log_init_level</function>. Finally comes the format
7133 string and additional values passed to <function>yaz_log</function>
7136 The log level <literal>YLOG_LOGLVL</literal>, enabled by the string
7137 <literal>loglevel</literal>, will log all the log-level affecting
7138 operations. This can come in handy if you need to know what other log
7139 levels would be useful. Grep the logfile for <literal>[loglevel]</literal>.
7142 The log system is almost independent of the rest of &yaz;, the only
7143 important dependence is of <filename>nmem</filename>, and that only for
7144 using the semaphore definition there.
7147 The dynamic log levels and log rotation were introduced in &yaz; 2.0.28. At
7148 the same time, the log bit names were changed from
7149 <literal>LOG_something</literal> to <literal>YLOG_something</literal>,
7150 to avoid collision with <filename>syslog.h</filename>.
7156 YAZ provides a fast utility for working with MARC records.
7157 Early versions of the MARC utility only allowed decoding of ISO2709.
7158 Today the utility may both encode - and decode to a variety of formats.
7161 #include <yaz/marcdisp.h>
7163 /* create handler */
7164 yaz_marc_t yaz_marc_create(void);
7166 void yaz_marc_destroy(yaz_marc_t mt);
7168 /* set XML mode YAZ_MARC_LINE, YAZ_MARC_SIMPLEXML, ... */
7169 void yaz_marc_xml(yaz_marc_t mt, int xmlmode);
7170 #define YAZ_MARC_LINE 0
7171 #define YAZ_MARC_SIMPLEXML 1
7172 #define YAZ_MARC_OAIMARC 2
7173 #define YAZ_MARC_MARCXML 3
7174 #define YAZ_MARC_ISO2709 4
7175 #define YAZ_MARC_XCHANGE 5
7176 #define YAZ_MARC_CHECK 6
7177 #define YAZ_MARC_TURBOMARC 7
7178 #define YAZ_MARC_JSON 8
7180 /* supply iconv handle for character set conversion .. */
7181 void yaz_marc_iconv(yaz_marc_t mt, yaz_iconv_t cd);
7183 /* set debug level, 0=none, 1=more, 2=even more, .. */
7184 void yaz_marc_debug(yaz_marc_t mt, int level);
7186 /* decode MARC in buf of size bsize. Returns >0 on success; <=0 on failure.
7187 On success, result in *result with size *rsize. */
7188 int yaz_marc_decode_buf(yaz_marc_t mt, const char *buf, int bsize,
7189 const char **result, size_t *rsize);
7191 /* decode MARC in buf of size bsize. Returns >0 on success; <=0 on failure.
7192 On success, result in WRBUF */
7193 int yaz_marc_decode_wrbuf(yaz_marc_t mt, const char *buf,
7194 int bsize, WRBUF wrbuf);
7199 The synopsis is just a basic subset of all functionality. Refer
7200 to the actual header file <filename>marcdisp.h</filename> for
7205 A MARC conversion handle must be created by using
7206 <function>yaz_marc_create</function> and destroyed
7207 by calling <function>yaz_marc_destroy</function>.
7210 All other functions operate on a <literal>yaz_marc_t</literal> handle.
7211 The output is specified by a call to <function>yaz_marc_xml</function>.
7212 The <literal>xmlmode</literal> must be one of
7215 <term>YAZ_MARC_LINE</term>
7218 A simple line-by-line format suitable for display but not
7219 recommended for further (machine) processing.
7224 <term>YAZ_MARC_MARCXML</term>
7227 <ulink url="&url.marcxml;">MARCXML</ulink>.
7232 <term>YAZ_MARC_ISO2709</term>
7235 ISO2709 (sometimes just referred to as "MARC").
7240 <term>YAZ_MARC_XCHANGE</term>
7243 <ulink url="&url.marcxchange;">MarcXchange</ulink>.
7248 <term>YAZ_MARC_CHECK</term>
7251 Pseudo format for validation only. Does not generate
7252 any real output except diagnostics.
7257 <term>YAZ_MARC_TURBOMARC</term>
7260 XML format with same semantics as MARCXML but more compact
7261 and geared towards fast processing with XSLT. Refer to
7262 <xref linkend="tools.turbomarc"/> for more information.
7267 <term>YAZ_MARC_JSON</term>
7270 <ulink url="&url.marc_in_json;">MARC-in-JSON</ulink> format.
7277 The actual conversion functions are
7278 <function>yaz_marc_decode_buf</function> and
7279 <function>yaz_marc_decode_wrbuf</function> which decodes and encodes
7280 a MARC record. The former function operates on simple buffers, and
7281 stores the resulting record in a WRBUF handle (WRBUF is a simple string
7284 <example id="example.marc.display">
7285 <title>Display of MARC record</title>
7287 The following program snippet illustrates how the MARC API may
7288 be used to convert a MARC record to the line-by-line format:
7289 <programlisting><![CDATA[
7290 void print_marc(const char *marc_buf, int marc_buf_size)
7292 char *result; /* for result buf */
7293 size_t result_len; /* for size of result */
7294 yaz_marc_t mt = yaz_marc_create();
7295 yaz_marc_xml(mt, YAZ_MARC_LINE);
7296 yaz_marc_decode_buf(mt, marc_buf, marc_buf_size,
7297 &result, &result_len);
7298 fwrite(result, result_len, 1, stdout);
7299 yaz_marc_destroy(mt); /* note that result is now freed... */
7305 <sect2 id="tools.turbomarc">
7306 <title>TurboMARC</title>
7308 TurboMARC is yet another XML encoding of a MARC record. The format
7309 was designed for fast processing with XSLT.
7313 Pazpar2 uses XSLT to convert an XML encoded MARC record to an internal
7314 representation. This conversion mostly checks the tag of a MARC field
7315 to determine the basic rules in the conversion. This check is
7316 costly when that tag is encoded as an attribute in MARCXML.
7317 By having the tag value as the element instead, makes processing
7318 many times faster (at least for Libxslt).
7321 TurboMARC is encoded as follows:
7325 Record elements is part of namespace
7326 "<literal>http://www.indexdata.com/turbomarc</literal>".
7331 A record is enclosed in element <literal>r</literal>.
7336 A collection of records is enclosed in element
7337 <literal>collection</literal>.
7342 The leader is encoded as element <literal>l</literal> with the
7343 leader content as its (text) value.
7348 A control field is encoded as element <literal>c</literal> concatenated
7349 with the tag value of the control field if the tag value
7350 matches the regular expression <literal>[a-zA-Z0-9]*</literal>.
7351 If the tag value does not match the regular expression
7352 <literal>[a-zA-Z0-9]*</literal> the control field is encoded
7353 as element <literal>c</literal> and attribute <literal>code</literal>
7354 will hold the tag value.
7355 This rule ensures that in the rare cases where a tag value might
7356 result in a non-well-formed XML, then YAZ will encode it as a coded attribute
7360 The control field content is the text value of this element.
7361 Indicators are encoded as attribute names
7362 <literal>i1</literal>, <literal>i2</literal>, etc. and
7363 corresponding values for each indicator.
7368 A data field is encoded as element <literal>d</literal> concatenated
7369 with the tag value of the data field or using the attribute
7370 <literal>code</literal> as described in the rules for control fields.
7371 The children of the data field element are subfield elements.
7372 Each subfield element is encoded as <literal>s</literal>
7373 concatenated with the sub field code.
7374 The text of the subfield element is the contents of the subfield.
7375 Indicators are encoded as attributes for the data field element, similar
7376 to the encoding for control fields.
7383 <sect1 id="tools.retrieval">
7384 <title>Retrieval Facility</title>
7386 YAZ version 2.1.20 or later includes a Retrieval facility tool
7387 which allows a SRU/Z39.50 to describe itself and perform record
7388 conversions. The idea is the following:
7392 An SRU/Z39.50 client sends a retrieval request which includes
7393 a combination of the following parameters: syntax (format),
7394 schema (or element set name).
7399 The retrieval facility is invoked with parameters in a
7400 server/proxy. The retrieval facility matches the parameters a set of
7401 "supported" retrieval types.
7402 If there is no match, the retrieval signals an error
7403 (syntax and / or schema not supported).
7408 For a successful match, the backend is invoked with the same
7409 or altered retrieval parameters (syntax, schema). If
7410 a record is received from the backend, it is converted to the
7411 frontend name / syntax.
7416 The resulting record is sent back the client and tagged with
7417 the frontend syntax / schema.
7423 The Retrieval facility is driven by an XML configuration. The
7424 configuration is neither Z39.50 ZeeRex or SRU ZeeRex. But it
7425 should be easy to generate both of them from the XML configuration.
7426 (Unfortunately the two versions
7427 of ZeeRex differ substantially in this regard.)
7429 <sect2 id="tools.retrieval.format">
7430 <title>Retrieval XML format</title>
7432 All elements should be covered by namespace
7433 <literal>http://indexdata.com/yaz</literal> .
7434 The root element node must be <literal>retrievalinfo</literal>.
7437 The <literal>retrievalinfo</literal> must include one or
7438 more <literal>retrieval</literal> elements. Each
7439 <literal>retrieval</literal> defines specific combination of
7440 syntax, name and identifier supported by this retrieval service.
7443 The <literal>retrieval</literal> element may include any of the
7444 following attributes:
7446 <varlistentry><term><literal>syntax</literal> (REQUIRED)</term>
7449 Defines the record syntax. Possible values is any
7450 of the names defined in YAZ' OID database or a raw
7455 <varlistentry><term><literal>name</literal> (OPTIONAL)</term>
7458 Defines the name of the retrieval format. This can be
7459 any string. For SRU, the value is equivalent to schema (short-hand);
7460 for Z39.50 it's equivalent to simple element set name.
7461 For YAZ 3.0.24 and later this name may be specified as a glob
7462 expression with operators
7463 <literal>*</literal> and <literal>?</literal>.
7467 <varlistentry><term><literal>identifier</literal> (OPTIONAL)</term>
7470 Defines the URI schema name of the retrieval format. This can be
7471 any string. For SRU, the value is equivalent to URI schema.
7472 For Z39.50, there is no equivalent.
7479 The <literal>retrieval</literal> may include one
7480 <literal>backend</literal> element. If a <literal>backend</literal>
7481 element is given, it specifies how the records are retrieved by
7482 some backend and how the records are converted from the backend to
7486 The attributes, <literal>name</literal> and <literal>syntax</literal>
7487 may be specified for the <literal>backend</literal> element. The
7488 semantics of these attributes is equivalent to those for the
7489 <literal>retrieval</literal>. However, these values are passed to
7493 The <literal>backend</literal> element may include one or more
7494 conversion instructions (as children elements). The supported
7497 <varlistentry><term><literal>marc</literal></term>
7500 The <literal>marc</literal> element specifies a conversion
7501 to - and from ISO2709 encoded MARC and
7502 <ulink url="&url.marcxml;">&acro.marcxml;</ulink>/MarcXchange.
7503 The following attributes may be specified:
7506 <term><literal>inputformat</literal> (REQUIRED)</term>
7509 Format of input. Supported values are
7510 <literal>marc</literal> (for ISO2709), <literal>xml</literal>
7511 (MARCXML/MarcXchange) and <literal>json</literal>
7512 (<ulink url="&url.marc_in_json;">MARC-in-JSON</ulink>).
7517 <term><literal>outputformat</literal> (REQUIRED)</term>
7520 Format of output. Supported values are
7521 <literal>line</literal> (MARC line format);
7522 <literal>marcxml</literal> (for MARCXML),
7523 <literal>marc</literal> (ISO2709),
7524 <literal>marcxhcange</literal> (for MarcXchange),
7525 or <literal>json</literal>
7526 (<ulink url="&url.marc_in_json;">MARC-in-JSON </ulink>).
7531 <term><literal>inputcharset</literal> (OPTIONAL)</term>
7534 Encoding of input. For XML input formats, this need not
7535 be given, but for ISO2709 based inputformats, this should
7536 be set to the encoding used. For MARC21 records, a common
7537 inputcharset value would be <literal>marc-8</literal>.
7542 <term><literal>outputcharset</literal> (OPTIONAL)</term>
7545 Encoding of output. If outputformat is XML based, it is
7546 strongly recommened to use <literal>utf-8</literal>.
7555 <term><literal>select</literal></term>
7558 The <literal>select</literal> selects one or more text nodes
7559 and decodes them as XML.
7560 The following attributes may be specified:
7562 <varlistentry><term><literal>path</literal> (REQUIRED)</term>
7565 X-Path expression for selecting text nodes.
7572 This conversion is available in YAZ 5.8.0 and later.
7577 <term><literal>solrmarc</literal></term>
7580 The <literal>solrmarc</literal> decodes solrmarc records.
7581 It assumes that the input is pure solrmarc text (no escaping)
7582 and will convert all sequences of the form #XX; to a single
7583 character of the hexadecimal value as given by XX. The output,
7584 presumably, is a valid ISO2709 buffer.
7587 This conversion is available in YAZ 5.0.21 and later.
7592 <term><literal>xslt</literal></term>
7595 The <literal>xslt</literal> element specifies a conversion
7596 via &acro.xslt;. The following attributes may be specified:
7598 <varlistentry><term><literal>stylesheet</literal> (REQUIRED)</term>
7612 <sect2 id="tools.retrieval.examples">
7613 <title>Retrieval Facility Examples</title>
7614 <example id="tools.retrieval.marc21">
7615 <title>MARC21 backend</title>
7617 A typical way to use the retrieval facility is to enable XML
7618 for servers that only supports ISO2709 encoded MARC21 records.
7620 <programlisting><![CDATA[
7622 <retrieval syntax="usmarc" name="F"/>
7623 <retrieval syntax="usmarc" name="B"/>
7624 <retrieval syntax="xml" name="marcxml"
7625 identifier="info:srw/schema/1/marcxml-v1.1">
7626 <backend syntax="usmarc" name="F">
7627 <marc inputformat="marc" outputformat="marcxml"
7628 inputcharset="marc-8"/>
7631 <retrieval syntax="xml" name="dc">
7632 <backend syntax="usmarc" name="F">
7633 <marc inputformat="marc" outputformat="marcxml"
7634 inputcharset="marc-8"/>
7635 <xslt stylesheet="MARC21slim2DC.xsl"/>
7642 This means that our frontend supports:
7646 MARC21 F(ull) records.
7651 MARC21 B(rief) records.
7661 Dublin core records.
7667 <example id="tools.retrieval.marcxml">
7668 <title>MARCXML backend</title>
7670 SRW/SRU and Solr backends return records in XML.
7671 If they return MARCXML or MarcXchange, the retrieval module
7672 can convert those into ISO2709 formats, most commonly USMARC
7674 In this example, the backend returns MARCXML for schema="marcxml".
7676 <programlisting><![CDATA[
7678 <retrieval syntax="usmarc">
7679 <backend syntax="xml" name="marcxml">
7680 <marc inputformat="xml" outputformat="marc"
7681 outputcharset="marc-8"/>
7684 <retrieval syntax="xml" name="marcxml"
7685 identifier="info:srw/schema/1/marcxml-v1.1"/>
7686 <retrieval syntax="xml" name="dc">
7687 <backend syntax="xml" name="marcxml">
7688 <xslt stylesheet="MARC21slim2DC.xsl"/>
7695 This means that our frontend supports:
7699 MARC21 records (any element set name) in MARC-8 encoding.
7704 MARCXML records for element-set=marcxml
7709 Dublin core records for element-set=dc.
7716 <sect2 id="tools.retrieval.api">
7719 It should be easy to use the retrieval systems from applications. Refer
7721 <filename>yaz/retrieval.h</filename> and
7722 <filename>yaz/record_conv.h</filename>.
7726 <sect1 id="sorting">
7727 <title>Sorting</title>
7729 This chapter describes sorting and how it is supported in YAZ.
7730 Sorting applies to a result-set.
7732 <ulink url="http://www.loc.gov/z3950/agency/markup/05.html#3.2.7">
7733 Z39.50 sorting facility
7735 takes one or more input result-sets
7736 and one result-set as output. The most simple case is that
7737 the input-set is the same as the output-set.
7740 Z39.50 sorting has a separate APDU (service) that is, thus, performed
7741 following a search (two phases).
7744 In SRU/Solr, however, the model is different. Here, sorting is specified
7745 during the search operation. Note, however, that SRU might
7746 perform sort as separate search, by referring to an existing result-set
7747 in the query (result-set reference).
7750 <title>Using the Z39.50 sort service</title>
7752 yaz-client and the ZOOM API support the Z39.50 sort facility. In any
7753 case the sort sequence or sort critiera is using a string notation.
7754 This notation is a one-line notation suitable for being manually
7755 entered or generated, and allows for easy logging (one liner).
7756 For the ZOOM API, the sort is specified in the call to ZOOM_query_sortby
7757 function. For yaz-client the sort is performed and specified using
7758 the sort and sort+ commands. For description of the sort criteria notation
7759 refer to the <link linkend="sortspec">sort command</link> in the
7763 The ZOOM API might choose one of several sort strategies for
7764 sorting. Refer to <xref linkend="zoom-sort-strategy"/>.
7768 <title>Type-7 sort</title>
7770 Type-7 sort is an extension to the Bib-1 based RPN query where the
7771 sort specification is embedded as an Attribute-Plus-Term.
7774 The objectives for introducing Type-7 sorting is that it allows
7775 a client to perform sorting even if it does not implement/support
7776 Z39.50 sort. Virtually all Z39.50 client software supports
7777 RPN queries. It also may improve performance because the sort
7778 critieria is specified along with the search query.
7781 The sort is triggered by the presence of type 7, and the value of type 7
7783 <ulink url="http://www.loc.gov/z3950/agency/asn1.html#SortKeySpec">
7786 The value for type 7 is 1 for ascending and 2 for descending.
7788 <ulink url="http://www.loc.gov/z3950/agency/asn1.html#SortElement">
7791 only the generic part is handled. If generic sortKey is of type
7792 sortField, then attribute type 1 is present and the value is
7793 sortField (InternationalString). If generic sortKey is of type
7794 sortAttributes, then the attributes in the list are used. Generic sortKey
7795 of type elementSpec is not supported.
7798 The term in the sorting Attribute-Plus-Term combo should hold
7799 an integer. The value is 0 for primary sorting criteria, 1 for second
7805 <title>Facets</title>
7807 YAZ supports facets in the Solr, SRU 2.0 and Z39.50 protocols.
7810 Like Type-1/RPN, YAZ supports a string notation for specifying
7811 facets. For the API this is performed by
7812 <function>yaz_pqf_parse_facet_list</function>.
7815 For ZOOM C the facets are given by option "facets".
7816 For yaz-client it is used for the 'facets' command.
7819 The grammar of this specification is as follows:
7821 facet-spec ::= facet-list
7823 facet-list ::= facet-list ',' attr-spec | attr-spec
7825 attr-spec ::= attr-spec '@attr' string | '@attr' string
7828 The notation is inspired by PQF. The string following '@attr'
7829 must not include blanks and is of the form
7830 <replaceable>type</replaceable><literal>=</literal><replaceable>value</replaceable>,
7831 where <replaceable>type</replaceable> is an integer and
7832 <replaceable>value</replaceable> is a string or an integer.
7835 The Facets specification is not Bib-1. The following types apply:
7837 <table id="facet.attributes">
7838 <title>Facet attributes</title>
7840 <colspec colwidth="2*" colname="type"></colspec>
7841 <colspec colwidth="9*" colname="description"></colspec>
7845 <entry>Description</entry>
7852 Field-name. This is often a string, e.g. "Author", "Year", etc.
7858 Sort order. Value should be an integer.
7859 Value 0: count descending (frequency). Value 1: alpha ascending.
7865 Number of terms requested.
7880 <title>The ODR Module</title>
7881 <sect1 id="odr.introduction">
7882 <title>Introduction</title>
7884 &odr; is the BER-encoding/decoding subsystem of &yaz;. Care has been taken
7885 to isolate &odr; from the rest of the package - specifically from the
7886 transport interface. &odr; may be used in any context where basic
7887 ASN.1/BER representations are used.
7890 If you are only interested in writing a Z39.50 implementation based on
7891 the PDUs that are already provided with &yaz;, you only need to concern
7892 yourself with the section on managing ODR streams
7893 (<xref linkend="odr.use"/>). Only if you need to
7894 implement ASN.1 beyond that which has been provided, should you
7895 worry about the second half of the documentation
7896 (<xref linkend="odr.programming"/>).
7897 If you use one of the higher-level interfaces, you can skip this
7901 This is important, so we'll repeat it for emphasis: <emphasis>You do
7902 not need to read <xref linkend="odr.programming"/>
7903 to implement Z39.50 with &yaz;.</emphasis>
7906 If you need a part of the protocol that isn't already in &yaz;, you
7907 should contact the authors before going to work on it yourself: We
7908 might already be working on it. Conversely, if you implement a useful
7909 part of the protocol before us, we'd be happy to include it in a
7913 <sect1 id="odr.use">
7914 <title>Using ODR</title>
7915 <sect2 id="odr.streams">
7916 <title>ODR Streams</title>
7918 Conceptually, the ODR stream is the source of encoded data in the
7919 decoding mode; when encoding, it is the receptacle for the encoded
7920 data. Before you can use an ODR stream it must be allocated. This is
7921 done with the function
7924 ODR odr_createmem(int direction);
7927 The <function>odr_createmem()</function> function takes as argument one
7928 of three manifest constants: <literal>ODR_ENCODE</literal>,
7929 <literal>ODR_DECODE</literal>, or <literal>ODR_PRINT</literal>.
7930 An &odr; stream can be in only one mode - it is not possible to change
7931 its mode once it's selected. Typically, your program will allocate
7932 at least two ODR streams - one for decoding, and one for encoding.
7935 When you're done with the stream, you can use
7938 void odr_destroy(ODR o);
7941 to release the resources allocated for the stream.
7944 <sect2 id="odr.memory.management">
7945 <title id="memory">Memory Management</title>
7947 Two forms of memory management take place in the &odr; system. The first
7948 one, which has to do with allocating little bits of memory (sometimes
7949 quite large bits of memory, actually) when a protocol package is
7950 decoded, and turned into a complex of interlinked structures. This
7951 section deals with this system, and how you can use it for your own
7952 purposes. The next section deals with the memory management which is
7953 required when encoding data - to make sure that a large enough buffer is
7954 available to hold the fully encoded PDU.
7957 The &odr; module has its own memory management system, which is
7958 used whenever memory is required. Specifically, it is used to allocate
7959 space for data when decoding incoming PDUs. You can use the memory
7960 system for your own purposes, by using the function
7963 void *odr_malloc(ODR o, size_t size);
7966 You can't use the normal <function>free(2)</function> routine to free
7967 memory allocated by this function, and &odr; doesn't provide a parallel
7968 function. Instead, you can call
7971 void odr_reset(ODR o);
7974 when you are done with the
7975 memory: Everything allocated since the last call to
7976 <function>odr_reset()</function> is released.
7977 The <function>odr_reset()</function> call is also required to clear
7978 up an error condition on a stream.
7984 size_t odr_total(ODR o);
7987 returns the number of bytes allocated on the stream since the last call to
7988 <function>odr_reset()</function>.
7991 The memory subsystem of &odr; is fairly efficient at allocating and
7992 releasing little bits of memory. Rather than managing the individual,
7993 small bits of space, the system maintains a free-list of larger chunks
7994 of memory, which are handed out in small bits. This scheme is
7995 generally known as a <emphasis>nibble-memory</emphasis> system.
7996 It is very useful for maintaining short-lived constructions such
8000 If you want to retain a bit of memory beyond the next call to
8001 <function>odr_reset()</function>, you can use the function
8004 ODR_MEM odr_extract_mem(ODR o);
8007 This function will give you control of the memory recently allocated
8008 on the ODR stream. The memory will live (past calls to
8009 <function>odr_reset()</function>), until you call the function
8012 void odr_release_mem(ODR_MEM p);
8015 The opaque <literal>ODR_MEM</literal> handle has no other purpose than
8016 referencing the memory block for you until you want to release it.
8019 You can use <function>odr_extract_mem()</function> repeatedly between
8020 allocating data, to retain individual control of separate chunks of data.
8023 <sect2 id="odr.encoding.and.decoding">
8024 <title>Encoding and Decoding Data</title>
8026 When encoding data, the ODR stream will write the encoded octet string
8027 in an internal buffer. To retrieve the data, use the function
8030 char *odr_getbuf(ODR o, int *len, int *size);
8033 The integer pointed to by len is set to the length of the encoded
8034 data, and a pointer to that data is returned. <literal>*size</literal>
8035 is set to the size of the buffer (unless <literal>size</literal> is null,
8036 signaling that you are not interested in the size). The next call to
8037 a primitive function using the same &odr; stream will overwrite the
8038 data, unless a different buffer has been supplied using the call
8041 void odr_setbuf(ODR o, char *buf, int len, int can_grow);
8044 which sets the encoding (or decoding) buffer used by
8045 <literal>o</literal> to <literal>buf</literal>, using the length
8046 <literal>len</literal>.
8047 Before a call to an encoding function, you can use
8048 <function>odr_setbuf()</function> to provide the stream with an encoding
8049 buffer of sufficient size (length). The <literal>can_grow</literal>
8050 parameter tells the encoding &odr; stream whether it is allowed to use
8051 <function>realloc(2)</function> to increase the size of the buffer when
8052 necessary. The default condition of a new encoding stream is equivalent
8053 to the results of calling
8056 odr_setbuf(stream, 0, 0, 1);
8059 In this case, the stream will allocate and reallocate memory as
8060 necessary. The stream reallocates memory by repeatedly doubling the
8061 size of the buffer - the result is that the buffer will typically
8062 reach its maximum, working size with only a small number of reallocation
8063 operations. The memory is freed by the stream when the latter is destroyed,
8064 unless it was assigned by the user with the <literal>can_grow</literal>
8065 parameter set to zero (in this case, you are expected to retain
8066 control of the memory yourself).
8069 To assume full control of an encoded buffer, you must first call
8070 <function>odr_getbuf()</function> to fetch the buffer and its length.
8071 Next, you should call <function>odr_setbuf()</function> to provide a
8072 different buffer (or a null pointer) to the stream. In the simplest
8073 case, you will reuse the same buffer over and over again, and you
8074 will just need to call <function>odr_getbuf()</function> after each
8075 encoding operation to get the length and address of the buffer.
8076 Note that the stream may reallocate the buffer during an encoding
8077 operation, so it is necessary to retrieve the correct address after
8078 each encoding operation.
8081 It is important to realize that the ODR stream will not release this
8082 memory when you call <function>odr_reset()</function>: It will
8083 merely update its internal pointers to prepare for the encoding of a
8085 When the stream is released by the <function>odr_destroy()</function>
8086 function, the memory given to it by <function>odr_setbuf</function> will
8087 be released <emphasis>only</emphasis> if the <literal>can_grow</literal>
8088 parameter to <function>odr_setbuf()</function> was nonzero. The
8089 <literal>can_grow</literal> parameter, in other words, is a way of
8090 signaling who is to own the buffer, you or the ODR stream. If you never call
8091 <function>odr_setbuf()</function> on your encoding stream, which is
8092 typically the case, the buffer allocated by the stream will belong to
8093 the stream by default.
8096 When you wish to decode data, you should first call
8097 <function>odr_setbuf()</function>, to tell the decoding stream
8098 where to find the encoded data, and how long the buffer is
8099 (the <literal>can_grow</literal> parameter is ignored by a decoding
8100 stream). After this, you can call the function corresponding to the
8101 data you wish to decode (eg, <function>odr_integer()</function> odr
8102 <function>z_APDU()</function>).
8104 <example id="example.odr.encoding.and.decoding.functions">
8105 <title>Encoding and decoding functions</title>
8107 int odr_integer(ODR o, Odr_int **p, int optional, const char *name);
8109 int z_APDU(ODR o, Z_APDU **p, int optional, const char *name);
8113 If the data is absent (or doesn't match the tag corresponding to
8114 the type), the return value will be either 0 or 1 depending on the
8115 <literal>optional</literal> flag. If <literal>optional</literal>
8116 is 0 and the data is absent, an error flag will be raised in the
8117 stream, and you'll need to call <function>odr_reset()</function> before
8118 you can use the stream again. If <literal>optional</literal> is
8119 nonzero, the pointer <emphasis>pointed</emphasis> to/ by
8120 <literal>p</literal> will be set to the null value, and the function
8122 The <literal>name</literal> argument is used to pretty-print the
8123 tag in question. It may be set to <literal>NULL</literal> if
8124 pretty-printing is not desired.
8127 If the data value is found where it's expected, the pointer
8128 <emphasis>pointed to</emphasis> by the <literal>p</literal> argument
8129 will be set to point to the decoded type.
8130 The space for the type will be allocated and owned by the &odr;
8131 stream, and it will live until you call
8132 <function>odr_reset()</function> on the stream. You cannot use
8133 <function>free(2)</function> to release the memory.
8134 You can decode several data elements (by repeated calls to
8135 <function>odr_setbuf()</function> and your decoding function), and
8136 new memory will be allocated each time. When you do call
8137 <function>odr_reset()</function>, everything decoded since the
8138 last call to <function>odr_reset()</function> will be released.
8140 <example id="example.odr.encoding.of.integer">
8141 <title>Encoding and decoding of an integer</title>
8143 The use of the double indirection can be a little confusing at first
8144 (its purpose will become clear later on, hopefully),
8145 so an example is in order. We'll encode an integer value, and
8146 immediately decode it again using a different stream. A useless, but
8147 informative operation.
8149 <programlisting><![CDATA[
8150 void do_nothing_useful(Odr_int value)
8153 Odr_int *valp, *resvalp;
8157 /* allocate streams */
8158 if (!(encode = odr_createmem(ODR_ENCODE)))
8160 if (!(decode = odr_createmem(ODR_DECODE)))
8164 if (odr_integer(encode, &valp, 0, 0) == 0)
8166 printf("encoding went bad\n");
8169 bufferp = odr_getbuf(encode, &len, 0);
8170 printf("length of encoded data is %d\n", len);
8172 /* now let's decode the thing again */
8173 odr_setbuf(decode, bufferp, len, 0);
8174 if (odr_integer(decode, &resvalp, 0, 0) == 0)
8176 printf("decoding went bad\n");
8179 /* ODR_INT_PRINTF format for printf (such as %d) */
8180 printf("the value is " ODR_INT_PRINTF "\n", *resvalp);
8183 odr_destroy(encode);
8184 odr_destroy(decode);
8189 This looks like a lot of work, offhand. In practice, the &odr; streams
8190 will typically be allocated once, in the beginning of your program
8191 (or at the beginning of a new network session), and the encoding
8192 and decoding will only take place in a few, isolated places in your
8193 program, so the overhead is quite manageable.
8197 <sect2 id="odr.printing">
8198 <title>Printing</title>
8200 When an ODR stream is created of type <literal>ODR_PRINT</literal>
8201 the ODR module will print the contents of a PDU in a readable format.
8202 By default output is written to the <literal>stderr</literal> stream.
8203 This behavior can be changed, however, by calling the function
8205 odr_setprint(ODR o, FILE *file);
8207 before encoders or decoders are being invoked.
8208 It is also possible to direct the output to a buffer (or indeed
8209 another file), by using the more generic mechanism:
8211 void odr_set_stream(ODR o, void *handle,
8212 void (*stream_write)(ODR o, void *handle, int type,
8213 const char *buf, int len),
8214 void (*stream_close)(void *handle));
8216 Here the user provides an opaque handle and two handlers,
8217 <replaceable>stream_write</replaceable> for writing,
8218 and <replaceable>stream_close</replaceable> which is supposed
8219 to close/free resources associated with handle.
8220 The <replaceable>stream_close</replaceable> handler is optional and
8221 if NULL for the function is provided, it will not be invoked.
8222 The <replaceable>stream_write</replaceable> takes the ODR handle
8223 as parameter, the user-defined handle, a type
8224 <literal>ODR_OCTETSTRING</literal>, <literal>ODR_VISIBLESTRING</literal>
8225 which indicates the type of contents being written.
8228 Another utility useful for diagnostics (error handling) or as
8229 part of the printing facilities is:
8231 const char **odr_get_element_path(ODR o);
8233 which returns a list of current elements that ODR deals with at the
8234 moment. For the returned array, say <literal>ar</literal>,
8235 then <literal>ar[0]</literal> is the top level element,
8236 <literal>ar[n]</literal> is the last. The last element has the
8237 property that <literal>ar[n+1] == NULL</literal>.
8239 <example id="example.odr.element.path.record">
8240 <title>Element Path for record</title>
8242 For a database record part of a PresentResponse the
8243 array returned by <function>odr_get_element</function>
8244 is <literal>presentResponse</literal>, <literal>databaseOrSurDiagnostics</literal>, <literal>?</literal>, <literal>record</literal>, <literal>?</literal>, <literal>databaseRecord</literal> . The question mark appears due to
8245 unnamed constructions.
8249 <sect2 id="odr.diagnostics">
8250 <title>Diagnostics</title>
8252 The encoding/decoding functions all return 0 when an error occurs.
8253 Until you call <function>odr_reset()</function>, you cannot use the
8254 stream again, and any function called will immediately return 0.
8257 To provide information to the programmer or administrator, the function
8260 void odr_perror(ODR o, char *message);
8263 is provided, which prints the <literal>message</literal> argument to
8264 <literal>stderr</literal> along with an error message from the stream.
8267 You can also use the function
8270 int odr_geterror(ODR o);
8273 to get the current error number from the screen. The number will be
8274 one of these constants:
8276 <table frame="top" id="odr.error.codes">
8277 <title>ODR Error codes</title>
8282 <entry>Description</entry>
8287 <entry>OMEMORY</entry><entry>Memory allocation failed.</entry>
8290 <entry>OSYSERR</entry><entry>A system- or library call has failed.
8291 The standard diagnostic variable <literal>errno</literal> should be
8292 examined to determine the actual error.</entry>
8295 <entry>OSPACE</entry><entry>No more space for encoding.
8296 This will only occur when the user has explicitly provided a
8297 buffer for an encoding stream without allowing the system to
8298 allocate more space.</entry>
8301 <entry>OREQUIRED</entry><entry>This is a common protocol error; A
8302 required data element was missing during encoding or decoding.</entry>
8305 <entry>OUNEXPECTED</entry><entry>An unexpected data element was
8306 found during decoding.</entry>
8309 <entry>OOTHER</entry><entry>Other error. This is typically an
8310 indication of misuse of the &odr; system by the programmer, and also
8311 that the diagnostic system isn't as good as it should be, yet.</entry>
8317 The character string array
8323 can be indexed by the error code to obtain a human-readable
8324 representation of the problem.
8327 <sect2 id="odr.summary.and.synopsis">
8328 <title>Summary and Synopsis</title>
8330 #include <yaz/odr.h>
8332 ODR odr_createmem(int direction);
8334 void odr_destroy(ODR o);
8336 void odr_reset(ODR o);
8338 char *odr_getbuf(ODR o, int *len, int *size);
8340 void odr_setbuf(ODR o, char *buf, int len, int can_grow);
8342 void *odr_malloc(ODR o, int size);
8344 NMEM odr_extract_mem(ODR o);
8346 int odr_geterror(ODR o);
8348 void odr_perror(ODR o, const char *message);
8350 extern char *odr_errlist[];
8354 <sect1 id="odr.programming">
8355 <title>Programming with ODR</title>
8357 The API of &odr; is designed to reflect the structure of ASN.1, rather
8358 than BER itself. Future releases may be able to represent data in
8359 other external forms.
8363 There is an ASN.1 tutorial available at
8364 <ulink url="&url.asn.1.tutorial;">this site</ulink>.
8365 This site also has standards for ASN.1 (X.680) and BER (X.690)
8366 <ulink url="&url.asn.1.standards;">online</ulink>.
8370 The ODR interface is based loosely on that of the Sun Microsystems
8372 Specifically, each function which corresponds to an ASN.1 primitive
8373 type has a dual function. Depending on the settings of the ODR
8374 stream which is supplied as a parameter, the function may be used
8375 either to encode or decode data. The functions that can be built
8376 using these primitive functions, to represent more complex data types,
8377 share this quality. The result is that you only have to enter the
8378 definition for a type once - and you have the functionality of encoding,
8379 decoding (and pretty-printing) all in one unit.
8380 The resulting C source code is quite compact, and is a pretty
8381 straightforward representation of the source ASN.1 specification.
8384 In many cases, the model of the XDR functions works quite well in this
8386 In others, it is less elegant. Most of the hassle comes from the optional
8387 SEQUENCE members which don't exist in XDR.
8389 <sect2 id="odr.primitive.asn1.types">
8390 <title>The Primitive ASN.1 Types</title>
8392 ASN.1 defines a number of primitive types (many of which correspond
8393 roughly to primitive types in structured programming languages, such as C).
8395 <sect3 id="odr.integer">
8396 <title>INTEGER</title>
8398 The &odr; function for encoding or decoding (or printing) the ASN.1
8399 INTEGER type looks like this:
8402 int odr_integer(ODR o, Odr_int **p, int optional, const char *name);
8405 The <literal>Odr_int</literal> is just a simple integer.
8408 This form is typical of the primitive &odr; functions. They are named
8409 after the type of data that they encode or decode. They take an &odr;
8410 stream, an indirect reference to the type in question, and an
8411 <literal>optional</literal> flag (corresponding to the OPTIONAL keyword
8412 of ASN.1) as parameters. They all return an integer value of either one
8414 When you use the primitive functions to construct encoders for complex
8415 types of your own, you should follow this model as well. This
8416 ensures that your new types can be reused as elements in yet more
8420 The <literal>o</literal> parameter should obviously refer to a properly
8421 initialized &odr; stream of the right type (encoding/decoding/printing)
8422 for the operation that you wish to perform.
8425 When encoding or printing, the function first looks at
8426 <literal>* p</literal>. If <literal>* p</literal> (the pointer pointed
8427 to by <literal>p</literal>) is a null pointer, this is taken to mean that
8428 the data element is absent. If the <literal>optional</literal> parameter
8429 is nonzero, the function will return one (signifying success) without
8430 any further processing. If the <literal>optional</literal> is zero, an
8431 internal error flag is set in the &odr; stream, and the function will
8432 return 0. No further operations can be carried out on the stream without
8433 a call to the function <function>odr_reset()</function>.
8436 If <literal>*p</literal> is not a null pointer, it is expected to
8437 point to an instance of the data type. The data will be subjected to
8438 the encoding rules, and the result will be placed in the buffer held
8439 by the &odr; stream.
8442 The other ASN.1 primitives have similar functions that operate in
8446 <sect3 id="odr.boolean">
8447 <title>BOOLEAN</title>
8449 int odr_bool(ODR o, Odr_bool **p, int optional, const char *name);
8452 <sect3 id="odr.real">
8458 <sect3 id="odr.null">
8461 int odr_null(ODR o, Odr_null **p, int optional, const char *name);
8464 In this case, the value of **p is not important. If <literal>*p</literal>
8465 is different from the null pointer, the null value is present, otherwise
8469 <sect3 id="odr.octet.string">
8470 <title>OCTET STRING</title>
8472 typedef struct odr_oct
8478 int odr_octetstring(ODR o, Odr_oct **p, int optional,
8482 The <literal>buf</literal> field should point to the character array
8483 that holds the octetstring. The <literal>len</literal> field holds the
8485 The character array need not be null-terminated.
8488 To make things a little easier, an alternative is given for string
8489 types that are not expected to contain embedded NULL characters (e.g.
8493 int odr_cstring(ODR o, char **p, int optional, const char *name);
8496 which encodes or decodes between OCTETSTRING representations and
8497 null-terminated C strings.
8500 Functions are provided for the derived string types, e.g.:
8503 int odr_visiblestring(ODR o, char **p, int optional,
8507 <sect3 id="odr.bit.string">
8508 <title>BIT STRING</title>
8510 int odr_bitstring(ODR o, Odr_bitmask **p, int optional,
8514 The opaque type <literal>Odr_bitmask</literal> is only suitable for
8515 holding relatively brief bit strings, e.g. for options fields, etc.
8516 The constant <literal>ODR_BITMASK_SIZE</literal> multiplied by 8
8517 gives the maximum possible number of bits.
8520 A set of macros are provided for manipulating the
8521 <literal>Odr_bitmask</literal> type:
8524 void ODR_MASK_ZERO(Odr_bitmask *b);
8526 void ODR_MASK_SET(Odr_bitmask *b, int bitno);
8528 void ODR_MASK_CLEAR(Odr_bitmask *b, int bitno);
8530 int ODR_MASK_GET(Odr_bitmask *b, int bitno);
8533 The functions are modeled after the manipulation functions that
8534 accompany the <literal>fd_set</literal> type used by the
8535 <function>select(2)</function> call.
8536 <literal>ODR_MASK_ZERO</literal> should always be called first on a
8537 new bitmask, to initialize the bits to zero.
8540 <sect3 id="odr.object.identifier">
8541 <title>OBJECT IDENTIFIER</title>
8543 int odr_oid(ODR o, Odr_oid **p, int optional, const char *name);
8546 The C OID representation is simply an array of integers, terminated by
8547 the value -1 (the <literal>Odr_oid</literal> type is synonymous with
8548 the <literal>short</literal> type).
8549 We suggest that you use the OID database module (see
8550 <xref linkend="tools.oid.database"/>) to handle object identifiers
8551 in your application.
8555 <sect2 id="odr.tagging.primitive.types">
8556 <title>Tagging Primitive Types</title>
8558 The simplest way of tagging a type is to use the
8559 <function>odr_implicit_tag()</function> or
8560 <function>odr_explicit_tag()</function> macros:
8563 int odr_implicit_tag(ODR o, Odr_fun fun, int class, int tag,
8564 int optional, const char *name);
8566 int odr_explicit_tag(ODR o, Odr_fun fun, int class, int tag,
8567 int optional, const char *name);
8570 To create a type derived from the integer type by implicit tagging, you
8574 MyInt ::= [210] IMPLICIT INTEGER
8577 In the &odr; system, this would be written like:
8580 int myInt(ODR o, Odr_int **p, int optional, const char *name)
8582 return odr_implicit_tag(o, odr_integer, p,
8583 ODR_CONTEXT, 210, optional, name);
8587 The function <function>myInt()</function> can then be used like any of
8588 the primitive functions provided by &odr;. Note that the behavior of
8589 <function>odr_explicit_tag()</function>
8590 and <function>odr_implicit_tag()</function> macros
8591 act exactly the same as the functions they are applied to - they
8592 respond to error conditions, etc, in the same manner - they
8593 simply have three extra parameters. The class parameter may
8594 take one of the values: <literal>ODR_CONTEXT</literal>,
8595 <literal>ODR_PRIVATE</literal>, <literal>ODR_UNIVERSAL</literal>, or
8596 <literal>/ODR_APPLICATION</literal>.
8599 <sect2 id="odr.constructed.types">
8600 <title>Constructed Types</title>
8602 Constructed types are created by combining primitive types. The
8603 &odr; system only implements the SEQUENCE and SEQUENCE OF constructions
8604 (although adding the rest of the container types should be simple
8605 enough, if the need arises).
8608 For implementing SEQUENCEs, the functions
8611 int odr_sequence_begin(ODR o, void *p, int size, const char *name);
8612 int odr_sequence_end(ODR o);
8618 The <function>odr_sequence_begin()</function> function should be
8619 called in the beginning of a function that implements a SEQUENCE type.
8620 Its parameters are the &odr; stream, a pointer (to a pointer to the type
8621 you're implementing), and the <literal>size</literal> of the type
8622 (typically a C structure). On encoding, it returns 1 if
8623 <literal>* p</literal> is a null pointer. The <literal>size</literal>
8624 parameter is ignored. On decoding, it returns 1 if the type is found in
8625 the data stream. <literal>size</literal> bytes of memory are allocated,
8626 and <literal>*p</literal> is set to point to this space.
8627 The <function>odr_sequence_end()</function> is called at the end of the
8628 complex function. Assume that a type is defined like this:
8631 MySequence ::= SEQUENCE {
8633 boolval BOOLEAN OPTIONAL
8637 The corresponding &odr; encoder/decoder function and the associated data
8638 structures could be written like this:
8641 typedef struct MySequence
8647 int mySequence(ODR o, MySequence **p, int optional, const char *name)
8649 if (odr_sequence_begin(o, p, sizeof(**p), name) == 0)
8650 return optional && odr_ok(o);
8652 odr_integer(o, &(*p)->intval, 0, "intval") &&
8653 odr_bool(o, &(*p)->boolval, 1, "boolval") &&
8654 odr_sequence_end(o);
8658 Note the 1 in the call to <function>odr_bool()</function>, to mark
8659 that the sequence member is optional.
8660 If either of the member types had been tagged, the macros
8661 <function>odr_implicit_tag()</function> or
8662 <function>odr_explicit_tag()</function>
8663 could have been used.
8664 The new function can be used exactly like the standard functions provided
8665 with &odr;. It will encode, decode or pretty-print a data value of the
8666 <literal>MySequence</literal> type. We like to name types with an
8667 initial capital, as done in ASN.1 definitions, and to name the
8668 corresponding function with the first character of the name in lower case.
8669 You could, of course, name your structures, types, and functions any way
8670 you please - as long as you're consistent, and your code is easily readable.
8671 <literal>odr_ok</literal> is just that - a predicate that returns the
8672 state of the stream. It is used to ensure that the behavior of the new
8673 type is compatible with the interface of the primitive types.
8676 <sect2 id="odr.tagging.constructed.types">
8677 <title>Tagging Constructed Types</title>
8680 See <xref linkend="odr.tagging.primitive.types"/> for information
8681 on how to tag the primitive types, as well as types that are
8685 <sect3 id="odr.implicit.tagging">
8686 <title>Implicit Tagging</title>
8688 Assume the type above had been defined as
8691 MySequence ::= [10] IMPLICIT SEQUENCE {
8693 boolval BOOLEAN OPTIONAL
8697 You would implement this in &odr; by calling the function
8700 int odr_implicit_settag(ODR o, int class, int tag);
8703 which overrides the tag of the type immediately following it. The
8704 macro <function>odr_implicit_tag()</function> works by calling
8705 <function>odr_implicit_settag()</function> immediately
8706 before calling the function pointer argument.
8707 Your type function could look like this:
8710 int mySequence(ODR o, MySequence **p, int optional, const char *name)
8712 if (odr_implicit_settag(o, ODR_CONTEXT, 10) == 0 ||
8713 odr_sequence_begin(o, p, sizeof(**p), name) == 0)
8714 return optional && odr_ok(o);
8716 odr_integer(o, &(*p)->intval, 0, "intval") &&
8717 odr_bool(o, &(*p)->boolval, 1, "boolval") &&
8718 odr_sequence_end(o);
8722 The definition of the structure <literal>MySequence</literal> would be
8726 <sect3 id="odr.explicit.tagging">
8727 <title>Explicit Tagging</title>
8729 Explicit tagging of constructed types is a little more complicated,
8730 since you are in effect adding a level of construction to the data.
8733 Assume the definition:
8736 MySequence ::= [10] IMPLICIT SEQUENCE {
8738 boolval BOOLEAN OPTIONAL
8742 Since the new type has an extra level of construction, two new functions
8743 are needed to encapsulate the base type:
8746 int odr_constructed_begin(ODR o, void *p, int class, int tag,
8749 int odr_constructed_end(ODR o);
8752 Assume that the IMPLICIT in the type definition above were replaced
8753 with EXPLICIT (or that the IMPLICIT keyword was simply deleted, which
8754 would be equivalent). The structure definition would look the same,
8755 but the function would look like this:
8758 int mySequence(ODR o, MySequence **p, int optional, const char *name)
8760 if (odr_constructed_begin(o, p, ODR_CONTEXT, 10, name) == 0)
8761 return optional && odr_ok(o);
8762 if (o->direction == ODR_DECODE)
8763 *p = odr_malloc(o, sizeof(**p));
8764 if (odr_sequence_begin(o, p, sizeof(**p), 0) == 0)
8766 *p = 0; /* this is almost certainly a protocol error */
8770 odr_integer(o, &(*p)->intval, 0, "intval") &&
8771 odr_bool(o, &(*p)->boolval, 1, "boolval") &&
8772 odr_sequence_end(o) &&
8773 odr_constructed_end(o);
8777 Notice that the interface here gets kind of nasty. The reason is
8778 simple: Explicitly tagged, constructed types are fairly rare in
8779 the protocols that we care about, so the
8780 aesthetic annoyance (not to mention the dangers of a cluttered
8781 interface) is less than the time that would be required to develop a
8782 better interface. Nevertheless, it is far from satisfying, and it's a
8783 point that will be worked on in the future. One option for you would
8784 be to simply apply the <function>odr_explicit_tag()</function> macro to
8785 the first function, and not
8786 have to worry about <function>odr_constructed_*</function> yourself.
8787 Incidentally, as you might have guessed, the
8788 <function>odr_sequence_</function> functions are themselves
8789 implemented using the <function>/odr_constructed_</function> functions.
8793 <sect2 id="odr.sequence.of">
8794 <title>SEQUENCE OF</title>
8796 To handle sequences (arrays) of a specific type, the function
8799 int odr_sequence_of(ODR o, int (*fun)(ODR o, void *p, int optional),
8800 void *p, int *num, const char *name);
8803 The <literal>fun</literal> parameter is a pointer to the decoder/encoder
8804 function of the type. <literal>p</literal> is a pointer to an array of
8805 pointers to your type. <literal>num</literal> is the number of elements
8812 MyArray ::= SEQUENCE OF INTEGER
8815 The C representation might be
8818 typedef struct MyArray
8825 And the function might look like
8828 int myArray(ODR o, MyArray **p, int optional, const char *name)
8830 if (o->direction == ODR_DECODE)
8831 *p = odr_malloc(o, sizeof(**p));
8832 if (odr_sequence_of(o, odr_integer, &(*p)->elements,
8833 &(*p)->num_elements, name))
8836 return optional && odr_ok(o);
8840 <sect2 id="odr.choice.types">
8841 <title>CHOICE Types</title>
8843 The choice type is used fairly often in some ASN.1 definitions, so
8844 some work has gone into streamlining its interface.
8847 CHOICE types are handled by the function:
8850 int odr_choice(ODR o, Odr_arm arm[], void *p, void *whichp,
8854 The <literal>arm</literal> array is used to describe each of the possible
8855 types that the CHOICE type may assume. Internally in your application,
8856 the CHOICE type is represented as a discriminated union. That is, a
8857 C union accompanied by an integer (or enum) identifying the active
8859 <literal>whichp</literal> is a pointer to the union discriminator.
8860 When encoding, it is examined to determine the current type.
8861 When decoding, it is set to reference the type that was found in
8865 The Odr_arm type is defined thus:
8868 typedef struct odr_arm
8879 The interpretation of the fields are:
8883 <term>tagmode</term>
8884 <listitem><para>Either <literal>ODR_IMPLICIT</literal>,
8885 <literal>ODR_EXPLICIT</literal>, or <literal>ODR_NONE</literal> (-1)
8886 to mark no tagging.</para></listitem>
8890 <listitem><para>The value of the discriminator that corresponds to
8891 this CHOICE element. Typically, it will be a #defined constant, or
8892 an enum member.</para></listitem>
8896 <listitem><para>A pointer to a function that implements the type of
8897 the CHOICE member. It may be either a standard &odr; type or a type
8898 defined by yourself.</para></listitem>
8902 <listitem><para>Name of tag.</para></listitem>
8906 A handy way to prepare the array for use by the
8907 <function>odr_choice()</function> function is to
8908 define it as a static, initialized array in the beginning of your
8909 decoding/encoding function. Assume the type definition:
8912 MyChoice ::= CHOICE {
8914 tagged [99] IMPLICIT INTEGER,
8919 Your C type might look like
8922 typedef struct MyChoice
8939 And your function could look like this:
8942 int myChoice(ODR o, MyChoice **p, int optional, const char *name)
8944 static Odr_arm arm[] =
8946 {-1, -1, -1, MyChoice_untagged, odr_integer, "untagged"},
8947 {ODR_IMPLICIT, ODR_CONTEXT, 99, MyChoice_tagged, odr_integer,
8949 {-1, -1, -1, MyChoice_other, odr_boolean, "other"},
8953 if (o->direction == ODR_DECODE)
8954 *p = odr_malloc(o, sizeof(**p);
8956 return optional && odr_ok(o);
8958 if (odr_choice(o, arm, &(*p)->u, &(*p)->which), name)
8961 return optional && odr_ok(o);
8965 In some cases (say, a non-optional choice which is a member of a
8966 sequence), you can "embed" the union and its discriminator in the
8967 structure belonging to the enclosing type, and you won't need to
8968 fiddle with memory allocation to create a separate structure to
8969 wrap the discriminator and union.
8972 The corresponding function is somewhat nicer in the Sun XDR interface.
8973 Most of the complexity of this interface comes from the possibility of
8974 declaring sequence elements (including CHOICEs) optional.
8977 The ASN.1 specifications naturally require that each member of a
8978 CHOICE have a distinct tag, so they can be told apart on decoding.
8979 Sometimes it can be useful to define a CHOICE that has multiple types
8980 that share the same tag. You'll need some other mechanism, perhaps
8981 keyed to the context of the CHOICE type. In effect, we would like to
8982 introduce a level of context-sensitiveness to our ASN.1 specification.
8983 When encoding an internal representation, we have no problem, as long
8984 as each CHOICE member has a distinct discriminator value. For
8985 decoding, we need a way to tell the choice function to look for a
8986 specific arm of the table. The function
8989 void odr_choice_bias(ODR o, int what);
8992 provides this functionality. When called, it leaves a notice for the next
8993 call to <function>odr_choice()</function> to be called on the decoding
8994 stream <literal>o</literal>, that only the <literal>arm</literal> entry with
8995 a <literal>which</literal> field equal to <literal>what</literal>
8999 The most important application (perhaps the only one, really) is in
9000 the definition of application-specific EXTERNAL encoders/decoders
9001 which will automatically decode an ANY member given the direct or
9006 <sect1 id="odr.debugging">
9007 <title>Debugging</title>
9009 The protocol modules are suffering somewhat from a lack of diagnostic
9010 tools at the moment. Specifically ways to pretty-print PDUs that
9011 aren't recognized by the system. We'll include something to this end
9012 in a not-too-distant release. In the meantime, what we do when we get
9013 packages we don't understand is to compile the ODR module with
9014 <literal>ODR_DEBUG</literal> defined. This causes the module to dump tracing
9015 information as it processes data units. With this output and the
9016 protocol specification (Z39.50), it is generally fairly easy to see
9021 <chapter id="comstack">
9022 <title>The COMSTACK Module</title>
9023 <sect1 id="comstack.synopsis">
9024 <title>Synopsis (blocking mode)</title>
9025 <programlisting><![CDATA[
9028 int size = 0, length_incoming;
9029 char server_address_str[] = "localhost:9999";
9030 void *server_address_ip;
9033 char *protocol_package = "GET / HTTP/1.0\r\n\r\n";
9034 int protocol_package_length = strlen(protocol_package);
9036 stack = cs_create(tcpip_type, 1, PROTO_HTTP);
9038 perror("cs_create"); /* use perror() here since we have no stack yet */
9042 server_address_ip = cs_straddr(stack, server_address_str);
9043 if (!server_address_ip) {
9044 fprintf(stderr, "cs_straddr: address could not be resolved\n");
9048 status = cs_connect(stack, server_address_ip);
9050 fprintf(stderr, "cs_connect: %s\n", cs_strerror(stack));
9054 status = cs_rcvconnect(stack);
9056 fprintf(stderr, "cs_rcvconnect: %s\n", cs_strerror(stack));
9060 status = cs_put(stack, protocol_package, protocol_package_length);
9062 fprintf(stderr, "cs_put: %s\n", cs_strerror(stack));
9066 /* Now get a response */
9067 length_incoming = cs_get(stack, &buf, &size);
9068 if (!length_incoming) {
9069 fprintf(stderr, "Connection closed\n");
9071 } else if (length_incoming < 0) {
9072 fprintf(stderr, "cs_get: %s\n", cs_strerror(stack));
9077 fwrite(buf, length_incoming, 1, stdout);
9088 <sect1 id="comstack.introduction">
9089 <title>Introduction</title>
9092 subsystem provides a transparent interface to different types of transport
9093 stacks for the exchange of BER-encoded data and HTTP packets.
9094 At present, the RFC1729 method (BER over TCP/IP), local UNIX socket and an
9095 experimental SSL stack are supported, but others may be added in time.
9096 The philosophy of the
9097 module is to provide a simple interface by hiding unused options and
9098 facilities of the underlying libraries. This is always done at the risk
9099 of losing generality, and it may prove that the interface will need
9104 There hasn't been interest in the XTImOSI stack for some years.
9105 Therefore, it is no longer supported.
9109 The interface is implemented in such a fashion that only the
9110 sub-layers constructed to the transport methods that you wish to
9111 use in your application are linked in.
9114 You will note that even though simplicity was a goal in the design,
9115 the interface is still orders of magnitudes more complex than the
9116 transport systems found in many other packages. One reason is that
9117 the interface needs to support the somewhat different requirements of
9118 the different lower-layer communications stacks; another important
9119 reason is that the interface seeks to provide a more or less
9120 industrial-strength approach to asynchronous event-handling.
9121 When no function is allowed to block, things get more complex -
9122 particularly on the server side.
9123 We urge you to have a look at the demonstration client and server
9124 provided with the package. They are meant to be easily readable and
9125 instructive, while still being at least moderately useful.
9128 <sect1 id="comstack.common">
9129 <title>Common Functions</title>
9130 <sect2 id="comstack.managing.endpoints">
9131 <title>Managing Endpoints</title>
9133 COMSTACK cs_create(CS_TYPE type, int blocking, int protocol);
9136 Creates an instance of the protocol stack - a communications endpoint.
9137 The <literal>type</literal> parameter determines the mode
9138 of communication. At present the following values are supported:
9142 <term><literal>tcpip_type</literal></term>
9143 <listitem><para>TCP/IP (BER over TCP/IP or HTTP over TCP/IP)
9147 <term><literal>ssl_type</literal></term>
9148 <listitem><para>Secure Socket Layer (SSL). This COMSTACK
9149 is experimental and is not fully implemented. If
9150 HTTP is used, this effectively is HTTPS.
9154 <term><literal>unix_type</literal></term>
9155 <listitem><para>Unix socket (unix only). Local Transfer via
9156 file socket. See <citerefentry><refentrytitle>unix</refentrytitle>
9157 <manvolnum>7</manvolnum></citerefentry>.
9162 The <function>cs_create</function> function returns a null-pointer
9163 if a system error occurs.
9164 The <literal>blocking</literal> parameter should be '1' if
9165 you wish the association to operate in blocking mode, and '0' otherwise.
9166 The <literal>protocol</literal> field should be
9167 <literal>PROTO_Z3950</literal> or <literal>PROTO_HTTP</literal>.
9168 Protocol <literal>PROTO_SR</literal> is no longer supported.
9171 void cs_close(COMSTACK handle);
9174 Closes the connection (as elegantly as the lower layers will permit),
9175 and releases the resources pointed to by the
9176 <literal>handle</literal>
9178 <literal>handle</literal>
9179 should not be referenced again after this call.
9183 We really need a soft disconnect, don't we?
9187 <sect2 id="comstack.data.exchange">
9188 <title>Data Exchange</title>
9190 int cs_put(COMSTACK handle, char *buf, int len);
9193 Sends <literal>buf</literal> down the wire.
9194 In blocking mode, this function will return only when a full buffer has
9195 been written, or an error has occurred. In nonblocking mode, it's
9196 possible that the function will be unable to send the full buffer
9197 at once, which will be indicated by a return value of 1.
9198 The function will keep track of the number of octets already written; you
9199 should call it repeatedly with the same values of <literal>buf</literal>
9200 and <literal>len</literal>, until the buffer has been transmitted.
9201 When a full buffer has been sent, the function will return 0 for
9202 success. The return value -1 indicates an error condition (see below).
9205 int cs_get(COMSTACK handle, char **buf, int *size);
9208 Receives a PDU or HTTP Response from the peer. Returns the number of
9210 In nonblocking mode, it is possible that not all of the packet can be
9211 read at once. In this case, the function returns 1. To simplify the
9212 interface, the function is
9213 responsible for managing the size of the buffer. It will be reallocated
9214 if necessary to contain large packages, and will sometimes be moved
9215 around internally by the subsystem when partial packages are read. Before
9217 <function>cs_get</function>
9218 for the first time, the buffer can be initialized to the null pointer,
9219 and the length should also be set to 0 (cs_get will perform a
9220 <function>malloc(2)</function>
9221 on the buffer for you). When a full buffer has been read, the size of
9222 the package is returned (which will always be greater than 1).
9223 The return value -1 indicates an error condition.
9226 See also the <function>cs_more()</function> function below.
9229 int cs_more(COMSTACK handle);
9232 The <function>cs_more()</function> function should be used in conjunction
9233 with <function>cs_get</function> and
9234 <function>select(2)</function>.
9235 The <function>cs_get()</function> function will sometimes
9236 (notably in the TCP/IP mode) read more than a single protocol package
9237 off the network. When this happens, the extra package is stored
9238 by the subsystem. After calling <function>cs_get()</function>, and before
9239 waiting for more input, You should always call
9240 <function>cs_more()</function>
9241 to check if there's a full protocol package already read. If
9242 <function>cs_more()</function>
9244 <function>cs_get()</function>
9245 can be used to immediately fetch the new package. For the
9247 subsystem, the function should always return 0, but if you want your
9248 stuff to be protocol independent, you should use it.
9252 The <function>cs_more()</function>
9253 function is required because the RFC1729-method
9254 does not provide a way of separating individual PDUs, short of
9255 partially decoding the BER. Some other implementations will carefully
9256 nibble at the packet by calling
9257 <function>read(2)</function>
9258 several times. This was felt to be too inefficient (or at least
9259 clumsy) - hence the call for this extra function.
9263 int cs_look(COMSTACK handle);
9266 This function is useful when you're operating in nonblocking
9268 <function>select(2)</function>
9269 tells you there's something happening on the line. It returns one of
9270 the following values:
9274 <term>CS_NONE</term>
9276 No event is pending. The data found on the line was not a
9281 <term>CS_CONNECT</term>
9283 A response to your connect request has been received. Call
9284 <function>cs_rcvconnect</function>
9285 to process the event and to finalize the connection establishment.
9289 <term>CS_DISCON</term>
9291 The other side has closed the connection (or maybe sent a disconnect
9292 request - but do we care? Maybe later). Call
9293 <function>cs_close</function> to close your end of the association
9298 <term>CS_LISTEN</term>
9300 A connect request has been received.
9301 Call <function>cs_listen</function> to process the event.
9305 <term>CS_DATA</term>
9307 There's data to be found on the line.
9308 Call <function>cs_get</function> to get it.
9314 You should be aware that even if
9315 <function>cs_look()</function>
9316 tells you that there's an event event pending, the corresponding
9317 function may still return and tell you there was nothing to be found.
9318 This means that only part of a package was available for reading. The
9319 same event will show up again, when more data has arrived.
9323 int cs_fileno(COMSTACK h);
9326 returns the file descriptor of the association. Use this when
9327 file-level operations on the endpoint are required
9328 (<function>select(2)</function> operations, specifically).
9332 <sect1 id="comstack.client">
9333 <title>Client Side</title>
9335 int cs_connect(COMSTACK handle, void *address);
9338 Initiate a connection with the target at <literal>address</literal>
9339 (more on addresses below). The function will return 0 on success, and 1 if
9340 the operation does not complete immediately (this will only
9341 happen on a nonblocking endpoint). In this case, use
9342 <function>cs_rcvconnect</function> to complete the operation,
9343 when <function>select(2)</function> or <function>poll(2)</function>
9344 reports input pending on the association.
9347 int cs_rcvconnect(COMSTACK handle);
9350 Complete a connect operation initiated by <function>cs_connect()</function>.
9351 It will return 0 on success; 1 if the operation has not yet completed (in
9352 this case, call the function again later); -1 if an error has occurred.
9355 <sect1 id="comstack.server">
9356 <title>Server Side</title>
9358 To establish a server under the <application>inetd</application>
9362 COMSTACK cs_createbysocket(int socket, CS_TYPE type, int blocking,
9366 The <literal>socket</literal> parameter is an established socket (when
9367 your application is invoked from <application>inetd</application>, the
9368 socket will typically be 0.
9369 The following parameters are identical to the ones for
9370 <function>cs_create</function>.
9373 int cs_bind(COMSTACK handle, void *address, int mode)
9376 Binds a local address to the endpoint. Read about addresses below. The
9377 <literal>mode</literal> parameter should be either
9378 <literal>CS_CLIENT</literal> or <literal>CS_SERVER</literal>.
9381 int cs_listen(COMSTACK handle, char *addr, int *addrlen);
9384 Call this to process incoming events on an endpoint that has been
9385 bound in listening mode. It will return 0 to indicate that the connect
9386 request has been received, 1 to signal a partial reception, and -1 to
9387 indicate an error condition.
9390 COMSTACK cs_accept(COMSTACK handle);
9393 This finalizes the server-side association establishment, after
9394 cs_listen has completed successfully. It returns a new connection
9395 endpoint, which represents the new association. The application will
9396 typically wish to fork off a process to handle the association at this
9397 point, and continue listen for new connections on the old
9398 <literal>handle</literal>.
9401 You can use the call
9404 const char *cs_addrstr(COMSTACK);
9407 on an established connection to retrieve the host-name of the remote host.
9411 You may need to use this function with some care if your
9412 name server service is slow or unreliable.
9416 <sect1 id="comstack.addresses">
9417 <title>Addresses</title>
9419 The low-level format of the addresses are different depending on the
9420 mode of communication you have chosen. A function is provided by each
9421 of the lower layers to map a user-friendly string-form address to the
9422 binary form required by the lower layers.
9425 void *cs_straddr(COMSTACK handle, const char *str);
9428 The format for TCP/IP and SSL addresses is:
9431 <host> [ ':' <portnum> ]
9434 The <literal>hostname</literal> can be either a domain name or an
9435 IP address. The port number, if omitted, defaults to 210.
9438 For TCP/IP and SSL, the special hostnames <literal>@</literal>,
9439 maps to <literal>IN6ADDR_ANY_INIT</literal> with
9440 IPV4 binding as well (bindv6only=0),
9441 The special hostname <literal>@4</literal> binds to
9442 <literal>INADDR_ANY</literal> (IPV4 only listener).
9443 The special hostname <literal>@6</literal> binds to
9444 <literal>IN6ADDR_ANY_INIT</literal> with bindv6only=1 (IPV6 only listener).
9447 For UNIX sockets, the format of an address is the socket filename.
9450 When a connection has been established, you can use
9453 const char *cs_addrstr(COMSTACK h);
9456 to retrieve the host name of the peer system. The function returns
9457 a pointer to a static area, which is overwritten on the next call
9461 A fairly recent addition to the &comstack; module is the utility
9465 COMSTACK cs_create_host (const char *str, int blocking, void **vp);
9468 which is just a wrapper for <function>cs_create</function> and
9469 <function>cs_straddr</function>. The <parameter>str</parameter>
9470 is similar to that described for <function>cs_straddr</function>
9471 but with a prefix denoting the &comstack; type. Prefixes supported
9472 are <literal>tcp:</literal> and <literal>unix:</literal> and
9473 <literal>ssl:</literal> for TCP/IP and UNIX and SSL respectively.
9474 If no prefix is given, then TCP/IP is used.
9475 The <parameter>blocking</parameter> is passed to
9476 function <function>cs_create</function>. The third parameter
9477 <parameter>vp</parameter> is a pointer to &comstack; stack type
9479 Parameter <parameter>vp</parameter> is reserved for future use.
9480 Set it to <literal>NULL</literal>.
9483 <sect1 id="comstack.ssl">
9487 void *cs_get_ssl(COMSTACK cs);
9489 Returns the SSL handle, <literal>SSL *</literal> for comstack. If comstack
9490 is not of type SSL, then NULL is returned.
9494 int cs_set_ssl_ctx(COMSTACK cs, void *ctx);
9496 Sets SSL context for comstack. The parameter is expected to be of type
9497 <literal>SSL_CTX *</literal>. This function should be called just
9498 after comstack has been created (before connect, bind, etc).
9499 This function returns 1 for success; 0 for failure.
9503 int cs_set_ssl_certificate_file(COMSTACK cs, const char *fname);
9505 Sets SSL certificate for comstack as a PEM file. This function
9506 returns 1 for success; 0 for failure.
9510 int cs_get_ssl_peer_certificate_x509(COMSTACK cs, char **buf, int *len);
9512 This function returns the peer certificate. If successful,
9513 <literal>*buf</literal> and <literal>*len</literal> holds
9514 X509 buffer and length respectively. Buffer should be freed
9515 with <literal>xfree</literal>. This function returns 1 for success;
9519 <sect1 id="comstack.diagnostics">
9520 <title>Diagnostics</title>
9522 All functions return -1 if an error occurs. Typically, the functions
9523 will return 0 on success, but the data exchange functions
9524 (<function>cs_get</function>, <function>cs_put</function>,
9525 <function>cs_more</function>) follow special rules. Consult their
9529 The error code for the COMSTACK can be retrieved using C macro
9530 <function>cs_errno</function> which will return one
9531 of the error codes <literal>CSYSERR</literal>,
9532 <literal>CSOUTSTATE</literal>,
9533 <literal>CSNODATA</literal>, ...
9536 int cs_errno(COMSTACK handle);
9539 You can the textual representation of the error code
9540 by using <function>cs_errmsg</function>, which
9541 works like <function>strerror(3)</function>.
9544 const char *cs_errmsg(int n);
9547 It is also possible to get straight to the textual representation
9548 without the error code, by using
9549 <function>cs_strerror</function>.
9552 const char *cs_strerror(COMSTACK h);
9555 <sect1 id="comstack.summary">
9556 <title>Summary and Synopsis</title>
9558 #include <yaz/comstack.h>
9560 #include <yaz/tcpip.h> /* this is for TCP/IP and SSL support */
9561 #include <yaz/unix.h> /* this is for UNIX socket support */
9563 COMSTACK cs_create(CS_TYPE type, int blocking, int protocol);
9565 COMSTACK cs_createbysocket(int s, CS_TYPE type, int blocking,
9567 COMSTACK cs_create_host(const char *str, int blocking,
9570 int cs_bind(COMSTACK handle, int mode);
9572 int cs_connect(COMSTACK handle, void *address);
9574 int cs_rcvconnect(COMSTACK handle);
9576 int cs_listen(COMSTACK handle);
9578 COMSTACK cs_accept(COMSTACK handle);
9580 int cs_put(COMSTACK handle, char *buf, int len);
9582 int cs_get(COMSTACK handle, char **buf, int *size);
9584 int cs_more(COMSTACK handle);
9586 void cs_close(COMSTACK handle);
9588 int cs_look(COMSTACK handle);
9590 void *cs_straddr(COMSTACK handle, const char *str);
9592 const char *cs_addrstr(COMSTACK h);
9597 <chapter id="future">
9598 <title>Future Directions</title>
9600 We have a new and better version of the front-end server on the drawing
9601 board. Resources and external commitments will govern when we'll be
9602 able to do something real with it. Features should include greater
9603 flexibility, greater support for access/resource control, and easy
9604 support for Explain (possibly with Zebra as an extra database engine).
9607 &yaz; is a BER toolkit and as such should support all protocols
9608 out there based on that. We'd like to see running ILL applications.
9609 It shouldn't be that hard. Another thing that would be interesting is
9610 LDAP. Maybe a generic framework for doing IR using both LDAP and
9611 Z39.50 transparently.
9614 The SOAP implementation is incomplete. In the future we hope
9615 to add more features to it. Perhaps make a WSDL/XML Schema compiler.
9616 The authors of libxml2 are already working on XML Schema / RelaxNG
9617 compilers so this may not be too hard.
9620 It would be neat to have a proper module mechanism for the Generic
9621 Frontend Server so that backend would be dynamically
9622 loaded (as shared objects / DLLs).
9625 Other than that, &yaz; generally moves in the directions which appear to
9626 make the most people happy (including ourselves, as prime users of the
9627 software). If there's something you'd like to see in here, then drop
9628 us a note and let's see what we can come up with.
9631 <reference id="reference">
9632 <title>Reference</title>
9633 <partintro id="reference-introduction">
9635 The material in this chapter is drawn directly from the individual
9641 <appendix id="list-oids">
9642 <title>List of Object Identifiers</title>
9644 These is a list of object identifiers that are built into YAZ.
9648 <appendix id="bib1-diagnostics">
9649 <title>Bib-1 diagnostics</title>
9651 List of Bib-1 diagnostics that are known to YAZ.
9655 <appendix id="sru-diagnostics">
9656 <title>SRU diagnostics</title>
9658 List of SRU diagnostics that are known to YAZ.
9662 <appendix id="license">
9663 <title>License</title>
9664 <sect1 id="license.indexdata">
9665 <title>Index Data Copyright</title>
9667 Copyright © ©right-year; Index Data.
9670 All rights reserved.
9673 Redistribution and use in source and binary forms, with or without
9674 modification, are permitted provided that the following conditions are met:
9679 Redistributions of source code must retain the above copyright
9680 notice, this list of conditions and the following disclaimer.
9685 Redistributions in binary form must reproduce the above copyright
9686 notice, this list of conditions and the following disclaimer in the
9687 documentation and/or other materials provided with the distribution.
9692 Neither the name of Index Data nor the names of its contributors
9693 may be used to endorse or promote products derived from this
9694 software without specific prior written permission.
9699 THIS SOFTWARE IS PROVIDED BY INDEX DATA ``AS IS'' AND ANY
9700 EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
9701 WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
9702 DISCLAIMED. IN NO EVENT SHALL INDEX DATA BE LIABLE FOR
9703 ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
9704 DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
9705 SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
9706 CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
9707 LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
9708 OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
9713 <appendix id="indexdata">
9714 <title>About Index Data</title>
9716 Index Data is a consulting and software-development enterprise that
9717 specializes in library and information management systems. Our
9718 interests and expertise span a broad range of related fields, and one
9719 of our primary, long-term objectives is the development of a powerful
9720 information management
9721 system with open network interfaces and hyper-media capabilities.
9723 We make this software available free of charge, on a fairly unrestrictive
9724 license; as a service to the networking community, and to further the
9725 development of quality software for open network communication.
9727 We'll be happy to answer questions about the software, and about ourselves
9733 <street>Amagerfælledvej 56</street>
9734 <postcode>2300 Copenhagen S</postcode>
9735 <country>Denmark</country>
9736 Email <email>info@indexdata.dk</email>
9740 The Hacker's Jargon File has the following to say about the
9742 prefix "YA" in the name of a software product.
9746 Yet Another. adj. 1. Of your own work: A
9747 humorous allusion often used in titles to acknowledge that the
9748 topic is not original, though the content is. As in "Yet Another
9749 AI Group" or "Yet Another Simulated Annealing Algorithm".
9751 others' work: Describes something of which there are already far
9756 <appendix id="credits">
9757 <title>Credits</title>
9759 This appendix lists individuals that have contributed in the development
9760 of &yaz;. Some have contributed with code, while others have provided bug
9761 fixes or suggestions. If we're missing somebody, of if you, for
9762 whatever reason, don't like to be listed here, let us know.
9772 Morten Bøgeskov
9793 Mads Bondo Dydensborg
9802 Morten Garkier Hendriksen
9859 Tom André Øverland
9865 <!-- Keep this comment at the end of the file
9868 nxml-child-indent: 1