1 <?xml version="1.0" standalone="no"?>
2 <!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.4//EN"
3 "http://www.oasis-open.org/docbook/xml/4.4/docbookx.dtd"
5 <!ENTITY % local SYSTEM "local.ent">
7 <!ENTITY % entities SYSTEM "entities.ent">
9 <!ENTITY % idcommon SYSTEM "common/common.ent">
14 <title>YAZ User's Guide and Reference</title>
16 <author><firstname>Sebastian</firstname><surname>Hammer</surname></author>
17 <author><firstname>Adam</firstname><surname>Dickmeiss</surname></author>
18 <author><firstname>Mike</firstname><surname>Taylor</surname></author>
19 <author><firstname>Heikki</firstname><surname>Levanto</surname></author>
20 <author><firstname>Dennis</firstname><surname>Schafroth</surname></author>
22 <releaseinfo>&version;</releaseinfo>
24 <year>©right-year;</year>
25 <holder>Index Data</holder>
29 This document is the programmer's guide and reference to the &yaz;
30 package version &version;. &yaz; is a compact toolkit that provides
31 access to the Z39.50 and SRU/Solr protocols, as well as a set of
32 higher-level tools for implementing the server and client
34 The documentation can be used on its own, or as a reference when
35 looking at the example applications provided with the package.
40 <imagedata fileref="common/id.png" format="PNG"/>
43 <imagedata fileref="common/id.eps" format="EPS"/>
48 <chapter id="introduction">
49 <title>Introduction</title>
51 &yaz; is a C/C++ library for information retrieval applications
52 using the Z39.50/SRU/Solr protocols for information retrieval.
60 <ulink url="&url.z39.50;">Z39.50</ulink> version 3 support.
61 Amendments and Z39.50-2002 revision is supported.
67 <ulink url="&url.sru;">SRU GET/POST/SOAP</ulink>
68 version 1.1, 1.2 and 2.0 (over HTTP and HTTPS).
73 Includes BER encoders/decoders for the
74 <ulink url="&url.ill;">ISO ILL</ulink>
81 <ulink url="&url.solr;">Apache Solr</ulink> Web Service version 1.4.x
87 Supports the following transports: BER over TCP/IP
88 (<ulink url="&url.ber.over.tcpip;">RFC1729</ulink>),
89 BER over unix local socket, and
90 <ulink url="&url.http.1.1;">HTTP 1.1</ulink>.
95 Secure Socket Layer support using
96 <ulink url="&url.gnutls;">GnuTLS</ulink>.
97 If enabled, &yaz; uses HTTPS transport (for SOAP) or
98 "Secure BER" (for Z39.50).
104 <ulink url="&url.zoom;">ZOOM</ulink> C API implementing
105 Z39.50, SRU and Solr Web Service.
110 The &yaz; library offers a set of useful utilities
111 related to the protocols, such as MARC (ISO2709) parser,
112 CCL (ISO8777) parser,
113 <ulink url="&url.cql;">CQL</ulink>
114 parser, memory management routines, character set conversion.
119 Portable code. &yaz; compiles out-of-the box on most Unixes and
120 on Windows using Microsoft Visual C++.
125 Fast operation. The C based BER encoders/decoders as well
126 as the server component of &yaz; is very fast.
131 Liberal license that allows for commercial use of &yaz;.
137 <sect1 id="introduction.reading">
138 <title>Reading this Manual</title>
140 Most implementors only need to read a fraction of the
141 material in this manual, so a quick walkthrough of the chapters
147 <xref linkend="installation"/> contains installation
148 instructions for &yaz;. You don't need to read this
149 if you expect to download &yaz; binaries.
150 However, the chapter contains information about how
151 to make <emphasis>your</emphasis> application link
157 <xref linkend="zoom"/> describes the ZOOM API of &yaz;.
158 This is definitely worth reading if you wish to develop a Z39.50/SRU
164 <xref linkend="server"/> describes the generic frontend server
165 and explains how to develop server Z39.50/SRU applications for &yaz;.
166 Obviously worth reading if you're to develop a server.
171 <xref linkend="yaz-client"/> describes how to use the &yaz; Z39.50
172 client. If you're a developer and wish to test your server
173 or a server from another party, you might find this chapter
179 <xref linkend="asn"/> documents the most commonly used Z39.50
180 C data structures offered by the &yaz; API. Client
181 developers using ZOOM and non-Z39.50 implementors may skip this.
186 <xref linkend="soap"/> describes how SRU and SOAP is used
187 in &yaz;. Only if you're developing SRU applications
188 this section is a must.
193 <xref linkend="tools"/> contains sections for the various
194 tools offered by &yaz;. Scan through the material quickly
195 and see what's relevant to you! SRU implementors
196 might find the <link linkend="cql">CQL</link> section
202 <xref linkend="odr"/> goes through the details of the
203 ODR module which is the work horse that encodes and decodes
204 BER packages. Implementors using ZOOM only, do <emphasis>not</emphasis>
206 Most other Z39.50 implementors only need to read the first two
207 sections (<xref linkend="odr.introduction"/> and
208 <xref linkend="odr.use"/>).
213 <xref linkend="comstack"/> describes the network layer module
214 COMSTACK. Implementors using ZOOM or the generic frontend server
215 may skip this. Others, presumably, handling client/server
216 communication on their own should read this.
221 <sect1 id="introduction.api">
222 <title>The API</title>
224 The <ulink url="&url.yaz;">&yaz;</ulink>
225 toolkit offers several different levels of access to the
226 <ulink url="&url.z39.50;">ISO23950/Z39.50</ulink>,
227 <ulink url="&url.ill;">ILL</ulink> and
228 <ulink url="&url.sru;">SRU</ulink>
230 The level that you need to use depends on your requirements, and
231 the role (server or client) that you want to implement.
232 If you're developing a client application you should consider the
233 <link linkend="zoom">ZOOM</link> API.
234 It is, by far, the easiest way to develop clients in C.
235 Server implementers should consider the
236 <link linkend="server">generic frontend server</link>.
237 None of those high-level APIs support the whole protocol, but
238 they do include most facilities used in existing Z39.50 applications.
241 If you're using 'exotic' functionality (meaning anything not included in
242 the high-level APIs), developing non-standard extensions to Z39.50 or
243 you're going to develop an ILL application you'll have to learn the lower
247 The YAZ toolkit modules are shown in figure <xref linkend="yaz.layer"/>.
249 <figure id="yaz.layer">
250 <title>YAZ layers</title>
253 <imagedata fileref="apilayer.png" format="PNG"/>
256 <imagedata fileref="apilayer.eps" format="EPS"/>
261 There are four layers.
264 <para>A client or server application (or both).
265 This layer includes ZOOM and the generic frontend server.
270 The second layer provides a C represenation of the
271 protocol units (packages) for Z39.50 ASN.1, ILL ASN.1,
277 The third layer encodes and decodes protocol data units to
278 simple packages (buffer with certain length). The &odr; module
279 encodes and decodes BER whereas the HTTP modules encodes and
280 decodes HTTP ruquests/responses.
285 The lowest layer is &comstack; which exchanges the encoded packages
286 with a peer process over a network.
292 The &asn; module represents the ASN.1 definition of
293 the Z39.50 protocol. It establishes a set of type and
294 structure definitions, with one structure for each of the top-level
295 PDUs, and one structure or type for each of the contained ASN.1 types.
296 For primitive types, or other types that are defined by the ASN.1
297 standard itself (such as the EXTERNAL type), the C representation is
298 provided by the &odr; (Open Data Representation) subsystem.
301 &odr; is a basic mechanism for representing an
302 ASN.1 type in the C programming language, and for implementing BER
303 encoders and decoders for values of that type. The types defined in
304 the &asn; module generally have the prefix <literal>Z_</literal>, and
305 a suffix corresponding to the name of the type in the ASN.1
306 specification of the protocol (generally Z39.50-1995). In the case of
307 base types (those originating in the ASN.1 standard itself), the prefix
308 <literal>Odr_</literal> is sometimes seen. Either way, look for
309 the actual definition in either <filename>z-core.h</filename> (for the types
310 from the protocol), <filename>odr.h</filename> (for the primitive ASN.1
312 The &asn; library also provides functions (which are, in turn,
313 defined using &odr; primitives) for encoding and decoding data values.
314 Their general form is
316 <funcprototype><funcdef>int <function>z_<replaceable>xxx</replaceable></function></funcdef>
317 <paramdef>ODR <parameter>o</parameter></paramdef>
318 <paramdef>Z_<replaceable>xxx</replaceable> **<parameter>p</parameter></paramdef>
319 <paramdef>int <parameter>optional</parameter></paramdef>
320 <paramdef>const char *<parameter>name</parameter></paramdef>
323 (note the lower-case "z" in the function name)
327 If you are using the premade definitions of the &asn; module, and you
328 are not adding a new protocol of your own, the only parts of &odr; that you
329 need to worry about are documented in
330 <xref linkend="odr.use"/>.
334 When you have created a BER-encoded buffer, you can use the &comstack;
335 subsystem to transmit (or receive) data over the network. The &comstack;
336 module provides simple functions for establishing a connection
337 (passively or actively, depending on the role of your application),
338 and for exchanging BER-encoded PDUs over that connection. When you
339 create a connection endpoint, you need to specify what transport to
340 use (TCP/IP, SSL or UNIX sockets).
341 For the remainder of the connection's lifetime, you don't have
342 to worry about the underlying transport protocol at all - the &comstack;
343 will ensure that the correct mechanism is used.
346 We call the combined interfaces to &odr;, &asn;, and &comstack; the service
347 level API. It's the API that most closely models the Z39.50
348 service/protocol definition, and it provides unlimited access to all
349 fields and facilities of the protocol definitions.
352 The reason that the &yaz; service-level API is a conglomerate of the
353 APIs from three different submodules is twofold. First, we wanted to allow
354 the user a choice of different options for each major task. For instance,
355 if you don't like the protocol API provided by &odr;/&asn;, you
356 can use SNACC or BERUtils instead, and still have the benefits of the
357 transparent transport approach of the &comstack; module. Secondly,
358 we realize that you may have to fit the toolkit into an existing
359 event-processing structure, in a way that is incompatible with
360 the &comstack; interface or some other part of &yaz;.
364 <chapter id="installation">
365 <title>Compilation and Installation</title>
366 <sect1 id="installation-introduction">
367 <title>Introduction</title>
369 The latest version of the software will generally be found at:
372 <ulink url="&url.yaz.download;"/>
375 We have tried our best to keep the software portable, and on many
376 platforms, you should be able to compile everything with little or
380 The software is regularly tested on
381 <ulink url="&url.debian;">Debian GNU/Linux</ulink>,
382 <ulink url="&url.centos;">CentOS</ulink>,
383 <ulink url="&url.ubuntu;">Ubuntu Linux</ulink>,
384 <ulink url="&url.freebsd;">FreeBSD (i386)</ulink>,
385 <ulink url="&url.macosx;">MAC OSX</ulink>,
386 <ulink url="&url.solaris;">Solaris</ulink>,
387 Windows 7, Windows XP.
390 Some versions have be known to work on HP/UX,
391 DEC Unix, <ulink url="&url.netbsd;">NetBSD</ulink>,
392 <ulink url="&url.openbsd;">OpenBSD</ulink>,
394 Data General DG/UX (with some CFLAGS tinkering),
395 SGI/IRIX, DDE Supermax, Apple Macintosh (using the Codewarrior programming
396 environment and the GUSI socket libraries),
400 If you move the software to other platforms, we'd be grateful if you'd
401 let us know about it. If you run into difficulties, we will try to help
402 if we can, and if you solve the problems, we would be happy to include
403 your fixes in the next release. So far, we have mostly avoided
404 <literal>#ifdefs</literal> for individual platforms, and we'd
405 like to keep it that way as far as it makes sense.
408 We maintain a mailing-list for the purpose of announcing new releases and
409 bug-fixes, as well as general discussion. Subscribe by
411 <ulink url="&url.yaz.mailinglist;">here</ulink>.
412 General questions and problems can be directed at
413 <ulink url="&url.yaz.mail;"/>, or the address given at the top of
417 <sect1 id="installation.unix"><title>UNIX</title>
420 <ulink url="&url.debian;">Debian GNU/Linux</ulink> (i386 and amd64),
421 <ulink url="&url.ubuntu;">Ubuntu</ulink> (i386 and amd64)
423 <ulink url="&url.centos;">CentOS</ulink> (amd64 only) packages for &yaz;.
424 You should be able to create packages for other CPUs by building
425 them from the source package.
428 YAZ is also part of several packages repositories. Some of them are
433 Solaris CSW: <ulink url="http://www.opencsw.org/packages/yaz/"/>
438 Solaris: <ulink url="http://unixpackages.com"/>
443 FreeBSD: <ulink url="http://www.freshports.org/net/yaz"/>
448 Debian: <ulink url="http://packages.debian.org/search?keywords=yaz"/>
453 Ubuntu: <ulink url="https://launchpad.net/ubuntu/+source/yaz"/>
459 <ulink url="http://ftp.netbsd.org/pub/pkgsrc/current/pkgsrc/net/yaz/README.html"/>
463 <sect2 id="installation.source.unix">
464 <title>Compiling from source on Unix</title>
466 Note that if your system doesn't have a native ANSI C compiler, you may
467 have to acquire one separately. We recommend
468 <ulink url="&url.gcc;">GCC</ulink>.
471 If you wish to use character set conversion facilities in &yaz; or if you
472 are compiling &yaz; for use with Zebra it is a good idea to ensure that
473 the iconv library is installed. Some Unixes today already have it
475 <ulink url="&url.libiconv;">GNU libiconv</ulink>.
478 YAZ 3.0.16 and later includes a wrapper for the
479 <ulink url="&url.icu;">ICU</ulink>
480 (International Components for Unicode).
481 In order to use this, the developer version of the ICU library
482 must be available. ICU support is recommended for applications
483 such as Pazpar2 and Zebra.
486 The <ulink url="&url.libxslt;">libxslt</ulink>,
487 <ulink url="&url.libxml2;">libxml2</ulink> librararies are required
488 if &yaz; is to support SRU/Solr.
489 These libraries are very portable and should compile out-of-the
490 box on virtually all Unix platforms. It is available in binary
491 forms for Linux and others.
495 <ulink url="&url.autoconf;">Autoconf</ulink>,
496 <ulink url="&url.automake;">Automake</ulink> and
497 <ulink url="&url.libtool;">Libtool</ulink>
498 are used to generate Makefiles and configure &yaz; for the system.
499 You do <emphasis>not</emphasis> need these tools unless you're using the
500 Git version of &yaz;.
503 The CQL parser for &yaz; is built using
504 GNU <ulink url="&url.bison;">Bison</ulink>.
505 This tool is only needed if you're using the Git version of &yaz;.
508 &yaz; includes a tiny ASN.1 compiler. This compiler is
509 written in <ulink url="&url.tcl;">Tcl</ulink>.
510 But as for Bison you do not need it unless you're using Git
511 version of &yaz; or you're using the compiler to build your own codecs
515 Generally it should be sufficient to run configure without options,
522 The configure script attempts to use use the C compiler specified by
523 the <literal>CC</literal> environment variable. If not set, GNU C will be
524 used if it is available. The <literal>CFLAGS</literal> environment
525 variable holds options to be passed to the C compiler. If you're using
526 Bourne-compatible shell you may pass something like this to use a
527 particular C compiler with optimization enabled:
530 CC=/opt/ccs/bin/cc CFLAGS=-O ./configure
533 To customize &yaz;, the configure script also accepts a set of options.
534 The most important are:
538 <literal>--prefix</literal>=<replaceable>prefix</replaceable>
541 <para>Specifies installation prefix for &yaz;. This is
542 only needed if you run <literal>make install</literal> later to
543 perform a "system" installation. The prefix is
544 <literal>/usr/local</literal> if not specified.
550 <literal>--enable-tcpd</literal>
553 <para>The front end server will be built using Wietse's
554 <ulink url="&url.tcpwrapper;">TCP wrapper library</ulink>.
555 It allows you to allow/deny clients depending on IP number.
556 The TCP wrapper library is often used in GNU/Linux and
560 <refentrytitle>hosts_access</refentrytitle>
561 <manvolnum>5</manvolnum>
565 <refentrytitle>tcpd</refentrytitle>
566 <manvolnum>8</manvolnum>
573 <literal>--enable-threads</literal>
576 <para>&yaz; will be built using POSIX threads.
577 Specifically, <constant>_REENTRANT</constant> will be defined during
584 <literal>--disable-shared</literal>
587 <para>The make process will not create shared
588 libraries (also known as shared objects <filename>.so</filename>).
589 By default, shared libraries are created -
590 equivalent to <literal>--enable-shared</literal>.
596 <literal>--disable-shared</literal>
599 <para>The make process will not create
600 static libraries (<filename>.a</filename>).
601 By default, static libraries are created -
602 equivalent to <literal>--enable-static</literal>.
608 <literal>--with-iconv</literal>[=<replaceable>prefix</replaceable>]
611 <para>Compile &yaz; with iconv library in directory
612 <replaceable>prefix</replaceable>. By default configure will
613 search for iconv on the system. Use this option if it
614 doesn't find iconv. Alternatively,
615 <literal>--without-iconv</literal>, can be used to force &yaz;
622 <literal>--with-xslt</literal>[=<replaceable>prefix</replaceable>]
625 <para>Compile &yaz; with
626 <ulink url="&url.libxslt;">libxslt</ulink> in directory
627 <replaceable>prefix</replaceable>.
628 Use this option if you want XSLT and XML support.
629 By default, configure will
630 search for libxslt on the system. Use this option if
631 libxslt is not found automatically. Alternatively,
632 <literal>--without-xslt</literal>, can be used to force &yaz;
639 <literal>--with-xml2</literal>[=<replaceable>prefix</replaceable>]
642 <para>Compile &yaz; with
643 <ulink url="&url.libxml2;">libxml2</ulink> in directory
644 <replaceable>prefix</replaceable>.
645 Use this option if you want &yaz; to use XML and support SRU/Solr.
646 By default, configure will
647 search for libxml2 on the system. Use this option if
648 libxml2 is not found automatically. Alternatively,
649 <literal>--without-xml2</literal>, can be used to force &yaz;
653 Note that option <literal>--with-xslt</literal>
654 also enables libxml2.
660 <literal>--with-gnutls</literal>[=<replaceable>prefix</replaceable>]
663 <para>&yaz; will be linked with the GNU TLS libraries and
664 an SSL COMSTACK will be provided. By default configure enables
665 SSL support for YAZ if the GNU TLS development libraries are found
672 <literal>--with-icu</literal>[=<replaceable>prefix</replaceable>]
675 <para>&yaz; will be linked the
676 <ulink url="&url.icu;">ICU</ulink> library in the prefix if given.
677 If prefix is not given, the libraries exposed by the script
678 <application>icu-config</application> will be used if found.
685 <literal>--with-libgcrypt</literal>[=<replaceable>prefix</replaceable>]
688 <para>&yaz; will be linked with
689 <ulink url="&url.libgcrypt;">Libgcrypt</ulink> in the prefix if given.
690 If prefix is not given, the libraries exposed by the script
691 <application>libgcrypt-config</application> will be used if found.
697 <literal>--with-memcached</literal>
700 <para>&yaz; will be linked with
701 <ulink url="&url.libmemcached;">libMemcached</ulink> to allow
702 for result-set caching for ZOOM.
703 The prefix can not be given. Note that YAZ will only search
704 for libMemcached if Libgcrypt is also enabled.
705 Note that 0.40 of libmemcached is required.
711 <literal>--with-redis</literal>
714 <para>&yaz; will be linked with the hiredis C library
715 to allow for result-set caching for ZOOM on a
716 <ulink url="&url.redis;">redis</ulink> server.
717 The prefix can not be given. Note that YAZ will only search
718 for hiredis if Libgcrypt is also enabled.
726 When configured, build the software by typing:
732 The following files are generated by the make process:
735 <term><filename>src/libyaz.la</filename></term>
737 Main &yaz; library. This is no ordinary library. It's
739 By default, &yaz; creates a static library in
740 <filename>lib/.libs/libyaz.a</filename>.
744 <term><filename>src/libyaz_server.la</filename></term>
746 Generic Frontend server. This is an add-on for libyaz.la.
747 Code in this library uses POSIX threads functions - if POSIX
748 threads are available on the platform.
752 <term><filename>src/libyaz_icu.la</filename></term>
754 Functions that wrap the ICU library.
758 <term><filename>ztest/yaz-ztest</filename></term>
759 <listitem><para>Test Z39.50 server.
763 <term><filename>client/yaz-client</filename></term>
764 <listitem><para>Z39.50 client for testing the protocol.
765 See chapter <link linkend="yaz-client">
766 YAZ client</link> for more information.
770 <term><filename>util/yaz-config</filename></term>
771 <listitem><para>A Bourne-shell script, generated by configure, that
772 specifies how external applications should compile - and link with
777 <term><filename>util/yaz-asncomp</filename></term>
778 <listitem><para>The ASN.1 compiler for &yaz;. Requires the
779 Tcl Shell, <application>tclsh</application>, in
780 <literal>PATH</literal> to operate.
784 <term><filename>util/yaz-iconv</filename></term>
785 <listitem><para>This program converts data in one character set to
786 another. This command exercises the YAZ character set
791 <term><filename>util/yaz-marcdump</filename></term>
792 <listitem><para>This program parses ISO2709 encoded MARC records
793 and prints them in line-format or XML.
797 <term><filename>util/yaz-icu</filename></term>
798 <listitem><para>This program exposes the ICU wrapper library if that
799 is enabled for YAZ. Only if ICU is available this program is
804 <term><filename>util/yaz-url</filename></term>
805 <listitem><para>This program is a simple HTTP page fetcher ala
810 <term><filename>zoom/zoomsh</filename></term>
812 A simple shell implemented on top of the
813 <link linkend="zoom">ZOOM</link> functions.
814 The shell is a command line application that allows you to enter
815 simple commands to perform ZOOM operations.
819 <term><filename>zoom/zoomtst1</filename>,
820 <filename>zoom/zoomtst2</filename>, ..</term>
822 Several small applications that demonstrates the ZOOM API.
828 If you wish to install &yaz; in system directories
829 <filename>/usr/local/bin</filename>,
830 <filename>/usr/local/lib</filename> .. etc, you can type:
836 You probably need to have root access in order to perform this.
837 You must specify the <literal>--prefix</literal> option for configure if
838 you wish to install &yaz; in other directories than the default
839 <filename>/usr/local/</filename>.
842 If you wish to perform an un-installation of &yaz;, use:
848 This will only work if you haven't reconfigured &yaz; (and therefore
849 changed installation prefix). Note that uninstall will not
850 remove directories created by make install, e.g.
851 <filename>/usr/local/include/yaz</filename>.
854 <sect2 id="installation-linking-yaz-unix">
855 <title>How to make apps using YAZ on UNIX</title>
857 This section describes how to compile - and link your own
858 applications using the &yaz; toolkit.
859 If you're used to Makefiles this shouldn't be hard. As for
860 other libraries you have used before, you need to set a proper include
861 path for your C/C++ compiler and specify the location of
862 &yaz; libraries. You can do it by hand, but generally we suggest
863 you use the <filename>yaz-config</filename> that is generated
864 by <filename>configure</filename>. This is especially
865 important if you're using the threaded version of &yaz; which
866 require you to pass more options to your linker/compiler.
869 The <filename>yaz-config</filename> script accepts command line
870 options that makes the <filename>yaz-config</filename> script print
871 options that you should use in your make process.
872 The most important ones are:
873 <literal>--cflags</literal>, <literal>--libs</literal>
874 which prints C compiler flags, and linker flags respectively.
877 A small and complete <literal>Makefile</literal> for a C
878 application consisting of one source file,
879 <filename>myprog.c</filename>, may look like this:
881 YAZCONFIG=/usr/local/bin/yaz-config
882 CFLAGS=`$(YAZCONFIG) --cflags`
883 LIBS=`$(YAZCONFIG) --libs`
885 $(CC) $(CFLAGS) -o myprog myprog.o $(LIBS)
889 The CFLAGS variable consists of a C compiler directive that will set
890 the include path to the <emphasis>parent</emphasis> directory
891 of <filename>yaz</filename>. That is, if &yaz; header files were
892 installed in <filename>/usr/local/include/yaz</filename>,
893 then include path is set to <filename>/usr/local/include</filename>.
894 Therefore, in your applications you should use
896 #include <yaz/proto.h>
898 and <emphasis>not</emphasis>
900 #include <proto.h>
904 For Libtool users, the <filename>yaz-config</filename> script provides
905 a different variant of option <literal>--libs</literal>, called
906 <literal>--lalibs</literal> that returns the name of the
907 Libtool archive(s) for &yaz; rather than the ordinary ones.
910 For applications using the threaded version of &yaz;,
911 specify <literal>threads</literal> after the
912 other options. When <literal>threads</literal> is given,
913 more flags and linker flags will be printed by
914 <filename>yaz-config</filename>. If our previous example was
915 using threads, you'd have to modify the lines that set
916 <literal>CFLAGS</literal> and <literal>LIBS</literal> as
919 CFLAGS=`$(YAZCONFIG) --cflags threads`
920 LIBS=`$(YAZCONFIG) --libs threads`
922 There is no need specify POSIX thread libraries in your Makefile.
923 The <literal>LIBS</literal> variable includes that as well.
927 <sect1 id="installation.win32">
928 <title>Windows</title>
929 <para>The easiest way to install YAZ on Windows is by downloading
931 <ulink url="&url.yaz.download.win32;">here</ulink>.
932 The installer comes with source too - in case you wish to
933 compile YAZ with different compiler options, etc.
936 <sect2 id="installation.win32.source">
937 <title>Compiling from Source on Windows</title>
939 &yaz; is shipped with "makefiles" for the NMAKE tool that comes
940 with <ulink url="&url.vstudio;">
941 Microsoft Visual Studio</ulink>. It has been tested with
942 Microsoft Visual Studio 2013.
945 Start a command prompt and switch the sub directory
946 <filename>WIN</filename> where the file <filename>makefile</filename>
947 is located. Customize the installation by editing the
948 <filename>makefile</filename> file (for example by using notepad).
949 The following summarizes the most important settings in that file:
952 <term><literal>DEBUG</literal></term>
954 If set to 1, the software is
955 compiled with debugging libraries (code generation is
956 multi-threaded debug DLL).
957 If set to 0, the software is compiled with release libraries
958 (code generation is multi-threaded DLL).
962 <term><literal>HAVE_TCL</literal>, <literal>TCL</literal></term>
964 If <literal>HAVE_TCL</literal> is set to 1, nmake will
965 use the ASN.1 compiler (<ulink url="&url.tcl;">Tcl</ulink> based).
966 You must set <literal>TCL</literal> to the full path of the Tcl
967 interpreter. A Windows version of Tcl is part of
968 <ulink url="&url.gitwindows;">Git for Windows</ulink>.
971 If you do not have Tcl installed, set
972 <literal>HAVE_TCL</literal> to 0.
976 <term><literal>HAVE_BISON</literal>,
977 <literal>BISON</literal></term>
979 If GNU Bison is present, you might set <literal>HAVE_BISON</literal>
980 to 1 and specify the Bison executable in <literal>BISON</literal>.
981 Bison is only required if you use the Git version of
982 YAZ or if you modify the grammar for CQL
983 (<filename>cql.y</filename>).
986 A Windows version of GNU Bison is part of
987 <ulink url="&url.gitwindows;">Git for Windows</ulink>.
991 <term><literal>HAVE_ICONV</literal>,
992 <literal>ICONV_DIR</literal></term>
994 If <literal>HAVE_ICONV</literal> is set to 1, YAZ is compiled
995 with iconv support. In this configuration, set
996 <literal>ICONV_DIR</literal> to the iconv source directory.
1000 <term><literal>HAVE_LIBXML2</literal>,
1001 <literal>LIBXML2_DIR</literal></term>
1004 If <literal>HAVE_LIBXML2</literal> is set to 1, YAZ is compiled
1005 with SRU support. In this configuration, set
1006 <literal>LIBXML2_DIR</literal> to the
1007 <ulink url="&url.libxml2;">libxml2</ulink> source directory.
1010 You can get pre-compiled Libxml2+Libxslt DLLs and headers from
1011 <ulink url="&url.libxml2.download.windows;">here</ulink>.
1012 Should you with to compile those libraries yourself, refer to
1013 to <xref linkend="installation.windows.libxml2"/>
1018 <term><literal>HAVE_LIBXSLT</literal>,
1019 <literal>LIBXSLT_DIR</literal></term>
1022 If <literal>HAVE_LIBXSLT</literal> is set to 1, YAZ is compiled
1023 with XSLT support. In this configuration, set
1024 <literal>LIBXSLT_DIR</literal> to the
1025 <ulink url="&url.libxslt;">libxslt</ulink> source directory.
1029 libxslt depends on libxml2.
1035 <term><literal>HAVE_ICU</literal>,
1036 <literal>ICU_DIR</literal></term>
1039 If <literal>HAVE_ICU</literal> is set to 1, YAZ is compiled
1040 with <ulink url="&url.icu;">ICU</ulink> support.
1041 In this configuration, set
1042 <literal>ICU_DIR</literal> to the
1043 <ulink url="&url.icu;">ICU</ulink> source directory.
1050 When satisfied with the settings in the makefile, type
1057 If the <filename>nmake</filename> command is not found on your system
1058 you probably haven't defined the environment variables required to
1059 use that tool. To fix that, find and run the batch file
1060 <filename>vcvars32.bat</filename>. You need to run it from within
1061 the command prompt or set the environment variables "globally";
1062 otherwise it doesn't work.
1066 If you wish to recompile &yaz; - for example if you modify
1067 settings in the <filename>makefile</filename> you can delete
1068 object files, etc by running.
1074 The following files are generated upon successful compilation:
1077 <term><filename>bin/yaz&soversion;.dll</filename> /
1078 <filename>bin/yaz&soversion;d.dll</filename></term>
1080 &yaz; Release/Debug DLL.
1084 <term><filename>lib/yaz&soversion;.lib</filename> /
1085 <filename>lib/yaz&soversion;d.lib</filename></term>
1087 Import library for <filename>yaz&soversion;.dll</filename> /
1088 <filename>yaz&soversion;d.dll</filename>.
1092 <term><filename>bin/yaz_cond&soversion;.dll</filename> /
1093 <filename>bin/yaz_cond&soversion;d.dll</filename></term>
1095 Release/Debug DLL for condition variable utilities (condvar.c).
1099 <term><filename>lib/yaz_cond&soversion;.lib</filename> /
1100 <filename>lib/yaz_cond&soversion;d.lib</filename></term>
1102 Import library for <filename>yaz_cond&soversion;.dll</filename> /
1103 <filename>yaz_cond&soversion;d.dll</filename>.
1107 <term><filename>bin/yaz_icu&soversion;.dll</filename> /
1108 <filename>bin/yaz_icu&soversion;d.dll</filename></term>
1110 Release/Debug DLL for the ICU wrapper utility.
1111 Only build if HAVE_ICU is 1.
1115 <term><filename>lib/yaz_icu&soversion;.lib</filename> /
1116 <filename>lib/yaz_icu&soversion;d.lib</filename></term>
1118 Import library for <filename>yaz_icu&soversion;.dll</filename> /
1119 <filename>yaz_icu&soversion;d.dll</filename>.
1123 <term><filename>bin/yaz-ztest.exe</filename></term>
1125 Z39.50 multi-threaded test/example server. It's a WIN32
1126 console application.
1130 <term><filename>bin/yaz-client.exe</filename></term>
1132 &yaz; Z39.50 client application. It's a WIN32 console application.
1133 See chapter <link linkend="yaz-client">YAZ client</link> for more
1138 <term><filename>bin/yaz-icu.exe</filename></term>
1139 <listitem><para>This program exposes the ICU wrapper library if that
1140 is enabled for YAZ. Only if ICU is available this program is
1145 <term><filename>bin/zoomsh.exe</filename></term>
1147 Simple console application implemented on top of the
1148 <link linkend="zoom">ZOOM</link> functions.
1149 The application is a command line shell that allows you to enter
1150 simple commands to perform ZOOM operations.
1154 <term><filename>bin/zoomtst1.exe</filename>,
1155 <filename>bin/zoomtst2.exe</filename>, ..</term>
1157 Several small applications that demonstrate the ZOOM API.
1164 <sect2 id="installation-linking-yaz-win32">
1165 <title>How to make apps using YAZ on Windows</title>
1167 This section will go though the process of linking your Windows
1168 applications with &yaz;.
1171 Some people are confused by the fact that we use the nmake
1172 tool to build &yaz;. They think they have to do that too - in order
1173 to make their Windows applications work with &yaz;. The good news is that
1174 you don't have to. You can use the integrated environment of
1175 Visual Studio if desired for your own application.
1178 When setting up a project or Makefile you have to set the following:
1181 <term>include path</term>
1183 Set it to the <filename>include</filename> directory of &yaz;.
1187 <term>import library <filename>yaz&soversion;.lib</filename></term>
1189 You must link with this library. It's located in the
1190 sub directory <filename>lib</filename> of &yaz;.
1191 If you want to link with the debug version of &yaz;, you must
1192 link against <filename>yaz&soversion;d.lib</filename> instead.
1196 <term>dynamic link library
1197 <filename>yaz&soversion;.dll</filename>
1200 This DLL must be in your execution path when you invoke
1201 your application. Specifically, you should distribute this
1202 DLL with your application.
1209 <sect2 id="installation.windows.libxml2">
1210 <title>Compiling Libxml2 and Libxslt on windows</title>
1212 Download libxml2 and Libxslt source and unpack it.
1213 In the example below we install Libxml2 2.9.2 and Libxslt 1.1.28
1214 for 32-bit, so we use the destination directories
1215 libxml2.2.9.2.win32 and libxslt-1.1.28.win32 to reflect both
1216 version and architecture.
1219 cscript configure.js prefix=c:\libxml2-2.9.2.win32 iconv=no
1226 There's an error in <filename>configure.js</filename> for Libxml2 2.9.2.
1227 Line 17 should be assigned to <filename>configure.ac</filename>
1228 rather than <filename>configure.in</filename>.
1232 For Libxslt it is similar. We must ensure that compilation of
1233 Libxslt links against the already installed libxml2.
1236 cscript configure.js prefix=c:\libxslt-1.1.28.win32 iconv=no \
1237 lib=c:\libxml2-2.9.2.win32\lib \
1238 include=c:\libxml2-2.9.2.win32\include\libxml2
1248 ### Still to document:
1249 ZOOM_connection_errcode(c)
1250 ZOOM_connection_errmsg(c)
1251 ZOOM_connection_addinfo(c)
1252 ZOOM_connection_addinfo(c)
1253 ZOOM_connection_diagset(c);
1254 ZOOM_connection_save_apdu_wrbuf
1255 ZOOM_diag_str(error)
1256 ZOOM_resultset_record_immediate(s, pos)
1257 ZOOM_resultset_cache_reset(r)
1258 ZOOM_options_set_callback(opt, function, handle)
1259 ZOOM_options_create_with_parent2(parent1, parent2)
1260 ZOOM_options_getl(opt, name, len)
1261 ZOOM_options_setl(opt, name, value, len)
1262 ZOOM_options_get_bool(opt, name, defa)
1263 ZOOM_options_get_int(opt, name, defa)
1264 ZOOM_options_set_int(opt, name, value)
1269 &zoom; is an acronym for 'Z39.50 Object-Orientation Model' and is
1270 an initiative started by Mike Taylor (Mike is from the UK, which
1271 explains the peculiar name of the model). The goal of &zoom; is to
1272 provide a common Z39.50 client API not bound to a particular
1273 programming language or toolkit.
1276 From YAZ version 2.1.12, <ulink url="&url.sru;">SRU</ulink> is supported.
1277 You can make SRU ZOOM connections by specifying scheme
1278 <literal>http://</literal> for the hostname for a connection.
1279 The dialect of SRU used is specified by the value of the
1280 connection's <literal>sru</literal> option, which may be SRU over
1281 HTTP GET (<literal>get</literal>),
1282 SRU over HTTP POST (<literal>post</literal>), (SRU over
1283 SOAP) (<literal>soap</literal>) or <literal>solr</literal>
1284 (<ulink url="&url.solr;">Solr</ulink> Web Service).
1285 Using the facility for embedding options in target strings, a
1286 connection can be forced to use SRU rather the SRW (the default) by
1287 prefixing the target string with <literal>sru=get,</literal>, like this:
1288 <literal>sru=get,http://sru.miketaylor.org.uk:80/sru.pl</literal>
1291 <ulink url="&url.solr;">Solr</ulink> protocol support was added to
1292 YAZ in version 4.1.0, as a dialect of a SRU protocol, since both are
1293 HTTP based protocols.
1296 The lack of a simple Z39.50 client API for &yaz; has become more
1297 and more apparent over time. So when the first &zoom; specification
1299 an implementation for &yaz; was quickly developed. For the first time, it is
1300 now as easy (or easier!) to develop clients as it is to develop
1301 servers with &yaz;. This
1302 chapter describes the &zoom; C binding. Before going further, please
1303 reconsider whether C is the right programming language for the job.
1304 There are other language bindings available for &yaz;, and still
1306 are in active development. See the
1307 <ulink url="&url.zoom;">ZOOM web-site</ulink> for
1311 In order to fully understand this chapter you should read and
1312 try the example programs <literal>zoomtst1.c</literal>,
1313 <literal>zoomtst2.c</literal>, .. in the <literal>zoom</literal>
1317 The C language misses features found in object oriented languages
1318 such as C++, Java, etc. For example, you'll have to manually,
1319 destroy all objects you create, even though you may think of them as
1320 temporary. Most objects have a <literal>_create</literal> - and a
1321 <literal>_destroy</literal> variant.
1322 All objects are in fact pointers to internal stuff, but you don't see
1323 that because of typedefs. All destroy methods should gracefully ignore a
1324 <literal>NULL</literal> pointer.
1327 In each of the sections below you'll find a sub section called
1328 protocol behavior, that describes how the API maps to the Z39.50
1331 <sect1 id="zoom-connections">
1332 <title>Connections</title>
1333 <para>The Connection object is a session with a target.
1336 #include <yaz/zoom.h>
1338 ZOOM_connection ZOOM_connection_new(const char *host, int portnum);
1340 ZOOM_connection ZOOM_connection_create(ZOOM_options options);
1342 void ZOOM_connection_connect(ZOOM_connection c, const char *host,
1344 void ZOOM_connection_destroy(ZOOM_connection c);
1347 Connection objects are created with either function
1348 <function>ZOOM_connection_new</function> or
1349 <function>ZOOM_connection_create</function>.
1350 The former creates and automatically attempts to establish a network
1351 connection with the target. The latter doesn't establish
1352 a connection immediately, thus allowing you to specify options
1353 before establishing network connection using the function
1354 <function>ZOOM_connection_connect</function>.
1355 If the port number, <literal>portnum</literal>, is zero, the
1356 <literal>host</literal> is consulted for a port specification.
1357 If no port is given, 210 is used. A colon denotes the beginning of
1358 a port number in the host string. If the host string includes a
1359 slash, the following part specifies a database for the connection.
1362 You can prefix the host with a scheme followed by colon. The
1363 default scheme is <literal>tcp</literal> (Z39.50 protocol).
1364 The scheme <literal>http</literal> selects SRU/get over HTTP by default,
1365 but can overridden to use SRU/post, SRW, and the Solr protocol.
1368 You can prefix the scheme-qualified host-string with one or more
1370 <literal><parameter>key</parameter>=<parameter>value</parameter></literal>
1371 sequences, each of which represents an option to be set into the
1372 connection structure <emphasis>before</emphasis> the
1373 protocol-level connection is forged and the initialization
1374 handshake takes place. This facility can be used to provide
1375 authentication credentials, as in host-strings such as:
1376 <literal>user=admin,password=halfAm4n,tcp:localhost:8017/db</literal>
1379 Connection objects should be destroyed using the function
1380 <function>ZOOM_connection_destroy</function>.
1383 void ZOOM_connection_option_set(ZOOM_connection c,
1384 const char *key, const char *val);
1386 void ZOOM_connection_option_setl(ZOOM_connection c,
1388 const char *val, int len);
1390 const char *ZOOM_connection_option_get(ZOOM_connection c,
1392 const char *ZOOM_connection_option_getl(ZOOM_connection c,
1397 The functions <function>ZOOM_connection_option_set</function> and
1398 <function>ZOOM_connection_option_setl</function> allows you to
1399 set an option given by <parameter>key</parameter> to the value
1400 <parameter>value</parameter> for the connection.
1401 For <function>ZOOM_connection_option_set</function>, the
1402 value is assumed to be a 0-terminated string. Function
1403 <function>ZOOM_connection_option_setl</function> specifies a
1404 value of a certain size (len).
1407 Functions <function>ZOOM_connection_option_get</function> and
1408 <function>ZOOM_connection_option_getl</function> returns
1409 the value for an option given by <parameter>key</parameter>.
1411 <table id="zoom-connection-options" frame="top">
1412 <title>ZOOM Connection Options</title>
1414 <colspec colwidth="4*" colname="name"></colspec>
1415 <colspec colwidth="7*" colname="description"></colspec>
1416 <colspec colwidth="3*" colname="default"></colspec>
1419 <entry>Option</entry>
1420 <entry>Description</entry>
1421 <entry>Default</entry>
1426 implementationName</entry><entry>Name of Your client
1427 </entry><entry>none</entry></row>
1429 user</entry><entry>Authentication user name
1430 </entry><entry>none</entry></row>
1432 group</entry><entry>Authentication group name
1433 </entry><entry>none</entry></row>
1435 password</entry><entry>Authentication password.
1436 </entry><entry>none</entry></row>
1438 authenticationMode</entry><entry>How authentication is encoded.
1439 </entry><entry>basic</entry></row>
1441 host</entry><entry>Target host. This setting is "read-only".
1442 It's automatically set internally when connecting to a target.
1443 </entry><entry>none</entry></row>
1445 proxy</entry><entry>Proxy host. If set, the logical host
1446 is encoded in the otherInfo area of the Z39.50 Init PDU
1447 with OID 1.2.840.10003.10.1000.81.1.
1448 </entry><entry>none</entry></row>
1450 clientIP</entry><entry>Client IP. If set, is
1451 encoded in the otherInfo area of a Z39.50 PDU with OID
1452 1.2.840.10003.10.1000.81.3. Holds the original IP addreses
1453 of a client. Is used if ZOOM is used in a gateway of some sort.
1454 </entry><entry>none</entry></row>
1456 async</entry><entry>If true (1) the connection operates in
1457 asynchronous operation which means that all calls are non-blocking
1459 <link linkend="zoom.events"><function>ZOOM_event</function></link>.
1460 </entry><entry>0</entry></row>
1462 maximumRecordSize</entry><entry> Maximum size of single record.
1463 </entry><entry>1 MB</entry></row>
1465 preferredMessageSize</entry><entry> Maximum size of multiple records.
1466 </entry><entry>1 MB</entry></row>
1468 lang</entry><entry> Language for negotiation.
1469 </entry><entry>none</entry></row>
1471 charset</entry><entry> Character set for negotiation.
1472 </entry><entry>none</entry></row>
1474 serverImplementationId</entry><entry>
1475 Implementation ID of server. (The old targetImplementationId
1476 option is also supported for the benefit of old applications.)
1477 </entry><entry>none</entry></row>
1479 targetImplementationName</entry><entry>
1480 Implementation Name of server. (The old
1481 targetImplementationName option is also supported for the
1482 benefit of old applications.)
1483 </entry><entry>none</entry></row>
1485 serverImplementationVersion</entry><entry>
1486 Implementation Version of server. (the old
1487 targetImplementationVersion option is also supported for the
1488 benefit of old applications.)
1489 </entry><entry>none</entry></row>
1491 databaseName</entry><entry>One or more database names
1492 separated by character plus (<literal>+</literal>), which is to
1493 be used by subsequent search requests on this Connection.
1494 </entry><entry>Default</entry></row>
1496 piggyback</entry><entry>True (1) if piggyback should be
1497 used in searches; false (0) if not.
1498 </entry><entry>1</entry></row>
1500 smallSetUpperBound</entry><entry>If hits is less than or equal to this
1501 value, then target will return all records using small element set name
1502 </entry><entry>0</entry></row>
1504 largeSetLowerBound</entry><entry>If hits is greater than this
1505 value, the target will return no records.
1506 </entry><entry>1</entry></row>
1508 mediumSetPresentNumber</entry><entry>This value represents
1509 the number of records to be returned as part of a search when
1510 hits is less than or equal to large set lower bound and if hits
1511 is greater than small set upper bound.
1512 </entry><entry>0</entry></row>
1514 smallSetElementSetName</entry><entry>
1515 The element set name to be used for small result sets.
1516 </entry><entry>none</entry></row>
1518 mediumSetElementSetName</entry><entry>
1519 The element set name to be used for medium-sized result sets.
1520 </entry><entry>none</entry></row>
1522 init_opt_search, init_opt_present, init_opt_delSet, etc.</entry><entry>
1523 After a successful Init, these options may be interrogated to
1524 discover whether the server claims to support the specified
1526 </entry><entry>none</entry></row>
1528 <entry>sru</entry><entry>
1529 SRU/Solr transport type. Must be either <literal>soap</literal>,
1530 <literal>get</literal>, <literal>post</literal>, or
1531 <literal>solr</literal>.
1532 </entry><entry>soap</entry></row>
1534 sru_version</entry><entry>
1535 SRU/SRW version. Should be <literal>1.1</literal>, or
1536 <literal>1.2</literal>. This is, prior to connect, the version
1537 to offer (highest version). And following connect (in fact
1538 first operation), holds the negotiated version with the server
1539 (same or lower version).
1540 </entry><entry>1.2</entry></row>
1541 <row id="zoom.extraArgs.option"><entry>
1542 extraArgs</entry><entry>
1543 Extra arguments for SRU/Solr URLs. The value must be
1544 URL encoded already.
1545 </entry><entry></entry></row>
1546 <row id="zoom.facets.option"><entry>
1547 facets</entry><entry>
1548 Requested or recommended facets may be given before a search is sent.
1549 The value of this setting is described in <xref linkend="facets"/>
1550 For inspection of the facets returned, refer to the functions
1551 described in <xref linkend="zoom.facets"/>.
1552 </entry><entry>none</entry></row>
1554 apdulog</entry><entry>
1555 If set to a true value such as "1", a log of low-level
1556 protocol packets is emitted on standard error stream. This
1557 can be very useful for debugging.
1558 </entry><entry>0</entry></row>
1560 saveAPDU</entry><entry>
1561 If set to a true value such as "1", a log of low-level
1562 protocol packets is saved. The log can be retrieved by reading
1563 option APDU. Setting saveAPDU always has the side effect of
1564 resetting the currently saved log. This setting is
1565 <emphasis>write-only</emphasis>. If read, NULL will be returned.
1566 It is only recognized in
1567 <function>ZOOM_connection_option_set</function>.
1568 </entry><entry>0</entry></row>
1571 Returns the log of protocol packets. Will be empty if logging
1572 is not enabled (see saveAPDU above). This setting is
1573 <emphasis>read-only</emphasis>. It is only recognized if used
1574 in call to <function>ZOOM_connection_option_get</function> or
1575 <function>ZOOM_connection_option_getl</function>.
1576 </entry><entry></entry></row>
1578 memcached</entry><entry>
1579 If given and non-empty,
1580 <ulink url="&url.libmemcached;">libMemcached</ulink>
1581 will be configured for the connection.
1582 This option is inspected by ZOOM when a connection is established.
1583 If the <literal>memcached</literal> option is given
1584 and YAZ is compiled without libMemcached support, an internal
1585 diagnostic (10018) will be thrown.
1586 libMemcached support is available for YAZ 5.0.13 or later. If this
1587 option is supplied for an earlier version of YAZ, it is
1588 <emphasis>ignored</emphasis>.
1589 The value of this option is a list options - each is of the
1590 form <literal>--name=value</literal>.
1591 Option <literal>--server=</literal>host[:port] specifies a memcached
1592 server. It may be repeated for multiple memcached servers.
1593 Option <literal>--expire=</literal>seconds sets expiry time in seconds
1594 for how long result sets are to be cached.
1595 </entry><entry>none</entry></row>
1597 redis</entry><entry>
1598 If given and non-empty,
1599 a <ulink url="&url.redis;">redis</ulink> context will be created
1601 This option is inspected by ZOOM when a connection is established.
1602 If the <literal>redis</literal> option is given
1603 and YAZ is compiled without redis support, an internal
1604 diagnostic (10018) will be thrown.
1605 redis support is available for YAZ 5.2.0 or later. If this
1606 option is supplied for an earlier version of YAZ, it is
1607 <emphasis>ignored</emphasis>.
1608 The value of this option is a set of options, similar to that
1609 of the memcached setting. At this stage only --server=host[:port]
1610 and --expire=seconds are supported.
1611 </entry><entry>none</entry></row>
1616 If either option <literal>lang</literal> or <literal>charset</literal>
1618 <ulink url="&url.z39.50.charneg;">
1619 Character Set and Language Negotiation</ulink> is in effect.
1622 int ZOOM_connection_error(ZOOM_connection c, const char **cp,
1623 const char **addinfo);
1624 int ZOOM_connection_error_x(ZOOM_connection c, const char **cp,
1625 const char **addinfo, const char **dset);
1628 Function <function>ZOOM_connection_error</function> checks for
1629 errors for the last operation(s) performed. The function returns
1630 zero if no errors occurred; non-zero otherwise indicating the error.
1631 Pointers <parameter>cp</parameter> and <parameter>addinfo</parameter>
1632 holds messages for the error and additional-info if passed as
1633 non-<literal>NULL</literal>. Function
1634 <function>ZOOM_connection_error_x</function> is an extended version
1635 of <function>ZOOM_connection_error</function> that is capable of
1636 returning name of diagnostic set in <parameter>dset</parameter>.
1638 <sect2 id="zoom-connection-z39.50">
1639 <title>Z39.50 Protocol behavior</title>
1641 The calls <function>ZOOM_connection_new</function> and
1642 <function>ZOOM_connection_connect</function> establishes a TCP/IP
1643 connection and sends an Initialize Request to the target if
1644 possible. In addition, the calls wait for an Initialize Response
1645 from the target and the result is inspected (OK or rejected).
1648 If <literal>proxy</literal> is set then the client will establish
1649 a TCP/IP connection with the peer as specified by the
1650 <literal>proxy</literal> host and the hostname as part of the
1651 connect calls will be set as part of the Initialize Request.
1652 The proxy server will then "forward" the PDU's transparently
1653 to the target behind the proxy.
1656 For the authentication parameters, if option <literal>user</literal>
1657 is set and both options <literal>group</literal> and
1658 <literal>pass</literal> are unset, then Open style
1659 authentication is used (Version 2/3) in which case the username
1660 is usually followed by a slash, then by a password.
1661 If either <literal>group</literal>
1662 or <literal>pass</literal> is set then idPass authentication
1663 (Version 3 only) is used. If none of the options are set, no
1664 authentication parameters are set as part of the Initialize Request
1668 When option <literal>async</literal> is 1, it really means that
1669 all network operations are postponed (and queued) until the
1670 function <literal>ZOOM_event</literal> is invoked. When doing so
1671 it doesn't make sense to check for errors after
1672 <literal>ZOOM_connection_new</literal> is called since that
1673 operation "connecting - and init" is still incomplete and the
1674 API cannot tell the outcome (yet).
1677 <sect2 id="zoom.sru.init.behavior">
1678 <title>SRU/Solr Protocol behavior</title>
1680 The HTTP based protocols (SRU, SRW, Solr) do not feature an
1681 Inititialize Request, so the connection phase merely establishes a
1682 TCP/IP connection with the HTTP server.
1684 <para>Most of the ZOOM connection options do not
1685 affect SRU/Solr and they are ignored. However, future versions
1686 of &yaz; might honor <literal>implementationName</literal> and
1687 put that as part of User-Agent header for HTTP requests.
1690 The <literal>charset</literal> is used in the Content-Type header
1694 Setting <literal>authentcationMode</literal> specifies how
1695 authentication parameters are encoded for HTTP. The default is
1696 "<literal>basic</literal>" where <literal>user</literal> and
1697 <literal>password</literal> are encoded by using HTTP basic
1701 If <literal>authentcationMode</literal> is "<literal>url</literal>", then
1702 user and password are encoded in the URL by parameters
1703 <literal>x-username</literal> and <literal>x-password</literal> as
1704 given by the SRU standard.
1708 <sect1 id="zoom.query">
1709 <title>Queries</title>
1711 Query objects represents queries.
1714 ZOOM_query ZOOM_query_create(void);
1716 void ZOOM_query_destroy(ZOOM_query q);
1718 int ZOOM_query_prefix(ZOOM_query q, const char *str);
1720 int ZOOM_query_cql(ZOOM_query s, const char *str);
1722 int ZOOM_query_sortby(ZOOM_query q, const char *criteria);
1724 int ZOOM_query_sortby2(ZOOM_query q, const char *strategy,
1725 const char *criteria);
1728 Create query objects using <function>ZOOM_query_create</function>
1729 and destroy them by calling <function>ZOOM_query_destroy</function>.
1730 RPN-queries can be specified in <link linkend="PQF">PQF</link>
1731 notation by using the
1732 function <function>ZOOM_query_prefix</function>.
1733 The <function>ZOOM_query_cql</function> specifies a CQL
1734 query to be sent to the server/target.
1735 More query types will be added in future versions of &yaz;, such as
1736 <link linkend="CCL">CCL</link> to RPN-mapping, native CCL query,
1737 etc. In addition to a search, a sort criteria may be set. Function
1738 <function>ZOOM_query_sortby</function> enables Z39.50 sorting and
1739 it takes sort criteria using the same string notation as
1740 yaz-client's <link linkend="sortspec">sort command</link>.
1742 <para id="zoom.query.sortby2">
1743 <function>ZOOM_query_sortby2</function> is similar to
1744 <function>ZOOM_query_sortby</function> but allows a strategy for
1745 sorting. The reason for the strategy parameter is that some
1746 protocols offer multiple ways of performing sorting.
1747 For example, Z39.50 has the standard sort, which is performed after
1748 search on an existing result set.
1749 It's also possible to use CQL in Z39.50 as the query type and use
1750 CQL's SORTBY keyword. Finally, Index Data's
1751 Zebra server also allows sorting to be specified as part of RPN (Type 7).
1753 <table id="zoom-sort-strategy" frame="top">
1754 <title>ZOOM sort strategy</title>
1756 <colspec colwidth="2*" colname="name"/>
1757 <colspec colwidth="5*" colname="description"/>
1761 <entry>Description</entry>
1766 <entry>z39.50</entry><entry>Z39.50 resultset sort</entry>
1769 <entry>type7</entry><entry>Sorting embedded in RPN(Type-7)</entry>
1772 <entry>cql</entry><entry>CQL SORTBY</entry>
1775 <entry>sru11</entry><entry>SRU sortKeys parameter</entry>
1778 <entry>solr</entry><entry>Solr sort</entry>
1781 <entry>embed</entry><entry>type7 for Z39.50, cql for SRU,
1782 solr for Solr protocol</entry>
1788 <sect1 id="zoom.resultsets"><title>Result sets</title>
1790 The result set object is a container for records returned from
1794 ZOOM_resultset ZOOM_connection_search(ZOOM_connection, ZOOM_query q);
1796 ZOOM_resultset ZOOM_connection_search_pqf(ZOOM_connection c,
1798 void ZOOM_resultset_destroy(ZOOM_resultset r);
1801 Function <function>ZOOM_connection_search</function> creates
1802 a result set, given a connection and query.
1803 Destroy a result set by calling
1804 <function>ZOOM_resultset_destroy</function>.
1805 Simple clients using PQF only, may use the function
1806 <function>ZOOM_connection_search_pqf</function> in which case
1807 creating query objects is not necessary.
1810 void ZOOM_resultset_option_set(ZOOM_resultset r,
1811 const char *key, const char *val);
1813 const char *ZOOM_resultset_option_get(ZOOM_resultset r, const char *key);
1815 size_t ZOOM_resultset_size(ZOOM_resultset r);
1818 Functions <function>ZOOM_resultset_options_set</function> and
1819 <function>ZOOM_resultset_get</function> sets and gets an option
1820 for a result set similar to <function>ZOOM_connection_option_get</function>
1821 and <function>ZOOM_connection_option_set</function>.
1824 The number of hits, also called result-count, is returned by
1825 function <function>ZOOM_resultset_size</function>.
1827 <table id="zoom.resultset.options"
1828 frame="top"><title>ZOOM Result set Options</title>
1830 <colspec colwidth="4*" colname="name"></colspec>
1831 <colspec colwidth="7*" colname="description"></colspec>
1832 <colspec colwidth="2*" colname="default"></colspec>
1835 <entry>Option</entry>
1836 <entry>Description</entry>
1837 <entry>Default</entry>
1842 start</entry><entry>Offset of first record to be
1843 retrieved from target. First record has offset 0 unlike the
1844 protocol specifications where first record has position 1.
1845 This option affects ZOOM_resultset_search and
1846 ZOOM_resultset_search_pqf and must be set before any of
1847 these functions are invoked. If a range of
1848 records must be fetched manually after search,
1849 function ZOOM_resultset_records should be used.
1850 </entry><entry>0</entry></row>
1852 count</entry><entry>Number of records to be retrieved.
1853 This option affects ZOOM_resultset_search and
1854 ZOOM_resultset_search_pqf and must be set before any of
1855 these functions are invoked.
1856 </entry><entry>0</entry></row>
1858 presentChunk</entry><entry>The number of records to be
1859 requested from the server in each chunk (present request). The
1860 value 0 means to request all the records in a single chunk.
1861 (The old <literal>step</literal>
1862 option is also supported for the benefit of old applications.)
1863 </entry><entry>0</entry></row>
1865 elementSetName</entry><entry>Element-Set name of records.
1866 Most targets should honor element set name <literal>B</literal>
1867 and <literal>F</literal> for brief and full respectively.
1868 </entry><entry>none</entry></row>
1870 preferredRecordSyntax</entry><entry>Preferred Syntax, such as
1871 <literal>USMARC</literal>, <literal>SUTRS</literal>, etc.
1872 </entry><entry>none</entry></row>
1874 schema</entry><entry>Schema for retrieval, such as
1875 <literal>Gils-schema</literal>, <literal>Geo-schema</literal>, etc.
1876 </entry><entry>none</entry></row>
1878 setname</entry><entry>Name of Result Set (Result Set ID).
1879 If this option isn't set, the ZOOM module will automatically
1880 allocate a result set name.
1881 </entry><entry>default</entry></row>
1883 rpnCharset</entry><entry>Character set for RPN terms.
1884 If this is set, ZOOM C will assume that the ZOOM application is
1885 running UTF-8. Terms in RPN queries are then converted to the
1886 rpnCharset. If this is unset, ZOOM C will not assume any encoding
1887 of RPN terms and no conversion is performed.
1888 </entry><entry>none</entry></row>
1893 For servers that support Search Info report, the following
1894 options may be read using <function>ZOOM_resultset_get</function>.
1895 This detailed information is read after a successful search has
1899 This information is a list of of items, where each item is
1900 information about a term or subquery. All items in the list
1902 <literal>SearchResult.</literal><replaceable>no</replaceable>
1903 where no presents the item number (0=first, 1=second).
1904 Read <literal>searchresult.size</literal> to determine the
1907 <table id="zoom.search.info.report.options"
1908 frame="top"><title>Search Info Report Options</title>
1910 <colspec colwidth="4*" colname="name"></colspec>
1911 <colspec colwidth="7*" colname="description"></colspec>
1914 <entry>Option</entry>
1915 <entry>Description</entry>
1920 <entry>searchresult.size</entry>
1922 number of search result entries. This option is non-existent
1923 if no entries are returned by the server.
1927 <entry>searchresult.<replaceable>no</replaceable>.id</entry>
1928 <entry>sub query ID</entry>
1931 <entry>searchresult.<replaceable>no</replaceable>.count</entry>
1932 <entry>result count for item (number of hits)</entry>
1935 <entry>searchresult.<replaceable>no</replaceable>.subquery.term</entry>
1936 <entry>subquery term</entry>
1940 searchresult.<replaceable>no</replaceable>.interpretation.term
1942 <entry>interpretation term</entry>
1946 searchresult.<replaceable>no</replaceable>.recommendation.term
1948 <entry>recommendation term</entry>
1953 <sect2 id="zoom.z3950.resultset.sort">
1954 <title>Z39.50 Result-set Sort</title>
1956 void ZOOM_resultset_sort(ZOOM_resultset r,
1957 const char *sort_type, const char *sort_spec);
1959 int ZOOM_resultset_sort1(ZOOM_resultset r,
1960 const char *sort_type, const char *sort_spec);
1963 <function>ZOOM_resultset_sort</function> and
1964 <function>ZOOM_resultset_sort1</function> both sort an existing
1965 result-set. The sort_type parameter is not used. Set it to "yaz".
1966 The sort_spec is same notation as ZOOM_query_sortby and identical
1967 to that offered by yaz-client's
1968 <link linkend="sortspec">sort command</link>.
1971 These functions only work for Z39.50. Use the more generic utility
1972 <link linkend="zoom.query.sortby2">
1973 <function>ZOOM_query_sortby2</function></link>
1974 for other protocols (and even Z39.50).
1977 <sect2 id="zoom.z3950.resultset.behavior">
1978 <title>Z39.50 Protocol behavior</title>
1980 The creation of a result set involves at least a SearchRequest
1981 - SearchResponse protocol handshake. Following that, if a sort
1982 criteria was specified as part of the query, a SortRequest -
1983 SortResponse handshake takes place. Note that it is necessary to
1984 perform sorting before any retrieval takes place, so no records will
1985 be returned from the target as part of the SearchResponse because these
1986 would be unsorted. Hence, piggyback is disabled when sort criteria
1987 are set. Following Search - and a possible sort - Retrieval takes
1988 place - as one or more Present Requests/Response pairs being
1992 The API allows for two different modes for retrieval. A high level
1993 mode which is somewhat more powerful and a low level one.
1994 The low level is enabled when searching on a Connection object
1995 for which the settings
1996 <literal>smallSetUpperBound</literal>,
1997 <literal>mediumSetPresentNumber</literal> and
1998 <literal>largeSetLowerBound</literal> are set. The low level mode
1999 thus allows you to precisely set how records are returned as part
2000 of a search response as offered by the Z39.50 protocol.
2001 Since the client may be retrieving records as part of the
2002 search response, this mode doesn't work well if sorting is used.
2005 The high-level mode allows you to fetch a range of records from
2006 the result set with a given start offset. When you use this mode
2007 the client will automatically use piggyback if that is possible
2008 with the target, and perform one or more present requests as needed.
2009 Even if the target returns fewer records as part of a present response
2010 because of a record size limit, etc. the client will repeat sending
2011 present requests. As an example, if option <literal>start</literal>
2012 is 0 (default) and <literal>count</literal> is 4, and
2013 <literal>piggyback</literal> is 1 (default) and no sorting criteria
2014 is specified, then the client will attempt to retrieve the 4
2015 records as part the search response (using piggyback). On the other
2016 hand, if either <literal>start</literal> is positive or if
2017 a sorting criteria is set, or if <literal>piggyback</literal>
2018 is 0, then the client will not perform piggyback but send Present
2022 If either of the options <literal>mediumSetElementSetName</literal> and
2023 <literal>smallSetElementSetName</literal> are unset, the value
2024 of option <literal>elementSetName</literal> is used for piggyback
2025 searches. This means that for the high-level mode you only have
2026 to specify one elementSetName option rather than three.
2029 <sect2 id="zoom.sru.resultset.behavior">
2030 <title>SRU Protocol behavior</title>
2032 Current version of &yaz; does not take advantage of a result set id
2033 returned by the SRU server. Future versions might do, however.
2034 Since the ZOOM driver does not save result set IDs, any
2035 present (retrieval) is transformed to a SRU SearchRetrieveRequest
2036 with same query but, possibly, different offsets.
2039 Option <literal>schema</literal> specifies SRU schema
2040 for retrieval. However, options <literal>elementSetName</literal> and
2041 <literal>preferredRecordSyntax</literal> are ignored.
2044 Options <literal>start</literal> and <literal>count</literal>
2045 are supported by SRU.
2046 The remaining options
2047 <literal>piggyback</literal>,
2048 <literal>smallSetUpperBound</literal>,
2049 <literal>largeSetLowerBound</literal>,
2050 <literal>mediumSetPresentNumber</literal>,
2051 <literal>mediumSetElementSetName</literal>,
2052 <literal>smallSetElementSetName</literal> are
2056 SRU supports CQL queries, <emphasis>not</emphasis> PQF.
2057 If PQF is used, however, the PQF query is transferred anyway
2058 using non-standard element <literal>pQuery</literal> in
2059 SRU SearchRetrieveRequest.
2062 Solr queries need to be done in Solr query format.
2065 Unfortunately, SRU and Solr do not define a database setting. Hence,
2066 <literal>databaseName</literal> is unsupported and ignored.
2067 However, the path part in host parameter for functions
2068 <function>ZOOM_connecton_new</function> and
2069 <function>ZOOM_connection_connect</function> acts as a
2070 database (at least for the &yaz; SRU server).
2074 <sect1 id="zoom.records">
2075 <title>Records</title>
2077 A record object is a retrieval record on the client side -
2078 created from result sets.
2081 void ZOOM_resultset_records(ZOOM_resultset r,
2083 size_t start, size_t count);
2084 ZOOM_record ZOOM_resultset_record(ZOOM_resultset s, size_t pos);
2086 const char *ZOOM_record_get(ZOOM_record rec, const char *type,
2089 int ZOOM_record_error(ZOOM_record rec, const char **msg,
2090 const char **addinfo, const char **diagset);
2092 ZOOM_record ZOOM_record_clone(ZOOM_record rec);
2094 void ZOOM_record_destroy(ZOOM_record rec);
2097 References to temporary records are returned by functions
2098 <function>ZOOM_resultset_records</function> or
2099 <function>ZOOM_resultset_record</function>.
2102 If a persistent reference to a record is desired
2103 <function>ZOOM_record_clone</function> should be used.
2104 It returns a record reference that should be destroyed
2105 by a call to <function>ZOOM_record_destroy</function>.
2108 A single record is returned by function
2109 <function>ZOOM_resultset_record</function> that takes a
2110 position as argument. First record has position zero.
2111 If no record could be obtained <literal>NULL</literal> is returned.
2114 Error information for a record can be checked with
2115 <function>ZOOM_record_error</function> which returns non-zero
2116 (error code) if record is in error, called <emphasis>Surrogate
2117 Diagnostics</emphasis> in Z39.50.
2120 Function <function>ZOOM_resultset_records</function> retrieves
2121 a number of records from a result set. Parameter <literal>start</literal>
2122 and <literal>count</literal> specifies the range of records to
2123 be returned. Upon completion, the array
2124 <literal>recs[0], ..recs[count-1]</literal>
2125 holds record objects for the records. The array of records
2126 <literal>recs</literal> should be allocated prior the call
2127 <function>ZOOM_resultset_records</function>. Note that for those
2128 records that couldn't be retrieved from the target,
2129 <literal>recs[ ..]</literal> is set to <literal>NULL</literal>.
2131 <para id="zoom.record.get">
2132 In order to extract information about a single record,
2133 <function>ZOOM_record_get</function> is provided. The
2134 function returns a pointer to certain record information. The
2135 nature (type) of the pointer depends on the parameter,
2136 <parameter>type</parameter>.
2139 The <parameter>type</parameter> is a string of the format:
2142 <replaceable>format</replaceable>[;charset=<replaceable>from</replaceable>[/<replaceable>opacfrom</replaceable>][,<replaceable>to</replaceable>]][;format=<replaceable>v</replaceable>][;base64=<replaceable>xpath</replaceable>]
2145 If <literal>charset</literal> is given, then <replaceable>from</replaceable>
2146 specifies the character set of the record in its original form
2147 (as returned by the server), <replaceable>to</replaceable> specifies
2148 the output (returned) character set encoding.
2149 If <replaceable>to</replaceable> is omitted, then UTF-8 is assumed.
2150 If charset is not given, then no character set conversion takes place.
2151 OPAC records may be returned in a different
2152 set from the bibliographic MARC record. If this is this the case,
2153 <replaceable>opacfrom</replaceable> should be set to the character set
2154 of the OPAC record part.
2158 The <literal>format</literal> is generic but can only be used to
2159 specify XML indentation when the value <replaceable>v</replaceable>
2160 is 1 (<literal>format=1</literal>).
2163 The <literal>base64</literal> allows a full record to be extracted
2164 from base64-encoded string in an XML document.
2168 Specifying the OPAC record character set requires YAZ 4.1.5 or later.
2171 Specifying the base64 parameter requires YAZ 4.2.35 or later.
2175 The format argument controls whether record data should be XML
2176 pretty-printed (post process operation).
2177 It is enabled only if format value <replaceable>v</replaceable> is
2178 <literal>1</literal> and the record content is XML well-formed.
2181 In addition, for certain types, the length
2182 <literal>len</literal> passed will be set to the size in bytes of
2183 the returned information.
2186 The following are the supported values for <replaceable>form</replaceable>.
2188 <varlistentry><term><literal>database</literal></term>
2189 <listitem><para>The Database of the record is returned
2190 as a C null-terminated string. Return type
2191 <literal>const char *</literal>.
2194 <varlistentry><term><literal>syntax</literal></term>
2195 <listitem><para>The transfer syntax of the record is returned
2196 as a C null-terminated string containing the symbolic name of
2197 the record syntax, e.g. <literal>Usmarc</literal>. Return type
2199 <literal>const char *</literal>.
2202 <varlistentry><term><literal>schema</literal></term>
2203 <listitem><para>The schema of the record is returned
2204 as a C null-terminated string. Return type is
2205 <literal>const char *</literal>.
2208 <varlistentry><term><literal>render</literal></term>
2209 <listitem><para>The record is returned in a display friendly
2210 format. Upon completion, buffer is returned
2211 (type <literal>const char *</literal>) and length is stored in
2212 <literal>*len</literal>.
2215 <varlistentry><term><literal>raw</literal></term>
2216 <listitem><para>The record is returned in the internal
2217 YAZ specific format. For GRS-1, Explain, and others, the
2218 raw data is returned as type
2219 <literal>Z_External *</literal> which is just the type for
2220 the member <literal>retrievalRecord</literal> in
2221 type <literal>NamePlusRecord</literal>.
2222 For SUTRS and octet aligned record (including all MARCs) the
2223 octet buffer is returned and the length of the buffer.
2226 <varlistentry><term><literal>xml</literal></term>
2227 <listitem><para>The record is returned in XML if possible.
2228 SRU, Solr and Z39.50 records with transfer syntax XML are
2229 returned verbatim. MARC records are returned in
2230 <ulink url="&url.marcxml;">
2233 (converted from ISO2709 to MARCXML by YAZ).
2234 OPAC records are also converted to XML and the
2235 bibliographic record is converted to MARCXML (when possible).
2236 GRS-1 records are not supported for this form.
2237 Upon completion, the XML buffer is returned
2238 (type <literal>const char *</literal>) and length is stored in
2239 <literal>*len</literal>.
2242 <varlistentry><term><literal>opac</literal></term>
2243 <listitem><para>OPAC information for record is returned in XML
2244 if an OPAC record is present at the position given. If no
2245 OPAC record is present, a NULL pointer is returned.
2248 <varlistentry><term><literal>txml</literal></term>
2249 <listitem><para>The record is returned in TurboMARC if possible.
2250 SRU and Z39.50 records with transfer syntax XML are
2251 returned verbatim. MARC records are returned in
2252 <link linkend="tools.turbomarc">
2255 (converted from ISO2709 to TurboMARC by YAZ).
2256 Upon completion, the XML buffer is returned
2257 (type <literal>const char *</literal>) and length is stored in
2258 <literal>*len</literal>.
2261 <varlistentry><term><literal>json</literal></term>
2262 <listitem><para>Like xml, but MARC records are converted to
2263 <ulink url="&url.marc_in_json;">MARC-in-JSON</ulink>.
2271 <ulink url="&url.marc21;">MARC21</ulink>
2273 <ulink url="&url.marc8;">MARC-8</ulink>
2274 character set encoding.
2275 An application that wishes to display in Latin-1 would use
2277 render; charset=marc8,iso-8859-1
2280 <sect2 id="zoom.z3950.record.behavior">
2281 <title>Z39.50 Protocol behavior</title>
2283 The functions <function>ZOOM_resultset_record</function> and
2284 <function>ZOOM_resultset_records</function> inspects the client-side
2285 record cache. Records not found in cache are fetched using
2287 The functions may block (and perform network I/O) - even though option
2288 <literal>async</literal> is 1, because they return records objects.
2289 (And there's no way to return records objects without retrieving them!)
2292 There is a trick, however, in the usage of function
2293 <function>ZOOM_resultset_records</function> that allows for
2294 delayed retrieval (and makes it non-blocking). By using
2295 a null pointer for <parameter>recs</parameter> you're indicating
2296 you're not interested in getting records objects
2297 <emphasis>now</emphasis>.
2300 <sect2 id="zoom.sru.record.behavior">
2301 <title>SRU/Solr Protocol behavior</title>
2303 The ZOOM driver for SRU/Solr treats records returned by a SRU/Solr server
2304 as if they where Z39.50 records with transfer syntax XML and
2305 no element set name or database name.
2309 <sect1 id="zoom.facets"><title>Facets</title>
2311 Facet operations is not part of the official ZOOM specification, but
2312 is an Index Data extension for YAZ-based Z39.50 targets,
2313 <ulink url="&url.solr;">Solr</ulink> and SRU 2.0 targets.
2315 Facets may be requestd by the
2316 <link linkend="zoom.facets.option">facets</link> option before a
2318 For inspection of the returned facets, the following functions are
2322 ZOOM_facet_field *ZOOM_resultset_facets(ZOOM_resultset r);
2324 ZOOM_facet_field ZOOM_resultset_get_facet_field(ZOOM_resultset r,
2325 const char *facet_name);
2327 ZOOM_facet_field ZOOM_resultset_get_facet_field_by_index(ZOOM_resultset r,
2330 size_t ZOOM_resultset_facets_size(ZOOM_resultset r);
2332 const char *ZOOM_facet_field_name(ZOOM_facet_field facet_field);
2334 size_t ZOOM_facet_field_term_count(ZOOM_facet_field facet_field);
2336 const char *ZOOM_facet_field_get_term(ZOOM_facet_field facet_field,
2337 size_t idx, int *freq);
2340 References to temporary structures are returned by all functions.
2341 They are only valid as long the Result set is valid.
2342 <function>ZOOM_resultset_get_facet_field</function> or
2343 <function>ZOOM_resultset_get_facet_field_by_index</function>.
2344 <function>ZOOM_resultset_facets</function>.
2345 <function>ZOOM_facet_field_name</function>.
2346 <function>ZOOM_facet_field_get_term</function>.
2348 <para id="zoom.resultset.get_facet_field">
2349 A single Facet field is returned by function
2350 <function>ZOOM_resultset_get_facet_field</function> or
2351 <function>ZOOM_resultset_get_facet_field_by_index</function> that takes
2352 a result set and facet name or positive index respectively. First
2353 facet has position zero. If no facet could be obtained (invalid name
2354 or index out of bounds) <literal>NULL</literal> is returned.
2356 <para id="zoom.resultset.facets">
2357 An array of facets field can be returned by
2358 <function>ZOOM_resultset_facets</function>. The length of the array is
2359 given by <function>ZOOM_resultset_facets_size</function>. The array is
2360 zero-based and the last entry will be at
2361 <function>ZOOM_resultset_facets_size(result_set)</function>-1.
2363 <para id="zoom.resultset.facets_names">
2364 It is possible to interate over facets by name, by calling
2365 <function>ZOOM_resultset_facets_names</function>.
2366 This will return a const array of char * where each string can be used
2367 as parameter for <function>ZOOM_resultset_get_facet_field</function>.
2370 Function <function>ZOOM_facet_field_name</function> gets the request
2371 facet name from a returned facet field.
2374 Function <function>ZOOM_facet_field_get_term</function> returns the
2375 idx'th term and term count for a facet field.
2376 Idx must between 0 and
2377 <function>ZOOM_facet_field_term_count</function>-1, otherwise the
2378 returned reference will be <literal>NULL</literal>. On a valid idx, the
2379 value of the freq reference will be the term count.
2380 The <literal>freq</literal> parameter must be valid pointer to integer.
2383 <sect1 id="zoom.scan"><title>Scan</title>
2385 This section describes an interface for Scan. Scan is not an
2386 official part of the ZOOM model yet. The result of a scan operation
2387 is the <literal>ZOOM_scanset</literal> which is a set of terms
2388 returned by a target.
2392 The Scan interface is supported for both Z39.50, SRU and Solr.
2396 ZOOM_scanset ZOOM_connection_scan(ZOOM_connection c,
2397 const char *startpqf);
2399 ZOOM_scanset ZOOM_connection_scan1(ZOOM_connection c,
2402 size_t ZOOM_scanset_size(ZOOM_scanset scan);
2404 const char *ZOOM_scanset_term(ZOOM_scanset scan, size_t pos,
2405 size_t *occ, size_t *len);
2407 const char *ZOOM_scanset_display_term(ZOOM_scanset scan, size_t pos,
2408 size_t *occ, size_t *len);
2410 void ZOOM_scanset_destroy(ZOOM_scanset scan);
2412 const char *ZOOM_scanset_option_get(ZOOM_scanset scan,
2415 void ZOOM_scanset_option_set(ZOOM_scanset scan, const char *key,
2419 The scan set is created by function
2420 <function>ZOOM_connection_scan</function> which performs a scan
2421 operation on the connection using the specified
2422 <parameter>startpqf</parameter>.
2423 If the operation was successful, the size of the scan set can be
2424 retrieved by a call to <function>ZOOM_scanset_size</function>.
2425 Like result sets, the items are numbered 0..size-1.
2426 To obtain information about a particular scan term, call function
2427 <function>ZOOM_scanset_term</function>. This function takes
2428 a scan set offset <literal>pos</literal> and returns a pointer
2429 to a <emphasis>raw term</emphasis> or <literal>NULL</literal> if
2431 If present, the <literal>occ</literal> and <literal>len</literal>
2432 are set to the number of occurrences and the length
2433 of the actual term respectively.
2434 <function>ZOOM_scanset_display_term</function> is similar to
2435 <function>ZOOM_scanset_term</function> except that it returns
2436 the <emphasis>display term</emphasis> rather than the raw term.
2437 In a few cases, the term is different from display term. Always
2438 use the display term for display and the raw term for subsequent
2439 scan operations (to get more terms, next scan result, etc).
2442 A scan set may be freed by a call to function
2443 <function>ZOOM_scanset_destroy</function>.
2444 Functions <function>ZOOM_scanset_option_get</function> and
2445 <function>ZOOM_scanset_option_set</function> retrieves and sets
2446 an option respectively.
2449 The <parameter>startpqf</parameter> is a subset of PQF, namely
2450 the Attributes+Term part. Multiple <literal>@attr</literal> can
2451 be used. For example to scan in title (complete) phrases:
2453 @attr 1=4 @attr 6=2 "science o"
2457 The <function>ZOOM_connecton_scan1</function> is a newer and
2458 more generic alternative to <function>ZOOM_connection_scan</function>
2459 which allows to use both CQL and PQF for Scan.
2461 <table frame="top" id="zoom.scanset.options">
2462 <title>ZOOM Scan Set Options</title>
2464 <colspec colwidth="4*" colname="name"></colspec>
2465 <colspec colwidth="7*" colname="description"></colspec>
2466 <colspec colwidth="2*" colname="default"></colspec>
2469 <entry>Option</entry>
2470 <entry>Description</entry>
2471 <entry>Default</entry>
2476 number</entry><entry>Number of Scan Terms requested in next scan.
2477 After scan it holds the actual number of terms returned.
2478 </entry><entry>20</entry></row>
2480 position</entry><entry>Preferred Position of term in response
2481 in next scan; actual position after completion of scan.
2482 </entry><entry>1</entry></row>
2484 stepSize</entry><entry>Step Size
2485 </entry><entry>0</entry></row>
2487 scanStatus</entry><entry>An integer indicating the Scan Status
2489 </entry><entry>0</entry></row>
2491 rpnCharset</entry><entry>Character set for RPN terms.
2492 If this is set, ZOOM C will assume that the ZOOM application is
2493 running UTF-8. Terms in RPN queries are then converted to the
2494 rpnCharset. If this is unset, ZOOM C will not assume any encoding
2495 of RPN terms and no conversion is performed.
2496 </entry><entry>none</entry></row>
2501 <sect1 id="zoom.extendedservices">
2502 <title>Extended Services</title>
2504 ZOOM offers an interface to a subset of the Z39.50 extended services
2505 as well as a few privately defined ones:
2510 Z39.50 Item Order (ILL).
2511 See <xref linkend="zoom.item.order"/>.
2516 Record Update. This allows a client to insert, modify or delete
2518 See <xref linkend="zoom.record.update"/>.
2523 Database Create. This a non-standard feature. Allows a client
2524 to create a database.
2525 See <xref linkend="zoom.database.create"/>.
2530 Database Drop. This a non-standard feature. Allows a client
2531 to delete/drop a database.
2532 See <xref linkend="zoom.database.drop"/>.
2537 Commit operation. This a non-standard feature. Allows a client
2538 to commit operations.
2539 See <xref linkend="zoom.commit"/>.
2542 <!-- all the ILL PDU options should go here too -->
2545 To create an extended service operation, a <literal>ZOOM_package</literal>
2546 must be created. The operation is a five step operation. The
2547 package is created, package is configured by means of options,
2548 the package is sent, result is inspected (by means of options),
2549 the package is destroyed.
2552 ZOOM_package ZOOM_connection_package(ZOOM_connection c,
2553 ZOOM_options options);
2555 const char *ZOOM_package_option_get(ZOOM_package p,
2557 void ZOOM_package_option_set(ZOOM_package p, const char *key,
2559 void ZOOM_package_send(ZOOM_package p, const char *type);
2561 void ZOOM_package_destroy(ZOOM_package p);
2564 The <function>ZOOM_connection_package</function> creates a
2565 package for the connection given using the options specified.
2568 Functions <function>ZOOM_package_option_get</function> and
2569 <function>ZOOM_package_option_set</function> gets and sets
2573 <function>ZOOM_package_send</function> sends
2574 the package the via connection specified in
2575 <function>ZOOM_connection_package</function>.
2576 The <parameter>type</parameter> specifies the actual extended service
2577 package type to be sent.
2579 <table frame="top" id="zoom.extendedservices.type">
2580 <title>Extended Service Type</title>
2582 <colspec colwidth="3*" colname="value"></colspec>
2583 <colspec colwidth="7*" colname="description"></colspec>
2587 <entry>Description</entry>
2592 <entry>itemorder</entry><entry>Item Order</entry>
2595 <entry>update</entry><entry>Record Update</entry>
2598 <entry>create</entry><entry>Database Create</entry>
2601 <entry>drop</entry><entry>Database Drop</entry>
2604 <entry>commit</entry><entry>Commit Operation</entry>
2610 <table frame="top" id="zoom.extendedservices.options">
2611 <title>Extended Service Common Options</title>
2613 <colspec colwidth="4*" colname="name"></colspec>
2614 <colspec colwidth="7*" colname="description"></colspec>
2615 <colspec colwidth="3*" colname="default"></colspec>
2618 <entry>Option</entry>
2619 <entry>Description</entry>
2620 <entry>Default</entry>
2625 <entry>package-name</entry>
2626 <entry>Extended Service Request package name. Must be specified
2627 as part of a request.</entry>
2631 <entry>user-id</entry>
2632 <entry>User ID of Extended Service Package. Is a request option.</entry>
2636 <entry>function</entry>
2638 Function of package - one of <literal>create</literal>,
2639 <literal>delete</literal>, <literal>modify</literal>. Is
2642 <entry><literal>create</literal></entry>
2645 <entry>waitAction</entry>
2647 Wait action for package. Possible values:
2648 <literal>wait</literal>, <literal>waitIfPossible</literal>,
2649 <literal>dontWait</literal> or <literal>dontReturnPackage</literal>.
2651 <entry><literal>waitIfPossible</literal></entry>
2654 <entry>targetReference</entry>
2656 Target Reference. This is part of the response as returned
2657 by the server. Read it after a successful operation.
2659 <entry><literal>none</literal></entry>
2664 <sect2 id="zoom.item.order">
2665 <title>Item Order</title>
2667 For Item Order, <literal>type</literal> must be set to
2668 <literal>itemorder</literal> in
2669 <function>ZOOM_package_send</function>.
2672 <table frame="top" id="zoom.item.order.options">
2673 <title>Item Order Options</title>
2675 <colspec colwidth="4*" colname="name"></colspec>
2676 <colspec colwidth="7*" colname="description"></colspec>
2677 <colspec colwidth="3*" colname="default"></colspec>
2680 <entry>Option</entry>
2681 <entry>Description</entry>
2682 <entry>Default</entry>
2687 <entry>contact-name</entry>
2688 <entry>ILL contact name</entry>
2692 <entry>contact-phone</entry>
2693 <entry>ILL contact phone</entry>
2697 <entry>contact-email</entry>
2698 <entry>ILL contact email</entry>
2702 <entry>itemorder-setname</entry>
2703 <entry>Name of result set for record</entry>
2704 <entry>default</entry>
2707 <entry>itemorder-item</entry>
2708 <entry>Position for item (record) requested. An integer</entry>
2715 There are two variants of item order: ILL-variant and
2716 XML document variant. In order to use the XML variant the setting
2717 <literal>doc</literal> must hold the XML item order document. If that
2718 setting is unset, the ILL-variant is used.
2721 <table frame="top" id="zoom.illrequest.options">
2722 <title>ILL Request Options</title>
2724 <colspec colwidth="4*" colname="name"></colspec>
2727 <entry>Option</entry>
2731 <row><entry>protocol-version-num</entry></row>
2732 <row><entry>transaction-id,initial-requester-id,person-or-institution-symbol,person</entry></row>
2733 <row><entry>transaction-id,initial-requester-id,person-or-institution-symbol,institution</entry></row>
2734 <row><entry>transaction-id,initial-requester-id,name-of-person-or-institution,name-of-person</entry></row>
2735 <row><entry>transaction-id,initial-requester-id,name-of-person-or-institution,name-of-institution</entry></row>
2736 <row><entry>transaction-id,transaction-group-qualifier</entry></row>
2737 <row><entry>transaction-id,transaction-qualifier</entry></row>
2738 <row><entry>transaction-id,sub-transaction-qualifier</entry></row>
2739 <row><entry>service-date-time,this,date</entry></row>
2740 <row><entry>service-date-time,this,time</entry></row>
2741 <row><entry>service-date-time,original,date</entry></row>
2742 <row><entry>service-date-time,original,time</entry></row>
2743 <row><entry>requester-id,person-or-institution-symbol,person</entry></row>
2744 <row><entry>requester-id,person-or-institution-symbol,institution</entry></row>
2745 <row><entry>requester-id,name-of-person-or-institution,name-of-person</entry></row>
2746 <row><entry>requester-id,name-of-person-or-institution,name-of-institution</entry></row>
2747 <row><entry>responder-id,person-or-institution-symbol,person</entry></row>
2748 <row><entry>responder-id,person-or-institution-symbol,institution</entry></row>
2749 <row><entry>responder-id,name-of-person-or-institution,name-of-person</entry></row>
2750 <row><entry>responder-id,name-of-person-or-institution,name-of-institution</entry></row>
2751 <row><entry>transaction-type</entry></row>
2752 <row><entry>delivery-address,postal-address,name-of-person-or-institution,name-of-person</entry></row>
2753 <row><entry>delivery-address,postal-address,name-of-person-or-institution,name-of-institution</entry></row>
2754 <row><entry>delivery-address,postal-address,extended-postal-delivery-address</entry></row>
2755 <row><entry>delivery-address,postal-address,street-and-number</entry></row>
2756 <row><entry>delivery-address,postal-address,post-office-box</entry></row>
2757 <row><entry>delivery-address,postal-address,city</entry></row>
2758 <row><entry>delivery-address,postal-address,region</entry></row>
2759 <row><entry>delivery-address,postal-address,country</entry></row>
2760 <row><entry>delivery-address,postal-address,postal-code</entry></row>
2761 <row><entry>delivery-address,electronic-address,telecom-service-identifier</entry></row>
2762 <row><entry>delivery-address,electronic-address,telecom-service-addreess</entry></row>
2763 <row><entry>billing-address,postal-address,name-of-person-or-institution,name-of-person</entry></row>
2764 <row><entry>billing-address,postal-address,name-of-person-or-institution,name-of-institution</entry></row>
2765 <row><entry>billing-address,postal-address,extended-postal-delivery-address</entry></row>
2766 <row><entry>billing-address,postal-address,street-and-number</entry></row>
2767 <row><entry>billing-address,postal-address,post-office-box</entry></row>
2768 <row><entry>billing-address,postal-address,city</entry></row>
2769 <row><entry>billing-address,postal-address,region</entry></row>
2770 <row><entry>billing-address,postal-address,country</entry></row>
2771 <row><entry>billing-address,postal-address,postal-code</entry></row>
2772 <row><entry>billing-address,electronic-address,telecom-service-identifier</entry></row>
2773 <row><entry>billing-address,electronic-address,telecom-service-addreess</entry></row>
2774 <row><entry>ill-service-type</entry></row>
2775 <row><entry>requester-optional-messages,can-send-RECEIVED</entry></row>
2776 <row><entry>requester-optional-messages,can-send-RETURNED</entry></row>
2777 <row><entry>requester-optional-messages,requester-SHIPPED</entry></row>
2778 <row><entry>requester-optional-messages,requester-CHECKED-IN</entry></row>
2779 <row><entry>search-type,level-of-service</entry></row>
2780 <row><entry>search-type,need-before-date</entry></row>
2781 <row><entry>search-type,expiry-date</entry></row>
2782 <row><entry>search-type,expiry-flag</entry></row>
2783 <row><entry>place-on-hold</entry></row>
2784 <row><entry>client-id,client-name</entry></row>
2785 <row><entry>client-id,client-status</entry></row>
2786 <row><entry>client-id,client-identifier</entry></row>
2787 <row><entry>item-id,item-type</entry></row>
2788 <row><entry>item-id,call-number</entry></row>
2789 <row><entry>item-id,author</entry></row>
2790 <row><entry>item-id,title</entry></row>
2791 <row><entry>item-id,sub-title</entry></row>
2792 <row><entry>item-id,sponsoring-body</entry></row>
2793 <row><entry>item-id,place-of-publication</entry></row>
2794 <row><entry>item-id,publisher</entry></row>
2795 <row><entry>item-id,series-title-number</entry></row>
2796 <row><entry>item-id,volume-issue</entry></row>
2797 <row><entry>item-id,edition</entry></row>
2798 <row><entry>item-id,publication-date</entry></row>
2799 <row><entry>item-id,publication-date-of-component</entry></row>
2800 <row><entry>item-id,author-of-article</entry></row>
2801 <row><entry>item-id,title-of-article</entry></row>
2802 <row><entry>item-id,pagination</entry></row>
2803 <row><entry>item-id,ISBN</entry></row>
2804 <row><entry>item-id,ISSN</entry></row>
2805 <row><entry>item-id,additional-no-letters</entry></row>
2806 <row><entry>item-id,verification-reference-source</entry></row>
2807 <row><entry>copyright-complicance</entry></row>
2808 <row><entry>retry-flag</entry></row>
2809 <row><entry>forward-flag</entry></row>
2810 <row><entry>requester-note</entry></row>
2811 <row><entry>forward-note</entry></row>
2816 <sect2 id="zoom.record.update">
2817 <title>Record Update</title>
2819 For Record Update, <literal>type</literal> must be set to
2820 <literal>update</literal> in
2821 <function>ZOOM_package_send</function>.
2823 <table frame="top" id="zoom.record.update.options">
2824 <title>Record Update Options</title>
2826 <colspec colwidth="4*" colname="name"></colspec>
2827 <colspec colwidth="7*" colname="description"></colspec>
2828 <colspec colwidth="3*" colname="default"></colspec>
2831 <entry>Option</entry>
2832 <entry>Description</entry>
2833 <entry>Default</entry>
2838 <entry>action</entry>
2840 The update action. One of
2841 <literal>specialUpdate</literal>,
2842 <literal>recordInsert</literal>,
2843 <literal>recordReplace</literal>,
2844 <literal>recordDelete</literal>,
2845 <literal>elementUpdate</literal>.
2847 <entry><literal>specialUpdate (recordInsert for updateVersion=1 which does not support specialUpdate)</literal></entry>
2850 <entry>recordIdOpaque</entry>
2851 <entry>Opaque Record ID</entry>
2855 <entry>recordIdNumber</entry>
2856 <entry>Record ID number</entry>
2860 <entry>record</entry>
2861 <entry>The record itself</entry>
2865 <entry>recordOpaque</entry>
2866 <entry>Specifies an opaque record which is
2867 encoded as an ASN.1 ANY type with the OID as tiven by option
2868 <literal>syntax</literal> (see below).
2869 Option <literal>recordOpaque</literal> is an alternative
2870 to record - and <literal>record</literal> option (above) is
2871 ignored if recordOpaque is set. This option is only available in
2872 YAZ 3.0.35 and later, and is meant to facilitate Updates with
2878 <entry>syntax</entry>
2879 <entry>The record syntax (transfer syntax). Is a string that
2880 is a known record syntax.
2882 <entry>no syntax</entry>
2885 <entry>databaseName</entry>
2886 <entry>Database from connection object</entry>
2887 <entry>Default</entry>
2890 <entry>correlationInfo.note</entry>
2891 <entry>Correlation Info Note (string)</entry>
2895 <entry>correlationInfo.id</entry>
2896 <entry>Correlation Info ID (integer)</entry>
2900 <entry>elementSetName</entry>
2901 <entry>Element Set for Record</entry>
2905 <entry>updateVersion</entry>
2906 <entry>Record Update version which holds one of the values
2907 1, 2 or 3. Each version has a distinct OID:
2909 (<ulink url="&url.z39.50.extupdate1;">first version</ulink>) ,
2911 (second version) and
2912 1.2.840.10003.9.5.1.1
2913 (<ulink url="&url.z39.50.extupdate3;">third and
2914 newest version</ulink>).
2924 <sect2 id="zoom.database.create"><title>Database Create</title>
2926 For Database Create, <literal>type</literal> must be set to
2927 <literal>create</literal> in
2928 <function>ZOOM_package_send</function>.
2931 <table frame="top" id="zoom.database.create.options">
2932 <title>Database Create Options</title>
2934 <colspec colwidth="4*" colname="name"></colspec>
2935 <colspec colwidth="7*" colname="description"></colspec>
2936 <colspec colwidth="3*" colname="default"></colspec>
2939 <entry>Option</entry>
2940 <entry>Description</entry>
2941 <entry>Default</entry>
2946 <entry>databaseName</entry>
2947 <entry>Database from connection object</entry>
2948 <entry>Default</entry>
2954 <sect2 id="zoom.database.drop">
2955 <title>Database Drop</title>
2957 For Database Drop, <literal>type</literal> must be set to
2958 <literal>drop</literal> in
2959 <function>ZOOM_package_send</function>.
2961 <table frame="top" id="zoom.database.drop.options">
2962 <title>Database Drop Options</title>
2964 <colspec colwidth="4*" colname="name"></colspec>
2965 <colspec colwidth="7*" colname="description"></colspec>
2966 <colspec colwidth="3*" colname="default"></colspec>
2969 <entry>Option</entry>
2970 <entry>Description</entry>
2971 <entry>Default</entry>
2976 <entry>databaseName</entry>
2977 <entry>Database from connection object</entry>
2978 <entry>Default</entry>
2984 <sect2 id="zoom.commit">
2985 <title>Commit Operation</title>
2987 For Commit, <literal>type</literal> must be set to
2988 <literal>commit</literal> in
2989 <function>ZOOM_package_send</function>.
2992 <sect2 id="zoom.extended.services.behavior">
2993 <title>Protocol behavior</title>
2995 All the extended services are Z39.50-only.
2999 The database create, drop, and commit services are privately defined
3001 Refer to <filename>esadmin.asn</filename> in YAZ for the ASN.1
3007 <sect1 id="zoom.options">
3008 <title>Options</title>
3010 Most &zoom; objects provide a way to specify options to change behavior.
3011 From an implementation point of view, a set of options is just like
3012 an associative array / hash.
3015 ZOOM_options ZOOM_options_create(void);
3017 ZOOM_options ZOOM_options_create_with_parent(ZOOM_options parent);
3019 void ZOOM_options_destroy(ZOOM_options opt);
3022 const char *ZOOM_options_get(ZOOM_options opt, const char *name);
3024 void ZOOM_options_set(ZOOM_options opt, const char *name,
3028 typedef const char *(*ZOOM_options_callback)
3029 (void *handle, const char *name);
3031 ZOOM_options_callback
3032 ZOOM_options_set_callback(ZOOM_options opt,
3033 ZOOM_options_callback c,
3037 <sect1 id="zoom.queryconversions">
3038 <title>Query conversions</title>
3040 int ZOOM_query_cql2rpn(ZOOM_query s, const char *cql_str,
3041 ZOOM_connection conn);
3043 int ZOOM_query_ccl2rpn(ZOOM_query s, const char *ccl_str,
3045 int *ccl_error, const char **error_string,
3049 <function>ZOOM_query_cql2rpn</function> translates the CQL string,
3050 client-side, into RPN which may be passed to the server.
3051 This is useful for servers that don't themselves
3052 support CQL, for which <function>ZOOM_query_cql</function> is useless.
3053 `conn' is used only as a place to stash diagnostics if compilation
3054 fails; if this information is not needed, a null pointer may be used.
3055 The CQL conversion is driven by option <literal>cqlfile</literal> from
3056 connection conn. This specifies a conversion file (e.g. pqf.properties)
3057 which <emphasis>must</emphasis> be present.
3060 <function>ZOOM_query_ccl2rpn</function> translates the CCL string,
3061 client-side, into RPN which may be passed to the server.
3062 The conversion is driven by the specification given by
3063 <literal>config</literal>. Upon completion 0 is returned on success; -1
3064 is returned on failure. On failure <literal>error_string</literal> and
3065 <literal>error_pos</literal> hold the error message and position of
3066 first error in original CCL string.
3069 <sect1 id="zoom.events"><title>Events</title>
3071 If you're developing non-blocking applications, you have to deal
3075 int ZOOM_event(int no, ZOOM_connection *cs);
3078 The <function>ZOOM_event</function> executes pending events for
3079 a number of connections. Supply the number of connections in
3080 <literal>no</literal> and an array of connections in
3081 <literal>cs</literal> (<literal>cs[0] ... cs[no-1]</literal>).
3082 A pending event could be sending a search, receiving a response,
3084 When an event has occurred for one of the connections, this function
3085 returns a positive integer <literal>n</literal> denoting that an event
3086 occurred for connection <literal>cs[n-1]</literal>.
3087 When no events are pending for the connections, a value of zero is
3089 To ensure that all outstanding requests are performed, call this function
3090 repeatedly until zero is returned.
3093 If <function>ZOOM_event</function> returns, and returns non-zero, the
3094 last event that occurred can be expected.
3097 int ZOOM_connection_last_event(ZOOM_connection cs);
3100 <function>ZOOM_connection_last_event</function> returns an event type
3101 (integer) for the last event.
3104 <table frame="top" id="zoom.event.ids">
3105 <title>ZOOM Event IDs</title>
3107 <colspec colwidth="4*" colname="name"></colspec>
3108 <colspec colwidth="7*" colname="description"></colspec>
3111 <entry>Event</entry>
3112 <entry>Description</entry>
3117 <entry>ZOOM_EVENT_NONE</entry>
3118 <entry>No event has occurred</entry>
3121 <entry>ZOOM_EVENT_CONNECT</entry>
3122 <entry>TCP/IP connect has initiated</entry>
3125 <entry>ZOOM_EVENT_SEND_DATA</entry>
3126 <entry>Data has been transmitted (sending)</entry>
3129 <entry>ZOOM_EVENT_RECV_DATA</entry>
3130 <entry>Data has been received</entry>
3133 <entry>ZOOM_EVENT_TIMEOUT</entry>
3134 <entry>Timeout</entry>
3137 <entry>ZOOM_EVENT_UNKNOWN</entry>
3138 <entry>Unknown event</entry>
3141 <entry>ZOOM_EVENT_SEND_APDU</entry>
3142 <entry>An APDU has been transmitted (sending)</entry>
3145 <entry>ZOOM_EVENT_RECV_APDU</entry>
3146 <entry>An APDU has been received</entry>
3149 <entry>ZOOM_EVENT_RECV_RECORD</entry>
3150 <entry>A result-set record has been received</entry>
3153 <entry>ZOOM_EVENT_RECV_SEARCH</entry>
3154 <entry>A search result has been received</entry>
3161 <chapter id="server">
3162 <title>Generic server</title>
3163 <sect1 id="server.introduction"><title>Introduction</title>
3165 If you aren't into documentation, a good way to learn how the
3166 back end interface works is to look at the <filename>backend.h</filename>
3167 file. Then, look at the small dummy-server in
3168 <filename>ztest/ztest.c</filename>. The <filename>backend.h</filename>
3169 file also makes a good reference, once you've chewed your way through
3170 the prose of this file.
3173 If you have a database system that you would like to make available by
3174 means of Z39.50 or SRU, &yaz; basically offers your two options. You
3175 can use the APIs provided by the &asn;, &odr;, and &comstack;
3177 create and decode PDUs, and exchange them with a client.
3178 Using this low-level interface gives you access to all fields and
3179 options of the protocol, and you can construct your server as close
3180 to your existing database as you like.
3181 It is also a fairly involved process, requiring
3182 you to set up an event-handling mechanism, protocol state machine,
3183 etc. To simplify server implementation, we have implemented a compact
3184 and simple, but reasonably full-functioned server-frontend that will
3185 handle most of the protocol mechanics, while leaving you to
3186 concentrate on your database interface.
3190 The backend interface was designed in anticipation of a specific
3191 integration task, while still attempting to achieve some degree of
3192 generality. We realize fully that there are points where the
3193 interface can be improved significantly. If you have specific
3194 functions or parameters that you think could be useful, send us a
3195 mail (or better, sign on to the mailing list referred to in the
3196 top-level README file). We will try to fit good suggestions into future
3197 releases, to the extent that it can be done without requiring
3198 too many structural changes in existing applications.
3203 The &yaz; server does not support XCQL.
3207 <sect1 id="server.frontend">
3208 <title>The Database Frontend</title>
3210 We refer to this software as a generic database frontend. Your
3211 database system is the <emphasis>backend database</emphasis>, and the
3212 interface between the two is called the <emphasis>backend API</emphasis>.
3213 The backend API consists of a small number of function handlers and
3214 structure definitions. You are required to provide the
3215 <function>main()</function> routine for the server (which can be
3216 quite simple), as well as a set of handlers to match each of the
3218 The interface functions that you write can use any mechanism you like
3219 to communicate with your database system: You might link the whole
3220 thing together with your database application and access it by
3221 function calls; you might use IPC to talk to a database server
3222 somewhere; or you might link with third-party software that handles
3223 the communication for you (like a commercial database client library).
3224 At any rate, the handlers will perform the tasks of:
3237 Scanning the database index (optional - if you wish to implement SCAN).
3240 Extended Services (optional).
3243 Result-Set Delete (optional).
3246 Result-Set Sort (optional).
3249 Return Explain for SRU (optional).
3253 (more functions will be added in time to support as much of
3254 Z39.50-1995 as possible).
3257 <sect1 id="server.backend">
3258 <title>The Backend API</title>
3260 The header file that you need to use the interface are in the
3261 <filename>include/yaz</filename> directory. It's called
3262 <filename>backend.h</filename>. It will include other files from
3263 the <filename>include/yaz</filename> directory, so you'll
3264 probably want to use the -I option of your compiler to tell it
3265 where to find the files. When you run
3266 <literal>make</literal> in the top-level &yaz; directory,
3267 everything you need to create your server is to link with the
3268 <filename>lib/libyaz.la</filename> library.
3271 <sect1 id="server.main">
3272 <title>Your main() Routine</title>
3274 As mentioned, your <function>main()</function> routine can be quite brief.
3275 If you want to initialize global parameters, or read global configuration
3276 tables, this is the place to do it. At the end of the routine, you should
3280 int statserv_main(int argc, char **argv,
3281 bend_initresult *(*bend_init)(bend_initrequest *r),
3282 void (*bend_close)(void *handle));
3285 The third and fourth arguments are pointers to handlers. Handler
3286 <function>bend_init</function> is called whenever the server receives
3287 an Initialize Request, so it serves as a Z39.50 session initializer. The
3288 <function>bend_close</function> handler is called when the session is
3292 <function>statserv_main</function> will establish listening sockets
3293 according to the parameters given. When connection requests are received,
3294 the event handler will typically <function>fork()</function> and
3295 create a sub-process to handle a new connection.
3296 Alternatively the server may be setup to create threads for each
3298 If you do use global variables and forking, you should be aware, then,
3299 that these cannot be shared between associations, unless you explicitly
3300 disable forking by command line parameters.
3303 The server provides a mechanism for controlling some of its behavior
3304 without using command-line options. The function
3307 statserv_options_block *statserv_getcontrol(void);
3310 will return a pointer to a <literal>struct statserv_options_block</literal>
3311 describing the current default settings of the server. The structure
3312 contains these elements:
3315 <term><literal>int dynamic</literal></term>
3317 A boolean value, which determines whether the server
3318 will fork on each incoming request (TRUE), or not (FALSE). Default is
3319 TRUE. This flag is only read by UNIX-based servers (WIN32 based servers
3324 <term><literal>int threads</literal></term>
3326 A boolean value, which determines whether the server
3327 will create a thread on each incoming request (TRUE), or not (FALSE).
3328 Default is FALSE. This flag is only read by UNIX-based servers
3329 that offer POSIX Threads support.
3330 WIN32-based servers always operate in threaded mode.
3334 <term><literal>int inetd</literal></term>
3336 A boolean value, which determines whether the server
3337 will operates under a UNIX INET daemon (inetd). Default is FALSE.
3341 <term><literal>char logfile[ODR_MAXNAME+1]</literal></term>
3342 <listitem><para>File for diagnostic output ("": stderr).
3346 <term><literal>char apdufile[ODR_MAXNAME+1]</literal></term>
3348 Name of file for logging incoming and outgoing APDUs
3349 ("": don't log APDUs, "-":
3350 <literal>stderr</literal>).
3354 <term><literal>char default_listen[1024]</literal></term>
3355 <listitem><para>Same form as the command-line specification of
3356 listener address. "": no default listener address.
3357 Default is to listen at "tcp:@:9999". You can only
3358 specify one default listener address in this fashion.
3362 <term><literal>enum oid_proto default_proto;</literal></term>
3363 <listitem><para>Either <literal>PROTO_Z3950</literal> or
3364 <literal>PROTO_SR</literal>.
3365 Default is <literal>PROTO_Z39_50</literal>.
3369 <term><literal>int idle_timeout;</literal></term>
3370 <listitem><para>Maximum session idle-time, in minutes. Zero indicates
3371 no (infinite) timeout. Default is 15 minutes.
3375 <term><literal>int maxrecordsize;</literal></term>
3376 <listitem><para>Maximum permissible record (message) size. Default
3377 is 64 MB. This amount of memory will only be allocated if a
3378 client requests a very large amount of records in one operation
3380 Set it to a lower number if you are worried about resource
3381 consumption on your host system.
3385 <term><literal>char configname[ODR_MAXNAME+1]</literal></term>
3386 <listitem><para>Passed to the backend when a new connection is received.
3390 <term><literal>char setuid[ODR_MAXNAME+1]</literal></term>
3391 <listitem><para>Set user id to the user specified, after binding
3392 the listener addresses.
3397 <literal>void (*bend_start)(struct statserv_options_block *p)</literal>
3399 <listitem><para>Pointer to function which is called after the
3400 command line options have been parsed - but before the server
3402 For forked UNIX servers this handler is called in the mother
3403 process; for threaded servers this handler is called in the
3405 The default value of this pointer is NULL in which case it
3406 isn't invoked by the frontend server.
3407 When the server operates as an NT service this handler is called
3408 whenever the service is started.
3413 <literal>void (*bend_stop)(struct statserv_options_block *p)</literal>
3415 <listitem><para>Pointer to function which is called whenever the server
3416 has stopped listening for incoming connections. This function pointer
3417 has a default value of NULL in which case it isn't called.
3418 When the server operates as an NT service this handler is called
3419 whenever the service is stopped.
3423 <term><literal>void *handle</literal></term>
3424 <listitem><para>User defined pointer (default value NULL).
3425 This is a per-server handle that can be used to specify "user-data".
3426 Do not confuse this with the session-handle as returned by bend_init.
3432 The pointer returned by <literal>statserv_getcontrol</literal> points to
3433 a static area. You are allowed to change the contents of the structure,
3434 but the changes will not take effect before you call
3437 void statserv_setcontrol(statserv_options_block *block);
3441 that you should generally update this structure before calling
3442 <function>statserv_main()</function>.
3446 <sect1 id="server.backendfunctions">
3447 <title>The Backend Functions</title>
3449 For each service of the protocol, the backend interface declares one or
3450 two functions. You are required to provide implementations of the
3451 functions representing the services that you wish to implement.
3453 <sect2 id="server.init">
3456 bend_initresult (*bend_init)(bend_initrequest *r);
3459 This handler is called once for each new connection request, after
3460 a new process/thread has been created, and an Initialize Request has
3461 been received from the client. The pointer to the
3462 <function>bend_init</function> handler is passed in the call to
3463 <function>statserv_start</function>.
3466 This handler is also called when operating in SRU mode - when
3467 a connection has been made (even though SRU does not offer
3471 Unlike previous versions of YAZ, the <function>bend_init</function> also
3472 serves as a handler that defines the Z39.50 services that the backend
3473 wish to support. Pointers to <emphasis>all</emphasis> service handlers,
3474 including search - and fetch must be specified here in this handler.
3477 The request - and result structures are defined as
3480 typedef struct bend_initrequest
3482 /** \brief user/name/password to be read */
3483 Z_IdAuthentication *auth;
3484 /** \brief encoding stream (for results) */
3486 /** \brief printing stream */
3488 /** \brief decoding stream (use stream for results) */
3490 /** \brief reference ID */
3491 Z_ReferenceId *referenceId;
3492 /** \brief peer address of client */
3495 /** \brief character set and language negotiation
3497 see include/yaz/z-charneg.h
3499 Z_CharSetandLanguageNegotiation *charneg_request;
3501 /** \brief character negotiation response */
3502 Z_External *charneg_response;
3504 /** \brief character set (encoding) for query terms
3506 This is NULL by default. It should be set to the native character
3507 set that the backend assumes for query terms */
3508 char *query_charset;
3510 /** \brief whehter query_charset also applies to recors
3512 Is 0 (No) by default. Set to 1 (yes) if records is in the same
3513 character set as queries. If in doubt, use 0 (No).
3515 int records_in_same_charset;
3517 char *implementation_id;
3518 char *implementation_name;
3519 char *implementation_version;
3521 /** \brief Z39.50 sort handler */
3522 int (*bend_sort)(void *handle, bend_sort_rr *rr);
3523 /** \brief SRU/Z39.50 search handler */
3524 int (*bend_search)(void *handle, bend_search_rr *rr);
3525 /** \brief SRU/Z39.50 fetch handler */
3526 int (*bend_fetch)(void *handle, bend_fetch_rr *rr);
3527 /** \brief SRU/Z39.50 present handler */
3528 int (*bend_present)(void *handle, bend_present_rr *rr);
3529 /** \brief Z39.50 extended services handler */
3530 int (*bend_esrequest) (void *handle, bend_esrequest_rr *rr);
3531 /** \brief Z39.50 delete result set handler */
3532 int (*bend_delete)(void *handle, bend_delete_rr *rr);
3533 /** \brief Z39.50 scan handler */
3534 int (*bend_scan)(void *handle, bend_scan_rr *rr);
3535 /** \brief Z39.50 segment facility handler */
3536 int (*bend_segment)(void *handle, bend_segment_rr *rr);
3537 /** \brief SRU explain handler */
3538 int (*bend_explain)(void *handle, bend_explain_rr *rr);
3539 /** \brief SRU scan handler */
3540 int (*bend_srw_scan)(void *handle, bend_scan_rr *rr);
3541 /** \brief SRU record update handler */
3542 int (*bend_srw_update)(void *handle, bend_update_rr *rr);
3544 /** \brief whether named result sets are supported (0=disable, 1=enable) */
3545 int named_result_sets;
3548 typedef struct bend_initresult
3550 int errcode; /* 0==OK */
3551 char *errstring; /* system error string or NULL */
3552 void *handle; /* private handle to the backend module */
3556 In general, the server frontend expects that the
3557 <literal>bend_*result</literal> pointer that you return is valid at
3558 least until the next call to a <literal>bend_* function</literal>.
3559 This applies to all of the functions described herein. The parameter
3560 structure passed to you in the call belongs to the server frontend, and
3561 you should not make assumptions about its contents after the current
3562 function call has completed. In other words, if you want to retain any
3563 of the contents of a request structure, you should copy them.
3566 The <literal>errcode</literal> should be zero if the initialization of
3567 the backend went well. Any other value will be interpreted as an error.
3568 The <literal>errstring</literal> isn't used in the current version, but
3569 one option would be to stick it in the initResponse as a VisibleString.
3570 The <literal>handle</literal> is the most important parameter. It should
3571 be set to some value that uniquely identifies the current session to
3572 the backend implementation. It is used by the frontend server in any
3573 future calls to a backend function.
3574 The typical use is to set it to point to a dynamically allocated state
3575 structure that is private to your backend module.
3578 The <literal>auth</literal> member holds the authentication information
3579 part of the Z39.50 Initialize Request. Interpret this if your serves
3580 requires authentication.
3583 The members <literal>peer_name</literal>,
3584 <literal>implementation_id</literal>,
3585 <literal>implementation_name</literal> and
3586 <literal>implementation_version</literal> holds
3587 DNS of client, ID of implementor, name
3588 of client (Z39.50) implementation - and version.
3591 The <literal>bend_</literal> - members are set to NULL when
3592 <function>bend_init</function> is called. Modify the pointers by
3593 setting them to point to backend functions.
3596 <sect2 id="server.search.retrieve">
3597 <title>Search and Retrieve</title>
3599 We now describe the handlers that are required to support search -
3600 and retrieve. You must support two functions - one for search - and one
3601 for fetch (retrieval of one record). If desirable you can provide a
3602 third handler which is called when a present request is received which
3603 allows you to optimize retrieval of multiple-records.
3606 int (*bend_search) (void *handle, bend_search_rr *rr);
3609 char *setname; /* name to give to this set */
3610 int replace_set; /* replace set, if it already exists */
3611 int num_bases; /* number of databases in list */
3612 char **basenames; /* databases to search */
3613 Z_ReferenceId *referenceId;/* reference ID */
3614 Z_Query *query; /* query structure */
3615 ODR stream; /* encode stream */
3616 ODR decode; /* decode stream */
3617 ODR print; /* print stream */
3619 bend_request request;
3620 bend_association association;
3622 int hits; /* number of hits */
3623 int errcode; /* 0==OK */
3624 char *errstring; /* system error string or NULL */
3625 Z_OtherInformation *search_info; /* additional search info */
3626 char *srw_sortKeys; /* holds SRU/SRW sortKeys info */
3627 char *srw_setname; /* holds SRU/SRW generated resultsetID */
3628 int *srw_setnameIdleTime; /* holds SRU/SRW life-time */
3629 int estimated_hit_count; /* if hit count is estimated */
3630 int partial_resultset; /* if result set is partial */
3634 The <function>bend_search</function> handler is a fairly close
3635 approximation of a protocol Z39.50 Search Request - and Response PDUs
3636 The <literal>setname</literal> is the resultSetName from the protocol.
3637 You are required to establish a mapping between the set name and whatever
3638 your backend database likes to use.
3639 Similarly, the <literal>replace_set</literal> is a boolean value
3640 corresponding to the resultSetIndicator field in the protocol.
3641 <literal>num_bases/basenames</literal> is a length of/array of character
3642 pointers to the database names provided by the client.
3643 The <literal>query</literal> is the full query structure as defined in
3644 the protocol ASN.1 specification.
3645 It can be either of the possible query types, and it's up to you to
3646 determine if you can handle the provided query type.
3647 Rather than reproduce the C interface here, we'll refer you to the
3648 structure definitions in the file
3649 <filename>include/yaz/z-core.h</filename>. If you want to look at the
3650 attributeSetId OID of the RPN query, you can either match it against
3651 your own internal tables, or you can use the <link linkend="tools.oid">
3655 The structure contains a number of hits, and an
3656 <literal>errcode/errstring</literal> pair. If an error occurs
3657 during the search, or if you're unhappy with the request, you should
3658 set the errcode to a value from the BIB-1 diagnostic set. The value
3659 will then be returned to the user in a nonsurrogate diagnostic record
3660 in the response. The <literal>errstring</literal>, if provided, will
3661 go in the addinfo field. Look at the protocol definition for the
3662 defined error codes, and the suggested uses of the addinfo field.
3665 The <function>bend_search</function> handler is also called when
3666 the frontend server receives a SRU SearchRetrieveRequest.
3667 For SRU, a CQL query is usually provided by the client.
3668 The CQL query is available as part of <literal>Z_Query</literal>
3669 structure (note that CQL is now part of Z39.50 via an external).
3670 To support CQL in existing implementations that only do Type-1,
3671 we refer to the CQL-to-PQF tool described
3672 <link linkend="cql.to.pqf">here</link>.
3675 To maintain backwards compatibility, the frontend server
3676 of yaz always assume that error codes are BIB-1 diagnostics.
3677 For SRU operation, a Bib-1 diagnostic code is mapped to
3681 int (*bend_fetch) (void *handle, bend_fetch_rr *rr);
3683 typedef struct bend_fetch_rr {
3684 char *setname; /* set name */
3685 int number; /* record number */
3686 Z_ReferenceId *referenceId;/* reference ID */
3687 Odr_oid *request_format; /* format, transfer syntax (OID) */
3688 Z_RecordComposition *comp; /* Formatting instructions */
3689 ODR stream; /* encoding stream - memory source if req */
3690 ODR print; /* printing stream */
3692 char *basename; /* name of database that provided record */
3693 int len; /* length of record or -1 if structured */
3694 char *record; /* record */
3695 int last_in_set; /* is it? */
3696 Odr_oid *output_format; /* response format/syntax (OID) */
3697 int errcode; /* 0==success */
3698 char *errstring; /* system error string or NULL */
3699 int surrogate_flag; /* surrogate diagnostic */
3700 char *schema; /* string record schema input/output */
3704 The frontend server calls the <function>bend_fetch</function> handler
3705 when it needs database records to fulfill a Z39.50 Search Request, a
3706 Z39.50 Present Request or a SRU SearchRetrieveRequest.
3707 The <literal>setname</literal> is simply the name of the result set
3708 that holds the reference to the desired record.
3709 The <literal>number</literal> is the offset into the set (with 1
3710 being the first record in the set). The <literal>format</literal> field
3711 is the record format requested by the client (See
3712 <xref linkend="tools.oid"/>).
3713 A value of NULL for <literal>format</literal> indicates that the
3714 client did not request a specific format.
3715 The <literal>stream</literal> argument is an &odr; stream which
3716 should be used for allocating space for structured data records.
3717 The stream will be reset when all records have been assembled, and
3718 the response package has been transmitted.
3719 For unstructured data, the backend is responsible for maintaining a
3720 static or dynamic buffer for the record between calls.
3723 If a SRU SearchRetrieveRequest is received by the frontend server,
3724 the <literal>referenceId</literal> is NULL and the
3725 <literal>format</literal> (transfer syntax) is the OID for XML.
3726 The schema for SRU is stored in both the
3727 <literal>Z_RecordComposition</literal>
3728 structure and <literal>schema</literal> (simple string).
3731 In the structure, the <literal>basename</literal> is the name of the
3732 database that holds the
3733 record. <literal>len</literal> is the length of the record returned, in
3734 bytes, and <literal>record</literal> is a pointer to the record.
3735 <literal>last_in_set</literal> should be nonzero only if the record
3736 returned is the last one in the given result set.
3737 <literal>errcode</literal> and <literal>errstring</literal>, if
3738 given, will be interpreted as a global error pertaining to the
3739 set, and will be returned in a non-surrogate-diagnostic.
3740 If you wish to return the error as a surrogate-diagnostic
3741 (local error) you can do this by setting
3742 <literal>surrogate_flag</literal> to 1 also.
3745 If the <literal>len</literal> field has the value -1, then
3746 <literal>record</literal> is assumed to point to a constructed data
3747 type. The <literal>format</literal> field will be used to determine
3748 which encoder should be used to serialize the data.
3752 If your backend generates structured records, it should use
3753 <function>odr_malloc()</function> on the provided stream for allocating
3754 data: This allows the frontend server to keep track of the record sizes.
3758 The <literal>format</literal> field is mapped to an object identifier
3759 in the direct reference of the resulting EXTERNAL representation
3764 The current version of &yaz; only supports the direct reference mode.
3768 int (*bend_present) (void *handle, bend_present_rr *rr);
3771 char *setname; /* set name */
3773 int number; /* record number */
3774 Odr_oid *format; /* format, transfer syntax (OID) */
3775 Z_ReferenceId *referenceId;/* reference ID */
3776 Z_RecordComposition *comp; /* Formatting instructions */
3777 ODR stream; /* encoding stream - memory source if required */
3778 ODR print; /* printing stream */
3779 bend_request request;
3780 bend_association association;
3782 int hits; /* number of hits */
3783 int errcode; /* 0==OK */
3784 char *errstring; /* system error string or NULL */
3788 The <function>bend_present</function> handler is called when
3789 the server receives a Z39.50 Present Request.
3790 The <literal>setname</literal>,
3791 <literal>start</literal> and <literal>number</literal> is the
3792 name of the result set - start position - and number of records to
3793 be retrieved respectively. <literal>format</literal> and
3794 <literal>comp</literal> is the preferred transfer syntax and element
3795 specifications of the present request.
3798 Note that this is handler serves as a supplement for
3799 <function>bend_fetch</function> and need not to be defined in order to
3800 support search - and retrieve.
3803 <sect2 id="server.delete">
3804 <title>Delete</title>
3806 For back-ends that supports delete of a result set only one handler
3810 int (*bend_delete)(void *handle, bend_delete_rr *rr);
3812 typedef struct bend_delete_rr {
3816 Z_ReferenceId *referenceId;
3817 int delete_status; /* status for the whole operation */
3818 int *statuses; /* status each set - indexed as setnames */
3825 The delete set function definition is rather primitive, mostly because
3826 we have had no practical need for it as of yet. If someone wants
3827 to provide a full delete service, we'd be happy to add the
3828 extra parameters that are required. Are there clients out there
3829 that will actually delete sets they no longer need?
3833 <sect2 id="server.scan">
3836 For servers that wish to offer the scan service one handler
3840 int (*bend_scan)(void *handle, bend_scan_rr *rr);
3843 BEND_SCAN_SUCCESS, /* ok */
3844 BEND_SCAN_PARTIAL /* not all entries could be found */
3847 typedef struct bend_scan_rr {
3848 int num_bases; /* number of elements in databaselist */
3849 char **basenames; /* databases to search */
3850 Odr_oid *attributeset;
3851 Z_ReferenceId *referenceId; /* reference ID */
3852 Z_AttributesPlusTerm *term;
3853 ODR stream; /* encoding stream - memory source if required */
3854 ODR print; /* printing stream */
3856 int *step_size; /* step size */
3857 int term_position; /* desired index of term in result list/returned */
3858 int num_entries; /* number of entries requested/returned */
3860 /* scan term entries. The called handler does not have
3861 to allocate this. Size of entries is num_entries (see above) */
3862 struct scan_entry *entries;
3863 bend_scan_status status;
3866 char *scanClause; /* CQL scan clause */
3867 char *setname; /* Scan in result set (NULL if omitted) */
3871 This backend server handles both Z39.50 scan
3872 and SRU scan. In order for a handler to distinguish between SRU (CQL) scan
3873 Z39.50 Scan , it must check for a non-NULL value of
3874 <literal>scanClause</literal>.
3878 if designed today, it would be a choice using a union or similar,
3879 but that would break binary compatibility with existing servers.
3884 <sect1 id="server.invocation">
3885 <title>Application Invocation</title>
3887 The finished application has the following
3888 invocation syntax (by way of <function>statserv_main()</function>):
3896 A listener specification consists of a transport mode followed by a
3897 colon (:) followed by a listener address. The transport mode is
3898 either <literal>tcp</literal>, <literal>unix:</literal> or
3899 <literal>ssl</literal>.
3902 For TCP and SSL, an address has the form
3905 hostname | IP-number [: portnumber]
3908 The port number defaults to 210 (standard Z39.50 port).
3911 For UNIX, the address is the filename of socket.
3914 For TCP/IP and SSL, the special hostnames <literal>@</literal>,
3915 maps to <literal>IN6ADDR_ANY_INIT</literal> with
3916 IPV4 binding as well (bindv6only=0),
3917 The special hostname <literal>@4</literal> binds to
3918 <literal>INADDR_ANY</literal> (IPV4 only listener).
3919 The special hostname <literal>@6</literal> binds to
3920 <literal>IN6ADDR_ANY_INIT</literal> with bindv6only=1 (IPV6 only listener).
3922 <example id="server.example.running.unix">
3923 <title>Running the GFS on Unix</title>
3925 Assuming the server application <replaceable>appname</replaceable> is
3926 started as root, the following will make it listen on port 210.
3927 The server will change identity to <literal>nobody</literal>
3928 and write its log to <filename>/var/log/app.log</filename>.
3930 application -l /var/log/app.log -u nobody tcp:@:210
3934 The server will accept Z39.50 requests and offer SRU service on port 210.
3937 <example id="server.example.apache.sru">
3938 <title>Setting up Apache as SRU Frontend</title>
3940 If you use <ulink url="&url.apache;">Apache</ulink>
3941 as your public web server and want to offer HTTP port 80
3942 access to the YAZ server on 210, you can use the
3943 <ulink url="&url.apache.directive.proxypass;">
3944 <literal>ProxyPass</literal></ulink>
3946 If you have virtual host
3947 <literal>srw.mydomain</literal> you can use the following directives
3948 in Apache's httpd.conf:
3951 ErrorLog /home/srw/logs/error_log
3952 TransferLog /home/srw/logs/access_log
3953 ProxyPass / http://srw.mydomain:210/
3958 The above for the Apache 1.3 series.
3961 <example id="server.example.local.access">
3962 <title>Running a server with local access only</title>
3964 Servers that is only being accessed from the local host should listen
3965 on UNIX file socket rather than a Internet socket. To listen on
3966 <filename>/tmp/mysocket</filename> start the server as follows:
3968 application unix:/tmp/mysocket
3973 <sect1 id="server.vhosts">
3974 <title>GFS Configuration and Virtual Hosts</title>
3979 <title>The Z39.50 ASN.1 Module</title>
3980 <sect1 id="asn.introduction">
3981 <title>Introduction</title>
3983 The &asn; module provides you with a set of C struct definitions for the
3984 various PDUs of the Z39.50 protocol, as well as for the complex types
3985 appearing within the PDUs. For the primitive data types, the C
3986 representation often takes the form of an ordinary C language type,
3987 such as <literal>Odr_int</literal> which is equivalent to an integral
3988 C integer. For ASN.1 constructs that have no direct
3989 representation in C, such as general octet strings and bit strings,
3990 the &odr; module (see section <link linkend="odr">The ODR Module</link>)
3991 provides auxiliary definitions.
3994 The &asn; module is located in sub directory <filename>z39.50</filename>.
3995 There you'll find C files that implements encoders and decoders for the
3996 Z39.50 types. You'll also find the protocol definitions:
3997 <filename>z3950v3.asn</filename>, <filename>esupdate.asn</filename>,
4001 <sect1 id="asn.preparing">
4002 <title>Preparing PDUs</title>
4004 A structure representing a complex ASN.1 type doesn't in itself contain the
4005 members of that type. Instead, the structure contains
4006 <emphasis>pointers</emphasis> to the members of the type.
4007 This is necessary, in part, to allow a mechanism for specifying which
4008 of the optional structure (SEQUENCE) members are present, and which
4009 are not. It follows that you will need to somehow provide space for
4010 the individual members of the structure, and set the pointers to
4011 refer to the members.
4014 The conversion routines don't care how you allocate and maintain your
4015 C structures - they just follow the pointers that you provide.
4016 Depending on the complexity of your application, and your personal
4017 taste, there are at least three different approaches that you may take
4018 when you allocate the structures.
4021 You can use static or automatic local variables in the function that
4022 prepares the PDU. This is a simple approach, and it provides the most
4023 efficient form of memory management. While it works well for flat
4024 PDUs like the InitReqest, it will generally not be sufficient for say,
4025 the generation of an arbitrarily complex RPN query structure.
4028 You can individually create the structure and its members using the
4029 <function>malloc(2)</function> function. If you want to ensure that
4030 the data is freed when it is no longer needed, you will have to
4031 define a function that individually releases each member of a
4032 structure before freeing the structure itself.
4035 You can use the <function>odr_malloc()</function> function (see
4036 <xref linkend="odr.use"/> for details). When you use
4037 <function>odr_malloc()</function>, you can release all of the
4038 allocated data in a single operation, independent of any pointers and
4039 relations between the data. <function>odr_malloc()</function> is based on a
4040 "nibble-memory"
4041 scheme, in which large portions of memory are allocated, and then
4042 gradually handed out with each call to <function>odr_malloc()</function>.
4043 The next time you call <function>odr_reset()</function>, all of the
4044 memory allocated since the last call is recycled for future use (actually,
4045 it is placed on a free-list).
4048 You can combine all of the methods described here. This will often be
4049 the most practical approach. For instance, you might use
4050 <function>odr_malloc()</function> to allocate an entire structure and
4051 some of its elements, while you leave other elements pointing to global
4052 or per-session default variables.
4055 The &asn; module provides an important aid in creating new PDUs. For
4056 each of the PDU types (say, <function>Z_InitRequest</function>), a
4057 function is provided that allocates and initializes an instance of
4058 that PDU type for you. In the case of the InitRequest, the function is
4059 simply named <function>zget_InitRequest()</function>, and it sets up
4060 reasonable default value for all of the mandatory members. The optional
4061 members are generally initialized to null pointers. This last aspect
4062 is very important: it ensures that if the PDU definitions are
4063 extended after you finish your implementation (to accommodate
4064 new versions of the protocol, say), you won't get into trouble with
4065 uninitialized pointers in your structures. The functions use
4066 <function>odr_malloc()</function> to
4067 allocate the PDUs and its members, so you can free everything again with a
4068 single call to <function>odr_reset()</function>. We strongly recommend
4069 that you use the <literal>zget_*</literal>
4070 functions whenever you are preparing a PDU (in a C++ API, the
4071 <literal>zget_</literal>
4072 functions would probably be promoted to constructors for the
4076 The prototype for the individual PDU types generally look like this:
4079 Z_<type> *zget_<type>(ODR o);
4085 Z_InitRequest *zget_InitRequest(ODR o);
4088 The &odr; handle should generally be your encoding stream, but it
4092 As well as the individual PDU functions, a function
4093 <function>zget_APDU()</function> is provided, which allocates
4094 a top-level Z-APDU of the type requested:
4097 Z_APDU *zget_APDU(ODR o, int which);
4100 The <varname>which</varname> parameter is (of course) the discriminator
4101 belonging to the <varname>Z_APDU</varname> <literal>CHOICE</literal> type.
4102 All of the interface described here is provided by the &asn; module, and
4103 you access it through the <filename>proto.h</filename> header file.
4106 <sect1 id="asn.external">
4107 <title>EXTERNAL Data</title>
4109 In order to achieve extensibility and adaptability to different
4110 application domains, the new version of the protocol defines many
4111 structures outside of the main ASN.1 specification, referencing them
4112 through ASN.1 EXTERNAL constructs. To simplify the construction and
4113 access to the externally referenced data, the &asn; module defines a
4114 specialized version of the EXTERNAL construct, called
4115 <literal>Z_External</literal>.It is defined thus:
4118 typedef struct Z_External
4120 Odr_oid *direct_reference;
4121 int *indirect_reference;
4126 Z_External_single = 0,
4128 Z_External_arbitrary,
4130 /* Specific types */
4132 Z_External_explainRecord,
4133 Z_External_resourceReport1,
4134 Z_External_resourceReport2
4142 Odr_any *single_ASN1_type;
4143 Odr_oct *octet_aligned;
4144 Odr_bitmask *arbitrary;
4146 /* Specific types */
4148 Z_ExplainRecord *explainRecord;
4149 Z_ResourceReport1 *resourceReport1;
4150 Z_ResourceReport2 *resourceReport2;
4158 When decoding, the &asn; module will attempt to determine which
4159 syntax describes the data by looking at the reference fields
4160 (currently only the direct-reference). For ASN.1 structured data, you
4161 need only consult the <literal>which</literal> field to determine the
4162 type of data. You can the access the data directly through the union.
4163 When constructing data for encoding, you set the union pointer to point
4164 to the data, and set the <literal>which</literal> field accordingly.
4165 Remember also to set the direct (or indirect) reference to the correct
4166 OID for the data type.
4167 For non-ASN.1 data such as MARC records, use the
4168 <literal>octet_aligned</literal> arm of the union.
4171 Some servers return ASN.1 structured data values (eg. database
4172 records) as BER-encoded records placed in the
4173 <literal>octet-aligned</literal> branch of the EXTERNAL CHOICE.
4174 The ASN-module will <emphasis>not</emphasis> automatically decode
4175 these records. To help you decode the records in the application, the
4179 Z_ext_typeent *z_ext_gettypebyref(const oid *oid);
4182 Can be used to retrieve information about the known, external data
4183 types. The function return a pointer to a static area, or NULL, if no
4184 match for the given direct reference is found. The
4185 <literal>Z_ext_typeent</literal>
4189 typedef struct Z_ext_typeent
4191 int oid[OID_SIZE]; /* the direct-reference OID. */
4192 int what; /* discriminator value for the external CHOICE */
4193 Odr_fun fun; /* decoder function */
4197 The <literal>what</literal> member contains the
4198 <literal>Z_External</literal> union discriminator value for the
4199 given type: For the SUTRS record syntax, the value would be
4200 <literal>Z_External_sutrs</literal>.
4201 The <literal>fun</literal> member contains a pointer to the
4202 function which encodes/decodes the given type. Again, for the SUTRS
4203 record syntax, the value of <literal>fun</literal> would be
4204 <literal>z_SUTRS</literal> (a function pointer).
4207 If you receive an EXTERNAL which contains an octet-string value that
4208 you suspect of being an ASN.1-structured data value, you can use
4209 <literal>z_ext_gettypebyref</literal> to look for the provided
4211 If the return value is different from NULL, you can use the provided
4212 function to decode the BER string (see <xref linkend="odr.use"/>
4216 If you want to <emphasis>send</emphasis> EXTERNALs containing
4217 ASN.1-structured values in the occtet-aligned branch of the CHOICE, this
4218 is possible too. However, on the encoding phase, it requires a somewhat
4219 involved juggling around of the various buffers involved.
4222 If you need to add new, externally defined data types, you must update
4223 the struct above, in the source file <filename>prt-ext.h</filename>, as
4224 well as the encoder/decoder in the file <filename>prt-ext.c</filename>.
4225 When changing the latter, remember to update both the
4226 <literal>arm</literal> arrary and the list
4227 <literal>type_table</literal>, which drives the CHOICE biasing that
4228 is necessary to tell the different, structured types apart
4233 Eventually, the EXTERNAL processing will most likely
4234 automatically insert the correct OIDs or indirect-refs. First,
4235 however, we need to determine how application-context management
4236 (specifically the presentation-context-list) should fit into the
4241 <sect1 id="asn.pdu">
4242 <title>PDU Contents Table</title>
4244 We include, for reference, a listing of the fields of each top-level
4245 PDU, as well as their default settings.
4247 <table frame="top" id="asn.default.initialize.request">
4248 <title>Default settings for PDU Initialize Request</title>
4250 <colspec colwidth="7*" colname="field"></colspec>
4251 <colspec colwidth="5*" colname="type"></colspec>
4252 <colspec colwidth="7*" colname="value"></colspec>
4255 <entry>Field</entry>
4257 <entry>Default Value</entry>
4262 referenceId</entry><entry>Z_ReferenceId</entry><entry>NULL
4265 protocolVersion</entry><entry>Odr_bitmask</entry><entry>Empty bitmask
4268 options</entry><entry>Odr_bitmask</entry><entry>Empty bitmask
4271 preferredMessageSize</entry><entry>Odr_int</entry><entry>30*1024
4274 maximumRecordSize</entry><entry>Odr_int</entry><entry>30*1024
4277 idAuthentication</entry><entry>Z_IdAuthentication</entry><entry>NULL
4280 implementationId</entry><entry>char*</entry><entry>"81"
4283 implementationName</entry><entry>char*</entry><entry>"YAZ"
4286 implementationVersion</entry><entry>char*</entry><entry>YAZ_VERSION
4289 userInformationField</entry><entry>Z_UserInformation</entry><entry>NULL
4292 otherInfo</entry><entry>Z_OtherInformation</entry><entry>NULL
4297 <table frame="top" id="asn.default.initialize.response">
4298 <title>Default settings for PDU Initialize Response</title>
4300 <colspec colwidth="7*" colname="field"></colspec>
4301 <colspec colwidth="5*" colname="type"></colspec>
4302 <colspec colwidth="7*" colname="value"></colspec>
4305 <entry>Field</entry>
4307 <entry>Default Value</entry>
4312 referenceId</entry><entry>Z_ReferenceId</entry><entry>NULL
4315 protocolVersion</entry><entry>Odr_bitmask</entry><entry>Empty bitmask
4318 options</entry><entry>Odr_bitmask</entry><entry>Empty bitmask
4321 preferredMessageSize</entry><entry>Odr_int</entry><entry>30*1024
4324 maximumRecordSize</entry><entry>Odr_int</entry><entry>30*1024
4327 result</entry><entry>Odr_bool</entry><entry>TRUE
4330 implementationId</entry><entry>char*</entry><entry>"id)"
4333 implementationName</entry><entry>char*</entry><entry>"YAZ"
4336 implementationVersion</entry><entry>char*</entry><entry>YAZ_VERSION
4339 userInformationField</entry><entry>Z_UserInformation</entry><entry>NULL
4342 otherInfo</entry><entry>Z_OtherInformation</entry><entry>NULL
4347 <table frame="top" id="asn.default.search.request">
4348 <title>Default settings for PDU Search Request</title>
4350 <colspec colwidth="7*" colname="field"></colspec>
4351 <colspec colwidth="5*" colname="type"></colspec>
4352 <colspec colwidth="7*" colname="value"></colspec>
4355 <entry>Field</entry>
4357 <entry>Default Value</entry>
4362 referenceId</entry><entry>Z_ReferenceId</entry><entry>NULL
4365 smallSetUpperBound</entry><entry>Odr_int</entry><entry>0
4368 largeSetLowerBound</entry><entry>Odr_int</entry><entry>1
4371 mediumSetPresentNumber</entry><entry>Odr_int</entry><entry>0
4374 replaceIndicator</entry><entry>Odr_bool</entry><entry>TRUE
4377 resultSetName</entry><entry>char *</entry><entry>"default"
4380 num_databaseNames</entry><entry>Odr_int</entry><entry>0
4383 databaseNames</entry><entry>char **</entry><entry>NULL
4386 smallSetElementSetNames</entry><entry>Z_ElementSetNames
4390 mediumSetElementSetNames</entry><entry>Z_ElementSetNames
4394 preferredRecordSyntax</entry><entry>Odr_oid</entry><entry>NULL
4397 query</entry><entry>Z_Query</entry><entry>NULL
4400 additionalSearchInfo</entry><entry>Z_OtherInformation
4404 otherInfo</entry><entry>Z_OtherInformation</entry><entry>NULL
4409 <table frame="top" id="asn.default.search.response">
4410 <title>Default settings for PDU Search Response</title>
4412 <colspec colwidth="7*" colname="field"></colspec>
4413 <colspec colwidth="5*" colname="type"></colspec>
4414 <colspec colwidth="7*" colname="value"></colspec>
4417 <entry>Field</entry>
4419 <entry>Default Value</entry>
4424 referenceId</entry><entry>Z_ReferenceId</entry><entry>NULL
4427 resultCount</entry><entry>Odr_int</entry><entry>0
4430 numberOfRecordsReturned</entry><entry>Odr_int</entry><entry>0
4433 nextResultSetPosition</entry><entry>Odr_int</entry><entry>0
4436 searchStatus</entry><entry>Odr_bool</entry><entry>TRUE
4439 resultSetStatus</entry><entry>Odr_int</entry><entry>NULL
4442 presentStatus</entry><entry>Odr_int</entry><entry>NULL
4445 records</entry><entry>Z_Records</entry><entry>NULL
4448 additionalSearchInfo</entry>
4449 <entry>Z_OtherInformation</entry><entry>NULL
4452 otherInfo</entry><entry>Z_OtherInformation</entry><entry>NULL
4457 <table frame="top" id="asn.default.present.request">
4458 <title>Default settings for PDU Present Request</title>
4460 <colspec colwidth="7*" colname="field"></colspec>
4461 <colspec colwidth="5*" colname="type"></colspec>
4462 <colspec colwidth="7*" colname="value"></colspec>
4465 <entry>Field</entry>
4467 <entry>Default Value</entry>
4472 referenceId</entry><entry>Z_ReferenceId</entry><entry>NULL
4475 resultSetId</entry><entry>char*</entry><entry>"default"
4478 resultSetStartPoint</entry><entry>Odr_int</entry><entry>1
4481 numberOfRecordsRequested</entry><entry>Odr_int</entry><entry>10
4484 num_ranges</entry><entry>Odr_int</entry><entry>0
4487 additionalRanges</entry><entry>Z_Range</entry><entry>NULL
4490 recordComposition</entry><entry>Z_RecordComposition</entry><entry>NULL
4493 preferredRecordSyntax</entry><entry>Odr_oid</entry><entry>NULL
4496 maxSegmentCount</entry><entry>Odr_int</entry><entry>NULL
4499 maxRecordSize</entry><entry>Odr_int</entry><entry>NULL
4502 maxSegmentSize</entry><entry>Odr_int</entry><entry>NULL
4505 otherInfo</entry><entry>Z_OtherInformation</entry><entry>NULL
4510 <table frame="top" id="asn.default.present.response">
4511 <title>Default settings for PDU Present Response</title>
4513 <colspec colwidth="7*" colname="field"></colspec>
4514 <colspec colwidth="5*" colname="type"></colspec>
4515 <colspec colwidth="7*" colname="value"></colspec>
4518 <entry>Field</entry>
4520 <entry>Default Value</entry>
4525 referenceId</entry><entry>Z_ReferenceId</entry><entry>NULL
4528 numberOfRecordsReturned</entry><entry>Odr_int</entry><entry>0
4531 nextResultSetPosition</entry><entry>Odr_int</entry><entry>0
4534 presentStatus</entry><entry>Odr_int</entry><entry>Z_PresentStatus_success
4537 records</entry><entry>Z_Records</entry><entry>NULL
4540 otherInfo</entry><entry>Z_OtherInformation</entry><entry>NULL
4545 <table frame="top" id="asn.default.delete.result.set.request">
4546 <title>Default settings for Delete Result Set Request</title>
4548 <colspec colwidth="7*" colname="field"></colspec>
4549 <colspec colwidth="5*" colname="type"></colspec>
4550 <colspec colwidth="7*" colname="value"></colspec>
4553 <entry>Field</entry>
4555 <entry>Default Value</entry>
4559 <row><entry>referenceId
4560 </entry><entry>Z_ReferenceId</entry><entry>NULL
4563 deleteFunction</entry><entry>Odr_int</entry><entry>Z_DeleteResultSetRequest_list
4566 num_ids</entry><entry>Odr_int</entry><entry>0
4569 resultSetList</entry><entry>char**</entry><entry>NULL
4572 otherInfo</entry><entry>Z_OtherInformation</entry><entry>NULL
4577 <table frame="top" id="asn.default.delete.result.set.response">
4578 <title>Default settings for Delete Result Set Response</title>
4580 <colspec colwidth="7*" colname="field"></colspec>
4581 <colspec colwidth="5*" colname="type"></colspec>
4582 <colspec colwidth="7*" colname="value"></colspec>
4585 <entry>Field</entry>
4587 <entry>Default Value</entry>
4592 referenceId</entry><entry>Z_ReferenceId</entry><entry>NULL
4595 deleteOperationStatus</entry><entry>Odr_int</entry>
4596 <entry>Z_DeleteStatus_success</entry></row>
4598 num_statuses</entry><entry>Odr_int</entry><entry>0
4601 deleteListStatuses</entry><entry>Z_ListStatus**</entry><entry>NULL
4604 numberNotDeleted</entry><entry>Odr_int</entry><entry>NULL
4607 num_bulkStatuses</entry><entry>Odr_int</entry><entry>0
4610 bulkStatuses</entry><entry>Z_ListStatus</entry><entry>NUL
4613 deleteMessage</entry><entry>char*</entry><entry>NULL
4616 otherInfo</entry><entry>Z_OtherInformation</entry><entry>NULL
4621 <table frame="top" id="asn.default.scan.request">
4622 <title>Default settings for Scan Request</title>
4624 <colspec colwidth="7*" colname="field"></colspec>
4625 <colspec colwidth="5*" colname="type"></colspec>
4626 <colspec colwidth="7*" colname="value"></colspec>
4629 <entry>Field</entry>
4631 <entry>Default Value</entry>
4636 referenceId</entry><entry>Z_ReferenceId</entry><entry>NULL
4639 num_databaseNames</entry><entry>Odr_int</entry><entry>0
4642 databaseNames</entry><entry>char**</entry><entry>NULL
4645 attributeSet</entry><entry>Odr_oid</entry><entry>NULL
4648 termListAndStartPoint</entry><entry>Z_AttributesPlus...
4649 </entry><entry>NULL</entry></row>
4651 stepSize</entry><entry>Odr_int</entry><entry>NULL
4654 numberOfTermsRequested</entry><entry>Odr_int</entry><entry>20
4657 preferredPositionInResponse</entry><entry>Odr_int</entry><entry>NULL
4660 otherInfo</entry><entry>Z_OtherInformation</entry><entry>NULL
4665 <table frame="top" id="asn.default.scan.response">
4666 <title>Default settings for Scan Response</title>
4668 <colspec colwidth="7*" colname="field"></colspec>
4669 <colspec colwidth="5*" colname="type"></colspec>
4670 <colspec colwidth="7*" colname="value"></colspec>
4673 <entry>Field</entry>
4675 <entry>Default Value</entry>
4680 referenceId</entry><entry>Z_ReferenceId</entry><entry>NULL
4683 stepSize</entry><entry>Odr_int</entry><entry>NULL
4686 scanStatus</entry><entry>Odr_int</entry><entry>Z_Scan_success
4689 numberOfEntriesReturned</entry><entry>Odr_int</entry><entry>0
4692 positionOfTerm</entry><entry>Odr_int</entry><entry>NULL
4695 entries</entry><entry>Z_ListEntris</entry><entry>NULL
4698 attributeSet</entry><entry>Odr_oid</entry><entry>NULL
4701 otherInfo</entry><entry>Z_OtherInformation</entry><entry>NULL
4706 <table frame="top" id="asn.default.trigger.resource.control.request">
4707 <title>Default settings for Trigger Resource Control Request</title>
4709 <colspec colwidth="7*" colname="field"></colspec>
4710 <colspec colwidth="5*" colname="type"></colspec>
4711 <colspec colwidth="7*" colname="value"></colspec>
4714 <entry>Field</entry>
4716 <entry>Default Value</entry>
4721 referenceId</entry><entry>Z_ReferenceId</entry><entry>NULL
4724 requestedAction</entry><entry>Odr_int</entry><entry>
4725 Z_TriggerResourceCtrl_resou..
4728 prefResourceReportFormat</entry><entry>Odr_oid</entry><entry>NULL
4731 resultSetWanted</entry><entry>Odr_bool</entry><entry>NULL
4734 otherInfo</entry><entry>Z_OtherInformation</entry><entry>NULL
4739 <table frame="top" id="asn.default.resource.control.request">
4740 <title>Default settings for Resource Control Request</title>
4742 <colspec colwidth="7*" colname="field"></colspec>
4743 <colspec colwidth="5*" colname="type"></colspec>
4744 <colspec colwidth="7*" colname="value"></colspec>
4747 <entry>Field</entry>
4749 <entry>Default Value</entry>
4754 referenceId</entry><entry>Z_ReferenceId</entry><entry>NULL
4757 suspendedFlag</entry><entry>Odr_bool</entry><entry>NULL
4760 resourceReport</entry><entry>Z_External</entry><entry>NULL
4763 partialResultsAvailable</entry><entry>Odr_int</entry><entry>NULL
4766 responseRequired</entry><entry>Odr_bool</entry><entry>FALSE
4769 triggeredRequestFlag</entry><entry>Odr_bool</entry><entry>NULL
4772 otherInfo</entry><entry>Z_OtherInformation</entry><entry>NULL
4777 <table frame="top" id="asn.default.resource.control.response">
4778 <title>Default settings for Resource Control Response</title>
4780 <colspec colwidth="7*" colname="field"></colspec>
4781 <colspec colwidth="5*" colname="type"></colspec>
4782 <colspec colwidth="7*" colname="value"></colspec>
4785 <entry>Field</entry>
4787 <entry>Default Value</entry>
4792 referenceId</entry><entry>Z_ReferenceId</entry><entry>NULL
4795 continueFlag</entry><entry>bool_t</entry><entry>TRUE
4798 resultSetWanted</entry><entry>bool_t</entry><entry>NULL
4801 otherInfo</entry><entry>Z_OtherInformation</entry><entry>NULL
4806 <table frame="top" id="asn.default.access.control.request">
4807 <title>Default settings for Access Control Request</title>
4809 <colspec colwidth="7*" colname="field"></colspec>
4810 <colspec colwidth="5*" colname="type"></colspec>
4811 <colspec colwidth="7*" colname="value"></colspec>
4814 <entry>Field</entry>
4816 <entry>Default Value</entry>
4821 referenceId</entry><entry>Z_ReferenceId</entry><entry>NULL
4824 which</entry><entry>enum</entry><entry>Z_AccessRequest_simpleForm;
4827 u</entry><entry>union</entry><entry>NULL
4830 otherInfo</entry><entry>Z_OtherInformation</entry><entry>NULL
4835 <table frame="top" id="asn.default.access.control.response">
4836 <title>Default settings for Access Control Response</title>
4838 <colspec colwidth="7*" colname="field"></colspec>
4839 <colspec colwidth="5*" colname="type"></colspec>
4840 <colspec colwidth="7*" colname="value"></colspec>
4843 <entry>Field</entry>
4845 <entry>Default Value</entry>
4850 referenceId</entry><entry>Z_ReferenceId</entry><entry>NULL
4853 which</entry><entry>enum</entry><entry>Z_AccessResponse_simpleForm
4856 u</entry><entry>union</entry><entry>NULL
4859 diagnostic</entry><entry>Z_DiagRec</entry><entry>NULL
4862 otherInfo</entry><entry>Z_OtherInformation</entry><entry>NULL
4867 <table frame="top" id="asn.default.segment">
4868 <title>Default settings for Segment</title>
4870 <colspec colwidth="7*" colname="field"></colspec>
4871 <colspec colwidth="5*" colname="type"></colspec>
4872 <colspec colwidth="7*" colname="value"></colspec>
4875 <entry>Field</entry>
4877 <entry>Default Value</entry>
4882 referenceId</entry><entry>Z_ReferenceId</entry><entry>NULL
4885 numberOfRecordsReturned</entry><entry>Odr_int</entry><entry>value=0
4888 num_segmentRecords</entry><entry>Odr_int</entry><entry>0
4891 segmentRecords</entry><entry>Z_NamePlusRecord</entry><entry>NULL
4893 <row><entry>otherInfo</entry><entry>Z_OtherInformation</entry><entry>NULL
4898 <table frame="top" id="asn.default.close">
4899 <title>Default settings for Close</title>
4901 <colspec colwidth="7*" colname="field"></colspec>
4902 <colspec colwidth="5*" colname="type"></colspec>
4903 <colspec colwidth="7*" colname="value"></colspec>
4906 <entry>Field</entry>
4908 <entry>Default Value</entry>
4913 referenceId</entry><entry>Z_ReferenceId</entry><entry>NULL
4916 closeReason</entry><entry>Odr_int</entry><entry>Z_Close_finished
4919 diagnosticInformation</entry><entry>char*</entry><entry>NULL
4922 resourceReportFormat</entry><entry>Odr_oid</entry><entry>NULL
4925 resourceFormat</entry><entry>Z_External</entry><entry>NULL
4928 otherInfo</entry><entry>Z_OtherInformation</entry><entry>NULL
4936 <title>SOAP and SRU</title>
4937 <sect1 id="soap.introduction">
4938 <title>Introduction</title>
4940 &yaz; uses a very simple implementation of
4941 <ulink url="&url.soap;">SOAP</ulink> that only,
4942 currenly, supports what is sufficient to offer SRU SOAP functionality.
4943 The implementation uses the
4944 <ulink url="&url.libxml2.api.tree;">tree API</ulink> of
4945 libxml2 to encode and decode SOAP packages.
4948 Like the Z39.50 ASN.1 module, the &yaz; SRU implementation uses
4949 simple C structs to represent SOAP packages as well as
4953 <sect1 id="soap.http">
4956 &yaz; only offers HTTP as transport carrier for SOAP, but it is
4957 relatively easy to change that.
4960 The following definition of <literal>Z_GDU</literal> (Generic Data
4961 Unit) allows for both HTTP and Z39.50 in one packet.
4964 #include <yaz/zgdu.h>
4966 #define Z_GDU_Z3950 1
4967 #define Z_GDU_HTTP_Request 2
4968 #define Z_GDU_HTTP_Response 3
4973 Z_HTTP_Request *HTTP_Request;
4974 Z_HTTP_Response *HTTP_Response;
4979 The corresponding Z_GDU encoder/decoder is <function>z_GDU</function>.
4980 The <literal>z3950</literal> is any of the known BER encoded Z39.50
4982 <literal>HTTP_Request</literal> and <literal>HTTP_Response</literal>
4983 is the HTTP Request and Response respectively.
4986 <sect1 id="soap.xml">
4987 <title>SOAP Packages</title>
4989 Every SOAP package in &yaz; is represented as follows:
4991 #include <yaz/soap.h>
5005 #define Z_SOAP_fault 1
5006 #define Z_SOAP_generic 2
5007 #define Z_SOAP_error 3
5011 Z_SOAP_Fault *fault;
5012 Z_SOAP_Generic *generic;
5013 Z_SOAP_Fault *soap_error;
5020 The <literal>fault</literal> and <literal>soap_error</literal>
5021 arms represent both a SOAP fault - struct
5022 <literal>Z_SOAP_Fault</literal>. Any other generic
5023 (valid) package is represented by <literal>Z_SOAP_Generic</literal>.
5026 The <literal>ns</literal> as part of <literal>Z_SOAP</literal>
5027 is the namespace for SOAP itself and reflects the SOAP
5028 version. For version 1.1 it is
5029 <literal>http://schemas.xmlsoap.org/soap/envelope/</literal>,
5030 for version 1.2 it is
5031 <literal>http://www.w3.org/2001/06/soap-envelope</literal>.
5034 int z_soap_codec(ODR o, Z_SOAP **pp,
5035 char **content_buf, int *content_len,
5036 Z_SOAP_Handler *handlers);
5039 The <literal>content_buf</literal> and <literal>content_len</literal>
5040 is XML buffer and length of buffer respectively.
5043 The <literal>handlers</literal> is a list of SOAP codec
5044 handlers - one handler for each service namespace. For SRU SOAP, the
5045 namespace would be <literal>http://www.loc.gov/zing/srw/v1.0/</literal>.
5048 When decoding, the <function>z_soap_codec</function>
5049 inspects the XML content
5050 and tries to match one of the services namespaces of the
5051 supplied handlers. If there is a match a handler function
5052 is invoked which decodes that particular SOAP package.
5053 If successful, the returned <literal>Z_SOAP</literal> package will be
5054 of type <literal>Z_SOAP_Generic</literal>.
5055 Member <literal>no</literal> is
5056 set the offset of handler that matched; <literal>ns</literal>
5057 is set to namespace of matching handler; the void pointer
5058 <literal>p</literal> is set to the C data structure assocatiated
5062 When a NULL namespace is met (member <literal>ns</literal> bwlow),
5063 that specifies end-of-list.
5066 Each handler is defined as follows:
5074 The <literal>ns</literal> is namespace of service associated with
5075 handler <literal>f</literal>. <literal>client_data</literal>
5076 is user-defined data which is passed to handler.
5079 The prototype for a SOAP service handler is:
5081 int handler(ODR o, void * ptr, void **handler_data,
5082 void *client_data, const char *ns);
5084 The <parameter>o</parameter> specifies the mode (decode/encode)
5085 as usual. The second argument, <parameter>ptr</parameter>,
5086 is a libxml2 tree node pointer (<literal>xmlNodePtr</literal>)
5087 and is a pointer to the <literal>Body</literal> element
5088 of the SOAP package. The <parameter>handler_data</parameter>
5089 is an opaque pointer to a C definitions associated with the
5090 SOAP service. <parameter>client_data</parameter> is the pointer
5091 which was set as part of the <literal>Z_SOAP_handler</literal>.
5092 Finally, <parameter>ns</parameter> the service namespace.
5095 <sect1 id="soap.srw">
5098 SRU SOAP is just one implementation of a SOAP handler as described
5099 in the previous section.
5100 The encoder/decoder handler for SRU is defined as
5103 #include <yaz/srw.h>
5105 int yaz_srw_codec(ODR o, void * pptr,
5106 Z_SRW_GDU **handler_data,
5107 void *client_data, const char *ns);
5109 Here, <literal>Z_SRW_GDU</literal> is either
5110 searchRetrieveRequest or a searchRetrieveResponse.
5114 The xQuery and xSortKeys are not handled yet by
5115 the SRW implementation of &yaz;. Explain is also missing.
5116 Future versions of &yaz; will include these features.
5120 The definition of searchRetrieveRequest is:
5124 #define Z_SRW_query_type_cql 1
5125 #define Z_SRW_query_type_xcql 2
5126 #define Z_SRW_query_type_pqf 3
5134 #define Z_SRW_sort_type_none 1
5135 #define Z_SRW_sort_type_sort 2
5136 #define Z_SRW_sort_type_xSort 3
5144 int *maximumRecords;
5146 char *recordPacking;
5148 } Z_SRW_searchRetrieveRequest;
5150 Please observe that data of type xsd:string is represented
5151 as a char pointer (<literal>char *</literal>). A null pointer
5152 means that the element is absent.
5153 Data of type xsd:integer is representd as a pointer to
5154 an int (<literal>int *</literal>). Again, a null pointer
5155 us used for absent elements.
5158 The SearchRetrieveResponse has the following definition.
5161 int * numberOfRecords;
5163 int * resultSetIdleTime;
5165 Z_SRW_record *records;
5168 Z_SRW_diagnostic *diagnostics;
5169 int num_diagnostics;
5170 int *nextRecordPosition;
5171 } Z_SRW_searchRetrieveResponse;
5173 The <literal>num_records</literal> and <literal>num_diagnostics</literal>
5174 is number of returned records and diagnostics respectively and also
5175 correspond to the "size of" arrays <literal>records</literal>
5176 and <literal>diagnostics</literal>.
5179 A retrieval record is defined as follows:
5183 char *recordData_buf;
5185 int *recordPosition;
5188 The record data is defined as a buffer of some length so that
5189 data can be of any type. SRW 1.0 currenly doesn't allow for this
5190 (only XML), but future versions might do.
5193 And, a diagnostic as:
5203 <chapter id="tools">
5204 <title>Supporting Tools</title>
5206 In support of the service API - primarily the ASN module, which
5207 provides the pro-grammatic interface to the Z39.50 APDUs, &yaz; contains
5208 a collection of tools that support the development of applications.
5210 <sect1 id="tools.query">
5211 <title>Query Syntax Parsers</title>
5213 Since the type-1 (RPN) query structure has no direct, useful string
5214 representation, every origin application needs to provide some form of
5215 mapping from a local query notation or representation to a
5216 <token>Z_RPNQuery</token> structure. Some programmers will prefer to
5217 construct the query manually, perhaps using
5218 <function>odr_malloc()</function> to simplify memory management.
5219 The &yaz; distribution includes three separate, query-generating tools
5220 that may be of use to you.
5223 <title>Prefix Query Format</title>
5225 Since RPN or reverse polish notation is really just a fancy way of
5226 describing a suffix notation format (operator follows operands), it
5227 would seem that the confusion is total when we now introduce a prefix
5228 notation for RPN. The reason is one of simple laziness - it's somewhat
5229 simpler to interpret a prefix format, and this utility was designed
5230 for maximum simplicity, to provide a baseline representation for use
5231 in simple test applications and scripting environments (like Tcl). The
5232 demonstration client included with YAZ uses the PQF.
5236 The PQF have been adopted by other parties developing Z39.50
5237 software. It is often referred to as Prefix Query Notation
5242 The PQF is defined by the pquery module in the YAZ library.
5243 There are two sets of function that have similar behavior. First
5244 set operates on a PQF parser handle, second set doesn't. First set
5245 set of functions are more flexible than the second set. Second set
5246 is obsolete and is only provided to ensure backwards compatibility.
5249 First set of functions all operate on a PQF parser handle:
5252 #include <yaz/pquery.h>
5254 YAZ_PQF_Parser yaz_pqf_create(void);
5256 void yaz_pqf_destroy(YAZ_PQF_Parser p);
5258 Z_RPNQuery *yaz_pqf_parse(YAZ_PQF_Parser p, ODR o, const char *qbuf);
5260 Z_AttributesPlusTerm *yaz_pqf_scan(YAZ_PQF_Parser p, ODR o,
5261 Odr_oid **attributeSetId, const char *qbuf);
5263 int yaz_pqf_error(YAZ_PQF_Parser p, const char **msg, size_t *off);
5266 A PQF parser is created and destructed by functions
5267 <function>yaz_pqf_create</function> and
5268 <function>yaz_pqf_destroy</function> respectively.
5269 Function <function>yaz_pqf_parse</function> parses query given
5270 by string <literal>qbuf</literal>. If parsing was successful,
5271 a Z39.50 RPN Query is returned which is created using ODR stream
5272 <literal>o</literal>. If parsing failed, a NULL pointer is
5274 Function <function>yaz_pqf_scan</function> takes a scan query in
5275 <literal>qbuf</literal>. If parsing was successful, the function
5276 returns attributes plus term pointer and modifies
5277 <literal>attributeSetId</literal> to hold attribute set for the
5278 scan request - both allocated using ODR stream <literal>o</literal>.
5279 If parsing failed, yaz_pqf_scan returns a NULL pointer.
5280 Error information for bad queries can be obtained by a call to
5281 <function>yaz_pqf_error</function> which returns an error code and
5282 modifies <literal>*msg</literal> to point to an error description,
5283 and modifies <literal>*off</literal> to the offset within last
5284 query were parsing failed.
5287 The second set of functions are declared as follows:
5290 #include <yaz/pquery.h>
5292 Z_RPNQuery *p_query_rpn(ODR o, oid_proto proto, const char *qbuf);
5294 Z_AttributesPlusTerm *p_query_scan(ODR o, oid_proto proto,
5295 Odr_oid **attributeSetP, const char *qbuf);
5297 int p_query_attset(const char *arg);
5300 The function <function>p_query_rpn()</function> takes as arguments an
5301 &odr; stream (see section <link linkend="odr">The ODR Module</link>)
5302 to provide a memory source (the structure created is released on
5303 the next call to <function>odr_reset()</function> on the stream), a
5304 protocol identifier (one of the constants <token>PROTO_Z3950</token> and
5305 <token>PROTO_SR</token>), an attribute set reference, and
5306 finally a null-terminated string holding the query string.
5309 If the parse went well, <function>p_query_rpn()</function> returns a
5310 pointer to a <literal>Z_RPNQuery</literal> structure which can be
5311 placed directly into a <literal>Z_SearchRequest</literal>.
5312 If parsing failed, due to syntax error, a NULL pointer is returned.
5315 The <literal>p_query_attset</literal> specifies which attribute set
5316 to use if the query doesn't specify one by the
5317 <literal>@attrset</literal> operator.
5318 The <literal>p_query_attset</literal> returns 0 if the argument is a
5319 valid attribute set specifier; otherwise the function returns -1.
5322 The grammar of the PQF is as follows:
5325 query ::= top-set query-struct.
5327 top-set ::= [ '@attrset' string ]
5329 query-struct ::= attr-spec | simple | complex | '@term' term-type query
5331 attr-spec ::= '@attr' [ string ] string query-struct
5333 complex ::= operator query-struct query-struct.
5335 operator ::= '@and' | '@or' | '@not' | '@prox' proximity.
5337 simple ::= result-set | term.
5339 result-set ::= '@set' string.
5343 proximity ::= exclusion distance ordered relation which-code unit-code.
5345 exclusion ::= '1' | '0' | 'void'.
5347 distance ::= integer.
5349 ordered ::= '1' | '0'.
5351 relation ::= integer.
5353 which-code ::= 'known' | 'private' | integer.
5355 unit-code ::= integer.
5357 term-type ::= 'general' | 'numeric' | 'string' | 'oid' | 'datetime' | 'null'.
5360 You will note that the syntax above is a fairly faithful
5361 representation of RPN, except for the Attribute, which has been
5362 moved a step away from the term, allowing you to associate one or more
5363 attributes with an entire query structure. The parser will
5364 automatically apply the given attributes to each term as required.
5367 The @attr operator is followed by an attribute specification
5368 (<literal>attr-spec</literal> above). The specification consists
5369 of an optional attribute set, an attribute type-value pair and
5370 a sub-query. The attribute type-value pair is packed in one string:
5371 an attribute type, an equals sign, and an attribute value, like this:
5372 <literal>@attr 1=1003</literal>.
5373 The type is always an integer but the value may be either an
5374 integer or a string (if it doesn't start with a digit character).
5375 A string attribute-value is encoded as a Type-1 ``complex''
5376 attribute with the list of values containing the single string
5377 specified, and including no semantic indicators.
5380 Version 3 of the Z39.50 specification defines various encoding of terms.
5381 Use <literal>@term </literal> <replaceable>type</replaceable>
5382 <replaceable>string</replaceable>,
5383 where type is one of: <literal>general</literal>,
5384 <literal>numeric</literal> or <literal>string</literal>
5385 (for InternationalString).
5386 If no term type has been given, the <literal>general</literal> form
5387 is used. This is the only encoding allowed in both versions 2 and 3
5388 of the Z39.50 standard.
5390 <sect3 id="PQF-prox">
5391 <title>Using Proximity Operators with PQF</title>
5394 This is an advanced topic, describing how to construct
5395 queries that make very specific requirements on the
5396 relative location of their operands.
5397 You may wish to skip this section and go straight to
5398 <link linkend="pqf-examples">the example PQF queries</link>.
5403 Most Z39.50 servers do not support proximity searching, or
5404 support only a small subset of the full functionality that
5405 can be expressed using the PQF proximity operator. Be
5406 aware that the ability to <emphasis>express</emphasis> a
5407 query in PQF is no guarantee that any given server will
5408 be able to <emphasis>execute</emphasis> it.
5414 The proximity operator <literal>@prox</literal> is a special
5415 and more restrictive version of the conjunction operator
5416 <literal>@and</literal>. Its semantics are described in
5417 section 3.7.2 (Proximity) of Z39.50 the standard itself, which
5418 can be read on-line at
5419 <ulink url="&url.z39.50.proximity;"/>
5422 In PQF, the proximity operation is represented by a sequence
5425 @prox <replaceable>exclusion</replaceable> <replaceable>distance</replaceable> <replaceable>ordered</replaceable> <replaceable>relation</replaceable> <replaceable>which-code</replaceable> <replaceable>unit-code</replaceable>
5427 in which the meanings of the parameters are as described in in
5428 the standard, and they can take the following values:
5431 <formalpara><title>exclusion</title>
5433 0 = false (i.e. the proximity condition specified by the
5434 remaining parameters must be satisfied) or
5435 1 = true (the proximity condition specified by the
5436 remaining parameters must <emphasis>not</emphasis> be
5442 <formalpara><title>distance</title><para>
5443 An integer specifying the difference between the locations
5444 of the operands: e.g. two adjacent words would have
5445 distance=1 since their locations differ by one unit.
5447 </formalpara></listitem>
5449 <formalpara><title>ordered</title><para>
5450 1 = ordered (the operands must occur in the order the
5451 query specifies them) or
5452 0 = unordered (they may appear in either order).
5457 <formalpara><title>relation</title><para>
5458 Recognised values are
5460 2 (lessThanOrEqual),
5462 4 (greaterThanOrEqual),
5469 <formalpara><title>which-code</title><para>
5470 <literal>known</literal>
5472 <literal>k</literal>
5473 (the unit-code parameter is taken from the well-known list
5474 of alternatives described in below) or
5475 <literal>private</literal>
5477 <literal>p</literal>
5478 (the unit-code paramater has semantics specific to an
5479 out-of-band agreement such as a profile).
5484 <formalpara><title>unit-code</title><para>
5485 If the which-code parameter is <literal>known</literal>
5486 then the recognised values are
5496 10 (elementType) and
5498 If which-code is <literal>private</literal> then the
5499 acceptable values are determined by the profile.
5504 (The numeric values of the relation and well-known unit-code
5505 parameters are taken straight from
5506 <ulink url="&url.z39.50.proximity.asn1;"
5507 >the ASN.1</ulink> of the proximity structure in the standard.)
5510 <sect3 id="pqf-examples">
5511 <title>PQF queries</title>
5512 <example id="example.pqf.simple.terms">
5513 <title>PQF queries using simple terms</title>
5522 <example id="pqf.example.pqf.boolean.operators">
5523 <title>PQF boolean operators</title>
5526 @or "dylan" "zimmerman"
5528 @and @or dylan zimmerman when
5530 @and when @or dylan zimmerman
5534 <example id="example.pqf.result.sets">
5535 <title>PQF references to result sets</title>
5540 @and @set seta @set setb
5544 <example id="example.pqf.attributes">
5545 <title>Attributes for terms</title>
5550 @attr 1=4 @attr 4=1 "self portrait"
5552 @attrset exp1 @attr 1=1 CategoryList
5554 @attr gils 1=2008 Copenhagen
5556 @attr 1=/book/title computer
5560 <example id="example.pqf.proximity">
5561 <title>PQF Proximity queries</title>
5564 @prox 0 3 1 2 k 2 dylan zimmerman
5566 Here the parameters 0, 3, 1, 2, k and 2 represent exclusion,
5567 distance, ordered, relation, which-code and unit-code, in that
5571 <para>exclusion = 0: the proximity condition must hold</para>
5574 <para>distance = 3: the terms must be three units apart</para>
5578 ordered = 1: they must occur in the order they are specified
5583 relation = 2: lessThanOrEqual (to the distance of 3 units)
5588 which-code is ``known'', so the standard unit-codes are used
5592 <para>unit-code = 2: word.</para>
5595 So the whole proximity query means that the words
5596 <literal>dylan</literal> and <literal>zimmerman</literal> must
5597 both occur in the record, in that order, differing in position
5598 by three or fewer words (i.e. with two or fewer words between
5599 them.) The query would find ``Bob Dylan, aka. Robert
5600 Zimmerman'', but not ``Bob Dylan, born as Robert Zimmerman''
5601 since the distance in this case is four.
5604 <example id="example.pqf.search.term.type">
5605 <title>PQF specification of search term type</title>
5608 @term string "a UTF-8 string, maybe?"
5612 <example id="example.pqf.mixed.queries">
5613 <title>PQF mixed queries</title>
5616 @or @and bob dylan @set Result-1
5618 @attr 4=1 @and @attr 1=1 "bob dylan" @attr 1=4 "slow train coming"
5620 @and @attr 2=4 @attr gils 1=2038 -114 @attr 2=2 @attr gils 1=2039 -109
5622 The last of these examples is a spatial search: in
5623 <ulink url="http://www.gils.net/prof_v2.html#sec_7_4"
5624 >the GILS attribute set</ulink>,
5626 2038 indicates West Bounding Coordinate and
5627 2030 indicates East Bounding Coordinate,
5628 so the query is for areas extending from -114 degrees
5629 to no more than -109 degrees.
5634 <sect2 id="CCL"><title>CCL</title>
5636 Not all users enjoy typing in prefix query structures and numerical
5637 attribute values, even in a minimalistic test client. In the library
5638 world, the more intuitive Common Command Language - CCL (ISO 8777)
5639 has enjoyed some popularity - especially before the widespread
5640 availability of graphical interfaces. It is still useful in
5641 applications where you for some reason or other need to provide a
5642 symbolic language for expressing boolean query structures.
5644 <sect3 id="ccl.syntax">
5645 <title>CCL Syntax</title>
5647 The CCL parser obeys the following grammar for the FIND argument.
5648 The syntax is annotated by in the lines prefixed by
5649 <literal>--</literal>.
5652 CCL-Find ::= CCL-Find Op Elements
5655 Op ::= "and" | "or" | "not"
5656 -- The above means that Elements are separated by boolean operators.
5658 Elements ::= '(' CCL-Find ')'
5661 | Qualifiers Relation Terms
5662 | Qualifiers Relation '(' CCL-Find ')'
5663 | Qualifiers '=' string '-' string
5664 -- Elements is either a recursive definition, a result set reference, a
5665 -- list of terms, qualifiers followed by terms, qualifiers followed
5666 -- by a recursive definition or qualifiers in a range (lower - upper).
5668 Set ::= 'set' = string
5669 -- Reference to a result set
5671 Terms ::= Terms Prox Term
5673 -- Proximity of terms.
5675 Term ::= Term string
5677 -- This basically means that a term may include a blank
5679 Qualifiers ::= Qualifiers ',' string
5681 -- Qualifiers is a list of strings separated by comma
5683 Relation ::= '=' | '>=' | '<=' | '<>' | '>' | '<'
5684 -- Relational operators. This really doesn't follow the ISO8777
5688 -- Proximity operator
5691 <example id="example.ccl.queries">
5692 <title>CCL queries</title>
5694 The following queries are all valid:
5705 (dylan and bob) or set=1
5714 Assuming that the qualifiers <literal>ti</literal>,
5715 <literal>au</literal>
5716 and <literal>date</literal> are defined we may use:
5721 au=(bob dylan and slow train coming)
5723 date>1980 and (ti=((self portrait)))
5727 <sect3 id="ccl.qualifiers">
5728 <title>CCL Qualifiers</title>
5730 Qualifiers are used to direct the search to a particular searchable
5731 index, such as title (ti) and author indexes (au). The CCL standard
5732 itself doesn't specify a particular set of qualifiers, but it does
5733 suggest a few short-hand notations. You can customize the CCL parser
5734 to support a particular set of qualifiers to reflect the current target
5735 profile. Traditionally, a qualifier would map to a particular
5736 use-attribute within the BIB-1 attribute set. It is also
5737 possible to set other attributes, such as the structure
5741 A CCL profile is a set of predefined CCL qualifiers that may be
5742 read from a file or set in the CCL API.
5743 The YAZ client reads its CCL qualifiers from a file named
5744 <filename>default.bib</filename>. There are four types of
5745 lines in a CCL profile: qualifier specification,
5746 qualifier alias, comments and directives.
5748 <sect4 id="ccl.qualifier.specification">
5749 <title>Qualifier specification</title>
5751 A qualifier specification is of the form:
5754 <replaceable>qualifier-name</replaceable>
5755 [<replaceable>attributeset</replaceable><literal>,</literal>]<replaceable>type</replaceable><literal>=</literal><replaceable>val</replaceable>
5756 [<replaceable>attributeset</replaceable><literal>,</literal>]<replaceable>type</replaceable><literal>=</literal><replaceable>val</replaceable> ...
5759 where <replaceable>qualifier-name</replaceable> is the name of the
5760 qualifier to be used (eg. <literal>ti</literal>),
5761 <replaceable>type</replaceable> is attribute type in the attribute
5762 set (Bib-1 is used if no attribute set is given) and
5763 <replaceable>val</replaceable> is attribute value.
5764 The <replaceable>type</replaceable> can be specified as an
5765 integer or as it be specified either as a single-letter:
5766 <literal>u</literal> for use,
5767 <literal>r</literal> for relation,<literal>p</literal> for position,
5768 <literal>s</literal> for structure,<literal>t</literal> for truncation
5769 or <literal>c</literal> for completeness.
5770 The attributes for the special qualifier name <literal>term</literal>
5771 are used when no CCL qualifier is given in a query.
5772 <table id="ccl.common.bib1.attributes">
5773 <title>Common Bib-1 attributes</title>
5775 <colspec colwidth="2*" colname="type"></colspec>
5776 <colspec colwidth="9*" colname="description"></colspec>
5780 <entry>Description</entry>
5785 <entry><literal>u=</literal><replaceable>value</replaceable></entry>
5787 Use attribute (1). Common use attributes are
5788 1 Personal-name, 4 Title, 7 ISBN, 8 ISSN, 30 Date,
5789 62 Subject, 1003 Author), 1016 Any. Specify value
5794 <entry><literal>r=</literal><replaceable>value</replaceable></entry>
5796 Relation attribute (2). Common values are
5797 1 <, 2 <=, 3 =, 4 >=, 5 >, 6 <>,
5798 100 phonetic, 101 stem, 102 relevance, 103 always matches.
5802 <entry><literal>p=</literal><replaceable>value</replaceable></entry>
5804 Position attribute (3). Values: 1 first in field, 2
5805 first in any subfield, 3 any position in field.
5809 <entry><literal>s=</literal><replaceable>value</replaceable></entry>
5811 Structure attribute (4). Values: 1 phrase, 2 word,
5812 3 key, 4 year, 5 date, 6 word list, 100 date (un),
5813 101 name (norm), 102 name (un), 103 structure, 104 urx,
5814 105 free-form-text, 106 document-text, 107 local-number,
5815 108 string, 109 numeric string.
5819 <entry><literal>t=</literal><replaceable>value</replaceable></entry>
5821 Truncation attribute (5). Values: 1 right, 2 left,
5822 3 left& right, 100 none, 101 process #, 102 regular-1,
5823 103 regular-2, 104 CCL.
5827 <entry><literal>c=</literal><replaceable>value</replaceable></entry>
5829 Completeness attribute (6). Values: 1 incomplete subfield,
5830 2 complete subfield, 3 complete field.
5838 Refer to <xref linkend="bib1"/> or the complete
5839 <ulink url="&url.z39.50.attset.bib1;">list of Bib-1 attributes</ulink>
5842 It is also possible to specify non-numeric attribute values,
5843 which are used in combination with certain types.
5844 The special combinations are:
5845 <table id="ccl.special.attribute.combos">
5846 <title>Special attribute combos</title>
5848 <colspec colwidth="2*" colname="name"></colspec>
5849 <colspec colwidth="9*" colname="description"></colspec>
5853 <entry>Description</entry>
5858 <entry><literal>s=pw</literal></entry>
5860 The structure is set to either word or phrase depending
5861 on the number of tokens in a term (phrase-word).
5865 <entry><literal>s=al</literal></entry>
5867 Each token in the term is ANDed. (and-list).
5868 This does not set the structure at all.
5871 <row><entry><literal>s=ol</literal></entry>
5873 Each token in the term is ORed. (or-list).
5874 This does not set the structure at all.
5877 <row><entry><literal>s=ag</literal></entry>
5879 Tokens that appears as phrases (with blank in them) gets
5880 structure phrase attached (4=1). Tokens that appear to be words
5881 gets structure word attached (4=2). Phrases and words are
5882 ANDed. This is a variant of s=al and s=pw, with the main
5883 difference that words are not split (with operator AND)
5884 but instead kept in one RPN token. This facility appeared
5888 <row><entry><literal>s=sl</literal></entry>
5890 Tokens are split into sub-phrases of all combinations - in order.
5891 This facility appeared in YAZ 5.14.0.
5894 <row><entry><literal>r=o</literal></entry>
5896 Allows ranges and the operators greather-than, less-than, ...
5898 This sets Bib-1 relation attribute accordingly (relation
5899 ordered). A query construct is only treated as a range if
5900 dash is used and that is surrounded by white-space. So
5901 <literal>-1980</literal> is treated as term
5902 <literal>"-1980"</literal> not <literal><= 1980</literal>.
5903 If <literal>- 1980</literal> is used, however, that is
5907 <row><entry><literal>r=r</literal></entry>
5909 Similar to <literal>r=o</literal> but assumes that terms
5910 are non-negative (not prefixed with <literal>-</literal>).
5911 Thus, a dash will always be treated as a range.
5912 The construct <literal>1980-1990</literal> is
5913 treated as a range with <literal>r=r</literal> but as a
5914 single term <literal>"1980-1990"</literal> with
5915 <literal>r=o</literal>. The special attribute
5916 <literal>r=r</literal> is available in YAZ 2.0.24 or later.
5919 <row><entry><literal>r=omiteq</literal></entry>
5921 This will omit relation=equals (@attr 2=3) when r=o / r=r
5922 is used. This is useful for servers that somehow breaks
5923 when an explicit relation=equals is used. Omitting the
5924 relation is usually safe because "equals" is the default
5925 behavior. This tweak was added in YAZ version 5.1.2.
5928 <row><entry><literal>t=l</literal></entry>
5930 Allows term to be left-truncated.
5931 If term is of the form <literal>?x</literal>, the resulting
5932 Type-1 term is <literal>x</literal> and truncation is left.
5935 <row><entry><literal>t=r</literal></entry>
5937 Allows term to be right-truncated.
5938 If term is of the form <literal>x?</literal>, the resulting
5939 Type-1 term is <literal>x</literal> and truncation is right.
5942 <row><entry><literal>t=n</literal></entry>
5944 If term is does not include <literal>?</literal>, the
5945 truncation attribute is set to none (100).
5948 <row><entry><literal>t=b</literal></entry>
5950 Allows term to be both left&right truncated.
5951 If term is of the form <literal>?x?</literal>, the
5952 resulting term is <literal>x</literal> and trunctation is
5953 set to both left&right.
5956 <row><entry><literal>t=x</literal></entry>
5958 Allows masking anywhere in a term, thus fully supporting
5959 # (mask one character) and ? (zero or more of any).
5960 If masking is used, trunction is set to 102 (regexp-1 in term)
5961 and the term is converted accordingly to a regular expression.
5964 <row><entry><literal>t=z</literal></entry>
5966 Allows masking anywhere in a term, thus fully supporting
5967 # (mask one character) and ? (zero or more of any).
5968 If masking is used, trunction is set to 104 (Z39.58 in term)
5969 and the term is converted accordingly to Z39.58 masking term -
5970 actually the same truncation as CCL itself.
5977 <example id="example.ccl.profile">
5978 <title>CCL profile</title>
5980 Consider the following definition:
5990 <literal>ti</literal> and <literal>au</literal> both set
5991 structure attribute to phrase (s=1).
5992 <literal>ti</literal>
5993 sets the use-attribute to 4. <literal>au</literal> sets the
5995 When no qualifiers are used in the query the structure-attribute is
5996 set to free-form-text (105) (rule for <literal>term</literal>).
5997 The <literal>date</literal> sets the relation attribute to
5998 the relation used in the CCL query and sets the use attribute
6002 You can combine attributes. To Search for "ranked title" you
6005 ti,ranked=knuth computer
6007 which will set relation=ranked, use=title, structure=phrase.
6014 is a valid query. But
6022 <sect4 id="ccl.qualifier.alias">
6023 <title>Qualifier alias</title>
6025 A qualifier alias is of the form:
6028 <replaceable>q</replaceable>
6029 <replaceable>q1</replaceable> <replaceable>q2</replaceable> ..
6032 which declares <replaceable>q</replaceable> to
6033 be an alias for <replaceable>q1</replaceable>,
6034 <replaceable>q2</replaceable>... such that the CCL
6035 query <replaceable>q=x</replaceable> is equivalent to
6036 <replaceable>q1=x or q2=x or ...</replaceable>.
6039 <sect4 id="ccl.comments">
6040 <title>Comments</title>
6042 Lines with white space or lines that begin with
6043 character <literal>#</literal> are treated as comments.
6046 <sect4 id="ccl.directives">
6047 <title>Directives</title>
6049 Directive specifications takes the form
6051 <para><literal>@</literal><replaceable>directive</replaceable> <replaceable>value</replaceable>
6053 <table id="ccl.directives.table">
6054 <title>CCL directives</title>
6056 <colspec colwidth="2*" colname="name"></colspec>
6057 <colspec colwidth="8*" colname="description"></colspec>
6058 <colspec colwidth="1*" colname="default"></colspec>
6062 <entry>Description</entry>
6063 <entry>Default</entry>
6068 <entry>truncation</entry>
6069 <entry>Truncation character</entry>
6070 <entry><literal>?</literal></entry>
6074 <entry>Masking character. Requires YAZ 4.2.58 or later</entry>
6075 <entry><literal>#</literal></entry>
6078 <entry>field</entry>
6079 <entry>Specifies how multiple fields are to be
6080 combined. There are two modes: <literal>or</literal>:
6081 multiple qualifier fields are ORed,
6082 <literal>merge</literal>: attributes for the qualifier
6083 fields are merged and assigned to one term.
6085 <entry><literal>merge</literal></entry>
6089 <entry>Specifies if CCL operators and qualifiers should be
6090 compared with case sensitivity or not. Specify 1 for
6091 case sensitive; 0 for case insensitive.</entry>
6092 <entry><literal>1</literal></entry>
6096 <entry>Specifies token for CCL operator AND.</entry>
6097 <entry><literal>and</literal></entry>
6101 <entry>Specifies token for CCL operator OR.</entry>
6102 <entry><literal>or</literal></entry>
6106 <entry>Specifies token for CCL operator NOT.</entry>
6107 <entry><literal>not</literal></entry>
6111 <entry>Specifies token for CCL operator SET.</entry>
6112 <entry><literal>set</literal></entry>
6119 <sect3 id="ccl.api">
6120 <title>CCL API</title>
6122 All public definitions can be found in the header file
6123 <filename>ccl.h</filename>. A profile identifier is of type
6124 <literal>CCL_bibset</literal>. A profile must be created with the call
6125 to the function <function>ccl_qual_mk</function> which returns a profile
6126 handle of type <literal>CCL_bibset</literal>.
6129 To read a file containing qualifier definitions the function
6130 <function>ccl_qual_file</function> may be convenient. This function
6131 takes an already opened <literal>FILE</literal> handle pointer as
6132 argument along with a <literal>CCL_bibset</literal> handle.
6135 To parse a simple string with a FIND query use the function
6138 struct ccl_rpn_node *ccl_find_str(CCL_bibset bibset, const char *str,
6139 int *error, int *pos);
6142 which takes the CCL profile (<literal>bibset</literal>) and query
6143 (<literal>str</literal>) as input. Upon successful completion the RPN
6144 tree is returned. If an error occur, such as a syntax error, the integer
6145 pointed to by <literal>error</literal> holds the error code and
6146 <literal>pos</literal> holds the offset inside query string in which
6150 An English representation of the error may be obtained by calling
6151 the <literal>ccl_err_msg</literal> function. The error codes are
6152 listed in <filename>ccl.h</filename>.
6155 To convert the CCL RPN tree (type
6156 <literal>struct ccl_rpn_node *</literal>)
6157 to the Z_RPNQuery of YAZ the function <function>ccl_rpn_query</function>
6158 must be used. This function which is part of YAZ is implemented in
6159 <filename>yaz-ccl.c</filename>.
6160 After calling this function the CCL RPN tree is probably no longer
6161 needed. The <literal>ccl_rpn_delete</literal> destroys the CCL RPN tree.
6164 A CCL profile may be destroyed by calling the
6165 <function>ccl_qual_rm</function> function.
6168 The token names for the CCL operators may be changed by setting the
6169 globals (all type <literal>char *</literal>)
6170 <literal>ccl_token_and</literal>, <literal>ccl_token_or</literal>,
6171 <literal>ccl_token_not</literal> and <literal>ccl_token_set</literal>.
6172 An operator may have aliases, i.e. there may be more than one name for
6173 the operator. To do this, separate each alias with a space character.
6180 <ulink url="&url.cql;">CQL</ulink>
6181 - Common Query Language - was defined for the
6182 <ulink url="&url.sru;">SRU</ulink> protocol.
6183 In many ways CQL has a similar syntax to CCL.
6184 The objective of CQL is different. Where CCL aims to be
6185 an end-user language, CQL is <emphasis>the</emphasis> protocol
6186 query language for SRU.
6190 If you are new to CQL, read the
6191 <ulink url="&url.cql.intro;">Gentle Introduction</ulink>.
6195 The CQL parser in &yaz; provides the following:
6199 It parses and validates a CQL query.
6204 It generates a C structure that allows you to convert
6205 a CQL query to some other query language, such as SQL.
6210 The parser converts a valid CQL query to PQF, thus providing a
6211 way to use CQL for both SRU servers and Z39.50 targets at the
6217 The parser converts CQL to XCQL.
6218 XCQL is an XML representation of CQL.
6219 XCQL is part of the SRU specification. However, since SRU
6220 supports CQL only, we don't expect XCQL to be widely used.
6221 Furthermore, CQL has the advantage over XCQL that it is
6227 <sect3 id="cql.parsing">
6228 <title>CQL parsing</title>
6230 A CQL parser is represented by the <literal>CQL_parser</literal>
6231 handle. Its contents should be considered &yaz; internal (private).
6233 #include <yaz/cql.h>
6235 typedef struct cql_parser *CQL_parser;
6237 CQL_parser cql_parser_create(void);
6238 void cql_parser_destroy(CQL_parser cp);
6240 A parser is created by <function>cql_parser_create</function> and
6241 is destroyed by <function>cql_parser_destroy</function>.
6244 To parse a CQL query string, the following function
6247 int cql_parser_string(CQL_parser cp, const char *str);
6249 A CQL query is parsed by the <function>cql_parser_string</function>
6250 which takes a query <parameter>str</parameter>.
6251 If the query was valid (no syntax errors), then zero is returned;
6252 otherwise -1 is returned to indicate a syntax error.
6256 int cql_parser_stream(CQL_parser cp,
6257 int (*getbyte)(void *client_data),
6258 void (*ungetbyte)(int b, void *client_data),
6261 int cql_parser_stdio(CQL_parser cp, FILE *f);
6263 The functions <function>cql_parser_stream</function> and
6264 <function>cql_parser_stdio</function> parses a CQL query
6265 - just like <function>cql_parser_string</function>.
6266 The only difference is that the CQL query can be
6267 fed to the parser in different ways.
6268 The <function>cql_parser_stream</function> uses a generic
6269 byte stream as input. The <function>cql_parser_stdio</function>
6270 uses a <literal>FILE</literal> handle which is opened for reading.
6273 <sect3 id="cql.tree">
6274 <title>CQL tree</title>
6276 The the query string is valid, the CQL parser
6277 generates a tree representing the structure of the
6282 struct cql_node *cql_parser_result(CQL_parser cp);
6284 <function>cql_parser_result</function> returns the
6285 a pointer to the root node of the resulting tree.
6288 Each node in a CQL tree is represented by a
6289 <literal>struct cql_node</literal>.
6290 It is defined as follows:
6292 #define CQL_NODE_ST 1
6293 #define CQL_NODE_BOOL 2
6294 #define CQL_NODE_SORT 3
6304 struct cql_node *modifiers;
6308 struct cql_node *left;
6309 struct cql_node *right;
6310 struct cql_node *modifiers;
6314 struct cql_node *next;
6315 struct cql_node *modifiers;
6316 struct cql_node *search;
6321 There are three node types: search term (ST), boolean (BOOL)
6323 A modifier is treated as a search term too.
6326 The search term node has five members:
6330 <literal>index</literal>: index for search term.
6331 If an index is unspecified for a search term,
6332 <literal>index</literal> will be NULL.
6337 <literal>index_uri</literal>: index URi for search term
6338 or NULL if none could be resolved for the index.
6343 <literal>term</literal>: the search term itself.
6348 <literal>relation</literal>: relation for search term.
6353 <literal>relation_uri</literal>: relation URI for search term.
6358 <literal>modifiers</literal>: relation modifiers for search
6359 term. The <literal>modifiers</literal> list itself of cql_nodes
6360 each of type <literal>ST</literal>.
6366 The boolean node represents <literal>and</literal>,
6367 <literal>or</literal>, <literal>not</literal> +
6372 <literal>left</literal> and <literal>right</literal>: left
6373 - and right operand respectively.
6378 <literal>modifiers</literal>: proximity arguments.
6384 The sort node represents both the SORTBY clause.
6387 <sect3 id="cql.to.pqf">
6388 <title>CQL to PQF conversion</title>
6390 Conversion to PQF (and Z39.50 RPN) is tricky by the fact
6391 that the resulting RPN depends on the Z39.50 target
6392 capabilities (combinations of supported attributes).
6393 In addition, the CQL and SRU operates on index prefixes
6394 (URI or strings), whereas the RPN uses Object Identifiers
6398 The CQL library of &yaz; defines a <literal>cql_transform_t</literal>
6399 type. It represents a particular mapping between CQL and RPN.
6400 This handle is created and destroyed by the functions:
6402 cql_transform_t cql_transform_open_FILE (FILE *f);
6403 cql_transform_t cql_transform_open_fname(const char *fname);
6404 void cql_transform_close(cql_transform_t ct);
6406 The first two functions create a tranformation handle from
6407 either an already open FILE or from a filename respectively.
6410 The handle is destroyed by <function>cql_transform_close</function>
6411 in which case no further reference of the handle is allowed.
6414 When a <literal>cql_transform_t</literal> handle has been created
6415 you can convert to RPN.
6417 int cql_transform_buf(cql_transform_t ct,
6418 struct cql_node *cn, char *out, int max);
6420 This function converts the CQL tree <literal>cn</literal>
6421 using handle <literal>ct</literal>.
6422 For the resulting PQF, you supply a buffer <literal>out</literal>
6423 which must be able to hold at at least <literal>max</literal>
6427 If conversion failed, <function>cql_transform_buf</function>
6428 returns a non-zero SRU error code; otherwise zero is returned
6429 (conversion successful). The meanings of the numeric error
6430 codes are listed in the SRU specification somewhere (no
6431 direct link anymore).
6434 If conversion fails, more information can be obtained by calling
6436 int cql_transform_error(cql_transform_t ct, char **addinfop);
6438 This function returns the most recently returned numeric
6439 error-code and sets the string-pointer at
6440 <literal>*addinfop</literal> to point to a string containing
6441 additional information about the error that occurred: for
6442 example, if the error code is 15 (``Illegal or unsupported context
6443 set''), the additional information is the name of the requested
6444 context set that was not recognised.
6447 The SRU error-codes may be translated into brief human-readable
6448 error messages using
6450 const char *cql_strerror(int code);
6454 If you wish to be able to produce a PQF result in a different
6455 way, there are two alternatives.
6457 void cql_transform_pr(cql_transform_t ct,
6458 struct cql_node *cn,
6459 void (*pr)(const char *buf, void *client_data),
6462 int cql_transform_FILE(cql_transform_t ct,
6463 struct cql_node *cn, FILE *f);
6465 The former function produces output to a user-defined
6466 output stream. The latter writes the result to an already
6467 open <literal>FILE</literal>.
6470 <sect3 id="cql.to.rpn">
6471 <title>Specification of CQL to RPN mappings</title>
6473 The file supplied to functions
6474 <function>cql_transform_open_FILE</function>,
6475 <function>cql_transform_open_fname</function> follows
6476 a structure found in many Unix utilities.
6477 It consists of mapping specifications - one per line.
6478 Lines starting with <literal>#</literal> are ignored (comments).
6481 Each line is of the form
6483 <replaceable>CQL pattern</replaceable><literal> = </literal> <replaceable> RPN equivalent</replaceable>
6487 An RPN pattern is a simple attribute list. Each attribute pair
6490 [<replaceable>set</replaceable>] <replaceable>type</replaceable><literal>=</literal><replaceable>value</replaceable>
6492 The attribute <replaceable>set</replaceable> is optional.
6493 The <replaceable>type</replaceable> is the attribute type,
6494 <replaceable>value</replaceable> the attribute value.
6497 The character <literal>*</literal> (asterisk) has special meaning
6498 when used in the RPN pattern.
6499 Each occurrence of <literal>*</literal> is substituted with the
6500 CQL matching name (index, relation, qualifier etc).
6501 This facility can be used to copy a CQL name verbatim to the RPN result.
6504 The following CQL patterns are recognized:
6508 <literal>index.</literal><replaceable>set</replaceable><literal>.</literal><replaceable>name</replaceable>
6512 This pattern is invoked when a CQL index, such as
6513 dc.title is converted. <replaceable>set</replaceable>
6514 and <replaceable>name</replaceable> are the context set and index
6516 Typically, the RPN specifies an equivalent use attribute.
6519 For terms not bound by an index the pattern
6520 <literal>index.cql.serverChoice</literal> is used.
6521 Here, the prefix <literal>cql</literal> is defined as
6522 <literal>http://www.loc.gov/zing/cql/cql-indexes/v1.0/</literal>.
6523 If this pattern is not defined, the mapping will fail.
6527 <literal>index.</literal><replaceable>set</replaceable><literal>.*</literal>
6528 is used when no other index pattern is matched.
6534 <literal>qualifier.</literal><replaceable>set</replaceable><literal>.</literal><replaceable>name</replaceable>
6539 For backwards compatibility, this is recognised as a synonym of
6540 <literal>index.</literal><replaceable>set</replaceable><literal>.</literal><replaceable>name</replaceable>
6546 <literal>relation.</literal><replaceable>relation</replaceable>
6550 This pattern specifies how a CQL relation is mapped to RPN.
6551 <replaceable>pattern</replaceable> is name of relation
6552 operator. Since <literal>=</literal> is used as
6553 separator between CQL pattern and RPN, CQL relations
6554 including <literal>=</literal> cannot be
6555 used directly. To avoid a conflict, the names
6556 <literal>ge</literal>,
6557 <literal>eq</literal>,
6558 <literal>le</literal>,
6559 must be used for CQL operators, greater-than-or-equal,
6560 equal, less-than-or-equal respectively.
6561 The RPN pattern is supposed to include a relation attribute.
6564 For terms not bound by a relation, the pattern
6565 <literal>relation.scr</literal> is used. If the pattern
6566 is not defined, the mapping will fail.
6569 The special pattern, <literal>relation.*</literal> is used
6570 when no other relation pattern is matched.
6576 <literal>relationModifier.</literal><replaceable>mod</replaceable>
6580 This pattern specifies how a CQL relation modifier is mapped to RPN.
6581 The RPN pattern is usually a relation attribute.
6587 <literal>structure.</literal><replaceable>type</replaceable>
6591 This pattern specifies how a CQL structure is mapped to RPN.
6592 Note that this CQL pattern is somewhat to similar to
6593 CQL pattern <literal>relation</literal>.
6594 The <replaceable>type</replaceable> is a CQL relation.
6597 The pattern, <literal>structure.*</literal> is used
6598 when no other structure pattern is matched.
6599 Usually, the RPN equivalent specifies a structure attribute.
6605 <literal>position.</literal><replaceable>type</replaceable>
6609 This pattern specifies how the anchor (position) of
6610 CQL is mapped to RPN.
6611 The <replaceable>type</replaceable> is one
6612 of <literal>first</literal>, <literal>any</literal>,
6613 <literal>last</literal>, <literal>firstAndLast</literal>.
6616 The pattern, <literal>position.*</literal> is used
6617 when no other position pattern is matched.
6623 <literal>set.</literal><replaceable>prefix</replaceable>
6627 This specification defines a CQL context set for a given prefix.
6628 The value on the right hand side is the URI for the set -
6629 <emphasis>not</emphasis> RPN. All prefixes used in
6630 index patterns must be defined this way.
6636 <literal>set</literal>
6640 This specification defines a default CQL context set for index names.
6641 The value on the right hand side is the URI for the set.
6647 <example id="example.cql.to.rpn.mapping">
6648 <title>CQL to RPN mapping file</title>
6650 This simple file defines two context sets, three indexes and three
6651 relations, a position pattern and a default structure.
6653 <programlisting><![CDATA[
6654 set.cql = http://www.loc.gov/zing/cql/context-sets/cql/v1.1/
6655 set.dc = http://www.loc.gov/zing/cql/dc-indexes/v1.0/
6657 index.cql.serverChoice = 1=1016
6658 index.dc.title = 1=4
6659 index.dc.subject = 1=21
6665 position.any = 3=3 6=1
6671 With the mappings above, the CQL query
6675 is converted to the PQF:
6677 @attr 1=1016 @attr 2=3 @attr 4=1 @attr 3=3 @attr 6=1 "computer"
6679 by rules <literal>index.cql.serverChoice</literal>,
6680 <literal>relation.scr</literal>, <literal>structure.*</literal>,
6681 <literal>position.any</literal>.
6688 is rejected, since <literal>position.right</literal> is
6694 >my = "http://www.loc.gov/zing/cql/dc-indexes/v1.0/" my.title = x
6698 @attr 1=4 @attr 2=3 @attr 4=1 @attr 3=3 @attr 6=1 "x"
6702 <example id="example.cql.to.rpn.string">
6703 <title>CQL to RPN string attributes</title>
6705 In this example we allow any index to be passed to RPN as
6708 <programlisting><![CDATA[
6709 # Identifiers for prefixes used in this file. (index.*)
6710 set.cql = info:srw/cql-context-set/1/cql-v1.1
6711 set.rpn = http://bogus/rpn
6712 set = http://bogus/rpn
6714 # The default index when none is specified by the query
6715 index.cql.serverChoice = 1=any
6724 The <literal>http://bogus/rpn</literal> context set is also the default
6725 so we can make queries such as
6729 which is converted to
6731 @attr 2=3 @attr 4=1 @attr 3=3 @attr 1=title "a"
6735 <example id="example.cql.to.rpn.bathprofile">
6736 <title>CQL to RPN using Bath Profile</title>
6738 The file <filename>etc/pqf.properties</filename> has mappings from
6739 the Bath Profile and Dublin Core to RPN.
6740 If YAZ is installed as a package it's usually located
6741 in <filename>/usr/share/yaz/etc</filename> and part of the
6742 development package, such as <literal>libyaz-dev</literal>.
6746 <sect3 id="cql.xcql">
6747 <title>CQL to XCQL conversion</title>
6749 Conversion from CQL to XCQL is trivial and does not
6750 require a mapping to be defined.
6751 There three functions to choose from depending on the
6752 way you wish to store the resulting output (XML buffer
6755 int cql_to_xml_buf(struct cql_node *cn, char *out, int max);
6756 void cql_to_xml(struct cql_node *cn,
6757 void (*pr)(const char *buf, void *client_data),
6759 void cql_to_xml_stdio(struct cql_node *cn, FILE *f);
6761 Function <function>cql_to_xml_buf</function> converts
6762 to XCQL and stores result in a user supplied buffer of a given
6766 <function>cql_to_xml</function> writes the result in
6767 a user defined output stream.
6768 <function>cql_to_xml_stdio</function> writes to a
6772 <sect3 id="rpn.to.cql">
6773 <title>PQF to CQL conversion</title>
6775 Conversion from PQF to CQL is offered by the two functions shown
6776 below. The former uses a generic stream for result. The latter
6777 puts result in a WRBUF (string container).
6779 #include <yaz/rpn2cql.h>
6781 int cql_transform_rpn2cql_stream(cql_transform_t ct,
6782 void (*pr)(const char *buf, void *client_data),
6786 int cql_transform_rpn2cql_wrbuf(cql_transform_t ct,
6790 The configuration is the same as used in CQL to PQF conversions.
6795 <sect1 id="tools.oid">
6796 <title>Object Identifiers</title>
6798 The basic YAZ representation of an OID is an array of integers,
6799 terminated with the value -1. This integer is of type
6800 <literal>Odr_oid</literal>.
6803 Fundamental OID operations and the type <literal>Odr_oid</literal>
6804 are defined in <filename>yaz/oid_util.h</filename>.
6807 An OID can either be declared as a automatic variable or it can
6808 allocated using the memory utilities or ODR/NMEM. It's
6809 guaranteed that an OID can fit in <literal>OID_SIZE</literal> integers.
6811 <example id="tools.oid.bib1.1"><title>Create OID on stack</title>
6813 We can create an OID for the Bib-1 attribute set with:
6815 Odr_oid bib1[OID_SIZE];
6827 And OID may also be filled from a string-based representation using
6828 dots (.). This is achieved by function
6830 int oid_dotstring_to_oid(const char *name, Odr_oid *oid);
6832 This functions returns 0 if name could be converted; -1 otherwise.
6834 <example id="tools.oid.bib1.2"><title>Using oid_oiddotstring_to_oid</title>
6836 We can fill the Bib-1 attribute set OID easier with:
6838 Odr_oid bib1[OID_SIZE];
6839 oid_oiddotstring_to_oid("1.2.840.10003.3.1", bib1);
6844 We can also allocate an OID dynamically on a ODR stream with:
6846 Odr_oid *odr_getoidbystr(ODR o, const char *str);
6848 This creates an OID from string-based representation using dots.
6849 This function take an &odr; stream as parameter. This stream is used to
6850 allocate memory for the data elements, which is released on a
6851 subsequent call to <function>odr_reset()</function> on that stream.
6853 <example id="tools.oid.bib1.3">
6854 <title>Using odr_getoidbystr</title>
6856 We can create a OID for the Bib-1 attribute set with:
6858 Odr_oid *bib1 = odr_getoidbystr(odr, "1.2.840.10003.3.1");
6865 char *oid_oid_to_dotstring(const Odr_oid *oid, char *oidbuf)
6867 does the reverse of <function>oid_oiddotstring_to_oid</function>. It
6868 converts an OID to the string-based representation using dots.
6869 The supplied char buffer <literal>oidbuf</literal> holds the resulting
6870 string and must be at least <literal>OID_STR_MAX</literal> in size.
6873 OIDs can be copied with <function>oid_oidcpy</function> which takes
6874 two OID lists as arguments. Alternativly, an OID copy can be allocated
6875 on a ODR stream with:
6877 Odr_oid *odr_oiddup(ODR odr, const Odr_oid *o);
6881 OIDs can be compared with <function>oid_oidcmp</function> which returns
6882 zero if the two OIDs provided are identical; non-zero otherwise.
6884 <sect2 id="tools.oid.database">
6885 <title>OID database</title>
6887 From YAZ version 3 and later, the oident system has been replaced
6888 by an OID database. OID database is a misnomer .. the old odient
6889 system was also a database.
6892 The OID database is really just a map between named Object Identifiers
6893 (string) and their OID raw equivalents. Most operations either
6894 convert from string to OID or other way around.
6897 Unfortunately, whenever we supply a string we must also specify the
6898 <emphasis>OID class</emphasis>. The class is necessary because some
6899 strings correspond to multiple OIDs. An example of such a string is
6900 <literal>Bib-1</literal> which may either be an attribute-set
6901 or a diagnostic-set.
6904 Applications using the YAZ database should include
6905 <filename>yaz/oid_db.h</filename>.
6908 A YAZ database handle is of type <literal>yaz_oid_db_t</literal>.
6909 Actually that's a pointer. You need not think deal with that.
6910 YAZ has a built-in database which can be considered "constant" for
6912 We can get hold that by using function <function>yaz_oid_std</function>.
6915 All functions with prefix <function>yaz_string_to_oid</function>
6916 converts from class + string to OID. We have variants of this
6917 operation due to different memory allocation strategies.
6920 All functions with prefix
6921 <function>yaz_oid_to_string</function> converts from OID to string
6924 <example id="tools.oid.bib1.4">
6925 <title>Create OID with YAZ DB</title>
6927 We can create an OID for the Bib-1 attribute set on the ODR stream
6931 yaz_string_to_oid_odr(yaz_oid_std(), CLASS_ATTSET, "Bib-1", odr);
6933 This is more complex than using <function>odr_getoidbystr</function>.
6934 You would only use <function>yaz_string_to_oid_odr</function> when the
6935 string (here Bib-1) is supplied by a user or configuration.
6939 <sect2 id="tools.oid.std">
6940 <title>Standard OIDs</title>
6942 All the object identifers in the standard OID database as returned
6943 by <function>yaz_oid_std</function> can referenced directly in a
6944 program as a constant OID.
6945 Each constant OID is prefixed with <literal>yaz_oid_</literal> -
6946 followed by OID class (lowercase) - then by OID name (normalized and
6950 See <xref linkend="list-oids"/> for list of all object identifiers
6952 These are declared in <filename>yaz/oid_std.h</filename> but are
6953 included by <filename>yaz/oid_db.h</filename> as well.
6955 <example id="tools.oid.bib1.5">
6956 <title>Use a built-in OID</title>
6958 We can allocate our own OID filled with the constant OID for
6961 Odr_oid *bib1 = odr_oiddup(o, yaz_oid_attset_bib1);
6967 <sect1 id="tools.nmem">
6968 <title>Nibble Memory</title>
6970 Sometimes when you need to allocate and construct a large,
6971 interconnected complex of structures, it can be a bit of a pain to
6972 release the associated memory again. For the structures describing the
6973 Z39.50 PDUs and related structures, it is convenient to use the
6974 memory-management system of the &odr; subsystem (see
6975 <xref linkend="odr.use"/>). However, in some circumstances
6976 where you might otherwise benefit from using a simple nibble memory
6977 management system, it may be impractical to use
6978 <function>odr_malloc()</function> and <function>odr_reset()</function>.
6979 For this purpose, the memory manager which also supports the &odr;
6980 streams is made available in the NMEM module. The external interface
6981 to this module is given in the <filename>nmem.h</filename> file.
6984 The following prototypes are given:
6987 NMEM nmem_create(void);
6988 void nmem_destroy(NMEM n);
6989 void *nmem_malloc(NMEM n, size_t size);
6990 void nmem_reset(NMEM n);
6991 size_t nmem_total(NMEM n);
6992 void nmem_init(void);
6993 void nmem_exit(void);
6996 The <function>nmem_create()</function> function returns a pointer to a
6997 memory control handle, which can be released again by
6998 <function>nmem_destroy()</function> when no longer needed.
6999 The function <function>nmem_malloc()</function> allocates a block of
7000 memory of the requested size. A call to <function>nmem_reset()</function>
7001 or <function>nmem_destroy()</function> will release all memory allocated
7002 on the handle since it was created (or since the last call to
7003 <function>nmem_reset()</function>. The function
7004 <function>nmem_total()</function> returns the number of bytes currently
7005 allocated on the handle.
7008 The nibble memory pool is shared amongst threads. POSIX
7009 mutex'es and WIN32 Critical sections are introduced to keep the
7010 module thread safe. Function <function>nmem_init()</function>
7011 initializes the nibble memory library and it is called automatically
7012 the first time the <literal>YAZ.DLL</literal> is loaded. &yaz; uses
7013 function <function>DllMain</function> to achieve this. You should
7014 <emphasis>not</emphasis> call <function>nmem_init</function> or
7015 <function>nmem_exit</function> unless you're absolute sure what
7016 you're doing. Note that in previous &yaz; versions you'd have to call
7017 <function>nmem_init</function> yourself.
7020 <sect1 id="tools.log">
7023 &yaz; has evolved a fairly complex log system which should be useful both
7024 for debugging &yaz; itself, debugging applications that use &yaz;, and for
7025 production use of those applications.
7028 The log functions are declared in header <filename>yaz/log.h</filename>
7029 and implemented in <filename>src/log.c</filename>.
7030 Due to name clash with syslog and some math utilities the logging
7031 interface has been modified as of YAZ 2.0.29. The obsolete interface
7032 is still available if in header file <filename>yaz/log.h</filename>.
7033 The key points of the interface are:
7036 void yaz_log(int level, const char *fmt, ...)
7037 void yaz_log_init(int level, const char *prefix, const char *name);
7038 void yaz_log_init_file(const char *fname);
7039 void yaz_log_init_level(int level);
7040 void yaz_log_init_prefix(const char *prefix);
7041 void yaz_log_time_format(const char *fmt);
7042 void yaz_log_init_max_size(int mx);
7044 int yaz_log_mask_str(const char *str);
7045 int yaz_log_module_level(const char *name);
7048 The reason for the whole log module is the <function>yaz_log</function>
7049 function. It takes a bitmask indicating the log levels, a
7050 <literal>printf</literal>-like format string, and a variable number of
7054 The <literal>log level</literal> is a bit mask, that says on which level(s)
7055 the log entry should be made, and optionally set some behaviour of the
7056 logging. In the most simple cases, it can be one of <literal>YLOG_FATAL,
7057 YLOG_DEBUG, YLOG_WARN, YLOG_LOG</literal>. Those can be combined with bits
7058 that modify the way the log entry is written:<literal>YLOG_ERRNO,
7059 YLOG_NOTIME, YLOG_FLUSH</literal>.
7060 Most of the rest of the bits are deprecated, and should not be used. Use
7061 the dynamic log levels instead.
7064 Applications that use &yaz;, should not use the LOG_LOG for ordinary
7065 messages, but should make use of the dynamic loglevel system. This consists
7066 of two parts, defining the loglevel and checking it.
7069 To define the log levels, the (main) program should pass a string to
7070 <function>yaz_log_mask_str</function> to define which log levels are to be
7071 logged. This string should be a comma-separated list of log level names,
7072 and can contain both hard-coded names and dynamic ones. The log level
7073 calculation starts with <literal>YLOG_DEFAULT_LEVEL</literal> and adds a bit
7074 for each word it meets, unless the word starts with a '-', in which case it
7075 clears the bit. If the string <literal>'none'</literal> is found,
7076 all bits are cleared. Typically this string comes from the command-line,
7077 often identified by <literal>-v</literal>. The
7078 <function>yaz_log_mask_str</function> returns a log level that should be
7079 passed to <function>yaz_log_init_level</function> for it to take effect.
7082 Each module should check what log bits it should be used, by calling
7083 <function>yaz_log_module_level</function> with a suitable name for the
7084 module. The name is cleared from a preceding path and an extension, if any,
7085 so it is quite possible to use <literal>__FILE__</literal> for it. If the
7086 name has been passed to <function>yaz_log_mask_str</function>, the routine
7087 returns a non-zero bitmask, which should then be used in consequent calls
7088 to yaz_log. (It can also be tested, so as to avoid unnecessary calls to
7089 yaz_log, in time-critical places, or when the log entry would take time
7093 Yaz uses the following dynamic log levels:
7094 <literal>server, session, request, requestdetail</literal> for the server
7096 <literal>zoom</literal> for the zoom client api.
7097 <literal>ztest</literal> for the simple test server.
7098 <literal>malloc, nmem, odr, eventl</literal> for internal
7099 debugging of yaz itself.
7100 Of course, any program using yaz is welcome to define as many new
7104 By default the log is written to stderr, but this can be changed by a call
7105 to <function>yaz_log_init_file</function> or
7106 <function>yaz_log_init</function>. If the log is directed to a file, the
7107 file size is checked at every write, and if it exceeds the limit given in
7108 <function>yaz_log_init_max_size</function>, the log is rotated. The
7109 rotation keeps one old version (with a <literal>.1</literal> appended to
7110 the name). The size defaults to 1GB. Setting it to zero will disable the
7114 A typical yaz-log looks like this
7115 13:23:14-23/11 yaz-ztest(1) [session] Starting session from tcp:127.0.0.1 (pid=30968)
7116 13:23:14-23/11 yaz-ztest(1) [request] Init from 'YAZ' (81) (ver 2.0.28) OK
7117 13:23:17-23/11 yaz-ztest(1) [request] Search Z: @attrset Bib-1 foo OK:7 hits
7118 13:23:22-23/11 yaz-ztest(1) [request] Present: [1] 2+2 OK 2 records returned
7119 13:24:13-23/11 yaz-ztest(1) [request] Close OK
7122 The log entries start with a time stamp. This can be omitted by setting the
7123 <literal>YLOG_NOTIME</literal> bit in the loglevel. This way automatic tests
7124 can be hoped to produce identical log files, that are easy to diff. The
7125 format of the time stamp can be set with
7126 <function>yaz_log_time_format</function>, which takes a format string just
7127 like <function>strftime</function>.
7130 Next in a log line comes the prefix, often the name of the program. For
7131 yaz-based servers, it can also contain the session number. Then
7132 comes one or more logbits in square brackets, depending on the logging
7133 level set by <function>yaz_log_init_level</function> and the loglevel
7134 passed to <function>yaz_log_init_level</function>. Finally comes the format
7135 string and additional values passed to <function>yaz_log</function>
7138 The log level <literal>YLOG_LOGLVL</literal>, enabled by the string
7139 <literal>loglevel</literal>, will log all the log-level affecting
7140 operations. This can come in handy if you need to know what other log
7141 levels would be useful. Grep the logfile for <literal>[loglevel]</literal>.
7144 The log system is almost independent of the rest of &yaz;, the only
7145 important dependence is of <filename>nmem</filename>, and that only for
7146 using the semaphore definition there.
7149 The dynamic log levels and log rotation were introduced in &yaz; 2.0.28. At
7150 the same time, the log bit names were changed from
7151 <literal>LOG_something</literal> to <literal>YLOG_something</literal>,
7152 to avoid collision with <filename>syslog.h</filename>.
7158 YAZ provides a fast utility for working with MARC records.
7159 Early versions of the MARC utility only allowed decoding of ISO2709.
7160 Today the utility may both encode - and decode to a varity of formats.
7163 #include <yaz/marcdisp.h>
7165 /* create handler */
7166 yaz_marc_t yaz_marc_create(void);
7168 void yaz_marc_destroy(yaz_marc_t mt);
7170 /* set XML mode YAZ_MARC_LINE, YAZ_MARC_SIMPLEXML, ... */
7171 void yaz_marc_xml(yaz_marc_t mt, int xmlmode);
7172 #define YAZ_MARC_LINE 0
7173 #define YAZ_MARC_SIMPLEXML 1
7174 #define YAZ_MARC_OAIMARC 2
7175 #define YAZ_MARC_MARCXML 3
7176 #define YAZ_MARC_ISO2709 4
7177 #define YAZ_MARC_XCHANGE 5
7178 #define YAZ_MARC_CHECK 6
7179 #define YAZ_MARC_TURBOMARC 7
7180 #define YAZ_MARC_JSON 8
7182 /* supply iconv handle for character set conversion .. */
7183 void yaz_marc_iconv(yaz_marc_t mt, yaz_iconv_t cd);
7185 /* set debug level, 0=none, 1=more, 2=even more, .. */
7186 void yaz_marc_debug(yaz_marc_t mt, int level);
7188 /* decode MARC in buf of size bsize. Returns >0 on success; <=0 on failure.
7189 On success, result in *result with size *rsize. */
7190 int yaz_marc_decode_buf(yaz_marc_t mt, const char *buf, int bsize,
7191 const char **result, size_t *rsize);
7193 /* decode MARC in buf of size bsize. Returns >0 on success; <=0 on failure.
7194 On success, result in WRBUF */
7195 int yaz_marc_decode_wrbuf(yaz_marc_t mt, const char *buf,
7196 int bsize, WRBUF wrbuf);
7201 The synopsis is just a basic subset of all functionality. Refer
7202 to the actual header file <filename>marcdisp.h</filename> for
7207 A MARC conversion handle must be created by using
7208 <function>yaz_marc_create</function> and destroyed
7209 by calling <function>yaz_marc_destroy</function>.
7212 All other function operate on a <literal>yaz_marc_t</literal> handle.
7213 The output is specified by a call to <function>yaz_marc_xml</function>.
7214 The <literal>xmlmode</literal> must be one of
7217 <term>YAZ_MARC_LINE</term>
7220 A simple line-by-line format suitable for display but not
7221 recommend for further (machine) processing.
7226 <term>YAZ_MARC_MARCXML</term>
7229 <ulink url="&url.marcxml;">MARCXML</ulink>.
7234 <term>YAZ_MARC_ISO2709</term>
7237 ISO2709 (sometimes just referred to as "MARC").
7242 <term>YAZ_MARC_XCHANGE</term>
7245 <ulink url="&url.marcxchange;">MarcXchange</ulink>.
7250 <term>YAZ_MARC_CHECK</term>
7253 Pseudo format for validation only. Does not generate
7254 any real output except diagnostics.
7259 <term>YAZ_MARC_TURBOMARC</term>
7262 XML format with same semantics as MARCXML but more compact
7263 and geared towards fast processing with XSLT. Refer to
7264 <xref linkend="tools.turbomarc"/> for more information.
7269 <term>YAZ_MARC_JSON</term>
7272 <ulink url="&url.marc_in_json;">MARC-in_JSON</ulink> format.
7279 The actual conversion functions are
7280 <function>yaz_marc_decode_buf</function> and
7281 <function>yaz_marc_decode_wrbuf</function> which decodes and encodes
7282 a MARC record. The former function operates on simple buffers, the
7283 stores the resulting record in a WRBUF handle (WRBUF is a simple string
7286 <example id="example.marc.display">
7287 <title>Display of MARC record</title>
7289 The following program snippet illustrates how the MARC API may
7290 be used to convert a MARC record to the line-by-line format:
7291 <programlisting><![CDATA[
7292 void print_marc(const char *marc_buf, int marc_buf_size)
7294 char *result; /* for result buf */
7295 size_t result_len; /* for size of result */
7296 yaz_marc_t mt = yaz_marc_create();
7297 yaz_marc_xml(mt, YAZ_MARC_LINE);
7298 yaz_marc_decode_buf(mt, marc_buf, marc_buf_size,
7299 &result, &result_len);
7300 fwrite(result, result_len, 1, stdout);
7301 yaz_marc_destroy(mt); /* note that result is now freed... */
7307 <sect2 id="tools.turbomarc">
7308 <title>TurboMARC</title>
7310 TurboMARC is yet another XML encoding of a MARC record. The format
7311 was designed for fast processing with XSLT.
7315 Pazpar2 uses XSLT to convert an XML encoded MARC record to an internal
7316 representation. This conversion mostly check the tag of a MARC field
7317 to determine the basic rules in the conversion. This check is
7318 costly when that is tag is encoded as an attribute in MARCXML.
7319 By having the tag value as the element instead, makes processing
7320 many times faster (at least for Libxslt).
7323 TurboMARC is encoded as follows:
7327 Record elements is part of namespace
7328 "<literal>http://www.indexdata.com/turbomarc</literal>".
7333 A record is enclosed in element <literal>r</literal>.
7338 A collection of records is enclosed in element
7339 <literal>collection</literal>.
7344 The leader is encoded as element <literal>l</literal> with the
7345 leader content as its (text) value.
7350 A control field is encoded as element <literal>c</literal> concatenated
7351 with the tag value of the control field if the tag value
7352 matches the regular expression <literal>[a-zA-Z0-9]*</literal>.
7353 If the tag value do not match the regular expression
7354 <literal>[a-zA-Z0-9]*</literal> the control field is encoded
7355 as element <literal>c</literal> and attribute <literal>code</literal>
7356 will hold the tag value.
7357 This rule ensure that in the rare cases where a tag value might
7358 result in a non-wellformed XML YAZ encode it as a coded attribute
7362 The control field content is the the text value of this element.
7363 Indicators are encoded as attribute names
7364 <literal>i1</literal>, <literal>i2</literal>, etc.. and
7365 corresponding values for each indicator.
7370 A data field is encoded as element <literal>d</literal> concatenated
7371 with the tag value of the data field or using the attribute
7372 <literal>code</literal> as described in the rules for control fields.
7373 The children of the data field element is subfield elements.
7374 Each subfield element is encoded as <literal>s</literal>
7375 concatenated with the sub field code.
7376 The text of the subfield element is the contents of the subfield.
7377 Indicators are encoded as attributes for the data field element similar
7378 to the encoding for control fields.
7385 <sect1 id="tools.retrieval">
7386 <title>Retrieval Facility</title>
7388 YAZ version 2.1.20 or later includes a Retrieval facility tool
7389 which allows a SRU/Z39.50 to describe itself and perform record
7390 conversions. The idea is the following:
7394 An SRU/Z39.50 client sends a retrieval request which includes
7395 a combination of the following parameters: syntax (format),
7396 schema (or element set name).
7401 The retrieval facility is invoked with parameters in a
7402 server/proxy. The retrieval facility matches the parameters a set of
7403 "supported" retrieval types.
7404 If there is no match, the retrieval signals an error
7405 (syntax and / or schema not supported).
7410 For a successful match, the backend is invoked with the same
7411 or altered retrieval parameters (syntax, schema). If
7412 a record is received from the backend, it is converted to the
7413 frontend name / syntax.
7418 The resulting record is sent back the client and tagged with
7419 the frontend syntax / schema.
7425 The Retrieval facility is driven by an XML configuration. The
7426 configuration is neither Z39.50 ZeeRex or SRU ZeeRex. But it
7427 should be easy to generate both of them from the XML configuration.
7428 (unfortunately the two versions
7429 of ZeeRex differ substantially in this regard).
7431 <sect2 id="tools.retrieval.format">
7432 <title>Retrieval XML format</title>
7434 All elements should be covered by namespace
7435 <literal>http://indexdata.com/yaz</literal> .
7436 The root element node must be <literal>retrievalinfo</literal>.
7439 The <literal>retrievalinfo</literal> must include one or
7440 more <literal>retrieval</literal> elements. Each
7441 <literal>retrieval</literal> defines specific combination of
7442 syntax, name and identifier supported by this retrieval service.
7445 The <literal>retrieval</literal> element may include any of the
7446 following attributes:
7448 <varlistentry><term><literal>syntax</literal> (REQUIRED)</term>
7451 Defines the record syntax. Possible values is any
7452 of the names defined in YAZ' OID database or a raw
7457 <varlistentry><term><literal>name</literal> (OPTIONAL)</term>
7460 Defines the name of the retrieval format. This can be
7461 any string. For SRU, the value, is equivalent to schema (short-hand);
7462 for Z39.50 it's equivalent to simple element set name.
7463 For YAZ 3.0.24 and later this name may be specified as a glob
7464 expression with operators
7465 <literal>*</literal> and <literal>?</literal>.
7469 <varlistentry><term><literal>identifier</literal> (OPTIONAL)</term>
7472 Defines the URI schema name of the retrieval format. This can be
7473 any string. For SRU, the value, is equivalent to URI schema.
7474 For Z39.50, there is no equivalent.
7481 The <literal>retrieval</literal> may include one
7482 <literal>backend</literal> element. If a <literal>backend</literal>
7483 element is given, it specifies how the records are retrieved by
7484 some backend and how the records are converted from the backend to
7488 The attributes, <literal>name</literal> and <literal>syntax</literal>
7489 may be specified for the <literal>backend</literal> element. These
7490 semantics of these attributes is equivalent to those for the
7491 <literal>retrieval</literal>. However, these values are passed to
7495 The <literal>backend</literal> element may includes one or more
7496 conversion instructions (as children elements). The supported
7499 <varlistentry><term><literal>marc</literal></term>
7502 The <literal>marc</literal> element specifies a conversion
7503 to - and from ISO2709 encoded MARC and
7504 <ulink url="&url.marcxml;">&acro.marcxml;</ulink>/MarcXchange.
7505 The following attributes may be specified:
7508 <term><literal>inputformat</literal> (REQUIRED)</term>
7511 Format of input. Supported values are
7512 <literal>marc</literal> (for ISO2709), <literal>xml</literal>
7513 (MARCXML/MarcXchange) and <literal>json</literal>
7514 (<ulink url="&url.marc_in_json;">MARC-in_JSON</ulink>).
7519 <term><literal>outputformat</literal> (REQUIRED)</term>
7522 Format of output. Supported values are
7523 <literal>line</literal> (MARC line format);
7524 <literal>marcxml</literal> (for MARCXML),
7525 <literal>marc</literal> (ISO2709),
7526 <literal>marcxhcange</literal> (for MarcXchange),
7527 or <literal>json</literal>
7528 (<ulink url="&url.marc_in_json;">MARC-in_JSON </ulink>).
7533 <term><literal>inputcharset</literal> (OPTIONAL)</term>
7536 Encoding of input. For XML input formats, this need not
7537 be given, but for ISO2709 based inputformats, this should
7538 be set to the encoding used. For MARC21 records, a common
7539 inputcharset value would be <literal>marc-8</literal>.
7544 <term><literal>outputcharset</literal> (OPTIONAL)</term>
7547 Encoding of output. If outputformat is XML based, it is
7548 strongly recommened to use <literal>utf-8</literal>.
7557 <term><literal>select</literal></term>
7560 The <literal>select</literal> selects one or more text nodes
7561 and decodes them as XML.
7562 The following attributes may be specified:
7564 <varlistentry><term><literal>path</literal> (REQUIRED)</term>
7567 X-Path expression for selecting text nodes.
7574 This conversion is available in YAZ 5.8.0 and later.
7579 <term><literal>solrmarc</literal></term>
7582 The <literal>solrmarc</literal> decodes solrmarc records.
7583 It assumes that the input is pure solrmarc text (no escaping)
7584 and will convert all sequences of the form #XX; to a single
7585 character of the hexadecimal value as given by XX. The output,
7586 presumably, is a valid ISO2709 buffer.
7589 This conversion is available in YAZ 5.0.21 and later.
7594 <term><literal>xslt</literal></term>
7597 The <literal>xslt</literal> element specifies a conversion
7598 via &acro.xslt;. The following attributes may be specified:
7600 <varlistentry><term><literal>stylesheet</literal> (REQUIRED)</term>
7614 <sect2 id="tools.retrieval.examples">
7615 <title>Retrieval Facility Examples</title>
7616 <example id="tools.retrieval.marc21">
7617 <title>MARC21 backend</title>
7619 A typical way to use the retrieval facility is to enable XML
7620 for servers that only supports ISO2709 encoded MARC21 records.
7622 <programlisting><![CDATA[
7624 <retrieval syntax="usmarc" name="F"/>
7625 <retrieval syntax="usmarc" name="B"/>
7626 <retrieval syntax="xml" name="marcxml"
7627 identifier="info:srw/schema/1/marcxml-v1.1">
7628 <backend syntax="usmarc" name="F">
7629 <marc inputformat="marc" outputformat="marcxml"
7630 inputcharset="marc-8"/>
7633 <retrieval syntax="xml" name="dc">
7634 <backend syntax="usmarc" name="F">
7635 <marc inputformat="marc" outputformat="marcxml"
7636 inputcharset="marc-8"/>
7637 <xslt stylesheet="MARC21slim2DC.xsl"/>
7644 This means that our frontend supports:
7648 MARC21 F(ull) records.
7653 MARC21 B(rief) records.
7663 Dublin core records.
7669 <example id="tools.retrieval.marcxml">
7670 <title>MARCXML backend</title>
7672 SRW/SRU and Solr backends returns records in XML.
7673 If they return MARCXML or MarcXchange, the retrieval module
7674 can convert those into ISO2709 formats, most commonly USMARC
7676 In this example, the backend returns MARCXML for schema="marcxml".
7678 <programlisting><![CDATA[
7680 <retrieval syntax="usmarc">
7681 <backend syntax="xml" name="marcxml">
7682 <marc inputformat="xml" outputformat="marc"
7683 outputcharset="marc-8"/>
7686 <retrieval syntax="xml" name="marcxml"
7687 identifier="info:srw/schema/1/marcxml-v1.1"/>
7688 <retrieval syntax="xml" name="dc">
7689 <backend syntax="xml" name="marcxml">
7690 <xslt stylesheet="MARC21slim2DC.xsl"/>
7697 This means that our frontend supports:
7701 MARC21 records (any element set name) in MARC-8 encoding.
7706 MARCXML records for element-set=marcxml
7711 Dublin core records for element-set=dc.
7718 <sect2 id="tools.retrieval.api">
7721 It should be easy to use the retrieval systems from applications. Refer
7723 <filename>yaz/retrieval.h</filename> and
7724 <filename>yaz/record_conv.h</filename>.
7728 <sect1 id="sorting">
7729 <title>Sorting</title>
7731 This chapter describes sorting and how it is supported in YAZ.
7732 Sorting applies to a result-set.
7734 <ulink url="http://www.loc.gov/z3950/agency/markup/05.html#3.2.7">
7735 Z39.50 sorting facility
7737 takes one or more input result-sets
7738 and one result-set as output. The most simple case is that
7739 the input-set is the same as the output-set.
7742 Z39.50 sorting has a separate APDU (service) that is, thus, performed
7743 following a search (two phases).
7746 In SRU/Solr, however, the model is different. Here, sorting is specified
7747 during the the search operation. Note, however, that SRU might
7748 perform sort as separate search, by referring to an existing result-set
7749 in the query (result-set reference).
7752 <title>Using the Z39.50 sort service</title>
7754 yaz-client and the ZOOM API supports the Z39.50 sort facility. In any
7755 case the sort sequence or sort critiera is using a string notation.
7756 This notation is a one-line notation suitable for being manually
7757 entered or generated and allows for easy logging (one liner).
7758 For the ZOOM API, the sort is specified in the call to ZOOM_query_sortby
7759 function. For yaz-client the sort is performed and specified using
7760 the sort and sort+ commands. For description of the sort criteria notation
7761 refer to the <link linkend="sortspec">sort command</link> in the
7765 The ZOOM API might choose one of several sort strategies for
7766 sorting. Refer to <xref linkend="zoom-sort-strategy"/>.
7770 <title>Type-7 sort</title>
7772 Type-7 sort is an extension to the Bib-1 based RPN query where the
7773 sort specification is embedded as an Attribute-Plus-Term.
7776 The objectives for introducing Type-7 sorting is that it allows
7777 a client to perform sorting even if it does not implement/support
7778 Z39.50 sort. Virtually all Z39.50 client software supports
7779 RPN queries. It also may improve performance because the sort
7780 critieria is specified along with the search query.
7783 The sort is triggered by the presence of type 7 and the value of type 7
7785 <ulink url="http://www.loc.gov/z3950/agency/asn1.html#SortKeySpec">
7788 The value for type 7 is 1 for ascending and 2 for descending.
7790 <ulink url="http://www.loc.gov/z3950/agency/asn1.html#SortElement">
7793 only the generic part is handled. If generic sortKey is of type
7794 sortField, then attribute type 1 is present and the value is
7795 sortField (InternationalString). If generic sortKey is of type
7796 sortAttributes, then the attributes in list is used . generic sortKey
7797 of type elementSpec is not supported.
7800 The term in the sorting Attribute-Plus-Term combo should hold
7801 an integer. The value is 0 for primary sorting criteria, 1 for second
7807 <title>Facets</title>
7809 YAZ supports facets for in Solr, SRU 2.0 and Z39.50 protocols.
7812 Like Type-1/RPN, YAZ supports a string notation for specifying
7813 facets. For the API this is performed by
7814 <function>yaz_pqf_parse_facet_list</function>.
7817 For ZOOM C the facets are given by option "facets"
7818 For yaz-client it is used for the facets command.
7821 The grammar of this specification is as follows:
7823 facet-spec ::= facet-list
7825 facet-list ::= facet-list ',' attr-spec | attr-spec
7827 attr-spec ::= attr-spec '@attr' string | '@attr' string
7830 The notation is inspired by PQF. The string following '@attr'
7831 may not include blanks and is of the form
7832 <replaceable>type</replaceable><literal>=</literal><replaceable>value</replaceable>,
7833 where <replaceable>type</replaceable> is an integer and
7834 <replaceable>value</replaceable> is a string or an integer.
7837 The Facets specification is not Bib-1. The following types apply:
7839 <table id="facet.attributes">
7840 <title>Facet attributes</title>
7842 <colspec colwidth="2*" colname="type"></colspec>
7843 <colspec colwidth="9*" colname="description"></colspec>
7847 <entry>Description</entry>
7854 Field-name. This is often a string, eg "Author", "Year", etc.
7860 Sort order. Value should be an integer.
7861 Value 0: count descending (frequency). Value 1: alpha ascending.
7867 Number of terms requested.
7882 <title>The ODR Module</title>
7883 <sect1 id="odr.introduction">
7884 <title>Introduction</title>
7886 &odr; is the BER-encoding/decoding subsystem of &yaz;. Care as been taken
7887 to isolate &odr; from the rest of the package - specifically from the
7888 transport interface. &odr; may be used in any context where basic
7889 ASN.1/BER representations are used.
7892 If you are only interested in writing a Z39.50 implementation based on
7893 the PDUs that are already provided with &yaz;, you only need to concern
7894 yourself with the section on managing ODR streams
7895 (<xref linkend="odr.use"/>). Only if you need to
7896 implement ASN.1 beyond that which has been provided, should you
7897 worry about the second half of the documentation
7898 (<xref linkend="odr.programming"/>).
7899 If you use one of the higher-level interfaces, you can skip this
7903 This is important, so we'll repeat it for emphasis: <emphasis>You do
7904 not need to read <xref linkend="odr.programming"/>
7905 to implement Z39.50 with &yaz;.</emphasis>
7908 If you need a part of the protocol that isn't already in &yaz;, you
7909 should contact the authors before going to work on it yourself: We
7910 might already be working on it. Conversely, if you implement a useful
7911 part of the protocol before us, we'd be happy to include it in a
7915 <sect1 id="odr.use">
7916 <title>Using ODR</title>
7917 <sect2 id="odr.streams">
7918 <title>ODR Streams</title>
7920 Conceptually, the ODR stream is the source of encoded data in the
7921 decoding mode; when encoding, it is the receptacle for the encoded
7922 data. Before you can use an ODR stream it must be allocated. This is
7923 done with the function
7926 ODR odr_createmem(int direction);
7929 The <function>odr_createmem()</function> function takes as argument one
7930 of three manifest constants: <literal>ODR_ENCODE</literal>,
7931 <literal>ODR_DECODE</literal>, or <literal>ODR_PRINT</literal>.
7932 An &odr; stream can be in only one mode - it is not possible to change
7933 its mode once it's selected. Typically, your program will allocate
7934 at least two ODR streams - one for decoding, and one for encoding.
7937 When you're done with the stream, you can use
7940 void odr_destroy(ODR o);
7943 to release the resources allocated for the stream.
7946 <sect2 id="odr.memory.management">
7947 <title id="memory">Memory Management</title>
7949 Two forms of memory management take place in the &odr; system. The first
7950 one, which has to do with allocating little bits of memory (sometimes
7951 quite large bits of memory, actually) when a protocol package is
7952 decoded, and turned into a complex of interlinked structures. This
7953 section deals with this system, and how you can use it for your own
7954 purposes. The next section deals with the memory management which is
7955 required when encoding data - to make sure that a large enough buffer is
7956 available to hold the fully encoded PDU.
7959 The &odr; module has its own memory management system, which is
7960 used whenever memory is required. Specifically, it is used to allocate
7961 space for data when decoding incoming PDUs. You can use the memory
7962 system for your own purposes, by using the function
7965 void *odr_malloc(ODR o, size_t size);
7968 You can't use the normal <function>free(2)</function> routine to free
7969 memory allocated by this function, and &odr; doesn't provide a parallel
7970 function. Instead, you can call
7973 void odr_reset(ODR o);
7976 when you are done with the
7977 memory: Everything allocated since the last call to
7978 <function>odr_reset()</function> is released.
7979 The <function>odr_reset()</function> call is also required to clear
7980 up an error condition on a stream.
7986 size_t odr_total(ODR o);
7989 returns the number of bytes allocated on the stream since the last call to
7990 <function>odr_reset()</function>.
7993 The memory subsystem of &odr; is fairly efficient at allocating and
7994 releasing little bits of memory. Rather than managing the individual,
7995 small bits of space, the system maintains a free-list of larger chunks
7996 of memory, which are handed out in small bits. This scheme is
7997 generally known as a <emphasis>nibble memory</emphasis> system.
7998 It is very useful for maintaining short-lived constructions such
8002 If you want to retain a bit of memory beyond the next call to
8003 <function>odr_reset()</function>, you can use the function
8006 ODR_MEM odr_extract_mem(ODR o);
8009 This function will give you control of the memory recently allocated
8010 on the ODR stream. The memory will live (past calls to
8011 <function>odr_reset()</function>), until you call the function
8014 void odr_release_mem(ODR_MEM p);
8017 The opaque <literal>ODR_MEM</literal> handle has no other purpose than
8018 referencing the memory block for you until you want to release it.
8021 You can use <function>odr_extract_mem()</function> repeatedly between
8022 allocating data, to retain individual control of separate chunks of data.
8025 <sect2 id="odr.encoding.and.decoding">
8026 <title>Encoding and Decoding Data</title>
8028 When encoding data, the ODR stream will write the encoded octet string
8029 in an internal buffer. To retrieve the data, use the function
8032 char *odr_getbuf(ODR o, int *len, int *size);
8035 The integer pointed to by len is set to the length of the encoded
8036 data, and a pointer to that data is returned. <literal>*size</literal>
8037 is set to the size of the buffer (unless <literal>size</literal> is null,
8038 signaling that you are not interested in the size). The next call to
8039 a primitive function using the same &odr; stream will overwrite the
8040 data, unless a different buffer has been supplied using the call
8043 void odr_setbuf(ODR o, char *buf, int len, int can_grow);
8046 which sets the encoding (or decoding) buffer used by
8047 <literal>o</literal> to <literal>buf</literal>, using the length
8048 <literal>len</literal>.
8049 Before a call to an encoding function, you can use
8050 <function>odr_setbuf()</function> to provide the stream with an encoding
8051 buffer of sufficient size (length). The <literal>can_grow</literal>
8052 parameter tells the encoding &odr; stream whether it is allowed to use
8053 <function>realloc(2)</function> to increase the size of the buffer when
8054 necessary. The default condition of a new encoding stream is equivalent
8055 to the results of calling
8058 odr_setbuf(stream, 0, 0, 1);
8061 In this case, the stream will allocate and reallocate memory as
8062 necessary. The stream reallocates memory by repeatedly doubling the
8063 size of the buffer - the result is that the buffer will typically
8064 reach its maximum, working size with only a small number of reallocation
8065 operations. The memory is freed by the stream when the latter is destroyed,
8066 unless it was assigned by the user with the <literal>can_grow</literal>
8067 parameter set to zero (in this case, you are expected to retain
8068 control of the memory yourself).
8071 To assume full control of an encoded buffer, you must first call
8072 <function>odr_getbuf()</function> to fetch the buffer and its length.
8073 Next, you should call <function>odr_setbuf()</function> to provide a
8074 different buffer (or a null pointer) to the stream. In the simplest
8075 case, you will reuse the same buffer over and over again, and you
8076 will just need to call <function>odr_getbuf()</function> after each
8077 encoding operation to get the length and address of the buffer.
8078 Note that the stream may reallocate the buffer during an encoding
8079 operation, so it is necessary to retrieve the correct address after
8080 each encoding operation.
8083 It is important to realize that the ODR stream will not release this
8084 memory when you call <function>odr_reset()</function>: It will
8085 merely update its internal pointers to prepare for the encoding of a
8087 When the stream is released by the <function>odr_destroy()</function>
8088 function, the memory given to it by <function>odr_setbuf</function> will
8089 be released <emphasis>only</emphasis> if the <literal>can_grow</literal>
8090 parameter to <function>odr_setbuf()</function> was nonzero. The
8091 <literal>can_grow</literal> parameter, in other words, is a way of
8092 signaling who is to own the buffer, you or the ODR stream. If you never call
8093 <function>odr_setbuf()</function> on your encoding stream, which is
8094 typically the case, the buffer allocated by the stream will belong to
8095 the stream by default.
8098 When you wish to decode data, you should first call
8099 <function>odr_setbuf()</function>, to tell the decoding stream
8100 where to find the encoded data, and how long the buffer is
8101 (the <literal>can_grow</literal> parameter is ignored by a decoding
8102 stream). After this, you can call the function corresponding to the
8103 data you wish to decode (eg, <function>odr_integer()</function> odr
8104 <function>z_APDU()</function>).
8106 <example id="example.odr.encoding.and.decoding.functions">
8107 <title>Encoding and decoding functions</title>
8109 int odr_integer(ODR o, Odr_int **p, int optional, const char *name);
8111 int z_APDU(ODR o, Z_APDU **p, int optional, const char *name);
8115 If the data is absent (or doesn't match the tag corresponding to
8116 the type), the return value will be either 0 or 1 depending on the
8117 <literal>optional</literal> flag. If <literal>optional</literal>
8118 is 0 and the data is absent, an error flag will be raised in the
8119 stream, and you'll need to call <function>odr_reset()</function> before
8120 you can use the stream again. If <literal>optional</literal> is
8121 nonzero, the pointer <emphasis>pointed</emphasis> to/ by
8122 <literal>p</literal> will be set to the null value, and the function
8124 The <literal>name</literal> argument is used to pretty-print the
8125 tag in question. It may be set to <literal>NULL</literal> if
8126 pretty-printing is not desired.
8129 If the data value is found where it's expected, the pointer
8130 <emphasis>pointed to</emphasis> by the <literal>p</literal> argument
8131 will be set to point to the decoded type.
8132 The space for the type will be allocated and owned by the &odr;
8133 stream, and it will live until you call
8134 <function>odr_reset()</function> on the stream. You cannot use
8135 <function>free(2)</function> to release the memory.
8136 You can decode several data elements (by repeated calls to
8137 <function>odr_setbuf()</function> and your decoding function), and
8138 new memory will be allocated each time. When you do call
8139 <function>odr_reset()</function>, everything decoded since the
8140 last call to <function>odr_reset()</function> will be released.
8142 <example id="example.odr.encoding.of.integer">
8143 <title>Encoding and decoding of an integer</title>
8145 The use of the double indirection can be a little confusing at first
8146 (its purpose will become clear later on, hopefully),
8147 so an example is in order. We'll encode an integer value, and
8148 immediately decode it again using a different stream. A useless, but
8149 informative operation.
8151 <programlisting><![CDATA[
8152 void do_nothing_useful(Odr_int value)
8155 Odr_int *valp, *resvalp;
8159 /* allocate streams */
8160 if (!(encode = odr_createmem(ODR_ENCODE)))
8162 if (!(decode = odr_createmem(ODR_DECODE)))
8166 if (odr_integer(encode, &valp, 0, 0) == 0)
8168 printf("encoding went bad\n");
8171 bufferp = odr_getbuf(encode, &len, 0);
8172 printf("length of encoded data is %d\n", len);
8174 /* now let's decode the thing again */
8175 odr_setbuf(decode, bufferp, len, 0);
8176 if (odr_integer(decode, &resvalp, 0, 0) == 0)
8178 printf("decoding went bad\n");
8181 /* ODR_INT_PRINTF format for printf (such as %d) */
8182 printf("the value is " ODR_INT_PRINTF "\n", *resvalp);
8185 odr_destroy(encode);
8186 odr_destroy(decode);
8191 This looks like a lot of work, offhand. In practice, the &odr; streams
8192 will typically be allocated once, in the beginning of your program
8193 (or at the beginning of a new network session), and the encoding
8194 and decoding will only take place in a few, isolated places in your
8195 program, so the overhead is quite manageable.
8199 <sect2 id="odr.printing">
8200 <title>Printing</title>
8202 When an ODR stream is created of type <literal>ODR_PRINT</literal>
8203 the ODR module will print the contents of a PDU in a readable format.
8204 By default output is written to the <literal>stderr</literal> stream.
8205 This behavior can be changed, however, by calling the function
8207 odr_setprint(ODR o, FILE *file);
8209 before encoders or decoders are being invoked.
8210 It is also possible to direct the output to a buffer (of indeed
8211 another file), by using the more generic mechanism:
8213 void odr_set_stream(ODR o, void *handle,
8214 void (*stream_write)(ODR o, void *handle, int type,
8215 const char *buf, int len),
8216 void (*stream_close)(void *handle));
8218 Here the user provides an opaque handle and two handlers,
8219 <replaceable>stream_write</replaceable> for writing,
8220 and <replaceable>stream_close</replaceable> which is supposed
8221 to close/free resources associated with handle.
8222 The <replaceable>stream_close</replaceable> handler is optional and
8223 if NULL for the function is provided, it will not be invoked.
8224 The <replaceable>stream_write</replaceable> takes the ODR handle
8225 as parameter, the user defined handle, a type
8226 <literal>ODR_OCTETSTRING</literal>, <literal>ODR_VISIBLESTRING</literal>
8227 which indicates the type of contents is being written.
8230 Another utility useful for diagnostics (error handling) or as
8231 part of the printing facilities is:
8233 const char **odr_get_element_path(ODR o);
8235 which returns a list of current elements that ODR deals with at the
8236 moment. For the returned array, say <literal>ar</literal>,
8237 <literal>ar[0]</literal> is the top level element,
8238 <literal>ar[n]</literal> is the last. The last element has the
8239 property that <literal>ar[n+1] == NULL</literal>.
8241 <example id="example.odr.element.path.record">
8242 <title>Element Path for record</title>
8244 For a database record part of a PresentResponse the
8245 array returned by <function>odr_get_element</function>
8246 is <literal>presentResponse</literal>, <literal>databaseOrSurDiagnostics</literal>, <literal>?</literal>, <literal>record</literal>, <literal>?</literal>, <literal>databaseRecord</literal> . The question mark appears due to
8247 unnamed constructions.
8251 <sect2 id="odr.diagnostics">
8252 <title>Diagnostics</title>
8254 The encoding/decoding functions all return 0 when an error occurs.
8255 Until you call <function>odr_reset()</function>, you cannot use the
8256 stream again, and any function called will immediately return 0.
8259 To provide information to the programmer or administrator, the function
8262 void odr_perror(ODR o, char *message);
8265 is provided, which prints the <literal>message</literal> argument to
8266 <literal>stderr</literal> along with an error message from the stream.
8269 You can also use the function
8272 int odr_geterror(ODR o);
8275 to get the current error number from the screen. The number will be
8276 one of these constants:
8278 <table frame="top" id="odr.error.codes">
8279 <title>ODR Error codes</title>
8284 <entry>Description</entry>
8289 <entry>OMEMORY</entry><entry>Memory allocation failed.</entry>
8292 <entry>OSYSERR</entry><entry>A system- or library call has failed.
8293 The standard diagnostic variable <literal>errno</literal> should be
8294 examined to determine the actual error.</entry>
8297 <entry>OSPACE</entry><entry>No more space for encoding.
8298 This will only occur when the user has explicitly provided a
8299 buffer for an encoding stream without allowing the system to
8300 allocate more space.</entry>
8303 <entry>OREQUIRED</entry><entry>This is a common protocol error; A
8304 required data element was missing during encoding or decoding.</entry>
8307 <entry>OUNEXPECTED</entry><entry>An unexpected data element was
8308 found during decoding.</entry>
8311 <entry>OOTHER</entry><entry>Other error. This is typically an
8312 indication of misuse of the &odr; system by the programmer, and also
8313 that the diagnostic system isn't as good as it should be, yet.</entry>
8319 The character string array
8325 can be indexed by the error code to obtain a human-readable
8326 representation of the problem.
8329 <sect2 id="odr.summary.and.synopsis">
8330 <title>Summary and Synopsis</title>
8332 #include <yaz/odr.h>
8334 ODR odr_createmem(int direction);
8336 void odr_destroy(ODR o);
8338 void odr_reset(ODR o);
8340 char *odr_getbuf(ODR o, int *len, int *size);
8342 void odr_setbuf(ODR o, char *buf, int len, int can_grow);
8344 void *odr_malloc(ODR o, int size);
8346 NMEM odr_extract_mem(ODR o);
8348 int odr_geterror(ODR o);
8350 void odr_perror(ODR o, const char *message);
8352 extern char *odr_errlist[];
8356 <sect1 id="odr.programming">
8357 <title>Programming with ODR</title>
8359 The API of &odr; is designed to reflect the structure of ASN.1, rather
8360 than BER itself. Future releases may be able to represent data in
8361 other external forms.
8365 There is an ASN.1 tutorial available at
8366 <ulink url="&url.asn.1.tutorial;">this site</ulink>.
8367 This site also has standards for ASN.1 (X.680) and BER (X.690)
8368 <ulink url="&url.asn.1.standards;">online</ulink>.
8372 The ODR interface is based loosely on that of the Sun Microsystems
8374 Specifically, each function which corresponds to an ASN.1 primitive
8375 type has a dual function. Depending on the settings of the ODR
8376 stream which is supplied as a parameter, the function may be used
8377 either to encode or decode data. The functions that can be built
8378 using these primitive functions, to represent more complex data types,
8379 share this quality. The result is that you only have to enter the
8380 definition for a type once - and you have the functionality of encoding,
8381 decoding (and pretty-printing) all in one unit.
8382 The resulting C source code is quite compact, and is a pretty
8383 straightforward representation of the source ASN.1 specification.
8386 In many cases, the model of the XDR functions works quite well in this
8388 In others, it is less elegant. Most of the hassle comes from the optional
8389 SEQUENCE members which don't exist in XDR.
8391 <sect2 id="odr.primitive.asn1.types">
8392 <title>The Primitive ASN.1 Types</title>
8394 ASN.1 defines a number of primitive types (many of which correspond
8395 roughly to primitive types in structured programming languages, such as C).
8397 <sect3 id="odr.integer">
8398 <title>INTEGER</title>
8400 The &odr; function for encoding or decoding (or printing) the ASN.1
8401 INTEGER type looks like this:
8404 int odr_integer(ODR o, Odr_int **p, int optional, const char *name);
8407 The <literal>Odr_int</literal> is just a simple integer.
8410 This form is typical of the primitive &odr; functions. They are named
8411 after the type of data that they encode or decode. They take an &odr;
8412 stream, an indirect reference to the type in question, and an
8413 <literal>optional</literal> flag (corresponding to the OPTIONAL keyword
8414 of ASN.1) as parameters. They all return an integer value of either one
8416 When you use the primitive functions to construct encoders for complex
8417 types of your own, you should follow this model as well. This
8418 ensures that your new types can be reused as elements in yet more
8422 The <literal>o</literal> parameter should obviously refer to a properly
8423 initialized &odr; stream of the right type (encoding/decoding/printing)
8424 for the operation that you wish to perform.
8427 When encoding or printing, the function first looks at
8428 <literal>* p</literal>. If <literal>* p</literal> (the pointer pointed
8429 to by <literal>p</literal>) is a null pointer, this is taken to mean that
8430 the data element is absent. If the <literal>optional</literal> parameter
8431 is nonzero, the function will return one (signifying success) without
8432 any further processing. If the <literal>optional</literal> is zero, an
8433 internal error flag is set in the &odr; stream, and the function will
8434 return 0. No further operations can be carried out on the stream without
8435 a call to the function <function>odr_reset()</function>.
8438 If <literal>*p</literal> is not a null pointer, it is expected to
8439 point to an instance of the data type. The data will be subjected to
8440 the encoding rules, and the result will be placed in the buffer held
8441 by the &odr; stream.
8444 The other ASN.1 primitives have similar functions that operate in
8448 <sect3 id="odr.boolean">
8449 <title>BOOLEAN</title>
8451 int odr_bool(ODR o, Odr_bool **p, int optional, const char *name);
8454 <sect3 id="odr.real">
8460 <sect3 id="odr.null">
8463 int odr_null(ODR o, Odr_null **p, int optional, const char *name);
8466 In this case, the value of **p is not important. If <literal>*p</literal>
8467 is different from the null pointer, the null value is present, otherwise
8471 <sect3 id="odr.octet.string">
8472 <title>OCTET STRING</title>
8474 typedef struct odr_oct
8480 int odr_octetstring(ODR o, Odr_oct **p, int optional,
8484 The <literal>buf</literal> field should point to the character array
8485 that holds the octetstring. The <literal>len</literal> field holds the
8487 The character array need not be null terminated.
8490 To make things a little easier, an alternative is given for string
8491 types that are not expected to contain embedded NULL characters (eg.
8495 int odr_cstring(ODR o, char **p, int optional, const char *name);
8498 Which encoded or decodes between OCTETSTRING representations and
8499 null-terminates C strings.
8502 Functions are provided for the derived string types, eg:
8505 int odr_visiblestring(ODR o, char **p, int optional,
8509 <sect3 id="odr.bit.string">
8510 <title>BIT STRING</title>
8512 int odr_bitstring(ODR o, Odr_bitmask **p, int optional,
8516 The opaque type <literal>Odr_bitmask</literal> is only suitable for
8517 holding relatively brief bit strings, eg. for options fields, etc.
8518 The constant <literal>ODR_BITMASK_SIZE</literal> multiplied by 8
8519 gives the maximum possible number of bits.
8522 A set of macros are provided for manipulating the
8523 <literal>Odr_bitmask</literal> type:
8526 void ODR_MASK_ZERO(Odr_bitmask *b);
8528 void ODR_MASK_SET(Odr_bitmask *b, int bitno);
8530 void ODR_MASK_CLEAR(Odr_bitmask *b, int bitno);
8532 int ODR_MASK_GET(Odr_bitmask *b, int bitno);
8535 The functions are modeled after the manipulation functions that
8536 accompany the <literal>fd_set</literal> type used by the
8537 <function>select(2)</function> call.
8538 <literal>ODR_MASK_ZERO</literal> should always be called first on a
8539 new bitmask, to initialize the bits to zero.
8542 <sect3 id="odr.object.identifier">
8543 <title>OBJECT IDENTIFIER</title>
8545 int odr_oid(ODR o, Odr_oid **p, int optional, const char *name);
8548 The C OID representation is simply an array of integers, terminated by
8549 the value -1 (the <literal>Odr_oid</literal> type is synonymous with
8550 the <literal>short</literal> type).
8551 We suggest that you use the OID database module (see
8552 <xref linkend="tools.oid.database"/>) to handle object identifiers
8553 in your application.
8557 <sect2 id="odr.tagging.primitive.types">
8558 <title>Tagging Primitive Types</title>
8560 The simplest way of tagging a type is to use the
8561 <function>odr_implicit_tag()</function> or
8562 <function>odr_explicit_tag()</function> macros:
8565 int odr_implicit_tag(ODR o, Odr_fun fun, int class, int tag,
8566 int optional, const char *name);
8568 int odr_explicit_tag(ODR o, Odr_fun fun, int class, int tag,
8569 int optional, const char *name);
8572 To create a type derived from the integer type by implicit tagging, you
8576 MyInt ::= [210] IMPLICIT INTEGER
8579 In the &odr; system, this would be written like:
8582 int myInt(ODR o, Odr_int **p, int optional, const char *name)
8584 return odr_implicit_tag(o, odr_integer, p,
8585 ODR_CONTEXT, 210, optional, name);
8589 The function <function>myInt()</function> can then be used like any of
8590 the primitive functions provided by &odr;. Note that the behavior of
8591 <function>odr_explicit_tag()</function>
8592 and <function>odr_implicit_tag()</function> macros
8593 act exactly the same as the functions they are applied to - they
8594 respond to error conditions, etc, in the same manner - they
8595 simply have three extra parameters. The class parameter may
8596 take one of the values: <literal>ODR_CONTEXT</literal>,
8597 <literal>ODR_PRIVATE</literal>, <literal>ODR_UNIVERSAL</literal>, or
8598 <literal>/ODR_APPLICATION</literal>.
8601 <sect2 id="odr.constructed.types">
8602 <title>Constructed Types</title>
8604 Constructed types are created by combining primitive types. The
8605 &odr; system only implements the SEQUENCE and SEQUENCE OF constructions
8606 (although adding the rest of the container types should be simple
8607 enough, if the need arises).
8610 For implementing SEQUENCEs, the functions
8613 int odr_sequence_begin(ODR o, void *p, int size, const char *name);
8614 int odr_sequence_end(ODR o);
8620 The <function>odr_sequence_begin()</function> function should be
8621 called in the beginning of a function that implements a SEQUENCE type.
8622 Its parameters are the &odr; stream, a pointer (to a pointer to the type
8623 you're implementing), and the <literal>size</literal> of the type
8624 (typically a C structure). On encoding, it returns 1 if
8625 <literal>* p</literal> is a null pointer. The <literal>size</literal>
8626 parameter is ignored. On decoding, it returns 1 if the type is found in
8627 the data stream. <literal>size</literal> bytes of memory are allocated,
8628 and <literal>*p</literal> is set to point to this space.
8629 <function>odr_sequence_end()</function> is called at the end of the
8630 complex function. Assume that a type is defined like this:
8633 MySequence ::= SEQUENCE {
8635 boolval BOOLEAN OPTIONAL
8639 The corresponding &odr; encoder/decoder function and the associated data
8640 structures could be written like this:
8643 typedef struct MySequence
8649 int mySequence(ODR o, MySequence **p, int optional, const char *name)
8651 if (odr_sequence_begin(o, p, sizeof(**p), name) == 0)
8652 return optional && odr_ok(o);
8654 odr_integer(o, &(*p)->intval, 0, "intval") &&
8655 odr_bool(o, &(*p)->boolval, 1, "boolval") &&
8656 odr_sequence_end(o);
8660 Note the 1 in the call to <function>odr_bool()</function>, to mark
8661 that the sequence member is optional.
8662 If either of the member types had been tagged, the macros
8663 <function>odr_implicit_tag()</function> or
8664 <function>odr_explicit_tag()</function>
8665 could have been used.
8666 The new function can be used exactly like the standard functions provided
8667 with &odr;. It will encode, decode or pretty-print a data value of the
8668 <literal>MySequence</literal> type. We like to name types with an
8669 initial capital, as done in ASN.1 definitions, and to name the
8670 corresponding function with the first character of the name in lower case.
8671 You could, of course, name your structures, types, and functions any way
8672 you please - as long as you're consistent, and your code is easily readable.
8673 <literal>odr_ok</literal> is just that - a predicate that returns the
8674 state of the stream. It is used to ensure that the behavior of the new
8675 type is compatible with the interface of the primitive types.
8678 <sect2 id="odr.tagging.constructed.types">
8679 <title>Tagging Constructed Types</title>
8682 See <xref linkend="odr.tagging.primitive.types"/> for information
8683 on how to tag the primitive types, as well as types that are
8687 <sect3 id="odr.implicit.tagging">
8688 <title>Implicit Tagging</title>
8690 Assume the type above had been defined as
8693 MySequence ::= [10] IMPLICIT SEQUENCE {
8695 boolval BOOLEAN OPTIONAL
8699 You would implement this in &odr; by calling the function
8702 int odr_implicit_settag(ODR o, int class, int tag);
8705 which overrides the tag of the type immediately following it. The
8706 macro <function>odr_implicit_tag()</function> works by calling
8707 <function>odr_implicit_settag()</function> immediately
8708 before calling the function pointer argument.
8709 Your type function could look like this:
8712 int mySequence(ODR o, MySequence **p, int optional, const char *name)
8714 if (odr_implicit_settag(o, ODR_CONTEXT, 10) == 0 ||
8715 odr_sequence_begin(o, p, sizeof(**p), name) == 0)
8716 return optional && odr_ok(o);
8718 odr_integer(o, &(*p)->intval, 0, "intval") &&
8719 odr_bool(o, &(*p)->boolval, 1, "boolval") &&
8720 odr_sequence_end(o);
8724 The definition of the structure <literal>MySequence</literal> would be
8728 <sect3 id="odr.explicit.tagging">
8729 <title>Explicit Tagging</title>
8731 Explicit tagging of constructed types is a little more complicated,
8732 since you are in effect adding a level of construction to the data.
8735 Assume the definition:
8738 MySequence ::= [10] IMPLICIT SEQUENCE {
8740 boolval BOOLEAN OPTIONAL
8744 Since the new type has an extra level of construction, two new functions
8745 are needed to encapsulate the base type:
8748 int odr_constructed_begin(ODR o, void *p, int class, int tag,
8751 int odr_constructed_end(ODR o);
8754 Assume that the IMPLICIT in the type definition above were replaced
8755 with EXPLICIT (or that the IMPLICIT keyword were simply deleted, which
8756 would be equivalent). The structure definition would look the same,
8757 but the function would look like this:
8760 int mySequence(ODR o, MySequence **p, int optional, const char *name)
8762 if (odr_constructed_begin(o, p, ODR_CONTEXT, 10, name) == 0)
8763 return optional && odr_ok(o);
8764 if (o->direction == ODR_DECODE)
8765 *p = odr_malloc(o, sizeof(**p));
8766 if (odr_sequence_begin(o, p, sizeof(**p), 0) == 0)
8768 *p = 0; /* this is almost certainly a protocol error */
8772 odr_integer(o, &(*p)->intval, 0, "intval") &&
8773 odr_bool(o, &(*p)->boolval, 1, "boolval") &&
8774 odr_sequence_end(o) &&
8775 odr_constructed_end(o);
8779 Notice that the interface here gets kind of nasty. The reason is
8780 simple: Explicitly tagged, constructed types are fairly rare in
8781 the protocols that we care about, so the
8782 esthetic annoyance (not to mention the dangers of a cluttered
8783 interface) is less than the time that would be required to develop a
8784 better interface. Nevertheless, it is far from satisfying, and it's a
8785 point that will be worked on in the future. One option for you would
8786 be to simply apply the <function>odr_explicit_tag()</function> macro to
8787 the first function, and not
8788 have to worry about <function>odr_constructed_*</function> yourself.
8789 Incidentally, as you might have guessed, the
8790 <function>odr_sequence_</function> functions are themselves
8791 implemented using the <function>/odr_constructed_</function> functions.
8795 <sect2 id="odr.sequence.of">
8796 <title>SEQUENCE OF</title>
8798 To handle sequences (arrays) of a specific type, the function
8801 int odr_sequence_of(ODR o, int (*fun)(ODR o, void *p, int optional),
8802 void *p, int *num, const char *name);
8805 The <literal>fun</literal> parameter is a pointer to the decoder/encoder
8806 function of the type. <literal>p</literal> is a pointer to an array of
8807 pointers to your type. <literal>num</literal> is the number of elements
8814 MyArray ::= SEQUENCE OF INTEGER
8817 The C representation might be
8820 typedef struct MyArray
8827 And the function might look like
8830 int myArray(ODR o, MyArray **p, int optional, const char *name)
8832 if (o->direction == ODR_DECODE)
8833 *p = odr_malloc(o, sizeof(**p));
8834 if (odr_sequence_of(o, odr_integer, &(*p)->elements,
8835 &(*p)->num_elements, name))
8838 return optional && odr_ok(o);
8842 <sect2 id="odr.choice.types">
8843 <title>CHOICE Types</title>
8845 The choice type is used fairly often in some ASN.1 definitions, so
8846 some work has gone into streamlining its interface.
8849 CHOICE types are handled by the function:
8852 int odr_choice(ODR o, Odr_arm arm[], void *p, void *whichp,
8856 The <literal>arm</literal> array is used to describe each of the possible
8857 types that the CHOICE type may assume. Internally in your application,
8858 the CHOICE type is represented as a discriminated union. That is, a
8859 C union accompanied by an integer (or enum) identifying the active
8861 <literal>whichp</literal> is a pointer to the union discriminator.
8862 When encoding, it is examined to determine the current type.
8863 When decoding, it is set to reference the type that was found in
8867 The Odr_arm type is defined thus:
8870 typedef struct odr_arm
8881 The interpretation of the fields are:
8885 <term>tagmode</term>
8886 <listitem><para>Either <literal>ODR_IMPLICIT</literal>,
8887 <literal>ODR_EXPLICIT</literal>, or <literal>ODR_NONE</literal> (-1)
8888 to mark no tagging.</para></listitem>
8892 <listitem><para>The value of the discriminator that corresponds to
8893 this CHOICE element. Typically, it will be a #defined constant, or
8894 an enum member.</para></listitem>
8898 <listitem><para>A pointer to a function that implements the type of
8899 the CHOICE member. It may be either a standard &odr; type or a type
8900 defined by yourself.</para></listitem>
8904 <listitem><para>Name of tag.</para></listitem>
8908 A handy way to prepare the array for use by the
8909 <function>odr_choice()</function> function is to
8910 define it as a static, initialized array in the beginning of your
8911 decoding/encoding function. Assume the type definition:
8914 MyChoice ::= CHOICE {
8916 tagged [99] IMPLICIT INTEGER,
8921 Your C type might look like
8924 typedef struct MyChoice
8941 And your function could look like this:
8944 int myChoice(ODR o, MyChoice **p, int optional, const char *name)
8946 static Odr_arm arm[] =
8948 {-1, -1, -1, MyChoice_untagged, odr_integer, "untagged"},
8949 {ODR_IMPLICIT, ODR_CONTEXT, 99, MyChoice_tagged, odr_integer,
8951 {-1, -1, -1, MyChoice_other, odr_boolean, "other"},
8955 if (o->direction == ODR_DECODE)
8956 *p = odr_malloc(o, sizeof(**p);
8958 return optional && odr_ok(o);
8960 if (odr_choice(o, arm, &(*p)->u, &(*p)->which), name)
8963 return optional && odr_ok(o);
8967 In some cases (say, a non-optional choice which is a member of a
8968 sequence), you can "embed" the union and its discriminator in the
8969 structure belonging to the enclosing type, and you won't need to
8970 fiddle with memory allocation to create a separate structure to
8971 wrap the discriminator and union.
8974 The corresponding function is somewhat nicer in the Sun XDR interface.
8975 Most of the complexity of this interface comes from the possibility of
8976 declaring sequence elements (including CHOICEs) optional.
8979 The ASN.1 specifications naturally requires that each member of a
8980 CHOICE have a distinct tag, so they can be told apart on decoding.
8981 Sometimes it can be useful to define a CHOICE that has multiple types
8982 that share the same tag. You'll need some other mechanism, perhaps
8983 keyed to the context of the CHOICE type. In effect, we would like to
8984 introduce a level of context-sensitiveness to our ASN.1 specification.
8985 When encoding an internal representation, we have no problem, as long
8986 as each CHOICE member has a distinct discriminator value. For
8987 decoding, we need a way to tell the choice function to look for a
8988 specific arm of the table. The function
8991 void odr_choice_bias(ODR o, int what);
8994 provides this functionality. When called, it leaves a notice for the next
8995 call to <function>odr_choice()</function> to be called on the decoding
8996 stream <literal>o</literal> that only the <literal>arm</literal> entry with
8997 a <literal>which</literal> field equal to <literal>what</literal>
9001 The most important application (perhaps the only one, really) is in
9002 the definition of application-specific EXTERNAL encoders/decoders
9003 which will automatically decode an ANY member given the direct or
9008 <sect1 id="odr.debugging">
9009 <title>Debugging</title>
9011 The protocol modules are suffering somewhat from a lack of diagnostic
9012 tools at the moment. Specifically ways to pretty-print PDUs that
9013 aren't recognized by the system. We'll include something to this end
9014 in a not-too-distant release. In the meantime, what we do when we get
9015 packages we don't understand is to compile the ODR module with
9016 <literal>ODR_DEBUG</literal> defined. This causes the module to dump tracing
9017 information as it processes data units. With this output and the
9018 protocol specification (Z39.50), it is generally fairly easy to see
9023 <chapter id="comstack">
9024 <title>The COMSTACK Module</title>
9025 <sect1 id="comstack.synopsis">
9026 <title>Synopsis (blocking mode)</title>
9027 <programlisting><![CDATA[
9030 int size = 0, length_incoming;
9031 char server_address_str[] = "localhost:9999";
9032 void *server_address_ip;
9035 char *protocol_package = "GET / HTTP/1.0\r\n\r\n";
9036 int protocol_package_length = strlen(protocol_package);
9038 stack = cs_create(tcpip_type, 1, PROTO_HTTP);
9040 perror("cs_create"); /* use perror() here since we have no stack yet */
9044 server_address_ip = cs_straddr(stack, server_address_str);
9045 if (!server_address_ip) {
9046 fprintf(stderr, "cs_straddr: address could not be resolved\n");
9050 status = cs_connect(stack, server_address_ip);
9052 fprintf(stderr, "cs_connect: %s\n", cs_strerror(stack));
9056 status = cs_rcvconnect(stack);
9058 fprintf(stderr, "cs_rcvconnect: %s\n", cs_strerror(stack));
9062 status = cs_put(stack, protocol_package, protocol_package_length);
9064 fprintf(stderr, "cs_put: %s\n", cs_strerror(stack));
9068 /* Now get a response */
9069 length_incoming = cs_get(stack, &buf, &size);
9070 if (!length_incoming) {
9071 fprintf(stderr, "Connection closed\n");
9073 } else if (length_incoming < 0) {
9074 fprintf(stderr, "cs_get: %s\n", cs_strerror(stack));
9079 fwrite(buf, length_incoming, 1, stdout);
9090 <sect1 id="comstack.introduction">
9091 <title>Introduction</title>
9094 subsystem provides a transparent interface to different types of transport
9095 stacks for the exchange of BER-encoded data and HTTP packets.
9096 At present, the RFC1729 method (BER over TCP/IP), local UNIX socket and an
9097 experimental SSL stack are supported, but others may be added in time.
9098 The philosophy of the
9099 module is to provide a simple interface by hiding unused options and
9100 facilities of the underlying libraries. This is always done at the risk
9101 of losing generality, and it may prove that the interface will need
9106 There hasn't been interest in the XTImOSI stack for some years.
9107 Therefore, it is no longer supported.
9111 The interface is implemented in such a fashion that only the
9112 sub-layers constructed to the transport methods that you wish to
9113 use in your application are linked in.
9116 You will note that even though simplicity was a goal in the design,
9117 the interface is still orders of magnitudes more complex than the
9118 transport systems found in many other packages. One reason is that
9119 the interface needs to support the somewhat different requirements of
9120 the different lower-layer communications stacks; another important
9121 reason is that the interface seeks to provide a more or less
9122 industrial-strength approach to asynchronous event-handling.
9123 When no function is allowed to block, things get more complex -
9124 particularly on the server side.
9125 We urge you to have a look at the demonstration client and server
9126 provided with the package. They are meant to be easily readable and
9127 instructive, while still being at least moderately useful.
9130 <sect1 id="comstack.common">
9131 <title>Common Functions</title>
9132 <sect2 id="comstack.managing.endpoints">
9133 <title>Managing Endpoints</title>
9135 COMSTACK cs_create(CS_TYPE type, int blocking, int protocol);
9138 Creates an instance of the protocol stack - a communications endpoint.
9139 The <literal>type</literal> parameter determines the mode
9140 of communication. At present the following values are supported:
9144 <term><literal>tcpip_type</literal></term>
9145 <listitem><para>TCP/IP (BER over TCP/IP or HTTP over TCP/IP)
9149 <term><literal>ssl_type</literal></term>
9150 <listitem><para>Secure Socket Layer (SSL). This COMSTACK
9151 is experimental and is not fully implemented. If
9152 HTTP is used, this effectively is HTTPS.
9156 <term><literal>unix_type</literal></term>
9157 <listitem><para>Unix socket (unix only). Local Transfer via
9158 file socket. See <citerefentry><refentrytitle>unix</refentrytitle>
9159 <manvolnum>7</manvolnum></citerefentry>.
9164 The <function>cs_create</function> function returns a null-pointer
9165 if a system error occurs.
9166 The <literal>blocking</literal> parameter should be one if
9167 you wish the association to operate in blocking mode, zero otherwise.
9168 The <literal>protocol</literal> field should be
9169 <literal>PROTO_Z3950</literal> or <literal>PROTO_HTTP</literal>.
9170 Protocol <literal>PROTO_SR</literal> is no longer supported.
9173 void cs_close(COMSTACK handle);
9176 Closes the connection (as elegantly as the lower layers will permit),
9177 and releases the resources pointed to by the
9178 <literal>handle</literal>
9180 <literal>handle</literal>
9181 should not be referenced again after this call.
9185 We really need a soft disconnect, don't we?
9189 <sect2 id="comstack.data.exchange">
9190 <title>Data Exchange</title>
9192 int cs_put(COMSTACK handle, char *buf, int len);
9195 Sends <literal>buf</literal> down the wire.
9196 In blocking mode, this function will return only when a full buffer has
9197 been written, or an error has occurred. In nonblocking mode, it's
9198 possible that the function will be unable to send the full buffer
9199 at once, which will be indicated by a return value of 1.
9200 The function will keep track of the number of octets already written; you
9201 should call it repeatedly with the same values of <literal>buf</literal>
9202 and <literal>len</literal>, until the buffer has been transmitted.
9203 When a full buffer has been sent, the function will return 0 for
9204 success. -1 indicates an error condition (see below).
9207 int cs_get(COMSTACK handle, char **buf, int *size);
9210 Receives a PDU or HTTP Response from the peer. Returns the number of
9212 In nonblocking mode, it is possible that not all of the packet can be
9213 read at once. In this case, the function returns 1. To simplify the
9214 interface, the function is
9215 responsible for managing the size of the buffer. It will be reallocated
9216 if necessary to contain large packages, and will sometimes be moved
9217 around internally by the subsystem when partial packages are read. Before
9219 <function>cs_get</function>
9220 for the fist time, the buffer can be initialized to the null pointer,
9221 and the length should also be set to 0 - cs_get will perform a
9222 <function>malloc(2)</function>
9223 on the buffer for you. When a full buffer has been read, the size of
9224 the package is returned (which will always be greater than 1). -1
9225 indicates an error condition.
9228 See also the <function>cs_more()</function> function below.
9231 int cs_more(COMSTACK handle);
9234 The <function>cs_more()</function> function should be used in conjunction
9235 with <function>cs_get</function> and
9236 <function>select(2)</function>.
9237 The <function>cs_get()</function> function will sometimes
9238 (notably in the TCP/IP mode) read more than a single protocol package
9239 off the network. When this happens, the extra package is stored
9240 by the subsystem. After calling <function>cs_get()</function>, and before
9241 waiting for more input, You should always call
9242 <function>cs_more()</function>
9243 to check if there's a full protocol package already read. If
9244 <function>cs_more()</function>
9246 <function>cs_get()</function>
9247 can be used to immediately fetch the new package. For the
9249 subsystem, the function should always return 0, but if you want your
9250 stuff to be protocol independent, you should use it.
9254 The <function>cs_more()</function>
9255 function is required because the RFC1729-method
9256 does not provide a way of separating individual PDUs, short of
9257 partially decoding the BER. Some other implementations will carefully
9258 nibble at the packet by calling
9259 <function>read(2)</function>
9260 several times. This was felt to be too inefficient (or at least
9261 clumsy) - hence the call for this extra function.
9265 int cs_look(COMSTACK handle);
9268 This function is useful when you're operating in nonblocking
9270 <function>select(2)</function>
9271 tells you there's something happening on the line. It returns one of
9272 the following values:
9276 <term>CS_NONE</term>
9278 No event is pending. The data found on the line was not a
9283 <term>CS_CONNECT</term>
9285 A response to your connect request has been received. Call
9286 <function>cs_rcvconnect</function>
9287 to process the event and to finalize the connection establishment.
9291 <term>CS_DISCON</term>
9293 The other side has closed the connection (or maybe sent a disconnect
9294 request - but do we care? Maybe later). Call
9295 <function>cs_close</function> to close your end of the association
9300 <term>CS_LISTEN</term>
9302 A connect request has been received.
9303 Call <function>cs_listen</function> to process the event.
9307 <term>CS_DATA</term>
9309 There's data to be found on the line.
9310 Call <function>cs_get</function> to get it.
9316 You should be aware that even if
9317 <function>cs_look()</function>
9318 tells you that there's an event event pending, the corresponding
9319 function may still return and tell you there was nothing to be found.
9320 This means that only part of a package was available for reading. The
9321 same event will show up again, when more data has arrived.
9325 int cs_fileno(COMSTACK h);
9328 Returns the file descriptor of the association. Use this when
9329 file-level operations on the endpoint are required
9330 (<function>select(2)</function> operations, specifically).
9334 <sect1 id="comstack.client">
9335 <title>Client Side</title>
9337 int cs_connect(COMSTACK handle, void *address);
9340 Initiate a connection with the target at <literal>address</literal>
9341 (more on addresses below). The function will return 0 on success, and 1 if
9342 the operation does not complete immediately (this will only
9343 happen on a nonblocking endpoint). In this case, use
9344 <function>cs_rcvconnect</function> to complete the operation,
9345 when <function>select(2)</function> or <function>poll(2)</function>
9346 reports input pending on the association.
9349 int cs_rcvconnect(COMSTACK handle);
9352 Complete a connect operation initiated by <function>cs_connect()</function>.
9353 It will return 0 on success; 1 if the operation has not yet completed (in
9354 this case, call the function again later); -1 if an error has occurred.
9357 <sect1 id="comstack.server">
9358 <title>Server Side</title>
9360 To establish a server under the <application>inetd</application>
9364 COMSTACK cs_createbysocket(int socket, CS_TYPE type, int blocking,
9368 The <literal>socket</literal> parameter is an established socket (when
9369 your application is invoked from <application>inetd</application>, the
9370 socket will typically be 0.
9371 The following parameters are identical to the ones for
9372 <function>cs_create</function>.
9375 int cs_bind(COMSTACK handle, void *address, int mode)
9378 Binds a local address to the endpoint. Read about addresses below. The
9379 <literal>mode</literal> parameter should be either
9380 <literal>CS_CLIENT</literal> or <literal>CS_SERVER</literal>.
9383 int cs_listen(COMSTACK handle, char *addr, int *addrlen);
9386 Call this to process incoming events on an endpoint that has been
9387 bound in listening mode. It will return 0 to indicate that the connect
9388 request has been received, 1 to signal a partial reception, and -1 to
9389 indicate an error condition.
9392 COMSTACK cs_accept(COMSTACK handle);
9395 This finalizes the server-side association establishment, after
9396 cs_listen has completed successfully. It returns a new connection
9397 endpoint, which represents the new association. The application will
9398 typically wish to fork off a process to handle the association at this
9399 point, and continue listen for new connections on the old
9400 <literal>handle</literal>.
9403 You can use the call
9406 const char *cs_addrstr(COMSTACK);
9409 on an established connection to retrieve the host-name of the remote host.
9413 You may need to use this function with some care if your
9414 name server service is slow or unreliable
9418 <sect1 id="comstack.addresses">
9419 <title>Addresses</title>
9421 The low-level format of the addresses are different depending on the
9422 mode of communication you have chosen. A function is provided by each
9423 of the lower layers to map a user-friendly string-form address to the
9424 binary form required by the lower layers.
9427 void *cs_straddr(COMSTACK handle, const char *str);
9430 The format for TCP/IP and SSL addresses is:
9433 <host> [ ':' <portnum> ]
9436 The <literal>hostname</literal> can be either a domain name or an
9437 IP address. The port number, if omitted, defaults to 210.
9440 For TCP/IP and SSL, the special hostnames <literal>@</literal>,
9441 maps to <literal>IN6ADDR_ANY_INIT</literal> with
9442 IPV4 binding as well (bindv6only=0),
9443 The special hostname <literal>@4</literal> binds to
9444 <literal>INADDR_ANY</literal> (IPV4 only listener).
9445 The special hostname <literal>@6</literal> binds to
9446 <literal>IN6ADDR_ANY_INIT</literal> with bindv6only=1 (IPV6 only listener).
9449 For UNIX sockets, the format of an address is the socket filename.
9452 When a connection has been established, you can use
9455 const char *cs_addrstr(COMSTACK h);
9458 to retrieve the host name of the peer system. The function returns
9459 a pointer to a static area, which is overwritten on the next call
9463 A fairly recent addition to the &comstack; module is the utility
9467 COMSTACK cs_create_host (const char *str, int blocking, void **vp);
9470 which is just a wrapper for <function>cs_create</function> and
9471 <function>cs_straddr</function>. The <parameter>str</parameter>
9472 is similar to that described for <function>cs_straddr</function>
9473 but with a prefix denoting the &comstack; type. Prefixes supported
9474 are <literal>tcp:</literal>, <literal>unix:</literal> and
9475 <literal>ssl:</literal> for TCP/IP, UNIX and SSL respectively.
9476 If no prefix is given, then TCP/IP is used.
9477 The <parameter>blocking</parameter> is passed to
9478 function <function>cs_create</function>. The third parameter
9479 <parameter>vp</parameter> is a pointer to &comstack; stack type
9481 Parameter <parameter>vp</parameter> is reserved for future use.
9482 Set it to <literal>NULL</literal>.
9485 <sect1 id="comstack.ssl">
9489 void *cs_get_ssl(COMSTACK cs);
9491 Returns the SSL handle, <literal>SSL *</literal> for comstack. If comstack
9492 is not of type SSL, NULL is returned.
9496 int cs_set_ssl_ctx(COMSTACK cs, void *ctx);
9498 Sets SSL context for comstack. The parameter is expected to be of type
9499 <literal>SSL_CTX *</literal>. This function should be called just
9500 after comstack has been created (before connect, bind, etc).
9501 This function returns 1 for success; 0 for failure.
9505 int cs_set_ssl_certificate_file(COMSTACK cs, const char *fname);
9507 Sets SSL certificate for comstack as a PEM file. This function
9508 returns 1 for success; 0 for failure.
9512 int cs_get_ssl_peer_certificate_x509(COMSTACK cs, char **buf, int *len);
9514 This function returns the peer certificate. If successful,
9515 <literal>*buf</literal> and <literal>*len</literal> holds
9516 X509 buffer and length respectively. Buffer should be freed
9517 with <literal>xfree</literal>. This function returns 1 for success;
9521 <sect1 id="comstack.diagnostics">
9522 <title>Diagnostics</title>
9524 All functions return -1 if an error occurs. Typically, the functions
9525 will return 0 on success, but the data exchange functions
9526 (<function>cs_get</function>, <function>cs_put</function>,
9527 <function>cs_more</function>) follow special rules. Consult their
9531 The error code for the COMSTACK can be retrieved using C macro
9532 <function>cs_errno</function> which will return one
9533 of the error codes <literal>CSYSERR</literal>,
9534 <literal>CSOUTSTATE</literal>,
9535 <literal>CSNODATA</literal>, ...
9538 int cs_errno(COMSTACK handle);
9541 You can the textual representation of the error code
9542 by using <function>cs_errmsg</function> - which
9543 works like <function>strerror(3)</function>
9546 const char *cs_errmsg(int n);
9549 It is also possible to get straight to the textual represenataion
9550 without the error code by using
9551 <function>cs_strerror</function>.
9554 const char *cs_strerror(COMSTACK h);
9557 <sect1 id="comstack.summary">
9558 <title>Summary and Synopsis</title>
9560 #include <yaz/comstack.h>
9562 #include <yaz/tcpip.h> /* this is for TCP/IP and SSL support */
9563 #include <yaz/unix.h> /* this is for UNIX socket support */
9565 COMSTACK cs_create(CS_TYPE type, int blocking, int protocol);
9567 COMSTACK cs_createbysocket(int s, CS_TYPE type, int blocking,
9569 COMSTACK cs_create_host(const char *str, int blocking,
9572 int cs_bind(COMSTACK handle, int mode);
9574 int cs_connect(COMSTACK handle, void *address);
9576 int cs_rcvconnect(COMSTACK handle);
9578 int cs_listen(COMSTACK handle);
9580 COMSTACK cs_accept(COMSTACK handle);
9582 int cs_put(COMSTACK handle, char *buf, int len);
9584 int cs_get(COMSTACK handle, char **buf, int *size);
9586 int cs_more(COMSTACK handle);
9588 void cs_close(COMSTACK handle);
9590 int cs_look(COMSTACK handle);
9592 void *cs_straddr(COMSTACK handle, const char *str);
9594 const char *cs_addrstr(COMSTACK h);
9599 <chapter id="future">
9600 <title>Future Directions</title>
9602 We have a new and better version of the front-end server on the drawing
9603 board. Resources and external commitments will govern when we'll be
9604 able to do something real with it. Features should include greater
9605 flexibility, greater support for access/resource control, and easy
9606 support for Explain (possibly with Zebra as an extra database engine).
9609 &yaz; is a BER toolkit and as such should support all protocols
9610 out there based on that. We'd like to see running ILL applications.
9611 It shouldn't be that hard. Another thing that would be interesting is
9612 LDAP. Maybe a generic framework for doing IR using both LDAP and
9613 Z39.50 transparently.
9616 The SOAP implementation is incomplete. In the future we hope
9617 to add more features to it. Perhaps make a WSDL/XML Schema compiler.
9618 The authors of libxml2 are already working on XML Schema / RelaxNG
9619 compilers so this may not be too hard.
9622 It would be neat to have a proper module mechanism for the Generic
9623 Frontend Server so that backend would be dynamically
9624 loaded (as shared objects / DLLs).
9627 Other than that, &yaz; generally moves in the directions which appear to
9628 make the most people happy (including ourselves, as prime users of the
9629 software). If there's something you'd like to see in here, then drop
9630 us a note and let's see what we can come up with.
9633 <reference id="reference">
9634 <title>Reference</title>
9635 <partintro id="reference-introduction">
9637 The material in this chapter is drawn directly from the individual
9643 <appendix id="list-oids">
9644 <title>List of Object Identifiers</title>
9646 These is a list of object identifiers that are built into YAZ.
9650 <appendix id="bib1-diagnostics">
9651 <title>Bib-1 diagnostics</title>
9653 List of Bib-1 diagnostics that are known to YAZ.
9657 <appendix id="sru-diagnostics">
9658 <title>SRU diagnostics</title>
9660 List of SRU diagnostics that are known to YAZ.
9664 <appendix id="license">
9665 <title>License</title>
9666 <sect1 id="license.indexdata">
9667 <title>Index Data Copyright</title>
9669 Copyright © ©right-year; Index Data.
9672 All rights reserved.
9675 Redistribution and use in source and binary forms, with or without
9676 modification, are permitted provided that the following conditions are met:
9681 Redistributions of source code must retain the above copyright
9682 notice, this list of conditions and the following disclaimer.
9687 Redistributions in binary form must reproduce the above copyright
9688 notice, this list of conditions and the following disclaimer in the
9689 documentation and/or other materials provided with the distribution.
9694 Neither the name of Index Data nor the names of its contributors
9695 may be used to endorse or promote products derived from this
9696 software without specific prior written permission.
9701 THIS SOFTWARE IS PROVIDED BY INDEX DATA ``AS IS'' AND ANY
9702 EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
9703 WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
9704 DISCLAIMED. IN NO EVENT SHALL INDEX DATA BE LIABLE FOR
9705 ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
9706 DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
9707 SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
9708 CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
9709 LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
9710 OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
9715 <appendix id="indexdata">
9716 <title>About Index Data</title>
9718 Index Data is a consulting and software-development enterprise that
9719 specializes in library and information management systems. Our
9720 interests and expertise span a broad range of related fields, and one
9721 of our primary, long-term objectives is the development of a powerful
9722 information management
9723 system with open network interfaces and hyper-media capabilities.
9725 We make this software available free of charge, on a fairly unrestrictive
9726 license; as a service to the networking community, and to further the
9727 development of quality software for open network communication.
9729 We'll be happy to answer questions about the software, and about ourselves
9735 <street>Amagerfælledvej 56</street>
9736 <postcode>2300 Copenhagen S</postcode>
9737 <country>Denmark</country>
9738 Email <email>info@indexdata.dk</email>
9742 The Hacker's Jargon File has the following to say about the
9744 prefix "YA" in the name of a software product.
9748 Yet Another. adj. 1. Of your own work: A
9749 humorous allusion often used in titles to acknowledge that the
9750 topic is not original, though the content is. As in "Yet Another
9751 AI Group" or "Yet Another Simulated Annealing Algorithm".
9753 others' work: Describes something of which there are already far
9758 <appendix id="credits">
9759 <title>Credits</title>
9761 This appendix lists individuals that have contributed in the development
9762 of &yaz;. Some have contributed with code, while others have provided bug
9763 fixes or suggestions. If we're missing somebody, of if you, for
9764 whatever reason, don't like to be listed here, let us know.
9774 Morten Bøgeskov
9795 Mads Bondo Dydensborg
9804 Morten Garkier Hendriksen
9861 Tom André Øverland
9867 <!-- Keep this comment at the end of the file
9870 nxml-child-indent: 1