1 <?xml version="1.0" standalone="no"?>
2 <!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.4//EN"
3 "http://www.oasis-open.org/docbook/xml/4.4/docbookx.dtd"
5 <!ENTITY % local SYSTEM "local.ent">
7 <!ENTITY % entities SYSTEM "entities.ent">
9 <!ENTITY % idcommon SYSTEM "common/common.ent">
14 <title>YAZ User's Guide and Reference</title>
16 <author><firstname>Sebastian</firstname><surname>Hammer</surname></author>
17 <author><firstname>Adam</firstname><surname>Dickmeiss</surname></author>
18 <author><firstname>Mike</firstname><surname>Taylor</surname></author>
19 <author><firstname>Heikki</firstname><surname>Levanto</surname></author>
20 <author><firstname>Dennis</firstname><surname>Schafroth</surname></author>
22 <releaseinfo>&version;</releaseinfo>
24 <year>©right-year;</year>
25 <holder>Index Data</holder>
29 This document is the programmer's guide and reference to the &yaz;
30 package version &version;. &yaz; is a compact toolkit that provides
31 access to the Z39.50 and SRU/Solr protocols, as well as a set of
32 higher-level tools for implementing the server and client
34 The documentation can be used on its own, or as a reference when
35 looking at the example applications provided with the package.
40 <imagedata fileref="common/id.png" format="PNG"/>
43 <imagedata fileref="common/id.eps" format="EPS"/>
48 <chapter id="introduction">
49 <title>Introduction</title>
51 &yaz; is a C/C++ library for information retrieval applications
52 using the Z39.50/SRU/Solr protocols for information retrieval.
60 <ulink url="&url.z39.50;">Z39.50</ulink> version 3 support.
61 Amendments and Z39.50-2002 revision is supported.
67 <ulink url="&url.sru;">SRU GET/POST/SOAP</ulink>
68 version 1.1, 1.2 and 2.0 (over HTTP and HTTPS).
73 Includes BER encoders/decoders for the
74 <ulink url="&url.ill;">ISO ILL</ulink>
81 <ulink url="&url.solr;">Solr</ulink> Web Service version 1.4.x
87 Supports the following transports: BER over TCP/IP
88 (<ulink url="&url.ber.over.tcpip;">RFC1729</ulink>),
89 BER over unix local socket, and
90 <ulink url="&url.http.1.1;">HTTP 1.1</ulink>.
95 Secure Socket Layer support using
96 <ulink url="&url.gnutls;">GnuTLS</ulink>.
97 If enabled, &yaz; uses HTTPS transport (for SOAP) or
98 "Secure BER" (for Z39.50).
104 <ulink url="&url.zoom;">ZOOM</ulink> C API implementing
105 Z39.50, SRU and Solr Web Service.
110 The &yaz; library offers a set of useful utilities
111 related to the protocols, such as MARC (ISO2709) parser,
112 CCL (ISO8777) parser,
113 <ulink url="&url.cql;">CQL</ulink>
114 parser, memory management routines, character set conversion.
119 Portable code. &yaz; compiles out-of-the box on most Unixes and
120 on Windows using Microsoft Visual C++.
125 Fast operation. The C based BER encoders/decoders as well
126 as the server component of &yaz; is very fast.
131 Liberal license that allows for commercial use of &yaz;.
137 <sect1 id="introduction.reading">
138 <title>Reading this Manual</title>
140 Most implementors only need to read a fraction of the
141 material in thie manual, so a quick walkthrough of the chapters
147 <xref linkend="installation"/> contains installation
148 instructions for &yaz;. You don't need reading this
149 if you expect to download &yaz; binaries.
150 However, the chapter contains information about how
151 to make <emphasis>your</emphasis> application link
157 <xref linkend="zoom"/> describes the ZOOM API of &yaz;.
158 This is definitely worth a read if you wish to develop a Z39.50/SRU
164 <xref linkend="server"/> describes the generic frontend server
165 and explains how to develop server Z39.50/SRU applications for &yaz;.
166 Obviously worth reading if you're to develop a server.
171 <xref linkend="yaz-client"/> describes how to use the &yaz; Z39.50
172 client. If you're developer and wish to test your server
173 or a server from another party, you might find this chapter
179 <xref linkend="asn"/> documents the most commonly used Z39.50
180 C data structures offered by the &yaz; API. Client
181 developers using ZOOM and non-Z39.50 implementors may skip this.
186 <xref linkend="soap"/> describes how SRU and SOAP is used
187 in &yaz;. Only if you're developing SRU applications
188 this section is a must.
193 <xref linkend="tools"/> contains sections for the various
194 tools offered by &yaz;. Scan through the material quickly
195 and see what's relevant to you! SRU implementors
196 might find the <link linkend="cql">CQL</link> section
202 <xref linkend="odr"/> goes through the details of the
203 ODR module which is the work horse that encodes and decodes
204 BER packages. Implementors using ZOOM only, do <emphasis>not</emphasis>
206 Most other Z39.50 implementors only need to read the first two
207 sections (<xref linkend="odr.introduction"/> and
208 <xref linkend="odr.use"/>).
213 <xref linkend="comstack"/> describes the network layer module
214 COMSTACK. Implementors using ZOOM or the generic frontend server
215 may skip this. Others, presumably, handling client/server
216 communication on their own should read this.
221 <sect1 id="introduction.api">
222 <title>The API</title>
224 The <ulink url="&url.yaz;">&yaz;</ulink>
225 toolkit offers several different levels of access to the
226 <ulink url="&url.z39.50;">ISO23950/Z39.50</ulink>,
227 <ulink url="&url.ill;">ILL</ulink> and
228 <ulink url="&url.sru;">SRU</ulink>
230 The level that you need to use depends on your requirements, and
231 the role (server or client) that you want to implement.
232 If you're developing a client application you should consider the
233 <link linkend="zoom">ZOOM</link> API.
234 It is, by far, the easiest way to develop clients in C.
235 Server implementers should consider the
236 <link linkend="server">generic frontend server</link>.
237 None of those high-level APIs support the whole protocol, but
238 they do include most facilities used in existing Z39.50 applications.
241 If you're using 'exotic' functionality (meaning anything not included in
242 the high-level APIs), developing non-standard extensions to Z39.50 or
243 you're going to develop an ILL application you'll have to learn the lower
247 The YAZ toolkit modules is shown in figure <xref linkend="yaz.layer"/>.
249 <figure id="yaz.layer">
250 <title>YAZ layers</title>
253 <imagedata fileref="apilayer.png" format="PNG"/>
256 <imagedata fileref="apilayer.eps" format="EPS"/>
261 There are four layers.
264 <para>A client or server application (or both).
265 This layer includes ZOOM and the generic frontend server.
270 The second layer provides a C represenation of the
271 protocol units (packages) for Z39.50 ASN.1, ILL ASN.1,
277 The third layer encodes and decodes protocol data units to
278 simple packages (buffer with certain length). The &odr; module
279 encodes and decodes BER whereas the HTTP modules encodes and
280 decodes HTTP ruquests/responses.
285 The lowest layer is &comstack; which exchanges the encoded packages
286 with a peer process over a network.
292 The &asn; module represents the ASN.1 definition of
293 the Z39.50 protocol. It establishes a set of type and
294 structure definitions, with one structure for each of the top-level
295 PDUs, and one structure or type for each of the contained ASN.1 types.
296 For primitive types, or other types that are defined by the ASN.1
297 standard itself (such as the EXTERNAL type), the C representation is
298 provided by the &odr; (Open Data Representation) subsystem.
301 &odr; is a basic mechanism for representing an
302 ASN.1 type in the C programming language, and for implementing BER
303 encoders and decoders for values of that type. The types defined in
304 the &asn; module generally have the prefix <literal>Z_</literal>, and
305 a suffix corresponding to the name of the type in the ASN.1
306 specification of the protocol (generally Z39.50-1995). In the case of
307 base types (those originating in the ASN.1 standard itself), the prefix
308 <literal>Odr_</literal> is sometimes seen. Either way, look for
309 the actual definition in either <filename>z-core.h</filename> (for the types
310 from the protocol), <filename>odr.h</filename> (for the primitive ASN.1
312 The &asn; library also provides functions (which are, in turn,
313 defined using &odr; primitives) for encoding and decoding data values.
314 Their general form is
316 <funcprototype><funcdef>int <function>z_<replaceable>xxx</replaceable></function></funcdef>
317 <paramdef>ODR <parameter>o</parameter></paramdef>
318 <paramdef>Z_<replaceable>xxx</replaceable> **<parameter>p</parameter></paramdef>
319 <paramdef>int <parameter>optional</parameter></paramdef>
320 <paramdef>const char *<parameter>name</parameter></paramdef>
323 (note the lower-case "z" in the function name)
327 If you are using the premade definitions of the &asn; module, and you
328 are not adding new protocol of your own, the only parts of &odr; that you
329 need to worry about are documented in
330 <xref linkend="odr.use"/>.
334 When you have created a BER-encoded buffer, you can use the &comstack;
335 subsystem to transmit (or receive) data over the network. The &comstack;
336 module provides simple functions for establishing a connection
337 (passively or actively, depending on the role of your application),
338 and for exchanging BER-encoded PDUs over that connection. When you
339 create a connection endpoint, you need to specify what transport to
340 use (TCP/IP, SSL or UNIX sockets).
341 For the remainder of the connection's lifetime, you don't have
342 to worry about the underlying transport protocol at all - the &comstack;
343 will ensure that the correct mechanism is used.
346 We call the combined interfaces to &odr;, &asn;, and &comstack; the service
347 level API. It's the API that most closely models the Z39.50
348 service/protocol definition, and it provides unlimited access to all
349 fields and facilities of the protocol definitions.
352 The reason that the &yaz; service-level API is a conglomerate of the
353 APIs from three different submodules is twofold. First, we wanted to allow
354 the user a choice of different options for each major task. For instance,
355 if you don't like the protocol API provided by &odr;/&asn;, you
356 can use SNACC or BERUtils instead, and still have the benefits of the
357 transparent transport approach of the &comstack; module. Secondly,
358 we realize that you may have to fit the toolkit into an existing
359 event-processing structure, in a way that is incompatible with
360 the &comstack; interface or some other part of &yaz;.
364 <chapter id="installation">
365 <title>Compilation and Installation</title>
366 <sect1 id="installation-introduction">
367 <title>Introduction</title>
369 The latest version of the software will generally be found at:
372 <ulink url="&url.yaz.download;"/>
375 We have tried our best to keep the software portable, and on many
376 platforms, you should be able to compile everything with little or
380 The software is regularly tested on
381 <ulink url="&url.debian;">Debian GNU/Linux</ulink>,
382 <ulink url="&url.centos;">CentOS</ulink>,
383 <ulink url="&url.ubuntu;">Ubuntu Linux</ulink>,
384 <ulink url="&url.freebsd;">FreeBSD (i386)</ulink>,
385 <ulink url="&url.macosx;">MAC OSX</ulink>,
386 <ulink url="&url.solaris;">Solaris</ulink>,
387 Windows 7, Windows XP.
390 Some versions have be known to work on HP/UX,
391 DEC Unix, <ulink url="&url.netbsd;">NetBSD</ulink>,
392 <ulink url="&url.openbsd;">OpenBSD</ulink>,
394 Data General DG/UX (with some CFLAGS tinkering),
395 SGI/IRIX, DDE Supermax, Apple Macintosh (using the Codewarrior programming
396 environment and the GUSI socket libraries),
400 If you move the software to other platforms, we'd be grateful if you'd
401 let us know about it. If you run into difficulties, we will try to help
402 if we can, and if you solve the problems, we would be happy to include
403 your fixes in the next release. So far, we have mostly avoided
404 <literal>#ifdefs</literal> for individual platforms, and we'd
405 like to keep it that way as far as it makes sense.
408 We maintain a mailing-list for the purpose of announcing new releases and
409 bug-fixes, as well as general discussion. Subscribe by
411 <ulink url="&url.yaz.mailinglist;">here</ulink>.
412 General questions and problems can be directed at
413 <ulink url="&url.yaz.mail;"/>, or the address given at the top of
417 <sect1 id="installation.unix"><title>UNIX</title>
420 <ulink url="&url.debian;">Debian GNU/Linux</ulink> (i386 and amd64),
421 <ulink url="&url.ubuntu;">Ubuntu</ulink> (i386 and amd64)
423 <ulink url="&url.centos;">CentOS</ulink> (amd64 only) packages for &yaz;.
424 You should be able to create packages for other CPUs by building
425 them from the source package.
428 YAZ is also part of several packages repositories. Some of them are
433 Solaris CSW: <ulink url="http://www.opencsw.org/packages/yaz/"/>
438 Solaris: <ulink url="http://unixpackages.com"/>
443 FreeBSD: <ulink url="http://www.freshports.org/net/yaz"/>
448 Debian: <ulink url="http://packages.debian.org/search?keywords=yaz"/>
453 Ubuntu: <ulink url="https://launchpad.net/ubuntu/+source/yaz"/>
459 <ulink url="http://ftp.netbsd.org/pub/pkgsrc/current/pkgsrc/net/yaz/README.html"/>
463 <sect2 id="installation.source.unix">
464 <title>Compiling from source on Unix</title>
466 Note that if your system doesn't have a native ANSI C compiler, you may
467 have to acquire one separately. We recommend
468 <ulink url="&url.gcc;">GCC</ulink>.
471 If you wish to use character set conversion facilities in &yaz; or if you
472 are compiling &yaz; for use with Zebra it is a good idea to ensure that
473 the iconv library is installed. Some Unixes today already have it
475 <ulink url="&url.libiconv;">GNU libiconv</ulink>.
478 YAZ 3.0.16 and later includes a wrapper for the
479 <ulink url="&url.icu;">ICU</ulink>
480 (International Components for Unicode).
481 In order to use this, the developer version of the ICU library
482 must be available. ICU support is recommended for applications
483 such as Pazpar2 and Zebra.
486 The <ulink url="&url.libxslt;">libxslt</ulink>,
487 <ulink url="&url.libxml2;">libxml2</ulink> librararies are required
488 if &yaz; is to support SRU/Solr.
489 These libraries are very portable and should compile out-of-the
490 box on virtually all Unix platforms. It is available in binary
491 forms for Linux and others.
495 <ulink url="&url.autoconf;">Autoconf</ulink>,
496 <ulink url="&url.automake;">Automake</ulink> and
497 <ulink url="&url.libtool;">Libtool</ulink>
498 are used to generate Makefiles and configure &yaz; for the system.
499 You do <emphasis>not</emphasis> these tools unless you're using the
500 Git version of &yaz;.
503 The CQL parser for &yaz; is built using
504 GNU <ulink url="&url.bison;">Bison</ulink>.
505 This tool is only needed if you're using the Git version of &yaz;.
508 &yaz; includes a tiny ASN.1 compiler. This compiler is
509 written in <ulink url="&url.tcl;">Tcl</ulink>.
510 But as for Bison you do not need it unless you're using Git
511 version of &yaz; or you're using the compiler to built own codecs
515 Generally it should be sufficient to run configure without options,
522 The configure script attempts to use use the C compiler specified by
523 the <literal>CC</literal> environment variable. If not set, GNU C will be
524 used if it is available. The <literal>CFLAGS</literal> environment
525 variable holds options to be passed to the C compiler. If you're using
526 Bourne-compatible shell you may pass something like this to use a
527 particular C compiler with optimization enabled:
530 CC=/opt/ccs/bin/cc CFLAGS=-O ./configure
533 To customize &yaz;, the configure script also accepts a set of options.
534 The most important are:
538 <literal>--prefix</literal>=<replaceable>prefix</replaceable>
541 <para>Specifies installation prefix for &yaz;. This is
542 only needed if you run <literal>make install</literal> later to
543 perform a "system" installation. The prefix is
544 <literal>/usr/local</literal> if not specified.
550 <literal>--enable-tcpd</literal>
553 <para>The front end server will be built using Wietse's
554 <ulink url="&url.tcpwrapper;">TCP wrapper library</ulink>.
555 It allows you to allow/deny clients depending on IP number.
556 The TCP wrapper library is often used in GNU/Linux and
560 <refentrytitle>hosts_access</refentrytitle>
561 <manvolnum>5</manvolnum>
565 <refentrytitle>tcpd</refentrytitle>
566 <manvolnum>8</manvolnum>
573 <literal>--enable-threads</literal>
576 <para>&yaz; will be built using POSIX threads.
577 Specifically, <constant>_REENTRANT</constant> will be defined during
584 <literal>--disable-shared</literal>
587 <para>The make process will not create shared
588 libraries (also known as shared objects <filename>.so</filename>).
589 By default, shared libraries are created -
590 equivalent to <literal>--enable-shared</literal>.
596 <literal>--disable-shared</literal>
599 <para>The make process will not create
600 static libraries (<filename>.a</filename>).
601 By default, static libraries are created -
602 equivalent to <literal>--enable-static</literal>.
608 <literal>--with-iconv</literal>[=<replaceable>prefix</replaceable>]
611 <para>Compile &yaz; with iconv library in directory
612 <replaceable>prefix</replaceable>. By default configure will
613 search for iconv on the system. Use this option if it
614 doesn't find iconv. Alternatively,
615 <literal>--without-iconv</literal>, can be uset to force &yaz;
622 <literal>--with-xslt</literal>[=<replaceable>prefix</replaceable>]
625 <para>Compile &yaz; with
626 <ulink url="&url.libxslt;">libxslt</ulink> in directory
627 <replaceable>prefix</replaceable>.
628 Use this option if you want XSLT and XML support.
629 By default, configure will
630 search for libxslt on the system. Use this option if it
631 libxslt is not found automatically. Alternatively,
632 <literal>--without-xslt</literal>, can be used to force &yaz;
639 <literal>--with-xml2</literal>[=<replaceable>prefix</replaceable>]
642 <para>Compile &yaz; with
643 <ulink url="&url.libxml2;">libxml2</ulink> in directory
644 <replaceable>prefix</replaceable>.
645 Use this option if you want &yaz; to use XML and support SRU/Solr.
646 By default, configure will
647 search for libxml2 on the system. Use this option if it
648 libxml2 is not found automatically. Alternatively,
649 <literal>--without-xml2</literal>, can be used to force &yaz;
653 Note that option <literal>--with-xslt</literal>
654 also enables libxml2.
660 <literal>--with-gnutls</literal>[=<replaceable>prefix</replaceable>]
663 <para>&yaz; will be linked with the GNU TLS libraries and
664 an SSL COMSTACK will be provided. By default configure enables
665 SSL support for YAZ if the GNU TLS development libraries are found
672 <literal>--with-icu</literal>[=<replaceable>prefix</replaceable>]
675 <para>&yaz; will be linked the
676 <ulink url="&url.icu;">ICU</ulink> library in the prefix if given.
677 If prefix is not given, the libraries exposed by the script
678 <application>icu-config</application> will be used if found.
685 <literal>--with-libgcrypt</literal>[=<replaceable>prefix</replaceable>]
688 <para>&yaz; will be linked with
689 <ulink url="&url.libgcrypt;">Libgcrypt</ulink> in the prefix if given.
690 If prefix is not given, the libraries exposed by the script
691 <application>libgcrypt-config</application> will be used if found.
697 <literal>--with-memcached</literal>
700 <para>&yaz; will be linked with
701 <ulink url="&url.libmemcached;">libMemcached</ulink> to allow
702 for result-set caching for ZOOM.
703 The prefix can not be given. Note that YAZ will only search
704 for libMemcached if Libgcrypt is also enabled.
705 Note that 0.40 of libmemcached is required.
711 <literal>--with-redis</literal>
714 <para>&yaz; will be linked with the hiredis C library
715 to allow for result-set caching for ZOOM on a
716 <ulink url="&url.redis;">redis</ulink> server.
717 The prefix can not be given. Note that YAZ will only search
718 for hiredis if Libgcrypt is also enabled.
726 When configured, build the software by typing:
732 The following files are generated by the make process:
735 <term><filename>src/libyaz.la</filename></term>
737 Main &yaz; library. This is no ordinary library. It's
739 By default, &yaz; creates a static library in
740 <filename>lib/.libs/libyaz.a</filename>.
744 <term><filename>src/libyaz_server.la</filename></term>
746 Generic Frontend server. This is an add-on for libyaz.la.
747 Code in this library uses POSIX threads functions - if POSIX
748 threads are available on the platform.
752 <term><filename>src/libyaz_icu.la</filename></term>
754 Functions that wrap the ICU library.
758 <term><filename>ztest/yaz-ztest</filename></term>
759 <listitem><para>Test Z39.50 server.
763 <term><filename>client/yaz-client</filename></term>
764 <listitem><para>Z39.50 client for testing the protocol.
765 See chapter <link linkend="yaz-client">
766 YAZ client</link> for more information.
770 <term><filename>util/yaz-config</filename></term>
771 <listitem><para>A Bourne-shell script, generated by configure, that
772 specifies how external applications should compile - and link with
777 <term><filename>util/yaz-asncomp</filename></term>
778 <listitem><para>The ASN.1 compiler for &yaz;. Requires the
779 Tcl Shell, <application>tclsh</application>, in
780 <literal>PATH</literal> to operate.
784 <term><filename>util/yaz-iconv</filename></term>
785 <listitem><para>This program converts data in one character set to
786 another. This command exercises the YAZ character set
791 <term><filename>util/yaz-marcdump</filename></term>
792 <listitem><para>This program parses ISO2709 encoded MARC records
793 and prints them in line-format or XML.
797 <term><filename>util/yaz-icu</filename></term>
798 <listitem><para>This program exposes the ICU wrapper library if that
799 is enabled for YAZ. Only if ICU is available this program is
804 <term><filename>util/yaz-url</filename></term>
805 <listitem><para>This program is a simple HTTP page fetcher ala
810 <term><filename>zoom/zoomsh</filename></term>
812 A simple shell implemented on top of the
813 <link linkend="zoom">ZOOM</link> functions.
814 The shell is a command line application that allows you to enter
815 simple commands to perform ZOOM operations.
819 <term><filename>zoom/zoomtst1</filename>,
820 <filename>zoom/zoomtst2</filename>, ..</term>
822 Several small applications that demonstrates the ZOOM API.
828 If you wish to install &yaz; in system directories
829 <filename>/usr/local/bin</filename>,
830 <filename>/usr/local/lib</filename> .. etc, you can type:
836 You probably need to have root access in order to perform this.
837 You must specify the <literal>--prefix</literal> option for configure if
838 you wish to install &yaz; in other directories than the default
839 <filename>/usr/local/</filename>.
842 If you wish to perform an un-installation of &yaz;, use:
848 This will only work if you haven't reconfigured &yaz; (and therefore
849 changed installation prefix). Note that uninstall will not
850 remove directories created by make install, e.g.
851 <filename>/usr/local/include/yaz</filename>.
854 <sect2 id="installation-linking-yaz-unix">
855 <title>How to make apps using YAZ on UNIX</title>
857 This section describes how to compile - and link your own
858 applications using the &yaz; toolkit.
859 If you're used to Makefiles this shouldn't be hard. As for
860 other libraries you have used before, you have to set a proper include
861 path for your C/C++ compiler and specify the location of
862 &yaz; libraries. You can do it by hand, but generally we suggest
863 you use the <filename>yaz-config</filename> that is generated
864 by <filename>configure</filename>. This is especially
865 important if you're using the threaded version of &yaz; which
866 require you to pass more options to your linker/compiler.
869 The <filename>yaz-config</filename> script accepts command line
870 options that makes the <filename>yaz-config</filename> script print
871 options that you should use in your make process.
872 The most important ones are:
873 <literal>--cflags</literal>, <literal>--libs</literal>
874 which prints C compiler flags, and linker flags respectively.
877 A small and complete <literal>Makefile</literal> for a C
878 application consisting of one source file,
879 <filename>myprog.c</filename>, may look like this:
881 YAZCONFIG=/usr/local/bin/yaz-config
882 CFLAGS=`$(YAZCONFIG) --cflags`
883 LIBS=`$(YAZCONFIG) --libs`
885 $(CC) $(CFLAGS) -o myprog myprog.o $(LIBS)
889 The CFLAGS variable consists of a C compiler directive that will set
890 the include path to the <emphasis>parent</emphasis> directory
891 of <filename>yaz</filename>. That is, if &yaz; header files were
892 installed in <filename>/usr/local/include/yaz</filename>,
893 then include path is set to <filename>/usr/local/include</filename>.
894 Therefore, in your applications you should use
896 #include <yaz/proto.h>
898 and <emphasis>not</emphasis>
900 #include <proto.h>
904 For Libtool users, the <filename>yaz-config</filename> script provides
905 a different variant of option <literal>--libs</literal>, called
906 <literal>--lalibs</literal> that returns the name of the
907 Libtool archive(s) for &yaz; rather than the ordinary ones.
910 For applications using the threaded version of &yaz;,
911 specify <literal>threads</literal> after the
912 other options. When <literal>threads</literal> is given,
913 more flags and linker flags will be printed by
914 <filename>yaz-config</filename>. If our previous example was
915 using threads, you'd have to modify the lines that set
916 <literal>CFLAGS</literal> and <literal>LIBS</literal> as
919 CFLAGS=`$(YAZCONFIG) --cflags threads`
920 LIBS=`$(YAZCONFIG) --libs threads`
922 There is no need specify POSIX thread libraries in your Makefile.
923 The <literal>LIBS</literal> variable includes that as well.
927 <sect1 id="installation.win32">
929 <para>The easiest way to install YAZ on Windows is by downloading
931 <ulink url="&url.yaz.download.win32;">here</ulink>.
932 The installer comes with source too - in case you wish to
933 compile YAZ with different compiler options, etc.
936 <sect2 id="installation.win32.source">
937 <title>Compiling from Source on WIN32</title>
939 &yaz; is shipped with "makefiles" for the NMAKE tool that comes
940 with <ulink url="&url.vstudio;">
941 Microsoft Visual Studio</ulink>. It has been tested with
942 Microsoft Visual Studio 2003/2005/2008.
945 Start a command prompt and switch the sub directory
946 <filename>WIN</filename> where the file <filename>makefile</filename>
947 is located. Customize the installation by editing the
948 <filename>makefile</filename> file (for example by using notepad).
949 The following summarizes the most important settings in that file:
952 <term><literal>DEBUG</literal></term>
954 If set to 1, the software is
955 compiled with debugging libraries (code generation is
956 multi-threaded debug DLL).
957 If set to 0, the software is compiled with release libraries
958 (code generation is multi-threaded DLL).
962 <term><literal>HAVE_TCL</literal>, <literal>TCL</literal></term>
964 If <literal>HAVE_TCL</literal> is set to 1, nmake will
965 use the ASN.1 compiler (<ulink url="&url.tcl;">Tcl</ulink> based).
966 You must set <literal>TCL</literal> to the full path of the Tcl
967 interpreter. A Windows version of Tcl is part of
968 <ulink url="&url.gitwindows;">Git for Windows</ulink>.
971 If you do not have Tcl installed, set
972 <literal>HAVE_TCL</literal> to 0.
976 <term><literal>HAVE_BISON</literal>,
977 <literal>BISON</literal></term>
979 If GNU Bison is present, you might set <literal>HAVE_BISON</literal>
980 to 1 and specify the Bison executable in <literal>BISON</literal>.
981 Bison is only required if you use the Git version of
982 YAZ or if you modify the grammar for CQL
983 (<filename>cql.y</filename>).
986 A Windows version of GNU Bison is part of
987 <ulink url="&url.gitwindows;">Git for Windows</ulink>.
991 <term><literal>HAVE_ICONV</literal>,
992 <literal>ICONV_DIR</literal></term>
994 If <literal>HAVE_ICONV</literal> is set to 1, YAZ is compiled
995 with iconv support. In this configuration, set
996 <literal>ICONV_DIR</literal> to the iconv source directory.
1000 <term><literal>HAVE_LIBXML2</literal>,
1001 <literal>LIBXML2_DIR</literal></term>
1004 If <literal>HAVE_LIBXML2</literal> is set to 1, YAZ is compiled
1005 with SRU support. In this configuration, set
1006 <literal>LIBXML2_DIR</literal> to the
1007 <ulink url="&url.libxml2;">libxml2</ulink> source directory.
1010 Refer to <xref linkend="installation.win32.libxml2"/>.
1015 <term><literal>HAVE_LIBXSLT</literal>,
1016 <literal>LIBXSLT_DIR</literal></term>
1019 If <literal>HAVE_LIBXSLT</literal> is set to 1, YAZ is compiled
1020 with XSLT support. In this configuration, set
1021 <literal>LIBXSLT_DIR</literal> to the
1022 <ulink url="&url.libxslt;">libxslt</ulink> source directory.
1026 libxslt depends libxml2.
1032 <term><literal>HAVE_ICU</literal>,
1033 <literal>ICU_DIR</literal></term>
1036 If <literal>HAVE_ICU</literal> is set to 1, YAZ is compiled
1037 with <ulink url="&url.icu;">ICU</ulink> support.
1038 In this configuration, set
1039 <literal>ICU_DIR</literal> to the
1040 <ulink url="&url.icu;">ICU</ulink> source directory.
1047 When satisfied with the settings in the makefile, type
1054 If the <filename>nmake</filename> command is not found on your system
1055 you probably haven't defined the environment variables required to
1056 use that tool. To fix that, find and run the batch file
1057 <filename>vcvars32.bat</filename>. You need to run it from within
1058 the command prompt or set the environment variables "globally";
1059 otherwise it doesn't work.
1063 If you wish to recompile &yaz; - for example if you modify
1064 settings in the <filename>makefile</filename> you can delete
1065 object files, etc by running.
1071 The following files are generated upon successful compilation:
1074 <term><filename>bin/yaz&soversion;.dll</filename> /
1075 <filename>bin/yaz&soversion;d.dll</filename></term>
1077 &yaz; Release/Debug DLL.
1081 <term><filename>lib/yaz&soversion;.lib</filename> /
1082 <filename>lib/yaz&soversion;d.lib</filename></term>
1084 Import library for <filename>yaz&soversion;.dll</filename> /
1085 <filename>yaz&soversion;d.dll</filename>.
1089 <term><filename>bin/yaz_cond&soversion;.dll</filename> /
1090 <filename>bin/yaz_cond&soversion;d.dll</filename></term>
1092 Release/Debug DLL for condition variable utilities (condvar.c).
1096 <term><filename>lib/yaz_cond&soversion;.lib</filename> /
1097 <filename>lib/yaz_cond&soversion;d.lib</filename></term>
1099 Import library for <filename>yaz_cond&soversion;.dll</filename> /
1100 <filename>yaz_cond&soversion;d.dll</filename>.
1104 <term><filename>bin/yaz_icu&soversion;.dll</filename> /
1105 <filename>bin/yaz_icu&soversion;d.dll</filename></term>
1107 Release/Debug DLL for the ICU wrapper utility.
1108 Only build if HAVE_ICU is 1.
1112 <term><filename>lib/yaz_icu&soversion;.lib</filename> /
1113 <filename>lib/yaz_icu&soversion;d.lib</filename></term>
1115 Import library for <filename>yaz_icu&soversion;.dll</filename> /
1116 <filename>yaz_icu&soversion;d.dll</filename>.
1120 <term><filename>bin/yaz-ztest.exe</filename></term>
1122 Z39.50 multi-threaded test/example server. It's a WIN32
1123 console application.
1127 <term><filename>bin/yaz-client.exe</filename></term>
1129 &yaz; Z39.50 client application. It's a WIN32 console application.
1130 See chapter <link linkend="yaz-client">YAZ client</link> for more
1135 <term><filename>bin/yaz-icu.exe</filename></term>
1136 <listitem><para>This program exposes the ICU wrapper library if that
1137 is enabled for YAZ. Only if ICU is available this program is
1142 <term><filename>bin/zoomsh.exe</filename></term>
1144 Simple console application implemented on top of the
1145 <link linkend="zoom">ZOOM</link> functions.
1146 The application is a command line shell that allows you to enter
1147 simple commands to perform ZOOM operations.
1151 <term><filename>bin/zoomtst1.exe</filename>,
1152 <filename>bin/zoomtst2.exe</filename>, ..</term>
1154 Several small applications that demonstrates the ZOOM API.
1161 <sect2 id="installation-linking-yaz-win32">
1162 <title>How to make apps using YAZ on Windows</title>
1164 This section will go though the process of linking your Windows
1165 applications with &yaz;.
1168 Some people are confused by the fact that we use the nmake
1169 tool to build &yaz;. They think they have to do that too - in order
1170 to make their Windows applications work with &yaz;. The good news is that
1171 you don't have to. You can use the integrated environment of
1172 Visual Studio if desired for your own application.
1175 When setting up a project or Makefile you have to set the following:
1178 <term>include path</term>
1180 Set it to the <filename>include</filename> directory of &yaz;.
1184 <term>import library <filename>yaz&soversion;.lib</filename></term>
1186 You must link with this library. It's located in the
1187 sub directory <filename>lib</filename> of &yaz;.
1188 If you want to link with the debug version of &yaz;, you must
1189 link against <filename>yaz&soversion;d.lib</filename> instead.
1193 <term>dynamic link library
1194 <filename>yaz&soversion;.dll</filename>
1197 This DLL must be in your execution path when you invoke
1198 your application. Specifically, you should distribute this
1199 DLL with your application.
1206 <sect2 id="installation.win32.libxml2">
1207 <title>Compiling Libxml2 and Libxslt on windows</title>
1209 Download libxml2 and Libxslt source and unpack it.
1210 In the example below we install Libxml2 2.9.2 and Libxslt 1.1.28
1211 for 32-bit, so we use the destination directories
1212 libxml2.2.9.2.win32 and libxslt-1.1.28.win32 to reflect both
1213 version and architecture.
1216 cscript configure.js prefix=c:\libxml2-2.9.2.win32 iconv=no
1222 For Libxslt it is similar. We must ensure that compilation of
1223 Libxslt links against the already installed libxml2.
1226 cscript configure.js prefix=c:\libxslt-1.1.28.win32 iconv=no \
1227 lib=c:\libxmlt-2.9.2.win32\lib \
1228 include=c:\libxmlt-2.9.2.win32\include\libxml2
1238 ### Still to document:
1239 ZOOM_connection_errcode(c)
1240 ZOOM_connection_errmsg(c)
1241 ZOOM_connection_addinfo(c)
1242 ZOOM_connection_addinfo(c)
1243 ZOOM_connection_diagset(c);
1244 ZOOM_connection_save_apdu_wrbuf
1245 ZOOM_diag_str(error)
1246 ZOOM_resultset_record_immediate(s, pos)
1247 ZOOM_resultset_cache_reset(r)
1248 ZOOM_options_set_callback(opt, function, handle)
1249 ZOOM_options_create_with_parent2(parent1, parent2)
1250 ZOOM_options_getl(opt, name, len)
1251 ZOOM_options_setl(opt, name, value, len)
1252 ZOOM_options_get_bool(opt, name, defa)
1253 ZOOM_options_get_int(opt, name, defa)
1254 ZOOM_options_set_int(opt, name, value)
1259 &zoom; is an acronym for 'Z39.50 Object-Orientation Model' and is
1260 an initiative started by Mike Taylor (Mike is from the UK, which
1261 explains the peculiar name of the model). The goal of &zoom; is to
1262 provide a common Z39.50 client API not bound to a particular
1263 programming language or toolkit.
1266 From YAZ version 2.1.12, <ulink url="&url.sru;">SRU</ulink> is supported.
1267 You can make SRU ZOOM connections by specifying scheme
1268 <literal>http://</literal> for the hostname for a connection.
1269 The dialect of SRU used is specified by the value of the
1270 connection's <literal>sru</literal> option, which may be SRU over
1271 HTTP GET (<literal>get</literal>),
1272 SRU over HTTP POST (<literal>post</literal>), (SRU over
1273 SOAP) (<literal>soap</literal>) or <literal>solr</literal>
1274 (<ulink url="&url.solr;">Solr</ulink> Web Service).
1275 Using the facility for embedding options in target strings, a
1276 connection can be forced to use SRU rather the SRW (the default) by
1277 prefixing the target string with <literal>sru=get,</literal>, like this:
1278 <literal>sru=get,http://sru.miketaylor.org.uk:80/sru.pl</literal>
1281 <ulink url="&url.solr;">Solr</ulink> protocol support was added to
1282 YAZ in version 4.1.0, as a dialect of a SRU protocol, since both are
1283 HTTP based protocols.
1286 The lack of a simple Z39.50 client API for &yaz; has become more
1287 and more apparent over time. So when the first &zoom; specification
1289 an implementation for &yaz; was quickly developed. For the first time, it is
1290 now as easy (or easier!) to develop clients than servers with &yaz;. This
1291 chapter describes the &zoom; C binding. Before going further, please
1292 reconsider whether C is the right programming language for the job.
1293 There are other language bindings available for &yaz;, and still
1295 are in active development. See the
1296 <ulink url="&url.zoom;">ZOOM web-site</ulink> for
1300 In order to fully understand this chapter you should read and
1301 try the example programs <literal>zoomtst1.c</literal>,
1302 <literal>zoomtst2.c</literal>, .. in the <literal>zoom</literal>
1306 The C language misses features found in object oriented languages
1307 such as C++, Java, etc. For example, you'll have to manually,
1308 destroy all objects you create, even though you may think of them as
1309 temporary. Most objects has a <literal>_create</literal> - and a
1310 <literal>_destroy</literal> variant.
1311 All objects are in fact pointers to internal stuff, but you don't see
1312 that because of typedefs. All destroy methods should gracefully ignore a
1313 <literal>NULL</literal> pointer.
1316 In each of the sections below you'll find a sub section called
1317 protocol behavior, that describes how the API maps to the Z39.50
1320 <sect1 id="zoom-connections">
1321 <title>Connections</title>
1322 <para>The Connection object is a session with a target.
1325 #include <yaz/zoom.h>
1327 ZOOM_connection ZOOM_connection_new(const char *host, int portnum);
1329 ZOOM_connection ZOOM_connection_create(ZOOM_options options);
1331 void ZOOM_connection_connect(ZOOM_connection c, const char *host,
1333 void ZOOM_connection_destroy(ZOOM_connection c);
1336 Connection objects are created with either function
1337 <function>ZOOM_connection_new</function> or
1338 <function>ZOOM_connection_create</function>.
1339 The former creates and automatically attempts to establish a network
1340 connection with the target. The latter doesn't establish
1341 a connection immediately, thus allowing you to specify options
1342 before establishing network connection using the function
1343 <function>ZOOM_connection_connect</function>.
1344 If the port number, <literal>portnum</literal>, is zero, the
1345 <literal>host</literal> is consulted for a port specification.
1346 If no port is given, 210 is used. A colon denotes the beginning of
1347 a port number in the host string. If the host string includes a
1348 slash, the following part specifies a database for the connection.
1351 You can prefix the host with a scheme followed by colon. The
1352 default scheme is <literal>tcp</literal> (Z39.50 protocol).
1353 The scheme <literal>http</literal> selects SRU/get over HTTP by default,
1354 but can overridded to use SRU/post, SRW and the Solr protocol.
1357 You can prefix the scheme-qualified host-string with one or more
1359 <literal><parameter>key</parameter>=<parameter>value</parameter></literal>
1360 sequences, each of which represents an option to be set into the
1361 connection structure <emphasis>before</emphasis> the
1362 protocol-level connection is forged and the initialization
1363 handshake takes place. This facility can be used to provide
1364 authentication credentials, as in host-strings such as:
1365 <literal>user=admin,password=halfAm4n,tcp:localhost:8017/db</literal>
1368 Connection objects should be destroyed using the function
1369 <function>ZOOM_connection_destroy</function>.
1372 void ZOOM_connection_option_set(ZOOM_connection c,
1373 const char *key, const char *val);
1375 void ZOOM_connection_option_setl(ZOOM_connection c,
1377 const char *val, int len);
1379 const char *ZOOM_connection_option_get(ZOOM_connection c,
1381 const char *ZOOM_connection_option_getl(ZOOM_connection c,
1386 The functions <function>ZOOM_connection_option_set</function> and
1387 <function>ZOOM_connection_option_setl</function> allows you to
1388 set an option given by <parameter>key</parameter> to the value
1389 <parameter>value</parameter> for the connection.
1390 For <function>ZOOM_connection_option_set</function>, the
1391 value is assumed to be a 0-terminated string. Function
1392 <function>ZOOM_connection_option_setl</function> specifies a
1393 value of a certain size (len).
1396 Functions <function>ZOOM_connection_option_get</function> and
1397 <function>ZOOM_connection_option_getl</function> returns
1398 the value for an option given by <parameter>key</parameter>.
1400 <table id="zoom-connection-options" frame="top">
1401 <title>ZOOM Connection Options</title>
1403 <colspec colwidth="4*" colname="name"></colspec>
1404 <colspec colwidth="7*" colname="description"></colspec>
1405 <colspec colwidth="3*" colname="default"></colspec>
1408 <entry>Option</entry>
1409 <entry>Description</entry>
1410 <entry>Default</entry>
1415 implementationName</entry><entry>Name of Your client
1416 </entry><entry>none</entry></row>
1418 user</entry><entry>Authentication user name
1419 </entry><entry>none</entry></row>
1421 group</entry><entry>Authentication group name
1422 </entry><entry>none</entry></row>
1424 password</entry><entry>Authentication password.
1425 </entry><entry>none</entry></row>
1427 authenticationMode</entry><entry>How authentication is encoded.
1428 </entry><entry>basic</entry></row>
1430 host</entry><entry>Target host. This setting is "read-only".
1431 It's automatically set internally when connecting to a target.
1432 </entry><entry>none</entry></row>
1434 proxy</entry><entry>Proxy host. If set, the logical host
1435 is encoded in the otherInfo area of the Z39.50 Init PDU
1436 with OID 1.2.840.10003.10.1000.81.1.
1437 </entry><entry>none</entry></row>
1439 clientIP</entry><entry>Client IP. If set, is
1440 encoded in the otherInfo area of a Z39.50 PDU with OID
1441 1.2.840.10003.10.1000.81.3. Holds the original IP addreses
1442 of a client. Is used of ZOOM is used in a gateway of some sort.
1443 </entry><entry>none</entry></row>
1445 async</entry><entry>If true (1) the connection operates in
1446 asynchronous operation which means that all calls are non-blocking
1448 <link linkend="zoom.events"><function>ZOOM_event</function></link>.
1449 </entry><entry>0</entry></row>
1451 maximumRecordSize</entry><entry> Maximum size of single record.
1452 </entry><entry>1 MB</entry></row>
1454 preferredMessageSize</entry><entry> Maximum size of multiple records.
1455 </entry><entry>1 MB</entry></row>
1457 lang</entry><entry> Language for negotiation.
1458 </entry><entry>none</entry></row>
1460 charset</entry><entry> Character set for negotiation.
1461 </entry><entry>none</entry></row>
1463 serverImplementationId</entry><entry>
1464 Implementation ID of server. (The old targetImplementationId
1465 option is also supported for the benefit of old applications.)
1466 </entry><entry>none</entry></row>
1468 targetImplementationName</entry><entry>
1469 Implementation Name of server. (The old
1470 targetImplementationName option is also supported for the
1471 benefit of old applications.)
1472 </entry><entry>none</entry></row>
1474 serverImplementationVersion</entry><entry>
1475 Implementation Version of server. (the old
1476 targetImplementationVersion option is also supported for the
1477 benefit of old applications.)
1478 </entry><entry>none</entry></row>
1480 databaseName</entry><entry>One or more database names
1481 separated by character plus (<literal>+</literal>), which to
1482 be used by subsequent search requests on this Connection.
1483 </entry><entry>Default</entry></row>
1485 piggyback</entry><entry>True (1) if piggyback should be
1486 used in searches; false (0) if not.
1487 </entry><entry>1</entry></row>
1489 smallSetUpperBound</entry><entry>If hits is less than or equal to this
1490 value, then target will return all records using small element set name
1491 </entry><entry>0</entry></row>
1493 largeSetLowerBound</entry><entry>If hits is greater than this
1494 value, the target will return no records.
1495 </entry><entry>1</entry></row>
1497 mediumSetPresentNumber</entry><entry>This value represents
1498 the number of records to be returned as part of a search when when
1499 hits is less than or equal to large set lower bound and if hits
1500 is greater than small set upper bound.
1501 </entry><entry>0</entry></row>
1503 smallSetElementSetName</entry><entry>
1504 The element set name to be used for small result sets.
1505 </entry><entry>none</entry></row>
1507 mediumSetElementSetName</entry><entry>
1508 The element set name to be for medium-sized result sets.
1509 </entry><entry>none</entry></row>
1511 init_opt_search, init_opt_present, init_opt_delSet, etc.</entry><entry>
1512 After a successful Init, these options may be interrogated to
1513 discover whether the server claims to support the specified
1515 </entry><entry>none</entry></row>
1517 <entry>sru</entry><entry>
1518 SRU/Solr transport type. Must be either <literal>soap</literal>,
1519 <literal>get</literal>, <literal>post</literal>, or
1520 <literal>solr</literal>.
1521 </entry><entry>soap</entry></row>
1523 sru_version</entry><entry>
1524 SRU/SRW version. Should be <literal>1.1</literal>, or
1525 <literal>1.2</literal>. This is , prior to connect, the version
1526 to offer (highest version). And following connect (in fact
1527 first operation), holds the negotiated version with the server
1528 (same or lower version).
1529 </entry><entry>1.2</entry></row>
1530 <row id="zoom.facets.option"><entry>
1531 facets</entry><entry>
1532 Requested or recommend facets may be given before a search is sent.
1533 The value of this setting is described in <xref linkend="facets"/>
1534 For inspection of the facets returned, refer to the functions
1535 described in <xref linkend="zoom.facets"/>.
1536 </entry><entry>none</entry></row>
1538 apdulog</entry><entry>
1539 If set to a true value such as "1", a log of low-level
1540 protocol packets is emitted on standard error stream. This
1541 can be very useful for debugging.
1542 </entry><entry>0</entry></row>
1544 saveAPDU</entry><entry>
1545 If set to a true value such as "1", a log of low-level
1546 protocol packets is saved. The log can be retrieved by reading
1547 option APDU. Setting saveAPDU always has the side effect of
1548 resetting the currently saved log. This setting is
1549 <emphasis>write-only</emphasis>. If read, NULL will be returned.
1550 It is only recognized in
1551 <function>ZOOM_connection_option_set</function>.
1552 </entry><entry>0</entry></row>
1555 Returns the log of protocol packets. Will be empty if logging
1556 is not enabled (see saveAPDU above). This setting is
1557 <emphasis>read-only</emphasis>. It is only recognized if used
1558 in call to <function>ZOOM_connection_option_get</function> or
1559 <function>ZOOM_connection_option_getl</function>.
1560 </entry><entry></entry></row>
1562 memcached</entry><entry>
1563 If given and non-empty,
1564 <ulink url="&url.libmemcached;">libMemcached</ulink>
1565 will be configured for the connection.
1566 This option is inspected by ZOOM when a connection is established.
1567 If the <literal>memcached</literal> option is given
1568 and YAZ is compiled without libMemcached support, an internal
1569 diagnostic (10018) will be thrown.
1570 libMemcached support is available for YAZ 5.0.13 or later. If this
1571 option is supplied for an earlier version of YAZ, it is
1572 <emphasis>ignored</emphasis>.
1573 The value of this option is a list options - each is of the
1574 form <literal>--name=value</literal>.
1575 Option <literal>--server=</literal>host[:port] specifies a memcached
1576 server. It may be repeated for multiple memcached servers.
1577 Option <literal>--expire=</literal>seconds sets expiry time in seconds
1578 for how long result sets are to be cached.
1579 </entry><entry>none</entry></row>
1581 redis</entry><entry>
1582 If given and non-empty,
1583 a <ulink url="&url.redis;">redis</ulink> context will be created
1585 This option is inspected by ZOOM when a connection is established.
1586 If the <literal>redis</literal> option is given
1587 and YAZ is compiled without redis support, an internal
1588 diagnostic (10018) will be thrown.
1589 redis support is available for YAZ 5.2.0 or later. If this
1590 option is supplied for an earlier version of YAZ, it is
1591 <emphasis>ignored</emphasis>.
1592 The value of this option is a set options, similar to that
1593 of the memcached setting. At this stage only --server=host[:port]
1594 and --expire=seconds is supported.
1595 </entry><entry>none</entry></row>
1600 If either option <literal>lang</literal> or <literal>charset</literal>
1602 <ulink url="&url.z39.50.charneg;">
1603 Character Set and Language Negotiation</ulink> is in effect.
1606 int ZOOM_connection_error(ZOOM_connection c, const char **cp,
1607 const char **addinfo);
1608 int ZOOM_connection_error_x(ZOOM_connection c, const char **cp,
1609 const char **addinfo, const char **dset);
1612 Function <function>ZOOM_connection_error</function> checks for
1613 errors for the last operation(s) performed. The function returns
1614 zero if no errors occurred; non-zero otherwise indicating the error.
1615 Pointers <parameter>cp</parameter> and <parameter>addinfo</parameter>
1616 holds messages for the error and additional-info if passed as
1617 non-<literal>NULL</literal>. Function
1618 <function>ZOOM_connection_error_x</function> is an extended version
1619 of <function>ZOOM_connection_error</function> that is capable of
1620 returning name of diagnostic set in <parameter>dset</parameter>.
1622 <sect2 id="zoom-connection-z39.50">
1623 <title>Z39.50 Protocol behavior</title>
1625 The calls <function>ZOOM_connection_new</function> and
1626 <function>ZOOM_connection_connect</function> establishes a TCP/IP
1627 connection and sends an Initialize Request to the target if
1628 possible. In addition, the calls waits for an Initialize Response
1629 from the target and the result is inspected (OK or rejected).
1632 If <literal>proxy</literal> is set then the client will establish
1633 a TCP/IP connection with the peer as specified by the
1634 <literal>proxy</literal> host and the hostname as part of the
1635 connect calls will be set as part of the Initialize Request.
1636 The proxy server will then "forward" the PDU's transparently
1637 to the target behind the proxy.
1640 For the authentication parameters, if option <literal>user</literal>
1641 is set and both options <literal>group</literal> and
1642 <literal>pass</literal> are unset, then Open style
1643 authentication is used (Version 2/3) in which case the username
1644 is usually followed by a slash, then by a password.
1645 If either <literal>group</literal>
1646 or <literal>pass</literal> is set then idPass authentication
1647 (Version 3 only) is used. If none of the options are set, no
1648 authentication parameters are set as part of the Initialize Request
1652 When option <literal>async</literal> is 1, it really means that
1653 all network operations are postponed (and queued) until the
1654 function <literal>ZOOM_event</literal> is invoked. When doing so
1655 it doesn't make sense to check for errors after
1656 <literal>ZOOM_connection_new</literal> is called since that
1657 operation "connecting - and init" is still incomplete and the
1658 API cannot tell the outcome (yet).
1661 <sect2 id="zoom.sru.init.behavior">
1662 <title>SRU/Solr Protocol behavior</title>
1664 The HTTP based protocols (SRU, SRW, Solr) doesn't feature an
1665 Inititialize Request, so the connection phase merely establishes a
1666 TCP/IP connection with the HTTP server.
1668 <para>Most of the ZOOM connection options do not
1669 affect SRU/Solr and they are ignored. However, future versions
1670 of &yaz; might honor <literal>implementationName</literal> and
1671 put that as part of User-Agent header for HTTP requests.
1674 The <literal>charset</literal> is used in the Content-Type header
1678 Setting <literal>authentcationMode</literal> specifies how
1679 authentication parameters are encoded for HTTP. The default is
1680 "<literal>basic</literal>" where <literal>user</literal> and
1681 <literal>password</literal> are encoded by using HTTP basic
1685 If <literal>authentcationMode</literal> is "<literal>url</literal>", then
1686 user and password are encoded in the URL by parameters
1687 <literal>x-username</literal> and <literal>x-password</literal> as
1688 given by the SRU standard.
1692 <sect1 id="zoom.query">
1693 <title>Queries</title>
1695 Query objects represents queries.
1698 ZOOM_query ZOOM_query_create(void);
1700 void ZOOM_query_destroy(ZOOM_query q);
1702 int ZOOM_query_prefix(ZOOM_query q, const char *str);
1704 int ZOOM_query_cql(ZOOM_query s, const char *str);
1706 int ZOOM_query_sortby(ZOOM_query q, const char *criteria);
1708 int ZOOM_query_sortby2(ZOOM_query q, const char *strategy,
1709 const char *criteria);
1712 Create query objects using <function>ZOOM_query_create</function>
1713 and destroy them by calling <function>ZOOM_query_destroy</function>.
1714 RPN-queries can be specified in <link linkend="PQF">PQF</link>
1715 notation by using the
1716 function <function>ZOOM_query_prefix</function>.
1717 The <function>ZOOM_query_cql</function> specifies a CQL
1718 query to be sent to the server/target.
1719 More query types will be added in future versions of &yaz;, such as
1720 <link linkend="CCL">CCL</link> to RPN-mapping, native CCL query,
1721 etc. In addition to a search, a sort criteria may be set. Function
1722 <function>ZOOM_query_sortby</function> enables Z39.50 sorting and
1723 it takes sort criteria using the same string notation as
1724 yaz-client's <link linkend="sortspec">sort command</link>.
1726 <para id="zoom.query.sortby2">
1727 <function>ZOOM_query_sortby2</function> is similar to
1728 <function>ZOOM_query_sortby</function> but allows a strategy for
1729 sorting. The reason for the strategy parameter is that some
1730 protocols offers multiple ways of performing sorting.
1731 For example, Z39.50 has the standard sort, which is performed after
1732 search on an existing result set.
1733 It's also possible to use CQL in Z39.50 as the query type and use
1734 CQL's SORTBY keyword. Finally, Index Data's
1735 Zebra server also allows sorting to be specified as part of RPN (Type 7).
1737 <table id="zoom-sort-strategy" frame="top">
1738 <title>ZOOM sort strategy</title>
1740 <colspec colwidth="2*" colname="name"/>
1741 <colspec colwidth="5*" colname="description"/>
1745 <entry>Description</entry>
1750 <entry>z39.50</entry><entry>Z39.50 resultset sort</entry>
1753 <entry>type7</entry><entry>Sorting embedded in RPN(Type-7)</entry>
1756 <entry>cql</entry><entry>CQL SORTBY</entry>
1759 <entry>sru11</entry><entry>SRU sortKeys parameter</entry>
1762 <entry>solr</entry><entry>Solr sort</entry>
1765 <entry>embed</entry><entry>type7 for Z39.50, cql for SRU,
1766 solr for Solr protocol</entry>
1772 <sect1 id="zoom.resultsets"><title>Result sets</title>
1774 The result set object is a container for records returned from
1778 ZOOM_resultset ZOOM_connection_search(ZOOM_connection, ZOOM_query q);
1780 ZOOM_resultset ZOOM_connection_search_pqf(ZOOM_connection c,
1782 void ZOOM_resultset_destroy(ZOOM_resultset r);
1785 Function <function>ZOOM_connection_search</function> creates
1786 a result set given a connection and query.
1787 Destroy a result set by calling
1788 <function>ZOOM_resultset_destroy</function>.
1789 Simple clients may using PQF only may use function
1790 <function>ZOOM_connection_search_pqf</function> in which case
1791 creating query objects is not necessary.
1794 void ZOOM_resultset_option_set(ZOOM_resultset r,
1795 const char *key, const char *val);
1797 const char *ZOOM_resultset_option_get(ZOOM_resultset r, const char *key);
1799 size_t ZOOM_resultset_size(ZOOM_resultset r);
1802 Functions <function>ZOOM_resultset_options_set</function> and
1803 <function>ZOOM_resultset_get</function> sets and gets an option
1804 for a result set similar to <function>ZOOM_connection_option_get</function>
1805 and <function>ZOOM_connection_option_set</function>.
1808 The number of hits also called result-count is returned by
1809 function <function>ZOOM_resultset_size</function>.
1811 <table id="zoom.resultset.options"
1812 frame="top"><title>ZOOM Result set Options</title>
1814 <colspec colwidth="4*" colname="name"></colspec>
1815 <colspec colwidth="7*" colname="description"></colspec>
1816 <colspec colwidth="2*" colname="default"></colspec>
1819 <entry>Option</entry>
1820 <entry>Description</entry>
1821 <entry>Default</entry>
1826 start</entry><entry>Offset of first record to be
1827 retrieved from target. First record has offset 0 unlike the
1828 protocol specifications where first record has position 1.
1829 This option affects ZOOM_resultset_search and
1830 ZOOM_resultset_search_pqf and must be set before any of
1831 these functions are invoked. If a range of
1832 records must be fetched manually after search,
1833 function ZOOM_resultset_records should be used.
1834 </entry><entry>0</entry></row>
1836 count</entry><entry>Number of records to be retrieved.
1837 This option affects ZOOM_resultset_search and
1838 ZOOM_resultset_search_pqf and must be set before any of
1839 these functions are invoked.
1840 </entry><entry>0</entry></row>
1842 presentChunk</entry><entry>The number of records to be
1843 requested from the server in each chunk (present request). The
1844 value 0 means to request all the records in a single chunk.
1845 (The old <literal>step</literal>
1846 option is also supported for the benefit of old applications.)
1847 </entry><entry>0</entry></row>
1849 elementSetName</entry><entry>Element-Set name of records.
1850 Most targets should honor element set name <literal>B</literal>
1851 and <literal>F</literal> for brief and full respectively.
1852 </entry><entry>none</entry></row>
1854 preferredRecordSyntax</entry><entry>Preferred Syntax, such as
1855 <literal>USMARC</literal>, <literal>SUTRS</literal>, etc.
1856 </entry><entry>none</entry></row>
1858 schema</entry><entry>Schema for retrieval, such as
1859 <literal>Gils-schema</literal>, <literal>Geo-schema</literal>, etc.
1860 </entry><entry>none</entry></row>
1862 setname</entry><entry>Name of Result Set (Result Set ID).
1863 If this option isn't set, the ZOOM module will automatically
1864 allocate a result set name.
1865 </entry><entry>default</entry></row>
1867 rpnCharset</entry><entry>Character set for RPN terms.
1868 If this is set, ZOOM C will assume that the ZOOM application is
1869 running UTF-8. Terms in RPN queries are then converted to the
1870 rpnCharset. If this is unset, ZOOM C will not assume any encoding
1871 of RPN terms and no conversion is performed.
1872 </entry><entry>none</entry></row>
1877 For servers that support Search Info report, the following
1878 options may be read using <function>ZOOM_resultset_get</function>.
1879 This detailed information is read after a successful search has
1883 This information is a list of of items, where each item is
1884 information about a term or subquery. All items in the list
1886 <literal>SearchResult.</literal><replaceable>no</replaceable>
1887 where no presents the item number (0=first, 1=second).
1888 Read <literal>searchresult.size</literal> to determine the
1891 <table id="zoom.search.info.report.options"
1892 frame="top"><title>Search Info Report Options</title>
1894 <colspec colwidth="4*" colname="name"></colspec>
1895 <colspec colwidth="7*" colname="description"></colspec>
1898 <entry>Option</entry>
1899 <entry>Description</entry>
1904 <entry>searchresult.size</entry>
1906 number of search result entries. This option is-nonexistant
1907 if no entries are returned by the server.
1911 <entry>searchresult.<replaceable>no</replaceable>.id</entry>
1912 <entry>sub query ID</entry>
1915 <entry>searchresult.<replaceable>no</replaceable>.count</entry>
1916 <entry>result count for item (number of hits)</entry>
1919 <entry>searchresult.<replaceable>no</replaceable>.subquery.term</entry>
1920 <entry>subquery term</entry>
1924 searchresult.<replaceable>no</replaceable>.interpretation.term
1926 <entry>interpretation term</entry>
1930 searchresult.<replaceable>no</replaceable>.recommendation.term
1932 <entry>recommendation term</entry>
1937 <sect2 id="zoom.z3950.resultset.sort">
1938 <title>Z39.50 Result-set Sort</title>
1940 void ZOOM_resultset_sort(ZOOM_resultset r,
1941 const char *sort_type, const char *sort_spec);
1943 int ZOOM_resultset_sort1(ZOOM_resultset r,
1944 const char *sort_type, const char *sort_spec);
1947 <function>ZOOM_resultset_sort</function> and
1948 <function>ZOOM_resultset_sort1</function> both sort an existing
1949 result-set. The sort_type parameter is not use. Set it to "yaz".
1950 The sort_spec is same notation as ZOOM_query_sortby and identical
1951 to that offered by yaz-client's
1952 <link linkend="sortspec">sort command</link>.
1955 These functions only work for Z39.50. Use the more generic utility
1956 <link linkend="zoom.query.sortby2">
1957 <function>ZOOM_query_sortby2</function></link>
1958 for other protocols (and even Z39.50).
1961 <sect2 id="zoom.z3950.resultset.behavior">
1962 <title>Z39.50 Protocol behavior</title>
1964 The creation of a result set involves at least a SearchRequest
1965 - SearchResponse protocol handshake. Following that, if a sort
1966 criteria was specified as part of the query, a SortRequest -
1967 SortResponse handshake takes place. Note that it is necessary to
1968 perform sorting before any retrieval takes place, so no records will
1969 be returned from the target as part of the SearchResponse because these
1970 would be unsorted. Hence, piggyback is disabled when sort criteria
1971 are set. Following Search - and a possible sort - Retrieval takes
1972 place - as one or more Present Requests/Response pairs being
1976 The API allows for two different modes for retrieval. A high level
1977 mode which is somewhat more powerful and a low level one.
1978 The low level is enabled when searching on a Connection object
1979 for which the settings
1980 <literal>smallSetUpperBound</literal>,
1981 <literal>mediumSetPresentNumber</literal> and
1982 <literal>largeSetLowerBound</literal> are set. The low level mode
1983 thus allows you to precisely set how records are returned as part
1984 of a search response as offered by the Z39.50 protocol.
1985 Since the client may be retrieving records as part of the
1986 search response, this mode doesn't work well if sorting is used.
1989 The high-level mode allows you to fetch a range of records from
1990 the result set with a given start offset. When you use this mode
1991 the client will automatically use piggyback if that is possible
1992 with the target and perform one or more present requests as needed.
1993 Even if the target returns fewer records as part of a present response
1994 because of a record size limit, etc. the client will repeat sending
1995 present requests. As an example, if option <literal>start</literal>
1996 is 0 (default) and <literal>count</literal> is 4, and
1997 <literal>piggyback</literal> is 1 (default) and no sorting criteria
1998 is specified, then the client will attempt to retrieve the 4
1999 records as part the search response (using piggyback). On the other
2000 hand, if either <literal>start</literal> is positive or if
2001 a sorting criteria is set, or if <literal>piggyback</literal>
2002 is 0, then the client will not perform piggyback but send Present
2006 If either of the options <literal>mediumSetElementSetName</literal> and
2007 <literal>smallSetElementSetName</literal> are unset, the value
2008 of option <literal>elementSetName</literal> is used for piggyback
2009 searches. This means that for the high-level mode you only have
2010 to specify one elementSetName option rather than three.
2013 <sect2 id="zoom.sru.resultset.behavior">
2014 <title>SRU Protocol behavior</title>
2016 Current version of &yaz; does not take advantage of a result set id
2017 returned by the SRU server. Future versions might do, however.
2018 Since, the ZOOM driver does not save result set IDs any
2019 present (retrieval) is transformed to a SRU SearchRetrieveRequest
2020 with same query but, possibly, different offsets.
2023 Option <literal>schema</literal> specifies SRU schema
2024 for retrieval. However, options <literal>elementSetName</literal> and
2025 <literal>preferredRecordSyntax</literal> are ignored.
2028 Options <literal>start</literal> and <literal>count</literal>
2029 are supported by SRU.
2030 The remaining options
2031 <literal>piggyback</literal>,
2032 <literal>smallSetUpperBound</literal>,
2033 <literal>largeSetLowerBound</literal>,
2034 <literal>mediumSetPresentNumber</literal>,
2035 <literal>mediumSetElementSetName</literal>,
2036 <literal>smallSetElementSetName</literal> are
2040 SRU supports CQL queries, <emphasis>not</emphasis> PQF.
2041 If PQF is used, however, the PQF query is transferred anyway
2042 using non-standard element <literal>pQuery</literal> in
2043 SRU SearchRetrieveRequest.
2046 Solr queries has to be done in Solr query format.
2049 Unfortunately, SRU or Solr does not define a database setting. Hence,
2050 <literal>databaseName</literal> is unsupported and ignored.
2051 However, the path part in host parameter for functions
2052 <function>ZOOM_connecton_new</function> and
2053 <function>ZOOM_connection_connect</function> acts as a
2054 database (at least for the &yaz; SRU server).
2058 <sect1 id="zoom.records">
2059 <title>Records</title>
2061 A record object is a retrieval record on the client side -
2062 created from result sets.
2065 void ZOOM_resultset_records(ZOOM_resultset r,
2067 size_t start, size_t count);
2068 ZOOM_record ZOOM_resultset_record(ZOOM_resultset s, size_t pos);
2070 const char *ZOOM_record_get(ZOOM_record rec, const char *type,
2073 int ZOOM_record_error(ZOOM_record rec, const char **msg,
2074 const char **addinfo, const char **diagset);
2076 ZOOM_record ZOOM_record_clone(ZOOM_record rec);
2078 void ZOOM_record_destroy(ZOOM_record rec);
2081 References to temporary records are returned by functions
2082 <function>ZOOM_resultset_records</function> or
2083 <function>ZOOM_resultset_record</function>.
2086 If a persistent reference to a record is desired
2087 <function>ZOOM_record_clone</function> should be used.
2088 It returns a record reference that should be destroyed
2089 by a call to <function>ZOOM_record_destroy</function>.
2092 A single record is returned by function
2093 <function>ZOOM_resultset_record</function> that takes a
2094 position as argument. First record has position zero.
2095 If no record could be obtained <literal>NULL</literal> is returned.
2098 Error information for a record can be checked with
2099 <function>ZOOM_record_error</function> which returns non-zero
2100 (error code) if record is in error, called <emphasis>Surrogate
2101 Diagnostics</emphasis> in Z39.50.
2104 Function <function>ZOOM_resultset_records</function> retrieves
2105 a number of records from a result set. Parameter <literal>start</literal>
2106 and <literal>count</literal> specifies the range of records to
2107 be returned. Upon completion array
2108 <literal>recs[0], ..recs[count-1]</literal>
2109 holds record objects for the records. The array of records
2110 <literal>recs</literal> should be allocated prior the call
2111 <function>ZOOM_resultset_records</function>. Note that for those
2112 records that couldn't be retrieved from the target
2113 <literal>recs[ ..]</literal> is set to <literal>NULL</literal>.
2115 <para id="zoom.record.get">
2116 In order to extract information about a single record,
2117 <function>ZOOM_record_get</function> is provided. The
2118 function returns a pointer to certain record information. The
2119 nature (type) of the pointer depends on the parameter,
2120 <parameter>type</parameter>.
2123 The <parameter>type</parameter> is a string of the format:
2126 <replaceable>format</replaceable>[;charset=<replaceable>from</replaceable>[/<replaceable>opacfrom</replaceable>][,<replaceable>to</replaceable>]][;format=<replaceable>v</replaceable>]
2129 where <replaceable>format</replaceable> specifies the format of the
2130 returned record, <replaceable>from</replaceable>
2131 specifies the character set of the record in its original form
2132 (as returned by the server), <replaceable>to</replaceable> specifies
2133 the output (returned)
2134 character set encoding.
2135 If <replaceable>to</replaceable> is omitted UTF-8 is assumed.
2136 If charset is not given, then no character set conversion takes place.
2139 <para>OPAC records may be returned in a different
2140 set from the bibliographic MARC record. If this is this the case,
2141 <replaceable>opacfrom</replaceable> should be set to the character set
2142 of the OPAC record part.
2146 Specifying the OPAC record character set requires YAZ 4.1.5 or later.
2150 The format argument controls whether record data should be XML
2151 pretty-printed (post process operation).
2152 It is enabled only if format value <replaceable>v</replaceable> is
2153 <literal>1</literal> and the record content is XML well-formed.
2156 In addition, for certain types, the length
2157 <literal>len</literal> passed will be set to the size in bytes of
2158 the returned information.
2161 The following are the supported values for <replaceable>form</replaceable>.
2163 <varlistentry><term><literal>database</literal></term>
2164 <listitem><para>Database of record is returned
2165 as a C null-terminated string. Return type
2166 <literal>const char *</literal>.
2169 <varlistentry><term><literal>syntax</literal></term>
2170 <listitem><para>The transfer syntax of the record is returned
2171 as a C null-terminated string containing the symbolic name of
2172 the record syntax, e.g. <literal>Usmarc</literal>. Return type
2174 <literal>const char *</literal>.
2177 <varlistentry><term><literal>schema</literal></term>
2178 <listitem><para>The schema of the record is returned
2179 as a C null-terminated string. Return type is
2180 <literal>const char *</literal>.
2183 <varlistentry><term><literal>render</literal></term>
2184 <listitem><para>The record is returned in a display friendly
2185 format. Upon completion buffer is returned
2186 (type <literal>const char *</literal>) and length is stored in
2187 <literal>*len</literal>.
2190 <varlistentry><term><literal>raw</literal></term>
2191 <listitem><para>The record is returned in the internal
2192 YAZ specific format. For GRS-1, Explain, and others, the
2193 raw data is returned as type
2194 <literal>Z_External *</literal> which is just the type for
2195 the member <literal>retrievalRecord</literal> in
2196 type <literal>NamePlusRecord</literal>.
2197 For SUTRS and octet aligned record (including all MARCs) the
2198 octet buffer is returned and the length of the buffer.
2201 <varlistentry><term><literal>xml</literal></term>
2202 <listitem><para>The record is returned in XML if possible.
2203 SRU, Solr and Z39.50 records with transfer syntax XML are
2204 returned verbatim. MARC records are returned in
2205 <ulink url="&url.marcxml;">
2208 (converted from ISO2709 to MARCXML by YAZ).
2209 OPAC records are also converted to XML and the
2210 bibliographic record is converted to MARCXML (when possible).
2211 GRS-1 records are not supported for this form.
2212 Upon completion, the XML buffer is returned
2213 (type <literal>const char *</literal>) and length is stored in
2214 <literal>*len</literal>.
2217 <varlistentry><term><literal>opac</literal></term>
2218 <listitem><para>OPAC information for record is returned in XML
2219 if an OPAC record is present at the position given. If no
2220 OPAC record is present, a NULL pointer is returned.
2223 <varlistentry><term><literal>txml</literal></term>
2224 <listitem><para>The record is returned in TurboMARC if possible.
2225 SRU and Z39.50 records with transfer syntax XML are
2226 returned verbatim. MARC records are returned in
2227 <link linkend="tools.turbomarc">
2230 (converted from ISO2709 to TurboMARC by YAZ).
2231 Upon completion, the XML buffer is returned
2232 (type <literal>const char *</literal>) and length is stored in
2233 <literal>*len</literal>.
2236 <varlistentry><term><literal>json</literal></term>
2237 <listitem><para>Like xml, but MARC records are converted to
2238 <ulink url="&url.marc_in_json;">MARC-in-JSON</ulink>.
2246 <ulink url="&url.marc21;">MARC21</ulink>
2248 <ulink url="&url.marc8;">MARC-8</ulink>
2249 character set encoding.
2250 An application that wishes to display in Latin-1 would use
2252 render; charset=marc8,iso-8859-1
2255 <sect2 id="zoom.z3950.record.behavior">
2256 <title>Z39.50 Protocol behavior</title>
2258 The functions <function>ZOOM_resultset_record</function> and
2259 <function>ZOOM_resultset_records</function> inspects the client-side
2260 record cache. Records not found in cache are fetched using
2262 The functions may block (and perform network I/O) - even though option
2263 <literal>async</literal> is 1, because they return records objects.
2264 (and there's no way to return records objects without retrieving them!).
2267 There is a trick, however, in the usage of function
2268 <function>ZOOM_resultset_records</function> that allows for
2269 delayed retrieval (and makes it non-blocking). By using
2270 a null pointer for <parameter>recs</parameter> you're indicating
2271 you're not interested in getting records objects
2272 <emphasis>now</emphasis>.
2275 <sect2 id="zoom.sru.record.behavior">
2276 <title>SRU/Solr Protocol behavior</title>
2278 The ZOOM driver for SRU/Solr treats records returned by a SRU/Solr server
2279 as if they where Z39.50 records with transfer syntax XML and
2280 no element set name or database name.
2284 <sect1 id="zoom.facets"><title>Facets</title>
2286 Facet operations is not part of the official ZOOM specification, but
2287 is an Index Data extension for YAZ-based Z39.50 targets,
2288 <ulink url="&url.solr;">Solr</ulink> and SRU 2.0 targets.
2290 Facets may be requestd by the
2291 <link linkend="zoom.facets.option">facets</link> option before a
2293 For inspection of the returned facets, the following functions are
2297 ZOOM_facet_field *ZOOM_resultset_facets(ZOOM_resultset r);
2299 ZOOM_facet_field ZOOM_resultset_get_facet_field(ZOOM_resultset r,
2300 const char *facet_name);
2302 ZOOM_facet_field ZOOM_resultset_get_facet_field_by_index(ZOOM_resultset r,
2305 size_t ZOOM_resultset_facets_size(ZOOM_resultset r);
2307 const char *ZOOM_facet_field_name(ZOOM_facet_field facet_field);
2309 size_t ZOOM_facet_field_term_count(ZOOM_facet_field facet_field);
2311 const char *ZOOM_facet_field_get_term(ZOOM_facet_field facet_field,
2312 size_t idx, int *freq);
2315 References to temporary structures are returned by all functions.
2316 They are only valid as long the Result set is valid.
2317 <function>ZOOM_resultset_get_facet_field</function> or
2318 <function>ZOOM_resultset_get_facet_field_by_index</function>.
2319 <function>ZOOM_resultset_facets</function>.
2320 <function>ZOOM_facet_field_name</function>.
2321 <function>ZOOM_facet_field_get_term</function>.
2323 <para id="zoom.resultset.get_facet_field">
2324 A single Facet field is returned by function
2325 <function>ZOOM_resultset_get_facet_field</function> or
2326 <function>ZOOM_resultset_get_facet_field_by_index</function> that takes
2327 a result set and facet name or positive index respectively. First
2328 facet has position zero. If no facet could be obtained (invalid name
2329 or index out of bounds) <literal>NULL</literal> is returned.
2331 <para id="zoom.resultset.facets">
2332 An array of facets field can be returned by
2333 <function>ZOOM_resultset_facets</function>. The length of the array is
2334 given by <function>ZOOM_resultset_facets_size</function>. The array is
2335 zero-based and last entry will be at
2336 <function>ZOOM_resultset_facets_size(result_set)</function>-1.
2338 <para id="zoom.resultset.facets_names">
2339 It is possible to interate over facets by name, by calling
2340 <function>ZOOM_resultset_facets_names</function>.
2341 This will return an const array of char * where each string can be used
2342 as parameter for <function>ZOOM_resultset_get_facet_field</function>.
2345 Function <function>ZOOM_facet_field_name</function> gets the request
2346 facet name from a returned facet field.
2349 Function <function>ZOOM_facet_field_get_term</function> returns the
2350 idx'th term and term count for a facet field.
2351 Idx must between 0 and
2352 <function>ZOOM_facet_field_term_count</function>-1, otherwise the
2353 returned reference will be <literal>NULL</literal>. On a valid idx, the
2354 value of the freq reference will be the term count.
2355 The <literal>freq</literal> parameter must be valid pointer to integer.
2358 <sect1 id="zoom.scan"><title>Scan</title>
2360 This section describes an interface for Scan. Scan is not an
2361 official part of the ZOOM model yet. The result of a scan operation
2362 is the <literal>ZOOM_scanset</literal> which is a set of terms
2363 returned by a target.
2367 The Scan interface is supported for both Z39.50, SRU and Solr.
2371 ZOOM_scanset ZOOM_connection_scan(ZOOM_connection c,
2372 const char *startpqf);
2374 ZOOM_scanset ZOOM_connection_scan1(ZOOM_connection c,
2377 size_t ZOOM_scanset_size(ZOOM_scanset scan);
2379 const char *ZOOM_scanset_term(ZOOM_scanset scan, size_t pos,
2380 size_t *occ, size_t *len);
2382 const char *ZOOM_scanset_display_term(ZOOM_scanset scan, size_t pos,
2383 size_t *occ, size_t *len);
2385 void ZOOM_scanset_destroy(ZOOM_scanset scan);
2387 const char *ZOOM_scanset_option_get(ZOOM_scanset scan,
2390 void ZOOM_scanset_option_set(ZOOM_scanset scan, const char *key,
2394 The scan set is created by function
2395 <function>ZOOM_connection_scan</function> which performs a scan
2396 operation on the connection using the specified
2397 <parameter>startpqf</parameter>.
2398 If the operation was successful, the size of the scan set can be
2399 retrieved by a call to <function>ZOOM_scanset_size</function>.
2400 Like result sets, the items are numbered 0,..size-1.
2401 To obtain information about a particular scan term, call function
2402 <function>ZOOM_scanset_term</function>. This function takes
2403 a scan set offset <literal>pos</literal> and returns a pointer
2404 to a <emphasis>raw term</emphasis> or <literal>NULL</literal> if
2406 If present, the <literal>occ</literal> and <literal>len</literal>
2407 are set to the number of occurrences and the length
2408 of the actual term respectively.
2409 <function>ZOOM_scanset_display_term</function> is similar to
2410 <function>ZOOM_scanset_term</function> except that it returns
2411 the <emphasis>display term</emphasis> rather than the raw term.
2412 In a few cases, the term is different from display term. Always
2413 use the display term for display and the raw term for subsequent
2414 scan operations (to get more terms, next scan result, etc).
2417 A scan set may be freed by a call to function
2418 <function>ZOOM_scanset_destroy</function>.
2419 Functions <function>ZOOM_scanset_option_get</function> and
2420 <function>ZOOM_scanset_option_set</function> retrieves and sets
2421 an option respectively.
2424 The <parameter>startpqf</parameter> is a subset of PQF, namely
2425 the Attributes+Term part. Multiple <literal>@attr</literal> can
2426 be used. For example to scan in title (complete) phrases:
2428 @attr 1=4 @attr 6=2 "science o"
2432 The <function>ZOOM_connecton_scan1</function> is a newer and
2433 more generic alternative to <function>ZOOM_connection_scan</function>
2434 which allows to use both CQL and PQF for Scan.
2436 <table frame="top" id="zoom.scanset.options">
2437 <title>ZOOM Scan Set Options</title>
2439 <colspec colwidth="4*" colname="name"></colspec>
2440 <colspec colwidth="7*" colname="description"></colspec>
2441 <colspec colwidth="2*" colname="default"></colspec>
2444 <entry>Option</entry>
2445 <entry>Description</entry>
2446 <entry>Default</entry>
2451 number</entry><entry>Number of Scan Terms requested in next scan.
2452 After scan it holds the actual number of terms returned.
2453 </entry><entry>20</entry></row>
2455 position</entry><entry>Preferred Position of term in response
2456 in next scan; actual position after completion of scan.
2457 </entry><entry>1</entry></row>
2459 stepSize</entry><entry>Step Size
2460 </entry><entry>0</entry></row>
2462 scanStatus</entry><entry>An integer indicating the Scan Status
2464 </entry><entry>0</entry></row>
2466 rpnCharset</entry><entry>Character set for RPN terms.
2467 If this is set, ZOOM C will assume that the ZOOM application is
2468 running UTF-8. Terms in RPN queries are then converted to the
2469 rpnCharset. If this is unset, ZOOM C will not assume any encoding
2470 of RPN terms and no conversion is performed.
2471 </entry><entry>none</entry></row>
2476 <sect1 id="zoom.extendedservices">
2477 <title>Extended Services</title>
2479 ZOOM offers an interface to a subset of the Z39.50 extended services
2480 as well as a few privately defined ones:
2485 Z39.50 Item Order (ILL).
2486 See <xref linkend="zoom.item.order"/>.
2491 Record Update. This allows a client to insert, modify or delete
2493 See <xref linkend="zoom.record.update"/>.
2498 Database Create. This a non-standard feature. Allows a client
2499 to create a database.
2500 See <xref linkend="zoom.database.create"/>.
2505 Database Drop. This a non-standard feature. Allows a client
2506 to delete/drop a database.
2507 See <xref linkend="zoom.database.drop"/>.
2512 Commit operation. This a non-standard feature. Allows a client
2513 to commit operations.
2514 See <xref linkend="zoom.commit"/>.
2517 <!-- all the ILL PDU options should go here too -->
2520 To create an extended service operation a <literal>ZOOM_package</literal>
2521 must be created. The operation is a five step operation. The
2522 package is created, package is configured by means of options,
2523 the package is send, result is inspected (by means of options),
2524 the package is destroyed.
2527 ZOOM_package ZOOM_connection_package(ZOOM_connection c,
2528 ZOOM_options options);
2530 const char *ZOOM_package_option_get(ZOOM_package p,
2532 void ZOOM_package_option_set(ZOOM_package p, const char *key,
2534 void ZOOM_package_send(ZOOM_package p, const char *type);
2536 void ZOOM_package_destroy(ZOOM_package p);
2539 The <function>ZOOM_connection_package</function> creates a
2540 package for the connection given using the options specified.
2543 Functions <function>ZOOM_package_option_get</function> and
2544 <function>ZOOM_package_option_set</function> gets and sets
2548 <function>ZOOM_package_send</function> sends
2549 the package the via connection specified in
2550 <function>ZOOM_connection_package</function>.
2551 The <parameter>type</parameter> specifies the actual extended service
2552 package type to be sent.
2554 <table frame="top" id="zoom.extendedservices.options">
2555 <title>Extended Service Common Options</title>
2557 <colspec colwidth="4*" colname="name"></colspec>
2558 <colspec colwidth="7*" colname="description"></colspec>
2559 <colspec colwidth="3*" colname="default"></colspec>
2562 <entry>Option</entry>
2563 <entry>Description</entry>
2564 <entry>Default</entry>
2569 <entry>package-name</entry>
2570 <entry>Extended Service Request package name. Must be specified
2571 as part of a request</entry>
2575 <entry>user-id</entry>
2576 <entry>User ID of Extended Service Package. Is a request option</entry>
2580 <entry>function</entry>
2582 Function of package - one of <literal>create</literal>,
2583 <literal>delete</literal>, <literal>modify</literal>. Is
2586 <entry><literal>create</literal></entry>
2589 <entry>waitAction</entry>
2591 Wait action for package. Possible values:
2592 <literal>wait</literal>, <literal>waitIfPossible</literal>,
2593 <literal>dontWait</literal> or <literal>dontReturnPackage</literal>.
2595 <entry><literal>waitIfPossible</literal></entry>
2598 <entry>targetReference</entry>
2600 Target Reference. This is part of the response as returned
2601 by the server. Read it after a successful operation.
2603 <entry><literal>none</literal></entry>
2608 <sect2 id="zoom.item.order">
2609 <title>Item Order</title>
2611 For Item Order, type must be set to <literal>itemorder</literal> in
2612 <function>ZOOM_package_send</function>.
2615 <table frame="top" id="zoom.item.order.options">
2616 <title>Item Order Options</title>
2618 <colspec colwidth="4*" colname="name"></colspec>
2619 <colspec colwidth="7*" colname="description"></colspec>
2620 <colspec colwidth="3*" colname="default"></colspec>
2623 <entry>Option</entry>
2624 <entry>Description</entry>
2625 <entry>Default</entry>
2630 <entry>contact-name</entry>
2631 <entry>ILL contact name</entry>
2635 <entry>contact-phone</entry>
2636 <entry>ILL contact phone</entry>
2640 <entry>contact-email</entry>
2641 <entry>ILL contact email</entry>
2645 <entry>itemorder-item</entry>
2646 <entry>Position for item (record) requested. An integer</entry>
2653 <sect2 id="zoom.record.update">
2654 <title>Record Update</title>
2656 For Record Update, type must be set to <literal>update</literal> in
2657 <function>ZOOM_package_send</function>.
2659 <table frame="top" id="zoom.record.update.options">
2660 <title>Record Update Options</title>
2662 <colspec colwidth="4*" colname="name"></colspec>
2663 <colspec colwidth="7*" colname="description"></colspec>
2664 <colspec colwidth="3*" colname="default"></colspec>
2667 <entry>Option</entry>
2668 <entry>Description</entry>
2669 <entry>Default</entry>
2674 <entry>action</entry>
2676 The update action. One of
2677 <literal>specialUpdate</literal>,
2678 <literal>recordInsert</literal>,
2679 <literal>recordReplace</literal>,
2680 <literal>recordDelete</literal>,
2681 <literal>elementUpdate</literal>.
2683 <entry><literal>specialUpdate (recordInsert for updateVersion=1 which does not support specialUpdate)</literal></entry>
2686 <entry>recordIdOpaque</entry>
2687 <entry>Opaque Record ID</entry>
2691 <entry>recordIdNumber</entry>
2692 <entry>Record ID number</entry>
2696 <entry>record</entry>
2697 <entry>The record itself</entry>
2701 <entry>recordOpaque</entry>
2702 <entry>Specifies an opaque record which is
2703 encoded as an ASN.1 ANY type with the OID as tiven by option
2704 <literal>syntax</literal> (see below).
2705 Option <literal>recordOpaque</literal> is an alternative
2706 to record - and <literal>record</literal> option (above) is
2707 ignored if recordOpaque is set. This option is only available in
2708 YAZ 3.0.35 and later and is meant to facilitate Updates with
2714 <entry>syntax</entry>
2715 <entry>The record syntax (transfer syntax). Is a string that
2716 is a known record syntax.
2718 <entry>no syntax</entry>
2721 <entry>databaseName</entry>
2722 <entry>Database from connection object</entry>
2723 <entry>Default</entry>
2726 <entry>correlationInfo.note</entry>
2727 <entry>Correlation Info Note (string)</entry>
2731 <entry>correlationInfo.id</entry>
2732 <entry>Correlation Info ID (integer)</entry>
2736 <entry>elementSetName</entry>
2737 <entry>Element Set for Record</entry>
2741 <entry>updateVersion</entry>
2742 <entry>Record Update version which holds one of the values
2743 1, 2 or 3. Each version has a distinct OID:
2745 (<ulink url="&url.z39.50.extupdate1;">first version</ulink>) ,
2747 (second version) and
2748 1.2.840.10003.9.5.1.1
2749 (<ulink url="&url.z39.50.extupdate3;">third and
2750 newest version</ulink>).
2760 <sect2 id="zoom.database.create"><title>Database Create</title>
2762 For Database Create, type must be set to <literal>create</literal> in
2763 <function>ZOOM_package_send</function>.
2766 <table frame="top" id="zoom.database.create.options">
2767 <title>Database Create Options</title>
2769 <colspec colwidth="4*" colname="name"></colspec>
2770 <colspec colwidth="7*" colname="description"></colspec>
2771 <colspec colwidth="3*" colname="default"></colspec>
2774 <entry>Option</entry>
2775 <entry>Description</entry>
2776 <entry>Default</entry>
2781 <entry>databaseName</entry>
2782 <entry>Database from connection object</entry>
2783 <entry>Default</entry>
2789 <sect2 id="zoom.database.drop">
2790 <title>Database Drop</title>
2792 For Database Drop, type must be set to <literal>drop</literal> in
2793 <function>ZOOM_package_send</function>.
2795 <table frame="top" id="zoom.database.drop.options">
2796 <title>Database Drop Options</title>
2798 <colspec colwidth="4*" colname="name"></colspec>
2799 <colspec colwidth="7*" colname="description"></colspec>
2800 <colspec colwidth="3*" colname="default"></colspec>
2803 <entry>Option</entry>
2804 <entry>Description</entry>
2805 <entry>Default</entry>
2810 <entry>databaseName</entry>
2811 <entry>Database from connection object</entry>
2812 <entry>Default</entry>
2818 <sect2 id="zoom.commit">
2819 <title>Commit Operation</title>
2821 For Commit, type must be set to <literal>commit</literal> in
2822 <function>ZOOM_package_send</function>.
2825 <sect2 id="zoom.extended.services.behavior">
2826 <title>Protocol behavior</title>
2828 All the extended services are Z39.50-only.
2832 The database create, drop and commit services are privately defined
2834 Refer to <filename>esadmin.asn</filename> in YAZ for the ASN.1
2840 <sect1 id="zoom.options">
2841 <title>Options</title>
2843 Most &zoom; objects provide a way to specify options to change behavior.
2844 From an implementation point of view a set of options is just like
2845 an associative array / hash.
2848 ZOOM_options ZOOM_options_create(void);
2850 ZOOM_options ZOOM_options_create_with_parent(ZOOM_options parent);
2852 void ZOOM_options_destroy(ZOOM_options opt);
2855 const char *ZOOM_options_get(ZOOM_options opt, const char *name);
2857 void ZOOM_options_set(ZOOM_options opt, const char *name,
2861 typedef const char *(*ZOOM_options_callback)
2862 (void *handle, const char *name);
2864 ZOOM_options_callback
2865 ZOOM_options_set_callback(ZOOM_options opt,
2866 ZOOM_options_callback c,
2870 <sect1 id="zoom.queryconversions">
2871 <title>Query conversions</title>
2873 int ZOOM_query_cql2rpn(ZOOM_query s, const char *cql_str,
2874 ZOOM_connection conn);
2876 int ZOOM_query_ccl2rpn(ZOOM_query s, const char *ccl_str,
2878 int *ccl_error, const char **error_string,
2882 <function>ZOOM_query_cql2rpn</function> translates the CQL string,
2883 client-side, into RPN which may be passed to the server.
2884 This is useful for server's that don't themselves
2885 support CQL, for which <function>ZOOM_query_cql</function> is useless.
2886 `conn' is used only as a place to stash diagnostics if compilation
2887 fails; if this information is not needed, a null pointer may be used.
2888 The CQL conversion is driven by option <literal>cqlfile</literal> from
2889 connection conn. This specifies a conversion file (eg pqf.properties)
2890 which <emphasis>must</emphasis> be present.
2893 <function>ZOOM_query_ccl2rpn</function> translates the CCL string,
2894 client-side, into RPN which may be passed to the server.
2895 The conversion is driven by the specification given by
2896 <literal>config</literal>. Upon completion 0 is returned on success; -1
2897 is returned on on failure. Om failure <literal>error_string</literal> and
2898 <literal>error_pos</literal> holds error message and position of
2899 first error in original CCL string.
2902 <sect1 id="zoom.events"><title>Events</title>
2904 If you're developing non-blocking applications, you have to deal
2908 int ZOOM_event(int no, ZOOM_connection *cs);
2911 The <function>ZOOM_event</function> executes pending events for
2912 a number of connections. Supply the number of connections in
2913 <literal>no</literal> and an array of connections in
2914 <literal>cs</literal> (<literal>cs[0] ... cs[no-1]</literal>).
2915 A pending event could be a sending a search, receiving a response,
2917 When an event has occurred for one of the connections, this function
2918 returns a positive integer <literal>n</literal> denoting that an event
2919 occurred for connection <literal>cs[n-1]</literal>.
2920 When no events are pending for the connections, a value of zero is
2922 To ensure that all outstanding requests are performed call this function
2923 repeatedly until zero is returned.
2926 If <function>ZOOM_event</function> returns and returns non-zero, the
2927 last event that occurred can be expected.
2930 int ZOOM_connection_last_event(ZOOM_connection cs);
2933 <function>ZOOM_connection_last_event</function> returns an event type
2934 (integer) for the last event.
2937 <table frame="top" id="zoom.event.ids">
2938 <title>ZOOM Event IDs</title>
2940 <colspec colwidth="4*" colname="name"></colspec>
2941 <colspec colwidth="7*" colname="description"></colspec>
2944 <entry>Event</entry>
2945 <entry>Description</entry>
2950 <entry>ZOOM_EVENT_NONE</entry>
2951 <entry>No event has occurred</entry>
2954 <entry>ZOOM_EVENT_CONNECT</entry>
2955 <entry>TCP/IP connect has initiated</entry>
2958 <entry>ZOOM_EVENT_SEND_DATA</entry>
2959 <entry>Data has been transmitted (sending)</entry>
2962 <entry>ZOOM_EVENT_RECV_DATA</entry>
2963 <entry>Data has been received)</entry>
2966 <entry>ZOOM_EVENT_TIMEOUT</entry>
2967 <entry>Timeout</entry>
2970 <entry>ZOOM_EVENT_UNKNOWN</entry>
2971 <entry>Unknown event</entry>
2974 <entry>ZOOM_EVENT_SEND_APDU</entry>
2975 <entry>An APDU has been transmitted (sending)</entry>
2978 <entry>ZOOM_EVENT_RECV_APDU</entry>
2979 <entry>An APDU has been received</entry>
2982 <entry>ZOOM_EVENT_RECV_RECORD</entry>
2983 <entry>A result-set record has been received</entry>
2986 <entry>ZOOM_EVENT_RECV_SEARCH</entry>
2987 <entry>A search result been received</entry>
2994 <chapter id="server">
2995 <title>Generic server</title>
2996 <sect1 id="server.introduction"><title>Introduction</title>
2998 If you aren't into documentation, a good way to learn how the
2999 back end interface works is to look at the <filename>backend.h</filename>
3000 file. Then, look at the small dummy-server in
3001 <filename>ztest/ztest.c</filename>. The <filename>backend.h</filename>
3002 file also makes a good reference, once you've chewed your way through
3003 the prose of this file.
3006 If you have a database system that you would like to make available by
3007 means of Z39.50 or SRU, &yaz; basically offers your two options. You
3008 can use the APIs provided by the &asn;, &odr;, and &comstack;
3010 create and decode PDUs, and exchange them with a client.
3011 Using this low-level interface gives you access to all fields and
3012 options of the protocol, and you can construct your server as close
3013 to your existing database as you like.
3014 It is also a fairly involved process, requiring
3015 you to set up an event-handling mechanism, protocol state machine,
3016 etc. To simplify server implementation, we have implemented a compact
3017 and simple, but reasonably full-functioned server-frontend that will
3018 handle most of the protocol mechanics, while leaving you to
3019 concentrate on your database interface.
3023 The backend interface was designed in anticipation of a specific
3024 integration task, while still attempting to achieve some degree of
3025 generality. We realize fully that there are points where the
3026 interface can be improved significantly. If you have specific
3027 functions or parameters that you think could be useful, send us a
3028 mail (or better, sign on to the mailing list referred to in the
3029 top-level README file). We will try to fit good suggestions into future
3030 releases, to the extent that it can be done without requiring
3031 too many structural changes in existing applications.
3036 The &yaz; server does not support XCQL.
3040 <sect1 id="server.frontend">
3041 <title>The Database Frontend</title>
3043 We refer to this software as a generic database frontend. Your
3044 database system is the <emphasis>backend database</emphasis>, and the
3045 interface between the two is called the <emphasis>backend API</emphasis>.
3046 The backend API consists of a small number of function handlers and
3047 structure definitions. You are required to provide the
3048 <function>main()</function> routine for the server (which can be
3049 quite simple), as well as a set of handlers to match each of the
3051 The interface functions that you write can use any mechanism you like
3052 to communicate with your database system: You might link the whole
3053 thing together with your database application and access it by
3054 function calls; you might use IPC to talk to a database server
3055 somewhere; or you might link with third-party software that handles
3056 the communication for you (like a commercial database client library).
3057 At any rate, the handlers will perform the tasks of:
3070 Scanning the database index (optional - if you wish to implement SCAN).
3073 Extended Services (optional).
3076 Result-Set Delete (optional).
3079 Result-Set Sort (optional).
3082 Return Explain for SRU (optional).
3086 (more functions will be added in time to support as much of
3087 Z39.50-1995 as possible).
3090 <sect1 id="server.backend">
3091 <title>The Backend API</title>
3093 The header file that you need to use the interface are in the
3094 <filename>include/yaz</filename> directory. It's called
3095 <filename>backend.h</filename>. It will include other files from
3096 the <filename>include/yaz</filename> directory, so you'll
3097 probably want to use the -I option of your compiler to tell it
3098 where to find the files. When you run
3099 <literal>make</literal> in the top-level &yaz; directory,
3100 everything you need to create your server is to link with the
3101 <filename>lib/libyaz.la</filename> library.
3104 <sect1 id="server.main">
3105 <title>Your main() Routine</title>
3107 As mentioned, your <function>main()</function> routine can be quite brief.
3108 If you want to initialize global parameters, or read global configuration
3109 tables, this is the place to do it. At the end of the routine, you should
3113 int statserv_main(int argc, char **argv,
3114 bend_initresult *(*bend_init)(bend_initrequest *r),
3115 void (*bend_close)(void *handle));
3118 The third and fourth arguments are pointers to handlers. Handler
3119 <function>bend_init</function> is called whenever the server receives
3120 an Initialize Request, so it serves as a Z39.50 session initializer. The
3121 <function>bend_close</function> handler is called when the session is
3125 <function>statserv_main</function> will establish listening sockets
3126 according to the parameters given. When connection requests are received,
3127 the event handler will typically <function>fork()</function> and
3128 create a sub-process to handle a new connection.
3129 Alternatively the server may be setup to create threads for each
3131 If you do use global variables and forking, you should be aware, then,
3132 that these cannot be shared between associations, unless you explicitly
3133 disable forking by command line parameters.
3136 The server provides a mechanism for controlling some of its behavior
3137 without using command-line options. The function
3140 statserv_options_block *statserv_getcontrol(void);
3143 will return a pointer to a <literal>struct statserv_options_block</literal>
3144 describing the current default settings of the server. The structure
3145 contains these elements:
3148 <term><literal>int dynamic</literal></term>
3150 A boolean value, which determines whether the server
3151 will fork on each incoming request (TRUE), or not (FALSE). Default is
3152 TRUE. This flag is only read by UNIX-based servers (WIN32 based servers
3157 <term><literal>int threads</literal></term>
3159 A boolean value, which determines whether the server
3160 will create a thread on each incoming request (TRUE), or not (FALSE).
3161 Default is FALSE. This flag is only read by UNIX-based servers
3162 that offer POSIX Threads support.
3163 WIN32-based servers always operate in threaded mode.
3167 <term><literal>int inetd</literal></term>
3169 A boolean value, which determines whether the server
3170 will operates under a UNIX INET daemon (inetd). Default is FALSE.
3174 <term><literal>char logfile[ODR_MAXNAME+1]</literal></term>
3175 <listitem><para>File for diagnostic output ("": stderr).
3179 <term><literal>char apdufile[ODR_MAXNAME+1]</literal></term>
3181 Name of file for logging incoming and outgoing APDUs
3182 ("": don't log APDUs, "-":
3183 <literal>stderr</literal>).
3187 <term><literal>char default_listen[1024]</literal></term>
3188 <listitem><para>Same form as the command-line specification of
3189 listener address. "": no default listener address.
3190 Default is to listen at "tcp:@:9999". You can only
3191 specify one default listener address in this fashion.
3195 <term><literal>enum oid_proto default_proto;</literal></term>
3196 <listitem><para>Either <literal>PROTO_Z3950</literal> or
3197 <literal>PROTO_SR</literal>.
3198 Default is <literal>PROTO_Z39_50</literal>.
3202 <term><literal>int idle_timeout;</literal></term>
3203 <listitem><para>Maximum session idle-time, in minutes. Zero indicates
3204 no (infinite) timeout. Default is 15 minutes.
3208 <term><literal>int maxrecordsize;</literal></term>
3209 <listitem><para>Maximum permissible record (message) size. Default
3210 is 64 MB. This amount of memory will only be allocated if a
3211 client requests a very large amount of records in one operation
3213 Set it to a lower number if you are worried about resource
3214 consumption on your host system.
3218 <term><literal>char configname[ODR_MAXNAME+1]</literal></term>
3219 <listitem><para>Passed to the backend when a new connection is received.
3223 <term><literal>char setuid[ODR_MAXNAME+1]</literal></term>
3224 <listitem><para>Set user id to the user specified, after binding
3225 the listener addresses.
3230 <literal>void (*bend_start)(struct statserv_options_block *p)</literal>
3232 <listitem><para>Pointer to function which is called after the
3233 command line options have been parsed - but before the server
3235 For forked UNIX servers this handler is called in the mother
3236 process; for threaded servers this handler is called in the
3238 The default value of this pointer is NULL in which case it
3239 isn't invoked by the frontend server.
3240 When the server operates as an NT service this handler is called
3241 whenever the service is started.
3246 <literal>void (*bend_stop)(struct statserv_options_block *p)</literal>
3248 <listitem><para>Pointer to function which is called whenever the server
3249 has stopped listening for incoming connections. This function pointer
3250 has a default value of NULL in which case it isn't called.
3251 When the server operates as an NT service this handler is called
3252 whenever the service is stopped.
3256 <term><literal>void *handle</literal></term>
3257 <listitem><para>User defined pointer (default value NULL).
3258 This is a per-server handle that can be used to specify "user-data".
3259 Do not confuse this with the session-handle as returned by bend_init.
3265 The pointer returned by <literal>statserv_getcontrol</literal> points to
3266 a static area. You are allowed to change the contents of the structure,
3267 but the changes will not take effect before you call
3270 void statserv_setcontrol(statserv_options_block *block);
3274 that you should generally update this structure before calling
3275 <function>statserv_main()</function>.
3279 <sect1 id="server.backendfunctions">
3280 <title>The Backend Functions</title>
3282 For each service of the protocol, the backend interface declares one or
3283 two functions. You are required to provide implementations of the
3284 functions representing the services that you wish to implement.
3286 <sect2 id="server.init">
3289 bend_initresult (*bend_init)(bend_initrequest *r);
3292 This handler is called once for each new connection request, after
3293 a new process/thread has been created, and an Initialize Request has
3294 been received from the client. The pointer to the
3295 <function>bend_init</function> handler is passed in the call to
3296 <function>statserv_start</function>.
3299 This handler is also called when operating in SRU mode - when
3300 a connection has been made (even though SRU does not offer
3304 Unlike previous versions of YAZ, the <function>bend_init</function> also
3305 serves as a handler that defines the Z39.50 services that the backend
3306 wish to support. Pointers to <emphasis>all</emphasis> service handlers,
3307 including search - and fetch must be specified here in this handler.
3310 The request - and result structures are defined as
3313 typedef struct bend_initrequest
3315 /** \brief user/name/password to be read */
3316 Z_IdAuthentication *auth;
3317 /** \brief encoding stream (for results) */
3319 /** \brief printing stream */
3321 /** \brief decoding stream (use stream for results) */
3323 /** \brief reference ID */
3324 Z_ReferenceId *referenceId;
3325 /** \brief peer address of client */
3328 /** \brief character set and language negotiation
3330 see include/yaz/z-charneg.h
3332 Z_CharSetandLanguageNegotiation *charneg_request;
3334 /** \brief character negotiation response */
3335 Z_External *charneg_response;
3337 /** \brief character set (encoding) for query terms
3339 This is NULL by default. It should be set to the native character
3340 set that the backend assumes for query terms */
3341 char *query_charset;
3343 /** \brief whehter query_charset also applies to recors
3345 Is 0 (No) by default. Set to 1 (yes) if records is in the same
3346 character set as queries. If in doubt, use 0 (No).
3348 int records_in_same_charset;
3350 char *implementation_id;
3351 char *implementation_name;
3352 char *implementation_version;
3354 /** \brief Z39.50 sort handler */
3355 int (*bend_sort)(void *handle, bend_sort_rr *rr);
3356 /** \brief SRU/Z39.50 search handler */
3357 int (*bend_search)(void *handle, bend_search_rr *rr);
3358 /** \brief SRU/Z39.50 fetch handler */
3359 int (*bend_fetch)(void *handle, bend_fetch_rr *rr);
3360 /** \brief SRU/Z39.50 present handler */
3361 int (*bend_present)(void *handle, bend_present_rr *rr);
3362 /** \brief Z39.50 extended services handler */
3363 int (*bend_esrequest) (void *handle, bend_esrequest_rr *rr);
3364 /** \brief Z39.50 delete result set handler */
3365 int (*bend_delete)(void *handle, bend_delete_rr *rr);
3366 /** \brief Z39.50 scan handler */
3367 int (*bend_scan)(void *handle, bend_scan_rr *rr);
3368 /** \brief Z39.50 segment facility handler */
3369 int (*bend_segment)(void *handle, bend_segment_rr *rr);
3370 /** \brief SRU explain handler */
3371 int (*bend_explain)(void *handle, bend_explain_rr *rr);
3372 /** \brief SRU scan handler */
3373 int (*bend_srw_scan)(void *handle, bend_scan_rr *rr);
3374 /** \brief SRU record update handler */
3375 int (*bend_srw_update)(void *handle, bend_update_rr *rr);
3377 /** \brief whether named result sets are supported (0=disable, 1=enable) */
3378 int named_result_sets;
3381 typedef struct bend_initresult
3383 int errcode; /* 0==OK */
3384 char *errstring; /* system error string or NULL */
3385 void *handle; /* private handle to the backend module */
3389 In general, the server frontend expects that the
3390 <literal>bend_*result</literal> pointer that you return is valid at
3391 least until the next call to a <literal>bend_* function</literal>.
3392 This applies to all of the functions described herein. The parameter
3393 structure passed to you in the call belongs to the server frontend, and
3394 you should not make assumptions about its contents after the current
3395 function call has completed. In other words, if you want to retain any
3396 of the contents of a request structure, you should copy them.
3399 The <literal>errcode</literal> should be zero if the initialization of
3400 the backend went well. Any other value will be interpreted as an error.
3401 The <literal>errstring</literal> isn't used in the current version, but
3402 one option would be to stick it in the initResponse as a VisibleString.
3403 The <literal>handle</literal> is the most important parameter. It should
3404 be set to some value that uniquely identifies the current session to
3405 the backend implementation. It is used by the frontend server in any
3406 future calls to a backend function.
3407 The typical use is to set it to point to a dynamically allocated state
3408 structure that is private to your backend module.
3411 The <literal>auth</literal> member holds the authentication information
3412 part of the Z39.50 Initialize Request. Interpret this if your serves
3413 requires authentication.
3416 The members <literal>peer_name</literal>,
3417 <literal>implementation_id</literal>,
3418 <literal>implementation_name</literal> and
3419 <literal>implementation_version</literal> holds
3420 DNS of client, ID of implementor, name
3421 of client (Z39.50) implementation - and version.
3424 The <literal>bend_</literal> - members are set to NULL when
3425 <function>bend_init</function> is called. Modify the pointers by
3426 setting them to point to backend functions.
3429 <sect2 id="server.search.retrieve">
3430 <title>Search and Retrieve</title>
3432 We now describe the handlers that are required to support search -
3433 and retrieve. You must support two functions - one for search - and one
3434 for fetch (retrieval of one record). If desirable you can provide a
3435 third handler which is called when a present request is received which
3436 allows you to optimize retrieval of multiple-records.
3439 int (*bend_search) (void *handle, bend_search_rr *rr);
3442 char *setname; /* name to give to this set */
3443 int replace_set; /* replace set, if it already exists */
3444 int num_bases; /* number of databases in list */
3445 char **basenames; /* databases to search */
3446 Z_ReferenceId *referenceId;/* reference ID */
3447 Z_Query *query; /* query structure */
3448 ODR stream; /* encode stream */
3449 ODR decode; /* decode stream */
3450 ODR print; /* print stream */
3452 bend_request request;
3453 bend_association association;
3455 int hits; /* number of hits */
3456 int errcode; /* 0==OK */
3457 char *errstring; /* system error string or NULL */
3458 Z_OtherInformation *search_info; /* additional search info */
3459 char *srw_sortKeys; /* holds SRU/SRW sortKeys info */
3460 char *srw_setname; /* holds SRU/SRW generated resultsetID */
3461 int *srw_setnameIdleTime; /* holds SRU/SRW life-time */
3462 int estimated_hit_count; /* if hit count is estimated */
3463 int partial_resultset; /* if result set is partial */
3467 The <function>bend_search</function> handler is a fairly close
3468 approximation of a protocol Z39.50 Search Request - and Response PDUs
3469 The <literal>setname</literal> is the resultSetName from the protocol.
3470 You are required to establish a mapping between the set name and whatever
3471 your backend database likes to use.
3472 Similarly, the <literal>replace_set</literal> is a boolean value
3473 corresponding to the resultSetIndicator field in the protocol.
3474 <literal>num_bases/basenames</literal> is a length of/array of character
3475 pointers to the database names provided by the client.
3476 The <literal>query</literal> is the full query structure as defined in
3477 the protocol ASN.1 specification.
3478 It can be either of the possible query types, and it's up to you to
3479 determine if you can handle the provided query type.
3480 Rather than reproduce the C interface here, we'll refer you to the
3481 structure definitions in the file
3482 <filename>include/yaz/z-core.h</filename>. If you want to look at the
3483 attributeSetId OID of the RPN query, you can either match it against
3484 your own internal tables, or you can use the <link linkend="tools.oid">
3488 The structure contains a number of hits, and an
3489 <literal>errcode/errstring</literal> pair. If an error occurs
3490 during the search, or if you're unhappy with the request, you should
3491 set the errcode to a value from the BIB-1 diagnostic set. The value
3492 will then be returned to the user in a nonsurrogate diagnostic record
3493 in the response. The <literal>errstring</literal>, if provided, will
3494 go in the addinfo field. Look at the protocol definition for the
3495 defined error codes, and the suggested uses of the addinfo field.
3498 The <function>bend_search</function> handler is also called when
3499 the frontend server receives a SRU SearchRetrieveRequest.
3500 For SRU, a CQL query is usually provided by the client.
3501 The CQL query is available as part of <literal>Z_Query</literal>
3502 structure (note that CQL is now part of Z39.50 via an external).
3503 To support CQL in existing implementations that only do Type-1,
3504 we refer to the CQL-to-PQF tool described
3505 <link linkend="cql.to.pqf">here</link>.
3508 To maintain backwards compatibility, the frontend server
3509 of yaz always assume that error codes are BIB-1 diagnostics.
3510 For SRU operation, a Bib-1 diagnostic code is mapped to
3514 int (*bend_fetch) (void *handle, bend_fetch_rr *rr);
3516 typedef struct bend_fetch_rr {
3517 char *setname; /* set name */
3518 int number; /* record number */
3519 Z_ReferenceId *referenceId;/* reference ID */
3520 Odr_oid *request_format; /* format, transfer syntax (OID) */
3521 Z_RecordComposition *comp; /* Formatting instructions */
3522 ODR stream; /* encoding stream - memory source if req */
3523 ODR print; /* printing stream */
3525 char *basename; /* name of database that provided record */
3526 int len; /* length of record or -1 if structured */
3527 char *record; /* record */
3528 int last_in_set; /* is it? */
3529 Odr_oid *output_format; /* response format/syntax (OID) */
3530 int errcode; /* 0==success */
3531 char *errstring; /* system error string or NULL */
3532 int surrogate_flag; /* surrogate diagnostic */
3533 char *schema; /* string record schema input/output */
3537 The frontend server calls the <function>bend_fetch</function> handler
3538 when it needs database records to fulfill a Z39.50 Search Request, a
3539 Z39.50 Present Request or a SRU SearchRetrieveRequest.
3540 The <literal>setname</literal> is simply the name of the result set
3541 that holds the reference to the desired record.
3542 The <literal>number</literal> is the offset into the set (with 1
3543 being the first record in the set). The <literal>format</literal> field
3544 is the record format requested by the client (See
3545 <xref linkend="tools.oid"/>).
3546 A value of NULL for <literal>format</literal> indicates that the
3547 client did not request a specific format.
3548 The <literal>stream</literal> argument is an &odr; stream which
3549 should be used for allocating space for structured data records.
3550 The stream will be reset when all records have been assembled, and
3551 the response package has been transmitted.
3552 For unstructured data, the backend is responsible for maintaining a
3553 static or dynamic buffer for the record between calls.
3556 If a SRU SearchRetrieveRequest is received by the frontend server,
3557 the <literal>referenceId</literal> is NULL and the
3558 <literal>format</literal> (transfer syntax) is the OID for XML.
3559 The schema for SRU is stored in both the
3560 <literal>Z_RecordComposition</literal>
3561 structure and <literal>schema</literal> (simple string).
3564 In the structure, the <literal>basename</literal> is the name of the
3565 database that holds the
3566 record. <literal>len</literal> is the length of the record returned, in
3567 bytes, and <literal>record</literal> is a pointer to the record.
3568 <literal>last_in_set</literal> should be nonzero only if the record
3569 returned is the last one in the given result set.
3570 <literal>errcode</literal> and <literal>errstring</literal>, if
3571 given, will be interpreted as a global error pertaining to the
3572 set, and will be returned in a non-surrogate-diagnostic.
3573 If you wish to return the error as a surrogate-diagnostic
3574 (local error) you can do this by setting
3575 <literal>surrogate_flag</literal> to 1 also.
3578 If the <literal>len</literal> field has the value -1, then
3579 <literal>record</literal> is assumed to point to a constructed data
3580 type. The <literal>format</literal> field will be used to determine
3581 which encoder should be used to serialize the data.
3585 If your backend generates structured records, it should use
3586 <function>odr_malloc()</function> on the provided stream for allocating
3587 data: This allows the frontend server to keep track of the record sizes.
3591 The <literal>format</literal> field is mapped to an object identifier
3592 in the direct reference of the resulting EXTERNAL representation
3597 The current version of &yaz; only supports the direct reference mode.
3601 int (*bend_present) (void *handle, bend_present_rr *rr);
3604 char *setname; /* set name */
3606 int number; /* record number */
3607 Odr_oid *format; /* format, transfer syntax (OID) */
3608 Z_ReferenceId *referenceId;/* reference ID */
3609 Z_RecordComposition *comp; /* Formatting instructions */
3610 ODR stream; /* encoding stream - memory source if required */
3611 ODR print; /* printing stream */
3612 bend_request request;
3613 bend_association association;
3615 int hits; /* number of hits */
3616 int errcode; /* 0==OK */
3617 char *errstring; /* system error string or NULL */
3621 The <function>bend_present</function> handler is called when
3622 the server receives a Z39.50 Present Request.
3623 The <literal>setname</literal>,
3624 <literal>start</literal> and <literal>number</literal> is the
3625 name of the result set - start position - and number of records to
3626 be retrieved respectively. <literal>format</literal> and
3627 <literal>comp</literal> is the preferred transfer syntax and element
3628 specifications of the present request.
3631 Note that this is handler serves as a supplement for
3632 <function>bend_fetch</function> and need not to be defined in order to
3633 support search - and retrieve.
3636 <sect2 id="server.delete">
3637 <title>Delete</title>
3639 For back-ends that supports delete of a result set only one handler
3643 int (*bend_delete)(void *handle, bend_delete_rr *rr);
3645 typedef struct bend_delete_rr {
3649 Z_ReferenceId *referenceId;
3650 int delete_status; /* status for the whole operation */
3651 int *statuses; /* status each set - indexed as setnames */
3658 The delete set function definition is rather primitive, mostly because
3659 we have had no practical need for it as of yet. If someone wants
3660 to provide a full delete service, we'd be happy to add the
3661 extra parameters that are required. Are there clients out there
3662 that will actually delete sets they no longer need?
3666 <sect2 id="server.scan">
3669 For servers that wish to offer the scan service one handler
3673 int (*bend_scan)(void *handle, bend_scan_rr *rr);
3676 BEND_SCAN_SUCCESS, /* ok */
3677 BEND_SCAN_PARTIAL /* not all entries could be found */
3680 typedef struct bend_scan_rr {
3681 int num_bases; /* number of elements in databaselist */
3682 char **basenames; /* databases to search */
3683 Odr_oid *attributeset;
3684 Z_ReferenceId *referenceId; /* reference ID */
3685 Z_AttributesPlusTerm *term;
3686 ODR stream; /* encoding stream - memory source if required */
3687 ODR print; /* printing stream */
3689 int *step_size; /* step size */
3690 int term_position; /* desired index of term in result list/returned */
3691 int num_entries; /* number of entries requested/returned */
3693 /* scan term entries. The called handler does not have
3694 to allocate this. Size of entries is num_entries (see above) */
3695 struct scan_entry *entries;
3696 bend_scan_status status;
3699 char *scanClause; /* CQL scan clause */
3700 char *setname; /* Scan in result set (NULL if omitted) */
3704 This backend server handles both Z39.50 scan
3705 and SRU scan. In order for a handler to distinguish between SRU (CQL) scan
3706 Z39.50 Scan , it must check for a non-NULL value of
3707 <literal>scanClause</literal>.
3711 if designed today, it would be a choice using a union or similar,
3712 but that would break binary compatibility with existing servers.
3717 <sect1 id="server.invocation">
3718 <title>Application Invocation</title>
3720 The finished application has the following
3721 invocation syntax (by way of <function>statserv_main()</function>):
3729 A listener specification consists of a transport mode followed by a
3730 colon (:) followed by a listener address. The transport mode is
3731 either <literal>tcp</literal>, <literal>unix:</literal> or
3732 <literal>ssl</literal>.
3735 For TCP and SSL, an address has the form
3738 hostname | IP-number [: portnumber]
3741 The port number defaults to 210 (standard Z39.50 port).
3744 For UNIX, the address is the filename of socket.
3747 For TCP/IP and SSL, the special hostnames <literal>@</literal>,
3748 maps to <literal>IN6ADDR_ANY_INIT</literal> with
3749 IPV4 binding as well (bindv6only=0),
3750 The special hostname <literal>@4</literal> binds to
3751 <literal>INADDR_ANY</literal> (IPV4 only listener).
3752 The special hostname <literal>@6</literal> binds to
3753 <literal>IN6ADDR_ANY_INIT</literal> with bindv6only=1 (IPV6 only listener).
3755 <example id="server.example.running.unix">
3756 <title>Running the GFS on Unix</title>
3758 Assuming the server application <replaceable>appname</replaceable> is
3759 started as root, the following will make it listen on port 210.
3760 The server will change identity to <literal>nobody</literal>
3761 and write its log to <filename>/var/log/app.log</filename>.
3763 application -l /var/log/app.log -u nobody tcp:@:210
3767 The server will accept Z39.50 requests and offer SRU service on port 210.
3770 <example id="server.example.apache.sru">
3771 <title>Setting up Apache as SRU Frontend</title>
3773 If you use <ulink url="&url.apache;">Apache</ulink>
3774 as your public web server and want to offer HTTP port 80
3775 access to the YAZ server on 210, you can use the
3776 <ulink url="&url.apache.directive.proxypass;">
3777 <literal>ProxyPass</literal></ulink>
3779 If you have virtual host
3780 <literal>srw.mydomain</literal> you can use the following directives
3781 in Apache's httpd.conf:
3784 ErrorLog /home/srw/logs/error_log
3785 TransferLog /home/srw/logs/access_log
3786 ProxyPass / http://srw.mydomain:210/
3791 The above for the Apache 1.3 series.
3794 <example id="server.example.local.access">
3795 <title>Running a server with local access only</title>
3797 Servers that is only being accessed from the local host should listen
3798 on UNIX file socket rather than a Internet socket. To listen on
3799 <filename>/tmp/mysocket</filename> start the server as follows:
3801 application unix:/tmp/mysocket
3806 <sect1 id="server.vhosts">
3807 <title>GFS Configuration and Virtual Hosts</title>
3812 <title>The Z39.50 ASN.1 Module</title>
3813 <sect1 id="asn.introduction">
3814 <title>Introduction</title>
3816 The &asn; module provides you with a set of C struct definitions for the
3817 various PDUs of the Z39.50 protocol, as well as for the complex types
3818 appearing within the PDUs. For the primitive data types, the C
3819 representation often takes the form of an ordinary C language type,
3820 such as <literal>Odr_int</literal> which is equivalent to an integral
3821 C integer. For ASN.1 constructs that have no direct
3822 representation in C, such as general octet strings and bit strings,
3823 the &odr; module (see section <link linkend="odr">The ODR Module</link>)
3824 provides auxiliary definitions.
3827 The &asn; module is located in sub directory <filename>z39.50</filename>.
3828 There you'll find C files that implements encoders and decoders for the
3829 Z39.50 types. You'll also find the protocol definitions:
3830 <filename>z3950v3.asn</filename>, <filename>esupdate.asn</filename>,
3834 <sect1 id="asn.preparing">
3835 <title>Preparing PDUs</title>
3837 A structure representing a complex ASN.1 type doesn't in itself contain the
3838 members of that type. Instead, the structure contains
3839 <emphasis>pointers</emphasis> to the members of the type.
3840 This is necessary, in part, to allow a mechanism for specifying which
3841 of the optional structure (SEQUENCE) members are present, and which
3842 are not. It follows that you will need to somehow provide space for
3843 the individual members of the structure, and set the pointers to
3844 refer to the members.
3847 The conversion routines don't care how you allocate and maintain your
3848 C structures - they just follow the pointers that you provide.
3849 Depending on the complexity of your application, and your personal
3850 taste, there are at least three different approaches that you may take
3851 when you allocate the structures.
3854 You can use static or automatic local variables in the function that
3855 prepares the PDU. This is a simple approach, and it provides the most
3856 efficient form of memory management. While it works well for flat
3857 PDUs like the InitReqest, it will generally not be sufficient for say,
3858 the generation of an arbitrarily complex RPN query structure.
3861 You can individually create the structure and its members using the
3862 <function>malloc(2)</function> function. If you want to ensure that
3863 the data is freed when it is no longer needed, you will have to
3864 define a function that individually releases each member of a
3865 structure before freeing the structure itself.
3868 You can use the <function>odr_malloc()</function> function (see
3869 <xref linkend="odr.use"/> for details). When you use
3870 <function>odr_malloc()</function>, you can release all of the
3871 allocated data in a single operation, independent of any pointers and
3872 relations between the data. <function>odr_malloc()</function> is based on a
3873 "nibble-memory"
3874 scheme, in which large portions of memory are allocated, and then
3875 gradually handed out with each call to <function>odr_malloc()</function>.
3876 The next time you call <function>odr_reset()</function>, all of the
3877 memory allocated since the last call is recycled for future use (actually,
3878 it is placed on a free-list).
3881 You can combine all of the methods described here. This will often be
3882 the most practical approach. For instance, you might use
3883 <function>odr_malloc()</function> to allocate an entire structure and
3884 some of its elements, while you leave other elements pointing to global
3885 or per-session default variables.
3888 The &asn; module provides an important aid in creating new PDUs. For
3889 each of the PDU types (say, <function>Z_InitRequest</function>), a
3890 function is provided that allocates and initializes an instance of
3891 that PDU type for you. In the case of the InitRequest, the function is
3892 simply named <function>zget_InitRequest()</function>, and it sets up
3893 reasonable default value for all of the mandatory members. The optional
3894 members are generally initialized to null pointers. This last aspect
3895 is very important: it ensures that if the PDU definitions are
3896 extended after you finish your implementation (to accommodate
3897 new versions of the protocol, say), you won't get into trouble with
3898 uninitialized pointers in your structures. The functions use
3899 <function>odr_malloc()</function> to
3900 allocate the PDUs and its members, so you can free everything again with a
3901 single call to <function>odr_reset()</function>. We strongly recommend
3902 that you use the <literal>zget_*</literal>
3903 functions whenever you are preparing a PDU (in a C++ API, the
3904 <literal>zget_</literal>
3905 functions would probably be promoted to constructors for the
3909 The prototype for the individual PDU types generally look like this:
3912 Z_<type> *zget_<type>(ODR o);
3918 Z_InitRequest *zget_InitRequest(ODR o);
3921 The &odr; handle should generally be your encoding stream, but it
3925 As well as the individual PDU functions, a function
3926 <function>zget_APDU()</function> is provided, which allocates
3927 a top-level Z-APDU of the type requested:
3930 Z_APDU *zget_APDU(ODR o, int which);
3933 The <varname>which</varname> parameter is (of course) the discriminator
3934 belonging to the <varname>Z_APDU</varname> <literal>CHOICE</literal> type.
3935 All of the interface described here is provided by the &asn; module, and
3936 you access it through the <filename>proto.h</filename> header file.
3939 <sect1 id="asn.external">
3940 <title>EXTERNAL Data</title>
3942 In order to achieve extensibility and adaptability to different
3943 application domains, the new version of the protocol defines many
3944 structures outside of the main ASN.1 specification, referencing them
3945 through ASN.1 EXTERNAL constructs. To simplify the construction and
3946 access to the externally referenced data, the &asn; module defines a
3947 specialized version of the EXTERNAL construct, called
3948 <literal>Z_External</literal>.It is defined thus:
3951 typedef struct Z_External
3953 Odr_oid *direct_reference;
3954 int *indirect_reference;
3959 Z_External_single = 0,
3961 Z_External_arbitrary,
3963 /* Specific types */
3965 Z_External_explainRecord,
3966 Z_External_resourceReport1,
3967 Z_External_resourceReport2
3975 Odr_any *single_ASN1_type;
3976 Odr_oct *octet_aligned;
3977 Odr_bitmask *arbitrary;
3979 /* Specific types */
3981 Z_ExplainRecord *explainRecord;
3982 Z_ResourceReport1 *resourceReport1;
3983 Z_ResourceReport2 *resourceReport2;
3991 When decoding, the &asn; module will attempt to determine which
3992 syntax describes the data by looking at the reference fields
3993 (currently only the direct-reference). For ASN.1 structured data, you
3994 need only consult the <literal>which</literal> field to determine the
3995 type of data. You can the access the data directly through the union.
3996 When constructing data for encoding, you set the union pointer to point
3997 to the data, and set the <literal>which</literal> field accordingly.
3998 Remember also to set the direct (or indirect) reference to the correct
3999 OID for the data type.
4000 For non-ASN.1 data such as MARC records, use the
4001 <literal>octet_aligned</literal> arm of the union.
4004 Some servers return ASN.1 structured data values (eg. database
4005 records) as BER-encoded records placed in the
4006 <literal>octet-aligned</literal> branch of the EXTERNAL CHOICE.
4007 The ASN-module will <emphasis>not</emphasis> automatically decode
4008 these records. To help you decode the records in the application, the
4012 Z_ext_typeent *z_ext_gettypebyref(const oid *oid);
4015 Can be used to retrieve information about the known, external data
4016 types. The function return a pointer to a static area, or NULL, if no
4017 match for the given direct reference is found. The
4018 <literal>Z_ext_typeent</literal>
4022 typedef struct Z_ext_typeent
4024 int oid[OID_SIZE]; /* the direct-reference OID. */
4025 int what; /* discriminator value for the external CHOICE */
4026 Odr_fun fun; /* decoder function */
4030 The <literal>what</literal> member contains the
4031 <literal>Z_External</literal> union discriminator value for the
4032 given type: For the SUTRS record syntax, the value would be
4033 <literal>Z_External_sutrs</literal>.
4034 The <literal>fun</literal> member contains a pointer to the
4035 function which encodes/decodes the given type. Again, for the SUTRS
4036 record syntax, the value of <literal>fun</literal> would be
4037 <literal>z_SUTRS</literal> (a function pointer).
4040 If you receive an EXTERNAL which contains an octet-string value that
4041 you suspect of being an ASN.1-structured data value, you can use
4042 <literal>z_ext_gettypebyref</literal> to look for the provided
4044 If the return value is different from NULL, you can use the provided
4045 function to decode the BER string (see <xref linkend="odr.use"/>
4049 If you want to <emphasis>send</emphasis> EXTERNALs containing
4050 ASN.1-structured values in the occtet-aligned branch of the CHOICE, this
4051 is possible too. However, on the encoding phase, it requires a somewhat
4052 involved juggling around of the various buffers involved.
4055 If you need to add new, externally defined data types, you must update
4056 the struct above, in the source file <filename>prt-ext.h</filename>, as
4057 well as the encoder/decoder in the file <filename>prt-ext.c</filename>.
4058 When changing the latter, remember to update both the
4059 <literal>arm</literal> arrary and the list
4060 <literal>type_table</literal>, which drives the CHOICE biasing that
4061 is necessary to tell the different, structured types apart
4066 Eventually, the EXTERNAL processing will most likely
4067 automatically insert the correct OIDs or indirect-refs. First,
4068 however, we need to determine how application-context management
4069 (specifically the presentation-context-list) should fit into the
4074 <sect1 id="asn.pdu">
4075 <title>PDU Contents Table</title>
4077 We include, for reference, a listing of the fields of each top-level
4078 PDU, as well as their default settings.
4080 <table frame="top" id="asn.default.initialize.request">
4081 <title>Default settings for PDU Initialize Request</title>
4083 <colspec colwidth="7*" colname="field"></colspec>
4084 <colspec colwidth="5*" colname="type"></colspec>
4085 <colspec colwidth="7*" colname="value"></colspec>
4088 <entry>Field</entry>
4090 <entry>Default Value</entry>
4095 referenceId</entry><entry>Z_ReferenceId</entry><entry>NULL
4098 protocolVersion</entry><entry>Odr_bitmask</entry><entry>Empty bitmask
4101 options</entry><entry>Odr_bitmask</entry><entry>Empty bitmask
4104 preferredMessageSize</entry><entry>Odr_int</entry><entry>30*1024
4107 maximumRecordSize</entry><entry>Odr_int</entry><entry>30*1024
4110 idAuthentication</entry><entry>Z_IdAuthentication</entry><entry>NULL
4113 implementationId</entry><entry>char*</entry><entry>"81"
4116 implementationName</entry><entry>char*</entry><entry>"YAZ"
4119 implementationVersion</entry><entry>char*</entry><entry>YAZ_VERSION
4122 userInformationField</entry><entry>Z_UserInformation</entry><entry>NULL
4125 otherInfo</entry><entry>Z_OtherInformation</entry><entry>NULL
4130 <table frame="top" id="asn.default.initialize.response">
4131 <title>Default settings for PDU Initialize Response</title>
4133 <colspec colwidth="7*" colname="field"></colspec>
4134 <colspec colwidth="5*" colname="type"></colspec>
4135 <colspec colwidth="7*" colname="value"></colspec>
4138 <entry>Field</entry>
4140 <entry>Default Value</entry>
4145 referenceId</entry><entry>Z_ReferenceId</entry><entry>NULL
4148 protocolVersion</entry><entry>Odr_bitmask</entry><entry>Empty bitmask
4151 options</entry><entry>Odr_bitmask</entry><entry>Empty bitmask
4154 preferredMessageSize</entry><entry>Odr_int</entry><entry>30*1024
4157 maximumRecordSize</entry><entry>Odr_int</entry><entry>30*1024
4160 result</entry><entry>Odr_bool</entry><entry>TRUE
4163 implementationId</entry><entry>char*</entry><entry>"id)"
4166 implementationName</entry><entry>char*</entry><entry>"YAZ"
4169 implementationVersion</entry><entry>char*</entry><entry>YAZ_VERSION
4172 userInformationField</entry><entry>Z_UserInformation</entry><entry>NULL
4175 otherInfo</entry><entry>Z_OtherInformation</entry><entry>NULL
4180 <table frame="top" id="asn.default.search.request">
4181 <title>Default settings for PDU Search Request</title>
4183 <colspec colwidth="7*" colname="field"></colspec>
4184 <colspec colwidth="5*" colname="type"></colspec>
4185 <colspec colwidth="7*" colname="value"></colspec>
4188 <entry>Field</entry>
4190 <entry>Default Value</entry>
4195 referenceId</entry><entry>Z_ReferenceId</entry><entry>NULL
4198 smallSetUpperBound</entry><entry>Odr_int</entry><entry>0
4201 largeSetLowerBound</entry><entry>Odr_int</entry><entry>1
4204 mediumSetPresentNumber</entry><entry>Odr_int</entry><entry>0
4207 replaceIndicator</entry><entry>Odr_bool</entry><entry>TRUE
4210 resultSetName</entry><entry>char *</entry><entry>"default"
4213 num_databaseNames</entry><entry>Odr_int</entry><entry>0
4216 databaseNames</entry><entry>char **</entry><entry>NULL
4219 smallSetElementSetNames</entry><entry>Z_ElementSetNames
4223 mediumSetElementSetNames</entry><entry>Z_ElementSetNames
4227 preferredRecordSyntax</entry><entry>Odr_oid</entry><entry>NULL
4230 query</entry><entry>Z_Query</entry><entry>NULL
4233 additionalSearchInfo</entry><entry>Z_OtherInformation
4237 otherInfo</entry><entry>Z_OtherInformation</entry><entry>NULL
4242 <table frame="top" id="asn.default.search.response">
4243 <title>Default settings for PDU Search Response</title>
4245 <colspec colwidth="7*" colname="field"></colspec>
4246 <colspec colwidth="5*" colname="type"></colspec>
4247 <colspec colwidth="7*" colname="value"></colspec>
4250 <entry>Field</entry>
4252 <entry>Default Value</entry>
4257 referenceId</entry><entry>Z_ReferenceId</entry><entry>NULL
4260 resultCount</entry><entry>Odr_int</entry><entry>0
4263 numberOfRecordsReturned</entry><entry>Odr_int</entry><entry>0
4266 nextResultSetPosition</entry><entry>Odr_int</entry><entry>0
4269 searchStatus</entry><entry>Odr_bool</entry><entry>TRUE
4272 resultSetStatus</entry><entry>Odr_int</entry><entry>NULL
4275 presentStatus</entry><entry>Odr_int</entry><entry>NULL
4278 records</entry><entry>Z_Records</entry><entry>NULL
4281 additionalSearchInfo</entry>
4282 <entry>Z_OtherInformation</entry><entry>NULL
4285 otherInfo</entry><entry>Z_OtherInformation</entry><entry>NULL
4290 <table frame="top" id="asn.default.present.request">
4291 <title>Default settings for PDU Present Request</title>
4293 <colspec colwidth="7*" colname="field"></colspec>
4294 <colspec colwidth="5*" colname="type"></colspec>
4295 <colspec colwidth="7*" colname="value"></colspec>
4298 <entry>Field</entry>
4300 <entry>Default Value</entry>
4305 referenceId</entry><entry>Z_ReferenceId</entry><entry>NULL
4308 resultSetId</entry><entry>char*</entry><entry>"default"
4311 resultSetStartPoint</entry><entry>Odr_int</entry><entry>1
4314 numberOfRecordsRequested</entry><entry>Odr_int</entry><entry>10
4317 num_ranges</entry><entry>Odr_int</entry><entry>0
4320 additionalRanges</entry><entry>Z_Range</entry><entry>NULL
4323 recordComposition</entry><entry>Z_RecordComposition</entry><entry>NULL
4326 preferredRecordSyntax</entry><entry>Odr_oid</entry><entry>NULL
4329 maxSegmentCount</entry><entry>Odr_int</entry><entry>NULL
4332 maxRecordSize</entry><entry>Odr_int</entry><entry>NULL
4335 maxSegmentSize</entry><entry>Odr_int</entry><entry>NULL
4338 otherInfo</entry><entry>Z_OtherInformation</entry><entry>NULL
4343 <table frame="top" id="asn.default.present.response">
4344 <title>Default settings for PDU Present Response</title>
4346 <colspec colwidth="7*" colname="field"></colspec>
4347 <colspec colwidth="5*" colname="type"></colspec>
4348 <colspec colwidth="7*" colname="value"></colspec>
4351 <entry>Field</entry>
4353 <entry>Default Value</entry>
4358 referenceId</entry><entry>Z_ReferenceId</entry><entry>NULL
4361 numberOfRecordsReturned</entry><entry>Odr_int</entry><entry>0
4364 nextResultSetPosition</entry><entry>Odr_int</entry><entry>0
4367 presentStatus</entry><entry>Odr_int</entry><entry>Z_PresentStatus_success
4370 records</entry><entry>Z_Records</entry><entry>NULL
4373 otherInfo</entry><entry>Z_OtherInformation</entry><entry>NULL
4378 <table frame="top" id="asn.default.delete.result.set.request">
4379 <title>Default settings for Delete Result Set Request</title>
4381 <colspec colwidth="7*" colname="field"></colspec>
4382 <colspec colwidth="5*" colname="type"></colspec>
4383 <colspec colwidth="7*" colname="value"></colspec>
4386 <entry>Field</entry>
4388 <entry>Default Value</entry>
4392 <row><entry>referenceId
4393 </entry><entry>Z_ReferenceId</entry><entry>NULL
4396 deleteFunction</entry><entry>Odr_int</entry><entry>Z_DeleteResultSetRequest_list
4399 num_ids</entry><entry>Odr_int</entry><entry>0
4402 resultSetList</entry><entry>char**</entry><entry>NULL
4405 otherInfo</entry><entry>Z_OtherInformation</entry><entry>NULL
4410 <table frame="top" id="asn.default.delete.result.set.response">
4411 <title>Default settings for Delete Result Set Response</title>
4413 <colspec colwidth="7*" colname="field"></colspec>
4414 <colspec colwidth="5*" colname="type"></colspec>
4415 <colspec colwidth="7*" colname="value"></colspec>
4418 <entry>Field</entry>
4420 <entry>Default Value</entry>
4425 referenceId</entry><entry>Z_ReferenceId</entry><entry>NULL
4428 deleteOperationStatus</entry><entry>Odr_int</entry>
4429 <entry>Z_DeleteStatus_success</entry></row>
4431 num_statuses</entry><entry>Odr_int</entry><entry>0
4434 deleteListStatuses</entry><entry>Z_ListStatus**</entry><entry>NULL
4437 numberNotDeleted</entry><entry>Odr_int</entry><entry>NULL
4440 num_bulkStatuses</entry><entry>Odr_int</entry><entry>0
4443 bulkStatuses</entry><entry>Z_ListStatus</entry><entry>NUL
4446 deleteMessage</entry><entry>char*</entry><entry>NULL
4449 otherInfo</entry><entry>Z_OtherInformation</entry><entry>NULL
4454 <table frame="top" id="asn.default.scan.request">
4455 <title>Default settings for Scan Request</title>
4457 <colspec colwidth="7*" colname="field"></colspec>
4458 <colspec colwidth="5*" colname="type"></colspec>
4459 <colspec colwidth="7*" colname="value"></colspec>
4462 <entry>Field</entry>
4464 <entry>Default Value</entry>
4469 referenceId</entry><entry>Z_ReferenceId</entry><entry>NULL
4472 num_databaseNames</entry><entry>Odr_int</entry><entry>0
4475 databaseNames</entry><entry>char**</entry><entry>NULL
4478 attributeSet</entry><entry>Odr_oid</entry><entry>NULL
4481 termListAndStartPoint</entry><entry>Z_AttributesPlus...
4482 </entry><entry>NULL</entry></row>
4484 stepSize</entry><entry>Odr_int</entry><entry>NULL
4487 numberOfTermsRequested</entry><entry>Odr_int</entry><entry>20
4490 preferredPositionInResponse</entry><entry>Odr_int</entry><entry>NULL
4493 otherInfo</entry><entry>Z_OtherInformation</entry><entry>NULL
4498 <table frame="top" id="asn.default.scan.response">
4499 <title>Default settings for Scan Response</title>
4501 <colspec colwidth="7*" colname="field"></colspec>
4502 <colspec colwidth="5*" colname="type"></colspec>
4503 <colspec colwidth="7*" colname="value"></colspec>
4506 <entry>Field</entry>
4508 <entry>Default Value</entry>
4513 referenceId</entry><entry>Z_ReferenceId</entry><entry>NULL
4516 stepSize</entry><entry>Odr_int</entry><entry>NULL
4519 scanStatus</entry><entry>Odr_int</entry><entry>Z_Scan_success
4522 numberOfEntriesReturned</entry><entry>Odr_int</entry><entry>0
4525 positionOfTerm</entry><entry>Odr_int</entry><entry>NULL
4528 entries</entry><entry>Z_ListEntris</entry><entry>NULL
4531 attributeSet</entry><entry>Odr_oid</entry><entry>NULL
4534 otherInfo</entry><entry>Z_OtherInformation</entry><entry>NULL
4539 <table frame="top" id="asn.default.trigger.resource.control.request">
4540 <title>Default settings for Trigger Resource Control Request</title>
4542 <colspec colwidth="7*" colname="field"></colspec>
4543 <colspec colwidth="5*" colname="type"></colspec>
4544 <colspec colwidth="7*" colname="value"></colspec>
4547 <entry>Field</entry>
4549 <entry>Default Value</entry>
4554 referenceId</entry><entry>Z_ReferenceId</entry><entry>NULL
4557 requestedAction</entry><entry>Odr_int</entry><entry>
4558 Z_TriggerResourceCtrl_resou..
4561 prefResourceReportFormat</entry><entry>Odr_oid</entry><entry>NULL
4564 resultSetWanted</entry><entry>Odr_bool</entry><entry>NULL
4567 otherInfo</entry><entry>Z_OtherInformation</entry><entry>NULL
4572 <table frame="top" id="asn.default.resource.control.request">
4573 <title>Default settings for Resource Control Request</title>
4575 <colspec colwidth="7*" colname="field"></colspec>
4576 <colspec colwidth="5*" colname="type"></colspec>
4577 <colspec colwidth="7*" colname="value"></colspec>
4580 <entry>Field</entry>
4582 <entry>Default Value</entry>
4587 referenceId</entry><entry>Z_ReferenceId</entry><entry>NULL
4590 suspendedFlag</entry><entry>Odr_bool</entry><entry>NULL
4593 resourceReport</entry><entry>Z_External</entry><entry>NULL
4596 partialResultsAvailable</entry><entry>Odr_int</entry><entry>NULL
4599 responseRequired</entry><entry>Odr_bool</entry><entry>FALSE
4602 triggeredRequestFlag</entry><entry>Odr_bool</entry><entry>NULL
4605 otherInfo</entry><entry>Z_OtherInformation</entry><entry>NULL
4610 <table frame="top" id="asn.default.resource.control.response">
4611 <title>Default settings for Resource Control Response</title>
4613 <colspec colwidth="7*" colname="field"></colspec>
4614 <colspec colwidth="5*" colname="type"></colspec>
4615 <colspec colwidth="7*" colname="value"></colspec>
4618 <entry>Field</entry>
4620 <entry>Default Value</entry>
4625 referenceId</entry><entry>Z_ReferenceId</entry><entry>NULL
4628 continueFlag</entry><entry>bool_t</entry><entry>TRUE
4631 resultSetWanted</entry><entry>bool_t</entry><entry>NULL
4634 otherInfo</entry><entry>Z_OtherInformation</entry><entry>NULL
4639 <table frame="top" id="asn.default.access.control.request">
4640 <title>Default settings for Access Control Request</title>
4642 <colspec colwidth="7*" colname="field"></colspec>
4643 <colspec colwidth="5*" colname="type"></colspec>
4644 <colspec colwidth="7*" colname="value"></colspec>
4647 <entry>Field</entry>
4649 <entry>Default Value</entry>
4654 referenceId</entry><entry>Z_ReferenceId</entry><entry>NULL
4657 which</entry><entry>enum</entry><entry>Z_AccessRequest_simpleForm;
4660 u</entry><entry>union</entry><entry>NULL
4663 otherInfo</entry><entry>Z_OtherInformation</entry><entry>NULL
4668 <table frame="top" id="asn.default.access.control.response">
4669 <title>Default settings for Access Control Response</title>
4671 <colspec colwidth="7*" colname="field"></colspec>
4672 <colspec colwidth="5*" colname="type"></colspec>
4673 <colspec colwidth="7*" colname="value"></colspec>
4676 <entry>Field</entry>
4678 <entry>Default Value</entry>
4683 referenceId</entry><entry>Z_ReferenceId</entry><entry>NULL
4686 which</entry><entry>enum</entry><entry>Z_AccessResponse_simpleForm
4689 u</entry><entry>union</entry><entry>NULL
4692 diagnostic</entry><entry>Z_DiagRec</entry><entry>NULL
4695 otherInfo</entry><entry>Z_OtherInformation</entry><entry>NULL
4700 <table frame="top" id="asn.default.segment">
4701 <title>Default settings for Segment</title>
4703 <colspec colwidth="7*" colname="field"></colspec>
4704 <colspec colwidth="5*" colname="type"></colspec>
4705 <colspec colwidth="7*" colname="value"></colspec>
4708 <entry>Field</entry>
4710 <entry>Default Value</entry>
4715 referenceId</entry><entry>Z_ReferenceId</entry><entry>NULL
4718 numberOfRecordsReturned</entry><entry>Odr_int</entry><entry>value=0
4721 num_segmentRecords</entry><entry>Odr_int</entry><entry>0
4724 segmentRecords</entry><entry>Z_NamePlusRecord</entry><entry>NULL
4726 <row><entry>otherInfo</entry><entry>Z_OtherInformation</entry><entry>NULL
4731 <table frame="top" id="asn.default.close">
4732 <title>Default settings for Close</title>
4734 <colspec colwidth="7*" colname="field"></colspec>
4735 <colspec colwidth="5*" colname="type"></colspec>
4736 <colspec colwidth="7*" colname="value"></colspec>
4739 <entry>Field</entry>
4741 <entry>Default Value</entry>
4746 referenceId</entry><entry>Z_ReferenceId</entry><entry>NULL
4749 closeReason</entry><entry>Odr_int</entry><entry>Z_Close_finished
4752 diagnosticInformation</entry><entry>char*</entry><entry>NULL
4755 resourceReportFormat</entry><entry>Odr_oid</entry><entry>NULL
4758 resourceFormat</entry><entry>Z_External</entry><entry>NULL
4761 otherInfo</entry><entry>Z_OtherInformation</entry><entry>NULL
4769 <title>SOAP and SRU</title>
4770 <sect1 id="soap.introduction">
4771 <title>Introduction</title>
4773 &yaz; uses a very simple implementation of
4774 <ulink url="&url.soap;">SOAP</ulink> that only,
4775 currenly, supports what is sufficient to offer SRU SOAP functionality.
4776 The implementation uses the
4777 <ulink url="&url.libxml2.api.tree;">tree API</ulink> of
4778 libxml2 to encode and decode SOAP packages.
4781 Like the Z39.50 ASN.1 module, the &yaz; SRU implementation uses
4782 simple C structs to represent SOAP packages as well as
4786 <sect1 id="soap.http">
4789 &yaz; only offers HTTP as transport carrier for SOAP, but it is
4790 relatively easy to change that.
4793 The following definition of <literal>Z_GDU</literal> (Generic Data
4794 Unit) allows for both HTTP and Z39.50 in one packet.
4797 #include <yaz/zgdu.h>
4799 #define Z_GDU_Z3950 1
4800 #define Z_GDU_HTTP_Request 2
4801 #define Z_GDU_HTTP_Response 3
4806 Z_HTTP_Request *HTTP_Request;
4807 Z_HTTP_Response *HTTP_Response;
4812 The corresponding Z_GDU encoder/decoder is <function>z_GDU</function>.
4813 The <literal>z3950</literal> is any of the known BER encoded Z39.50
4815 <literal>HTTP_Request</literal> and <literal>HTTP_Response</literal>
4816 is the HTTP Request and Response respectively.
4819 <sect1 id="soap.xml">
4820 <title>SOAP Packages</title>
4822 Every SOAP package in &yaz; is represented as follows:
4824 #include <yaz/soap.h>
4838 #define Z_SOAP_fault 1
4839 #define Z_SOAP_generic 2
4840 #define Z_SOAP_error 3
4844 Z_SOAP_Fault *fault;
4845 Z_SOAP_Generic *generic;
4846 Z_SOAP_Fault *soap_error;
4853 The <literal>fault</literal> and <literal>soap_error</literal>
4854 arms represent both a SOAP fault - struct
4855 <literal>Z_SOAP_Fault</literal>. Any other generic
4856 (valid) package is represented by <literal>Z_SOAP_Generic</literal>.
4859 The <literal>ns</literal> as part of <literal>Z_SOAP</literal>
4860 is the namespace for SOAP itself and reflects the SOAP
4861 version. For version 1.1 it is
4862 <literal>http://schemas.xmlsoap.org/soap/envelope/</literal>,
4863 for version 1.2 it is
4864 <literal>http://www.w3.org/2001/06/soap-envelope</literal>.
4867 int z_soap_codec(ODR o, Z_SOAP **pp,
4868 char **content_buf, int *content_len,
4869 Z_SOAP_Handler *handlers);
4872 The <literal>content_buf</literal> and <literal>content_len</literal>
4873 is XML buffer and length of buffer respectively.
4876 The <literal>handlers</literal> is a list of SOAP codec
4877 handlers - one handler for each service namespace. For SRU SOAP, the
4878 namespace would be <literal>http://www.loc.gov/zing/srw/v1.0/</literal>.
4881 When decoding, the <function>z_soap_codec</function>
4882 inspects the XML content
4883 and tries to match one of the services namespaces of the
4884 supplied handlers. If there is a match a handler function
4885 is invoked which decodes that particular SOAP package.
4886 If successful, the returned <literal>Z_SOAP</literal> package will be
4887 of type <literal>Z_SOAP_Generic</literal>.
4888 Member <literal>no</literal> is
4889 set the offset of handler that matched; <literal>ns</literal>
4890 is set to namespace of matching handler; the void pointer
4891 <literal>p</literal> is set to the C data structure assocatiated
4895 When a NULL namespace is met (member <literal>ns</literal> bwlow),
4896 that specifies end-of-list.
4899 Each handler is defined as follows:
4907 The <literal>ns</literal> is namespace of service associated with
4908 handler <literal>f</literal>. <literal>client_data</literal>
4909 is user-defined data which is passed to handler.
4912 The prototype for a SOAP service handler is:
4914 int handler(ODR o, void * ptr, void **handler_data,
4915 void *client_data, const char *ns);
4917 The <parameter>o</parameter> specifies the mode (decode/encode)
4918 as usual. The second argument, <parameter>ptr</parameter>,
4919 is a libxml2 tree node pointer (<literal>xmlNodePtr</literal>)
4920 and is a pointer to the <literal>Body</literal> element
4921 of the SOAP package. The <parameter>handler_data</parameter>
4922 is an opaque pointer to a C definitions associated with the
4923 SOAP service. <parameter>client_data</parameter> is the pointer
4924 which was set as part of the <literal>Z_SOAP_handler</literal>.
4925 Finally, <parameter>ns</parameter> the service namespace.
4928 <sect1 id="soap.srw">
4931 SRU SOAP is just one implementation of a SOAP handler as described
4932 in the previous section.
4933 The encoder/decoder handler for SRU is defined as
4936 #include <yaz/srw.h>
4938 int yaz_srw_codec(ODR o, void * pptr,
4939 Z_SRW_GDU **handler_data,
4940 void *client_data, const char *ns);
4942 Here, <literal>Z_SRW_GDU</literal> is either
4943 searchRetrieveRequest or a searchRetrieveResponse.
4947 The xQuery and xSortKeys are not handled yet by
4948 the SRW implementation of &yaz;. Explain is also missing.
4949 Future versions of &yaz; will include these features.
4953 The definition of searchRetrieveRequest is:
4957 #define Z_SRW_query_type_cql 1
4958 #define Z_SRW_query_type_xcql 2
4959 #define Z_SRW_query_type_pqf 3
4967 #define Z_SRW_sort_type_none 1
4968 #define Z_SRW_sort_type_sort 2
4969 #define Z_SRW_sort_type_xSort 3
4977 int *maximumRecords;
4979 char *recordPacking;
4981 } Z_SRW_searchRetrieveRequest;
4983 Please observe that data of type xsd:string is represented
4984 as a char pointer (<literal>char *</literal>). A null pointer
4985 means that the element is absent.
4986 Data of type xsd:integer is representd as a pointer to
4987 an int (<literal>int *</literal>). Again, a null pointer
4988 us used for absent elements.
4991 The SearchRetrieveResponse has the following definition.
4994 int * numberOfRecords;
4996 int * resultSetIdleTime;
4998 Z_SRW_record *records;
5001 Z_SRW_diagnostic *diagnostics;
5002 int num_diagnostics;
5003 int *nextRecordPosition;
5004 } Z_SRW_searchRetrieveResponse;
5006 The <literal>num_records</literal> and <literal>num_diagnostics</literal>
5007 is number of returned records and diagnostics respectively and also
5008 correspond to the "size of" arrays <literal>records</literal>
5009 and <literal>diagnostics</literal>.
5012 A retrieval record is defined as follows:
5016 char *recordData_buf;
5018 int *recordPosition;
5021 The record data is defined as a buffer of some length so that
5022 data can be of any type. SRW 1.0 currenly doesn't allow for this
5023 (only XML), but future versions might do.
5026 And, a diagnostic as:
5036 <chapter id="tools">
5037 <title>Supporting Tools</title>
5039 In support of the service API - primarily the ASN module, which
5040 provides the pro-grammatic interface to the Z39.50 APDUs, &yaz; contains
5041 a collection of tools that support the development of applications.
5043 <sect1 id="tools.query">
5044 <title>Query Syntax Parsers</title>
5046 Since the type-1 (RPN) query structure has no direct, useful string
5047 representation, every origin application needs to provide some form of
5048 mapping from a local query notation or representation to a
5049 <token>Z_RPNQuery</token> structure. Some programmers will prefer to
5050 construct the query manually, perhaps using
5051 <function>odr_malloc()</function> to simplify memory management.
5052 The &yaz; distribution includes three separate, query-generating tools
5053 that may be of use to you.
5056 <title>Prefix Query Format</title>
5058 Since RPN or reverse polish notation is really just a fancy way of
5059 describing a suffix notation format (operator follows operands), it
5060 would seem that the confusion is total when we now introduce a prefix
5061 notation for RPN. The reason is one of simple laziness - it's somewhat
5062 simpler to interpret a prefix format, and this utility was designed
5063 for maximum simplicity, to provide a baseline representation for use
5064 in simple test applications and scripting environments (like Tcl). The
5065 demonstration client included with YAZ uses the PQF.
5069 The PQF have been adopted by other parties developing Z39.50
5070 software. It is often referred to as Prefix Query Notation
5075 The PQF is defined by the pquery module in the YAZ library.
5076 There are two sets of function that have similar behavior. First
5077 set operates on a PQF parser handle, second set doesn't. First set
5078 set of functions are more flexible than the second set. Second set
5079 is obsolete and is only provided to ensure backwards compatibility.
5082 First set of functions all operate on a PQF parser handle:
5085 #include <yaz/pquery.h>
5087 YAZ_PQF_Parser yaz_pqf_create(void);
5089 void yaz_pqf_destroy(YAZ_PQF_Parser p);
5091 Z_RPNQuery *yaz_pqf_parse(YAZ_PQF_Parser p, ODR o, const char *qbuf);
5093 Z_AttributesPlusTerm *yaz_pqf_scan(YAZ_PQF_Parser p, ODR o,
5094 Odr_oid **attributeSetId, const char *qbuf);
5096 int yaz_pqf_error(YAZ_PQF_Parser p, const char **msg, size_t *off);
5099 A PQF parser is created and destructed by functions
5100 <function>yaz_pqf_create</function> and
5101 <function>yaz_pqf_destroy</function> respectively.
5102 Function <function>yaz_pqf_parse</function> parses query given
5103 by string <literal>qbuf</literal>. If parsing was successful,
5104 a Z39.50 RPN Query is returned which is created using ODR stream
5105 <literal>o</literal>. If parsing failed, a NULL pointer is
5107 Function <function>yaz_pqf_scan</function> takes a scan query in
5108 <literal>qbuf</literal>. If parsing was successful, the function
5109 returns attributes plus term pointer and modifies
5110 <literal>attributeSetId</literal> to hold attribute set for the
5111 scan request - both allocated using ODR stream <literal>o</literal>.
5112 If parsing failed, yaz_pqf_scan returns a NULL pointer.
5113 Error information for bad queries can be obtained by a call to
5114 <function>yaz_pqf_error</function> which returns an error code and
5115 modifies <literal>*msg</literal> to point to an error description,
5116 and modifies <literal>*off</literal> to the offset within last
5117 query were parsing failed.
5120 The second set of functions are declared as follows:
5123 #include <yaz/pquery.h>
5125 Z_RPNQuery *p_query_rpn(ODR o, oid_proto proto, const char *qbuf);
5127 Z_AttributesPlusTerm *p_query_scan(ODR o, oid_proto proto,
5128 Odr_oid **attributeSetP, const char *qbuf);
5130 int p_query_attset(const char *arg);
5133 The function <function>p_query_rpn()</function> takes as arguments an
5134 &odr; stream (see section <link linkend="odr">The ODR Module</link>)
5135 to provide a memory source (the structure created is released on
5136 the next call to <function>odr_reset()</function> on the stream), a
5137 protocol identifier (one of the constants <token>PROTO_Z3950</token> and
5138 <token>PROTO_SR</token>), an attribute set reference, and
5139 finally a null-terminated string holding the query string.
5142 If the parse went well, <function>p_query_rpn()</function> returns a
5143 pointer to a <literal>Z_RPNQuery</literal> structure which can be
5144 placed directly into a <literal>Z_SearchRequest</literal>.
5145 If parsing failed, due to syntax error, a NULL pointer is returned.
5148 The <literal>p_query_attset</literal> specifies which attribute set
5149 to use if the query doesn't specify one by the
5150 <literal>@attrset</literal> operator.
5151 The <literal>p_query_attset</literal> returns 0 if the argument is a
5152 valid attribute set specifier; otherwise the function returns -1.
5155 The grammar of the PQF is as follows:
5158 query ::= top-set query-struct.
5160 top-set ::= [ '@attrset' string ]
5162 query-struct ::= attr-spec | simple | complex | '@term' term-type query
5164 attr-spec ::= '@attr' [ string ] string query-struct
5166 complex ::= operator query-struct query-struct.
5168 operator ::= '@and' | '@or' | '@not' | '@prox' proximity.
5170 simple ::= result-set | term.
5172 result-set ::= '@set' string.
5176 proximity ::= exclusion distance ordered relation which-code unit-code.
5178 exclusion ::= '1' | '0' | 'void'.
5180 distance ::= integer.
5182 ordered ::= '1' | '0'.
5184 relation ::= integer.
5186 which-code ::= 'known' | 'private' | integer.
5188 unit-code ::= integer.
5190 term-type ::= 'general' | 'numeric' | 'string' | 'oid' | 'datetime' | 'null'.
5193 You will note that the syntax above is a fairly faithful
5194 representation of RPN, except for the Attribute, which has been
5195 moved a step away from the term, allowing you to associate one or more
5196 attributes with an entire query structure. The parser will
5197 automatically apply the given attributes to each term as required.
5200 The @attr operator is followed by an attribute specification
5201 (<literal>attr-spec</literal> above). The specification consists
5202 of an optional attribute set, an attribute type-value pair and
5203 a sub-query. The attribute type-value pair is packed in one string:
5204 an attribute type, an equals sign, and an attribute value, like this:
5205 <literal>@attr 1=1003</literal>.
5206 The type is always an integer but the value may be either an
5207 integer or a string (if it doesn't start with a digit character).
5208 A string attribute-value is encoded as a Type-1 ``complex''
5209 attribute with the list of values containing the single string
5210 specified, and including no semantic indicators.
5213 Version 3 of the Z39.50 specification defines various encoding of terms.
5214 Use <literal>@term </literal> <replaceable>type</replaceable>
5215 <replaceable>string</replaceable>,
5216 where type is one of: <literal>general</literal>,
5217 <literal>numeric</literal> or <literal>string</literal>
5218 (for InternationalString).
5219 If no term type has been given, the <literal>general</literal> form
5220 is used. This is the only encoding allowed in both versions 2 and 3
5221 of the Z39.50 standard.
5223 <sect3 id="PQF-prox">
5224 <title>Using Proximity Operators with PQF</title>
5227 This is an advanced topic, describing how to construct
5228 queries that make very specific requirements on the
5229 relative location of their operands.
5230 You may wish to skip this section and go straight to
5231 <link linkend="pqf-examples">the example PQF queries</link>.
5236 Most Z39.50 servers do not support proximity searching, or
5237 support only a small subset of the full functionality that
5238 can be expressed using the PQF proximity operator. Be
5239 aware that the ability to <emphasis>express</emphasis> a
5240 query in PQF is no guarantee that any given server will
5241 be able to <emphasis>execute</emphasis> it.
5247 The proximity operator <literal>@prox</literal> is a special
5248 and more restrictive version of the conjunction operator
5249 <literal>@and</literal>. Its semantics are described in
5250 section 3.7.2 (Proximity) of Z39.50 the standard itself, which
5251 can be read on-line at
5252 <ulink url="&url.z39.50.proximity;"/>
5255 In PQF, the proximity operation is represented by a sequence
5258 @prox <replaceable>exclusion</replaceable> <replaceable>distance</replaceable> <replaceable>ordered</replaceable> <replaceable>relation</replaceable> <replaceable>which-code</replaceable> <replaceable>unit-code</replaceable>
5260 in which the meanings of the parameters are as described in in
5261 the standard, and they can take the following values:
5264 <formalpara><title>exclusion</title>
5266 0 = false (i.e. the proximity condition specified by the
5267 remaining parameters must be satisfied) or
5268 1 = true (the proximity condition specified by the
5269 remaining parameters must <emphasis>not</emphasis> be
5275 <formalpara><title>distance</title><para>
5276 An integer specifying the difference between the locations
5277 of the operands: e.g. two adjacent words would have
5278 distance=1 since their locations differ by one unit.
5280 </formalpara></listitem>
5282 <formalpara><title>ordered</title><para>
5283 1 = ordered (the operands must occur in the order the
5284 query specifies them) or
5285 0 = unordered (they may appear in either order).
5290 <formalpara><title>relation</title><para>
5291 Recognised values are
5293 2 (lessThanOrEqual),
5295 4 (greaterThanOrEqual),
5302 <formalpara><title>which-code</title><para>
5303 <literal>known</literal>
5305 <literal>k</literal>
5306 (the unit-code parameter is taken from the well-known list
5307 of alternatives described in below) or
5308 <literal>private</literal>
5310 <literal>p</literal>
5311 (the unit-code paramater has semantics specific to an
5312 out-of-band agreement such as a profile).
5317 <formalpara><title>unit-code</title><para>
5318 If the which-code parameter is <literal>known</literal>
5319 then the recognised values are
5329 10 (elementType) and
5331 If which-code is <literal>private</literal> then the
5332 acceptable values are determined by the profile.
5337 (The numeric values of the relation and well-known unit-code
5338 parameters are taken straight from
5339 <ulink url="&url.z39.50.proximity.asn1;"
5340 >the ASN.1</ulink> of the proximity structure in the standard.)
5343 <sect3 id="pqf-examples">
5344 <title>PQF queries</title>
5345 <example id="example.pqf.simple.terms">
5346 <title>PQF queries using simple terms</title>
5355 <example id="pqf.example.pqf.boolean.operators">
5356 <title>PQF boolean operators</title>
5359 @or "dylan" "zimmerman"
5361 @and @or dylan zimmerman when
5363 @and when @or dylan zimmerman
5367 <example id="example.pqf.result.sets">
5368 <title>PQF references to result sets</title>
5373 @and @set seta @set setb
5377 <example id="example.pqf.attributes">
5378 <title>Attributes for terms</title>
5383 @attr 1=4 @attr 4=1 "self portrait"
5385 @attrset exp1 @attr 1=1 CategoryList
5387 @attr gils 1=2008 Copenhagen
5389 @attr 1=/book/title computer
5393 <example id="example.pqf.proximity">
5394 <title>PQF Proximity queries</title>
5397 @prox 0 3 1 2 k 2 dylan zimmerman
5399 Here the parameters 0, 3, 1, 2, k and 2 represent exclusion,
5400 distance, ordered, relation, which-code and unit-code, in that
5404 <para>exclusion = 0: the proximity condition must hold</para>
5407 <para>distance = 3: the terms must be three units apart</para>
5411 ordered = 1: they must occur in the order they are specified
5416 relation = 2: lessThanOrEqual (to the distance of 3 units)
5421 which-code is ``known'', so the standard unit-codes are used
5425 <para>unit-code = 2: word.</para>
5428 So the whole proximity query means that the words
5429 <literal>dylan</literal> and <literal>zimmerman</literal> must
5430 both occur in the record, in that order, differing in position
5431 by three or fewer words (i.e. with two or fewer words between
5432 them.) The query would find ``Bob Dylan, aka. Robert
5433 Zimmerman'', but not ``Bob Dylan, born as Robert Zimmerman''
5434 since the distance in this case is four.
5437 <example id="example.pqf.search.term.type">
5438 <title>PQF specification of search term type</title>
5441 @term string "a UTF-8 string, maybe?"
5445 <example id="example.pqf.mixed.queries">
5446 <title>PQF mixed queries</title>
5449 @or @and bob dylan @set Result-1
5451 @attr 4=1 @and @attr 1=1 "bob dylan" @attr 1=4 "slow train coming"
5453 @and @attr 2=4 @attr gils 1=2038 -114 @attr 2=2 @attr gils 1=2039 -109
5455 The last of these examples is a spatial search: in
5456 <ulink url="http://www.gils.net/prof_v2.html#sec_7_4"
5457 >the GILS attribute set</ulink>,
5459 2038 indicates West Bounding Coordinate and
5460 2030 indicates East Bounding Coordinate,
5461 so the query is for areas extending from -114 degrees
5462 to no more than -109 degrees.
5467 <sect2 id="CCL"><title>CCL</title>
5469 Not all users enjoy typing in prefix query structures and numerical
5470 attribute values, even in a minimalistic test client. In the library
5471 world, the more intuitive Common Command Language - CCL (ISO 8777)
5472 has enjoyed some popularity - especially before the widespread
5473 availability of graphical interfaces. It is still useful in
5474 applications where you for some reason or other need to provide a
5475 symbolic language for expressing boolean query structures.
5477 <sect3 id="ccl.syntax">
5478 <title>CCL Syntax</title>
5480 The CCL parser obeys the following grammar for the FIND argument.
5481 The syntax is annotated by in the lines prefixed by
5482 <literal>--</literal>.
5485 CCL-Find ::= CCL-Find Op Elements
5488 Op ::= "and" | "or" | "not"
5489 -- The above means that Elements are separated by boolean operators.
5491 Elements ::= '(' CCL-Find ')'
5494 | Qualifiers Relation Terms
5495 | Qualifiers Relation '(' CCL-Find ')'
5496 | Qualifiers '=' string '-' string
5497 -- Elements is either a recursive definition, a result set reference, a
5498 -- list of terms, qualifiers followed by terms, qualifiers followed
5499 -- by a recursive definition or qualifiers in a range (lower - upper).
5501 Set ::= 'set' = string
5502 -- Reference to a result set
5504 Terms ::= Terms Prox Term
5506 -- Proximity of terms.
5508 Term ::= Term string
5510 -- This basically means that a term may include a blank
5512 Qualifiers ::= Qualifiers ',' string
5514 -- Qualifiers is a list of strings separated by comma
5516 Relation ::= '=' | '>=' | '<=' | '<>' | '>' | '<'
5517 -- Relational operators. This really doesn't follow the ISO8777
5521 -- Proximity operator
5524 <example id="example.ccl.queries">
5525 <title>CCL queries</title>
5527 The following queries are all valid:
5538 (dylan and bob) or set=1
5547 Assuming that the qualifiers <literal>ti</literal>,
5548 <literal>au</literal>
5549 and <literal>date</literal> are defined we may use:
5554 au=(bob dylan and slow train coming)
5556 date>1980 and (ti=((self portrait)))
5560 <sect3 id="ccl.qualifiers">
5561 <title>CCL Qualifiers</title>
5563 Qualifiers are used to direct the search to a particular searchable
5564 index, such as title (ti) and author indexes (au). The CCL standard
5565 itself doesn't specify a particular set of qualifiers, but it does
5566 suggest a few short-hand notations. You can customize the CCL parser
5567 to support a particular set of qualifiers to reflect the current target
5568 profile. Traditionally, a qualifier would map to a particular
5569 use-attribute within the BIB-1 attribute set. It is also
5570 possible to set other attributes, such as the structure
5574 A CCL profile is a set of predefined CCL qualifiers that may be
5575 read from a file or set in the CCL API.
5576 The YAZ client reads its CCL qualifiers from a file named
5577 <filename>default.bib</filename>. There are four types of
5578 lines in a CCL profile: qualifier specification,
5579 qualifier alias, comments and directives.
5581 <sect4 id="ccl.qualifier.specification">
5582 <title>Qualifier specification</title>
5584 A qualifier specification is of the form:
5587 <replaceable>qualifier-name</replaceable>
5588 [<replaceable>attributeset</replaceable><literal>,</literal>]<replaceable>type</replaceable><literal>=</literal><replaceable>val</replaceable>
5589 [<replaceable>attributeset</replaceable><literal>,</literal>]<replaceable>type</replaceable><literal>=</literal><replaceable>val</replaceable> ...
5592 where <replaceable>qualifier-name</replaceable> is the name of the
5593 qualifier to be used (eg. <literal>ti</literal>),
5594 <replaceable>type</replaceable> is attribute type in the attribute
5595 set (Bib-1 is used if no attribute set is given) and
5596 <replaceable>val</replaceable> is attribute value.
5597 The <replaceable>type</replaceable> can be specified as an
5598 integer or as it be specified either as a single-letter:
5599 <literal>u</literal> for use,
5600 <literal>r</literal> for relation,<literal>p</literal> for position,
5601 <literal>s</literal> for structure,<literal>t</literal> for truncation
5602 or <literal>c</literal> for completeness.
5603 The attributes for the special qualifier name <literal>term</literal>
5604 are used when no CCL qualifier is given in a query.
5605 <table id="ccl.common.bib1.attributes">
5606 <title>Common Bib-1 attributes</title>
5608 <colspec colwidth="2*" colname="type"></colspec>
5609 <colspec colwidth="9*" colname="description"></colspec>
5613 <entry>Description</entry>
5618 <entry><literal>u=</literal><replaceable>value</replaceable></entry>
5620 Use attribute (1). Common use attributes are
5621 1 Personal-name, 4 Title, 7 ISBN, 8 ISSN, 30 Date,
5622 62 Subject, 1003 Author), 1016 Any. Specify value
5627 <entry><literal>r=</literal><replaceable>value</replaceable></entry>
5629 Relation attribute (2). Common values are
5630 1 <, 2 <=, 3 =, 4 >=, 5 >, 6 <>,
5631 100 phonetic, 101 stem, 102 relevance, 103 always matches.
5635 <entry><literal>p=</literal><replaceable>value</replaceable></entry>
5637 Position attribute (3). Values: 1 first in field, 2
5638 first in any subfield, 3 any position in field.
5642 <entry><literal>s=</literal><replaceable>value</replaceable></entry>
5644 Structure attribute (4). Values: 1 phrase, 2 word,
5645 3 key, 4 year, 5 date, 6 word list, 100 date (un),
5646 101 name (norm), 102 name (un), 103 structure, 104 urx,
5647 105 free-form-text, 106 document-text, 107 local-number,
5648 108 string, 109 numeric string.
5652 <entry><literal>t=</literal><replaceable>value</replaceable></entry>
5654 Truncation attribute (5). Values: 1 right, 2 left,
5655 3 left& right, 100 none, 101 process #, 102 regular-1,
5656 103 regular-2, 104 CCL.
5660 <entry><literal>c=</literal><replaceable>value</replaceable></entry>
5662 Completeness attribute (6). Values: 1 incomplete subfield,
5663 2 complete subfield, 3 complete field.
5671 Refer to <xref linkend="bib1"/> or the complete
5672 <ulink url="&url.z39.50.attset.bib1;">list of Bib-1 attributes</ulink>
5675 It is also possible to specify non-numeric attribute values,
5676 which are used in combination with certain types.
5677 The special combinations are:
5678 <table id="ccl.special.attribute.combos">
5679 <title>Special attribute combos</title>
5681 <colspec colwidth="2*" colname="name"></colspec>
5682 <colspec colwidth="9*" colname="description"></colspec>
5686 <entry>Description</entry>
5691 <entry><literal>s=pw</literal></entry>
5693 The structure is set to either word or phrase depending
5694 on the number of tokens in a term (phrase-word).
5698 <entry><literal>s=al</literal></entry>
5700 Each token in the term is ANDed. (and-list).
5701 This does not set the structure at all.
5704 <row><entry><literal>s=ol</literal></entry>
5706 Each token in the term is ORed. (or-list).
5707 This does not set the structure at all.
5710 <row><entry><literal>s=ag</literal></entry>
5712 Tokens that appears as phrases (with blank in them) gets
5713 structure phrase attached (4=1). Tokens that appear to be words
5714 gets structure word attached (4=2). Phrases and words are
5715 ANDed. This is a variant of s=al and s=pw, with the main
5716 difference that words are not split (with operator AND)
5717 but instead kept in one RPN token. This facility appeared
5721 <row><entry><literal>r=o</literal></entry>
5723 Allows ranges and the operators greather-than, less-than, ...
5725 This sets Bib-1 relation attribute accordingly (relation
5726 ordered). A query construct is only treated as a range if
5727 dash is used and that is surrounded by white-space. So
5728 <literal>-1980</literal> is treated as term
5729 <literal>"-1980"</literal> not <literal><= 1980</literal>.
5730 If <literal>- 1980</literal> is used, however, that is
5734 <row><entry><literal>r=r</literal></entry>
5736 Similar to <literal>r=o</literal> but assumes that terms
5737 are non-negative (not prefixed with <literal>-</literal>).
5738 Thus, a dash will always be treated as a range.
5739 The construct <literal>1980-1990</literal> is
5740 treated as a range with <literal>r=r</literal> but as a
5741 single term <literal>"1980-1990"</literal> with
5742 <literal>r=o</literal>. The special attribute
5743 <literal>r=r</literal> is available in YAZ 2.0.24 or later.
5746 <row><entry><literal>r=omiteq</literal></entry>
5748 This will omit relation=equals (@attr 2=3) when r=o / r=r
5749 is used. This is useful for servers that somehow breaks
5750 when an explicit relation=equals is used. Omitting the
5751 relation is usually safe because "equals" is the default
5752 behavior. This tweak was added in YAZ version 5.1.2.
5755 <row><entry><literal>t=l</literal></entry>
5757 Allows term to be left-truncated.
5758 If term is of the form <literal>?x</literal>, the resulting
5759 Type-1 term is <literal>x</literal> and truncation is left.
5762 <row><entry><literal>t=r</literal></entry>
5764 Allows term to be right-truncated.
5765 If term is of the form <literal>x?</literal>, the resulting
5766 Type-1 term is <literal>x</literal> and truncation is right.
5769 <row><entry><literal>t=n</literal></entry>
5771 If term is does not include <literal>?</literal>, the
5772 truncation attribute is set to none (100).
5775 <row><entry><literal>t=b</literal></entry>
5777 Allows term to be both left&right truncated.
5778 If term is of the form <literal>?x?</literal>, the
5779 resulting term is <literal>x</literal> and trunctation is
5780 set to both left&right.
5783 <row><entry><literal>t=x</literal></entry>
5785 Allows masking anywhere in a term, thus fully supporting
5786 # (mask one character) and ? (zero or more of any).
5787 If masking is used, trunction is set to 102 (regexp-1 in term)
5788 and the term is converted accordingly to a regular expression.
5791 <row><entry><literal>t=z</literal></entry>
5793 Allows masking anywhere in a term, thus fully supporting
5794 # (mask one character) and ? (zero or more of any).
5795 If masking is used, trunction is set to 104 (Z39.58 in term)
5796 and the term is converted accordingly to Z39.58 masking term -
5797 actually the same truncation as CCL itself.
5804 <example id="example.ccl.profile">
5805 <title>CCL profile</title>
5807 Consider the following definition:
5817 <literal>ti</literal> and <literal>au</literal> both set
5818 structure attribute to phrase (s=1).
5819 <literal>ti</literal>
5820 sets the use-attribute to 4. <literal>au</literal> sets the
5822 When no qualifiers are used in the query the structure-attribute is
5823 set to free-form-text (105) (rule for <literal>term</literal>).
5824 The <literal>date</literal> sets the relation attribute to
5825 the relation used in the CCL query and sets the use attribute
5829 You can combine attributes. To Search for "ranked title" you
5832 ti,ranked=knuth computer
5834 which will set relation=ranked, use=title, structure=phrase.
5841 is a valid query. But
5849 <sect4 id="ccl.qualifier.alias">
5850 <title>Qualifier alias</title>
5852 A qualifier alias is of the form:
5855 <replaceable>q</replaceable>
5856 <replaceable>q1</replaceable> <replaceable>q2</replaceable> ..
5859 which declares <replaceable>q</replaceable> to
5860 be an alias for <replaceable>q1</replaceable>,
5861 <replaceable>q2</replaceable>... such that the CCL
5862 query <replaceable>q=x</replaceable> is equivalent to
5863 <replaceable>q1=x or q2=x or ...</replaceable>.
5866 <sect4 id="ccl.comments">
5867 <title>Comments</title>
5869 Lines with white space or lines that begin with
5870 character <literal>#</literal> are treated as comments.
5873 <sect4 id="ccl.directives">
5874 <title>Directives</title>
5876 Directive specifications takes the form
5878 <para><literal>@</literal><replaceable>directive</replaceable> <replaceable>value</replaceable>
5880 <table id="ccl.directives.table">
5881 <title>CCL directives</title>
5883 <colspec colwidth="2*" colname="name"></colspec>
5884 <colspec colwidth="8*" colname="description"></colspec>
5885 <colspec colwidth="1*" colname="default"></colspec>
5889 <entry>Description</entry>
5890 <entry>Default</entry>
5895 <entry>truncation</entry>
5896 <entry>Truncation character</entry>
5897 <entry><literal>?</literal></entry>
5901 <entry>Masking character. Requires YAZ 4.2.58 or later</entry>
5902 <entry><literal>#</literal></entry>
5905 <entry>field</entry>
5906 <entry>Specifies how multiple fields are to be
5907 combined. There are two modes: <literal>or</literal>:
5908 multiple qualifier fields are ORed,
5909 <literal>merge</literal>: attributes for the qualifier
5910 fields are merged and assigned to one term.
5912 <entry><literal>merge</literal></entry>
5916 <entry>Specifies if CCL operators and qualifiers should be
5917 compared with case sensitivity or not. Specify 1 for
5918 case sensitive; 0 for case insensitive.</entry>
5919 <entry><literal>1</literal></entry>
5923 <entry>Specifies token for CCL operator AND.</entry>
5924 <entry><literal>and</literal></entry>
5928 <entry>Specifies token for CCL operator OR.</entry>
5929 <entry><literal>or</literal></entry>
5933 <entry>Specifies token for CCL operator NOT.</entry>
5934 <entry><literal>not</literal></entry>
5938 <entry>Specifies token for CCL operator SET.</entry>
5939 <entry><literal>set</literal></entry>
5946 <sect3 id="ccl.api">
5947 <title>CCL API</title>
5949 All public definitions can be found in the header file
5950 <filename>ccl.h</filename>. A profile identifier is of type
5951 <literal>CCL_bibset</literal>. A profile must be created with the call
5952 to the function <function>ccl_qual_mk</function> which returns a profile
5953 handle of type <literal>CCL_bibset</literal>.
5956 To read a file containing qualifier definitions the function
5957 <function>ccl_qual_file</function> may be convenient. This function
5958 takes an already opened <literal>FILE</literal> handle pointer as
5959 argument along with a <literal>CCL_bibset</literal> handle.
5962 To parse a simple string with a FIND query use the function
5965 struct ccl_rpn_node *ccl_find_str(CCL_bibset bibset, const char *str,
5966 int *error, int *pos);
5969 which takes the CCL profile (<literal>bibset</literal>) and query
5970 (<literal>str</literal>) as input. Upon successful completion the RPN
5971 tree is returned. If an error occur, such as a syntax error, the integer
5972 pointed to by <literal>error</literal> holds the error code and
5973 <literal>pos</literal> holds the offset inside query string in which
5977 An English representation of the error may be obtained by calling
5978 the <literal>ccl_err_msg</literal> function. The error codes are
5979 listed in <filename>ccl.h</filename>.
5982 To convert the CCL RPN tree (type
5983 <literal>struct ccl_rpn_node *</literal>)
5984 to the Z_RPNQuery of YAZ the function <function>ccl_rpn_query</function>
5985 must be used. This function which is part of YAZ is implemented in
5986 <filename>yaz-ccl.c</filename>.
5987 After calling this function the CCL RPN tree is probably no longer
5988 needed. The <literal>ccl_rpn_delete</literal> destroys the CCL RPN tree.
5991 A CCL profile may be destroyed by calling the
5992 <function>ccl_qual_rm</function> function.
5995 The token names for the CCL operators may be changed by setting the
5996 globals (all type <literal>char *</literal>)
5997 <literal>ccl_token_and</literal>, <literal>ccl_token_or</literal>,
5998 <literal>ccl_token_not</literal> and <literal>ccl_token_set</literal>.
5999 An operator may have aliases, i.e. there may be more than one name for
6000 the operator. To do this, separate each alias with a space character.
6007 <ulink url="&url.cql;">CQL</ulink>
6008 - Common Query Language - was defined for the
6009 <ulink url="&url.sru;">SRU</ulink> protocol.
6010 In many ways CQL has a similar syntax to CCL.
6011 The objective of CQL is different. Where CCL aims to be
6012 an end-user language, CQL is <emphasis>the</emphasis> protocol
6013 query language for SRU.
6017 If you are new to CQL, read the
6018 <ulink url="&url.cql.intro;">Gentle Introduction</ulink>.
6022 The CQL parser in &yaz; provides the following:
6026 It parses and validates a CQL query.
6031 It generates a C structure that allows you to convert
6032 a CQL query to some other query language, such as SQL.
6037 The parser converts a valid CQL query to PQF, thus providing a
6038 way to use CQL for both SRU servers and Z39.50 targets at the
6044 The parser converts CQL to XCQL.
6045 XCQL is an XML representation of CQL.
6046 XCQL is part of the SRU specification. However, since SRU
6047 supports CQL only, we don't expect XCQL to be widely used.
6048 Furthermore, CQL has the advantage over XCQL that it is
6054 <sect3 id="cql.parsing">
6055 <title>CQL parsing</title>
6057 A CQL parser is represented by the <literal>CQL_parser</literal>
6058 handle. Its contents should be considered &yaz; internal (private).
6060 #include <yaz/cql.h>
6062 typedef struct cql_parser *CQL_parser;
6064 CQL_parser cql_parser_create(void);
6065 void cql_parser_destroy(CQL_parser cp);
6067 A parser is created by <function>cql_parser_create</function> and
6068 is destroyed by <function>cql_parser_destroy</function>.
6071 To parse a CQL query string, the following function
6074 int cql_parser_string(CQL_parser cp, const char *str);
6076 A CQL query is parsed by the <function>cql_parser_string</function>
6077 which takes a query <parameter>str</parameter>.
6078 If the query was valid (no syntax errors), then zero is returned;
6079 otherwise -1 is returned to indicate a syntax error.
6083 int cql_parser_stream(CQL_parser cp,
6084 int (*getbyte)(void *client_data),
6085 void (*ungetbyte)(int b, void *client_data),
6088 int cql_parser_stdio(CQL_parser cp, FILE *f);
6090 The functions <function>cql_parser_stream</function> and
6091 <function>cql_parser_stdio</function> parses a CQL query
6092 - just like <function>cql_parser_string</function>.
6093 The only difference is that the CQL query can be
6094 fed to the parser in different ways.
6095 The <function>cql_parser_stream</function> uses a generic
6096 byte stream as input. The <function>cql_parser_stdio</function>
6097 uses a <literal>FILE</literal> handle which is opened for reading.
6100 <sect3 id="cql.tree">
6101 <title>CQL tree</title>
6103 The the query string is valid, the CQL parser
6104 generates a tree representing the structure of the
6109 struct cql_node *cql_parser_result(CQL_parser cp);
6111 <function>cql_parser_result</function> returns the
6112 a pointer to the root node of the resulting tree.
6115 Each node in a CQL tree is represented by a
6116 <literal>struct cql_node</literal>.
6117 It is defined as follows:
6119 #define CQL_NODE_ST 1
6120 #define CQL_NODE_BOOL 2
6121 #define CQL_NODE_SORT 3
6131 struct cql_node *modifiers;
6135 struct cql_node *left;
6136 struct cql_node *right;
6137 struct cql_node *modifiers;
6141 struct cql_node *next;
6142 struct cql_node *modifiers;
6143 struct cql_node *search;
6148 There are three node types: search term (ST), boolean (BOOL)
6150 A modifier is treated as a search term too.
6153 The search term node has five members:
6157 <literal>index</literal>: index for search term.
6158 If an index is unspecified for a search term,
6159 <literal>index</literal> will be NULL.
6164 <literal>index_uri</literal>: index URi for search term
6165 or NULL if none could be resolved for the index.
6170 <literal>term</literal>: the search term itself.
6175 <literal>relation</literal>: relation for search term.
6180 <literal>relation_uri</literal>: relation URI for search term.
6185 <literal>modifiers</literal>: relation modifiers for search
6186 term. The <literal>modifiers</literal> list itself of cql_nodes
6187 each of type <literal>ST</literal>.
6193 The boolean node represents <literal>and</literal>,
6194 <literal>or</literal>, <literal>not</literal> +
6199 <literal>left</literal> and <literal>right</literal>: left
6200 - and right operand respectively.
6205 <literal>modifiers</literal>: proximity arguments.
6211 The sort node represents both the SORTBY clause.
6214 <sect3 id="cql.to.pqf">
6215 <title>CQL to PQF conversion</title>
6217 Conversion to PQF (and Z39.50 RPN) is tricky by the fact
6218 that the resulting RPN depends on the Z39.50 target
6219 capabilities (combinations of supported attributes).
6220 In addition, the CQL and SRU operates on index prefixes
6221 (URI or strings), whereas the RPN uses Object Identifiers
6225 The CQL library of &yaz; defines a <literal>cql_transform_t</literal>
6226 type. It represents a particular mapping between CQL and RPN.
6227 This handle is created and destroyed by the functions:
6229 cql_transform_t cql_transform_open_FILE (FILE *f);
6230 cql_transform_t cql_transform_open_fname(const char *fname);
6231 void cql_transform_close(cql_transform_t ct);
6233 The first two functions create a tranformation handle from
6234 either an already open FILE or from a filename respectively.
6237 The handle is destroyed by <function>cql_transform_close</function>
6238 in which case no further reference of the handle is allowed.
6241 When a <literal>cql_transform_t</literal> handle has been created
6242 you can convert to RPN.
6244 int cql_transform_buf(cql_transform_t ct,
6245 struct cql_node *cn, char *out, int max);
6247 This function converts the CQL tree <literal>cn</literal>
6248 using handle <literal>ct</literal>.
6249 For the resulting PQF, you supply a buffer <literal>out</literal>
6250 which must be able to hold at at least <literal>max</literal>
6254 If conversion failed, <function>cql_transform_buf</function>
6255 returns a non-zero SRU error code; otherwise zero is returned
6256 (conversion successful). The meanings of the numeric error
6257 codes are listed in the SRU specification somewhere (no
6258 direct link anymore).
6261 If conversion fails, more information can be obtained by calling
6263 int cql_transform_error(cql_transform_t ct, char **addinfop);
6265 This function returns the most recently returned numeric
6266 error-code and sets the string-pointer at
6267 <literal>*addinfop</literal> to point to a string containing
6268 additional information about the error that occurred: for
6269 example, if the error code is 15 (``Illegal or unsupported context
6270 set''), the additional information is the name of the requested
6271 context set that was not recognised.
6274 The SRU error-codes may be translated into brief human-readable
6275 error messages using
6277 const char *cql_strerror(int code);
6281 If you wish to be able to produce a PQF result in a different
6282 way, there are two alternatives.
6284 void cql_transform_pr(cql_transform_t ct,
6285 struct cql_node *cn,
6286 void (*pr)(const char *buf, void *client_data),
6289 int cql_transform_FILE(cql_transform_t ct,
6290 struct cql_node *cn, FILE *f);
6292 The former function produces output to a user-defined
6293 output stream. The latter writes the result to an already
6294 open <literal>FILE</literal>.
6297 <sect3 id="cql.to.rpn">
6298 <title>Specification of CQL to RPN mappings</title>
6300 The file supplied to functions
6301 <function>cql_transform_open_FILE</function>,
6302 <function>cql_transform_open_fname</function> follows
6303 a structure found in many Unix utilities.
6304 It consists of mapping specifications - one per line.
6305 Lines starting with <literal>#</literal> are ignored (comments).
6308 Each line is of the form
6310 <replaceable>CQL pattern</replaceable><literal> = </literal> <replaceable> RPN equivalent</replaceable>
6314 An RPN pattern is a simple attribute list. Each attribute pair
6317 [<replaceable>set</replaceable>] <replaceable>type</replaceable><literal>=</literal><replaceable>value</replaceable>
6319 The attribute <replaceable>set</replaceable> is optional.
6320 The <replaceable>type</replaceable> is the attribute type,
6321 <replaceable>value</replaceable> the attribute value.
6324 The character <literal>*</literal> (asterisk) has special meaning
6325 when used in the RPN pattern.
6326 Each occurrence of <literal>*</literal> is substituted with the
6327 CQL matching name (index, relation, qualifier etc).
6328 This facility can be used to copy a CQL name verbatim to the RPN result.
6331 The following CQL patterns are recognized:
6335 <literal>index.</literal><replaceable>set</replaceable><literal>.</literal><replaceable>name</replaceable>
6339 This pattern is invoked when a CQL index, such as
6340 dc.title is converted. <replaceable>set</replaceable>
6341 and <replaceable>name</replaceable> are the context set and index
6343 Typically, the RPN specifies an equivalent use attribute.
6346 For terms not bound by an index the pattern
6347 <literal>index.cql.serverChoice</literal> is used.
6348 Here, the prefix <literal>cql</literal> is defined as
6349 <literal>http://www.loc.gov/zing/cql/cql-indexes/v1.0/</literal>.
6350 If this pattern is not defined, the mapping will fail.
6354 <literal>index.</literal><replaceable>set</replaceable><literal>.*</literal>
6355 is used when no other index pattern is matched.
6361 <literal>qualifier.</literal><replaceable>set</replaceable><literal>.</literal><replaceable>name</replaceable>
6366 For backwards compatibility, this is recognised as a synonym of
6367 <literal>index.</literal><replaceable>set</replaceable><literal>.</literal><replaceable>name</replaceable>
6373 <literal>relation.</literal><replaceable>relation</replaceable>
6377 This pattern specifies how a CQL relation is mapped to RPN.
6378 <replaceable>pattern</replaceable> is name of relation
6379 operator. Since <literal>=</literal> is used as
6380 separator between CQL pattern and RPN, CQL relations
6381 including <literal>=</literal> cannot be
6382 used directly. To avoid a conflict, the names
6383 <literal>ge</literal>,
6384 <literal>eq</literal>,
6385 <literal>le</literal>,
6386 must be used for CQL operators, greater-than-or-equal,
6387 equal, less-than-or-equal respectively.
6388 The RPN pattern is supposed to include a relation attribute.
6391 For terms not bound by a relation, the pattern
6392 <literal>relation.scr</literal> is used. If the pattern
6393 is not defined, the mapping will fail.
6396 The special pattern, <literal>relation.*</literal> is used
6397 when no other relation pattern is matched.
6403 <literal>relationModifier.</literal><replaceable>mod</replaceable>
6407 This pattern specifies how a CQL relation modifier is mapped to RPN.
6408 The RPN pattern is usually a relation attribute.
6414 <literal>structure.</literal><replaceable>type</replaceable>
6418 This pattern specifies how a CQL structure is mapped to RPN.
6419 Note that this CQL pattern is somewhat to similar to
6420 CQL pattern <literal>relation</literal>.
6421 The <replaceable>type</replaceable> is a CQL relation.
6424 The pattern, <literal>structure.*</literal> is used
6425 when no other structure pattern is matched.
6426 Usually, the RPN equivalent specifies a structure attribute.
6432 <literal>position.</literal><replaceable>type</replaceable>
6436 This pattern specifies how the anchor (position) of
6437 CQL is mapped to RPN.
6438 The <replaceable>type</replaceable> is one
6439 of <literal>first</literal>, <literal>any</literal>,
6440 <literal>last</literal>, <literal>firstAndLast</literal>.
6443 The pattern, <literal>position.*</literal> is used
6444 when no other position pattern is matched.
6450 <literal>set.</literal><replaceable>prefix</replaceable>
6454 This specification defines a CQL context set for a given prefix.
6455 The value on the right hand side is the URI for the set -
6456 <emphasis>not</emphasis> RPN. All prefixes used in
6457 index patterns must be defined this way.
6463 <literal>set</literal>
6467 This specification defines a default CQL context set for index names.
6468 The value on the right hand side is the URI for the set.
6474 <example id="example.cql.to.rpn.mapping">
6475 <title>CQL to RPN mapping file</title>
6477 This simple file defines two context sets, three indexes and three
6478 relations, a position pattern and a default structure.
6480 <programlisting><![CDATA[
6481 set.cql = http://www.loc.gov/zing/cql/context-sets/cql/v1.1/
6482 set.dc = http://www.loc.gov/zing/cql/dc-indexes/v1.0/
6484 index.cql.serverChoice = 1=1016
6485 index.dc.title = 1=4
6486 index.dc.subject = 1=21
6492 position.any = 3=3 6=1
6498 With the mappings above, the CQL query
6502 is converted to the PQF:
6504 @attr 1=1016 @attr 2=3 @attr 4=1 @attr 3=3 @attr 6=1 "computer"
6506 by rules <literal>index.cql.serverChoice</literal>,
6507 <literal>relation.scr</literal>, <literal>structure.*</literal>,
6508 <literal>position.any</literal>.
6515 is rejected, since <literal>position.right</literal> is
6521 >my = "http://www.loc.gov/zing/cql/dc-indexes/v1.0/" my.title = x
6525 @attr 1=4 @attr 2=3 @attr 4=1 @attr 3=3 @attr 6=1 "x"
6529 <example id="example.cql.to.rpn.string">
6530 <title>CQL to RPN string attributes</title>
6532 In this example we allow any index to be passed to RPN as
6535 <programlisting><![CDATA[
6536 # Identifiers for prefixes used in this file. (index.*)
6537 set.cql = info:srw/cql-context-set/1/cql-v1.1
6538 set.rpn = http://bogus/rpn
6539 set = http://bogus/rpn
6541 # The default index when none is specified by the query
6542 index.cql.serverChoice = 1=any
6551 The <literal>http://bogus/rpn</literal> context set is also the default
6552 so we can make queries such as
6556 which is converted to
6558 @attr 2=3 @attr 4=1 @attr 3=3 @attr 1=title "a"
6562 <example id="example.cql.to.rpn.bathprofile">
6563 <title>CQL to RPN using Bath Profile</title>
6565 The file <filename>etc/pqf.properties</filename> has mappings from
6566 the Bath Profile and Dublin Core to RPN.
6567 If YAZ is installed as a package it's usually located
6568 in <filename>/usr/share/yaz/etc</filename> and part of the
6569 development package, such as <literal>libyaz-dev</literal>.
6573 <sect3 id="cql.xcql">
6574 <title>CQL to XCQL conversion</title>
6576 Conversion from CQL to XCQL is trivial and does not
6577 require a mapping to be defined.
6578 There three functions to choose from depending on the
6579 way you wish to store the resulting output (XML buffer
6582 int cql_to_xml_buf(struct cql_node *cn, char *out, int max);
6583 void cql_to_xml(struct cql_node *cn,
6584 void (*pr)(const char *buf, void *client_data),
6586 void cql_to_xml_stdio(struct cql_node *cn, FILE *f);
6588 Function <function>cql_to_xml_buf</function> converts
6589 to XCQL and stores result in a user supplied buffer of a given
6593 <function>cql_to_xml</function> writes the result in
6594 a user defined output stream.
6595 <function>cql_to_xml_stdio</function> writes to a
6599 <sect3 id="rpn.to.cql">
6600 <title>PQF to CQL conversion</title>
6602 Conversion from PQF to CQL is offered by the two functions shown
6603 below. The former uses a generic stream for result. The latter
6604 puts result in a WRBUF (string container).
6606 #include <yaz/rpn2cql.h>
6608 int cql_transform_rpn2cql_stream(cql_transform_t ct,
6609 void (*pr)(const char *buf, void *client_data),
6613 int cql_transform_rpn2cql_wrbuf(cql_transform_t ct,
6617 The configuration is the same as used in CQL to PQF conversions.
6622 <sect1 id="tools.oid">
6623 <title>Object Identifiers</title>
6625 The basic YAZ representation of an OID is an array of integers,
6626 terminated with the value -1. This integer is of type
6627 <literal>Odr_oid</literal>.
6630 Fundamental OID operations and the type <literal>Odr_oid</literal>
6631 are defined in <filename>yaz/oid_util.h</filename>.
6634 An OID can either be declared as a automatic variable or it can
6635 allocated using the memory utilities or ODR/NMEM. It's
6636 guaranteed that an OID can fit in <literal>OID_SIZE</literal> integers.
6638 <example id="tools.oid.bib1.1"><title>Create OID on stack</title>
6640 We can create an OID for the Bib-1 attribute set with:
6642 Odr_oid bib1[OID_SIZE];
6654 And OID may also be filled from a string-based representation using
6655 dots (.). This is achieved by function
6657 int oid_dotstring_to_oid(const char *name, Odr_oid *oid);
6659 This functions returns 0 if name could be converted; -1 otherwise.
6661 <example id="tools.oid.bib1.2"><title>Using oid_oiddotstring_to_oid</title>
6663 We can fill the Bib-1 attribute set OID easier with:
6665 Odr_oid bib1[OID_SIZE];
6666 oid_oiddotstring_to_oid("1.2.840.10003.3.1", bib1);
6671 We can also allocate an OID dynamically on a ODR stream with:
6673 Odr_oid *odr_getoidbystr(ODR o, const char *str);
6675 This creates an OID from string-based representation using dots.
6676 This function take an &odr; stream as parameter. This stream is used to
6677 allocate memory for the data elements, which is released on a
6678 subsequent call to <function>odr_reset()</function> on that stream.
6680 <example id="tools.oid.bib1.3">
6681 <title>Using odr_getoidbystr</title>
6683 We can create a OID for the Bib-1 attribute set with:
6685 Odr_oid *bib1 = odr_getoidbystr(odr, "1.2.840.10003.3.1");
6692 char *oid_oid_to_dotstring(const Odr_oid *oid, char *oidbuf)
6694 does the reverse of <function>oid_oiddotstring_to_oid</function>. It
6695 converts an OID to the string-based representation using dots.
6696 The supplied char buffer <literal>oidbuf</literal> holds the resulting
6697 string and must be at least <literal>OID_STR_MAX</literal> in size.
6700 OIDs can be copied with <function>oid_oidcpy</function> which takes
6701 two OID lists as arguments. Alternativly, an OID copy can be allocated
6702 on a ODR stream with:
6704 Odr_oid *odr_oiddup(ODR odr, const Odr_oid *o);
6708 OIDs can be compared with <function>oid_oidcmp</function> which returns
6709 zero if the two OIDs provided are identical; non-zero otherwise.
6711 <sect2 id="tools.oid.database">
6712 <title>OID database</title>
6714 From YAZ version 3 and later, the oident system has been replaced
6715 by an OID database. OID database is a misnomer .. the old odient
6716 system was also a database.
6719 The OID database is really just a map between named Object Identifiers
6720 (string) and their OID raw equivalents. Most operations either
6721 convert from string to OID or other way around.
6724 Unfortunately, whenever we supply a string we must also specify the
6725 <emphasis>OID class</emphasis>. The class is necessary because some
6726 strings correspond to multiple OIDs. An example of such a string is
6727 <literal>Bib-1</literal> which may either be an attribute-set
6728 or a diagnostic-set.
6731 Applications using the YAZ database should include
6732 <filename>yaz/oid_db.h</filename>.
6735 A YAZ database handle is of type <literal>yaz_oid_db_t</literal>.
6736 Actually that's a pointer. You need not think deal with that.
6737 YAZ has a built-in database which can be considered "constant" for
6739 We can get hold that by using function <function>yaz_oid_std</function>.
6742 All functions with prefix <function>yaz_string_to_oid</function>
6743 converts from class + string to OID. We have variants of this
6744 operation due to different memory allocation strategies.
6747 All functions with prefix
6748 <function>yaz_oid_to_string</function> converts from OID to string
6751 <example id="tools.oid.bib1.4">
6752 <title>Create OID with YAZ DB</title>
6754 We can create an OID for the Bib-1 attribute set on the ODR stream
6758 yaz_string_to_oid_odr(yaz_oid_std(), CLASS_ATTSET, "Bib-1", odr);
6760 This is more complex than using <function>odr_getoidbystr</function>.
6761 You would only use <function>yaz_string_to_oid_odr</function> when the
6762 string (here Bib-1) is supplied by a user or configuration.
6766 <sect2 id="tools.oid.std">
6767 <title>Standard OIDs</title>
6769 All the object identifers in the standard OID database as returned
6770 by <function>yaz_oid_std</function> can referenced directly in a
6771 program as a constant OID.
6772 Each constant OID is prefixed with <literal>yaz_oid_</literal> -
6773 followed by OID class (lowercase) - then by OID name (normalized and
6777 See <xref linkend="list-oids"/> for list of all object identifiers
6779 These are declared in <filename>yaz/oid_std.h</filename> but are
6780 included by <filename>yaz/oid_db.h</filename> as well.
6782 <example id="tools.oid.bib1.5">
6783 <title>Use a built-in OID</title>
6785 We can allocate our own OID filled with the constant OID for
6788 Odr_oid *bib1 = odr_oiddup(o, yaz_oid_attset_bib1);
6794 <sect1 id="tools.nmem">
6795 <title>Nibble Memory</title>
6797 Sometimes when you need to allocate and construct a large,
6798 interconnected complex of structures, it can be a bit of a pain to
6799 release the associated memory again. For the structures describing the
6800 Z39.50 PDUs and related structures, it is convenient to use the
6801 memory-management system of the &odr; subsystem (see
6802 <xref linkend="odr.use"/>). However, in some circumstances
6803 where you might otherwise benefit from using a simple nibble memory
6804 management system, it may be impractical to use
6805 <function>odr_malloc()</function> and <function>odr_reset()</function>.
6806 For this purpose, the memory manager which also supports the &odr;
6807 streams is made available in the NMEM module. The external interface
6808 to this module is given in the <filename>nmem.h</filename> file.
6811 The following prototypes are given:
6814 NMEM nmem_create(void);
6815 void nmem_destroy(NMEM n);
6816 void *nmem_malloc(NMEM n, size_t size);
6817 void nmem_reset(NMEM n);
6818 size_t nmem_total(NMEM n);
6819 void nmem_init(void);
6820 void nmem_exit(void);
6823 The <function>nmem_create()</function> function returns a pointer to a
6824 memory control handle, which can be released again by
6825 <function>nmem_destroy()</function> when no longer needed.
6826 The function <function>nmem_malloc()</function> allocates a block of
6827 memory of the requested size. A call to <function>nmem_reset()</function>
6828 or <function>nmem_destroy()</function> will release all memory allocated
6829 on the handle since it was created (or since the last call to
6830 <function>nmem_reset()</function>. The function
6831 <function>nmem_total()</function> returns the number of bytes currently
6832 allocated on the handle.
6835 The nibble memory pool is shared amongst threads. POSIX
6836 mutex'es and WIN32 Critical sections are introduced to keep the
6837 module thread safe. Function <function>nmem_init()</function>
6838 initializes the nibble memory library and it is called automatically
6839 the first time the <literal>YAZ.DLL</literal> is loaded. &yaz; uses
6840 function <function>DllMain</function> to achieve this. You should
6841 <emphasis>not</emphasis> call <function>nmem_init</function> or
6842 <function>nmem_exit</function> unless you're absolute sure what
6843 you're doing. Note that in previous &yaz; versions you'd have to call
6844 <function>nmem_init</function> yourself.
6847 <sect1 id="tools.log">
6850 &yaz; has evolved a fairly complex log system which should be useful both
6851 for debugging &yaz; itself, debugging applications that use &yaz;, and for
6852 production use of those applications.
6855 The log functions are declared in header <filename>yaz/log.h</filename>
6856 and implemented in <filename>src/log.c</filename>.
6857 Due to name clash with syslog and some math utilities the logging
6858 interface has been modified as of YAZ 2.0.29. The obsolete interface
6859 is still available if in header file <filename>yaz/log.h</filename>.
6860 The key points of the interface are:
6863 void yaz_log(int level, const char *fmt, ...)
6864 void yaz_log_init(int level, const char *prefix, const char *name);
6865 void yaz_log_init_file(const char *fname);
6866 void yaz_log_init_level(int level);
6867 void yaz_log_init_prefix(const char *prefix);
6868 void yaz_log_time_format(const char *fmt);
6869 void yaz_log_init_max_size(int mx);
6871 int yaz_log_mask_str(const char *str);
6872 int yaz_log_module_level(const char *name);
6875 The reason for the whole log module is the <function>yaz_log</function>
6876 function. It takes a bitmask indicating the log levels, a
6877 <literal>printf</literal>-like format string, and a variable number of
6881 The <literal>log level</literal> is a bit mask, that says on which level(s)
6882 the log entry should be made, and optionally set some behaviour of the
6883 logging. In the most simple cases, it can be one of <literal>YLOG_FATAL,
6884 YLOG_DEBUG, YLOG_WARN, YLOG_LOG</literal>. Those can be combined with bits
6885 that modify the way the log entry is written:<literal>YLOG_ERRNO,
6886 YLOG_NOTIME, YLOG_FLUSH</literal>.
6887 Most of the rest of the bits are deprecated, and should not be used. Use
6888 the dynamic log levels instead.
6891 Applications that use &yaz;, should not use the LOG_LOG for ordinary
6892 messages, but should make use of the dynamic loglevel system. This consists
6893 of two parts, defining the loglevel and checking it.
6896 To define the log levels, the (main) program should pass a string to
6897 <function>yaz_log_mask_str</function> to define which log levels are to be
6898 logged. This string should be a comma-separated list of log level names,
6899 and can contain both hard-coded names and dynamic ones. The log level
6900 calculation starts with <literal>YLOG_DEFAULT_LEVEL</literal> and adds a bit
6901 for each word it meets, unless the word starts with a '-', in which case it
6902 clears the bit. If the string <literal>'none'</literal> is found,
6903 all bits are cleared. Typically this string comes from the command-line,
6904 often identified by <literal>-v</literal>. The
6905 <function>yaz_log_mask_str</function> returns a log level that should be
6906 passed to <function>yaz_log_init_level</function> for it to take effect.
6909 Each module should check what log bits it should be used, by calling
6910 <function>yaz_log_module_level</function> with a suitable name for the
6911 module. The name is cleared from a preceding path and an extension, if any,
6912 so it is quite possible to use <literal>__FILE__</literal> for it. If the
6913 name has been passed to <function>yaz_log_mask_str</function>, the routine
6914 returns a non-zero bitmask, which should then be used in consequent calls
6915 to yaz_log. (It can also be tested, so as to avoid unnecessary calls to
6916 yaz_log, in time-critical places, or when the log entry would take time
6920 Yaz uses the following dynamic log levels:
6921 <literal>server, session, request, requestdetail</literal> for the server
6923 <literal>zoom</literal> for the zoom client api.
6924 <literal>ztest</literal> for the simple test server.
6925 <literal>malloc, nmem, odr, eventl</literal> for internal
6926 debugging of yaz itself.
6927 Of course, any program using yaz is welcome to define as many new
6931 By default the log is written to stderr, but this can be changed by a call
6932 to <function>yaz_log_init_file</function> or
6933 <function>yaz_log_init</function>. If the log is directed to a file, the
6934 file size is checked at every write, and if it exceeds the limit given in
6935 <function>yaz_log_init_max_size</function>, the log is rotated. The
6936 rotation keeps one old version (with a <literal>.1</literal> appended to
6937 the name). The size defaults to 1GB. Setting it to zero will disable the
6941 A typical yaz-log looks like this
6942 13:23:14-23/11 yaz-ztest(1) [session] Starting session from tcp:127.0.0.1 (pid=30968)
6943 13:23:14-23/11 yaz-ztest(1) [request] Init from 'YAZ' (81) (ver 2.0.28) OK
6944 13:23:17-23/11 yaz-ztest(1) [request] Search Z: @attrset Bib-1 foo OK:7 hits
6945 13:23:22-23/11 yaz-ztest(1) [request] Present: [1] 2+2 OK 2 records returned
6946 13:24:13-23/11 yaz-ztest(1) [request] Close OK
6949 The log entries start with a time stamp. This can be omitted by setting the
6950 <literal>YLOG_NOTIME</literal> bit in the loglevel. This way automatic tests
6951 can be hoped to produce identical log files, that are easy to diff. The
6952 format of the time stamp can be set with
6953 <function>yaz_log_time_format</function>, which takes a format string just
6954 like <function>strftime</function>.
6957 Next in a log line comes the prefix, often the name of the program. For
6958 yaz-based servers, it can also contain the session number. Then
6959 comes one or more logbits in square brackets, depending on the logging
6960 level set by <function>yaz_log_init_level</function> and the loglevel
6961 passed to <function>yaz_log_init_level</function>. Finally comes the format
6962 string and additional values passed to <function>yaz_log</function>
6965 The log level <literal>YLOG_LOGLVL</literal>, enabled by the string
6966 <literal>loglevel</literal>, will log all the log-level affecting
6967 operations. This can come in handy if you need to know what other log
6968 levels would be useful. Grep the logfile for <literal>[loglevel]</literal>.
6971 The log system is almost independent of the rest of &yaz;, the only
6972 important dependence is of <filename>nmem</filename>, and that only for
6973 using the semaphore definition there.
6976 The dynamic log levels and log rotation were introduced in &yaz; 2.0.28. At
6977 the same time, the log bit names were changed from
6978 <literal>LOG_something</literal> to <literal>YLOG_something</literal>,
6979 to avoid collision with <filename>syslog.h</filename>.
6985 YAZ provides a fast utility for working with MARC records.
6986 Early versions of the MARC utility only allowed decoding of ISO2709.
6987 Today the utility may both encode - and decode to a varity of formats.
6990 #include <yaz/marcdisp.h>
6992 /* create handler */
6993 yaz_marc_t yaz_marc_create(void);
6995 void yaz_marc_destroy(yaz_marc_t mt);
6997 /* set XML mode YAZ_MARC_LINE, YAZ_MARC_SIMPLEXML, ... */
6998 void yaz_marc_xml(yaz_marc_t mt, int xmlmode);
6999 #define YAZ_MARC_LINE 0
7000 #define YAZ_MARC_SIMPLEXML 1
7001 #define YAZ_MARC_OAIMARC 2
7002 #define YAZ_MARC_MARCXML 3
7003 #define YAZ_MARC_ISO2709 4
7004 #define YAZ_MARC_XCHANGE 5
7005 #define YAZ_MARC_CHECK 6
7006 #define YAZ_MARC_TURBOMARC 7
7007 #define YAZ_MARC_JSON 8
7009 /* supply iconv handle for character set conversion .. */
7010 void yaz_marc_iconv(yaz_marc_t mt, yaz_iconv_t cd);
7012 /* set debug level, 0=none, 1=more, 2=even more, .. */
7013 void yaz_marc_debug(yaz_marc_t mt, int level);
7015 /* decode MARC in buf of size bsize. Returns >0 on success; <=0 on failure.
7016 On success, result in *result with size *rsize. */
7017 int yaz_marc_decode_buf(yaz_marc_t mt, const char *buf, int bsize,
7018 const char **result, size_t *rsize);
7020 /* decode MARC in buf of size bsize. Returns >0 on success; <=0 on failure.
7021 On success, result in WRBUF */
7022 int yaz_marc_decode_wrbuf(yaz_marc_t mt, const char *buf,
7023 int bsize, WRBUF wrbuf);
7028 The synopsis is just a basic subset of all functionality. Refer
7029 to the actual header file <filename>marcdisp.h</filename> for
7034 A MARC conversion handle must be created by using
7035 <function>yaz_marc_create</function> and destroyed
7036 by calling <function>yaz_marc_destroy</function>.
7039 All other function operate on a <literal>yaz_marc_t</literal> handle.
7040 The output is specified by a call to <function>yaz_marc_xml</function>.
7041 The <literal>xmlmode</literal> must be one of
7044 <term>YAZ_MARC_LINE</term>
7047 A simple line-by-line format suitable for display but not
7048 recommend for further (machine) processing.
7053 <term>YAZ_MARC_MARCXML</term>
7056 <ulink url="&url.marcxml;">MARCXML</ulink>.
7061 <term>YAZ_MARC_ISO2709</term>
7064 ISO2709 (sometimes just referred to as "MARC").
7069 <term>YAZ_MARC_XCHANGE</term>
7072 <ulink url="&url.marcxchange;">MarcXchange</ulink>.
7077 <term>YAZ_MARC_CHECK</term>
7080 Pseudo format for validation only. Does not generate
7081 any real output except diagnostics.
7086 <term>YAZ_MARC_TURBOMARC</term>
7089 XML format with same semantics as MARCXML but more compact
7090 and geared towards fast processing with XSLT. Refer to
7091 <xref linkend="tools.turbomarc"/> for more information.
7096 <term>YAZ_MARC_JSON</term>
7099 <ulink url="&url.marc_in_json;">MARC-in_JSON</ulink> format.
7106 The actual conversion functions are
7107 <function>yaz_marc_decode_buf</function> and
7108 <function>yaz_marc_decode_wrbuf</function> which decodes and encodes
7109 a MARC record. The former function operates on simple buffers, the
7110 stores the resulting record in a WRBUF handle (WRBUF is a simple string
7113 <example id="example.marc.display">
7114 <title>Display of MARC record</title>
7116 The following program snippet illustrates how the MARC API may
7117 be used to convert a MARC record to the line-by-line format:
7118 <programlisting><![CDATA[
7119 void print_marc(const char *marc_buf, int marc_buf_size)
7121 char *result; /* for result buf */
7122 size_t result_len; /* for size of result */
7123 yaz_marc_t mt = yaz_marc_create();
7124 yaz_marc_xml(mt, YAZ_MARC_LINE);
7125 yaz_marc_decode_buf(mt, marc_buf, marc_buf_size,
7126 &result, &result_len);
7127 fwrite(result, result_len, 1, stdout);
7128 yaz_marc_destroy(mt); /* note that result is now freed... */
7134 <sect2 id="tools.turbomarc">
7135 <title>TurboMARC</title>
7137 TurboMARC is yet another XML encoding of a MARC record. The format
7138 was designed for fast processing with XSLT.
7142 Pazpar2 uses XSLT to convert an XML encoded MARC record to an internal
7143 representation. This conversion mostly check the tag of a MARC field
7144 to determine the basic rules in the conversion. This check is
7145 costly when that is tag is encoded as an attribute in MARCXML.
7146 By having the tag value as the element instead, makes processing
7147 many times faster (at least for Libxslt).
7150 TurboMARC is encoded as follows:
7154 Record elements is part of namespace
7155 "<literal>http://www.indexdata.com/turbomarc</literal>".
7160 A record is enclosed in element <literal>r</literal>.
7165 A collection of records is enclosed in element
7166 <literal>collection</literal>.
7171 The leader is encoded as element <literal>l</literal> with the
7172 leader content as its (text) value.
7177 A control field is encoded as element <literal>c</literal> concatenated
7178 with the tag value of the control field if the tag value
7179 matches the regular expression <literal>[a-zA-Z0-9]*</literal>.
7180 If the tag value do not match the regular expression
7181 <literal>[a-zA-Z0-9]*</literal> the control field is encoded
7182 as element <literal>c</literal> and attribute <literal>code</literal>
7183 will hold the tag value.
7184 This rule ensure that in the rare cases where a tag value might
7185 result in a non-wellformed XML YAZ encode it as a coded attribute
7189 The control field content is the the text value of this element.
7190 Indicators are encoded as attribute names
7191 <literal>i1</literal>, <literal>i2</literal>, etc.. and
7192 corresponding values for each indicator.
7197 A data field is encoded as element <literal>d</literal> concatenated
7198 with the tag value of the data field or using the attribute
7199 <literal>code</literal> as described in the rules for control fields.
7200 The children of the data field element is subfield elements.
7201 Each subfield element is encoded as <literal>s</literal>
7202 concatenated with the sub field code.
7203 The text of the subfield element is the contents of the subfield.
7204 Indicators are encoded as attributes for the data field element similar
7205 to the encoding for control fields.
7212 <sect1 id="tools.retrieval">
7213 <title>Retrieval Facility</title>
7215 YAZ version 2.1.20 or later includes a Retrieval facility tool
7216 which allows a SRU/Z39.50 to describe itself and perform record
7217 conversions. The idea is the following:
7221 An SRU/Z39.50 client sends a retrieval request which includes
7222 a combination of the following parameters: syntax (format),
7223 schema (or element set name).
7228 The retrieval facility is invoked with parameters in a
7229 server/proxy. The retrieval facility matches the parameters a set of
7230 "supported" retrieval types.
7231 If there is no match, the retrieval signals an error
7232 (syntax and / or schema not supported).
7237 For a successful match, the backend is invoked with the same
7238 or altered retrieval parameters (syntax, schema). If
7239 a record is received from the backend, it is converted to the
7240 frontend name / syntax.
7245 The resulting record is sent back the client and tagged with
7246 the frontend syntax / schema.
7252 The Retrieval facility is driven by an XML configuration. The
7253 configuration is neither Z39.50 ZeeRex or SRU ZeeRex. But it
7254 should be easy to generate both of them from the XML configuration.
7255 (unfortunately the two versions
7256 of ZeeRex differ substantially in this regard).
7258 <sect2 id="tools.retrieval.format">
7259 <title>Retrieval XML format</title>
7261 All elements should be covered by namespace
7262 <literal>http://indexdata.com/yaz</literal> .
7263 The root element node must be <literal>retrievalinfo</literal>.
7266 The <literal>retrievalinfo</literal> must include one or
7267 more <literal>retrieval</literal> elements. Each
7268 <literal>retrieval</literal> defines specific combination of
7269 syntax, name and identifier supported by this retrieval service.
7272 The <literal>retrieval</literal> element may include any of the
7273 following attributes:
7275 <varlistentry><term><literal>syntax</literal> (REQUIRED)</term>
7278 Defines the record syntax. Possible values is any
7279 of the names defined in YAZ' OID database or a raw
7284 <varlistentry><term><literal>name</literal> (OPTIONAL)</term>
7287 Defines the name of the retrieval format. This can be
7288 any string. For SRU, the value, is equivalent to schema (short-hand);
7289 for Z39.50 it's equivalent to simple element set name.
7290 For YAZ 3.0.24 and later this name may be specified as a glob
7291 expression with operators
7292 <literal>*</literal> and <literal>?</literal>.
7296 <varlistentry><term><literal>identifier</literal> (OPTIONAL)</term>
7299 Defines the URI schema name of the retrieval format. This can be
7300 any string. For SRU, the value, is equivalent to URI schema.
7301 For Z39.50, there is no equivalent.
7308 The <literal>retrieval</literal> may include one
7309 <literal>backend</literal> element. If a <literal>backend</literal>
7310 element is given, it specifies how the records are retrieved by
7311 some backend and how the records are converted from the backend to
7315 The attributes, <literal>name</literal> and <literal>syntax</literal>
7316 may be specified for the <literal>backend</literal> element. These
7317 semantics of these attributes is equivalent to those for the
7318 <literal>retrieval</literal>. However, these values are passed to
7322 The <literal>backend</literal> element may includes one or more
7323 conversion instructions (as children elements). The supported
7326 <varlistentry><term><literal>marc</literal></term>
7329 The <literal>marc</literal> element specifies a conversion
7330 to - and from ISO2709 encoded MARC and
7331 <ulink url="&url.marcxml;">&acro.marcxml;</ulink>/MarcXchange.
7332 The following attributes may be specified:
7335 <term><literal>inputformat</literal> (REQUIRED)</term>
7338 Format of input. Supported values are
7339 <literal>marc</literal> (for ISO2709), <literal>xml</literal>
7340 (MARCXML/MarcXchange) and <literal>json</literal>
7341 (<ulink url="&url.marc_in_json;">MARC-in_JSON</ulink>).
7346 <term><literal>outputformat</literal> (REQUIRED)</term>
7349 Format of output. Supported values are
7350 <literal>line</literal> (MARC line format);
7351 <literal>marcxml</literal> (for MARCXML),
7352 <literal>marc</literal> (ISO2709),
7353 <literal>marcxhcange</literal> (for MarcXchange),
7354 or <literal>json</literal>
7355 (<ulink url="&url.marc_in_json;">MARC-in_JSON </ulink>).
7360 <term><literal>inputcharset</literal> (OPTIONAL)</term>
7363 Encoding of input. For XML input formats, this need not
7364 be given, but for ISO2709 based inputformats, this should
7365 be set to the encoding used. For MARC21 records, a common
7366 inputcharset value would be <literal>marc-8</literal>.
7371 <term><literal>outputcharset</literal> (OPTIONAL)</term>
7374 Encoding of output. If outputformat is XML based, it is
7375 strongly recommened to use <literal>utf-8</literal>.
7384 <term><literal>xslt</literal></term>
7387 The <literal>xslt</literal> element specifies a conversion
7388 via &acro.xslt;. The following attributes may be specified:
7390 <varlistentry><term><literal>stylesheet</literal> (REQUIRED)</term>
7402 <term><literal>solrmarc</literal></term>
7405 The <literal>solrmarc</literal> decodes solrmarc records.
7406 It assumes that the input is pure solrmarc text (no escaping)
7407 and will convert all sequences of the form #XX; to a single
7408 character of the hexadecimal value as given by XX. The output,
7409 presumably, is a valid ISO2709 buffer.
7412 This conversion is available in YAZ 5.0.21 and later.
7419 <sect2 id="tools.retrieval.examples">
7420 <title>Retrieval Facility Examples</title>
7421 <example id="tools.retrieval.marc21">
7422 <title>MARC21 backend</title>
7424 A typical way to use the retrieval facility is to enable XML
7425 for servers that only supports ISO2709 encoded MARC21 records.
7427 <programlisting><![CDATA[
7429 <retrieval syntax="usmarc" name="F"/>
7430 <retrieval syntax="usmarc" name="B"/>
7431 <retrieval syntax="xml" name="marcxml"
7432 identifier="info:srw/schema/1/marcxml-v1.1">
7433 <backend syntax="usmarc" name="F">
7434 <marc inputformat="marc" outputformat="marcxml"
7435 inputcharset="marc-8"/>
7438 <retrieval syntax="xml" name="dc">
7439 <backend syntax="usmarc" name="F">
7440 <marc inputformat="marc" outputformat="marcxml"
7441 inputcharset="marc-8"/>
7442 <xslt stylesheet="MARC21slim2DC.xsl"/>
7449 This means that our frontend supports:
7453 MARC21 F(ull) records.
7458 MARC21 B(rief) records.
7468 Dublin core records.
7474 <example id="tools.retrieval.marcxml">
7475 <title>MARCXML backend</title>
7477 SRW/SRU and Solr backends returns records in XML.
7478 If they return MARCXML or MarcXchange, the retrieval module
7479 can convert those into ISO2709 formats, most commonly USMARC
7481 In this example, the backend returns MARCXML for schema="marcxml".
7483 <programlisting><![CDATA[
7485 <retrieval syntax="usmarc">
7486 <backend syntax="xml" name="marcxml">
7487 <marc inputformat="xml" outputformat="marc"
7488 outputcharset="marc-8"/>
7491 <retrieval syntax="xml" name="marcxml"
7492 identifier="info:srw/schema/1/marcxml-v1.1"/>
7493 <retrieval syntax="xml" name="dc">
7494 <backend syntax="xml" name="marcxml">
7495 <xslt stylesheet="MARC21slim2DC.xsl"/>
7502 This means that our frontend supports:
7506 MARC21 records (any element set name) in MARC-8 encoding.
7511 MARCXML records for element-set=marcxml
7516 Dublin core records for element-set=dc.
7523 <sect2 id="tools.retrieval.api">
7526 It should be easy to use the retrieval systems from applications. Refer
7528 <filename>yaz/retrieval.h</filename> and
7529 <filename>yaz/record_conv.h</filename>.
7533 <sect1 id="sorting">
7534 <title>Sorting</title>
7536 This chapter describes sorting and how it is supported in YAZ.
7537 Sorting applies to a result-set.
7539 <ulink url="http://www.loc.gov/z3950/agency/markup/05.html#3.2.7">
7540 Z39.50 sorting facility
7542 takes one or more input result-sets
7543 and one result-set as output. The most simple case is that
7544 the input-set is the same as the output-set.
7547 Z39.50 sorting has a separate APDU (service) that is, thus, performed
7548 following a search (two phases).
7551 In SRU/Solr, however, the model is different. Here, sorting is specified
7552 during the the search operation. Note, however, that SRU might
7553 perform sort as separate search, by referring to an existing result-set
7554 in the query (result-set reference).
7557 <title>Using the Z39.50 sort service</title>
7559 yaz-client and the ZOOM API supports the Z39.50 sort facility. In any
7560 case the sort sequence or sort critiera is using a string notation.
7561 This notation is a one-line notation suitable for being manually
7562 entered or generated and allows for easy logging (one liner).
7563 For the ZOOM API, the sort is specified in the call to ZOOM_query_sortby
7564 function. For yaz-client the sort is performed and specified using
7565 the sort and sort+ commands. For description of the sort criteria notation
7566 refer to the <link linkend="sortspec">sort command</link> in the
7570 The ZOOM API might choose one of several sort strategies for
7571 sorting. Refer to <xref linkend="zoom-sort-strategy"/>.
7575 <title>Type-7 sort</title>
7577 Type-7 sort is an extension to the Bib-1 based RPN query where the
7578 sort specification is embedded as an Attribute-Plus-Term.
7581 The objectives for introducing Type-7 sorting is that it allows
7582 a client to perform sorting even if it does not implement/support
7583 Z39.50 sort. Virtually all Z39.50 client software supports
7584 RPN queries. It also may improve performance because the sort
7585 critieria is specified along with the search query.
7588 The sort is triggered by the presence of type 7 and the value of type 7
7590 <ulink url="http://www.loc.gov/z3950/agency/asn1.html#SortKeySpec">
7593 The value for type 7 is 1 for ascending and 2 for descending.
7595 <ulink url="http://www.loc.gov/z3950/agency/asn1.html#SortElement">
7598 only the generic part is handled. If generic sortKey is of type
7599 sortField, then attribute type 1 is present and the value is
7600 sortField (InternationalString). If generic sortKey is of type
7601 sortAttributes, then the attributes in list is used . generic sortKey
7602 of type elementSpec is not supported.
7605 The term in the sorting Attribute-Plus-Term combo should hold
7606 an integer. The value is 0 for primary sorting criteria, 1 for second
7612 <title>Facets</title>
7614 YAZ supports facets for in Solr, SRU 2.0 and Z39.50 protocols.
7617 Like Type-1/RPN, YAZ supports a string notation for specifying
7618 facets. For the API this is performed by
7619 <function>yaz_pqf_parse_facet_list</function>.
7622 For ZOOM C the facets are given by option "facets"
7623 For yaz-client it is used for the facets command.
7626 The grammar of this specification is as follows:
7628 facet-spec ::= facet-list
7630 facet-list ::= facet-list ',' attr-spec | attr-spec
7632 attr-spec ::= attr-spec '@attr' string | '@attr' string
7635 The notation is inspired by PQF. The string following '@attr'
7636 may not include blanks and is of the form
7637 <replaceable>type</replaceable><literal>=</literal><replaceable>value</replaceable>,
7638 where <replaceable>type</replaceable> is an integer and
7639 <replaceable>value</replaceable> is a string or an integer.
7642 The Facets specification is not Bib-1. The following types apply:
7644 <table id="facet.attributes">
7645 <title>Facet attributes</title>
7647 <colspec colwidth="2*" colname="type"></colspec>
7648 <colspec colwidth="9*" colname="description"></colspec>
7652 <entry>Description</entry>
7659 Field-name. This is often a string, eg "Author", "Year", etc.
7665 Sort order. Value should be an integer.
7666 Value 0: count descending (frequency). Value 1: alpha ascending.
7672 Number of terms requested.
7687 <title>The ODR Module</title>
7688 <sect1 id="odr.introduction">
7689 <title>Introduction</title>
7691 &odr; is the BER-encoding/decoding subsystem of &yaz;. Care as been taken
7692 to isolate &odr; from the rest of the package - specifically from the
7693 transport interface. &odr; may be used in any context where basic
7694 ASN.1/BER representations are used.
7697 If you are only interested in writing a Z39.50 implementation based on
7698 the PDUs that are already provided with &yaz;, you only need to concern
7699 yourself with the section on managing ODR streams
7700 (<xref linkend="odr.use"/>). Only if you need to
7701 implement ASN.1 beyond that which has been provided, should you
7702 worry about the second half of the documentation
7703 (<xref linkend="odr.programming"/>).
7704 If you use one of the higher-level interfaces, you can skip this
7708 This is important, so we'll repeat it for emphasis: <emphasis>You do
7709 not need to read <xref linkend="odr.programming"/>
7710 to implement Z39.50 with &yaz;.</emphasis>
7713 If you need a part of the protocol that isn't already in &yaz;, you
7714 should contact the authors before going to work on it yourself: We
7715 might already be working on it. Conversely, if you implement a useful
7716 part of the protocol before us, we'd be happy to include it in a
7720 <sect1 id="odr.use">
7721 <title>Using ODR</title>
7722 <sect2 id="odr.streams">
7723 <title>ODR Streams</title>
7725 Conceptually, the ODR stream is the source of encoded data in the
7726 decoding mode; when encoding, it is the receptacle for the encoded
7727 data. Before you can use an ODR stream it must be allocated. This is
7728 done with the function
7731 ODR odr_createmem(int direction);
7734 The <function>odr_createmem()</function> function takes as argument one
7735 of three manifest constants: <literal>ODR_ENCODE</literal>,
7736 <literal>ODR_DECODE</literal>, or <literal>ODR_PRINT</literal>.
7737 An &odr; stream can be in only one mode - it is not possible to change
7738 its mode once it's selected. Typically, your program will allocate
7739 at least two ODR streams - one for decoding, and one for encoding.
7742 When you're done with the stream, you can use
7745 void odr_destroy(ODR o);
7748 to release the resources allocated for the stream.
7751 <sect2 id="odr.memory.management">
7752 <title id="memory">Memory Management</title>
7754 Two forms of memory management take place in the &odr; system. The first
7755 one, which has to do with allocating little bits of memory (sometimes
7756 quite large bits of memory, actually) when a protocol package is
7757 decoded, and turned into a complex of interlinked structures. This
7758 section deals with this system, and how you can use it for your own
7759 purposes. The next section deals with the memory management which is
7760 required when encoding data - to make sure that a large enough buffer is
7761 available to hold the fully encoded PDU.
7764 The &odr; module has its own memory management system, which is
7765 used whenever memory is required. Specifically, it is used to allocate
7766 space for data when decoding incoming PDUs. You can use the memory
7767 system for your own purposes, by using the function
7770 void *odr_malloc(ODR o, size_t size);
7773 You can't use the normal <function>free(2)</function> routine to free
7774 memory allocated by this function, and &odr; doesn't provide a parallel
7775 function. Instead, you can call
7778 void odr_reset(ODR o);
7781 when you are done with the
7782 memory: Everything allocated since the last call to
7783 <function>odr_reset()</function> is released.
7784 The <function>odr_reset()</function> call is also required to clear
7785 up an error condition on a stream.
7791 size_t odr_total(ODR o);
7794 returns the number of bytes allocated on the stream since the last call to
7795 <function>odr_reset()</function>.
7798 The memory subsystem of &odr; is fairly efficient at allocating and
7799 releasing little bits of memory. Rather than managing the individual,
7800 small bits of space, the system maintains a free-list of larger chunks
7801 of memory, which are handed out in small bits. This scheme is
7802 generally known as a <emphasis>nibble memory</emphasis> system.
7803 It is very useful for maintaining short-lived constructions such
7807 If you want to retain a bit of memory beyond the next call to
7808 <function>odr_reset()</function>, you can use the function
7811 ODR_MEM odr_extract_mem(ODR o);
7814 This function will give you control of the memory recently allocated
7815 on the ODR stream. The memory will live (past calls to
7816 <function>odr_reset()</function>), until you call the function
7819 void odr_release_mem(ODR_MEM p);
7822 The opaque <literal>ODR_MEM</literal> handle has no other purpose than
7823 referencing the memory block for you until you want to release it.
7826 You can use <function>odr_extract_mem()</function> repeatedly between
7827 allocating data, to retain individual control of separate chunks of data.
7830 <sect2 id="odr.encoding.and.decoding">
7831 <title>Encoding and Decoding Data</title>
7833 When encoding data, the ODR stream will write the encoded octet string
7834 in an internal buffer. To retrieve the data, use the function
7837 char *odr_getbuf(ODR o, int *len, int *size);
7840 The integer pointed to by len is set to the length of the encoded
7841 data, and a pointer to that data is returned. <literal>*size</literal>
7842 is set to the size of the buffer (unless <literal>size</literal> is null,
7843 signaling that you are not interested in the size). The next call to
7844 a primitive function using the same &odr; stream will overwrite the
7845 data, unless a different buffer has been supplied using the call
7848 void odr_setbuf(ODR o, char *buf, int len, int can_grow);
7851 which sets the encoding (or decoding) buffer used by
7852 <literal>o</literal> to <literal>buf</literal>, using the length
7853 <literal>len</literal>.
7854 Before a call to an encoding function, you can use
7855 <function>odr_setbuf()</function> to provide the stream with an encoding
7856 buffer of sufficient size (length). The <literal>can_grow</literal>
7857 parameter tells the encoding &odr; stream whether it is allowed to use
7858 <function>realloc(2)</function> to increase the size of the buffer when
7859 necessary. The default condition of a new encoding stream is equivalent
7860 to the results of calling
7863 odr_setbuf(stream, 0, 0, 1);
7866 In this case, the stream will allocate and reallocate memory as
7867 necessary. The stream reallocates memory by repeatedly doubling the
7868 size of the buffer - the result is that the buffer will typically
7869 reach its maximum, working size with only a small number of reallocation
7870 operations. The memory is freed by the stream when the latter is destroyed,
7871 unless it was assigned by the user with the <literal>can_grow</literal>
7872 parameter set to zero (in this case, you are expected to retain
7873 control of the memory yourself).
7876 To assume full control of an encoded buffer, you must first call
7877 <function>odr_getbuf()</function> to fetch the buffer and its length.
7878 Next, you should call <function>odr_setbuf()</function> to provide a
7879 different buffer (or a null pointer) to the stream. In the simplest
7880 case, you will reuse the same buffer over and over again, and you
7881 will just need to call <function>odr_getbuf()</function> after each
7882 encoding operation to get the length and address of the buffer.
7883 Note that the stream may reallocate the buffer during an encoding
7884 operation, so it is necessary to retrieve the correct address after
7885 each encoding operation.
7888 It is important to realize that the ODR stream will not release this
7889 memory when you call <function>odr_reset()</function>: It will
7890 merely update its internal pointers to prepare for the encoding of a
7892 When the stream is released by the <function>odr_destroy()</function>
7893 function, the memory given to it by <function>odr_setbuf</function> will
7894 be released <emphasis>only</emphasis> if the <literal>can_grow</literal>
7895 parameter to <function>odr_setbuf()</function> was nonzero. The
7896 <literal>can_grow</literal> parameter, in other words, is a way of
7897 signaling who is to own the buffer, you or the ODR stream. If you never call
7898 <function>odr_setbuf()</function> on your encoding stream, which is
7899 typically the case, the buffer allocated by the stream will belong to
7900 the stream by default.
7903 When you wish to decode data, you should first call
7904 <function>odr_setbuf()</function>, to tell the decoding stream
7905 where to find the encoded data, and how long the buffer is
7906 (the <literal>can_grow</literal> parameter is ignored by a decoding
7907 stream). After this, you can call the function corresponding to the
7908 data you wish to decode (eg, <function>odr_integer()</function> odr
7909 <function>z_APDU()</function>).
7911 <example id="example.odr.encoding.and.decoding.functions">
7912 <title>Encoding and decoding functions</title>
7914 int odr_integer(ODR o, Odr_int **p, int optional, const char *name);
7916 int z_APDU(ODR o, Z_APDU **p, int optional, const char *name);
7920 If the data is absent (or doesn't match the tag corresponding to
7921 the type), the return value will be either 0 or 1 depending on the
7922 <literal>optional</literal> flag. If <literal>optional</literal>
7923 is 0 and the data is absent, an error flag will be raised in the
7924 stream, and you'll need to call <function>odr_reset()</function> before
7925 you can use the stream again. If <literal>optional</literal> is
7926 nonzero, the pointer <emphasis>pointed</emphasis> to/ by
7927 <literal>p</literal> will be set to the null value, and the function
7929 The <literal>name</literal> argument is used to pretty-print the
7930 tag in question. It may be set to <literal>NULL</literal> if
7931 pretty-printing is not desired.
7934 If the data value is found where it's expected, the pointer
7935 <emphasis>pointed to</emphasis> by the <literal>p</literal> argument
7936 will be set to point to the decoded type.
7937 The space for the type will be allocated and owned by the &odr;
7938 stream, and it will live until you call
7939 <function>odr_reset()</function> on the stream. You cannot use
7940 <function>free(2)</function> to release the memory.
7941 You can decode several data elements (by repeated calls to
7942 <function>odr_setbuf()</function> and your decoding function), and
7943 new memory will be allocated each time. When you do call
7944 <function>odr_reset()</function>, everything decoded since the
7945 last call to <function>odr_reset()</function> will be released.
7947 <example id="example.odr.encoding.of.integer">
7948 <title>Encoding and decoding of an integer</title>
7950 The use of the double indirection can be a little confusing at first
7951 (its purpose will become clear later on, hopefully),
7952 so an example is in order. We'll encode an integer value, and
7953 immediately decode it again using a different stream. A useless, but
7954 informative operation.
7956 <programlisting><![CDATA[
7957 void do_nothing_useful(Odr_int value)
7960 Odr_int *valp, *resvalp;
7964 /* allocate streams */
7965 if (!(encode = odr_createmem(ODR_ENCODE)))
7967 if (!(decode = odr_createmem(ODR_DECODE)))
7971 if (odr_integer(encode, &valp, 0, 0) == 0)
7973 printf("encoding went bad\n");
7976 bufferp = odr_getbuf(encode, &len, 0);
7977 printf("length of encoded data is %d\n", len);
7979 /* now let's decode the thing again */
7980 odr_setbuf(decode, bufferp, len, 0);
7981 if (odr_integer(decode, &resvalp, 0, 0) == 0)
7983 printf("decoding went bad\n");
7986 /* ODR_INT_PRINTF format for printf (such as %d) */
7987 printf("the value is " ODR_INT_PRINTF "\n", *resvalp);
7990 odr_destroy(encode);
7991 odr_destroy(decode);
7996 This looks like a lot of work, offhand. In practice, the &odr; streams
7997 will typically be allocated once, in the beginning of your program
7998 (or at the beginning of a new network session), and the encoding
7999 and decoding will only take place in a few, isolated places in your
8000 program, so the overhead is quite manageable.
8004 <sect2 id="odr.printing">
8005 <title>Printing</title>
8007 When an ODR stream is created of type <literal>ODR_PRINT</literal>
8008 the ODR module will print the contents of a PDU in a readable format.
8009 By default output is written to the <literal>stderr</literal> stream.
8010 This behavior can be changed, however, by calling the function
8012 odr_setprint(ODR o, FILE *file);
8014 before encoders or decoders are being invoked.
8015 It is also possible to direct the output to a buffer (of indeed
8016 another file), by using the more generic mechanism:
8018 void odr_set_stream(ODR o, void *handle,
8019 void (*stream_write)(ODR o, void *handle, int type,
8020 const char *buf, int len),
8021 void (*stream_close)(void *handle));
8023 Here the user provides an opaque handle and two handlers,
8024 <replaceable>stream_write</replaceable> for writing,
8025 and <replaceable>stream_close</replaceable> which is supposed
8026 to close/free resources associated with handle.
8027 The <replaceable>stream_close</replaceable> handler is optional and
8028 if NULL for the function is provided, it will not be invoked.
8029 The <replaceable>stream_write</replaceable> takes the ODR handle
8030 as parameter, the user defined handle, a type
8031 <literal>ODR_OCTETSTRING</literal>, <literal>ODR_VISIBLESTRING</literal>
8032 which indicates the type of contents is being written.
8035 Another utility useful for diagnostics (error handling) or as
8036 part of the printing facilities is:
8038 const char **odr_get_element_path(ODR o);
8040 which returns a list of current elements that ODR deals with at the
8041 moment. For the returned array, say <literal>ar</literal>,
8042 <literal>ar[0]</literal> is the top level element,
8043 <literal>ar[n]</literal> is the last. The last element has the
8044 property that <literal>ar[n+1] == NULL</literal>.
8046 <example id="example.odr.element.path.record">
8047 <title>Element Path for record</title>
8049 For a database record part of a PresentResponse the
8050 array returned by <function>odr_get_element</function>
8051 is <literal>presentResponse</literal>, <literal>databaseOrSurDiagnostics</literal>, <literal>?</literal>, <literal>record</literal>, <literal>?</literal>, <literal>databaseRecord</literal> . The question mark appears due to
8052 unnamed constructions.
8056 <sect2 id="odr.diagnostics">
8057 <title>Diagnostics</title>
8059 The encoding/decoding functions all return 0 when an error occurs.
8060 Until you call <function>odr_reset()</function>, you cannot use the
8061 stream again, and any function called will immediately return 0.
8064 To provide information to the programmer or administrator, the function
8067 void odr_perror(ODR o, char *message);
8070 is provided, which prints the <literal>message</literal> argument to
8071 <literal>stderr</literal> along with an error message from the stream.
8074 You can also use the function
8077 int odr_geterror(ODR o);
8080 to get the current error number from the screen. The number will be
8081 one of these constants:
8083 <table frame="top" id="odr.error.codes">
8084 <title>ODR Error codes</title>
8089 <entry>Description</entry>
8094 <entry>OMEMORY</entry><entry>Memory allocation failed.</entry>
8097 <entry>OSYSERR</entry><entry>A system- or library call has failed.
8098 The standard diagnostic variable <literal>errno</literal> should be
8099 examined to determine the actual error.</entry>
8102 <entry>OSPACE</entry><entry>No more space for encoding.
8103 This will only occur when the user has explicitly provided a
8104 buffer for an encoding stream without allowing the system to
8105 allocate more space.</entry>
8108 <entry>OREQUIRED</entry><entry>This is a common protocol error; A
8109 required data element was missing during encoding or decoding.</entry>
8112 <entry>OUNEXPECTED</entry><entry>An unexpected data element was
8113 found during decoding.</entry>
8116 <entry>OOTHER</entry><entry>Other error. This is typically an
8117 indication of misuse of the &odr; system by the programmer, and also
8118 that the diagnostic system isn't as good as it should be, yet.</entry>
8124 The character string array
8130 can be indexed by the error code to obtain a human-readable
8131 representation of the problem.
8134 <sect2 id="odr.summary.and.synopsis">
8135 <title>Summary and Synopsis</title>
8137 #include <yaz/odr.h>
8139 ODR odr_createmem(int direction);
8141 void odr_destroy(ODR o);
8143 void odr_reset(ODR o);
8145 char *odr_getbuf(ODR o, int *len, int *size);
8147 void odr_setbuf(ODR o, char *buf, int len, int can_grow);
8149 void *odr_malloc(ODR o, int size);
8151 NMEM odr_extract_mem(ODR o);
8153 int odr_geterror(ODR o);
8155 void odr_perror(ODR o, const char *message);
8157 extern char *odr_errlist[];
8161 <sect1 id="odr.programming">
8162 <title>Programming with ODR</title>
8164 The API of &odr; is designed to reflect the structure of ASN.1, rather
8165 than BER itself. Future releases may be able to represent data in
8166 other external forms.
8170 There is an ASN.1 tutorial available at
8171 <ulink url="&url.asn.1.tutorial;">this site</ulink>.
8172 This site also has standards for ASN.1 (X.680) and BER (X.690)
8173 <ulink url="&url.asn.1.standards;">online</ulink>.
8177 The ODR interface is based loosely on that of the Sun Microsystems
8179 Specifically, each function which corresponds to an ASN.1 primitive
8180 type has a dual function. Depending on the settings of the ODR
8181 stream which is supplied as a parameter, the function may be used
8182 either to encode or decode data. The functions that can be built
8183 using these primitive functions, to represent more complex data types,
8184 share this quality. The result is that you only have to enter the
8185 definition for a type once - and you have the functionality of encoding,
8186 decoding (and pretty-printing) all in one unit.
8187 The resulting C source code is quite compact, and is a pretty
8188 straightforward representation of the source ASN.1 specification.
8191 In many cases, the model of the XDR functions works quite well in this
8193 In others, it is less elegant. Most of the hassle comes from the optional
8194 SEQUENCE members which don't exist in XDR.
8196 <sect2 id="odr.primitive.asn1.types">
8197 <title>The Primitive ASN.1 Types</title>
8199 ASN.1 defines a number of primitive types (many of which correspond
8200 roughly to primitive types in structured programming languages, such as C).
8202 <sect3 id="odr.integer">
8203 <title>INTEGER</title>
8205 The &odr; function for encoding or decoding (or printing) the ASN.1
8206 INTEGER type looks like this:
8209 int odr_integer(ODR o, Odr_int **p, int optional, const char *name);
8212 The <literal>Odr_int</literal> is just a simple integer.
8215 This form is typical of the primitive &odr; functions. They are named
8216 after the type of data that they encode or decode. They take an &odr;
8217 stream, an indirect reference to the type in question, and an
8218 <literal>optional</literal> flag (corresponding to the OPTIONAL keyword
8219 of ASN.1) as parameters. They all return an integer value of either one
8221 When you use the primitive functions to construct encoders for complex
8222 types of your own, you should follow this model as well. This
8223 ensures that your new types can be reused as elements in yet more
8227 The <literal>o</literal> parameter should obviously refer to a properly
8228 initialized &odr; stream of the right type (encoding/decoding/printing)
8229 for the operation that you wish to perform.
8232 When encoding or printing, the function first looks at
8233 <literal>* p</literal>. If <literal>* p</literal> (the pointer pointed
8234 to by <literal>p</literal>) is a null pointer, this is taken to mean that
8235 the data element is absent. If the <literal>optional</literal> parameter
8236 is nonzero, the function will return one (signifying success) without
8237 any further processing. If the <literal>optional</literal> is zero, an
8238 internal error flag is set in the &odr; stream, and the function will
8239 return 0. No further operations can be carried out on the stream without
8240 a call to the function <function>odr_reset()</function>.
8243 If <literal>*p</literal> is not a null pointer, it is expected to
8244 point to an instance of the data type. The data will be subjected to
8245 the encoding rules, and the result will be placed in the buffer held
8246 by the &odr; stream.
8249 The other ASN.1 primitives have similar functions that operate in
8253 <sect3 id="odr.boolean">
8254 <title>BOOLEAN</title>
8256 int odr_bool(ODR o, Odr_bool **p, int optional, const char *name);
8259 <sect3 id="odr.real">
8265 <sect3 id="odr.null">
8268 int odr_null(ODR o, Odr_null **p, int optional, const char *name);
8271 In this case, the value of **p is not important. If <literal>*p</literal>
8272 is different from the null pointer, the null value is present, otherwise
8276 <sect3 id="odr.octet.string">
8277 <title>OCTET STRING</title>
8279 typedef struct odr_oct
8285 int odr_octetstring(ODR o, Odr_oct **p, int optional,
8289 The <literal>buf</literal> field should point to the character array
8290 that holds the octetstring. The <literal>len</literal> field holds the
8292 The character array need not be null terminated.
8295 To make things a little easier, an alternative is given for string
8296 types that are not expected to contain embedded NULL characters (eg.
8300 int odr_cstring(ODR o, char **p, int optional, const char *name);
8303 Which encoded or decodes between OCTETSTRING representations and
8304 null-terminates C strings.
8307 Functions are provided for the derived string types, eg:
8310 int odr_visiblestring(ODR o, char **p, int optional,
8314 <sect3 id="odr.bit.string">
8315 <title>BIT STRING</title>
8317 int odr_bitstring(ODR o, Odr_bitmask **p, int optional,
8321 The opaque type <literal>Odr_bitmask</literal> is only suitable for
8322 holding relatively brief bit strings, eg. for options fields, etc.
8323 The constant <literal>ODR_BITMASK_SIZE</literal> multiplied by 8
8324 gives the maximum possible number of bits.
8327 A set of macros are provided for manipulating the
8328 <literal>Odr_bitmask</literal> type:
8331 void ODR_MASK_ZERO(Odr_bitmask *b);
8333 void ODR_MASK_SET(Odr_bitmask *b, int bitno);
8335 void ODR_MASK_CLEAR(Odr_bitmask *b, int bitno);
8337 int ODR_MASK_GET(Odr_bitmask *b, int bitno);
8340 The functions are modeled after the manipulation functions that
8341 accompany the <literal>fd_set</literal> type used by the
8342 <function>select(2)</function> call.
8343 <literal>ODR_MASK_ZERO</literal> should always be called first on a
8344 new bitmask, to initialize the bits to zero.
8347 <sect3 id="odr.object.identifier">
8348 <title>OBJECT IDENTIFIER</title>
8350 int odr_oid(ODR o, Odr_oid **p, int optional, const char *name);
8353 The C OID representation is simply an array of integers, terminated by
8354 the value -1 (the <literal>Odr_oid</literal> type is synonymous with
8355 the <literal>short</literal> type).
8356 We suggest that you use the OID database module (see
8357 <xref linkend="tools.oid.database"/>) to handle object identifiers
8358 in your application.
8362 <sect2 id="odr.tagging.primitive.types">
8363 <title>Tagging Primitive Types</title>
8365 The simplest way of tagging a type is to use the
8366 <function>odr_implicit_tag()</function> or
8367 <function>odr_explicit_tag()</function> macros:
8370 int odr_implicit_tag(ODR o, Odr_fun fun, int class, int tag,
8371 int optional, const char *name);
8373 int odr_explicit_tag(ODR o, Odr_fun fun, int class, int tag,
8374 int optional, const char *name);
8377 To create a type derived from the integer type by implicit tagging, you
8381 MyInt ::= [210] IMPLICIT INTEGER
8384 In the &odr; system, this would be written like:
8387 int myInt(ODR o, Odr_int **p, int optional, const char *name)
8389 return odr_implicit_tag(o, odr_integer, p,
8390 ODR_CONTEXT, 210, optional, name);
8394 The function <function>myInt()</function> can then be used like any of
8395 the primitive functions provided by &odr;. Note that the behavior of
8396 <function>odr_explicit_tag()</function>
8397 and <function>odr_implicit_tag()</function> macros
8398 act exactly the same as the functions they are applied to - they
8399 respond to error conditions, etc, in the same manner - they
8400 simply have three extra parameters. The class parameter may
8401 take one of the values: <literal>ODR_CONTEXT</literal>,
8402 <literal>ODR_PRIVATE</literal>, <literal>ODR_UNIVERSAL</literal>, or
8403 <literal>/ODR_APPLICATION</literal>.
8406 <sect2 id="odr.constructed.types">
8407 <title>Constructed Types</title>
8409 Constructed types are created by combining primitive types. The
8410 &odr; system only implements the SEQUENCE and SEQUENCE OF constructions
8411 (although adding the rest of the container types should be simple
8412 enough, if the need arises).
8415 For implementing SEQUENCEs, the functions
8418 int odr_sequence_begin(ODR o, void *p, int size, const char *name);
8419 int odr_sequence_end(ODR o);
8425 The <function>odr_sequence_begin()</function> function should be
8426 called in the beginning of a function that implements a SEQUENCE type.
8427 Its parameters are the &odr; stream, a pointer (to a pointer to the type
8428 you're implementing), and the <literal>size</literal> of the type
8429 (typically a C structure). On encoding, it returns 1 if
8430 <literal>* p</literal> is a null pointer. The <literal>size</literal>
8431 parameter is ignored. On decoding, it returns 1 if the type is found in
8432 the data stream. <literal>size</literal> bytes of memory are allocated,
8433 and <literal>*p</literal> is set to point to this space.
8434 <function>odr_sequence_end()</function> is called at the end of the
8435 complex function. Assume that a type is defined like this:
8438 MySequence ::= SEQUENCE {
8440 boolval BOOLEAN OPTIONAL
8444 The corresponding &odr; encoder/decoder function and the associated data
8445 structures could be written like this:
8448 typedef struct MySequence
8454 int mySequence(ODR o, MySequence **p, int optional, const char *name)
8456 if (odr_sequence_begin(o, p, sizeof(**p), name) == 0)
8457 return optional && odr_ok(o);
8459 odr_integer(o, &(*p)->intval, 0, "intval") &&
8460 odr_bool(o, &(*p)->boolval, 1, "boolval") &&
8461 odr_sequence_end(o);
8465 Note the 1 in the call to <function>odr_bool()</function>, to mark
8466 that the sequence member is optional.
8467 If either of the member types had been tagged, the macros
8468 <function>odr_implicit_tag()</function> or
8469 <function>odr_explicit_tag()</function>
8470 could have been used.
8471 The new function can be used exactly like the standard functions provided
8472 with &odr;. It will encode, decode or pretty-print a data value of the
8473 <literal>MySequence</literal> type. We like to name types with an
8474 initial capital, as done in ASN.1 definitions, and to name the
8475 corresponding function with the first character of the name in lower case.
8476 You could, of course, name your structures, types, and functions any way
8477 you please - as long as you're consistent, and your code is easily readable.
8478 <literal>odr_ok</literal> is just that - a predicate that returns the
8479 state of the stream. It is used to ensure that the behavior of the new
8480 type is compatible with the interface of the primitive types.
8483 <sect2 id="odr.tagging.constructed.types">
8484 <title>Tagging Constructed Types</title>
8487 See <xref linkend="odr.tagging.primitive.types"/> for information
8488 on how to tag the primitive types, as well as types that are
8492 <sect3 id="odr.implicit.tagging">
8493 <title>Implicit Tagging</title>
8495 Assume the type above had been defined as
8498 MySequence ::= [10] IMPLICIT SEQUENCE {
8500 boolval BOOLEAN OPTIONAL
8504 You would implement this in &odr; by calling the function
8507 int odr_implicit_settag(ODR o, int class, int tag);
8510 which overrides the tag of the type immediately following it. The
8511 macro <function>odr_implicit_tag()</function> works by calling
8512 <function>odr_implicit_settag()</function> immediately
8513 before calling the function pointer argument.
8514 Your type function could look like this:
8517 int mySequence(ODR o, MySequence **p, int optional, const char *name)
8519 if (odr_implicit_settag(o, ODR_CONTEXT, 10) == 0 ||
8520 odr_sequence_begin(o, p, sizeof(**p), name) == 0)
8521 return optional && odr_ok(o);
8523 odr_integer(o, &(*p)->intval, 0, "intval") &&
8524 odr_bool(o, &(*p)->boolval, 1, "boolval") &&
8525 odr_sequence_end(o);
8529 The definition of the structure <literal>MySequence</literal> would be
8533 <sect3 id="odr.explicit.tagging">
8534 <title>Explicit Tagging</title>
8536 Explicit tagging of constructed types is a little more complicated,
8537 since you are in effect adding a level of construction to the data.
8540 Assume the definition:
8543 MySequence ::= [10] IMPLICIT SEQUENCE {
8545 boolval BOOLEAN OPTIONAL
8549 Since the new type has an extra level of construction, two new functions
8550 are needed to encapsulate the base type:
8553 int odr_constructed_begin(ODR o, void *p, int class, int tag,
8556 int odr_constructed_end(ODR o);
8559 Assume that the IMPLICIT in the type definition above were replaced
8560 with EXPLICIT (or that the IMPLICIT keyword were simply deleted, which
8561 would be equivalent). The structure definition would look the same,
8562 but the function would look like this:
8565 int mySequence(ODR o, MySequence **p, int optional, const char *name)
8567 if (odr_constructed_begin(o, p, ODR_CONTEXT, 10, name) == 0)
8568 return optional && odr_ok(o);
8569 if (o->direction == ODR_DECODE)
8570 *p = odr_malloc(o, sizeof(**p));
8571 if (odr_sequence_begin(o, p, sizeof(**p), 0) == 0)
8573 *p = 0; /* this is almost certainly a protocol error */
8577 odr_integer(o, &(*p)->intval, 0, "intval") &&
8578 odr_bool(o, &(*p)->boolval, 1, "boolval") &&
8579 odr_sequence_end(o) &&
8580 odr_constructed_end(o);
8584 Notice that the interface here gets kind of nasty. The reason is
8585 simple: Explicitly tagged, constructed types are fairly rare in
8586 the protocols that we care about, so the
8587 esthetic annoyance (not to mention the dangers of a cluttered
8588 interface) is less than the time that would be required to develop a
8589 better interface. Nevertheless, it is far from satisfying, and it's a
8590 point that will be worked on in the future. One option for you would
8591 be to simply apply the <function>odr_explicit_tag()</function> macro to
8592 the first function, and not
8593 have to worry about <function>odr_constructed_*</function> yourself.
8594 Incidentally, as you might have guessed, the
8595 <function>odr_sequence_</function> functions are themselves
8596 implemented using the <function>/odr_constructed_</function> functions.
8600 <sect2 id="odr.sequence.of">
8601 <title>SEQUENCE OF</title>
8603 To handle sequences (arrays) of a specific type, the function
8606 int odr_sequence_of(ODR o, int (*fun)(ODR o, void *p, int optional),
8607 void *p, int *num, const char *name);
8610 The <literal>fun</literal> parameter is a pointer to the decoder/encoder
8611 function of the type. <literal>p</literal> is a pointer to an array of
8612 pointers to your type. <literal>num</literal> is the number of elements
8619 MyArray ::= SEQUENCE OF INTEGER
8622 The C representation might be
8625 typedef struct MyArray
8632 And the function might look like
8635 int myArray(ODR o, MyArray **p, int optional, const char *name)
8637 if (o->direction == ODR_DECODE)
8638 *p = odr_malloc(o, sizeof(**p));
8639 if (odr_sequence_of(o, odr_integer, &(*p)->elements,
8640 &(*p)->num_elements, name))
8643 return optional && odr_ok(o);
8647 <sect2 id="odr.choice.types">
8648 <title>CHOICE Types</title>
8650 The choice type is used fairly often in some ASN.1 definitions, so
8651 some work has gone into streamlining its interface.
8654 CHOICE types are handled by the function:
8657 int odr_choice(ODR o, Odr_arm arm[], void *p, void *whichp,
8661 The <literal>arm</literal> array is used to describe each of the possible
8662 types that the CHOICE type may assume. Internally in your application,
8663 the CHOICE type is represented as a discriminated union. That is, a
8664 C union accompanied by an integer (or enum) identifying the active
8666 <literal>whichp</literal> is a pointer to the union discriminator.
8667 When encoding, it is examined to determine the current type.
8668 When decoding, it is set to reference the type that was found in
8672 The Odr_arm type is defined thus:
8675 typedef struct odr_arm
8686 The interpretation of the fields are:
8690 <term>tagmode</term>
8691 <listitem><para>Either <literal>ODR_IMPLICIT</literal>,
8692 <literal>ODR_EXPLICIT</literal>, or <literal>ODR_NONE</literal> (-1)
8693 to mark no tagging.</para></listitem>
8697 <listitem><para>The value of the discriminator that corresponds to
8698 this CHOICE element. Typically, it will be a #defined constant, or
8699 an enum member.</para></listitem>
8703 <listitem><para>A pointer to a function that implements the type of
8704 the CHOICE member. It may be either a standard &odr; type or a type
8705 defined by yourself.</para></listitem>
8709 <listitem><para>Name of tag.</para></listitem>
8713 A handy way to prepare the array for use by the
8714 <function>odr_choice()</function> function is to
8715 define it as a static, initialized array in the beginning of your
8716 decoding/encoding function. Assume the type definition:
8719 MyChoice ::= CHOICE {
8721 tagged [99] IMPLICIT INTEGER,
8726 Your C type might look like
8729 typedef struct MyChoice
8746 And your function could look like this:
8749 int myChoice(ODR o, MyChoice **p, int optional, const char *name)
8751 static Odr_arm arm[] =
8753 {-1, -1, -1, MyChoice_untagged, odr_integer, "untagged"},
8754 {ODR_IMPLICIT, ODR_CONTEXT, 99, MyChoice_tagged, odr_integer,
8756 {-1, -1, -1, MyChoice_other, odr_boolean, "other"},
8760 if (o->direction == ODR_DECODE)
8761 *p = odr_malloc(o, sizeof(**p);
8763 return optional && odr_ok(o);
8765 if (odr_choice(o, arm, &(*p)->u, &(*p)->which), name)
8768 return optional && odr_ok(o);
8772 In some cases (say, a non-optional choice which is a member of a
8773 sequence), you can "embed" the union and its discriminator in the
8774 structure belonging to the enclosing type, and you won't need to
8775 fiddle with memory allocation to create a separate structure to
8776 wrap the discriminator and union.
8779 The corresponding function is somewhat nicer in the Sun XDR interface.
8780 Most of the complexity of this interface comes from the possibility of
8781 declaring sequence elements (including CHOICEs) optional.
8784 The ASN.1 specifications naturally requires that each member of a
8785 CHOICE have a distinct tag, so they can be told apart on decoding.
8786 Sometimes it can be useful to define a CHOICE that has multiple types
8787 that share the same tag. You'll need some other mechanism, perhaps
8788 keyed to the context of the CHOICE type. In effect, we would like to
8789 introduce a level of context-sensitiveness to our ASN.1 specification.
8790 When encoding an internal representation, we have no problem, as long
8791 as each CHOICE member has a distinct discriminator value. For
8792 decoding, we need a way to tell the choice function to look for a
8793 specific arm of the table. The function
8796 void odr_choice_bias(ODR o, int what);
8799 provides this functionality. When called, it leaves a notice for the next
8800 call to <function>odr_choice()</function> to be called on the decoding
8801 stream <literal>o</literal> that only the <literal>arm</literal> entry with
8802 a <literal>which</literal> field equal to <literal>what</literal>
8806 The most important application (perhaps the only one, really) is in
8807 the definition of application-specific EXTERNAL encoders/decoders
8808 which will automatically decode an ANY member given the direct or
8813 <sect1 id="odr.debugging">
8814 <title>Debugging</title>
8816 The protocol modules are suffering somewhat from a lack of diagnostic
8817 tools at the moment. Specifically ways to pretty-print PDUs that
8818 aren't recognized by the system. We'll include something to this end
8819 in a not-too-distant release. In the meantime, what we do when we get
8820 packages we don't understand is to compile the ODR module with
8821 <literal>ODR_DEBUG</literal> defined. This causes the module to dump tracing
8822 information as it processes data units. With this output and the
8823 protocol specification (Z39.50), it is generally fairly easy to see
8828 <chapter id="comstack">
8829 <title>The COMSTACK Module</title>
8830 <sect1 id="comstack.synopsis">
8831 <title>Synopsis (blocking mode)</title>
8832 <programlisting><![CDATA[
8835 int size = 0, length_incoming;
8836 char server_address_str[] = "localhost:9999";
8837 void *server_address_ip;
8840 char *protocol_package = "GET / HTTP/1.0\r\n\r\n";
8841 int protocol_package_length = strlen(protocol_package);
8843 stack = cs_create(tcpip_type, 1, PROTO_HTTP);
8845 perror("cs_create"); /* use perror() here since we have no stack yet */
8849 server_address_ip = cs_straddr(stack, server_address_str);
8850 if (!server_address_ip) {
8851 fprintf(stderr, "cs_straddr: address could not be resolved\n");
8855 status = cs_connect(stack, server_address_ip);
8857 fprintf(stderr, "cs_connect: %s\n", cs_strerror(stack));
8861 status = cs_rcvconnect(stack);
8863 fprintf(stderr, "cs_rcvconnect: %s\n", cs_strerror(stack));
8867 status = cs_put(stack, protocol_package, protocol_package_length);
8869 fprintf(stderr, "cs_put: %s\n", cs_strerror(stack));
8873 /* Now get a response */
8874 length_incoming = cs_get(stack, &buf, &size);
8875 if (!length_incoming) {
8876 fprintf(stderr, "Connection closed\n");
8878 } else if (length_incoming < 0) {
8879 fprintf(stderr, "cs_get: %s\n", cs_strerror(stack));
8884 fwrite(buf, length_incoming, 1, stdout);
8895 <sect1 id="comstack.introduction">
8896 <title>Introduction</title>
8899 subsystem provides a transparent interface to different types of transport
8900 stacks for the exchange of BER-encoded data and HTTP packets.
8901 At present, the RFC1729 method (BER over TCP/IP), local UNIX socket and an
8902 experimental SSL stack are supported, but others may be added in time.
8903 The philosophy of the
8904 module is to provide a simple interface by hiding unused options and
8905 facilities of the underlying libraries. This is always done at the risk
8906 of losing generality, and it may prove that the interface will need
8911 There hasn't been interest in the XTImOSI stack for some years.
8912 Therefore, it is no longer supported.
8916 The interface is implemented in such a fashion that only the
8917 sub-layers constructed to the transport methods that you wish to
8918 use in your application are linked in.
8921 You will note that even though simplicity was a goal in the design,
8922 the interface is still orders of magnitudes more complex than the
8923 transport systems found in many other packages. One reason is that
8924 the interface needs to support the somewhat different requirements of
8925 the different lower-layer communications stacks; another important
8926 reason is that the interface seeks to provide a more or less
8927 industrial-strength approach to asynchronous event-handling.
8928 When no function is allowed to block, things get more complex -
8929 particularly on the server side.
8930 We urge you to have a look at the demonstration client and server
8931 provided with the package. They are meant to be easily readable and
8932 instructive, while still being at least moderately useful.
8935 <sect1 id="comstack.common">
8936 <title>Common Functions</title>
8937 <sect2 id="comstack.managing.endpoints">
8938 <title>Managing Endpoints</title>
8940 COMSTACK cs_create(CS_TYPE type, int blocking, int protocol);
8943 Creates an instance of the protocol stack - a communications endpoint.
8944 The <literal>type</literal> parameter determines the mode
8945 of communication. At present the following values are supported:
8949 <term><literal>tcpip_type</literal></term>
8950 <listitem><para>TCP/IP (BER over TCP/IP or HTTP over TCP/IP)
8954 <term><literal>ssl_type</literal></term>
8955 <listitem><para>Secure Socket Layer (SSL). This COMSTACK
8956 is experimental and is not fully implemented. If
8957 HTTP is used, this effectively is HTTPS.
8961 <term><literal>unix_type</literal></term>
8962 <listitem><para>Unix socket (unix only). Local Transfer via
8963 file socket. See <citerefentry><refentrytitle>unix</refentrytitle>
8964 <manvolnum>7</manvolnum></citerefentry>.
8969 The <function>cs_create</function> function returns a null-pointer
8970 if a system error occurs.
8971 The <literal>blocking</literal> parameter should be one if
8972 you wish the association to operate in blocking mode, zero otherwise.
8973 The <literal>protocol</literal> field should be
8974 <literal>PROTO_Z3950</literal> or <literal>PROTO_HTTP</literal>.
8975 Protocol <literal>PROTO_SR</literal> is no longer supported.
8978 void cs_close(COMSTACK handle);
8981 Closes the connection (as elegantly as the lower layers will permit),
8982 and releases the resources pointed to by the
8983 <literal>handle</literal>
8985 <literal>handle</literal>
8986 should not be referenced again after this call.
8990 We really need a soft disconnect, don't we?
8994 <sect2 id="comstack.data.exchange">
8995 <title>Data Exchange</title>
8997 int cs_put(COMSTACK handle, char *buf, int len);
9000 Sends <literal>buf</literal> down the wire.
9001 In blocking mode, this function will return only when a full buffer has
9002 been written, or an error has occurred. In nonblocking mode, it's
9003 possible that the function will be unable to send the full buffer
9004 at once, which will be indicated by a return value of 1.
9005 The function will keep track of the number of octets already written; you
9006 should call it repeatedly with the same values of <literal>buf</literal>
9007 and <literal>len</literal>, until the buffer has been transmitted.
9008 When a full buffer has been sent, the function will return 0 for
9009 success. -1 indicates an error condition (see below).
9012 int cs_get(COMSTACK handle, char **buf, int *size);
9015 Receives a PDU or HTTP Response from the peer. Returns the number of
9017 In nonblocking mode, it is possible that not all of the packet can be
9018 read at once. In this case, the function returns 1. To simplify the
9019 interface, the function is
9020 responsible for managing the size of the buffer. It will be reallocated
9021 if necessary to contain large packages, and will sometimes be moved
9022 around internally by the subsystem when partial packages are read. Before
9024 <function>cs_get</function>
9025 for the fist time, the buffer can be initialized to the null pointer,
9026 and the length should also be set to 0 - cs_get will perform a
9027 <function>malloc(2)</function>
9028 on the buffer for you. When a full buffer has been read, the size of
9029 the package is returned (which will always be greater than 1). -1
9030 indicates an error condition.
9033 See also the <function>cs_more()</function> function below.
9036 int cs_more(COMSTACK handle);
9039 The <function>cs_more()</function> function should be used in conjunction
9040 with <function>cs_get</function> and
9041 <function>select(2)</function>.
9042 The <function>cs_get()</function> function will sometimes
9043 (notably in the TCP/IP mode) read more than a single protocol package
9044 off the network. When this happens, the extra package is stored
9045 by the subsystem. After calling <function>cs_get()</function>, and before
9046 waiting for more input, You should always call
9047 <function>cs_more()</function>
9048 to check if there's a full protocol package already read. If
9049 <function>cs_more()</function>
9051 <function>cs_get()</function>
9052 can be used to immediately fetch the new package. For the
9054 subsystem, the function should always return 0, but if you want your
9055 stuff to be protocol independent, you should use it.
9059 The <function>cs_more()</function>
9060 function is required because the RFC1729-method
9061 does not provide a way of separating individual PDUs, short of
9062 partially decoding the BER. Some other implementations will carefully
9063 nibble at the packet by calling
9064 <function>read(2)</function>
9065 several times. This was felt to be too inefficient (or at least
9066 clumsy) - hence the call for this extra function.
9070 int cs_look(COMSTACK handle);
9073 This function is useful when you're operating in nonblocking
9075 <function>select(2)</function>
9076 tells you there's something happening on the line. It returns one of
9077 the following values:
9081 <term>CS_NONE</term>
9083 No event is pending. The data found on the line was not a
9088 <term>CS_CONNECT</term>
9090 A response to your connect request has been received. Call
9091 <function>cs_rcvconnect</function>
9092 to process the event and to finalize the connection establishment.
9096 <term>CS_DISCON</term>
9098 The other side has closed the connection (or maybe sent a disconnect
9099 request - but do we care? Maybe later). Call
9100 <function>cs_close</function> to close your end of the association
9105 <term>CS_LISTEN</term>
9107 A connect request has been received.
9108 Call <function>cs_listen</function> to process the event.
9112 <term>CS_DATA</term>
9114 There's data to be found on the line.
9115 Call <function>cs_get</function> to get it.
9121 You should be aware that even if
9122 <function>cs_look()</function>
9123 tells you that there's an event event pending, the corresponding
9124 function may still return and tell you there was nothing to be found.
9125 This means that only part of a package was available for reading. The
9126 same event will show up again, when more data has arrived.
9130 int cs_fileno(COMSTACK h);
9133 Returns the file descriptor of the association. Use this when
9134 file-level operations on the endpoint are required
9135 (<function>select(2)</function> operations, specifically).
9139 <sect1 id="comstack.client">
9140 <title>Client Side</title>
9142 int cs_connect(COMSTACK handle, void *address);
9145 Initiate a connection with the target at <literal>address</literal>
9146 (more on addresses below). The function will return 0 on success, and 1 if
9147 the operation does not complete immediately (this will only
9148 happen on a nonblocking endpoint). In this case, use
9149 <function>cs_rcvconnect</function> to complete the operation,
9150 when <function>select(2)</function> or <function>poll(2)</function>
9151 reports input pending on the association.
9154 int cs_rcvconnect(COMSTACK handle);
9157 Complete a connect operation initiated by <function>cs_connect()</function>.
9158 It will return 0 on success; 1 if the operation has not yet completed (in
9159 this case, call the function again later); -1 if an error has occurred.
9162 <sect1 id="comstack.server">
9163 <title>Server Side</title>
9165 To establish a server under the <application>inetd</application>
9169 COMSTACK cs_createbysocket(int socket, CS_TYPE type, int blocking,
9173 The <literal>socket</literal> parameter is an established socket (when
9174 your application is invoked from <application>inetd</application>, the
9175 socket will typically be 0.
9176 The following parameters are identical to the ones for
9177 <function>cs_create</function>.
9180 int cs_bind(COMSTACK handle, void *address, int mode)
9183 Binds a local address to the endpoint. Read about addresses below. The
9184 <literal>mode</literal> parameter should be either
9185 <literal>CS_CLIENT</literal> or <literal>CS_SERVER</literal>.
9188 int cs_listen(COMSTACK handle, char *addr, int *addrlen);
9191 Call this to process incoming events on an endpoint that has been
9192 bound in listening mode. It will return 0 to indicate that the connect
9193 request has been received, 1 to signal a partial reception, and -1 to
9194 indicate an error condition.
9197 COMSTACK cs_accept(COMSTACK handle);
9200 This finalizes the server-side association establishment, after
9201 cs_listen has completed successfully. It returns a new connection
9202 endpoint, which represents the new association. The application will
9203 typically wish to fork off a process to handle the association at this
9204 point, and continue listen for new connections on the old
9205 <literal>handle</literal>.
9208 You can use the call
9211 const char *cs_addrstr(COMSTACK);
9214 on an established connection to retrieve the host-name of the remote host.
9218 You may need to use this function with some care if your
9219 name server service is slow or unreliable
9223 <sect1 id="comstack.addresses">
9224 <title>Addresses</title>
9226 The low-level format of the addresses are different depending on the
9227 mode of communication you have chosen. A function is provided by each
9228 of the lower layers to map a user-friendly string-form address to the
9229 binary form required by the lower layers.
9232 void *cs_straddr(COMSTACK handle, const char *str);
9235 The format for TCP/IP and SSL addresses is:
9238 <host> [ ':' <portnum> ]
9241 The <literal>hostname</literal> can be either a domain name or an
9242 IP address. The port number, if omitted, defaults to 210.
9245 For TCP/IP and SSL, the special hostnames <literal>@</literal>,
9246 maps to <literal>IN6ADDR_ANY_INIT</literal> with
9247 IPV4 binding as well (bindv6only=0),
9248 The special hostname <literal>@4</literal> binds to
9249 <literal>INADDR_ANY</literal> (IPV4 only listener).
9250 The special hostname <literal>@6</literal> binds to
9251 <literal>IN6ADDR_ANY_INIT</literal> with bindv6only=1 (IPV6 only listener).
9254 For UNIX sockets, the format of an address is the socket filename.
9257 When a connection has been established, you can use
9260 const char *cs_addrstr(COMSTACK h);
9263 to retrieve the host name of the peer system. The function returns
9264 a pointer to a static area, which is overwritten on the next call
9268 A fairly recent addition to the &comstack; module is the utility
9272 COMSTACK cs_create_host (const char *str, int blocking, void **vp);
9275 which is just a wrapper for <function>cs_create</function> and
9276 <function>cs_straddr</function>. The <parameter>str</parameter>
9277 is similar to that described for <function>cs_straddr</function>
9278 but with a prefix denoting the &comstack; type. Prefixes supported
9279 are <literal>tcp:</literal>, <literal>unix:</literal> and
9280 <literal>ssl:</literal> for TCP/IP, UNIX and SSL respectively.
9281 If no prefix is given, then TCP/IP is used.
9282 The <parameter>blocking</parameter> is passed to
9283 function <function>cs_create</function>. The third parameter
9284 <parameter>vp</parameter> is a pointer to &comstack; stack type
9286 Parameter <parameter>vp</parameter> is reserved for future use.
9287 Set it to <literal>NULL</literal>.
9290 <sect1 id="comstack.ssl">
9294 void *cs_get_ssl(COMSTACK cs);
9296 Returns the SSL handle, <literal>SSL *</literal> for comstack. If comstack
9297 is not of type SSL, NULL is returned.
9301 int cs_set_ssl_ctx(COMSTACK cs, void *ctx);
9303 Sets SSL context for comstack. The parameter is expected to be of type
9304 <literal>SSL_CTX *</literal>. This function should be called just
9305 after comstack has been created (before connect, bind, etc).
9306 This function returns 1 for success; 0 for failure.
9310 int cs_set_ssl_certificate_file(COMSTACK cs, const char *fname);
9312 Sets SSL certificate for comstack as a PEM file. This function
9313 returns 1 for success; 0 for failure.
9317 int cs_get_ssl_peer_certificate_x509(COMSTACK cs, char **buf, int *len);
9319 This function returns the peer certificate. If successful,
9320 <literal>*buf</literal> and <literal>*len</literal> holds
9321 X509 buffer and length respectively. Buffer should be freed
9322 with <literal>xfree</literal>. This function returns 1 for success;
9326 <sect1 id="comstack.diagnostics">
9327 <title>Diagnostics</title>
9329 All functions return -1 if an error occurs. Typically, the functions
9330 will return 0 on success, but the data exchange functions
9331 (<function>cs_get</function>, <function>cs_put</function>,
9332 <function>cs_more</function>) follow special rules. Consult their
9336 The error code for the COMSTACK can be retrieved using C macro
9337 <function>cs_errno</function> which will return one
9338 of the error codes <literal>CSYSERR</literal>,
9339 <literal>CSOUTSTATE</literal>,
9340 <literal>CSNODATA</literal>, ...
9343 int cs_errno(COMSTACK handle);
9346 You can the textual representation of the error code
9347 by using <function>cs_errmsg</function> - which
9348 works like <function>strerror(3)</function>
9351 const char *cs_errmsg(int n);
9354 It is also possible to get straight to the textual represenataion
9355 without the error code by using
9356 <function>cs_strerror</function>.
9359 const char *cs_strerror(COMSTACK h);
9362 <sect1 id="comstack.summary">
9363 <title>Summary and Synopsis</title>
9365 #include <yaz/comstack.h>
9367 #include <yaz/tcpip.h> /* this is for TCP/IP and SSL support */
9368 #include <yaz/unix.h> /* this is for UNIX socket support */
9370 COMSTACK cs_create(CS_TYPE type, int blocking, int protocol);
9372 COMSTACK cs_createbysocket(int s, CS_TYPE type, int blocking,
9374 COMSTACK cs_create_host(const char *str, int blocking,
9377 int cs_bind(COMSTACK handle, int mode);
9379 int cs_connect(COMSTACK handle, void *address);
9381 int cs_rcvconnect(COMSTACK handle);
9383 int cs_listen(COMSTACK handle);
9385 COMSTACK cs_accept(COMSTACK handle);
9387 int cs_put(COMSTACK handle, char *buf, int len);
9389 int cs_get(COMSTACK handle, char **buf, int *size);
9391 int cs_more(COMSTACK handle);
9393 void cs_close(COMSTACK handle);
9395 int cs_look(COMSTACK handle);
9397 void *cs_straddr(COMSTACK handle, const char *str);
9399 const char *cs_addrstr(COMSTACK h);
9404 <chapter id="future">
9405 <title>Future Directions</title>
9407 We have a new and better version of the front-end server on the drawing
9408 board. Resources and external commitments will govern when we'll be
9409 able to do something real with it. Features should include greater
9410 flexibility, greater support for access/resource control, and easy
9411 support for Explain (possibly with Zebra as an extra database engine).
9414 &yaz; is a BER toolkit and as such should support all protocols
9415 out there based on that. We'd like to see running ILL applications.
9416 It shouldn't be that hard. Another thing that would be interesting is
9417 LDAP. Maybe a generic framework for doing IR using both LDAP and
9418 Z39.50 transparently.
9421 The SOAP implementation is incomplete. In the future we hope
9422 to add more features to it. Perhaps make a WSDL/XML Schema compiler.
9423 The authors of libxml2 are already working on XML Schema / RelaxNG
9424 compilers so this may not be too hard.
9427 It would be neat to have a proper module mechanism for the Generic
9428 Frontend Server so that backend would be dynamically
9429 loaded (as shared objects / DLLs).
9432 Other than that, &yaz; generally moves in the directions which appear to
9433 make the most people happy (including ourselves, as prime users of the
9434 software). If there's something you'd like to see in here, then drop
9435 us a note and let's see what we can come up with.
9438 <reference id="reference">
9439 <title>Reference</title>
9440 <partintro id="reference-introduction">
9442 The material in this chapter is drawn directly from the individual
9448 <appendix id="list-oids">
9449 <title>List of Object Identifiers</title>
9451 These is a list of object identifiers that are built into YAZ.
9455 <appendix id="bib1-diagnostics">
9456 <title>Bib-1 diagnostics</title>
9458 List of Bib-1 diagnostics that are known to YAZ.
9462 <appendix id="sru-diagnostics">
9463 <title>SRU diagnostics</title>
9465 List of SRU diagnostics that are known to YAZ.
9469 <appendix id="license">
9470 <title>License</title>
9471 <sect1 id="license.indexdata">
9472 <title>Index Data Copyright</title>
9474 Copyright © ©right-year; Index Data.
9477 All rights reserved.
9480 Redistribution and use in source and binary forms, with or without
9481 modification, are permitted provided that the following conditions are met:
9486 Redistributions of source code must retain the above copyright
9487 notice, this list of conditions and the following disclaimer.
9492 Redistributions in binary form must reproduce the above copyright
9493 notice, this list of conditions and the following disclaimer in the
9494 documentation and/or other materials provided with the distribution.
9499 Neither the name of Index Data nor the names of its contributors
9500 may be used to endorse or promote products derived from this
9501 software without specific prior written permission.
9506 THIS SOFTWARE IS PROVIDED BY INDEX DATA ``AS IS'' AND ANY
9507 EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
9508 WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
9509 DISCLAIMED. IN NO EVENT SHALL INDEX DATA BE LIABLE FOR
9510 ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
9511 DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
9512 SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
9513 CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
9514 LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
9515 OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
9520 <appendix id="indexdata">
9521 <title>About Index Data</title>
9523 Index Data is a consulting and software-development enterprise that
9524 specializes in library and information management systems. Our
9525 interests and expertise span a broad range of related fields, and one
9526 of our primary, long-term objectives is the development of a powerful
9527 information management
9528 system with open network interfaces and hyper-media capabilities.
9530 We make this software available free of charge, on a fairly unrestrictive
9531 license; as a service to the networking community, and to further the
9532 development of quality software for open network communication.
9534 We'll be happy to answer questions about the software, and about ourselves
9540 <street>Amagerfælledvej 56</street>
9541 <postcode>2300 Copenhagen S</postcode>
9542 <country>Denmark</country>
9543 Email <email>info@indexdata.dk</email>
9547 The Hacker's Jargon File has the following to say about the
9549 prefix "YA" in the name of a software product.
9553 Yet Another. adj. 1. Of your own work: A
9554 humorous allusion often used in titles to acknowledge that the
9555 topic is not original, though the content is. As in "Yet Another
9556 AI Group" or "Yet Another Simulated Annealing Algorithm".
9558 others' work: Describes something of which there are already far
9563 <appendix id="credits">
9564 <title>Credits</title>
9566 This appendix lists individuals that have contributed in the development
9567 of &yaz;. Some have contributed with code, while others have provided bug
9568 fixes or suggestions. If we're missing somebody, of if you, for
9569 whatever reason, don't like to be listed here, let us know.
9579 Morten Bøgeskov
9600 Mads Bondo Dydensborg
9609 Morten Garkier Hendriksen
9666 Tom André Øverland
9672 <!-- Keep this comment at the end of the file
9675 nxml-child-indent: 1