1 <!-- $Id: frontend.xml,v 1.8 2001-10-26 20:13:44 adam Exp $ -->
2 <chapter id="server"><title>Generic server</title>
3 <sect1><title>Introduction</title>
6 If you aren't into documentation, a good way to learn how the
7 back end interface works is to look at the <filename>backend.h</filename>
8 file. Then, look at the small dummy-server in
9 <filename>ztest/ztest.c</filename>. The <filename>backend.h</filename>
10 file also makes a good reference, once you've chewed your way through
11 the prose of this file.
15 If you have a database system that you would like to make available by
16 means of Z39.50, &yaz; basically offers your two options. You
17 can use the APIs provided by the &asn;, &odr;, and &comstack;
19 create and decode PDUs, and exchange them with a client.
20 Using this low-level interface gives you access to all fields and
21 options of the protocol, and you can construct your server as close
22 to your existing database as you like.
23 It is also a fairly involved process, requiring
24 you to set up an event-handling mechanism, protocol state machine,
25 etc. To simplify server implementation, we have implemented a compact
26 and simple, but reasonably full-functioned server-frontend that will
27 handle most of the protocol mechanics, while leaving you to
28 concentrate on your database interface.
33 The backend interface was designed in anticipation of a specific
34 integration task, while still attempting to achieve some degree of
35 generality. We realize fully that there are points where the
36 interface can be improved significantly. If you have specific
37 functions or parameters that you think could be useful, send us a
38 mail (or better, sign on to the mailing list referred to in the
39 top-level README file). We will try to fit good suggestions into future
40 releases, to the extent that it can be done without requiring
41 too many structural changes in existing applications.
46 <sect1 id="server.frontend"><title>The Database Frontend</title>
49 We refer to this software as a generic database frontend. Your
50 database system is the <emphasis>backend database</emphasis>, and the
51 interface between the two is called the <emphasis>backend API</emphasis>.
52 The backend API consists of a small number of function handlers and
53 structure definitions. You are required to provide the
54 <function>main()</function> routine for the server (which can be
55 quite simple), as well as a set of handlers to match each of the prototypes.
56 The interface functions that you write can use any mechanism you like
57 to communicate with your database system: You might link the whole
58 thing together with your database application and access it by
59 function calls; you might use IPC to talk to a database server
60 somewhere; or you might link with third-party software that handles
61 the communication for you (like a commercial database client library).
62 At any rate, the handlers will perform the tasks of:
80 Scanning the database index (optional - if you wish to implement SCAN).
84 Extended Services (optional).
88 Result-Set Delete (optional).
92 Result-Set Sort (optional).
98 (more functions will be added in time to support as much of
99 Z39.50-1995 as possible).
103 <sect1 id="server.backend"><title>The Backend API</title>
106 The header file that you need to use the interface are in the
107 <filename>include/yaz</filename> directory. It's called
108 <filename>backend.h</filename>. It will include other files from
109 the <filename>include/yaz</filename> directory, so you'll
110 probably want to use the -I option of your compiler to tell it
111 where to find the files. When you run
112 <literal>make</literal> in the top-level &yaz; directory,
113 everything you need to create your server is to link with the
114 <filename>lib/libyaz.la</filename> library.
118 <sect1 id="server.main"><title>Your main() Routine</title>
121 As mentioned, your <function>main()</function> routine can be quite brief.
122 If you want to initialize global parameters, or read global configuration
123 tables, this is the place to do it. At the end of the routine, you should
128 int statserv_main(int argc, char **argv,
129 bend_initresult *(*bend_init)(bend_initrequest *r),
130 void (*bend_close)(void *handle));
134 The third and fourth arguments are pointers to handlers. Handler
135 <function>bend_init</function> is called whenever the server receives
136 an Initialize Request, so it serves as a Z39.50 session initializer. The
137 <function>bend_close</function> handler is called when the session is
142 <function>statserv_main</function> will establish listening sockets
143 according to the parameters given. When connection requests are received,
144 the event handler will typically <function>fork()</function> and
145 create a sub-process to handle a new connection.
146 Alternatively the server may be setup to create threads for each
148 If you do use global variables and forking, you should be aware, then,
149 that these cannot be shared between associations, unless you explicitly
150 disable forking by command line parameters.
154 The server provides a mechanism for controlling some of its behavior
155 without using command-line options. The function
159 statserv_options_block *statserv_getcontrol(void);
163 will return a pointer to a <literal>struct statserv_options_block</literal>
164 describing the current default settings of the server. The structure
165 contains these elements:
169 <literal>int dynamic</literal></term><listitem><para>
170 A boolean value, which determines whether the server
171 will fork on each incoming request (TRUE), or not (FALSE). Default is
172 TRUE. This flag is only read by UNIX-based servers (WIN32 based servers
174 </para></listitem></varlistentry>
177 <literal>int threads</literal></term><listitem><para>
178 A boolean value, which determines whether the server
179 will create a thread on each incoming request (TRUE), or not (FALSE).
180 Default is FALSE. This flag is only read by UNIX-based servers
181 that offer POSIX Threads support.
182 WIN32-based servers always operate in threaded mode.
183 </para></listitem></varlistentry>
186 <literal>int inetd</literal></term><listitem><para>
187 A boolean value, which determines whether the server
188 will operates under a UNIX INET daemon (inetd). Default is FALSE.
189 </para></listitem></varlistentry>
192 <literal>int loglevel</literal></term><listitem><para>
193 Set this by ORing the constants defined in
194 <filename>include/yaz/yaz-log.h</filename>.
195 </para></listitem></varlistentry>
198 <literal>char logfile[ODR_MAXNAME+1]</literal></term>
199 <listitem><para>File for diagnostic output ("": stderr).
200 </para></listitem></varlistentry>
203 <literal>char apdufile[ODR_MAXNAME+1]</literal></term>
205 Name of file for logging incoming and outgoing APDUs
206 ("": don't log APDUs, "-":
207 <literal>stderr</literal>).
208 </para></listitem></varlistentry>
211 <literal>char default_listen[1024]</literal></term>
212 <listitem><para>Same form as the command-line specification of
213 listener address. "": no default listener address.
214 Default is to listen at "tcp:@:9999". You can only
215 specify one default listener address in this fashion.
216 </para></listitem></varlistentry>
219 <literal>enum oid_proto default_proto;</literal></term>
220 <listitem><para>Either <literal>PROTO_SR</literal> or
221 <literal>PROTO_Z3950</literal>.
222 Default is <literal>PROTO_Z39_50</literal>.
223 </para></listitem></varlistentry>
226 <literal>int idle_timeout;</literal></term>
227 <listitem><para>Maximum session idletime, in minutes. Zero indicates
228 no (infinite) timeout. Default is 120 minutes.
229 </para></listitem></varlistentry>
232 <literal>int maxrecordsize;</literal></term>
233 <listitem><para>Maximum permissible record (message) size. Default
234 is 1Mb. This amount of memory will only be allocated if a
235 client requests a very large amount of records in one operation
237 Set it to a lower number if you are worried about resource
238 consumption on your host system.
239 </para></listitem></varlistentry>
242 <literal>char configname[ODR_MAXNAME+1]</literal></term>
243 <listitem><para>Passed to the backend when a new connection is received.
244 </para></listitem></varlistentry>
247 <literal>char setuid[ODR_MAXNAME+1]</literal></term>
248 <listitem><para>Set user id to the user specified, after binding
249 the listener addresses.
250 </para></listitem></varlistentry>
253 <literal>void (*bend_start)(struct statserv_options_block *p)</literal>
255 <listitem><para>Pointer to function which is called after the
256 command line options have been parsed - but before the server
258 For forked UNIX servers this handler is called in the mother
259 process; for threaded servers this handler is called in the
261 The default value of this pointer is NULL in which case it
262 isn't invoked by the frontend server.
263 When the server operates as an NT service this handler is called
264 whenever the service is started.
265 </para></listitem></varlistentry>
268 <literal>void (*bend_stop)(struct statserv_options_block *p)</literal>
270 <listitem><para>Pointer to function which is called whenever the server
271 has stopped listening for incoming connections. This function pointer
272 has a default value of NULL in which case it isn't called.
273 When the server operates as an NT service this handler is called
274 whenever the service is stopped.
275 </para></listitem></varlistentry>
278 <literal>void *handle</literal></term>
279 <listitem><para>User defined pointer (default value NULL).
280 This is a per-server handle that can be used to specify "user-data".
281 Do not confuse this with the session-handle as returned by bend_init.
282 </para></listitem></varlistentry>
288 The pointer returned by <literal>statserv_getcontrol</literal> points to
289 a static area. You are allowed to change the contents of the structure,
290 but the changes will not take effect before you call
294 void statserv_setcontrol(statserv_options_block *block);
299 that you should generally update this structure before calling
300 <function>statserv_main()</function>.
305 <sect1 id="server.backendfunctions"><title>The Backend Functions</title>
308 For each service of the protocol, the backend interface declares one or
309 two functions. You are required to provide implementations of the
310 functions representing the services that you wish to implement.
313 <sect2><title>Init</title>
316 bend_initresult (*bend_init)(bend_initrequest *r);
320 This handler is called once for each new connection request, after
321 a new process/thread has been created, and an Initialize Request has
322 been received from the client. The pointer to the
323 <function>bend_init</function> handler is passed in the call to
324 <function>statserv_start</function>.
327 Unlike previous versions of YAZ, the <function>bend_init</function> also
328 serves as a handler that defines the Z39.50 services that the backend
329 wish to support. Pointers to <emphasis>all</emphasis> service handlers,
330 including search - and fetch must be specified here in this handler.
333 The request - and result structures are defined as
337 typedef struct bend_initrequest
339 Z_IdAuthentication *auth;
340 ODR stream; /* encoding stream */
341 ODR print; /* printing stream */
342 Z_ReferenceId *referenceId;/* reference ID */
343 char *peer_name; /* dns host of peer (client) */
345 char *implementation_name;
346 char *implementation_version;
347 int (*bend_sort) (void *handle, bend_sort_rr *rr);
348 int (*bend_search) (void *handle, bend_search_rr *rr);
349 int (*bend_fetch) (void *handle, bend_fetch_rr *rr);
350 int (*bend_present) (void *handle, bend_present_rr *rr);
351 int (*bend_esrequest) (void *handle, bend_esrequest_rr *rr);
352 int (*bend_delete)(void *handle, bend_delete_rr *rr);
353 int (*bend_scan)(void *handle, bend_scan_rr *rr);
354 int (*bend_segment)(void *handle, bend_segment_rr *rr);
357 typedef struct bend_initresult
359 int errcode; /* 0==OK */
360 char *errstring; /* system error string or NULL */
361 void *handle; /* private handle to the backend module */
366 In general, the server frontend expects that the
367 <literal>bend_*result</literal> pointer that you return is valid at
368 least until the next call to a <literal>bend_* function</literal>.
369 This applies to all of the functions described herein. The parameter
370 structure passed to you in the call belongs to the server frontend, and
371 you should not make assumptions about its contents after the current
372 function call has completed. In other words, if you want to retain any
373 of the contents of a request structure, you should copy them.
377 The <literal>errcode</literal> should be zero if the initialization of
378 the backend went well. Any other value will be interpreted as an error.
379 The <literal>errstring</literal> isn't used in the current version, but
380 one option would be to stick it in the initResponse as a VisibleString.
381 The <literal>handle</literal> is the most important parameter. It should
382 be set to some value that uniquely identifies the current session to
383 the backend implementation. It is used by the frontend server in any
384 future calls to a backend function.
385 The typical use is to set it to point to a dynamically allocated state
386 structure that is private to your backend module.
390 The <literal>auth</literal> member holds the authentication information
391 part of the Z39.50 Initialize Request. Interpret this if your serves
392 requires authentication.
396 The members <literal>peer_name</literal>,
397 <literal>implementation_name</literal> and
398 <literal>implementation_version</literal> holds DNS of client, name
399 of client (Z39.50) implementation - and version.
403 The <literal>bend_</literal> - members are set to NULL when
404 <function>bend_init</function> is called. Modify the pointers by
405 setting them to point to backend functions.
410 <sect2><title>Search and retrieve</title>
412 <para>We now describe the handlers that are required to support search -
413 and retrieve. You must support two functions - one for search - and one
414 for fetch (retrieval of one record). If desirable you can provide a
415 third handler which is called when a present request is received which
416 allows you to optimize retrieval of multiple-records.
420 int (*bend_search) (void *handle, bend_search_rr *rr);
423 char *setname; /* name to give to this set */
424 int replace_set; /* replace set, if it already exists */
425 int num_bases; /* number of databases in list */
426 char **basenames; /* databases to search */
427 Z_ReferenceId *referenceId;/* reference ID */
428 Z_Query *query; /* query structure */
429 ODR stream; /* encode stream */
430 ODR decode; /* decode stream */
431 ODR print; /* print stream */
433 bend_request request;
434 bend_association association;
436 int hits; /* number of hits */
437 int errcode; /* 0==OK */
438 char *errstring; /* system error string or NULL */
444 The <function>bend_search</function> handler is a fairly close
445 approximation of a protocol Search Request - and Response PDUs
446 The <literal>setname</literal> is the resultSetName from the protocol.
447 You are required to establish a mapping between the set name and whatever
448 your backend database likes to use.
449 Similarly, the <literal>replace_set</literal> is a boolean value
450 corresponding to the resultSetIndicator field in the protocol.
451 <literal>num_bases/basenames</literal> is a length of/array of character
452 pointers to the database names provided by the client.
453 The <literal>query</literal> is the full query structure as defined in
454 the protocol ASN.1 specification.
455 It can be either of the possible query types, and it's up to you to
456 determine if you can handle the provided query type.
457 Rather than reproduce the C interface here, we'll refer you to the
458 structure definitions in the file
459 <filename>include/yaz/z-core.h</filename>. If you want to look at the
460 attributeSetId OID of the RPN query, you can either match it against
461 your own internal tables, or you can use the
462 <literal>oid_getentbyoid</literal> function provided by &yaz;.
466 The structure contains a number of hits, and an
467 <literal>errcode/errstring</literal> pair. If an error occurs
468 during the search, or if you're unhappy with the request, you should
469 set the errcode to a value from the BIB-1 diagnostic set. The value
470 will then be returned to the user in a nonsurrogate diagnostic record
471 in the response. The <literal>errstring</literal>, if provided, will
472 go in the addinfo field. Look at the protocol definition for the
473 defined error codes, and the suggested uses of the addinfo field.
478 int (*bend_fetch) (void *handle, bend_fetch_rr *rr);
480 typedef struct bend_fetch_rr {
481 char *setname; /* set name */
482 int number; /* record number */
483 Z_ReferenceId *referenceId;/* reference ID */
484 oid_value request_format; /* One of the CLASS_RECSYN members */
485 int *request_format_raw; /* same as above (raw OID) */
486 Z_RecordComposition *comp; /* Formatting instructions */
487 ODR stream; /* encoding stream - memory source if req */
488 ODR print; /* printing stream */
490 char *basename; /* name of database that provided record */
491 int len; /* length of record or -1 if structured */
492 char *record; /* record */
493 int last_in_set; /* is it? */
494 oid_value output_format; /* format */
495 int *output_format_raw; /* used instead of above if not-null */
496 int errcode; /* 0==success */
497 char *errstring; /* system error string or NULL */
498 int surrogate_flag; /* surrogate diagnostic */
503 The frontend server calls the <function>bend_fetch</function> handler
504 when it needs database records to fulfill a Search Request or a Present
506 The <literal>setname</literal> is simply the name of the result set
507 that holds the reference to the desired record.
508 The <literal>number</literal> is the offset into the set (with 1
509 being the first record in the set). The <literal>format</literal> field
510 is the record format requested by the client (See section
511 <link linkend="oid">Object Identifiers</link>). The value
512 <literal>VAL_NONE</literal> indicates that the client did not
513 request a specific format. The <literal>stream</literal> argument
514 is an &odr; stream which should be used for
515 allocating space for structured data records.
516 The stream will be reset when all records have been assembled, and
517 the response package has been transmitted.
518 For unstructured data, the backend is responsible for maintaining a
519 static or dynamic buffer for the record between calls.
523 In the structure, the <literal>basename</literal> is the name of the
524 database that holds the
525 record. <literal>len</literal> is the length of the record returned, in
526 bytes, and <literal>record</literal> is a pointer to the record.
527 <literal>Last_in_set</literal> should be nonzero only if the record
528 returned is the last one in the given result set.
529 <literal>errcode</literal> and <literal>errstring</literal>, if
530 given, will be interpreted as a global error pertaining to the
531 set, and will be returned in a non-surrogate-diagnostic.
532 If you wish to return the error as a surrogate-diagnostic
533 (local error) you can do this by setting
534 <literal>surrogate_flag</literal> to 1 also.
538 If the <literal>len</literal> field has the value -1, then
539 <literal>record</literal> is assumed to point to a constructed data
540 type. The <literal>format</literal> field will be used to determine
541 which encoder should be used to serialize the data.
546 If your backend generates structured records, it should use
547 <function>odr_malloc()</function> on the provided stream for allocating
548 data: This allows the frontend server to keep track of the record sizes.
553 The <literal>format</literal> field is mapped to an object identifier
554 in the direct reference of the resulting EXTERNAL representation
560 The current version of &yaz; only supports the direct reference mode.
565 int (*bend_present) (void *handle, bend_present_rr *rr);
568 char *setname; /* set name */
570 int number; /* record number */
571 oid_value format; /* One of the CLASS_RECSYN members */
572 Z_ReferenceId *referenceId;/* reference ID */
573 Z_RecordComposition *comp; /* Formatting instructions */
574 ODR stream; /* encoding stream */
575 ODR print; /* printing stream */
576 bend_request request;
577 bend_association association;
579 int hits; /* number of hits */
580 int errcode; /* 0==OK */
581 char *errstring; /* system error string or NULL */
586 The <function>bend_present</function> handler is called when
587 the server receives a Present Request. The <literal>setname</literal>,
588 <literal>start</literal> and <literal>number</literal> is the
589 name of the result set - start position - and number of records to
590 be retrieved respectively. <literal>format</literal> and
591 <literal>comp</literal> is the preferred transfer syntax and element
592 specifications of the present request.
595 Note that this is handler serves as a supplement for
596 <function>bend_fetch</function> and need not to be defined in order to
597 support search - and retrieve.
602 <sect2><title>Delete</title>
605 For back-ends that supports delete of a result set only one handler
610 int (*bend_delete)(void *handle, bend_delete_rr *rr);
612 typedef struct bend_delete_rr {
616 Z_ReferenceId *referenceId;
617 int delete_status; /* status for the whole operation */
618 int *statuses; /* status each set - indexed as setnames */
626 The delete set function definition is rather primitive, mostly because
627 we have had no practical need for it as of yet. If someone wants
628 to provide a full delete service, we'd be happy to add the
629 extra parameters that are required. Are there clients out there
630 that will actually delete sets they no longer need?
636 <sect2><title>scan</title>
639 For servers that wish to offer the scan service one handler
644 int (*bend_delete)(void *handle, bend_delete_rr *rr);
647 BEND_SCAN_SUCCESS, /* ok */
648 BEND_SCAN_PARTIAL /* not all entries could be found */
651 typedef struct bend_scan_rr {
652 int num_bases; /* number of elements in databaselist */
653 char **basenames; /* databases to search */
654 oid_value attributeset;
655 Z_ReferenceId *referenceId; /* reference ID */
656 Z_AttributesPlusTerm *term;
657 ODR stream; /* encoding stream - memory source if required */
658 ODR print; /* printing stream */
660 int *step_size; /* step size */
661 int term_position; /* desired index of term in result list/returned */
662 int num_entries; /* number of entries requested/returned */
664 struct scan_entry *entries;
665 bend_scan_status status;
673 <sect1 id="server.invocation"><title>Application Invocation</title>
676 The finished application has the following
677 invocation syntax (by way of <function>statserv_main()</function>):
681 <replaceable>appname</replaceable> [-szSiTu -a <replaceable>apdufile</replaceable> -l <replaceable>logfile</replaceable> -v <replaceable>loglevel</replaceable> -c <replaceable>config</replaceable>]
682 [listener ...]
690 <varlistentry><term>-a <replaceable>file</replaceable></term>
692 Specify a file for dumping PDUs (for diagnostic purposes).
693 The special name "-" sends output to
694 <literal>stderr</literal>.
695 </para></listitem></varlistentry>
697 <varlistentry><term>-S</term>
699 Don't fork or make threads on connection requests. This is good for
700 debugging, but not recommended for real operation: Although the
701 server is asynchronous and non-blocking, it can be nice to keep
702 a software malfunction (okay then, a crash) from affecting all
704 </para></listitem></varlistentry>
706 <varlistentry><term>-T</term>
708 Operate the server in threaded mode. The server creates a thread
709 for each connection rather than a fork a process. Only available
710 on UNIX systems that offers POSIX threads.
711 </para></listitem></varlistentry>
713 <varlistentry><term>-s</term>
715 Use the SR protocol (obsolete).
716 </para></listitem></varlistentry>
718 <varlistentry><term>-z</term>
720 Use the Z39.50 protocol (default). These two options complement
721 each other. You can use both multiple times on the same command
722 line, between listener-specifications (see below). This way, you
723 can set up the server to listen for connections in both protocols
724 concurrently, on different local ports.
725 </para></listitem></varlistentry>
727 <varlistentry><term>-l <replaceable>file</replaceable></term>
728 <listitem><para>The logfile.
729 </para></listitem></varlistentry>
731 <varlistentry><term>-c <replaceable>config</replaceable></term>
732 <listitem><para>A user option that serves as a specifier for some
733 sort of configuration, e.g. a filename.
734 The argument to this option is transferred to member
735 <literal>configname</literal>of the
736 <literal>statserv_options_block</literal>.
737 </para></listitem></varlistentry>
739 <varlistentry><term>-v <replaceable>level</replaceable></term>
741 The log level. Use a comma-separated list of members of the set
742 {fatal,debug,warn,log,all,none}.
743 </para></listitem></varlistentry>
745 <varlistentry><term>-u <replaceable>userid</replaceable></term>
747 Set user ID. Sets the real UID of the server process to that of the
748 given user. It's useful if you aren't comfortable with having the
749 server run as root, but you need to start it as such to bind a
751 </para></listitem></varlistentry>
753 <varlistentry><term>-w <replaceable>dir</replaceable></term>
756 </para></listitem></varlistentry>
758 <varlistentry><term>-i</term>
760 Use this when running from the <application>inetd</application> server.
761 </para></listitem></varlistentry>
763 <varlistentry><term>-t <replaceable>minutes</replaceable></term>
765 Idle session timeout, in minutes.
766 </para></listitem></varlistentry>
768 <varlistentry><term>-k <replaceable>size</replaceable></term>
770 Maximum record size/message size, in kilobytes.
771 </para></listitem></varlistentry>
777 A listener specification consists of a transport mode followed by a
778 colon (:) followed by a listener address. The transport mode is
779 either <literal>osi</literal> or <literal>tcp</literal>.
783 For TCP, an address has the form
787 hostname | IP-number [: portnumber]
791 The port number defaults to 210 (standard Z39.50 port).
795 For osi, the address form is
799 [t-selector /] hostname | IP-number [: portnumber]
803 The transport selector is given as a string of hex digits (with an even
804 number of digits). The default port number is 102 (RFC1006 port).
814 osi:0402/dbserver.osiworld.com:3000
818 In both cases, the special hostname "@" is mapped to
819 the address INADDR_ANY, which causes the server to listen on any local
820 interface. To start the server listening on the registered ports for
821 Z39.50 and SR over OSI/RFC1006, and to drop root privileges once the
822 ports are bound, execute the server like this (from a root shell):
826 my-server -u daemon tcp:@ -s osi:@
830 You can replace <literal>daemon</literal> with another user, eg. your
831 own account, or a dedicated IR server account.
832 <literal>my-server</literal> should be the name of your
833 server application. You can test the procedure with the
834 <application>yaz-ztest</application> application.
840 <!-- Keep this comment at the end of the file
845 sgml-minimize-attributes:nil
846 sgml-always-quote-attributes:t
849 sgml-parent-document: "yaz.xml"
850 sgml-local-catalogs: nil
851 sgml-namecase-general:t