+ </example>
+ </section>
+
+ </chapter>
+
+ <chapter id="using">
+ <title>Using Pazpar2</title>
+ <para>
+ This chapter provides a general introduction to the use and
+ deployment of Pazpar2.
+ </para>
+
+ <section id="architecture">
+ <title>Pazpar2 and your systems architecture</title>
+ <para>
+ Pazpar2 is designed to provide asynchronous, behind-the-scenes
+ metasearching functionality to your application, exposing this
+ functionality using a simple webservice API that can be accessed
+ from any number of development environments. In particular, it is
+ possible to combine Pazpar2 either with your server-side dynamic
+ website scripting, with scripting or code running in the browser, or
+ with any combination of the two. Pazpar2 is an excellent tool for
+ building advanced, Ajax-based user interfaces for metasearch
+ functionality, but it isn't a requirement -- you can choose to use
+ Pazpar2 entirely as a backend to your regular server-side scripting.
+ When you do use Pazpar2 in conjunction
+ with browser scripting (JavaScript/Ajax, Flash, applets,
+ etc.), there are special considerations.
+ </para>
+
+ <para>
+ Pazpar2 implements a simple but efficient HTTP server, and it is
+ designed to interact directly with scripting running in the browser
+ for the best possible performance, and to limit overhead when
+ several browser clients generate numerous webservice requests.
+ However, it is still desirable to use a conventional webserver,
+ such as Apache, to serve up graphics, HTML documents, and
+ server-side scripting. Because the security sandbox environment of
+ most browser-side programming environments only allows communication
+ with the server from which the enclosing HTML page or object
+ originated, Pazpar2 is designed so that it can act as a transparent
+ proxy in front of an existing webserver (see <xref
+ linkend="pazpar2_conf"/> for details).
+ In this mode, all regular
+ HTTP requests are transparently passed through to your webserver,
+ while Pazpar2 only intercepts search-related webservice requests.
+ </para>
+
+ <para>
+ If you want to expose your combined service on port 80, you can
+ either run your regular webserver on a different port, a different
+ server, or a different IP address associated with the same server.
+ </para>
+
+ <para>
+ Pazpar2 can also work behind
+ a reverse Proxy. Refer to <xref linkend="installation.apache2proxy"/>)
+ for more information.
+ This allows your existing HTTP server to operate on port 80 as usual.
+ Pazpar2 can be started on another (internal) port.
+ </para>
+
+ <para>
+ Sometimes, it may be necessary to implement functionality on your
+ regular webserver that makes use of search results, for example to
+ implement data import functionality, emailing results, history
+ lists, personal citation lists, interlibrary loan functionality,
+ etc. Fortunately, it is simple to exchange information between
+ Pazpar2, your browser scripting, and backend server-side scripting.
+ You can send a session ID and possibly a record ID from your browser
+ code to your server code, and from there use Pazpar2s webservice API
+ to access result sets or individual records. You could even 'hide'
+ all of Pazpar2s functionality between your own API implemented on
+ the server-side, and access that from the browser or elsewhere. The
+ possibilities are just about endless.
+ </para>
+ </section>
+
+ <section id="data_model">
+ <title>Your data model</title>
+ <para>
+ Pazpar2 does not have a preconceived model of what makes up a data
+ model. There are no assumptions that records have specific fields or
+ that they are organized in any particular way. The only assumption
+ is that data comes packaged in a form that the software can work
+ with (presently, that means XML or MARC), and that you can provide
+ the necessary information to massage it into Pazpar2's internal
+ record abstraction.
+ </para>
+
+ <para>
+ Handling retrieval records in Pazpar2 is a two-step process. First,
+ you decide which data elements of the source record you are
+ interested in, and you specify any desired massaging or combining of
+ elements using an XSLT stylesheet (MARC records are automatically
+ normalized to <ulink url="&url.marcxml;">MARCXML</ulink> before this step).
+ If desired, you can run multiple XSLT stylesheets in series to accomplish
+ this, but the output of the last one should be a representation of the
+ record in a schema that Pazpar2 understands.
+ </para>
+
+ <para>
+ The intermediate, internal representation of the record looks like
+ this:
+ <screen><![CDATA[
+ <record xmlns="http://www.indexdata.com/pazpar2/1.0"
+ mergekey="title The Shining author King, Stephen">
+
+ <metadata type="title" rank="2">The Shining</metadata>
+
+ <metadata type="author">King, Stephen</metadata>
+
+ <metadata type="kind">ebook</metadata>
+
+ <!-- ... and so on -->
+ </record>
+ ]]></screen>
+
+ As you can see, there isn't much to it. There are really only a few
+ important elements to this file.
+ </para>
+
+ <para>
+ Elements should belong to the namespace
+ <literal>http://www.indexdata.com/pazpar2/1.0</literal>.
+ If the root node contains the
+ attribute 'mergekey', then every record that generates the same
+ merge key (normalized for case differences, white space, and
+ truncation) will be joined into a cluster. In other words, you
+ decide how records are merged. If you don't include a merge key,
+ records are never merged. The 'metadata' elements provide the meat
+ of the elements -- the content. the 'type' attribute is used to
+ match each element against processing rules that determine what
+ happens to the data element next. The attribute, 'rank' specifies
+ specifies a multipler for ranking for this element.
+ </para>
+
+ <para>
+ The next processing step is the extraction of metadata from the
+ intermediate representation of the record. This is governed by the
+ 'metadata' elements in the 'service' section of the configuration
+ file. See <xref linkend="config-server"/> for details. The metadata
+ in the retrieval record ultimately drives merging, sorting, ranking,
+ the extraction of browse facets, and display, all configurable.
+ </para>
+ </section>
+
+ <section id="client">
+ <title>Client development overview</title>
+ <para>
+ You can use Pazpar2 from any environment that allows you to use
+ webservices. The initial goal of the software was to support
+ Ajax-based applications, but there literally are no limits to what
+ you can do. You can use Pazpar2 from Javascript, Flash, Java, etc.,
+ on the browser side, and from any development environment on the
+ server side, and you can pass session tokens and record IDs freely
+ around between these environments to build sophisticated applications.
+ Use your imagination.
+ </para>
+
+ <para>
+ The webservice API of Pazpar2 is described in detail in <xref
+ linkend="pazpar2_protocol"/>.
+ </para>
+
+ <para>
+ In brief, you use the 'init' command to create a session, a
+ temporary workspace which carries information about the current
+ search. You start a new search using the 'search' command. Once the
+ search has been started, you can follow its progress using the
+ 'stat', 'bytarget', 'termlist', or 'show' commands. Detailed records
+ can be fetched using the 'record' command.
+ </para>
+ </section>
+
+ §-ajaxdev;
+
+ <section id="nonstandard">
+ <title>Connecting to non-standard resources</title>
+ <para>
+ Pazpar2 uses Z39.50 as its switchboard language -- i.e. as far as it
+ is concerned, all resources speak Z39.50, its webservices derivatives,
+ SRU/SRW and SOLR servers exposing Lucene indexes. It is, however, equipped
+ to handle a broad range of different server behavior, through
+ configurable query mapping and record normalization. If you develop
+ configuration, stylesheets, etc., for a new type of resources, we
+ encourage you to share your work. But you can also use Pazpar2 to
+ connect to hundreds of resources that do not support standard
+ protocols.
+ </para>
+
+ <para>
+ For a growing number of resources, Z39.50 is all you need. Over the
+ last few years, a number of commercial, full-text resources have
+ implemented Z39.50. These can be used through Pazpar2 with little or
+ no effort. Resources that use non-standard record formats will
+ require a bit of XSLT work, but that's all.
+ </para>
+
+ <para>
+ But what about resources that don't support Z39.50 at all?
+ Some resources might support OpenSearch, private, XML/HTTP-based
+ protocols, or something else entirely.
+ Some databases exist only as web user interfaces and
+ will require screen-scraping. Still others exist only as static
+ files, or perhaps as databases supporting the OAI-PMH protocol.
+ There is hope! Read on.
+ </para>
+
+ <para>
+ Index Data continues to advocate the support of open standards. We
+ work with database vendors to support standards, so you don't have
+ to worry about programming against non-standard services. We also
+ provide tools (see <ulink
+ url="http://www.indexdata.com/simpleserver">SimpleServer</ulink>)
+ which make it comparatively easy to build gateways against servers
+ with non-standard behavior. Again, we encourage you to share any
+ work you do in this direction.
+ </para>
+
+ <para>
+ But the bottom line is that working with non-standard resources in
+ metasearching is really, really hard. If you want to build a
+ project with Pazpar2, and you need access to resources with
+ non-standard interfaces, we can help. We run gateways to more than
+ 2,000 popular, commercial databases and other resources,
+ making it simple
+ to plug them directly into Pazpar2. For a small annual fee per
+ database, we can help you establish connections to your licensed
+ resources. Meanwhile, you can help! If you build your own
+ standards-compliant gateways, host them for others, or share the
+ code! And tell your vendors that they can save everybody money and
+ increase the appeal of their resources by supporting standards.
+ </para>
+
+ <para>
+ There are those who will ask us why we are using Z39.50 as our
+ switchboard language rather than a different protocol. Basically,
+ we believe that Z39.50 is presently the most widely implemented
+ information retrieval protocol that has the level of functionality
+ required to support a good metasearching experience (structured
+ searching, structured, well-defined results). It is also compact and
+ efficient, and there is a very broad range of tools available to
+ implement it.
+ </para>
+ </section>
+
+ <section id="unicode">
+ <title>Unicode Compliance</title>
+ <para>
+ Pazpar2 is Unicode compliant and language and locale aware but relies
+ on character encoding for the targets to be specified correctly if
+ the targets themselves are not UTF-8 based (most aren't).
+ Just a few bad behaving targets can spoil the search experience
+ considerably if for example Greek, Russian or otherwise non 7-bit ASCII
+ search terms are entered. In these cases some targets return
+ records irrelevant to the query, and the result screens will be
+ cluttered with noise.
+ </para>
+ <para>
+ While noise from misbehaving targets can not be removed, it can
+ be reduced using truly Unicode based ranking. This is an
+ option which is available to the system administrator if ICU
+ support is compiled into Pazpar2, see
+ <xref linkend="installation"/> for details.
+ </para>
+ <para>
+ In addition, the ICU tokenization and normalization rules must
+ be defined in the master configuration file described in
+ <xref linkend="config-server"/>.
+ </para>
+ </section>
+
+ <section id="load_balancing">
+ <title>Load balancing</title>
+ <para>
+ Just like any web server, Pazpar2, can be load balanced by a standard
+ hardware or software load balancer as long as the session stickiness
+ is ensured. If you are already running the Apache2 web server in front
+ of Pazpar2 and use the apache mod_proxy module to 'relay' client
+ requests to Pazpar2, this set up can be easily extended to include
+ load balancing capabilites.
+ To do so you need to enable the
+ <ulink url="http://httpd.apache.org/docs/2.2/mod/mod_proxy_balancer.html">
+ mod_proxy_balance
+ </ulink>
+ module in your Apache2 installation.
+ </para>
+
+ <para>
+ On a Debian based Apache 2 system, the relevant modules can
+ be enabled with: