1 <chapter id="introduction">
2 <!-- $Id: introduction.xml,v 1.11 2002-08-29 14:05:11 mike Exp $ -->
3 <title>Introduction</title>
6 <title>Overview</title>
9 <ulink url="http://www.indexdata.dk/zebra/">
11 is a high-performance, general-purpose structured text
12 indexing and retrieval engine. It reads structured records in a
13 variety of input formats (eg. email, XML, MARC) and provides access
14 to them through a powerful combination of boolean search
15 expressions and relevance-ranked free-text queries.
19 Zebra supports large databases (tens of millions of records,
20 tens of gigabytes of data). It allows safe, incremental
21 database updates on live systems. Because Zebra supports
22 the industry-standard information retrieval protocol, Z39.50,
23 you can search Zebra databases using an enormous variety of
24 programs and toolkits, both commercial and free, which understand
25 this protocol. Application libraries are available to allow
26 bespoke clients to be written in Perl, C, C++, Java, Tcl, Visual
27 Basic, Python, PHP and more - see
28 <ulink url="http://zoom.z3950.org/">the ZOOM web site</ulink>
29 for more information on some of these client toolkits.
33 This document is an introduction to the Zebra system. It explains
34 how to compile the software, how to prepare your first database,
35 and how to configure the server to give you the
36 functionality that you need.
40 If you use Zebra, you should visit its
41 <ulink url="http://www.indexdata.dk/zebra/">web site</ulink>,
42 where you can join the
43 <ulink url="http://www.indexdata.dk/mailman/listinfo/zebralist">
46 <email>### zebra-subscribe@mailman.indexdata.dk</email>
52 <title>Features</title>
55 This is an overview of some of Zebra's most important features:
63 Very large databases: files for indexes, etc. can be
64 automatically partitioned over multiple disks.
70 Arbitrarily complex records. The internal data format
71 is an structured format conceptually similar to XML or GRS-1,
72 which allows nested structured data elements and
73 variant forms of data.
79 Robust updating - records can be added and deleted ``on the fly''
80 without rebuilding the index from scratch.
81 Records can be safely updated even while users are accessing
83 The update procedure is tolerant to crashes or hard interrupts
84 during database updating - data can be reconstructed following
91 Configurable to understand many input formats.
92 A system of input filters driven by
93 regular expressions allows you to easily process most ASCII-based
94 data formats. SGML, XML, ISO2709 (MARC), and raw text are also
101 Searching supports a powerful combination of boolean queries as
102 well as relevance-ranking (free-text) queries. Truncation,
103 masking, full regular expression matching and "approximate
104 matching" (eg. spelling mistakes) are all supported.
110 Index-only databases: data can be, and usually is, imported
111 into Zebra's own storage, but Zebra can also refer to
112 external files, building and maintaining indexes of "live"
119 Zebra is written in portable C, so it runs on most Unix-like systems
120 as well as Windows NT. A binary distribution for Windows NT is
130 Z39.50 protocol support:
137 Protocol facilities: Init, Search, Present (retrieval), Delete,
138 Scan (index browsing) and Sort.
144 Piggy-backed presents are honored in the search-request.
150 Named result sets are supported.
156 Easily configured to support different application profiles, with
157 tables for attribute sets, tag sets, and abstract syntaxes.
158 Additional tables control facilities such as element mappings to
159 different schema (eg., GILS-to-USMARC).
165 Complex composition specifications using Espec-1 (partial support).
166 Element sets are defined using the Espec-1 capability,
167 and are specified in configuration files as simple element
168 requests (and, optionally, variant requests).
174 Multiple record syntaxes
175 for data retrieval: GRS-1, SUTRS,
176 XML, ISO2709 (MARC), etc. Records can be mapped between record syntaxes
177 and schemas on the fly.
188 <title>Applications</title>
190 Zebra has been deployed in numerous applications, in both the
191 academic and commercial worlds, in application domains as diverse
192 as bibliographic catalogues, geospatial information, structured
193 vocabulary browsing, government information locators, civic
194 information systems, environmental observations, museum information
198 Notable applications include the following:
202 <title>DADS - the DTV Article Database Service</title>
204 DADS is a huge database of more than ten million records, totalling
205 over ten gigabytes of data. The records are metadata about academic
206 journal articles, primarily scientific; about 10% of these
207 metadata records link to the full text of the articles they
208 describe, a body of about a terabyte of information (although the
209 full text is not indexed.)
212 It allows students and researchers at DTU (Danmarks Tekniske
213 Universitet, the Technical College of Denmark) to find and order
214 articles from multiple databases in a single query. The database
215 contains literature on all engineering subjects. It's available
216 on-line through a web gateway, though currently only to registered
220 More information can be found at
221 <ulink url="http://www.dtv.dk/help/dads/index_e.htm"/>
226 <title>Various web indexes</title>
228 Zebra has been used by a variety of institutions to construct
229 indexes of large web sites, typically in the region of tens of
230 millions of pages. In this role, it functions somewhat similarly
231 to the engine of google or altavista, but for a selected intranet
232 or subset of the whole Web.
235 ### examples, details and numbers, please!
241 <title>Future Directions</title>
244 These are some of the plans that we have for the software in the near
245 and far future, ordered approximately as we expect to work on them.
253 Improved support for XML in search and retrieval. Eventually,
254 the goal is for Zebra to pull double duty as a flexible
255 information retrieval engine and high-performance XML
262 Access to search engine through SOAP/RPC API to allow the
263 construction of applications without requiring Z39.50 tools.
269 Finalisation and documentation of Zebra's C programming
270 API, allowing updates, database management and other functions
271 not readily expressed in Z39.50. We will also consider
272 exposing the API through SOAP.
278 Improved free-text searching. We're first and foremost octet jockeys and
279 we're actively looking for organisations or people who'd like
280 to contribute experience in relevance ranking and text
289 Programmers thrive on user feedback. If you are interested in a
290 facility that you don't see mentioned here, or if there's something
291 you think we could do better, please drop us a mail. Better still,
292 implement it and send us the patches.
295 If you think it's all really neat, you're welcome to drop us a line
296 saying that, too. You'll find contact info at the end of this file.
301 <!-- Keep this comment at the end of the file
306 sgml-minimize-attributes:nil
307 sgml-always-quote-attributes:t
310 sgml-parent-document: "zebra.xml"
311 sgml-local-catalogs: nil
312 sgml-namecase-general:t