1 % Embedded metasearching with the MasterKey Widget Set
9 There are lots of practical problems in building resource discovery
10 solutions. One of the biggest, and most ubiquitous is incorporating
11 metasearching functionality into existing web-sites -- for example,
12 content-management systems, library catalogues or intranets. In
13 general, even when access to core metasearching functionality is
14 provided by simple web-services such as
15 [Pazpar2](http://www.indexdata.com/pazpar2), integration work is seen
16 as a major part of most projects.
18 Index Data provides several different toolkits for communicating with
19 its metasearching middleware, trading off varying degrees of
20 flexibility against convenience:
22 * pz2.js -- a low-level JavaScript library for interrogating the
23 Service Proxy and Pazpar2. It allows the HTML/JavaScript programmer
24 to create JavaScript applications display facets, records, etc. that
25 are fetched from the metasearching middleware.
27 * masterkey-ui-core -- a higher-level, complex JavaScript library that
28 uses pz2.js to provide the pieces needed for building a
29 full-featured JavaScript application.
31 * MasterKey Demo UI -- an example of a searching application built on
32 top of masterkey-ui-core. Available as a public demo at
33 http://mk2.indexdata.com/
35 * MKDru -- a toolkit for embedding MasterKey-like searching into
38 All of these approaches require programming to a greater or lesser
39 extent. Against this backdrop, we introduced MKWS (the MasterKey
40 Widget Set) -- a set of simple, very high-level HTML+CSS+JavaScript
41 components that can be incorporated into any web-site to provide
42 MasterKey searching facilities. By placing `<div>`s with well-known
43 identifiers in any HTML page, the various components of an application
44 can be embedded: search-boxes, results areas, target information, etc.
50 The following is a complete MKWS-based searching application:
54 <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
55 <title>MKWS demo client</title>
56 <script type="text/javascript" src="http://mkws.indexdata.com/mkws-complete.js"></script>
57 <link rel="stylesheet" href="http://mkws.indexdata.com/mkws.css" />
60 <div id="mkwsSearch"></div>
61 <div id="mkwsResults"></div>
65 Go ahead, try it! You don't even need a web-server. Just copy and
66 paste this HTML into a file on your computer -- `/tmp/magic.html`,
67 say -- and point your web-browser at it:
68 `file:///tmp/magic.html`. Just like that, you have working
75 If you know any HTML, the structure of the file will be familar to
76 you: the `<html>` element at the top level contains a `<head>` and a
77 `<body>`. In addition to whatever else you might want to put on your
78 page, you can add MKWS elements.
80 These fall into two categories. First, the prerequisites in the HTML
81 header, which are loaded from the tool site mkws.indexdata.com:
84 contains all the JavaScript needed by the widget-set.
87 provides the default CSS styling
89 Second, within the HTML body, `<div>` elements with special IDs that
90 begin `mkws` can be provided. These are filled in by the MKWS code,
91 and provide the components of the searching UI. The very simple
92 application above has only two such components: a search box and a
93 results area. But more are supported. The main `<div>`s are:
95 * `mkwsSearch` -- provides the search box and button.
97 * `mkwsResults` -- provides the results area, including a list of
98 brief records (which open out into full versions when clicked),
99 paging for large results sets, facets for refining a search,
100 sorting facilities, etc.
102 * `mkwsLang` -- provides links to switch between one of several
103 different UI languages. By default, English, Danish and German are
106 * `mkwsSwitch` -- provides links to switch between a view of the
107 result records and of the targets that provide them. Only
108 meaningful when `mkwsTargets` is also provided.
110 * `mkwsTargets` -- the area where per-target information will appear
111 when selected by the link in the `mkwsSwitch` area. Of interest
112 mostly for fault diagnosis rather than for end-users.
114 * `mkwsStat` --provides a status line summarising the statistics of
117 To see all of these working together, just put them all into the HTML
120 <div id="mkwsSwitch"></div>
121 <div id="mkwsLang"></div>
122 <div id="mkwsSearch"></div>
123 <div id="mkwsResults"></div>
124 <div id="mkwsTargets"></div>
125 <div id="mkwsStat"></div>
131 Many aspects of the behaviour of MKWS can be modified by setting
132 parameters into the `mkws_config` object. **This must be done *before*
133 including the MKWS JavaScript** so that when that code is executed it
134 can refer to the configuration values. So the HTML header looks like
137 <script type="text/javascript">
140 sort_default: "title",
144 <script type="text/javascript" src="http://mkws.indexdata.com/mkws-complete.js"></script>
146 This configuration sets the UI language to Danish (rather than the
147 default of English), initially sorts search results by title rather
148 than relevance (though as always this can be changed in the UI) and
149 makes the search box a bit wider than the default.
151 The full set of supported configuration items is described in the
152 reference guide below.
155 Control over HTML and CSS
156 =========================
158 More sophisticated applications will not simply place the `<div>`s
159 together, but position them carefully within an existing page
160 framework -- such as a Drupal template, an OPAC or a SharePoint page.
162 While it's convenient for simple applications to use a monolithic
163 `mkwsResults` area which contains record, facets, sorting options,
164 etc., customised layouts may wish to treat each of these components
165 separately. In this case, `mkwsResults` can be omitted, and the
166 following lower-level components provided instead:
168 * `mkwsTermlists` -- provides the facets
170 * `mkwsRanking` -- provides the options for how records are sorted and
171 how many are included on each page of results.
173 * `mkwsPager` -- provides the links for navigating back and forth
174 through the pages of records.
176 * `mkwsNavi` -- when a search result has been narrowed by one or more
177 facets, this area shows the names of those facets, and allows the
178 selected values to be clicked in order to remove them.
180 * `mkwsRecords` -- lists the actual result records.
182 Customisation of MKWS searching widgets can also be achieved by
183 overriding the styles set in the toolkit's CSS stylesheet. The default
184 styles can be inspected in `mkws.css` and overridden in any
185 styles that appears later in the HTML than that file. At the simplest
186 level, this might just mean changing fonts, sizes and colours, but
187 more fundamental changes are also possible.
189 To properly apply styles, it's necessary to understand how the HTML is
190 structured, e.g. which elements are nested within which
191 containers. The structures used by the widget-set are described in the
192 reference guide below.
202 Some applications might like to open with content in the area that
203 will subsequently be filled with result-records -- a message of the
204 day, a welcome message or a help page. This can be done by placing an
205 `mkwsMOTD` division anywhere on the page. It will be moved into the
206 `mkwsResults` area and initially displayed, but will be hidden when a
210 Customised display using Handlebars templates
211 ---------------------------------------------
213 Certain aspects of the widget-set's display can be customised by
214 providing Handlebars templates with well-known classes that begin with
215 the string `mkwsTemplate_`. At present, the supported templates are:
217 * `mkwsTemplate_Summary` -- used for each summary record in a list of
220 * `mkwsTemplate_Record` -- used when displaying a full record.
222 For both of these the metadata record is passed in, and its fields can
223 be referenced in the template. As well as the metadata fields
224 (`md-*`), two special fields are provided to the `mkwsTemplate_Summary`
225 template, for creating popup links for full records. These are `_id`,
226 which must be provided as the `id` attribute of a link tag, and
227 `_onclick`, which must be provided as the `onclick` attribute.
229 For example, an application can install a simple author+title summary
230 record in place of the usual one providing the following template:
232 <script class="mkwsTemplate_Summary" type="text/x-handlebars-template">
234 <span>{{md-author}}</span>
236 <a href="#" id="{{_id}}" onclick="{{_onclick}}">
241 For details of Handlebars template syntax, see
242 [the online documentation](http://handlebarsjs.com/).
248 Metasearching applications may need to appear differently on
249 small-screened mobile devices, or change their appearance when
250 screen-width changes (as when a small device is rotated). To achieve
251 this, MKWS supports responsive design which will move the termlists to
252 the bottom on narrow screens and to the sidebar on wide screens.
254 To turn on this behaviour, set the `responsive_design_width` to the desired
255 threshhold width in pixels. For example:
257 <script type="text/javascript">
259 responsive_design_width: 990
263 If individual result-related components are in use in place of the
264 all-in-one mkwsResults, then the redesigned application needs to
265 specify the locations where the termlists should appear in both
266 cases. In this case, wrap the wide-screen `mkwsTermlists` element in a
267 `mkwsTermlists-Container-wide` element; and provide an
268 `mkwsTermlists-Container-narrow` element in the place where the narrow-screen
269 termlists should appear.
272 Popup results with jQuery UI
273 ----------------------------
275 The [jQuery UI library](http://en.wikipedia.org/wiki/JQuery_UI)
276 can be used to construct MKWS applications in which the only component
277 generally visible on the page is a search box, and the results appear
278 in a popup. The key part of such an application is this invocation of
279 the MKWS jQuery plugin:
281 <div class="mkwsSearch"></div>
282 <div class="mkwsPopup" popup_width="1024" popup_height="650" popup_modal="0" popup_autoOpen="0" popup_button="input.mkwsButton">
283 <div class="mkwsSwitch"></div>
284 <div class="mkwsLang"></div>
285 <div class="mkwsResults"></div>
286 <div class="mkwsTargets"></div>
287 <div class="mkwsStat"></div>
290 The necessary scaffolding can be seen in an example application,
291 http://example.indexdata.com/index-popup.html
294 Authentication and target configuration
295 ---------------------------------------
297 By default, MKWS configures itself to use a demonstration account on a
298 service hosted by mkws.indexdata.com. This account (username `demo`,
299 password `demo`) provides access to about a dozen free data
300 sources. Authentication onto this service is via an authentication URL
301 on the same MKWS server, so no explicit configuration is needed.
303 In order to search in a customised set of targets, including
304 subscription resources, it's necessary to create an account with
305 Index Data's hosted service proxy, and protect that account with
306 authentication tokens (to prevent unauthorised use of subscription
307 resources). For information on how to do this, see the next section.
310 MKWS Target Selection
311 =====================
313 MKWS accesses targets using the Pazpar2 metasearching engine. Although
314 Pazpar2 can be used directly, using a statically configured set of
315 targets, this usage is unusual. More often, Pazpar2 is fronted by the
316 Service Proxy (SP), which manages authentication, sessions, target
319 This document assumes the SP is used, and explains how to go about
320 making a set of targets (a "library") available, how to connect your
321 MKWS application to that library, and how to choose which of the
322 available targets to use.
325 Maintaining the library
326 -----------------------
328 The service proxy accesses sets of targets that are known as
329 "libraries". In general, each customer will have their own library,
330 though some standard libraries may be shared between many customers --
331 for example, a library containing all open-access academic journals.
332 A library can also contain other configuration information, including
333 the set of categories by which targets are classified for the library.
335 Libraries are maintained using MKAdmin (MasterKey
336 Admin). Specifically, those used by MKWS are generally maintained on
337 the "MKX Admin" installation at
338 <http://mkx-admin.indexdata.com/console/>
340 In general, Index Data will create a library for each customer, then
341 give the customer a username/password pair that they can use to enter
342 MKAdmin and administrate that library.
344 Once logged in, customers can select which targets to include (from
345 the list of several thousand that MKAdmin knows about), and make
346 customer-specific modifications -- e.g. overriding the titles of the
349 Most importantly, customers' administrators can add authentication
350 credentials that the Service Proxy will used on their behalf when
351 accessing subscription resources -- username/password pairs or proxies
352 to use for IP-based authentication. Note that **it is then crucial to
353 secure the library from use by unauthorised clients**, otherwise the
354 customer's paid subscriptions will be exploited.
356 Access to libraries is managed by creating one or more "User Access"
357 records in MKAdmin, under the tab of that name. Each of these records
358 provides a combination of credentials and other data that allow an
359 incoming MKWS client to be identified as having legitimate access to
360 the library. The authentication process, described below, works by
361 searching for a matching User Access record.
364 Authenticating your MWKS application onto the library
365 -----------------------------------------------------
367 Some MKWS applications will be content to use the default library with
368 its selection of targets. Most, though, will want to define their own
369 library providing a different range of available targets. An important
370 case is that of applications that authenticate onto subscription
371 resources by means of back-end site credentials stored in MKAdmin:
372 precautions must be taken so that such library accounts do not allow
375 Setting up such a library is a process of several stages.
377 ### Create the User Access account
379 Log in to MKAdmin to add a User Access account for your library:
381 * Go to <http://mkx-admin.indexdata.com/console/>
382 * Enter the adminstrative username/password
383 * Go to the User Access tab
384 * Create an end-user account
385 * Depending on what authentication method it be used, set the
386 User Access account's username and password, or referring URL, or
387 Service Proxy hostname, or IP-address range.
389 If your MWKS application runs at a well-known, permanent address --
390 <http://yourname.com/app.html>, say -- you can set the User Access
391 record so that this originating URL is recognised by setting it into
392 the "Referring URL" field.
394 If your application accesses the Service Proxy by a unique virtual
395 hostname -- yourname.sp-mkws.indexdata.com, say -- you can tie the use
396 of this hostname to your library by setting the User Access record's
397 "Host Name" field to name of the host where the SP is accessed. **Note
398 that this is not secure, as other applications can use this virtual
399 hostname to gain access to your library.**
401 Or if your application's users are coming from a well-known range of
402 IP-address space, you can enter the range in the "IP Ranges"
403 field. The format of this field is as follows: it can contain any
404 number of ranges, separated by commas; each range is either a single
405 IP address or two addresses separated by a hyphen; each IP address is
406 four small integers separated by periods. For example,
407 `80.229.143.255-80.229.143.255, 5.57.0.0-5.57.255.255, 127.0.0.1`.
409 Alternatively, your application can authenticate by username and
410 password credentials. This is a useful approach in several situations,
411 including when you need to specify the use of a different library from
412 usual one. To arrange for this, set the username and password as a
413 single string separated by a slash -- e.g. "mike/swordfish" -- into
414 the User Access record's Authentication field.
416 You can set multiple fields into a single User Access record; or
417 create multiple User Access records. For example, a single User Access
418 record can specify both a Referring URL a username/password pair that
419 can be used when running an application from a different URL. But if
420 multiple Referring URLs are needed, then each must be specified in its
421 own User Access record.
423 ### Tell the application to use the library
425 In the HTML of the application, tell MKWS to authenticate on to the
426 Service Proxy. When referer-based or IP-based authentication is used,
429 <script type="text/javascript">
430 var mkws_config = { service_proxy_auth:
431 "//sp-mkws.indexdata.com/service-proxy/?command=auth&action=perconfig" };
434 > TODO This should be the default setting: see MKWS-251.
436 And ensure that access to the MWKS application is from the correct
437 Referrer URL or IP-range.
439 ### (Optional): access by a different virtual hostname
441 When hostname-based authentication is in use, it's necessary to access
442 the Service Proxy as the correctly named virtual host. This can be
443 done by setting the `service_proxy_auth` configuration item to a
444 URL containing that hostname, such as
445 `//yourname.sp-mkws.indexdata.com/service-proxy/?command=auth&action=perconfig`
447 > TODO It should be possible to change just the hostname without
448 > needing to repeat the rest of the URL (protocol, path, query): see
451 > TODO When changing the SP authentication URL, the Pazpar2 URL should
452 > in general change along with it: see MKWS-253.
454 ### (Optional): embed credentials for access to the library
456 When credential-based authentication is in use (username and
457 password), it's necessary to pass these credentials into the Service
458 Proxy when establishing the session. This can most simply be done just
459 by setting the `service_proxy_auth` configuration item to a URL such as
460 `//sp-mkws.indexdata.com/service-proxy/?command=auth&action=perconfig&username=mike&password=swordfish`
462 > TODO It should be possible to add the username and password to the
463 > configuration without needing to repeat the rest of the URL.
465 ### (Optional): conceal credentials from HTML source
467 Using a credential-based Service-Proxy authentication URL such as the
468 one above reveals the the credentials to public view -- to anyone who
469 does View Source on the MKWS application. This may be acceptable for
470 some libraries, but is intolerable for those which provide
471 authenticated access to subscription resources.
473 In these circumstances, a more elaborate approach is necessary. The
474 idea is to make a URL local to the customer that is used for
475 authentication onto the Service Proxy, hiding the credentials in a
476 local rewrite rule. Then local mechanisms can be used to limit access
477 to that local authentication URL. Here is one way to do it when
478 Apache2 is the application's web-server, which we will call
481 Step 1: add a rewriting authentication alias to the configuration:
484 RewriteRule /spauth/ http://mkws.indexdata.com/service-proxy/?command=auth&action=check,login&username=U&password=PW [P]
486 Step 2: set the MKWS configuration item `service_proxy_auth` to
487 <http://yourname.com/spauth/>
489 Step 3: protect access to the local path <http://yourname.com/spauth/>
490 (e.g. using a `.htaccess` file).
493 Choosing targets from the library
494 ---------------------------------
496 MKWS applications can choose what subset of the library's targets to
497 use, by means of several alternative settings on individual widgets or
498 in the `mkws_config` structure:
500 * `targets` -- contains a Pazpar2 targets string, typically of the form
501 "pz:id=" or "pz:id~" followed by a pipe-separated list of low-level
503 At present, these IDs can take one of two forms, depending on the
504 configuration of the Service Proxy being used: they may be based on
505 ZURLs (so a typical value would be something like
506 `pz:id=josiah.brown.edu:210/innopac|lui.indexdata.com:8080/solr4/select?fq=database:4902`)
507 or they may be UDBs (so a typical value would be something like
508 `pz:id=brown|artstor`)
510 * `targetfilter` -- contains a CQL query which is used to find relevant
511 targets from the relvant library. For example,
516 * `target` -- contains a single UDB, that of the sole target to be
519 This is merely syntactic sugar for "targetfilter" with the query
522 For example, a `Records` widget can be limited to searching only in
523 targets that have been categorised as news sources by providing an
524 attribute as follows:
526 <div class="mkwsRecords" targetfilter='categories=news'/>
535 The configuration object `mkws_config` may be created before including
536 the MKWS JavaScript code to modify default behaviour. This structure
537 is a key-value lookup table, whose entries are described in the table
538 below. All entries are optional, but if specified must be given values
539 of the specified type. If ommitted, each setting takes the indicated
540 default value; long default values are in footnotes to keep the table
544 Element Type Default Description
545 -------- ----- --------- ------------
546 debug_level int 1 Level of debugging output to emit. 0 = none, 1 = messages, 2 = messages with
547 datestamps, 3 = messages with datestamps and stack-traces.
549 facets array *Note 1* Ordered list of names of facets to display. Supported facet names are
550 `xtargets`, `subject` and `author`.
552 lang string en Code of the default language to display the UI in. Supported language codes are `en` =
553 English, `de` = German, `da` = Danish, and whatever additional languages are configured
554 using `language_*` entries (see below).
556 lang_options array [] A list of the languages to offer as options. If empty (the default), then all
557 configured languages are listed.
559 language_* hash Support for any number of languages can be added by providing entries whose name is
560 `language_` followed by the code of the language. See the separate section below for
563 pazpar2_url string *Note 2* The URL used to access the metasearch middleware. This service must be configured to
564 provide search results, facets, etc. It may be either unmediated or Pazpar2 the
565 MasterKey Service Proxy, which mediates access to an underlying Pazpar2 instance. In
566 the latter case, `service_proxy_auth` must be provided.
568 perpage_default string 20 The initial value for the number of records to show on each page.
570 perpage_options array *Note 3* A list of candidate page sizes. Users can choose between these to determine how many
571 records are displayed on each page of results.
573 query_width int 50 The width of the query box, in characters.
575 responsive_design_width int If defined, then the facets display moves between two locations as the screen-width
576 varies, as described above. The specified number is the threshhold width, in pixels,
577 at which the facets move between their two locations.
579 service_proxy_auth url *Note 4* A URL which, when `use_service_proxy` is true, is fetched once at the beginning of each
580 session to authenticate the user and establish a session that encompasses a defined set
581 of targets to search in.
583 service_proxy_auth_domain domain Can be set to the domain for which `service_proxy_auth` proxies authentication, so
584 that cookies are rewritten to appear to be from this domain. In general, this is not
585 necessary, as this setting defaults to the domain of `pazpar2_url`.
587 show_lang bool true Indicates whether or not to display the language menu.
589 show_perpage bool true Indicates whether or not to display the perpage menu.
591 show_sort bool true Indicates whether or not to display the sort menu.
593 show_switch bool true Indicates whether or not to display the switch menu, for switching between showing
594 retrieved records and target information.
596 sort_default string relevance The label of the default sort criterion to use. Must be one of those in the `sort`
599 sort_options array *Note 6* List of supported sort criteria. Each element of the list is itself a two-element list:
600 the first element of each sublist is a pazpar2 sort-expression such as `data:0` and
601 the second is a human-readable label such as `newest`.
603 use_service_proxy bool true If true, then a Service Proxy is used to deliver searching services rather than raw
607 Perhaps we should get rid of the `show_lang`, `show_perpage`,
608 `show_sort` and `show_switch` configuration items, and simply display the relevant menus
609 only when their containers are provided -- e.g. an `mkwsLang` element
610 for the language menu. But for now we retain these, as an easier route
611 to lightly customise the display than my changing providing a full HTML
616 1. ["sources", "subjects", "authors"]
618 2. /pazpar2/search.pz2
622 4. http://mkws.indexdata.com/service-proxy-auth
624 5. http://mkws.indexdata.com/service-proxy/
626 6. [["relevance"], ["title:1", "title"], ["date:0", "newest"], ["date:1", "oldest"]]
629 Language specification
630 ----------------------
632 Support for another UI language can be added by providing an entry in
633 the `mkws_config` object whose name is `language_` followed by the
634 name of the language: for example, `language_French` to support
635 French. Then value of this entry must be a key-value lookup table,
636 mapping the English-language strings of the UI into their equivalents
637 in the specified language. For example:
641 "Authors": "Auteurs",
642 "Subjects": "Sujets",
643 // ... and others ...
647 The following strings occurring in the UI can be translated:
663 In addition, facet names can be translated:
669 Finally, the names of fields in the full-record display can be
670 translated. These include, but may not be limited to:
680 jQuery UI popup invocation
681 --------------------------
683 The MasterKey Widget Set can be invoked in a popup window on top of the page.
685 Note that when using the `popup` layout, facilities from the jQuery UI
686 toolkit are used, so it's necessary to include both CSS and JavaScript
687 from that toolkit. The relevant lines are:
689 <script src="http://code.jquery.com/ui/1.10.3/jquery-ui.min.js"></script>
690 <link rel="stylesheet" type="text/css"
691 href="http://code.jquery.com/ui/1.10.3/themes/smoothness/jquery-ui.css" />
693 <div class="mkwsSearch"></div>
694 <div class="mkwsPopup" popup_width="1024" popup_height="650" popup_modal="0" popup_autoOpen="0" popup_button="input.mkwsButton">
695 <div class="mkwsSwitch"></div>
696 <div class="mkwsLang"></div>
697 <div class="mkwsResults"></div>
698 <div class="mkwsTargets"></div>
699 <div class="mkwsStat"></div>
703 Element Type Default Description
704 -------- ----- --------- ------------
705 popup_width string 880 Width of the popup window (if used), in
708 popup_height string 760 Height of the popup window (if used), in
711 popup_button string input.mkwsButton (Never change this.)
713 popup_modal string 0 Modal confirmation mode. Valid values are 0 or 1
715 popup_autoOpen string 1 Open popup window on load. Valid values are 0 or 1
720 The structure of the HTML generated by the MKWS widgets
721 -------------------------------------------------------
723 In order to override the default CSS styles provided by the MasterKey Widget
724 Set, it's necessary to understand that structure of the HTML elements that are
725 generated within the components. This knowledge make it possible, for example,
726 to style each `<div>` with class `term` but only when it occurs inside an
727 element with ID `#mkwsTermlists`, so as to avoid inadvertently styling other
728 elements using the same class in the non-MKWS parts of the page.
730 The HTML structure is as follows. As in CSS, #ID indicates a unique identifier
731 and .CLASS indicates an instance of a class.
741 input#mkwsQuery type=text
742 input#mkwsButton type=submit
745 (no contents -- used only for masking)
766 span (for sequence number)
768 span (for other information such as author)
769 div.details (sometimes)
792 Copyright (C) 2013-2014 by IndexData ApS, <http://www.indexdata.com>