% Embedded metasearching with the MasterKey Widget Set
% Mike Taylor
+% 30 July 2014
Introduction
-------------
+============
There are lots of practical problems in building resource discovery
solutions. One of the biggest, and most ubiquitous is incorporating
Widget Set) -- a set of simple, very high-level HTML+CSS+JavaScript
components that can be incorporated into any web-site to provide
MasterKey searching facilities. By placing `<div>`s with well-known
-identifiers in any HTML page, the various components of an application
+MKWS classes in any HTML page, the various components of an application
can be embedded: search-boxes, results areas, target information, etc.
Simple Example
---------------
+==============
The following is a complete MKWS-based searching application:
<link rel="stylesheet" href="http://mkws.indexdata.com/mkws.css" />
</head>
<body>
- <div id="mkwsSearch"></div>
- <div id="mkwsResults"></div>
+ <div class="mkwsSearch"></div>
+ <div class="mkwsResults"></div>
</body>
</html>
-Go ahead, try it! You don't even need a web-server. Just copy and
-paste this HTML into a file on your computer -- `/tmp/magic.html`,
-say -- and point your web-browser at it:
-`file:///tmp/magic.html`. Just like that, you have working
-metasearching.
-
+Go ahead, try it! Simply put the above in a file (e.g index.html),
+drop it into a folder accessible with an ordinary web-server (e.g Apache)
+and load it in your web browser (and no, usually, you can't just load the file
+directly from disk as some browsers, e.g Chrome, won't allow storing cookies).
+Just like that, you have working metasearching.
How the example works
---------------------
<div id="mkwsTargets"></div>
<div id="mkwsStat"></div>
+
Configuration
--------------
+=============
Many aspects of the behaviour of MKWS can be modified by setting
parameters into the `mkws_config` object. **This must be done *before*
Control over HTML and CSS
--------------------------
+=========================
More sophisticated applications will not simply place the `<div>`s
together, but position them carefully within an existing page
reference guide below.
+Customised display using Handlebars templates
+=============================================
+
+A lot can be done by styling widgets in CSS and changing basic MKWS config
+options. For further customisation, MKWS allows you to change the markup it
+outputs for any widget. This is done by overriding the
+[Handlebars](http://handlebarsjs.com/) template used to generate it. In general
+these consist of `{{things in double braces}}` that are replaced by values from
+the system. For details of Handlebars template syntax, see [the online
+documentation](http://handlebarsjs.com/).
+
+The templates used by the core widgets can be viewed in [our git
+repository](http://git.indexdata.com/?p=mkws.git;a=tree;f=src/mkws.templates;).
+Replacement values are documented in a comment at the top of each template so
+you can see what's going where. If all you want to do is add a CSS class to
+something or change a `span` to a `div` it's easy to just copy the existing
+template and make your edits.
+
+Overriding templates
+--------------------
+
+To override the template for a widget, include it inline in the document
+as a `<script>` tag marked with a class of `mkwsTemplate_Foo` where Foo is the
+name of the template you want to override (typically the name of the widget).
+Inline Handlebars templates are distinguished from Javascript via a
+`type="text/x-handlebars-template` attribute. For example, to override the
+Pager template you would include this in your document:
+
+ <script class="mkwsTemplate_Pager" type="text/x-handlebars-template">
+ ...new Pager template
+ </script>
+
+The Facet template has a special feature where you can override it on a
+per-facet basis by adding a dash and the facet name as a suffix eg.
+`Facet-Subjects` rather than `Facet`.
+
+You can also explicitly specify a different template for a particular instance
+of a widget by providing the name of your alternative (eg. SpecialPager) as the
+value of the `template` key in the MKWS config object for that widget
+(usually via the `data-mkws-config` attribute).
+
+Templates for MKWS can also be
+[precompiled](http://handlebarsjs.com/precompilation.html). If a precompiled
+template of the same name is found in the `Handlebars.templates` object, it
+will be used instead of the default.
+
+Templating metadata
+-------------------
+
+MKWS makes requests to Service Proxy or Pazpar2 that perform the actual
+searching. Depending on how these are configured and what is available from the
+targets you are searching there may be more data available than what is
+presented by the default templates.
+
+Handlebars offers a convenient log helper that will output the contents of a
+variable for you to inspect. This lets you look at exactly what is being
+returned by the back end without needing to use a Javascript debugger. For
+example, you might prepend `{{log hits}}` to the Records template in order to
+see what is being returned with each search result in the list. In order for
+this to work you'll need to enable verbose output from Handlebars which is done
+by including this line or similar:
+
+ <script>Handlebars.logger.level = 1;</script>
+
+Internationalisation
+--------------------
+
+If you would like your template to use the built in translation functionality,
+output locale specific text via the mkws-translate helper like so:
+`{{{mkws-translate "a few words"}}}`.
+
+Example
+-------
+
+Rather than use the included AJAX helpers to render record details inline,
+here's a Records template that will link directly to the source via the address
+provided in the metadata as the first element of `md-electronic-url`:
+
+ <script class="mkwsTemplate_Records" type="text/x-handlebars-template">
+ {{#each hits}}
+ <div class="{{containerClass}}">
+ <a href="{{md-electronic-url.[0]}}">
+ <b>{{md-title}}</b>
+ </a>
+ {{#if md-title-remainder}}
+ <span>{{md-title-remainder}}</span>
+ {{/if}}
+ {{#if md-title-responsibility}}
+ <span><i>{{md-title-responsibility}}</i></span>
+ {{/if}}
+ </div>
+ {{/each}}
+ </script>
+
+For a more involved example where markup for multiple widgets is decorated with
+[Bootstrap](http://getbootstrap.com/) classes and a custom Handlebars helper is
+employed, take a look at the source of
+[topic.html](http://example.indexdata.com/topic.html?q=water).
+
+
Refinements
------------
+===========
-### Message of the day
+Message of the day
+------------------
Some applications might like to open with content in the area that
will subsequently be filled with result-records -- a message of the
search is made.
-### Customised display using Handlebars templates
-
-Certain aspects of the widget-set's display can be customised by
-providing Handlebars templates with well-known classes that begin with
-the string `mkwsTemplate_`. At present, the supported templates are:
-
-* `mkwsTemplate_Summary` -- used for each summary record in a list of
- results.
-
-* `mkwsTemplate_Record` -- used when displaying a full record.
-
-For both of these the metadata record is passed in, and its fields can
-be referenced in the template. As well as the metadata fields
-(`md-*`), two special fields are provided to the `mkwsTemplate_Summary`
-template, for creating popup links for full records. These are `_id`,
-which must be provided as the `id` attribute of a link tag, and
-`_onclick`, which must be provided as the `onclick` attribute.
-
-For example, an application can install a simple author+title summary
-record in place of the usual one providing the following template:
-
- <script class="mkwsTemplate_Summary" type="text/x-handlebars-template">
- {{#if md-author}}
- <span>{{md-author}}</span>
- {{/if}}
- <a href="#" id="{{_id}}" onclick="{{_onclick}}">
- <b>{{md-title}}</b>
- </a>
- </script>
-
-For details of Handlebars template syntax, see
-[the online documentation](http://handlebarsjs.com/).
-
-
-### Responsive design
+Responsive design
+-----------------
Metasearching applications may need to appear differently on
small-screened mobile devices, or change their appearance when
termlists should appear.
-### Popup results with jQuery UI
+Popup results with jQuery UI
+----------------------------
The [jQuery UI library](http://en.wikipedia.org/wiki/JQuery_UI)
can be used to construct MKWS applications in which the only component
http://example.indexdata.com/index-popup.html
-### Authentication and target configuration
+Authentication and target configuration
+---------------------------------------
By default, MKWS configures itself to use a demonstration account on a
service hosted by mkws.indexdata.com. This account (username `demo`,
subscription resources, it's necessary to create an account with
Index Data's hosted service proxy, and protect that account with
authentication tokens (to prevent unauthorised use of subscription
-resources). For information on how to do this, see
-[MKWS Target Selection](library-configuration.html)
+resources). For information on how to do this, see the next section.
+
+
+MKWS Target Selection
+=====================
+
+MKWS accesses targets using the Pazpar2 metasearching engine. Although
+Pazpar2 can be used directly, using a statically configured set of
+targets, this usage is unusual. More often, Pazpar2 is fronted by the
+Service Proxy (SP), which manages authentication, sessions, target
+selection, etc.
+
+This document assumes the SP is used, and explains how to go about
+making a set of targets (a "library") available, how to connect your
+MKWS application to that library, and how to choose which of the
+available targets to use.
+
+
+Maintaining the library
+-----------------------
+
+The service proxy accesses sets of targets that are known as
+"libraries". In general, each customer will have their own library,
+though some standard libraries may be shared between many customers --
+for example, a library containing all open-access academic journals.
+A library can also contain other configuration information, including
+the set of categories by which targets are classified for the library.
+
+Libraries are maintained using MKAdmin (MasterKey
+Admin). Specifically, those used by MKWS are generally maintained on
+the "MKX Admin" installation at
+<http://mkx-admin.indexdata.com/console/>
+
+In general, Index Data will create a library for each customer, then
+give the customer a username/password pair that they can use to enter
+MKAdmin and administrate that library.
+
+Once logged in, customers can select which targets to include (from
+the list of several thousand that MKAdmin knows about), and make
+customer-specific modifications -- e.g. overriding the titles of the
+targets.
+
+Most importantly, customers' administrators can add authentication
+credentials that the Service Proxy will used on their behalf when
+accessing subscription resources -- username/password pairs or proxies
+to use for IP-based authentication. Note that **it is then crucial to
+secure the library from use by unauthorised clients**, otherwise the
+customer's paid subscriptions will be exploited.
+
+Access to libraries is managed by creating one or more "User Access"
+records in MKAdmin, under the tab of that name. Each of these records
+provides a combination of credentials and other data that allow an
+incoming MKWS client to be identified as having legitimate access to
+the library. The authentication process, described below, works by
+searching for a matching User Access record.
+
+
+Authenticating your MWKS application onto the library
+-----------------------------------------------------
+
+Some MKWS applications will be content to use the default library with
+its selection of targets. Most, though, will want to define their own
+library providing a different range of available targets. An important
+case is that of applications that authenticate onto subscription
+resources by means of back-end site credentials stored in MKAdmin:
+precautions must be taken so that such library accounts do not allow
+unauthorised access.
+
+Setting up such a library is a process of several stages.
+
+### Create the User Access account
+
+Log in to MKAdmin to add a User Access account for your library:
+
+* Go to <http://mkx-admin.indexdata.com/console/>
+* Enter the adminstrative username/password
+* Go to the User Access tab
+* Create an end-user account
+* Depending on what authentication method it be used, set the
+ User Access account's username and password, or referring URL, or
+ Service Proxy hostname, or IP-address range.
+
+If your MWKS application runs at a well-known, permanent address --
+<http://yourname.com/app.html>, say -- you can set the User Access
+record so that this originating URL is recognised by setting it into
+the "Referring URL" field.
+
+If your application accesses the Service Proxy by a unique virtual
+hostname -- yourname.sp-mkws.indexdata.com, say -- you can tie the use
+of this hostname to your library by setting the User Access record's
+"Host Name" field to name of the host where the SP is accessed. **Note
+that this is not secure, as other applications can use this virtual
+hostname to gain access to your library.**
+
+Or if your application's users are coming from a well-known range of
+IP-address space, you can enter the range in the "IP Ranges"
+field. The format of this field is as follows: it can contain any
+number of ranges, separated by commas; each range is either a single
+IP address or two addresses separated by a hyphen; each IP address is
+four small integers separated by periods. For example,
+`80.229.143.255-80.229.143.255, 5.57.0.0-5.57.255.255, 127.0.0.1`.
+
+Alternatively, your application can authenticate by username and
+password credentials. This is a useful approach in several situations,
+including when you need to specify the use of a different library from
+usual one. To arrange for this, set the username and password as a
+single string separated by a slash -- e.g. "mike/swordfish" -- into
+the User Access record's Authentication field.
+
+You can set multiple fields into a single User Access record; or
+create multiple User Access records. For example, a single User Access
+record can specify both a Referring URL a username/password pair that
+can be used when running an application from a different URL. But if
+multiple Referring URLs are needed, then each must be specified in its
+own User Access record.
+
+### Tell the application to use the library
+
+In the HTML of the application, tell MKWS to authenticate on to the
+Service Proxy. When referer-based or IP-based authentication is used,
+this is very simple:
+
+ <script type="text/javascript">
+ var mkws_config = { service_proxy_auth:
+ "//sp-mkws.indexdata.com/service-proxy/?command=auth&action=perconfig" };
+ </script>
+
+> TODO This should be the default setting: see **MKWS-251**.
+
+And ensure that access to the MWKS application is from the correct
+Referrer URL or IP-range.
+
+### (Optional): access by a different virtual hostname
+
+When hostname-based authentication is in use, it's necessary to access
+the Service Proxy as the correctly named virtual host. This can be
+done by setting the `service_proxy_auth` configuration item to a
+URL containing that hostname, such as
+`//yourname.sp-mkws.indexdata.com/service-proxy/?command=auth&action=perconfig`
+
+> TODO It should be possible to change just the hostname without
+> needing to repeat the rest of the URL (protocol, path, query): see
+> **MKWS-252**.
+
+> TODO When changing the SP authentication URL, the Pazpar2 URL should
+> in general change along with it: see **MKWS-253**.
+
+### (Optional): embed credentials for access to the library
+
+When credential-based authentication is in use (username and
+password), it's necessary to pass these credentials into the Service
+Proxy when establishing the session. This can most simply be done just
+by setting the `service_proxy_auth` configuration item to a URL such as
+`//sp-mkws.indexdata.com/service-proxy/?command=auth&action=perconfig&username=mike&password=swordfish`
+
+> TODO It should be possible to add the username and password to the
+> configuration without needing to repeat the rest of the URL: see
+> **MKWS-254**.
+
+### (Optional): conceal credentials from HTML source
+
+Using a credential-based Service-Proxy authentication URL such as the
+one above reveals the the credentials to public view -- to anyone who
+does View Source on the MKWS application. This may be acceptable for
+some libraries, but is intolerable for those which provide
+authenticated access to subscription resources.
+
+In these circumstances, a more elaborate approach is necessary. The
+idea is to make a URL local to the customer that is used for
+authentication onto the Service Proxy, hiding the credentials in a
+local rewrite rule. Then local mechanisms can be used to limit access
+to that local authentication URL. Here is one way to do it when
+Apache2 is the application's web-server, which we will call
+yourname.com:
+
+Step 1: add a rewriting authentication alias to the configuration:
+
+ RewriteEngine on
+ RewriteRule /spauth/ http://sp-mkws.indexdata.com/service-proxy/?command=auth&action=check,login&username=U&password=PW [P]
+
+Step 2: set the MKWS configuration item `service_proxy_auth` to
+<http://yourname.com/spauth/>
+
+Step 3: protect access to the local path <http://yourname.com/spauth/>
+(e.g. using a `.htaccess` file).
+
+
+Choosing targets from the library
+---------------------------------
+
+MKWS applications can choose what subset of the library's targets to
+use, by means of several alternative settings on individual widgets or
+in the `mkws_config` structure:
+
+* `targets` -- contains a Pazpar2 targets string, typically of the form
+ "pz:id=" or "pz:id~" followed by a pipe-separated list of low-level
+ target IDs.
+ At present, these IDs can take one of two forms, depending on the
+ configuration of the Service Proxy being used: they may be based on
+ ZURLs (so a typical value would be something like
+ `pz:id=josiah.brown.edu:210/innopac|lui.indexdata.com:8080/solr4/select?fq=database:4902`)
+ or they may be UDBs (so a typical value would be something like
+ `pz:id=brown|artstor`)
+
+* `targetfilter` -- contains a CQL query which is used to find relevant
+ targets from the relvant library. For example,
+ `udb==Google_Images`
+ or
+ `categories=news`
+
+* `target` -- contains a single UDB, that of the sole target to be
+ used. For example,
+ `Google_Images`.
+ This is merely syntactic sugar for "targetfilter" with the query
+ `udb==NAME`
+
+For example, a `Records` widget can be limited to searching only in
+targets that have been categorised as news sources by providing an
+attribute as follows:
+
+ <div class="mkwsRecords" targetfilter='categories=news'/>
Reference Guide
----------------
+===============
-### Configuration object
+Configuration object
+--------------------
The configuration object `mkws_config` may be created before including
the MKWS JavaScript code to modify default behaviour. This structure
default value; long default values are in footnotes to keep the table
reasonably narrow.
----
+----
Element Type Default Description
-------- ----- --------- ------------
debug_level int 1 Level of debugging output to emit. 0 = none, 1 = messages, 2 = messages with
use_service_proxy bool true If true, then a Service Proxy is used to deliver searching services rather than raw
Pazpar2.
----
+----
Perhaps we should get rid of the `show_lang`, `show_perpage`,
`show_sort` and `show_switch` configuration items, and simply display the relevant menus
to lightly customise the display than my changing providing a full HTML
structure.
-#### Notes
+### Notes
1. ["sources", "subjects", "authors"]
3. [10, 20, 30, 50]
-4. http://mkws.indexdata.com/service-proxy-auth
+4. http://sp-mkws.indexdata.com/service-proxy-auth
-5. http://mkws.indexdata.com/service-proxy/
+5. http://sp-mkws.indexdata.com/service-proxy/
6. [["relevance"], ["title:1", "title"], ["date:0", "newest"], ["date:1", "oldest"]]
-### Language specification
+Language specification
+----------------------
Support for another UI language can be added by providing an entry in
the `mkws_config` object whose name is `language_` followed by the
-### jQuery UI popup invocation
+jQuery UI popup invocation
+--------------------------
The MasterKey Widget Set can be invoked in a popup window on top of the page.
<div class="mkwsResults"></div>
<div class="mkwsTargets"></div>
<div class="mkwsStat"></div>
- </div
+ </div>
----
-Element Type Default Description
--------- ----- --------- ------------
-popup_width string 880 Width of the popup window (if used), in
- pixels.
+----
+Element Type Default Description
+-------- ----- ------- ------------
+popup_width string 880 Width of the popup window (if used), in
+ pixels.
-popup_height string 760 Height of the popup window (if used), in
- pixels.
+popup_height string 760 Height of the popup window (if used), in
+ pixels.
-popup_button string input.mkwsButton (Never change this.)
+popup_button string `input.mkwsButton` (Never change this.)
-popup_modal string 0 Modal confirmation mode. Valid values are 0 or 1
+popup_modal string 0 Modal confirmation mode. Valid values are 0 or 1
-popup_autoOpen string 1 Open popup window on load. Valid values are 0 or 1
+popup_autoOpen string 1 Open popup window on load. Valid values are 0 or 1
----
+----
-### The structure of the HTML generated by the MKWS widgets
+The structure of the HTML generated by the MKWS widgets
+-------------------------------------------------------
In order to override the default CSS styles provided by the MasterKey Widget
Set, it's necessary to understand that structure of the HTML elements that are
- - -
-Copyright (C) 2013-2014 by IndexData ApS, <http://www.indexdata.com>
+Copyright (C) 2013-2014 by Index Data ApS, <http://www.indexdata.com>