1 It seems wrong that in IRSpy we have a database of 3000 Z39.50
2 targets, but no easy way to make them available to MasterKey
3 applications. Since Jason built the IRSpy toroid, which exposes
4 IRSpy's ZeeRex database in Torus format, it seems that have had the
5 pieces we need to present those 3000 databases to MasterKey
6 administrators; but in reality there are several significant barriers
7 to using the toroid output.
12 Of the 3000 or so records in the IRSpy database, some large proportion
13 represent servers that no longer exist, or never existed, or for which
14 we do not have access credentials. At present we can not even
15 identify which records fall into this category, because the records in
16 the database do not contain a reliability score (and therefore
17 searching by the score is not possible): the score is calculated when
18 records are displayed in the web UI, and so is "display-only"
20 TASK: calculate reliability as part of the existing
21 record-transformation process, store the score and index is,
22 make it available as a search criterion.
24 Once bad servers have been identified, we will need to decide what to
25 do with them. One option of course is just to throw the records away,
26 but I am always reluctant to discard information. It would probably
27 be better to introduce a notion of status into the IRSpy database,
28 mark the relevant records Dormant or similar, and arrange that the
29 IRSpy toroid ignores all such records.
31 TASK: add a notion of record status to the IRSpy database,
32 supported in the Web UI and in Z39.50/SRU searches, and
33 arrange for regular record maintenance to modify this setting
34 for records as required.
37 2. OVERNIGHT TEST RUNS
39 Every night, the IRSpy host runs a large series of tests on registered
40 servers to determine which are still alive, whether their capabilities
41 have changed, etc. One seventh of the server are tested each night,
42 so that the database is traversed every week. For some time now,
43 though, the overnight tests are failing -- sometimes quickly,
44 sometimes after running for a long and just occasionally not at all --
45 due to an XML/XSLT problem:
47 runtime error: file ../lib/ZOOM/../../xsl/irspy2zeerex.xsl
48 line 174 element param
49 xsltApplyXSLTTemplate: A potential infinite template recursion
52 This error masks another that was starting to manifest with increasing
53 frequency, to do with invalid characters in XML. These obviously need
54 fixing, as the overnight runs are crucial for maintaining the quality
57 TASK: fix the "infinite recursion" XSTL problem in the
60 TASK: fix the invalid XML character problem as soon as it is
66 The overlap between the set of data generated by IRSpy and what is
67 required by MasterKey is surprisingly small: at present, the IRSpy
68 Toroid provides only two fields in the records that it propagates:
69 ZURL and displayName. Other MasterKey target description fields fall
72 * Some, such as authentication, could be propagated simply by
73 extending the toroid's zeerex2torus.xsl transformation
75 * Others pertain to information that is known to IRSpy, but which
76 require additional business logic to extract: for example, cclmap_au
77 could be set to @attr 1=1003 for targets that have been determined
78 to support that access point, and @attr 1=1 for that that support
79 this but not 1003, and left blank for others. Similarly,
80 requestSyntax could be chosen from among those syntaxes that are
83 * Other fields specify information which could in principle be
84 determined by IRSpy, but for which there are presently no tests --
85 for example, a carefully design test could probably determine what
86 query encoding is in use in a given Z39.50 server, and what record
87 encoding is used in returned records -- but no such tests have been
90 * Finally, there are yet other MasterKey field that IRSpy could not
91 even in principle hope to determine: for example, URL Recipe seems
92 to be a lost cause, as it relates to web-sites that are "parallel"
93 Z39.50/SRU servers as well as to those servers themselves. Such
94 fields will need to be maintained by hand.
96 Of the as-yet unsupported MasterKey fields, some have no equivalent in
97 ZeeRex and are therefore not representable in ZeeRex records. IRSpy
98 uses an extended ZeeRex scheme for its database, so this is not an
101 In general, not only are more tests needed within IRSpy, but more
102 intelligence is needed in transforming the data that IRSpy does
103 discover into information that MasterKey can use (and that
104 intelligence would perhaps not be best expressed in XSLT).
107 4. ADMIN CONSOLE'S HANDLING OF LARGE NUMBERS OF TARGETS
109 Up till now, the MasterKey Admin Console has been used exclusively
110 with small sets of targets, not exceeding 30 or perhaps 40. It has
111 therefore been simplest to present all targets together on a single
112 page -- a strategy that will certainly not work well when we start
113 using the Admin Console to choose targets from IRSpy's much larger
114 selection. (At a rough guess, perhaps half to two thirds of the
115 registered servers in IRSpy are active and functional, so we are
116 looking at a list of 1500-2000 targets.)
118 To handle this, we would need to re-tool the Admin Console so that, as
119 well as running in its current mode (which is still appropriate in
120 many situations) it can also run in a mode where it does not show
121 complete lists of targets, but invites administrators to search for
122 specific targets; and in which it limits the number of results on each
123 page and provides a means of stepping back and forth through the pages
124 representing the full list.
126 In short, it needs to present its data in a way that more closely
127 resembles the way the IRSpy web UI works. Happily, some of the
128 substrate code for this already exists, as I imagined a search model
129 rather than a browse model when building the earliest Admin Console.
130 The big missing area is paging through long result-lists.
133 5. MISCELLANEOUS BUGS
135 Finally, Bugzilla shows 18 open IRSpy bugs at
136 http://tinyurl.com/irspybugs
137 These are of very variable importance and difficulty, but
138 unfortunately all currently appear as P4s, as they have been
139 downgraded through time to reflect their lack of urgency. (In
140 retrospect, this demonstrates that we need to tweak our Bugzilla
141 practices so as to distinguish between urgency and importance.)
143 These bugs should be reviewed, and we should determine which of them
144 ought to be solved as part of a MasterKey integration project.
146 On top of this is the wishlist:
147 http://twiki.indexdata.dk/cgi-bin/twiki/view/ID/IRSpyWishList