READ ME

System desciption and installation instructions for
Wilkinson’s Gazetteer and Bibliography of the Mines and Quarries of North Wales

Introduction
System Design

Design objectives
Architecture
User interface

Implementation

User interface
Server software
Debugging
Directory organisation
External dependencies

Installation

Introduction

This document provides a system description of the on-line (WWW) searchable implementation (data and code) of Jeremy Wilkinson's Gazetteer and Bibliography of the Mines and Quarries of North Wales.

It is provided primarily for anyone who may have to maintain the data, the system as a whole or anyone who may have to port the system to a new server.

System Design

Design objectives

The following objectives were adopted:

The user interface as simple and intuitive to use as posible.
The user interface should be interactive in that alterations to queries should update the displayed response without the need for a complete page refresh.
Data items such as site or person names etc. should be cross referenced, automatically where possible.
Data maintenance should be as simple as possible taking the above objectives into consideration.
Client-server traffic (in terms of the number of request and response transactions) should be kept to a minimum.
External dependencies should be keep to a minimum.

Architecture

System functionality is partitioned between:

The user interface, implemented as a collection of web pages (HTML files) plus CSS, JavaScript, map marker images etc.
The server software, implemented as CGI scripts and included modules.

Communication between these subsystems in the form of HTTP requests and responses. Each search request from the UI generates only a single HTTP request and its response.

User interface

This is structured as:

a search query form
a response display area

The query form uses drop-down menus for the majority of search terms and the minimum of free text fields. The use of menus guides the user in formulating searches and restricts the amount of input validation required to just the two free text input fields. These fields are validated as follows:

‘name’ input field valid entries:

site names as text (spaces and punctuation are ignored) e.g. morfadu will match MORFA DU and MORFA-DU
regular expression delimited by ‘/’ e.g. /hafod?t/ will match HAFODTY and HAFOTTY
one or more numeric site identifiers (this is mainly for use for data maintenance)

if no sites are found that match the query text an appropriate message is displayed in the response area

‘place’ field valid entries:

alphanumberic British Ordnance Survey grid reference e.g. SH61701798
numeric coordinate pair British Ordnance Survey grid reference e.g. 261706, 317989
latitude, longitude e.g. 52.741474, -4.048794
gazetteer site name (or leading part thereof) e.g. Hafotty
place name (or leading part thereof) e.g. Llanaber
postcode e.g. LL42 1AL

unrecognised input will evoke an appropriate error message adjacent to the ‘place’ field

Responses to user queries in the form of text or map images are displayed in a delineated response display area.

Implementation

User interface

This is a collection of HTML, CSS, JavaScript and map marker image files. For each type of query (site, entity and grant) there is an HTML file and a corresponding JavaScript file. A further JavaScript file contains common functions etc. and there is a common CSS file to control overall styling.

The UI:

validates free-text input fields
formats a query for POSTing to the server software
receives the response from the server
displays the response to the user as text or markers on a map as specified by the response

The web browser window can be resized without affecting system functionality.

The UI has been tested and appears to operate correctly with Firefox, Google Chrome, Microsoft Edge, Opera and Safari web browsers on Apple Macinstosh and Safari on Apple iPad.

Server software

This is a collection of Perl CGI scripts and supporting Perl modules. Perl was chosen as the scripting language because of the author’s familiarity with it.

The server software:

receives a query from the UI as CGI arguments, either from the UI query settings or from links in previous responses
searches the appropriate data file for matches
formats matches for display by expanding them and, for gazetteer display, appending cross references, source references, notes etc.
returns the response to the UI

Data files are flat files which are searched sequentially. A database was not used on the grounds of simplicity of data maintenance and code complexity.

Each script is invoked once per user request and terminates once the request is satisfied.

Debugging

The optional URL argument DEBUG will invoke the following data maintenance and debugging facilities:

DEBUG=1	Used for data maintenance: site and entity ids are shown in output (in red) and a ‘Refresh’ button is shown in gazetteer entries so that changes to data can be seen as soon as those changes have been made and the display refreshed.
DEBUG=2	Used for UI development: as DEBUG=1 plus raw query, response and some other data are written to browser JavaScript console log.

Directory organisation

Top-level directory Wilkinson/

index.html	Gazetteer ‘front page’ with an introduction to the system and link to the gazetteer search page. It includes acknowledgments and copyright information.
README.html	This file

Sub-directory Wilkinson/info/

changeLog.html	System change log
gazetteerData.html	Background information for users about the origin of, and subsequent changes to, the data used by the gazetteer.

Sub-directory Wilkinson/ui/

User interface: HTML, CSS, JavaScript, map marker images etc.

index.html	Gazetteer user interface ‘front page’
entity.html	Search form and results display for individuals, companies and organisations
entity.js	JavaScript for entity.html
grant.html	Search form and results display for crown grants
grant.js	JavaScript for grant.html
site.html	Search form and results display for mine and quarry searches
site.js	JavaScript for site.html
wilkinson.css	Style sheet for above HTML files
wilkinson.js	Common JavaScript functions for the above HTML files
jquery.js	jQuery JavaScript API library (renamed from jQuery 3.5.1.js)
marker_blue.png marker_red.png	Image files for site markers on map display

Sub-directory Wilkinson/cgi/

Perl 5 CGI scripts (.cgi) and include files (.pm)

entity.cgi	Individual, company and organisation search
grant.cgi	Crown grant search
site.cgi	Mine and quarry site search
osgb.pm	Conversion routines for latitude/longitude to/from OS British national grid and alphanumberic numeric to/from numeric grid references. Used by site.cgi. The code used for conversions between GB OS grid and latitude/logitude and their supporting subroutines is directly derived from Toby Thurston's Perl cpan OSGB.pm package Copyright © Toby Thurston (toby@cpan.org) 2002–2013.
wilkinson.pm	Common functions for CGI scripts. This module contains file names and file record formats (used with unpack() expressions) for the above scripts.

These scripts extract data in response to the search criteria, collate references, generate cross-references and apply style directives to the data for display.

Identifiers in code generally have meaningful names. Comments are generally minimal.

CGI development has used Perl v5.018.004

Sub-directory Wilkinson/data/

The design philosophy is:

Files should be human-readable (and as far as possible human-understandable) and editable with a text editor.
There should be no need to pre-process files.
Errors in data should fail silently as far as the user is concerned. (The checking program checkData.cgi can be used to identify bad or questionable data.)

File content:

Files have a header comment record indicating field order and lengths.
Blank lines and lines with # as the first character (i.e. comments) are ignored.
Comments are provided to indicate specific requirements and constraints for data.
As a consideration to the user (and data maintainer), site and entity file records should ideally be ordered by alphabetical order of names in the data. For the convenience of data maintenance, reference, role, employment and production files should be sorted numerically. Other than that, record ordering is not generally significant.

ENTITY.dat	Individual, company and organisation names – each entity has a 5-digit identifier
GRANT.dat	Crown grants
SITE.dat	Mine and quarry sites – each site has a 5-digit identifier
EMPLOYMENT.dat	Numbers of men employed above and below ground at sites
OBJECTS.dat	Limited company objects clause text
POSTCODE.dat PLACE.dat	British grid references for post codes and place names – used by site.cgi when searching for sites within a specified distance of a postcode area or named place
PRODUCTION.dat	Production records for sites
REFERENCE.dat	Bibliographical references for sites. When displayed, reference page numbers are de-duplicated, page ranges are elided correctly, overlapping and contained page number ranges are normalised and references are sorted for output.
ROLE.dat	Associates entities with roles and years active at sites (e.g. agent) or in companies (e.g. company director)

Sub-directory Wilkinson/translations/

Used to expand various fixed-length codes in data files for county, parish, product etc. to human-readable text.

COUNTY.dat	County identifier in GRANT.dat and SITE.dat to county name e.g. M translates to ‘Merioneth’
GRANT_TYPE.dat	Grant type id in GRANT.dat to grant type e.g. 3 translates to ‘take note’
NLS_MAP.dat	Map series identifier and sheet number in REFERENCE.dat to NLS sheet file name and survey/revision and publication dates (used to create links to maps on NLS website)
PARISH.dat	Parish identifier in GRANT.dat and SITE.dat to parish name e.g. AM translates to ‘Llanaber’
PRODUCT.dat	Product identifier in SITE.dat to product name e.g. MN translates to ‘manganese’
ROLE_NAME.dat	Role identifier in ROLE.dat to role name e.g. AG translates to ‘Agent’
SOURCE.dat	Source identifier in REFERENCE.dat to source title e.g. 013 translates to ‘Inspector of Mines Reports’

Sub-directory Wilkinson/utilities/

Data maintenance utilities (not available on public server)

checkData.cgi

Error and consistency checks on gazetteer data. This checks for:

limited companies with no company information
site references that do not have a corresponding site record (as may happen if a site is deleted but its references are not deleted)
coded information such as product code, role id etc. that do not have a corresponding translation
duplicate entity and reference records
incomplete entity names
missing page references
unresolved ‘see’ and ‘see also’ references

wilkinson.pm

Common functions for CGI utility scripts (alias to wilkinson.pm in Wilkinson/cgi/ directory)

Sub-directory Wilkinson/search/

index.html

Redirection page following directory reorganisations 9 May 2020

External Dependencies

The main external dependency is the use of the Ordnance Survey's Data Hub map service to source OS map tiles used for maps on which site locations can be shown.

The Leaflet open-source JavaScript library is used to provide map display and related functionality.

The jQuery JavaScript library is used to simplify the JavaScript code.

The gazetteer reference data contain links to a number of external sites. In particular use is made of National Library of Scotland's online historic OS maps. All external links open in a new window and their non-availablity does not affect the operation of the gazetteer.

Installation

An appropriate Web server and Perl interpreter installation is required.

The directory should be put in a suitable location for the web server to serve the html pages and execute the CGI scripts. (The server configuration should allow execution of the CGI scripts.) The system footprint is approximately 2.3 MByte.

CGI scripts should have ‘execute’ permission set for the server process. (If in doubt use 'world' to allow execution by any process.)

An API key for the Ordnance Survey data is required.

Return to Gazetteer front page