Download NMMP ICES Tool Guide and Support

NMMP ICES Tool Guide and Support
Author: John Pearce
Last Updated: 10/05/17
Accepted by Product sign off
Signature:
Name:
Title:
Date:
DEFRA NMMP Interim Data Solution
DOCUMENT CONTROL
Amendment History
Date
Editor
Reason
Version
12-09-05
John Pearce
First Draft
0.1
20-09-05
Alasdair McLay
IBM PM Comments
1.0
12-10-05
Anis Patel
Defra NMMP PM Comments
1.1
Sign-Off List
Name
Responsibility
Signed off (Y/N)
Beth Greenaway
Marine & Waterways (NMMP DB) Project Manager
Reviewers
Name
Responsibility
Jo Rudge
IBM QA Manager
Alasdair McLay
IBM Project Manager
Distribution
Name
Responsibility
Beth Greenaway
Marine & Waterways (NMMP DB) Project Manager
Related Documentation
Ref
Document Title
Author
Version
1.
National Marine Monitoring Programme
Database APR Number: 221 Project
Summary
Alasdair McLay
2.0
2.
APR 221 – PCN 001
Alasdair McLay
1.0
___________________________
769865258
10/05/2017
DEFRA NMMP Interim Data Solution
CONTENTS
1.
INTRODUCTION....................................................................................................................................... 4
2.
DATABASE DESCRIPTION .................................................................................................................... 6
3.
THE NMMP ICES TOOL .......................................................................................................................... 7
4.
LINKING TO NMMP DATABASE AND DATA ...................................................................................... 12
5.
REFERENCE DATA ............................................................................................................................... 13
___________________________
769865258
10/05/2017
DEFRA NMMP Interim Data Solution
1.
INTRODUCTION
1.1.
Description
The National Marine Monitoring Programme supplies a sample data report to ICES on an annual basis to
fulfil the UK’s obligations under the OSPAR convention. Up to now this data has been partially collated and
formatted manually. The NMMP ICES Tool is a response to the APR 221 PS Deliverable and PCN NMMPD001 to supply a data formatting tool to carry out in the region of 75% of the data formatting work for the 2004
ICES submission. The tool was used by IBM and CEFAS to successfully generate the reports that were
submitted to ICES for the 1st August, 2005 deadline.
The tool was originally prototyped in Microsoft Excel, however, during the development stage it was found
that Excel had limited capabilities for matching and formatting the data, and therefore the development of the
tool was completed using Microsoft Access 2000.
The original scope for the tool was to provide a formatting and reporting solution for Biota, Sediment and
Water data. However, the tool has been developed to provide a solution for Biota and Sediment as:
o
the water data was incomplete in terms of quality with not all labs supplying data in the same way
o
it is possible to submit water data in different formats to ICES and more work needs to be done to
determine the most appropriate format
As NMMP has not previously submitted Water data to ICES it was agreed that the tool would be developed
to submit Biota and Sediment data only. The tool can be extended to include Water data at a later date but
this will require significant effort working with the labs to ensure that the appropriate data has been supplied
and to determine the best format for submission to ICES.
1.2.
Purpose
The purpose of the tool is to
o
import data supplied by the NMMP Bath Database,
o
import Laboratory AQC data
o
format the data
o
translate data against lookup data to conform to the ICES 2.2 format
The tool will generate two reports in a flat file text format for BIOTA and Sediment samples.
This document provides a short guide to the tool and user support notes.
1.3.
Assumptions and Dependencies
Ref
Assumption
1.
It is necessary to have installed on the execution machine a copy of Microsoft
Access 2000 or later
2.
The tool does not work with Microsoft Access 97
3.
All tables used for reference data can be edited for changing values, deleting
values or adding new values
4.
There are a number of other tables that can be used as reference data and used
for various functions during the data export process. The descriptions of these
tables are out of scope for this document
5.
The NMMP sample data is not held within the tool database itself but is link to
___________________________
769865258
10/05/2017
DEFRA NMMP Interim Data Solution
the original databases as supplied by the NMMP Bath Database
___________________________
769865258
10/05/2017
DEFRA NMMP Interim Data Solution
2.
DATABASE DESCRIPTION
The Database is a Microsoft Access 2000 database and has the filename: NMMP_Link.mdb. This database
contains all the code, reference data, and links to the original data supplied by NMMP Bath.
The data supplied by the NMMP Bath Database is held in two Microsoft Access 97 databases. This data is
linked to the NMMP_Link database as described in section 4 - Linking to NMMP Database and Data
The following list shows the database filenames and description
Filename
Description
NMMP_Link.mdb
This is the main Microsoft Access 2000 database and contains all the code,
reference data and data links.
This also contains the queries used to export the data in ICES 2.2 format.
ices2004_25jul2005.mdb
This Microsoft Access 97 database is supplied by the NMMP Bath Database and
contains a single table called ICES2004Data which holds the sample data for
2004 which was reported in August 2005.
ICES12jul05.mdb
This Microsoft Access 97 database is supplied by the NMMP Bath Database and
contains a number of tables of which only one is used via the link. This is table
ICES_2003DATA12Jul2005 which holds the sample data for 2003.
___________________________
769865258
10/05/2017
DEFRA NMMP Interim Data Solution
3.
THE NMMP ICES TOOL
Link NMMP
Bath database
Add data manually
Generate
Sequence
Numbers
new data
new data
Run Report
extracts
re-run reports
Process of data export/extraction
The above diagram illustrates the flow of instructions to enable
3.1.
o
data to be exported/extracted
o
reports to be generated
o
handle data updates
Export/Extract Data into ICES 2.2 Format
To extract the data from the database, it is necessary to run a query which exports the data to a file as
specified within the query. This file name has been pre-defined but it is possible to modify the file name and
this is described in the section 3.5 - Modify Extract Output File name.
It may be necessary to regenerate the ICES_SEQNO before running the export queries. This procedure will
need to be repeated if any new data is supplied or new stations added. For a more detailed description and
instructions for regenerating the sequences see section 3.4 - Regenerate ICES SEQNO sequence number.
To execute the export, open the database and navigate to the query page by clicking the
navigator on the left side as shown below:
Now follow the appropriate section below
Biota – see section: 3.2 - Extract BIOTA
___________________________
769865258
10/05/2017
DEFRA NMMP Interim Data Solution
Sediment – see section: 3.3 - Extract Sediment
3.2.
Extract BIOTA – (query: BIOTA_##_#Export)
Navigate to the Query Page as described above.
Select the
query and either click ‘Open’ or Double Click the item to execute.
You will be prompted by the following screen, select ‘Yes’
You will be prompted to enter the year as shown below.
Enter the year in ‘yyyy’ format, i.e. 2004.
The query will now execute. This may take a couple of minutes. Once the query has finished, you will be
prompted with a screen similar to the following:
To continue generating the export file, click ‘Yes’, or to cancel, click ‘No’
The export file will now be generated and saved in the location as specified within the query.
See section 3.55 - Modify Extract Output File name for more information on file names.
3.3.
Extract Sediment – (query: SED_##_#Export)
Navigate to the Query page as described in section 3.2.
Select the
query and either click ‘Open’ or Double Click the item to execute.
You will be prompted by the following screen, select ‘Yes’
___________________________
769865258
10/05/2017
DEFRA NMMP Interim Data Solution
You will be prompted to enter the year as shown below.
Enter the year in ‘yyyy’ format, i.e. 2004.
The query will now execute. This may take up to a couple of minutes.
Once the query has finished, you will be prompted with a screen similar to the following:
To continue generating the export file, click ‘Yes’ or to cancel, click ‘No’.
The export file will now be generated and saved in the location as specified within the query code. See
section 3.5 - Modify Extract Output File name for more information on file names.
3.4.
Regenerate ICES SEQNO sequence number.
The ICES 2.2 format requires each group of sample data to be sequentially numbered. This numbering
takes places in a table within the database tool, by inserting the table with the expected list of stations,
species group (Fish, ShellFish, Sediment) and dates.
This table will need refreshing once if any new sample data entered.
To generate the sequence numbers, it is necessary to delete the existing seqno tables and copy a template
of the same table to reset the numbering and then re-populate the table with the new list of stations by
running an insert query. To achieve this, the procedure below will need to be repeated once for BIOTA and
once for Sediment. Follow the following steps carefully.
NOTE: Before beginning this procedure first close the database and take a backup.
1. Select the Table page by clicking the
icon on the left toolbar.
2. Select the table BIOTA_ICES_SEQNO
___________________________
769865258
10/05/2017
DEFRA NMMP Interim Data Solution
3. Delete the table by pressing the delete key, or right clicking and selecting delete from the menu.
4. Select the table BIOTA_ICES_SEQNO_TEMPLATE and choose Edit | Copy from the menu.
5. Now choose Edit | Paste from the menu and the following dialogue box will appear
6. In the name field, type BIOTA_ICES_SEQNO
7. Choose Structure Only
8. Click OK. A new empty table is created in the correct format with the sequence numbering reset. It
is now necessary to populate the table with the station data, continue with the next step.
9. Select the Query page by clicking the
icon.
10. Select the BIOTA_#SEQNO_Insert query
11. Now run the query by either double clicking the query or clicking Open on the toolbar. Choose ‘Yes’
to the following dialogue box
12. Enter the year in ‘yyyy’ format e.g. 2004 in the next window
___________________________
769865258
10/05/2017
DEFRA NMMP Interim Data Solution
13. Click ‘Yes’ to the following window
14. The sequence numbering is now complete and you can run the respective Export query to output the
ICES data.
Repeat the above procedure replacing the sediment names where necessary with those shown in the table
below
BIOTA
SEDIMENT
BIOTA_ICES_SEQNO
SED_ICES_SEQNO
BIOTA_ICES_SEQNO_Template
SED_ICES_SEQNO_Template
BIOTA_#SEQNO_Insert
SED_#SEQNO_Insert
3.5.
Modify Extract Output File name
The file produced by an export query is defined within the query code. To view or modify the file name
and/or its path, it is necessary to open the query in design mode.
NOTE: If you intend to modify the filename or path, close the database and take a backup first.
To view of modify the path or file name, select the appropriate query, either
or
and click the <Design> button on the toolbar
The query will open in design mode and the code will be shown similar to below
SELECT [SED_##_#Output].SED_DATA INTO
[Text;HEADER=;DATABASE=C:\DOCUME~1\WP3USE~1\MYDOCU~1\DEFRA\Work;].[bluk04cs#txt]
FROM [SED_##_#Output];
You can change path and file name in the DATABASE= section. However, you cannot change the file
extension of ‘txt’ after the’#’ symbol or the query will not work.
Once changed, click <save> and close the window. Test the new file name by executing/open(ing) the query
as described in the previous sections and check the file has been generated as expected.
___________________________
769865258
10/05/2017
DEFRA NMMP Interim Data Solution
4.
LINKING TO NMMP DATABASE AND DATA
The NMMP sample data is not held within the tool database but rather the tool links to the original databases
supplied by NMMP Bath (see section 2 - Database Description).
There are two links in the database and can be seen and identified on the tables page by arrows next to the
names as shown below
The arrows show linked tables
If new data is made available, then the databases and table names must be the same for the links to
continue to work.
If the linked table names change or the database filename changes or the path to the linked databases
change then the links can be maintained by using the ‘Linked Table Manager’ on the Tools | Database
Utilities menu.
The two tables are used in the main query which unions the data into a continuous dataset by the
##DATA_ALL query found on the Query page.
If table names change or additional data is made available, for example, years pre 2003 or future years, e.g.
2005, then this query can be modified to include the additional linked tables. Providing reference data is
available for this additional data and years (see section 5 - Reference Data). All queries will continue to work
and data can be exported as previously described.
Altering and modifying this query is out of scope of this document as knowledge of Microsoft Access and
SQL will be required by the person making the modifications.
4.1.
Adding Additional Manual Sample Data
Addition Sample data can be included by adding records to the table ICES_Manual on the Tables page.
To open the table for editing, double click the table name or select it first and click Open. A data grid will be
displayed and data can be added, edited or deleted.
4.2.
Regenerate ICES SEQNO sequence number
If new sample data is received from the NMMP Bath Database or additional manual sample data is entered
and where either of these contains new Station Id's or Sample Groups it is necessary for the ICES Sequence
numbers to be re-generated as described in section 3.4.
___________________________
769865258
10/05/2017
DEFRA NMMP Interim Data Solution
5.
REFERENCE DATA
A number of tables are used which act as reference data for generating the ICES Formatted data. The
tables and purpose is described in the list below.
Table Name
Description/Purpose
AQC_Biota
This table contains all the data for the AQC sheets in respect of BIOTA. The
data was manually generated by copying the data from the supplied AQC sheets
from the labs
AQC_Conversions
This table contains the conversion factors for translating ACQ values into
scientific format as required by the ICES 2.2 format in the type 21 records.
AQC_Sed
This table contains all the data for the AQC sheets in respect of Sediment. The
data was manually generated by copying the data from the supplied AQC sheets
from the labs
Conversion
This table contains the conversion factors for translating PARAM values into
scientific format as reuired by the ICES 2.2 format for type 10 records
CruiseCodes
This table contains the lookup data for obtaining the Cruise Codes for the
Sampt_Id station numbers and is indexed on a date range should Cruise codes
change within a time period.
LabCodes
This table contains lookup information for translating the NMMP Lab codes into
ICES Lab codes.
lats and longs
This contains the Lats and Longs for each Sampt_Id station. The table is
indexed by species group type as lats and longs can be different depending on
whether sampling Fish, Shell Fish or Sediment.
Locations
This table contains the JMP number for the Sampt_id station numbers.
Rectangles
This table contains the Rectangle values for all the Sampt_id stations numbers
Sex
This is the lookup table for converting NMMP Sex values into ICES Format
ShipCodes
This table converts the Sampt_Id staions numbers into the respective
ShipCodes and is indexed by Species type for separation of shipcodes between
Fish, Shell Fish and Sediment.
Species
This table converts the NMMP Species code in the ICES Species code name.
StationType
This table contains the lookup values for the Sampt_id stations into the
StationType codes.
___________________________
769865258
10/05/2017