Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
NMMP ICES Tool Guide and Support Author: John Pearce Last Updated: 10/05/17 Accepted by Product sign off Signature: Name: Title: Date: DEFRA NMMP Interim Data Solution DOCUMENT CONTROL Amendment History Date Editor Reason Version 12-09-05 John Pearce First Draft 0.1 20-09-05 Alasdair McLay IBM PM Comments 1.0 12-10-05 Anis Patel Defra NMMP PM Comments 1.1 Sign-Off List Name Responsibility Signed off (Y/N) Beth Greenaway Marine & Waterways (NMMP DB) Project Manager Reviewers Name Responsibility Jo Rudge IBM QA Manager Alasdair McLay IBM Project Manager Distribution Name Responsibility Beth Greenaway Marine & Waterways (NMMP DB) Project Manager Related Documentation Ref Document Title Author Version 1. National Marine Monitoring Programme Database APR Number: 221 Project Summary Alasdair McLay 2.0 2. APR 221 – PCN 001 Alasdair McLay 1.0 ___________________________ 769865258 10/05/2017 DEFRA NMMP Interim Data Solution CONTENTS 1. INTRODUCTION....................................................................................................................................... 4 2. DATABASE DESCRIPTION .................................................................................................................... 6 3. THE NMMP ICES TOOL .......................................................................................................................... 7 4. LINKING TO NMMP DATABASE AND DATA ...................................................................................... 12 5. REFERENCE DATA ............................................................................................................................... 13 ___________________________ 769865258 10/05/2017 DEFRA NMMP Interim Data Solution 1. INTRODUCTION 1.1. Description The National Marine Monitoring Programme supplies a sample data report to ICES on an annual basis to fulfil the UK’s obligations under the OSPAR convention. Up to now this data has been partially collated and formatted manually. The NMMP ICES Tool is a response to the APR 221 PS Deliverable and PCN NMMPD001 to supply a data formatting tool to carry out in the region of 75% of the data formatting work for the 2004 ICES submission. The tool was used by IBM and CEFAS to successfully generate the reports that were submitted to ICES for the 1st August, 2005 deadline. The tool was originally prototyped in Microsoft Excel, however, during the development stage it was found that Excel had limited capabilities for matching and formatting the data, and therefore the development of the tool was completed using Microsoft Access 2000. The original scope for the tool was to provide a formatting and reporting solution for Biota, Sediment and Water data. However, the tool has been developed to provide a solution for Biota and Sediment as: o the water data was incomplete in terms of quality with not all labs supplying data in the same way o it is possible to submit water data in different formats to ICES and more work needs to be done to determine the most appropriate format As NMMP has not previously submitted Water data to ICES it was agreed that the tool would be developed to submit Biota and Sediment data only. The tool can be extended to include Water data at a later date but this will require significant effort working with the labs to ensure that the appropriate data has been supplied and to determine the best format for submission to ICES. 1.2. Purpose The purpose of the tool is to o import data supplied by the NMMP Bath Database, o import Laboratory AQC data o format the data o translate data against lookup data to conform to the ICES 2.2 format The tool will generate two reports in a flat file text format for BIOTA and Sediment samples. This document provides a short guide to the tool and user support notes. 1.3. Assumptions and Dependencies Ref Assumption 1. It is necessary to have installed on the execution machine a copy of Microsoft Access 2000 or later 2. The tool does not work with Microsoft Access 97 3. All tables used for reference data can be edited for changing values, deleting values or adding new values 4. There are a number of other tables that can be used as reference data and used for various functions during the data export process. The descriptions of these tables are out of scope for this document 5. The NMMP sample data is not held within the tool database itself but is link to ___________________________ 769865258 10/05/2017 DEFRA NMMP Interim Data Solution the original databases as supplied by the NMMP Bath Database ___________________________ 769865258 10/05/2017 DEFRA NMMP Interim Data Solution 2. DATABASE DESCRIPTION The Database is a Microsoft Access 2000 database and has the filename: NMMP_Link.mdb. This database contains all the code, reference data, and links to the original data supplied by NMMP Bath. The data supplied by the NMMP Bath Database is held in two Microsoft Access 97 databases. This data is linked to the NMMP_Link database as described in section 4 - Linking to NMMP Database and Data The following list shows the database filenames and description Filename Description NMMP_Link.mdb This is the main Microsoft Access 2000 database and contains all the code, reference data and data links. This also contains the queries used to export the data in ICES 2.2 format. ices2004_25jul2005.mdb This Microsoft Access 97 database is supplied by the NMMP Bath Database and contains a single table called ICES2004Data which holds the sample data for 2004 which was reported in August 2005. ICES12jul05.mdb This Microsoft Access 97 database is supplied by the NMMP Bath Database and contains a number of tables of which only one is used via the link. This is table ICES_2003DATA12Jul2005 which holds the sample data for 2003. ___________________________ 769865258 10/05/2017 DEFRA NMMP Interim Data Solution 3. THE NMMP ICES TOOL Link NMMP Bath database Add data manually Generate Sequence Numbers new data new data Run Report extracts re-run reports Process of data export/extraction The above diagram illustrates the flow of instructions to enable 3.1. o data to be exported/extracted o reports to be generated o handle data updates Export/Extract Data into ICES 2.2 Format To extract the data from the database, it is necessary to run a query which exports the data to a file as specified within the query. This file name has been pre-defined but it is possible to modify the file name and this is described in the section 3.5 - Modify Extract Output File name. It may be necessary to regenerate the ICES_SEQNO before running the export queries. This procedure will need to be repeated if any new data is supplied or new stations added. For a more detailed description and instructions for regenerating the sequences see section 3.4 - Regenerate ICES SEQNO sequence number. To execute the export, open the database and navigate to the query page by clicking the navigator on the left side as shown below: Now follow the appropriate section below Biota – see section: 3.2 - Extract BIOTA ___________________________ 769865258 10/05/2017 DEFRA NMMP Interim Data Solution Sediment – see section: 3.3 - Extract Sediment 3.2. Extract BIOTA – (query: BIOTA_##_#Export) Navigate to the Query Page as described above. Select the query and either click ‘Open’ or Double Click the item to execute. You will be prompted by the following screen, select ‘Yes’ You will be prompted to enter the year as shown below. Enter the year in ‘yyyy’ format, i.e. 2004. The query will now execute. This may take a couple of minutes. Once the query has finished, you will be prompted with a screen similar to the following: To continue generating the export file, click ‘Yes’, or to cancel, click ‘No’ The export file will now be generated and saved in the location as specified within the query. See section 3.55 - Modify Extract Output File name for more information on file names. 3.3. Extract Sediment – (query: SED_##_#Export) Navigate to the Query page as described in section 3.2. Select the query and either click ‘Open’ or Double Click the item to execute. You will be prompted by the following screen, select ‘Yes’ ___________________________ 769865258 10/05/2017 DEFRA NMMP Interim Data Solution You will be prompted to enter the year as shown below. Enter the year in ‘yyyy’ format, i.e. 2004. The query will now execute. This may take up to a couple of minutes. Once the query has finished, you will be prompted with a screen similar to the following: To continue generating the export file, click ‘Yes’ or to cancel, click ‘No’. The export file will now be generated and saved in the location as specified within the query code. See section 3.5 - Modify Extract Output File name for more information on file names. 3.4. Regenerate ICES SEQNO sequence number. The ICES 2.2 format requires each group of sample data to be sequentially numbered. This numbering takes places in a table within the database tool, by inserting the table with the expected list of stations, species group (Fish, ShellFish, Sediment) and dates. This table will need refreshing once if any new sample data entered. To generate the sequence numbers, it is necessary to delete the existing seqno tables and copy a template of the same table to reset the numbering and then re-populate the table with the new list of stations by running an insert query. To achieve this, the procedure below will need to be repeated once for BIOTA and once for Sediment. Follow the following steps carefully. NOTE: Before beginning this procedure first close the database and take a backup. 1. Select the Table page by clicking the icon on the left toolbar. 2. Select the table BIOTA_ICES_SEQNO ___________________________ 769865258 10/05/2017 DEFRA NMMP Interim Data Solution 3. Delete the table by pressing the delete key, or right clicking and selecting delete from the menu. 4. Select the table BIOTA_ICES_SEQNO_TEMPLATE and choose Edit | Copy from the menu. 5. Now choose Edit | Paste from the menu and the following dialogue box will appear 6. In the name field, type BIOTA_ICES_SEQNO 7. Choose Structure Only 8. Click OK. A new empty table is created in the correct format with the sequence numbering reset. It is now necessary to populate the table with the station data, continue with the next step. 9. Select the Query page by clicking the icon. 10. Select the BIOTA_#SEQNO_Insert query 11. Now run the query by either double clicking the query or clicking Open on the toolbar. Choose ‘Yes’ to the following dialogue box 12. Enter the year in ‘yyyy’ format e.g. 2004 in the next window ___________________________ 769865258 10/05/2017 DEFRA NMMP Interim Data Solution 13. Click ‘Yes’ to the following window 14. The sequence numbering is now complete and you can run the respective Export query to output the ICES data. Repeat the above procedure replacing the sediment names where necessary with those shown in the table below BIOTA SEDIMENT BIOTA_ICES_SEQNO SED_ICES_SEQNO BIOTA_ICES_SEQNO_Template SED_ICES_SEQNO_Template BIOTA_#SEQNO_Insert SED_#SEQNO_Insert 3.5. Modify Extract Output File name The file produced by an export query is defined within the query code. To view or modify the file name and/or its path, it is necessary to open the query in design mode. NOTE: If you intend to modify the filename or path, close the database and take a backup first. To view of modify the path or file name, select the appropriate query, either or and click the <Design> button on the toolbar The query will open in design mode and the code will be shown similar to below SELECT [SED_##_#Output].SED_DATA INTO [Text;HEADER=;DATABASE=C:\DOCUME~1\WP3USE~1\MYDOCU~1\DEFRA\Work;].[bluk04cs#txt] FROM [SED_##_#Output]; You can change path and file name in the DATABASE= section. However, you cannot change the file extension of ‘txt’ after the’#’ symbol or the query will not work. Once changed, click <save> and close the window. Test the new file name by executing/open(ing) the query as described in the previous sections and check the file has been generated as expected. ___________________________ 769865258 10/05/2017 DEFRA NMMP Interim Data Solution 4. LINKING TO NMMP DATABASE AND DATA The NMMP sample data is not held within the tool database but rather the tool links to the original databases supplied by NMMP Bath (see section 2 - Database Description). There are two links in the database and can be seen and identified on the tables page by arrows next to the names as shown below The arrows show linked tables If new data is made available, then the databases and table names must be the same for the links to continue to work. If the linked table names change or the database filename changes or the path to the linked databases change then the links can be maintained by using the ‘Linked Table Manager’ on the Tools | Database Utilities menu. The two tables are used in the main query which unions the data into a continuous dataset by the ##DATA_ALL query found on the Query page. If table names change or additional data is made available, for example, years pre 2003 or future years, e.g. 2005, then this query can be modified to include the additional linked tables. Providing reference data is available for this additional data and years (see section 5 - Reference Data). All queries will continue to work and data can be exported as previously described. Altering and modifying this query is out of scope of this document as knowledge of Microsoft Access and SQL will be required by the person making the modifications. 4.1. Adding Additional Manual Sample Data Addition Sample data can be included by adding records to the table ICES_Manual on the Tables page. To open the table for editing, double click the table name or select it first and click Open. A data grid will be displayed and data can be added, edited or deleted. 4.2. Regenerate ICES SEQNO sequence number If new sample data is received from the NMMP Bath Database or additional manual sample data is entered and where either of these contains new Station Id's or Sample Groups it is necessary for the ICES Sequence numbers to be re-generated as described in section 3.4. ___________________________ 769865258 10/05/2017 DEFRA NMMP Interim Data Solution 5. REFERENCE DATA A number of tables are used which act as reference data for generating the ICES Formatted data. The tables and purpose is described in the list below. Table Name Description/Purpose AQC_Biota This table contains all the data for the AQC sheets in respect of BIOTA. The data was manually generated by copying the data from the supplied AQC sheets from the labs AQC_Conversions This table contains the conversion factors for translating ACQ values into scientific format as required by the ICES 2.2 format in the type 21 records. AQC_Sed This table contains all the data for the AQC sheets in respect of Sediment. The data was manually generated by copying the data from the supplied AQC sheets from the labs Conversion This table contains the conversion factors for translating PARAM values into scientific format as reuired by the ICES 2.2 format for type 10 records CruiseCodes This table contains the lookup data for obtaining the Cruise Codes for the Sampt_Id station numbers and is indexed on a date range should Cruise codes change within a time period. LabCodes This table contains lookup information for translating the NMMP Lab codes into ICES Lab codes. lats and longs This contains the Lats and Longs for each Sampt_Id station. The table is indexed by species group type as lats and longs can be different depending on whether sampling Fish, Shell Fish or Sediment. Locations This table contains the JMP number for the Sampt_id station numbers. Rectangles This table contains the Rectangle values for all the Sampt_id stations numbers Sex This is the lookup table for converting NMMP Sex values into ICES Format ShipCodes This table converts the Sampt_Id staions numbers into the respective ShipCodes and is indexed by Species type for separation of shipcodes between Fish, Shell Fish and Sediment. Species This table converts the NMMP Species code in the ICES Species code name. StationType This table contains the lookup values for the Sampt_id stations into the StationType codes. ___________________________ 769865258 10/05/2017