Download IMOS/AODN Ocean Portal: tools for data delivery Roger Proctor

IMOS/AODN Ocean Portal: tools for data delivery Roger Proctor, Peter Blain, Sebastien Mancini Integrated Marine Observing System, University of Tasmania, Tas 7001 Introduction Established in 2007, Australia’s Integrated Marine Observing System (IMOS) has set out to build a collaborative research infrastructure for ocean observations. Over the last 4 years a robust information infrastructure has been developed to make all IMOS data discoverable, accessible and reusable. A broader definition of an information infrastructure could encompass the people, processes, procedures, tools, facilities, and technology which support the creation, use, and transport of information. However, the goal of an information infrastructure in the context of a regional ocean observing system should be to deliver the critical information required by society to meet multiple challenges. These range from government policy (supporting policy formulation, the monitoring of policy compliance and the assessment of policy effectiveness) to underpinning commercial decisions, and from science to public use. Important though it is, an information system needs to move beyond harmonized data exchange to promote a streamlined and efficient mechanism for meeting societal needs. IMOS is research infrastructure supporting the marine and climate research community and the IMOS Ocean portal (http://imos.aodn.org.au) aims to fulfil this function. However, Australia’s need for marine information extends beyond this community and encompasses Commonwealth agencies, State governments, private industry and the public. This paper describes the IMOS Ocean portal and how this information infrastructure has grown to support the wider community through the establishment of the Australian Ocean Data Network (AODN, http://portal.aodn.org.au), to make all marine and ocean climate data discoverable and accessible through a national ocean data commons. IMOS Ocean portal The intention in setting up the IMOS portal was to provide an intuitive, easy to use point of access for IMOS data. The approach developed a map‐based display where all IMOS data could be located and easily accessed underpinned by rich metadata which could be interrogated through a refined search capability. The features of the information infrastructure, with the discovery portal as the outlet, are simple. The guiding principles are to use Open source tools which are standards based (examples are Open Geospatial Consortium Web Map Service, Web Feature Service, Sensor Observation Service, Catalog Service for the Web, OPeNDAP, NetCDF, Open Archives Initiative Protocol for Metadata Harvesting (OAI‐PMH) with metadata conforming to the ISO19115 Marine Community Profile, with the vocabulary service linked to NASA’s Global Change Master Directory (GCMD)); its Governance is provided through information on usage policies and licence agreements, all of which are available on the IMOS website (www.imos.org.au) – it should be noted that all IMOS data is offered freely and unencumbered subject to an acknowledgement of use; it is a distributed data network connected through machine interfaces, employing (mostly) web services delivered through Geoserver, GeoNetwork, RAMADDA, and THREDDS. The IMOS distributed data network utilises the Data Fabric, Australia’s national data storage system. It enables groups and communities to easily store, maintain and share their data. The system is available to all Australian researchers and free within stated limits. The storage system, accessed through the iRODS middleware (www.irods.org), sits on top of the Australian Academic and Research Network (AARNET), the high‐capacity internet service (Fig. 1). The Data Fabric is currently managed by iVEC in Western Australia on behalf of the Research Data Storage Initiative (RDSI, http://www.rdsi.uq.edu.au/). IMOS data is distributed about this network (Fig. 2), to some extent mirroring the structure of the IMOS nodes. Data and metadata catalogs (locally known as the MEST (Metadata Entry and Search tool), the IMOS version of Geonetwork) are located around the country, and these catalogs are harvested to a master catalog maintained in Tasmania, home of the IMOS eMarine Information Infrastructure facility. The network provides IMOS with publicly available data directories, with much of the data stored in CF‐compliant netCDF data structures utilising OPeDAP and THREDDS servers, and a private archive for raw data. Delivering large datasets through THREDDS allows the utilisation of sub‐setting and aggregation services server‐side, thus reducing the data download traffic to the user. Fig. 2 – IMOS data distribution Fig. 1 – the AARNET connectivity The Australian Ocean Data Network (AODN) The vision of the AODN was articulated in 2005 by the Australian Ocean Data Centre Joint Facility (AODCJF), a joint venture between the six Commonwealth Agencies 1 with primary responsibility for marine data. This was “to put in place, by June 2011, an interoperable, online network of marine and coastal data resources, including data from the six AODCJF partner agencies, supported by standards‐based metadata, which will serve data to support Australia's science, education, environmental management and policy needs: Australia's digital ocean commons.” In 2010 the AODCJF looked to IMOS to turn this vision into a practical reality, utilising the IMOS information infrastructure to establish an AODN (http://portal.aodn.org.au). It became clear quite early on in this process that extending the infrastructure set up for IMOS, which had full control in specifying data formats and metadata descriptions, needed to be more flexible to cope with more diverse data, metadata, and delivery mechanisms. In particular, extending the distributed network to include web servers not part of the Data Fabric, and to include metadata not strictly adhering to ISO standards, caused a re‐think of the portal architecture, moving away from a Java ZK framework to a Groovy‐Grails‐extJS structure. This move also allowed us to incorporate the full Geonetwork searching capability, including facetted searches, and to set up regional views of the AODN. The first regional view is the Western Australian view, created in August 2012 1
Australian Antarctic Division, Australian Institute of Marine Science, Bureau of Meteorology, Commonwealth Science and Industrial Research Organisation, Geoscience Australia, Royal Australian Navy. (http://wa.aodn.org.au/portal/), which presents IMOS, Western Australian Marine Science Institution (WAMSI), University, large collaborative programs and State Government data in a regional context. The infrastructure (fig. 3) is now common to both IMOS and AODN. Fig. 3 – Overview of IMOS/AODN information infrastructure The list of contributors to the AODN now includes a wide range of institutions, regions and data types. Institutions now delivering data and / or metadata include Commonwealth agencies, Commonwealth government departments, State government, Universities; national‐scale programs include IMOS and CERF (Commonwealth Environmental Research Facilities) and the new National Environment Research Program’s Marine Biodiversity Hub has included AODN in its data management plan. AODN data now spans from ocean through to coastal regions and estuaries of Australia, and will soon link with the coastal program of the Terrestrial Ecosystem Research Network (TERN). Through an Australian‐New Zealand government agreement to enhance collaboration in marine research, AODN has worked with the New Zealand National Institute of Water and Atmospheric Research (NIWA) to make NIWA a node of the AODN. Data is available across a wide range of disciplines – physical, biogeochemical, biological, ecological, and, some initial socio‐
economic data – in a range of digital formats, including imagery and documents. The intention over time is to provide access to model datasets (simulation and climatology) as well as observations, and AODN has started this collection with CSIRO products CARS (CSIRO Atlas of Regional Seas), SETAS (Southeast Tasmanian pre‐operational model) and RIBBON (sample output from the whole‐Australia Coastal Ocean model). The recent establishment of a Marine and Climate node of the RDSI offers potential storage for a large collection of modelled products, which would be linked into AODN and provide resource datasets for MARVL, the Marine Virtual Laboratory funded through the National eResearch Collaboration Tools and Resources program (www.nectar.org). The functional diagram of user interaction with the AODN portal is shown in Fig. 4. A user either selects a ‘search’ query to discover what data is available, or uses the map to display data sources directly from a menu of contributing organisations/programs. The portal features an option to save searches and map displays for future use. Fig. 4 – Interacting with AODN An organisation wishing to become a node of the AODN, and make their data publicly available, is encouraged to follow a protocol involving recognised standards for both their datasets and the metadata, with delivery through a web service. To aid this process we have produced ‘the AODN Cookbook’. As an example of model / observation interaction Fig. 5 shows a simple overlay of different data sources highlighted in the layer panel. Fig. 5 – snapshot from AODN portal with data layers This brings together surface currents from the RIBBON model, sea surface temperatures from satellites, tracks from ships of opportunity showing phytoplankton abundance, tracks of glider deployments, locations of acoustic curtains for tagged fish detection and fixed mooring platforms. Further clicking on a map feature will bring up additional information about the feature and, frequently, direct access to the data. At the time of writing, the AODN contains more than 10,000 metadata references and more than half these have data or data products attached. References AARNET, http://www.aarnet.edu.au/ CF convention, http://cf‐pcmdi.llnl.gov/ extJS, http://www.sencha.com/products/extjs GCMD, http://gcmd.nasa.gov/ GeoNetwork, http://geonetwork‐opensource.org/ Geoserver, http://geoserver.org/display/GEOS/Welcome Grails, http://grails.org/ GROOVY, http://groovy.codehaus.org/ ISO19115 & MCP, https://marinemetadata.org/references/iso19115 netCDF, http://www.unidata.ucar.edu/software/netcdf/ OAI‐PMH, http://www.openarchives.org/pmh/ OPenDAP, http://www.opendap.org/ Open Geospatial Consortium, http://www.opengeospatial.org/ RAMADDA, http://www.unidata.ucar.edu/software/ramadda/ THREDDS, http://www.unidata.ucar.edu/projects/THREDDS/