Download ppt

Astronomical Data Archiving and Curation
Clive Page
AstroGrid Project
University of Leicester
2004 March 22
Importance of Data Archiving in Astronomy
• No observation can be repeated exactly, as the
sky is always changing
– After a violent event (e.g. supernova explosion)
earlier observations are crucial
• Observations over a long period can identify
– Variability
– Proper motions
• In recent years all data come in digital form
• Important earlier datasets on photographic
plates have now mostly been digitised.
Principal Data Types in Archives
•
•
•
•
Raw data from telescopes
Observing logs
Calibration datasets
Calibrated/reduced data:
– Images
– Spectra
– Time-series
• Derived data products:
– Source catalogues
– Sky survey image collections
Data Formats
• A variety, but FITS format predominates:
– FITS can store arrays and tables, and encapsulates
data and metadata, but…
• Standards have evolved, older FITS files less compatible
• Individual observatory conventions also exist
• Metadata vital - sometimes to be found only:
– In associated software packages or documentation
– In the heads of those developing the software
Important UK data archive sites
• Cambridge - Astronomical Survey Unit (CASU):
– INT wide-field survey, APM catalogue, VIZIER mirror,
UKIRT archive. In future: WFCAM, VISTA.
• Edinburgh – Wide-field Astronomy Unit (WFAU)
– SuperCOSMOS images and catalogue, 6df galaxy
survey, SLOAN DSS copy. In future: WFCAM, VISTA.
• Leicester - Data Archive Service (LEDAS):
– EXOSAT, GINGA, ASCA, ROSAT, XMM; Chandra
mirror, many optical datasets. In future: SWIFT,
SuperWASP source archive.
Important UK data archive sites (continued)
• Manchester - Jodrell Bank:
– Merlin, HI surveys, European VLBI datasets, pulsar
catalogues. Future: e-Merlin archive.
• Rutherford Laboratory:
– World Data Centre for STP, CLUSTER and ISO UK
data centres, Starlink software collection and data
archive. In future: SuperWASP image archive.
• UCL - Mullard Space Science Laboratory:
– YOHKOH, SOHO, TRACE, ReSIK and other
solar/STP archives.
Database management systems
• DBMS currently used by UK archives include:
–
–
–
–
–
–
–
–
–
BROWSE – written at ESOC/ESTEC in 1980s.
DB2 (IBM)
Ingres
miniSQL – free simple DBMS
MySQL – open source, supports many web sites
PostgreSQL – open source, good spatial indexing
SQL Server (Microsoft)
Sybase ASE
WFCtools – written at Harvard/SAO for accessing
large optical catalogues
User access methods
• Residual telnet/ssh services
– Allows registered users to perform DBMS operations
store their own subsets etc.
– Mostly obsolescent
• FTP access for large downloads
• Web interfaces use CGI with Perl, PHP, or
Python
– Results mostly returned as HTML tables/GIFs, with
some FITS and VOtable.
• No use (pre-AstroGrid) of XML-based Web
Services (Xforms, SOAP, WSDL etc.)
Problems – (1) technical
• Data storage: thanks to Moore’s Law, new datasets are
much bigger than old ones. May get adequate storage
for existing data from:
– new big projects like WFCAM, SWIFT, e-MERLIN, VISTA?
– SRIF funding?
• International Virtual Observatory Alliance (IVOA) is
developing new standards e.g. for tabular data, registry,
query language.
– These have to be implemented before fully stable.
• DBMS: freeware like MySQL, PostgreSQL improving
rapidly, probably adequate.
– If not, licence costs may be substantial.
• Database middleware (OGSA-DAI, ELDAS)
– still developing, not quite ready for large-scale use
Problems – (2) structural
• Data preservation requires migration to new
platforms, new DBMS every few years
• Many DBMS in use are incapable of supporting
functionality required e.g. no spatial indexing
– Also implies migration to new DBMS
• AstroGrid (and other VO projects) will supply the
middleware, but have no remit (and no funding)
to update the archives themselves.
• Serious data mining research will require serious
processing power near the data stores (e.g. an
Astronomical Data Warehouse).
Problems – (3) managerial
• VO software from AstroGrid includes MySpace:
a temporary user space on remote systems.
– Optional, but highly desirable because of need to
“shift the results not the data”
– will sites give space to users unknown to them?
– how to administer many ad-hoc groups of users?
• Creation of the VO Registry will require
considerable input from managers of existing
data archives – exact mechanism TBD.
Manpower
Additional manpower needed for:
• Migration of existing data collections to new
platforms, and often to new DBMS
• Installation of AstroGrid and other VO software
• Provision of metadata to the Registry
• Implementation and operation of MySpace
• Setting up astronomical data warehouse
facilities at a few sites
Funding problems
• SRIF funding is for hardware only, not
manpower
• AstroGrid2 bid failed to get support for elements
of data centre support
• PPARC grant applications to support data
archiving and curation have an unhappy history:
they tend to fall between research and projects
funding lines.
Summary
•
Archives have a vital role in astronomy
– They are basically in good shape in that no
important bits have been lost (as far as I know)
– But we have been muddling through
•
•
•
•
Technical problems look soluble
Data storage – we may be able to find enough
Much work needed on current archives for
them to survive into the VO era.
Additional skilled manpower will be essential
– sources of support for this are lacking
•
Continuity is vital for archives – this is a longterm problem with no obvious solution.