Download Gemini and the Virtual Observatory

Gemini and the Virtual
Observatory
Warrick Couch, UNSW
(aided and abetted by Colin
Aspin, Gemini Obs)
Australian Virtual Observatory – 2003 Workshop
ATNF, Marsfield, 17-18 Nov 2003
The Gemini Observatory
Gemini North – Mauna Kea
Gemini South – Cerro Pachon
• twin 8.2m telescopes, 4-5 instruments per telescope,
many different modes per instrument.
Visible
Near-IR
Mid-IR
Gemini North
GMOS – imager/spec NIRI – imager/spec
(l/slit, MOS, IFU) ALTAIR – AO feed
Michelle – imager/spec
Gemini South
GMOS – imager/spec GNIRS – l/slit spec
(l/slit, MOS, IFU) Phoenix – HiRes
spec
Australian Virtual Observatory – 2003 Workshop
ATNF, Marsfield, 17-18 Nov 2003
T-ReCS – imager/spec
The Gemini Observatory
Gemini North – Mauna Kea
Gemini South – Cerro Pachon
• 6 partner countries, 1 host country (Chile), 1 host
institution (UHawaii)  broad and diverse set of scientific
users & programs.
• programs (currently) involve ‘PI’ rather than ‘survey’
science.
• majority (80-90%) of observations done in ‘queue’ mode
 science data must come with good calibration & site
weather data… but greatly facilitates a good archive!
• standard proprietary period = 18 months
• all new instruments undergo ‘science verification’  data
available in 2-3 months!
Australian Virtual Observatory – 2003 Workshop
ATNF, Marsfield, 17-18 Nov 2003
The road to a Gemini Science Archive (GSA):
• Apr 1998 : GSC/Board endorse GSA resolution.
• Sep 1998 : Gemini sponsored GSA workshop and budget line
approved by Board/Director.
• Nov 2000 : Functional and Performance Requirements Document and
Operational Concept Definition Document issued by Gemini.
• Sep 2001 : Colin Aspin became GSA Project Scientist.
• Nov 2001 : Phase II Operations and Development Plan review at CADC.
• May 2002 : Concept Design Document issued by CADC.
• Jan 2003 : Content Overview Document and Meta-Data DataBase Definition
Document issued by Gemini.
• Feb 2003 : Phase II ODP contract signed!
10 November 2003: the Gemini Science Archive
prototype (GASp) goes public!!
Australian Virtual Observatory – 2003 Workshop
ATNF, Marsfield, 17-18 Nov 2003
The Basic Archive/Prototype
(http://cadc.hia.arc.ca/gemini)
• Basic retrieval of raw science & calibration data to
(registered) users after proprietary period.
• Web-based search engine at CADC with numerous
options for retrieval. Same functionality as CFHT,
JCMT, etc archives.
• All ingested data shown in ‘results page’ but only
non-proprietary data available for download.
• FITS headers: only meta-data in prototype.
• Data retrieved via e-transfer on extraction and
packaging.
• Operational: data ingest of GN (facility instrument)
data up to date, of GS data (visitor instruments) –
90% complete. GMOS-S ingestion under way.
Australian Virtual Observatory – 2003 Workshop
ATNF, Marsfield, 17-18 Nov 2003
Transferal of Data to Archive
• Current Status : data from GN and GS are
packaged on DVD-R & shipped each 2-4 weeks.
• Data Ingestion is therefore delayed by packaging
and mailing of hard media.
• Plan is to setup e-transfer of data from GN and GS to
CADC. VPN connection to be used for security.
Staging areas at GN and GS will be monitored by
CADC and when new data is found, e-transfer,
validation and ingestion takes place (by end 2003).
• Current I2 network gives ~150KB/sec/connection.
Tested up to 6 simultaneous connections ->
900KB/sec. (typical GMOS image is ~50MB).
Australian Virtual Observatory – 2003 Workshop
ATNF, Marsfield, 17-18 Nov 2003
Data Distribution to PIs
• Current Status : data from GN and GS are
packaged on CD-R or DAT and shipped directly to PI.
• Plan is to have PI obtain data from Queue or
Classical run via CADC GSA interface via e-transfer.
• If hard-media is required by PI then CADC would
make media from archive data and ship.
• Requires special/secure access for PIs to
proprietary data (their own).
• Requires access to image data, calibration data and
meta-data.
• Timescale : mid-2004.
Australian Virtual Observatory – 2003 Workshop
ATNF, Marsfield, 17-18 Nov 2003
The fully capable GSA:
• Addition of complete meta-data access using
MDDB. This includes ‘results page’ display of metadata.
• Ingestion of results of OLDP system as on-line
preview.
• Associations of datasets (e.g. J,H,K)
• Associations of calibrations datasets.
• Improved user-interface(s).
• Batch user-interface (e-mail).
• Network data transfer to CADC : PI distribution from
CADC.
Australian Virtual Observatory – 2003 Workshop
ATNF, Marsfield, 17-18 Nov 2003
The Meta-Data Data Base (MDDB)
• MDDB resides at CADC.
• Ingest tools (provided by CADC) run at
GN/GS for ingesting meta-data into MDDB.
• Meta-data :
– augmented FITS headers,
– environmental & weather data including images
(via Gemini weather Server),
– observing proposals/programs/sequences,
– observing logs,
– publication lists (continually updated?)
Australian Virtual Observatory – 2003 Workshop
ATNF, Marsfield, 17-18 Nov 2003
Data Integrity/Security
• Gemini makes and keeps ‘local’ copies of all data
on-site.
• Data sent/e-transferred to CADC and ingested.
• CADC implementing off-site backup at UVic for
science, calibration and meta- data.
• All data e-transfers done through VPN, encrypted.
• Checksum integrity check planned at all sites
(summits, base facilities, CADC, UVic).
• Likely setup effective ‘mirror’ of GSA at GS for
performance improvement/security.
Australian Virtual Observatory – 2003 Workshop
ATNF, Marsfield, 17-18 Nov 2003
Advanced Capabilities
(to be further discussed/finalized)
• On-line, on-the-fly, data processing using Gemini
iraf package for (re-)generation of optimally reduced
data.
• Observations Tables: archive-independent tables of
scientifically interesting data supersets. Allows cross
archive searches. (NGC1333 x-ray to sub-mm)
• Source/Object Tables: archive-independent tables
of sources extracted from Gemini data and other
archives. (SVS13 fluxes 0.5um to 1mm).
• GUI improvements to support above and other
developments.
Australian Virtual Observatory – 2003 Workshop
ATNF, Marsfield, 17-18 Nov 2003
GSA within the VO World
• Overall Goal: to produce Gemini ‘data products’
which are VO-compliant.
• Data products: conform with the VO concept that
they are independent of the telescope/instrument
data taken with, and conform with some (yet to be?)
agreed VO standard for packaging data + metadata +
anything else, so that it can be read by any VO
system.
Natural extension of GSA and advanced capabilities!
Australian Virtual Observatory – 2003 Workshop
ATNF, Marsfield, 17-18 Nov 2003
Documentation
• Gemini web-site:
http:/www.gemini.edu/sciops/data/dataArchive.html
• Prototype cookbook in preparation.
• Support for GSA through Gemini HelpDesk
– Tier 1 : National Gemini Offices
– Tier 2 : CADC
– Tier 3 : Gemini (Colin Aspin)
Australian Virtual Observatory – 2003 Workshop
ATNF, Marsfield, 17-18 Nov 2003
Possible Aus-VO involvement
• Biggest Bottle-neck to generating scientific
results from Gemini observations is in the
data reduction. (IRAF-based reduction
scripts are available but have proven to be
user-unfriendly and unreliable.)
• Problem recognised by Gemini and more
resources now being devoted to developing
“fully processed” data pipelines; however,
more FTEs needed and welcomed!!
• Channel some Aus-VO resources into
developing pipelines for instrument modes
popular with Aus Gemini users??
Australian Virtual Observatory – 2003 Workshop
ATNF, Marsfield, 17-18 Nov 2003
Two points to take away…..
• GSA  increased Aus access to Gemini
through making ALL science observations
taken across the partnership available
(remembering that HST archive has
increased its usefuless by ~ x3!!)
• Gemini must be a success story for Aus
only 8m telescope facility we have a direct
share in; it is not cheap (US$1/sec); need to
MAXIMISE its use (science time + GSA) and
scientific outcomes (papers, publicity etc)!
Australian Virtual Observatory – 2003 Workshop
ATNF, Marsfield, 17-18 Nov 2003