Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Science Overview Architectural Status (http://www.astrogrid.ac.u k) Multi-wavelength showing the jet in M87: from top to bottom – Chandra X-ray, HST optical, Gemini mid-IR, VLA radio Nicholas Walton IoA, Cambridge credits: “NASA / Chandra X-ray Observatory / Herman Marshall (MIT)”, “NASA/HST/Eric Perlman (UMBC), “Gemini Observatory/OSCIR”, “VLA/NSF/Eric Perlman (UMBC)/Fang Zhou, Biretta (STScI)/F Owen (NRA)” N A Walton: Science Overview: AstroGrid Collaboration Meeting, Dec 13-14, 2001 p1 Printed: 12/12/01 Drivers: The Sociology of Astronomy Continuing Collectivization Facility class (common user) instruments Central development of supporting s/w (e.g. Iraf) Calibrated archives and access tools (e.g. IPAC) Information services (e.g. ADS, NED, astro-ph) Consortium projects (e.g. MACHO, SLOAN, VISTA) Evolving Developments Inter-operable Data archives, joint queries (e.g. MAST) mining (exploration and analysis tools) Information discovery tools N A Walton: Science Overview: AstroGrid Collaboration Meeting, Dec 13-14, 2001 p2 Printed: 12/12/01 Drivers: Technological Change I The Growth of Data Significant increase in number and size of telescopes In the optical: ESO's 4x8-m VLT, Gemini's 2x8-m In the x-ray: XMM-Newton, Chandra In the mm: ALMA Significant increase in size and multiplex capabilities of associated instrumentation and detectors, e.g.: In the optical: VISTA will have a Gpixel array In the radio: e-Merlin with data rates of 320 Gbps will generate N A Walton: Science Overview: AstroGrid Collaboration Meeting, Dec 13-14, 2001 p3 Printed: 12/12/01 Drivers: Technological Change II The Growth of Data Archives Many observatories hold multi-TB archives New initiatives set-up to support new observing capabilities (e.g. TeraPix) In the optical, need to ingress all-sky survey's Whole sky at 0.1 arcsec/pix is 100TB Increasing Importance of Archival Data Time on expensive facilities (e.g. HST) only awarded if the archive has been searched before hand This trend is continuing, driving observatory and user demand for Meeting, more archival data N A Walton: Science Overview: AstroGrid Collaboration Dec accessible 13-14, 2001 p4 Printed: 12/12/01 Empowering Science Driven Observational Astronomy Break down the 'wavelength' barriers e.g. Greater focus on science driven proposals encouraging the use of data sets from across a wide range of wavelengths Increase Coordination across countries e.g. The UK joins ESO: improved knowledge transfer Increase Access e.g. East European countries may not be able to fund a major new telescope but can contribute to, and access 'Virtual Observatories' p5 Printed: 12/12/01 N A Walton: Science Overview: AstroGrid Collaboration Meeting, Dec 13-14, 2001 Astronomical Drivers: Enabling New Science Linking the near and distant Universe Comparison of rest frame samples to study evolution of galaxies Combination of UV, optical and IR datasets Creating the 'Digital Sky' Temporal data measures motions in the Galactic centre, probes the creation of our Galaxy The search for extra-solar planets The planet-transit technique using federated survey data for millions of stars N A Walton: Science Overview: AstroGrid Collaboration Meeting, Dec 13-14, 2001 p6 Printed: 12/12/01 Astronomical Drivers: New Era of Surveys SuperCOSMOS (UK: now till 2002) Based on Schmidt plates - Science database ~2TB Sloan Digital Sky Survey (US: now till 2005) Dedicated CCD survey telescope: Science database ~10TB UKIDSS (UK: from 2003) IR VISTA (UK: from 2005) IR camera on 4-m UKIRT: Science database ~30TB camera on 4-m VISTA: Science database ~300TB LSST (US: from 2007-8?) Dedicated ~8-m telescope: All sky/few nights:~5000TB/yr N A Walton: Science Overview: AstroGrid Collaboration Meeting, Dec 13-14, 2001 p7 Printed: 12/12/01 Astronomical Drivers: Pre-Discovery Mining Investigating the progenitors of sources that show variability Dark matter revealed by microlensing events Planets revealed by stellar variability Formation of neutron stars revealed by GRB's Death of massive stars revealed by Type II SN The progenitor of SN1999gi is <9 M: found from mining pre-discovery HST images. (Smartt et al, 2001, ApJ, 556, L29) N A Walton: Science Overview: AstroGrid Collaboration Meeting, Dec 13-14, 2001 p8 Printed: 12/12/01 Astronomical Drivers: Rare Objects Huge data sets open up the possibility to find rare objects Those that are hard to find: Brown dwarfs, have unique red colours Those that are intrinsically rare High-z quasars, stand out due to suppression of their blue colour by the Ly- forest Hi-z QSO's found from SDSS multi-colour data: the shaded area is domain of QSO's, solid line is track for increasing z (Fan et al, 2001, ApJ, 121, 31) N A Walton: Science Overview: AstroGrid Collaboration Meeting, Dec 13-14, 2001 p9 Printed: 12/12/01 Astronomical Drivers: New Objects Large datasets open the possibility to discover new objects Those that have been missed before because they are extremely rare or short-lived Those that have previously been misclassified, revealed as outliers in new parameter space correlations DPOSS group, during searches for high-z quasars, have discovered peculiar objects, this one perhaps a BAL QSO (Djorgovski et al, 2001, PASP, 225, 52) N A Walton: Science Overview: AstroGrid Collaboration Meeting, Dec 13-14, 2001 p10 Printed: 12/12/01 Astronomical Drivers: Cosmology thru SN Ia AstroGrid will enable rapid typing of search fields via galaxy photo-z's in support of UK/EU groups, e.g. EC RTN SN network. Issue: large scale image deconvolution matching candidates to z Example SCP data from Perlmutter, 2001 snap.lbl.gov/pubdocs/ SAUL.pdf N A Walton: Science Overview: AstroGrid Collaboration Meeting, Dec 13-14, 2001 p11 Printed: 12/12/01 Astronomical Drivers: The X-ray Universe AstroGrid will enhance the ability to classify new X-ray objects and speed object sample creation Issue: federating large scale multi-λ data The ELIAS team is using Chandra, XMM, ISO, SCUBA, optical imaging and VLA data to study a number of fields: see www.roe.ac.uk/~omar/survey.html N A Walton: Science Overview: AstroGrid Collaboration Meeting, Dec 13-14, 2001 p12 Printed: 12/12/01 Astronomical Drivers: WD's and Dark Matter AstroGrid products will enable the federation of multi-epoch data to increase the number of White Dwarf's known. With IR data cool objects will be found impacting on the Dark Matter problem. Issue: TB's of multi-epoch, multicolour data, finding moving objects WD0346+246, moving at 1.3''/year, was discovered by Hambly et al., (2000, Nature, 403, 57). It is cool, but has blue colours in the IR due to collision absorption by H2 N A Walton: Science Overview: AstroGrid Collaboration Meeting, Dec 13-14, 2001 p13 Printed: 12/12/01 Astronomical Drivers: Solar Complexity Solar-B will be launched in 2005, aimed at studying the interaction between the Sun's magnetic field and its corona Solar-B will be supported by complementary space/ ground data. Science often targets events with many multi-λ observations required Image from Marshall Space Flight Centre, NASA science.msfc.nasa.gov/ssl/pad/solar/Beyond_Solar-B.htm N A Walton: Science Overview: AstroGrid Collaboration Meeting, Dec 13-14, 2001 p14 Printed: 12/12/01 AstroGrid: framework for applications Enable future development and deployment of AstroGrid and External (e.g NVO) tools: Automated detection of outliers in SDSS two colour data (Connolly et al, 2001, AJ in press) A NVO prototype of an automated discovery tool for arcs developed by A Szalay N A Walton: Science Overview: AstroGrid Collaboration Meeting, Dec 13-14, 2001 p15 Printed: 12/12/01 AstroGrid: a Typical Challenge • Problems in matching multi-λ survey data: Differences in angular resolution, s/n ratios, backgrounds, etc (Djorgovski et al, 2001, astro-ph/0108346) N A Walton: Science Overview: AstroGrid Collaboration Meeting, Dec 13-14, 2001 p16 Printed: 12/12/01 Astronomical e-Science Initiatives AstroGrid www.astrogrid.ac.uk European Grid of Solar Observations www.mssl.ucl.ac.uk/gri/egso Astrophysical Virtual Observatory AstroVirtel www.eso.org/avo ecf.hq.eso.org/astrovirtel National Virtual Observatory www.us-vo.org The Digital Sky www.cacr.caltech.edu/SDA/digital_sky.html The Virtual Sky Project virtualsky.org Sloan Digital Sky Survey: SkyServer skyserver.fnal.gov/en N A Walton: Science Overview: AstroGrid Collaboration Meeting, Dec 13-14, 2001 Skyview p17 Printed: 12/12/01 skyview.gsfc.nasa.gov Towards Phase-B: The Science Case Rapid development of science requirements Emphasis Development of 'use cases' Example: Formation of large scale structure on user input via consultation Construct unbiased cluster of galaxies sample over z to test various cosmological models of galaxy formation Need to operate on large data sets, constructing simulated surveys to test for selection effects Generate predictions of observed sample properties and compare with observed samples Development of 'White Paper' developing AstroGrid's role for Phase-B and in the context of larger VO picture N A Walton: Science Overview: AstroGrid Collaboration Meeting, Dec 13-14, 2001 p18 Printed: 12/12/01 Developing the Science Requirements Recognising the specific need to support UK science and associated key datasets was the backdrop to the genesis of the AstroGrid program Focus on community involvement Development of a few scenarios for AstroGrid, highlighting there capabilities and application to science areas Invite comment from community Interaction channels will be via The PS visiting astronomy groups Presentations, Development both face to face and on-line of a suitable questionnaire Organise 2nd AstroGrid Workshop: Mar/Apr 2002. N A Walton: Science Overview: AstroGrid Collaboration Meeting, Dec 13-14, 2001 p19 Printed: 12/12/01 Use Case Development: practical req's In addition to development of the science requirements, generic use cases are being developed, e.g.: Log on remotely, operate remotely Donate Fix a small catalogue to the grid the WCS in a FITS image Find observations via instrument footprint Synthesis Matching colour data for a set of objects objects by position with errors As the science requirements are formalised, use cases will be derived from them N A Walton: Science Overview: AstroGrid Collaboration Meeting, Dec 13-14, 2001 p20 Printed: 12/12/01 A Baseline Architecture for AstroGrid Architecture must address Top Level Design Issues: Ease of use, emphasising a uniform appearance Functionality Speed of Response Relevance Cost Reliability Scalabilty Outreach Level of data-provider buy in N A Walton: Science Overview: AstroGrid Collaboration Meeting, Dec 13-14, 2001 p21 Printed: 12/12/01 Architecture: Quality Assurance AstroGrid federation capabilities will characterise quality parameters, e.g.: Positional accuracy →reference frames Photometric accuracy →std colour systems Completeness Temporal →weight maps resolution Wavelength resolution and coverage Photometric system Image distrotions Artifacts Catalogues →criteria used to ID objects, etc. N A Walton: Science Overview: AstroGrid Collaboration Meeting, Dec 13-14, 2001 p22 Printed: 12/12/01 Architecture: Data Inclusion Model AstroGrid will potentially enable any data sets to be accessed through AstroGrid services. However the data must be adequately described. Data any restricted AstroGrid any (described) How any N A Walton: Science Overview: AstroGrid Collaboration Meeting, Dec 13-14, 2001 restricted restricted p23 Printed: 12/12/01 Unified Modelling Language (UML) representatio n of the query architecture N A Walton: Science Overview: AstroGrid Collaboration Meeting, Dec 13-14, 2001 p24 Printed: 12/12/01 Architecture Overview Major elements in the query form a circular flow: > a command flow involving user initiation > an inner 'grid' loop with bulk data flows - Client s/w WP-A3 <=======< - Query Engine WP-A2 | | - Resource Management WP-A2 <====<====< | - Resource Mapping/index WP-A2 | | | - Data Harvesting WP-A2 | | | - Data Services WP-A4 | | | - Results Management WP-A4 | | | - Data Mining WP-A4 >====>====> | - Visualisation WP-A3 |=======^ Noted above are WP areas leading expansion of the architecture in that area N A Walton: Science Overview: AstroGrid Collaboration Meeting, Dec 13-14, 2001 p25 Printed: 12/12/01 Architecture Overview: UML view N A Walton: Science Overview: AstroGrid Collaboration Meeting, Dec 13-14, 2001 p26 Printed: 12/12/01 Detailing the Architecture The Data flow diagrams (DFD's) show a functional decomposition They do not say how functions are distributed across hardware or software Grid 'layer' diagrams are developed later when the design has been matched to the analysis The DFD's emphasise the flow of data (s/w distribution), the transformations of data (s/w reuse) and the nature of data between transformations (interoperability) Flows described in a Data Dictionary Transforms described in Class-ResponsibilityN A Walton: Science Overview: AstroGrid Collaboration Dec 13-14, 2001 p27 Printed: 12/12/01 Collaboration (CRC)Meeting, notes UML representation of the 'query' process Diagram to be read with: N A Walton: Science Overview: AstroGrid Collaboration Meeting, Dec 13-14, 2001 CRC Notes Data Dictionary p28 Printed: 12/12/01 UML representation of the 'Harvest' process Diagram to be read with: CRC Notes Data Dictionary p29 Printed: 12/12/01 N A Walton: Science Overview: AstroGrid Collaboration Meeting, Dec 13-14, 2001 Extract from the Data Dictionary Query A list of constraints on data to be extracted from the Grid Authorisation Something that indicates that a user is allowed to use a resource up to some certain level Table meta-data Describe a table of data sufficiently that the table can be understood by Grid s/w and transformed as necessary Formal catalogue-extract Tabular data in AstroGrid favoured form. The catalogue extracts need to be merged and transformed. A rich format such as XML is required. N A Walton: Science Overview: AstroGrid Collaboration Meeting, Dec 13-14, 2001 p30 Printed: 12/12/01 Extract from the 'query' CRC Query-form display Displays forms to capture queries, passes contents of form to Grid, passes user's credentials to the Grid. Query scheduler Determines whether a query can be run immediately; whether is must be run in batch; whether it needs prior approval for use in resources. Resource authoriser Determines whether a given user is allowed to use a given resource at a particular level. Probably needs to be built from scratch N A Walton: Science Overview: AstroGrid Collaboration Meeting, Dec 13-14, 2001 p31 Printed: 12/12/01 The NVO Model: A Comparison Usual 'Grid' fabric, resource & collective layers added to with astro-specfic user & connectivity layers NVO focussed on (mainly) optical and near-IR large scale survey data sets. NVO layer model, figure taken from nvo-proj.pdf at www.us-vo.org N A Walton: Science Overview: AstroGrid Collaboration Meeting, Dec 13-14, 2001 p32 Printed: 12/12/01 Concluding Remarks N A Walton: Science Overview: AstroGrid Collaboration Meeting, Dec 13-14, 2001 AstroGrid is a major new UK funded e-science initiative In partnership with EU centres it will play a lead role in the realisation of a European Virtual Observatory AstroGrid is poised to significantly enhance the opportunities of the UK astronomical researchp33 Printed: 12/12/01