Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Astrogrid Architecture Review Presenters: Elizabeth Auden David Giaretta Tony Linde University of Leicester Friday, 17 May 2002 Astrogrid Architecture Review Clive Page Guy Rixon University of Leicester 17 May 2002 Architecture Review: Agenda 09:50 10:00 10:10 Introduction Architecture Challenges Core Use Cases Tony Linde Elizabeth Auden Elizabeth Auden 10:20 11:00 11:15 11:45 12:10 Sequence Diagrams Tea and Coffee Domain Model IAA Workflow Elizabeth Auden 12:25 12:45 13:30 14:00 Databases Lunch (served in room) Component Model Technology Clive Page 14:30 What Next?: Architecture, Phase B, WP5, Training Finish Tony Linde 15:00 Astrogrid Architecture Review David Giaretta Guy Rixon Tony Linde Tony Linde Tony Linde University of Leicester 17 May 2002 Architecture Challenges 6 Architecture Challenges: All use cases: IAA, interoperability, • • • compute resources Most use cases: Catalogue queries Some use cases: Software interfaces • • • Astrogrid Architecture Review Certification: Identification, authorization, authentication Catalogues: Complex queries, searches, joins, correlation Distributed Catalogues: Distributed queries, distributed joins Compute Resources: Upload/execute code, scheduler, estimator Interfaces: Astronomical packages, client software Interoperability: Data, metadata University of Leicester 17 May 2002 Core Use Cases USE CASE CHALLENGE DISCIPLINE • Brown Dwarf Science Problem Catalogue search and join Optical • Deep Field Survey Correlation Complex query, correlation Radio, Optical • Galaxy Clustering Catalogue search, upload code Optical • High Z Quasars Complex query and join Optical, X-ray • Low Surface Brightness Catalogue search, upload code Optical • Magnetic Storm Onset Complex queries, models STP • Solar Coronal Waves Complex queries, upload code Solar • Solar / STP Event Coincidence Distributed queries, join Solar / STP • Solar / Stellar Flare Redshift Distributed queries, upload code Solar / Astro • Supernova Galaxy Environment Catalogue search, upload code Optical / IR Astrogrid Architecture Review University of Leicester 17 May 2002 Sequence Diagrams 3 Sample • Low Surface Brightness: Optical astrophysics: An astronomer wants to identify faint galaxies and extended objects by comparing source flux with background flux. • Solar Coronal Waves Solar physics: A solar physicist wants to examine solar activity in the hours preceding coronal waves and flares to identify causes of solar events. • Magnetic Storm Onset: Solar terrestrial physics: A solar terrestrial physicist wants to study the morphology of the Earth's magnetosphere tail during the onset of geomagnetic storms. Use Cases: Low Surface Brightness Solar Coronal Waves Magnetic Storm Onset Astrogrid Architecture Review University of Leicester 17 May 2002 Detail: Low Surface Brightness Astronomer Resource Catalogue Image Catalogue Compute Resource My Space Search for image catalogues with magnitude / depth metadata Return list of catalogues Search for images of point sources / bright structures Upload image data Upload algorithm to remove point sources / bright structures Upload smoothing algorithm Upload algorithm to find extended sources Return extended source list Astrogrid Architecture Review University of Leicester 17 May 2002 Detail: Solar Coronal Waves Astronomer Resource Catalogue Search for flare catalogues Catalogue Source Image / Spectra Compute Resource My Space Return catalogue list Complex Query: search flare catalogue for time intervals given an intensity inequality Return time intervals Search for catalogues with spectral / image data for intense flare time intervals Return catalogue list Search catalogue for spectra and images based on time interval and location criteria Upload images and spectra Upload user image difference algorithm Upload user pattern-matching algorithm Return list of difference images matching pattern Astrogrid Architecture Review Upload images University of Leicester 17 May 2002 Detail: Magnetic Storm Onset Physicist Resource Catalogue DST Catalogue Mag. Sphr. Model Spacecraft Metadata Instrument Data My Space Search for DST catalogues Return list of DST catalogues Query for time intervals where DST index > X Return time intervals Search for models of magnetosphere Return list of models Search for spacecraft with magnetometers Return list of spacecraft Complex Query: find regions observed during time intervals while magnetometer is turned on Return regions; tool converts coordinate system Get magnetometer data by region and time interval Run data through magnetosphere model Return result list Astrogrid Architecture Review University of Leicester 17 May 2002 Domain Model Astrogrid Architecture Review University of Leicester 17 May 2002 Low Surface Brightness Together Sequence Diagram: Astrogrid Architecture Review University of Leicester 17 May 2002 Low Surface Brightness Together Collaboration Diagram: Astrogrid Architecture Review University of Leicester 17 May 2002 Solar Coronal Waves Together Sequence Diagram: Astrogrid Architecture Review University of Leicester 17 May 2002 Solar Coronal Waves Together Collaboration Diagram: Astrogrid Architecture Review University of Leicester 17 May 2002 Magnetic Storm Onset Together Sequence Diagram: Astrogrid Architecture Review University of Leicester 17 May 2002 Magnetic Storm Onset Together Collaboration Diagram: Astrogrid Architecture Review University of Leicester 17 May 2002 Architecture Review: IAA Guy Rixon • Identification • Authorization • Identification Astrogrid Architecture Review University of Leicester 17 May 2002 Architecture Review: Workflow Workflow: 7 workflow scenarios illustrate interactions between Astrogrid, administrators, and users. Astrogrid Architecture Review • Create Action • Job Admin • Orchestrate Job • Program Admin • Resource ACL • Resource Admin • Run Job University of Leicester 17 May 2002 Workflow: Create Action Astrogrid Architecture Review University of Leicester 17 May 2002 Workflow: Job Admin Astrogrid Architecture Review University of Leicester 17 May 2002 Workflow: Orchestrate Job Astrogrid Architecture Review University of Leicester 17 May 2002 Workflow: Program Admin Astrogrid Architecture Review University of Leicester 17 May 2002 Workflow: Resource ACL Astrogrid Architecture Review University of Leicester 17 May 2002 Workflow: Resource Admin Astrogrid Architecture Review University of Leicester 17 May 2002 Workflow: Run Job Astrogrid Architecture Review University of Leicester 17 May 2002 Database Technology Database Techonology: Internet DBMS • Aims and Elements • Queries • Resource Discovery • Interoperability • DBMS • Sky Indexing Astrogrid Applications Astrogrid Architecture Review University of Leicester 17 May 2002 Database Tech: Aims / Elements Assumed Astrogrid Aims •To federate existing data archives •To provide data mining facilities (anything that can't be done in-situ) •To devise a scaleable infrastructure for archives suitable for future projects like WFCAM, VISTA... Assumed Architectural Elements •Data Portal - provides user interface and intelligence, knows about resource registry at least. •Resource Registry - replicated directory of archives - level of detail under debate. •Data Warehouse - provides cpu power and disc space for operations not feasible elsewhere. Expect more than one of each of these, many examples will be co-located with main archive sites. Astrogrid Architecture Review University of Leicester 17 May 2002 Database Tech: Queries Types of Query •Positional queries - specify (RA, DEC, radius) - or several of them. May query any of: •Source catalogues •Image repositories •Observatory archives •Bibliographical archives •Non-positional queries •Cross-matching sources in two (or more) catalogues - fuzzy join is basic step for many data mining operations. •Selection from tables by reference to properties - may require sequential scan •Statistical operations on tables - ditto •Data mining queries requiring special algorithms (clustering, time-series, density functions, etc) Astrogrid Architecture Review University of Leicester 17 May 2002 Database Tech: Resource Discovery Astrogrid, AVO, NVO: agreement on need for unique but replicated Astronomical Resource Registry. NVO seems to be thinking of a human-managed registry. Could the registry be updated by a robot which uses WSDL descriptions of each archive? • UDDI has some of the right properties, but very business-oriented. • Level of detail subject of active debate. Possible Registry Information: • Name of service • URL • Physical location • Contact details of responsible person • Web interfaces supported (CGI, ASU, SOAP, WSDL, ...) • Holding type (source catalogues, images, observing logs, raw data, bibliographic info., …) • Waveband (radio, IR, optical, UV, X-ray, ...) • Sky coverage (limits in Dec or Galactic latitude, or details of each pointing?) • Access restrictions (by date of observation, location of user, max download size) • Spatial resolution (images), or positional error (catalogues) • Epoch of observation • Data volume • Export formats supported (FITS, VOtable, CSV, PNG, ...) Astrogrid Architecture Review University of Leicester 17 May 2002 Database Tech: Interoperability Implementation of resource discovery will rely on the adoption of common standards by Astrogrid, AVO, NVO, and the others. The essential interfaces are mainly those to databases. • Standards for queries. SQL inadequacies include: •Too batch-oriented, hard to save intermediate results to refine a query •Hard to do positional queries •Poor support for statistical functions •Text output only in most cases - need binary, interfaces to visualisation packages, etc. •Ignores or loses metadata. • Astronomical Query Language: • Develop our own? Interpreted by the portal or passed to the DBMS? • Standards for results: •VOtable for tabular results: v1.0 drafted. •Need similar standards for images and bulk data: discussed on US-VO metadata lists. • Standards for metadata: useful starts are FITS, HFWG, UCD, etc. Much more needed. • Database Task Force developing Web Services wrappers around databases (using JDBC). Astrogrid Architecture Review University of Leicester 17 May 2002 Database Tech: DBMS There are plans to evaluate several database management systems: • MySQL - fast, basic, used by several astronomical archives sites already • Postgres - object-relational, has R-tree indexing • DB2 (IBM) - heavyweight, free/cheap for research use - may get help in testing • Oracle - market leader, all possible features - may get help in testing • Sybase - used by many astronomical data archive sites already • Microsoft SQL Server - used by Sloan/JHU to replace Objectivity, easy to use; however, available for Windows only. Principal types of query to be evaluated: • I/O intensive queries, e.g. scans of entire table, statistical queries. • Index-intensive queries, e.g. join of two large catalogues (fuzzy join). Other important features: • Licence cost • Ease of use • Support for parallelism • Easy upload of users' own data, easy download/interface with astronomical applications. Astrogrid Architecture Review University of Leicester 17 May 2002 Database Tech: Sky Indexing Indexing the sky is surprisingly difficult to index in the 2-d plane, but it is even harder to index spherical surface. Two promising methods: •Postgres and Oracle support R-tree indexes on rectangles - should evaluate. •Alternatively can map 2-d surface to points on a line then use an ordinary Btree •Suitable mapping functions include HTM and HEALPix, simpler ones possible. •Devised PCODE method which allows fuzzy join to be done efficiently. Astrogrid Architecture Review University of Leicester 17 May 2002 Interface Model Astrogrid Architecture Review University of Leicester 17 May 2002 Deployment Model Astrogrid Architecture Review University of Leicester 17 May 2002 Astrogrid Technology • Technology to implement Astrogrid challenges. Astrogrid Architecture Review University of Leicester 17 May 2002 What Next? Astrogrid Architecture Review • Architecture • Phase B • WP 5 • Training University of Leicester 17 May 2002