Download The Heliophysics Integrated Observatory

Heliophysics Integrated Observatory (HELIO)
HELIO
The
Heliophysics Integrated Observatory
Bob Bentley (UCL-MSSL)
Andre Csillaghy (FHNW)
Jean Abourdarham (Obs. Paris)
20 November 2008
European Space Weather Week, Brussels
Heliophysics
Heliophysics Integrated Observatory (HELIO)
Heliophysics explores the Sun-Solar System Connection
Space weather is a subset of Heliophysics – SWx++
A virtual observatory that supports Heliophysics must facilitate
access to data from a number of communities
Solar, heliospheric, magnetospheric and ionospheric physics,
aeronomy…
Throughout the solar system – not just Sun-Earth
Globally – beyond national or European level
Heliophysics sits in the boundary between communities
Astrophysics and Planetary sciences (including Earth sciences)
A VO for heliophysics must be aware of the need to support the
interests of these communities
Heliophysics Integrated Observatory (HELIO)
Proposal submitted under EC’s FP7
2
Driven by science
Heliophysics Integrated Observatory (HELIO)
Desire to solve science problems that
span disciplinary boundaries is driving
the need to provide integrated
access to data across the
communities
The communities have evolved
independently over decades
Each has very different ways of
describing, storing and exploiting the
data from their observations, varying
use of standards
To facilitate access, we need to find
ways to:
Tie the data together through searches
across all the domains
Present any results in a form that does
not require a detailed understanding
of each discipline
U.Alaska
3
HELIO Concept
Heliophysics Integrated Observatory (HELIO)
HELIO concept implemented with a service oriented architecture
Service to curate and access search metadata
Search engine implemented as a service
Services to access data repositories
Services to extract and process required observations
Workflow tool used to bind the services together
User interface closely liked with this tool
Domain interoperability facilitated by semantic-driven approach
Single data model for all domains not practical
Service oriented architecture allows a researcher to use components
of HELIO as they wish – as a chain or as individual services
Chaining of services will provide maximum support for new users and
allows a diverse community to study complex science problems in
heliophysics
Users may only require access to metadata and employ their own search
tools or may just wish to use the dataset location capabilities
Format of data products can be tailored to meet community needs
4
Issues related to Metadata
Heliophysics Integrated Observatory (HELIO)
Metadata are the key to accessing observations
Rapidly increasing volumes make good metadata essential
There are many issues related to existing metadata
Poor quality or missing, lack of interoperability…
This affects ability of all users to do science – not just HELIO
To resolve the issues requires the re-evaluation of capabilities
provided within each community and some corrective action
HELIO plans to work closely with the community on this
Metadata can be grouped in several ways, one is:
Search metadata
o
Metadata used to identify interesting time intervals and locations
Observational metadata
o
Metadata used to describes the observations, e.g. FITS headers
Storage metadata
o
Metadata that describes how the data are stored and accessed
Administrative metadata
o
Metadata that allows the system to exploit the available resources
5
Data Storage
Heliophysics Integrated Observatory (HELIO)
Access to data should be a matter of mechanics and generic
Cheap storage and the Internet have greatly enhanced access
Well established protocols for access – http, ftp, …
How data are stored within a data source can make a lot of difference
to their accessibility
A variety of file formats should be accommodated
EGSO has the concept of resource-rich and resource-poor providers
Resource-rich – should provide what is needed in response to simple query
Resource-poor – may only be able to make data accessible over Internet
Guidelines/standards on ways that data should be organized could
improve capabilities of all providers
Providing data following simple naming conventions in an ordered
directory structure would make them simpler to access
Simple catalogue (textual?) might provide additional information
Discussed at VOiG in 2007; IAU WG discussion
6
Observational metadata
Heliophysics Integrated Observatory (HELIO)
Provides information about how the observation was made
Important for exploiting the data; key for HELIO
Often quality issues related to the metadata that is provided
Parameters sometimes missing, or wrong
Inconsistent use of information, “synonyms” for keywords
In solar data, space-based observations much better described than their
ground-based counterparts
o
Ground-based observations are only source in some wavelengths and need
access to as many observatories as possible
Researchers often used to deficiencies in their own domain
Difficult for machines to handle if it is not quantified properly
Consequence can be that the data are “unusable”
Need to encourage all organizations that generate data to adopt and
comply with agreed standards
Where possible the standards should have generic parts to facilitate
interoperability
7
Heliophysics Integrated Observatory (HELIO)
Searches
In heliophysics, we are interested in how an event on the solar surface
can propagate through the heliosphere and affect planetary
environments
May also want to work backwards and look for the cause of an effect –
what solar event caused this ionospheric activity…
Searches should identify interesting time intervals based on a
combination of event, features, etc. metadata
Light curves and images my also be used to augment the search
Location of observer affects whether phenomena seen
Each community of some combination of these metadata
Differences in how some quantities are expressed, what are included
There are concerns about the quality and integrity of these metadata
and whether they are adequate to support the searches HELIO would
like to undertake
8
Heliophysics Integrated Observatory (HELIO)
Solar search metadata
Searches in solar physics are mainly event driven
Phenomena occur on or near the solar surface
Event data gives time and location of phenomena
Feature data provides details of location and size of structures that may be
relevant
Time information can be expressed in many ways
Essentially these are the same, with simple transformations
Spatial information can be expressed in terms of:
Coordinates in the observing frame – e.g. arcsecs from disc centre
Coordinates on the rotating body of the Sun – Carrington coordinates
The location of the observer largely ignored
o
Helio-seismology is an exception
In the bigger picture of Heliophysics, also need to include the viewing
perspective (c.f. STEREO)
9
Heliophysics Integrated Observatory (HELIO)
Other search data
Observation of phenomena in the heliosphere and near/on planets
are more complex
For in-situ observations in the heliosphere
Time is when a phenomena affected (passed) the observer
Position of the observer relative to the Sun is key to understanding
When the in-situ observations are made on/near a planet
Position of the observer relative to the planet is also important
Relating events that are defined from in-situ data to those on/near the
Sun requires an understanding of how events propagate
Details of the velocity structure of CMEs and the solar wind are not easy to
determine…
HELIO plans to develop a tool that will use even/feature data to refine
a model to trace effects forwards from causes, etc.
10
Heliophysics Integrated Observatory (HELIO)
Simple, but not so simple
In principle this all seems fairly obvious, but lets look in detail at some
common solar event data
On 20 January 2005 there was an X7.1 flare that was intensely geoeffective.
The flare was associated with particle event and a CME; it was also
observed by ground-level neutron monitors – a GLE.
Many superlatives were used to describe the event
o
o
o
o
"The solar energetic particle event of January 20 2005 has been called, by some
measures, the most intense in 15 years..." (Mewaldt et al., 2005)
”The fastest rising SEP event of current cycle [cycle 23]" (Rawat et al., 2006)
”The most spectacular [solar event] of the Space Age" (Tylka et al., 2006)
”The largest GLE [GLE 69] in half a century" (Bartol Research Institute)
But event is absent from the NOAA SEC list of "Solar Proton Events Affecting
the Earth Environment"
When you look at the data and how lists are created, you realize that
the lists are deficient in several ways
Humans and SmFCACs can understand what happened, but
It is harder for machines...
11
Heliophysics Integrated Observatory (HELIO)
X7.1 of 20 Jan 2005
The event was one of several
from AR 10720
Two other X class flares and
several M class flares occurred
in previous 3 days; others
before this
12
Heliophysics Integrated Observatory (HELIO)
X7.1 of 20 Jan 2005
13
Heliophysics Integrated Observatory (HELIO)
X7.1 of 20 Jan 2005
At the time of the event, the
proton levels had not returned to
normal after previous events
The criteria fails to recognize a
new event
o
NOAA lists event on 16 Jan
The X-ray data also suffers from
problems
The end of an event is defined by
when the counts drop to 50%
o
New events can “interrupt”
existing events
The shape and true duration of
the decay phase are lost
o
NOAA gives start 0636; end 0726
Not all locations are tagged!!
Significant brightenings seen on
images not declared as flares
14
Some of the problems
Heliophysics Integrated Observatory (HELIO)
Automated searches are difficult when major events can be “missed”
A search for long duration events would yield spurious results
If the locations of all flares are not known (in a timely fashion), it is
impossible to know whether they will be geo-effective
Instrument flare lists have gaps – nights, off times, etc. – but the reason
for a null result is not included
15
Some of the problems
Heliophysics Integrated Observatory (HELIO)
Automated searches are difficult when major events can be “missed”
A search for long duration events would yield spurious results
If the locations of all flares are not known (in a timely fashion), it is
impossible to know whether they will be geo-effective
Instrument flare lists have gaps – nights, off times, etc. – but the reason
for a null result is not included
16
Improving Metadata
Heliophysics Integrated Observatory (HELIO)
Existing event lists can give a distorted picture what has occurred
Such deficiencies make it difficult for non-experts to use them
The community “knowledge” is not written down
Need to re-evaluate/regenerate the event data in all domains with
the idea that they will be used in a joint search across the domains
Ensure events are described more accurately
Include information that might explain null results
Metadata should comply with agreed standards that have generic
components to ensure interoperability
The situation has changed with enhancements to technology
o
Providers need to ensure they are more compliant
Virtual observatories should try to handle problems with old metadata
o
Limits to what can be achieved if metadata is poorly formed
Standards need to be developed in collaboration with the community
and funding agencies
17
Heliophysics Integrated Observatory (HELIO)
Conclusions
We have the technology, but developing a virtual observatory
to support heliophysics will not be simple
Cooperation of the community is essential if we are to succeed
Some of the possible problems have been highlighted
The quality of observational metadata needs to be improved
We must improve the quality and content of the search metadata
These problems affect the Space Weather community in a similar
way to the Heliophysics community
To address these issues we need to engage the communities in
all the domains that constitute heliophysics and develop
standards that will facilitate the process
HELIO has a large Networking component that is targeted at
addressing these issues and fully involving the community
18
19
Heliophysics Integrated Observatory (HELIO)