Download sdss-fnal - Sloan Digital Sky Survey

The Sloan Digital
Sky Survey
Alex Szalay
Department of Physics and Astronomy
The Johns Hopkins University
and the SDSS Project
The Sloan Digital Sky Survey
A project run by the Astrophysical Research Consortium (ARC)
The University of Chicago
Princeton University
The Johns Hopkins University
The University of Washington
Fermi National Accelerator Laboratory
US Naval Observatory
The Japanese Participation Group
The Institute for Advanced Study
Max Planck Inst, Heidelberg
SLOAN Foundation, NSF, DOE, NASA
Goal: To create a detailed multicolor map of the Northern Sky
over 5 years, with a budget of approximately $80M
Data Size: 40 TB raw, 1 TB processed
Alex Szalay, JHU
Scientific Motivation
Create the ultimate map of the Universe:
 The Cosmic Genome Project!
Study the distribution of galaxies:
 What is the origin of fluctuations?
 What is the topology of the distribution?
Measure the global properties of the Universe:
 How much dark matter is there?
Local census of the galaxy population:
 How did galaxies form?
Find the most distant objects in the Universe:
 What are the highest quasar redshifts?
Alex Szalay, JHU
The Cosmic Genome Project
The SDSS will create the ultimate map
of the Universe, with much more detail
than any other measurement before
daCosta
etal 1995
deLapparent, Geller and Huchra
1986
Gregory and Thompson 1978
Alex Szalay, JHU
SDSS Collaboration 2002
Area and Size of Redshift Surveys
1.00E+09
SDSS
photo-z
1.00E+08
No of objects
1.00E+07
SDSS
main
SDSS
abs line
1.00E+06
SDSS
red
1.00E+05
CfA+
SSRS
2dF
LCRS
1.00E+04
SAPM
1.00E+03
1.00E+04
2dFR
1.00E+05
1.00E+06
QDOT
1.00E+07
1.00E+08
Volume in M pc 3
Alex Szalay, JHU
1.00E+09
1.00E+10
1.00E+11
Clustering of Galaxies
We will measure the spectrum of the
density fluctuations to high precision
even on very large scales
The error in the amplitude of
the fluctuation spectrum
1970
1990
1995
1998
1999
2002
x100
x2
±0.4
±0.2
±0.1
±0.02
Alex Szalay, JHU
Finding the Most Distant Objects
Intermediate and high redshift QSOs
Multicolor selection function.
Luminosity functions and spatial clustering.
High redshift QSO’s (z>5).
Alex Szalay, JHU
Features of the SDSS
Special 2.5m telescope, located at Apache Point, NM
3 degree field of view.
Zero distortion focal plane.
Two surveys in one:
Photometric survey in 5 bands.
Spectroscopic redshift survey.
Huge CCD Mosaic
30 CCDs 2K x 2K (imaging)
22 CCDs 2K x 400 (astrometry)
Two high resolution spectrographs
2 x 320 fibers, with 3 arcsec diameter.
R=2000 resolution with 4096 pixels.
Spectral coverage from 3900Å to 9200Å.
Automated data reduction
Over 100 man-years of development effort.
(Fermilab + collaboration scientists)
Very high data volume
Expect over 40 TB of raw data.
About 1 TB processed catalogs.
Data made available to the public.
Alex Szalay, JHU
Apache Point Observatory
Located in New Mexico,
near White Sands National Monument
Alex Szalay, JHU
The Telescope
Special 2.5m telescope
3 degree field of view
Zero distortion focal plane
Wind screen moved separately
Alex Szalay, JHU
The Photometric Survey
Northern Galactic Cap
5 broad-band filters ( u', g', r',
i', z’ )
limiting magnitudes (22.3, 23.3, 23.1, 22.3, 20.8)
drift scan of 10,000 square degrees
55 sec exposure time
40 TB raw imaging data -> pipeline ->
100,000,000 galaxies
50,000,000 stars
calibration to 2% at r'=19.8
only done in the best seeing (20 nights/yr)
pixel size is 0.4 arcsec,
astrometric precision is 60 milliarcsec
Southern Galactic Cap
multiple scans (> 30 times) of the same stripe
Continuous data rate of 8 Mbytes/sec
Alex Szalay, JHU
Survey Strategy
Overlapping 2.5 degree wide stripes
Avoiding the Galactic Plane (dust)
Multiple exposures on the three
Southern stripes
Alex Szalay, JHU
The Spectroscopic Survey
Measure redshifts of objects  distance
SDSS Redshift Survey:
1 million galaxies
100,000 quasars
100,000 stars
Two high throughput spectrographs
spectral range 3900-9200 Å.
640 spectra simultaneously.
R=2000 resolution.
Automated reduction of spectra
Very high sampling density and completeness
Objects in other catalogs also targeted
Alex Szalay, JHU
The Mosaic Camera
Alex Szalay, JHU
First Light Images
Telescope:
First light May 9th 1998
Equatorial scans
Alex Szalay, JHU
The First Stripes
Camera:
5 color imaging of >100 square degrees
Multiple scans across the same fields
Photometric limits as expected
Alex Szalay, JHU
NGC 2068
Alex Szalay, JHU
UGC 3214
Alex Szalay, JHU
NGC 6070
Alex Szalay, JHU
The First Quasars
The four highest redshift
quasars have been found in the
first SDSS test data !
Alex Szalay, JHU
Methane/T Dwarf
Discovery of several new
objects by SDSS & 2MASS
SDSS T-dwarf
(June 1999)
Alex Szalay, JHU
Detection of Gravitational Lensing
28,000 foreground galaxies and 2,045,000 background galaxies in test data
(McKay etal 1999)
Alex Szalay, JHU
SDSS Data Flow
Alex Szalay, JHU
Data Processing Pipelines
Alex Szalay, JHU
Concept of the SDSS Archive
Operational
Archive
Science Archive
(products accessible to users)
(raw + processed data)
Other
Archives
Other
OtherArchives
Archives
Alex Szalay, JHU
Distributed Collaboration
Fermilab
U.Washington
ESNET
U.Chicago
I. Advanced
Study
VBNS
Japan
Apache Point
Observatory
Princeton U.
JHU
NMSU
USNO
Alex Szalay, JHU
SDSS Data Products
Object catalog
parameters of >108 objects
Redshift Catalog
parameters of 106 objects
400 GB
1 GB
Atlas Images
5 color cutouts of >108 objects
1.5 TB
Spectra
in a one-dimensional form
60 GB
Derived Catalogs
- clusters
- QSO absorption lines
20 GB
4x4 Pixel All-Sky Map
heavily compressed
60 GB
All raw data saved in a tape vault at Fermilab
Alex Szalay, JHU
Geometric Indexing
“Divide and Conquer”
Partitioning
Attributes
Number
Sky Position
Multiband Fluxes
Other
3
N = 5+
M= 100+
3NM
Hierarchical
Triangular
Mesh
Split as k-d tree
Stored as r-tree
of bounding boxes
Alex Szalay, JHU
Using regular
indexing
techniques
Distributed Implementation
User Interface
Analysis Engine
Master
SX Engine
Objectivity Federation
Objectivity
Slave
Slave
Slave
Objectivity
Slave
Objectivity
RAID
Objectivity
RAID
Objectivity
RAID
RAID
Alex Szalay, JHU
Collaboration with Particle Physics
Collaboration with the Analysis Data Grid:
proposal to the NSF KDI program by
JHU, Fermilab and Caltech (H. Newman, J. Bunn) +
Objectivity, Intel and Microsoft (Jim Gray)
Involves computer scientists, astronomers and particle physicists
Accessing Large Distributed Archives in Astronomy and Particle Physics
experiment with scalable server architectures,
create middleware of intelligent query agents,
apply to both particle physics and astrophysics data sets
Status:
3 year proposal just funded
Alex Szalay, JHU
The Age of Mega-Surveys
The next generation of astronomical archives with
Terabyte catalogs will dramatically change astronomy
top-down design
large sky coverage
built on sound statistical plans
uniform, homogeneous, well calibrated
well controlled and documented systematics
The technology to acquire, store and index the data is here
we are riding Moore’s Law
Data mining in such vast archives will be a challenge,
but possibilities are quite unimaginable
Integrating these archives into a single entity is a
project for the whole community
=> Virtual National Observatory
Alex Szalay, JHU
New Astronomy – Different!
Systematic Data Exploration
will have a central role in the New Astronomy
Digital Archives of the Sky
will be the main access to data
Data “Avalanche”
the flood of Terabytes of data is already happening,
whether we like it or not!
Transition to the new
may be organized or chaotic
Alex Szalay, JHU
NVO: The Challenges
Size of the archived data
• 40,000 square degrees is 2 trillion pixels
• One band:
4 Terabytes
• Multi-wavelength: 10-100 Terabytes
• Time dimension:
few Petabytes
The development of
• new archival methods
• new analysis tools
• new standards
(metadata, interchange formats)
Hardware/networking requirements
Training the next generation!
Alex Szalay, JHU
Summary
The SDSS project combines astronomy, physics, and computer science
It promises to fundamentally change our view of the universe
It will determine how the largest structures in the universe were formed
It will serve as the standard astronomy reference for several decades
Its ‘virtual universe’ can be explored by both scientists and the public
Through its archive it will create a new paradigm in astronomy
Alex Szalay, JHU
Alex Szalay, JHU