Download PPTX

Reducing IFU Data
Nic Scott
ITSO DR Workshop, Thursday 5th May
Overview
›Part 1 – Overview of IFU data reduction
- Useful resources, different kinds of IFU, “theory,” and
overview of main steps
›Part 2 – Hands-on with WiFeS data
- Understanding calibrations, reducing data and checking
output
›Part 3 – Tips and tricks
- Extra steps, common issues and solutions, and comments
for specific IFUs
2
Part 1 - Overview
Useful websites
› The IFS wiki: http://ifs.wikidot.com/welcome
› Instrument specific sites:
- OSIRIS: http://www2.keck.hawaii.edu/inst/osiris/
- WiFeS: http://rsaa.anu.edu.au/observatories/instruments/wide-field-spectrographwifes
- KOALA: https://www.aao.gov.au/science/instruments/current/koala/overview
- NIFS: https://www.gemini.edu/sciops/instruments/nifs/
- GMOS IFU: https://www.gemini.edu/sciops/instruments/gmos/integral-fieldspectroscopy
› IFS Surveys
- Atlas3D: http://www-astro.physics.ox.ac.uk/atlas3d/
- CALIFA: http://califa.caha.es/
- SAMI: http://sami-survey.org/
- MANGA: http://www.sdss.org/surveys/manga/
4
Useful resources
› Fits image/cube viewers:
- QFitsView: http://www.mpe.mpg.de/~ott/QFitsView/
- ds9: http://ds9.si.edu/site/Home.html
- Fv: http://heasarc.gsfc.nasa.gov/ftools/fv/
› Data reduction tools:
- Iraf 😟
- Python/IDL
- p3d: http://p3d.sourceforge.net/
- Instrument-specific packages and pipelines
› Many tools developed for radio astronomy can be applied to optical IFS
data – cube viewing software, disk fitting, line analysis etc.
5
Different kinds of IFU
SAMI
› Fibre-fed
- Much like a fibre-fed multi-object
spectrograph
› Image slicer
- Uses an array of mirrors to ‘slice’ the
MUSE
image into multiple long-slit spectra
› Lenselet array
- Uses a lenslet array in the focal plane
to divide up field, then carefully
arranges spectra on CCD
SAURON
6
Fibre-feb IFUs
› Fibre-fed IFUs use multiple
closely-spaced fibres to act as
spaxels and capture individual
spectra
› Pros: cheap, extremely
flexible, easily reconfigurable,
easy to arrange spectra on
the CCD, can take advantage
of fibre-MOS software and
techniques
› Cons: low efficiency, fibres
can’t be too close together,
circular spaxels
› Can be coupled with a
lenselet array to solve some
of these problems, but with
further efficiency loss
› Examples include: PPAK,
KOALA, SAMI, GMOS
Croom et al. (2012)
7
Lenselet array IFUs
› Use an array of closely-packed lenses, a
lenselet array, in the image plane to act as
spaxels
› Pros: reasonable efficiency, square pixels,
variable spatial scales, high fill-factor
› Cons: reasonable efficiency, arranging
spectra on CCD can be challenging,
crosstalk between spectra
› Examples: SAURON, OSIRIS
Bacon et al. (2000)
Larkin et al. (2010)
8
Image slicer IFUs
› Use an array of mirrors to ‘slice’ the
image plane into a series of long-slit
spectra. Each slice acts as a single
y-column, with the pixels along the
slit providing the x-direction
› Pros: High efficiency (only
reflections), can apply long-slit
techniques and software, high fillfactor, fill CCD easily
› Cons: Expensive, complicated
› Examples: WiFeS, MUSE, SINFONI
Bacon et al. (2010/12)
9
Steps in IFS data reduction
› 1: General calibrations
- Standard steps to account for CCD and instrument effects: bias subtraction, flat
fielding etc. Essentially the same as for non-IFS data
› 2: Extraction of spectra
- Identify and extract individual spectra from the CCD. This step is highly specific
to different kinds of IFU
› 3: Reassembly of spectra into a datacube
- Identify the relative on-sky positions and wavelength calibration of the extracted
spectra. Put all spectra on a common wavelength and spatial scale then
assemble into a cube
10
General calibrations
› Goal is to correct for instrumental effects i.e. go from electrons to photons
emitted by the source
› Effects to correct for:
› Calibration frames required:
- CCD bias level
- Bias
- Dark current
- Dark
- CCD sensitivity variations
- Flat lamp and/or twilight flat
- CCD illumination
- Spectrophotometric standard
star
- Cosmic rays
- Hot pixels and bad columns
- Total telescope and instrument
throughput
› Most of these steps can be done early on in the reduction process, and even benefit from being done
before spectra extracted
› Throughput correction typically done as a later stage
11
Extraction of spectra
› The ‘trick’ of IFS is arranging three
dimensions of data onto a two
dimensional detector
› This can be done in very different
ways depending on the IFS design:
- Lenselet systems use a rectification
matrix – a predetermined relationship
between detector pixel and lenselets
- Fibre-based systems typically use a
tramline map combined with a known
fibre profile
- Slicer systems fit the slitlet profile,
extracting each slitlet as an independent
long-slit spectrum
Bacon et al (2000)
Childress et al (2014)
› The end goal is a set of extracted
spectra, either as Row Stacked
Spectra (RSS), or multiple long-slit
spectra
Sharp et al (2015)
12
Assemble data cube
› The many one-dimensional (or two-dimensional for slicer IFUs) spectra
need to be arranged into a single three-dimensional datacube (x,y,λ)
› All spectra need to be placed on a common wavelength scale – typically
done by fitting an arc spectrum then interpolating to a constant Å/pixel
scale
› Each spectrum needs to be assigned an (x,y) coordinate position in the
datacube. This process depends on the IFU type:
- For lenselet arrays, the position of each lenselet on the sky is known in advance
and a simple matrix is used to arrange the extracted spectra into a cube
- For fibre-based IFUs, spectra are arranged into a cube based on the known
positions of each fibre
- For image slicers, a trace or wire calibration is used to match the spatial position
of pixels in each slitlet
13
Optional extra steps
› Sky subtraction
- Many options for this e.g. nod-and-shuffle, offset sky, dedicated sky fibres or
lenselets
- Can be done before or after assembling data cube
› Flux calibration
- Best done after cube creation
- IFS data has very good flux calibration accuracy compared to fibre or long-slit
data. Can fit model of star in 2d, therefore determine very accurate total flux
› Combining exposures
- Can be done at or after cubing
- Can be done to increase field-of-view (mosaicing), increase effective spatial
resolution (dithering) or simply increase S/N
14
Part 2 – Hands on with WiFeS data
Preparation
› Checks that pywifes.py is in your python path:
- Terminal -> ipython --pylab -> import pywifes
› If not: export PYTHONPATH =
$PYTHONPATH:/path/to/code/folder
› Missing python dependencies? Install them with pip
› Make sure you have a fits viewer application installed
16
Overview of the pywifes pipeline
› Pywifes is a Python-based pipeline for reducing WiFeS data written mainly
by Mike Childress – see Childress et al. (2014).
› Its really good! Easy to use, largely automated, produces high-quality data
products
› You can download the latest version here:
http://www.mso.anu.edu.au/pywifes/doku.php
› NB We are not using the latest version today. The WiFeS data provided
was taken in 2012, instrument and pipeline upgrades since then mean an
older pipeline version is required to reduce this data. Your own
installation of pywifes will likely not work for this exercise.
› Pywifes can simply reduce your data in one go. However, we’re going to
take it one step at a time so we can check the output at each stage.
17
Check your raw data!
› Before starting reduction visually inspect the raw data (this should
probably have been done when observing)
› Things to look out for:
- Saturated exposures
- Files that aren’t what they claim to be
- Exposures with no or very low flux
- Excessive amounts of cosmic rays
- Weird artifacts
› Calibration files with issues must be removed or fixed before starting
reduction – including bad calibrations makes your final data worse
18
Setting up the reduction
› NB I’ve already done this step for
you
› Need to tell the DR pipeline which
files you want to use
› Pipeline scripts will automatically
identify file types – but check that it
has done this right
› Manually edit file to stop the
pipeline using them in reduction –
alternatively, simply move them out
of the reduction folder
19
Running the reduction script
› Open reduce_blue_data.py in
the reduction scripts folder
› Set the first step, ‘run’:True. All
others should be False.
› Execute the reduction script:
› Reduced files will be created in
the Data/Reduced/ folder
› *.p??.fits indicates which step in
the reduction process a file is
› NB To disable a reduction step,
comment it out using #s in the
reduce_blue_data.py file
20
Bias subtraction
› Overscan_sub and bias_sub subtract
the zero-point of the CCD from all
reduction files
› Overscan_sub does this on a region
of the CCD that is not observed. Bias
does this based on a separate bias
combined observation
› Check the combined bias frame for
artifacts!
21
Flat creation
› These steps combine the lamp and
twilight observations into single highS/N combined frames
› Then extract the flat frame for each
slitlet
› Inspect the super_domeflat and
super_twiflat frames to check for
artifacts
› Check one or two of the extensions in
the super_domeflat_mef and
super_twiflat_mef files to check
extraction worked (use Open as…
Multi-Extension Cube… in ds9)
22
Spectral extraction
› These steps determine the slitlet
profile using the flat frames
› Once the profiles are determined,
each slitlet is extracted and
repackaged as a Mulit-Extension Fits
(MEF) file
› This step is also applied to the super
flat frames
› NB The slitlet_defs file claims to be
.fits but is really a .pkl. Use a text
editor to view it and modify if
necessary (this is fixed in current
versions of pywifes)
23
Wavelength calibration
› Using the arc frame, fits to the
positions of peaks
› Matches these to a line list for the
given lamp
› Fits a smooth polynomial to the list of
peak positions vs line wavelengths
and stores this as the wavelength
solution for each slitlet
› Check terminal output has sensible
values, and wave_soln.fits image
varies smoothly
24
Wire solution
› Wire frame is a flat with a single-pixel
width mask placed in the focal plane
› Purpose is to match corresponding ypixels in different slitlets
› Position of the wire (a trough) is fit in
each slitlet then recorded in wire_soln
› Output is slightly counterintuitive – it’s
a nλ * nslitlets “image” where the value
of each pixel is the fitted position of
the wire at that position
› Check to see wire positions are
consistent from slitlet to slitlet and
don’t vary much with wavelength
25
Flat fielding
› Attempt to mask cosmic rays using
the LACOSMIC routine
› Pretty good, but not perfect at finding
cosmics
› If any frame has an excessive
number of cosmics (very long
exposure red/NIR frames for
example) consider disabling them
› Apply the previously calculated flat
response to each slitlet
26
Cube creation – part 1
› Apply the wavelength and wire
solutions to your science
observations
› Interpolates each slitlet onto a
common wavelength and spatial
scale
› Aligns each slitlet using the wire
solution
› Produces another MEF file
27
Cube creation – part2
› For convenience, convert the MEF “cube” into a 3D datacube
› NB In later versions of pywifes this function is built in
› With this version:
- ipython --pylab
- import wifes_utilities
- wifes_utilities.wifes_mef_to_cube(infile,outlife)
› Inspect your final data cube using QFitsview or other
datacube viewer
28
Flux calibration
› To flux calibrate our data, we require
observations of spectrophotometric
standard stars
› These functions extract star spectra,
compare those spectra to the known
‘standard’ spectrum and compute a
polynomial transformation
› This polynomial is then applied to the
science frames
29
Bits and bobs and summary
Variance
› Because IFS data reduction is complex, tracking variance is also complex
› The key is to treat your variance image exactly as you treat your flux
- Dividing by a flat field? Divide your variance by the square of the same frame
- Flux calibrating? Rescale your variance in the same way
› Not quite that simple – many steps add additional variance
- Subtracting the bias? Bias frames have noise so add variance
- Extracting spectra? Cross-talk and inexact extraction apertures add variance
› Good news is most IFS reduction packages do this for you
› Understanding your variance is critical for taking best advantage of your
data!
31
Correlation and Covariance
› Not all flux pixels are independent. Not all variance pixels are independent
either
› Spatial correlation:
- Simplest example is seeing, happens in all IFUs
- Crosstalk between spectra located close to one another on the CCD – common in
fibre and lenselet IFUs
- Resampling onto a common spatial scale
› Spectral correlation:
- Resampling onto a common wavelength scale
› Covariance – or correlated noise:
- Similar to correlated flux, but more complicated to deal with
- Dealing with it properly requires a full covariance matrix – some DR packages will
provide this
- Often simpler to partially account for covariance by rescaling your variance by an
appropriate factor – or when model fitting penalise for fewer independent datapoints
32
Atmospheric Refraction
› Light of different wavelengths is
refracted by different amounts as it
passes through the atmosphere
› This means the position of an object
in IFS varies systematically as a
function of wavelength
› Sometimes telescopes have an ADC
(atmospheric dispersion corrector)
built in
› If not, you can manually shift each
wavelength to have a common
centroid, either by fitting to the object
centroid or using a DAR model
› This effect is worse at blue
wavelengths, can be safely ignored in
the NIR
33
Combining data
› Several reasons to combine data
cubes: improve S/N, increase field-ofview, attempt to increase spatial
resolution
› Can centre based on object centroid or
just telescope pointing for faint objects
› Easy mode: use native sampling and
combine to the nearest pixel – this is
often good enough
› Hard mode: combine to sub-pixel
accuracy by resampling data onto a
finer grid – have to do this when trying
to increase effective resolution via
drizzling
› Caution – resampling correlates your
data AND your noise. May be more
trouble than its worth
Sharp et al. (2015)
34
Closing remarks
› IFS is an extremely powerful technique with a broad range of astrophysical
applications. It has revolutionised optical astronomy in the last ~15 years
› Different IFUs have different wavelength ranges, fields-of-view, spectral
and spatial resolutions, efficiencies, access to AO etc. Pick the best tool to
suit your science goals
› However, the instruments and therefore the data are complex, and an
understanding of how the data are produced is critical to get the most out
of it
› Different IFU designs come with different pros and cons, as well as
different issues affect their data products
› Many IFUs have an excellent pipeline – I highly recommend using these!
› If your IFU doesn’t have a pipeline, find a collaborator with experience
reducing data from that instrument
35
Questions?