Download ppt - IRAF

NOAO Mosaic Pipeline
Technical Presentation
December 16, 2002
NOAO Mosaic Pipeline CoDR
1
Outline of Technical Presentation
•
•
•
•
•
Introduction
Contexts
Capabilities
Architecture
Implementation
December 16, 2002
NOAO Mosaic Pipeline CoDR
2
Presentation Goals
Convince you that:
• we understand the
–
–
–
–
problem
requirements
resources
components
• and that the project
– is feasible
– has a solution for the primary application
– has a flexible design for expansion and wider application
December 16, 2002
NOAO Mosaic Pipeline CoDR
3
Guiding Principles
•
•
•
•
Modest project
Part of Data Products Program
(NOAO) Mosaic Imaging Data
Dedicated pipeline
December 16, 2002
NOAO Mosaic Pipeline CoDR
4
Principles: Modest Project
• Reuse as much software as possible
• Keep it simple software
December 16, 2002
NOAO Mosaic Pipeline CoDR
5
Principles: DPP
•
•
•
•
MDHS: Mosaic Data Handling System
IRAF: Image Reduction and Analysis Facility
NSA: NOAO Science Archive
DTS: Data Transport System
• OPUS: AURA sister institution (STScI)
• GONG: AURA sister institution (NSO)
December 16, 2002
NOAO Mosaic Pipeline CoDR
6
Principles: (NOAO) Mosaic Data
• Use experience of Mosaic Survey Teams
• Need to deal with specific peculiarities
– Crosstalk, pupil reflections
• Allow for high performance per exposure
(for real-time telescope context) by
capitalizing on the inherent data parallel
nature of mosaic imaging data
December 16, 2002
NOAO Mosaic Pipeline CoDR
7
Principles: Dedicated Pipeline
• Network of similar computers
• No competition with general users
December 16, 2002
NOAO Mosaic Pipeline CoDR
8
What does this project encompass?
Algorithms, interfaces, and software for:
•
•
•
•
•
•
•
Pipeline infrastructure
CCD mosaic data reduction
Data quality assessment
Image differencing
Catalog production
Database entry and querying
Source merging/classification
December 16, 2002
•
•
•
•
•
•
•
Archive ingest and retrieval
Alerts
Monitoring
Data transport
High performance computing
Parallel computing
More …
NOAO Mosaic Pipeline CoDR
9
Contexts
In what contexts will the pipeline run?
Can we design a pipeline to satisfy multiple contexts?
December 16, 2002
NOAO Mosaic Pipeline CoDR
10
Contexts
• NOAO
– Telescope/operational context
– Archive/NVO context
• Community
– NOAO Mosaic surveys and observers
– Other mosaic instruments
December 16, 2002
NOAO Mosaic Pipeline CoDR
11
Priorities
1. NOAO Archive
2. NOAO Mosaic observers
− telescope
− downtown
− home institution
3. NOAO Mosaic observers at home
4. Community
December 16, 2002
NOAO Mosaic Pipeline CoDR
12
NOAO Contexts
• Downtown center fed from telescope
• Mountain at telescope
• Archive on-the-fly reprocessing
December 16, 2002
NOAO Mosaic Pipeline CoDR
13
Pipeline Locations
Pipeline
Pipeline
Tucson
Archive
La Serena
Archive
Kitt Peak
December 16, 2002
Pipeline
Cerro Tololo
NOAO Mosaic Pipeline CoDR
Pipeline
14
Context: Downtown Pipeline
Data Spool
and Transport
Observer
DCA
Pipeline
User @
telescope,
downtown,
home
December 16, 2002
DSC
Archive
NOAO Mosaic Pipeline CoDR
15
DTS
Context: Mountain Pipeline
Data Spool
and Transport
DCA
Pipeline
Archive
User @
telescope
December 16, 2002
NOAO Mosaic Pipeline CoDR
16
DTS
Context: Archive Pipeline
Pipeline
User @
home
December 16, 2002
Archive
NOAO Mosaic Pipeline CoDR
17
DTS
Context: User Pipeline
User @
home
December 16, 2002
Pipeline
@ home
NOAO Mosaic Pipeline CoDR
18
More
Proposed Context
• Downtown pipeline for NOAO archive
• Observer may subscribe to data products
– At telescope, downtown, home
– Images, catalogs, alerts, …
• Observer may connect to DQ monitors
• Pipeline software available at telescope with
minimal support
• DQ task/monitors may run at telescope
December 16, 2002
NOAO Mosaic Pipeline CoDR
19
Data Requirements
The pipeline design is dependent on the
information available about the input data.
• Basically we require data with the current NOAO
Mosaic readout format that includes:
– identification of exposure type (object, etc)
– description of regions (data, overscan)
– an approximate world coordinate system
December 16, 2002
NOAO Mosaic Pipeline CoDR
21
Data Requirements
There may be additional information that the
pipeline will use if present.
• Associations: type, ID, total and index
SEQUENCE = ‘zero2002-12-16T043244.20.3’
SEQUENCE = ‘dither2002-12-16T043244.5.2’
If not present heuristics will be used based on a
requirement that data enters in time order
December 16, 2002
NOAO Mosaic Pipeline CoDR
22
Capabilities
• Capabilities
• Major Features and Goals
• Data Products
– Basic
– Advanced
• Data Quality Assessment
• Instrumental Calibration
December 16, 2002
NOAO Mosaic Pipeline CoDR
27
Capabilities
•
•
•
•
•
•
•
•
•
•
Calibrate mosaic exposures
Update instrumental calibrations
Identify potential bad data (data quality assessment)
Monitor trends and maintain database
Stack dither sets
Catalog and classify objects and artifacts
Get and subtract reference image and detect sources
Identify interesting sources
Automatically provide data products to subscribers
Keep up with observing given sufficient CPU resources
December 16, 2002
NOAO Mosaic Pipeline CoDR
28
Major Features and Goals
•
•
•
•
•
•
•
•
•
•
•
Data products for NOAO archive and NVO node
Data products for observers (by subscription)
Pipeline for NOAO and mosaic community
Basic CCD mosaic calibrations
Advanced time-domain data products
Real-time data quality assessment and monitoring
High performance, data parallel system
LSST testbed
Fairly generic pipeline infrastructure (NEWFIRM, …)
Automated operation
Thorough processing history and data documentation
December 16, 2002
NOAO Mosaic Pipeline CoDR
29
Data Products: Basic
•
•
•
•
•
•
•
•
•
•
Instrument calibrated mosaic exposures
Rough photometric zero point
Astrometric calibrations
Data quality evaluations
Updated calibrations
Bad pixel, saturated, bleed trail masks
Object catalogs
Object masks
Observing logs
Processing information
– logs
– graphs
December 16, 2002
NOAO Mosaic Pipeline CoDR
30
Data Products: Advanced
•
•
•
•
Dither stacks
Exposure masks
Field Catalogs
Difference image detections
– Relative to dither stack
– Relative to archive or catalog reference
•
•
•
•
Light curves
Variable object detections
Unusual object alerts
Moving object trajectories
December 16, 2002
NOAO Mosaic Pipeline CoDR
31
Data Quality Assessment
•
•
•
•
•
•
Instrument
Telemetry
Crosstalk
Overscan
Bias, flat
Noise
Focus / Distortions
Sky
•
•
•
•
•
Seeing (PSF)
Sky brightness
Approx. zero point
Twilight
Moon up / distance
Data quality measures are monitored against preset and user limits as well as adaptive time series limits.
Some quantities include mean, sigma, and spatial variations.
December 16, 2002
NOAO Mosaic Pipeline CoDR
32
Instrumental Calibrations
•
•
•
•
•
•
Crosstalk [1]
CCD defects [2,4,5]
Saturated pixels [2,4,5]
Bleed trails [2,4,5]
Cosmic rays [2,4,5]
WCS update [3]
•
•
•
•
•
•
Overscan [2]
Bias [2]
Flat field [2]
Pupil pattern [3]
Fringing [3]
Approx. zero point [3]
1.
Requires image data from full mosaic (non-parallel)
2.
Each image element independent of others (parallel)
3.
Global calculation on measurements images (parallel and non-parallel)
4.
Interpolate in data
5.
Flag in mask
December 16, 2002
NOAO Mosaic Pipeline CoDR
33
Instrumental Calibrations
Two-pass calibration for telescope context:
1. Nighttime pass for immediate and nearly
complete calibrated exposures
2. Daytime pass for calibration update from the
full night’s data set
December 16, 2002
NOAO Mosaic Pipeline CoDR
34
Nighttime Pass
• Perform standard CCD calibrations:
– Use afternoon master bias
– Use most recent flat field
• Apply pupil and fringe correction
– Use most recent pupil and fringe templates
• Apply global coordinate calibration
December 16, 2002
NOAO Mosaic Pipeline CoDR
35
Daytime Pass
• Determine if night’s data is suitable for deriving
updates to library calibrations
• Derive new pupil, fringe, and sky flat calibrations
• Evaluate changes and significance of new
calibrations
• Update library calibrations for next night
• Update night’s exposures with new calibrations
• Combine afternoon biases into new master bias
• Combine afternoon dome flats if no library flat
December 16, 2002
NOAO Mosaic Pipeline CoDR
36
Other Contexts
• For archive data will either already have
best calibration from library or will be
derived by requesting raw data for night
• At home or in the community raw data will
be queued as at telescope
• Documentation and support (data ingest
applications) will be provided
December 16, 2002
NOAO Mosaic Pipeline CoDR
37
Data Products Subscription
• Capability of the DPP system
– Not necessarily specific to the pipeline but
requires interfacing with DTS
• Allows external software to request
notification of new data products
• Allows flexibility and broader access
– Has implications for the pipeline context
December 16, 2002
NOAO Mosaic Pipeline CoDR
38
Architecture
• What is a pipeline?
• Mosaic Pipeline Architecture Concept
• Pipeline Components
– Controls and Monitors
– Modules
– Calibrations and Database (Rafael Hiriart)
– Archive (Robyn Allsman)
December 16, 2002
NOAO Mosaic Pipeline CoDR
39
What is a Pipeline?
System to transform input data to output data
•
•
•
•
•
Automated
Composed of processing steps (modules)
Steps connected by rules (triggers)
Provides monitoring and alerts
Error tolerant (continue with next input data)
December 16, 2002
NOAO Mosaic Pipeline CoDR
40
Mosaic Pipeline Architecture Concept
• Multiple CPUs but no dependency on N
• Multiple types of sub-pipelines by function
– One for operations over all mosaic elements
– One for operations on individual elements
– One for cataloging
– One for image differencing
• All types on all CPUs: no master!
• Sub-pipelines triggered by files
December 16, 2002
NOAO Mosaic Pipeline CoDR
41
Mosaic Pipeline Architecture Concept
• All CPUs with identical pipeline software,
possibly on common NFS disk
• Assign work by minimum data backlog
• Transfer data to local CPU disk: not NFS!
– Optimize by modules writing to next trigger
directory
• Controls connected to operator console
• Monitors viewed via network by multiple
parties
December 16, 2002
NOAO Mosaic Pipeline CoDR
42
Network of Sub-pipelines and CPUs
Pipeline
CPU
CPU
CPU
MEF
MEF
MEF
SIF
SIF
SIF
CPU
CPU
CPU
SIF
SIF
SIF
MEF
MEF
MEF
MEF: pipeline for operations over all mosaic extensions; eg crosstalk, global WCS correction
SIF: pipeline for single CCD images; eg ccdproc, masking
December 16, 2002
NOAO Mosaic Pipeline CoDR
43
Data Flow Concept
Last module in one pipeline writes output
directly to the data directories of the host
for next pipeline, with the host selected by
having the minimum number of waiting
data files.
December 16, 2002
NOAO Mosaic Pipeline CoDR
44
Data Flow Algorithm
• Search list of potential hosts:
– Check if host is up
– Check number of trigger files
– Assign output filename to data directory of
host with least number of data files
– Network filenames are used: (eg.
host!directory/filename
• Module runs and writes output files
December 16, 2002
NOAO Mosaic Pipeline CoDR
45
Data Flow Networking
• Use a daemon automatically spawned the
first time data is transferred to a host
• Daemon provides portability across
platforms; eg. Unix and VMS
December 16, 2002
NOAO Mosaic Pipeline CoDR
46
Data Flow Networking: Example
• Crosstalk input is Obj123.fits with 2 extensions
• Output names are generated from Host.dat:
– Host1 has two waiting files, Host2 has one, Host3 is
down, Host4 has none
– Host2!Obj123.1, Host4!Obj123.2
• Crosstalk module runs and writes output files
directly to the hosts
• There are no extra network copy or splitting steps
December 16, 2002
NOAO Mosaic Pipeline CoDR
47
Data Flow Networking: Example
Host1:
Obj456.1
Obj321.2
Host2!Obj123.2
Obj123
Host0:
Crosstalk
Host2:
Obj567.2
Obj123.2
Host4:
DOWN
Host3!Obj123.1
December 16, 2002
Host3:
Obj123.1
NOAO Mosaic Pipeline CoDR
48
Pipeline Components
Controls & Monitors
Data Source
(DTS, user)
Pipeline
raw
data
Module
data
Module
products
Data Sink
(DTS, user)
Calibrations &
Databases
December 16, 2002
NOAO Mosaic Pipeline CoDR
49
Pipeline Modules
Pipeline
December 16, 2002
Module
Module
Module
CLSH
CSH
API
NOAO Mosaic Pipeline CoDR
50
Data Parallel Modules
Some algorithms may need to be (re-)implemented
specifically for a data parallel pipeline.
One type is where measurements are made
across the mosaic for a global calibration.
Rather than requiring all pieces to be in one
pipeline arrange for measurements made in
parallel to be collected for the global calibration
and then apply the global calibration to the
pieces in parallel.
December 16, 2002
NOAO Mosaic Pipeline CoDR
51
Data Parallel Modules
WCS Example
• Catalog objects in each CCD in parallel
• Bring catalogs (not images) together
– Only need x/y coordinates of brighter stars
• Match sources to ref. catalog (eg. USNO)
• Compute global correction (shift, scale, etc.)
• Return correction coefficients to parallel
pipelines to be applied to each CCD
• Cataloging and correction stages can be
separated and run asynchronously with other
stages
December 16, 2002
NOAO Mosaic Pipeline CoDR
52
Data Parallel Modules
Fringe/Pupil Example
• Determine best global scaling of pupil and
fringe templates to each exposure and
then subtract scaled template
– Compute statistics over each CCD in parallel
– Combine statistics to get global scale factor
– Subtract template with global scale from each
CCD in parallel
December 16, 2002
NOAO Mosaic Pipeline CoDR
53
Pipeline Triggers
•
•
•
•
•
•
Files:
Flags:
Timers:
File contents:
Messages:
Resources:
trigger on appearance of files
trigger on particular set of flags
trigger at times or intervals
trigger on keywords, etc
trigger on messages
trigger on resources
May be more but one type can mimic others
December 16, 2002
NOAO Mosaic Pipeline CoDR
54
Pipeline Triggers
• File triggers useful for initiating a pipeline
• Flag triggers useful within a pipeline to
communicate success of previous steps
• Flag triggers also useful for waiting for
completion of parallel steps
• Timer triggers useful in telescope pipeline
for performing different daytime/nighttime
steps
December 16, 2002
NOAO Mosaic Pipeline CoDR
55
File Triggers
Pipeline
Data
Directory
Trigger
Directory
Module
obj123.fits
obj123.trig
GO
May contain
information
such as
output path
December 16, 2002
NOAO Mosaic Pipeline CoDR
56
More
Flag Triggers and Merging
Pipeline
December 16, 2002
Data
Directory
Trigger
Directory
Module
obj123b.fits
obj123d.fits
obj123a.fits
obj123c.fits
obj123b.trig
obj123d.trig
obj123a.trig
obj123c.trig
GO
NOAO Mosaic Pipeline CoDR
-d
-d
-d
-d
Module
GO
57
Timer Triggers and Two-Passes
• Nighttime pipeline runs and leaves data in
starting directory for daytime pipeline
• Daytime pipeline is triggered at end of
night by timer
December 16, 2002
NOAO Mosaic Pipeline CoDR
58
Controls & Monitors
Process Manager
Status Monitor
Obs. Manager
Keyword Monitor
Pipeline
Module
To
Database
December 16, 2002
NOAO Mosaic Pipeline CoDR
59
Data Manager
• Interacts with the pipeline, operator, and potentially other
parts of the system such as archives or external
applications
• Record
– New calibrations from pipeline or operator
– New parameters from operator
– Processing information from pipeline
• Responds to queries for
– Calibrations
– Parameters
– Processing history
– Documentation and reports for data products
December 16, 2002
NOAO Mosaic Pipeline CoDR
61
Data Manager Architecture
December 16, 2002
NOAO Mosaic Pipeline CoDR
62
What do we want to store in the
database?
December 16, 2002
NOAO Mosaic Pipeline CoDR
63
Where is Data Manager?
December 16, 2002
NOAO Mosaic Pipeline CoDR
64
Calibrations
• The Data Manager responds to requests from pipeline
for current calibration for a particular date, filter, etc.
• Updates calibrations produced by pipeline (or externally)
for a particular date, filter, etc. Calibration updates may
require operator confirmation.
• Calibrations include
–
–
–
–
Biases and flat fields
Pupil and fringe templates
Standard star data
Astrometry coordinates
• Some queries are satisfied through secondary queries to
other databases such as USNO, GSC2, Landolt, etc.
December 16, 2002
NOAO Mosaic Pipeline CoDR
65
Parameters
• Responds to requests from pipeline for current
parameters for
– Pipeline module
– Observation date, filter, exposure type, etc.
– Position on sky
• Updates parameters supplied by operator
December 16, 2002
NOAO Mosaic Pipeline CoDR
66
Processing Information
• All information produced by the pipeline is recorded
(keyed by a data identifier). This includes all the
information provided to the keyword monitor as well as
other data processing sources (logs, graphics, etc.)
• Pipeline requests processing information for a pipeline
execution packaged as an associated data product for
the archive.
• The operator can query processing information for
diagnostic purposes.
December 16, 2002
NOAO Mosaic Pipeline CoDR
67
Reports
• Produces reports for a particular data product
• Documentation is created from processing information
according to some template and desired format (eg xml,
html)
December 16, 2002
NOAO Mosaic Pipeline CoDR
68
Pipeline/Archive Ingest Interface
• Desirable traits
– Independence of database semantics
– Use of self-describing data description
standards
– Hiding data’s physical location
December 16, 2002
NOAO Mosaic Pipeline CoDR
69
Archive Ingest
Who, What, Where
Authority
Payload
Data
Receiving
Agent
Archive
Ingest
Manager
Data Store
December 16, 2002
NOAO Mosaic Pipeline CoDR
70
Strawman Implementation
December 16, 2002
NOAO Mosaic Pipeline CoDR
71
Things We Looked At / Aware Of
•
•
•
•
•
•
•
•
•
•
•
Macho pipeline
SM/SN pipeline
Sloan pipeline
Pan-Starrs: IMCAT, Vista
IRAF: Core, IMRED pipelines, STSDAS, PYRAF, etc
MIDAS: Mosaic Imager Data Archive System
Linda and descendants
Elixar (CFH), Terapix (CFH), Subaru, ESO WFI, INT WFI
Condor / PVM / NOAO message bus
Opus pipelines: HST. MSSO, GONG
Databases: MySQL, Postgres
December 16, 2002
NOAO Mosaic Pipeline CoDR
72
Software and Systems
(Blue Ribbon)
• OPUS
• IRAF System
– CLSH (enhanced), KI, OBM/GUI
• IRAF Tasks
– MSCRED, ACE
•
•
•
•
SM/SN Alard/Lupton Algorithm
POSTGRES
DTS
NSA
December 16, 2002
NOAO Mosaic Pipeline CoDR
73
Software and Systems
(Honorable Mention)
• PVM
• Condor
• Other scripting languages and systems
– PYRAF and Python
– Perl
– MLCL
December 16, 2002
NOAO Mosaic Pipeline CoDR
74
Pipeline Modules
Pipeline
Module
Module
Module
CLSH
CSH
OAPI
MSCRED, etc
December 16, 2002
NOAO Mosaic Pipeline CoDR
75
Controls & Monitors
Process Manager
Status Monitor
Opus
IRAF GUI
Obs. Manager
Keyword Monitor
Opus
IRAF GUI
Pipeline
Module
Opus
December 16, 2002
NOAO Mosaic Pipeline CoDR
76
Switchboard Server
Keyword
Monitor
Status
Monitor
Database
Manager
Other types
Or instances
Switchboard address set by
environment variable
CPU
CPU
Pipeline
Pipeline
Module
Module
Pipeline
Module
Module
Switchboard
Server
Module
Module
Pipeline
Module
Module
Backup
December 16, 2002
NOAO Mosaic Pipeline CoDR
77
Triggers
OPUS provides:
• Files: trigger on appearance of files
– Data entry pipeline initiation
• Flags: trigger on “blackboard” flags
– Internal sequencing of modules
– Parallel to Global sequencing
• Timers: trigger at certain times or intervals
– Nighttime/Daytime Two-Pass Control
December 16, 2002
NOAO Mosaic Pipeline CoDR
78
Monitoring IRAF Tasks
• IRAF tasks, including scripts, will open a
messaging connection and write status and
monitor information
• Minimal changes will be required to tasks
• If a server is not running or disappears the tasks
will continue to run with output spooled locally
December 16, 2002
NOAO Mosaic Pipeline CoDR
79
Monitoring IRAF Tasks
• Initially the broadcasting will be a socket
connection with a server that multiple clients
may connect to for rebroadcast
• The monitor tasks are IRAF GUI tasks which
provide flexibility for changes to the GUI or
functionality
December 16, 2002
NOAO Mosaic Pipeline CoDR
80
Monitoring IRAF Tasks
• The GUI monitors will include:
– Adaptive alarms
– Adaptive heartbeat monitoring
– Advanced graphics
December 16, 2002
NOAO Mosaic Pipeline CoDR
81
IRAF Keyword Monitor Prototype
December 16, 2002
NOAO Mosaic Pipeline CoDR
82
IRAF Keyword Monitor Prototype
December 16, 2002
NOAO Mosaic Pipeline CoDR
83
IRAF Status Monitor Prototype
December 16, 2002
NOAO Mosaic Pipeline CoDR
84
NOAO Mosaic Pipeline
Development Plan
1. Basic Calibration Pipeline
2. Advanced Time-Domain Pipeline
December 16, 2002
NOAO Mosaic Pipeline CoDR
85
1. Basic Calibration Pipeline
•
•
•
•
•
Basic single exposure calibrations
Data quality assessment and monitoring
High-performance pipeline infrastructure
Simple data transport system
Connection to the NOAO Science Archive
December 16, 2002
NOAO Mosaic Pipeline CoDR
86
2. Advanced Time-Domain Pipeline
•
•
•
•
•
•
Catalogs
Image difference detections
Multiple detection ident. and merging
Time series
Alerts
Archiving of new data products
December 16, 2002
NOAO Mosaic Pipeline CoDR
87
Timeline Targets
• Test version of basic calibration pipeline
– July 2003
• Operational
– September 2003
• Test version of time-domain pipeline
– July 2004
• Operational
– September 2004
December 16, 2002
NOAO Mosaic Pipeline CoDR
88
Work Breakdown
•
•
•
•
•
•
Pipeline
Monitors
Data Manager
Input and Output
Data Products
Archive
December 16, 2002
NOAO Mosaic Pipeline CoDR
89
Work Breakdown
• Pipeline
– Define methods for running IRAF tasks in OPUS
• Parameters
• Error handling
• I/O
– Define and verify data flow balancing method
– Define, develop, and implement DQ methods
– Develop data parallel algorithm steps for
• WCS
• Fringe/pupil removal
– Develop data parallel OPUS architecture
– Setup development system of at least two machines
December 16, 2002
NOAO Mosaic Pipeline CoDR
90
Work Breakdown
• Monitors
– Develop status monitor
• Experiment with different GUI formats
– Develop keyword monitor
• Experiment with different GUI formats
– Develop switchboard server
December 16, 2002
NOAO Mosaic Pipeline CoDR
91
Work Breakdown
• Data Manager
– Define interfaces
•
•
•
•
–
–
–
–
–
Pipeline
DBMS
NVO/web services
External clients
Define database structures
Define archive data products
Design processing reports
Design calibration library storage and methods
Design and implement manager application
• Include GUI monitor and operator interface
– Install and configure DBMS
December 16, 2002
NOAO Mosaic Pipeline CoDR
92
Work Breakdown
• Input and Output Services
– Contribute to DTS
– Implement interim data transport, staging, and
queuing
December 16, 2002
NOAO Mosaic Pipeline CoDR
93
Work Breakdown
• Archive
– Contribute to NSA development of automatic
ingest
– Adjust data product specification to include
NSA requirements
December 16, 2002
NOAO Mosaic Pipeline CoDR
94
Work Breakdown
• Specify Data Products
– File types
– Headers
– Documentation
December 16, 2002
NOAO Mosaic Pipeline CoDR
95
Implementation Plan
This project will proceed using a Phased Delivery
lifecycle development model – i.e., phases of
detailed design-develop-test as core functionality is
built up. This is especially appropriate when:
• It is important to deliver core functionality quickly
• Some technologies are new (to the development team)
• Delivery timeframe is short
December 16, 2002
NOAO Mosaic Pipeline CoDR
96
Management Plan
Key elements of the management plan are:
•
•
•
•
Management/staffing
Work Breakdown [covered by FV]
Schedule
Risk Management
December 16, 2002
NOAO Mosaic Pipeline CoDR
97
Personnel
Staff
Member
Role
Alloca
tion
Dick Shaw
Project Manager
5%
Schedule development, resource planning
Frank Valdes
Team Lead
40%
Allocation of work, tracking technical progress, lead
designer, documentation
Chris Smith
Project Scientist
10%
Definition of requirements, use cases, verification &
validation, documentation
Rafael Hiriart
S/W Engineer
25%
Database & infrastructure design & development, use
case development
Robyn Allsman
S/W Sys. Eng.
5%
Archive interface definition, archive system updates,
data storage planning, consultant
F. Pierfederici
Scientific Progr.
30%
Implementation, testing
TBD (U. MD)
S/W Engineer
50%
Implementation, testing
December 16, 2002
Responsibilities
NOAO Mosaic Pipeline CoDR
98
Staffing Profile
180%
160%
140%
U.MD
Pierfederici
Allsman
Hiriart
Smith
Valdes
Shaw
120%
100%
80%
60%
40%
20%
0%
2002 2003 2003 2003 2003 2004 2004
Q4
Q1
Q2
Q3
Q4
Q1
Q2
December 16, 2002
NOAO Mosaic Pipeline CoDR
99
December 16, 2002
NOAO Mosaic Pipeline CoDR
100
Risk Management
• Heavily matrixed staff
– New staff will also off-load other work from team lead
• Staff distributed across continents & institutions
– Project leadership remains in Tucson
– Extended visits by new remote staff
– Weekly videoconferences
• New staff has limited experience in problem domain
– Project leader to work closely with new staff
• Use of new/third-party software
– Make effective use of expertise from external partners
December 16, 2002
NOAO Mosaic Pipeline CoDR
101