Download Data Catalog Web Interface

Fermi (previously
GLAST) Gamma-Ray
Space Telescope
Processing Pipeline
and Data Catalog
Tony Johnson
[email protected]
Launched 11 June 2008 – LAT activated 25 June
T. Johnson
FERMI Data Processing Pipeline
CHEP 2010, Taipei, Taiwan
2/18
Fermi MISSION ELEMENTS
•
•
GPS
msec
Large Area Telescope
& GBM
DELTA
7920H
-
• Telemetry 1 kbps
•
Fermi Spacecraft
TDRSS SN
S & Ku
•
•
S
•
GN
•
Schedules
Mission Operations
Center (MOC)
GRB
Coordinates Network
Fermi Science
Support Center
White Sands
HEASARC
Schedules
Alerts
LAT Instrument
Science
Operations Center
(SLAC)
GBM Instrument
Operations Center
Data, Command Loads
T. Johnson
FERMI Data Processing Pipeline
CHEP 2010, Taipei, Taiwan
3/18
Design Goals
•
Experiment Goals
• Completely automate Fermi data processing at SLAC
• 10 year lifetime for experiment
• Unlike HEP data astrophysics data is time critical
• NASA requirement to make data public in 24 hours
• Internal goal to complete processing of each downlink within 3
hours
• Support submission and bookkeeping of simulation and data reprocessing
• Maintain full history of all data processing
• Data catalog to keep track of all data products
• Provide access to data for collaborators
T. Johnson
FERMI Data Processing Pipeline
CHEP 2010, Taipei, Taiwan
4/18
Pipeline Design Goals
•
•
•
•
•
Automated submission and monitoring of batch jobs
– Very high reliability
Ability to define arbitrary graph of jobs to be run
– Ability to parallelize processing tasks
– Ability to perform simple computations as part of job graph
• E.g. Compute how many parallel streams to create as a function of the
number of events to be processed
Ability to “Roll Back” jobs (whether successful or not)
– Capability to automatically compute sub-graph of jobs to rerun
Web interface for monitoring jobs and submitting new tasks
– Worldwide collaboration
– Plus command line client, and programmatic API
Avoid tight coupling to specific experiment to allow for reuse
T. Johnson
FERMI Data Processing Pipeline
CHEP 2010, Taipei, Taiwan
5/18
Data Catalog design goals
• Support multiple storage formats
– AFS, NFS, xrootd, ….
• Support multiple file locations for same datasets
– SLAC, IN2P2, …
• Allow arbitrary meta-data to be stored with datasets
– As much meta-data as possible should be extracted from the
file itself to ensure integrity
• Dataset access from web, command line, API
– Including search based on meta-data
• Avoid tight coupling to specific experiment to allow for reuse
– Avoid tight coupling with pipeline
T. Johnson
FERMI Data Processing Pipeline
CHEP 2010, Taipei, Taiwan
6/18
Pipeline and Data Catalog Components
SLAC
Command
Client Client
Oracle
Data
Catalog
Catalog Web
Interface Client
SLAC
Client
Line Mode
Client Client
Client
Pipeline
Web Interface
T. Johnson
Pipeline
Server
read-only
Oracle
FERMI Data Processing Pipeline
Client Data
Client Portal
SLAC
Job Control
Batch
Farm
IN2P3
Job Control
Batch
Farm
INFN / Grid?
Job Control
Batch
Farm
CHEP 2010, Taipei, Taiwan
7/18
Anatomy of a Pipeline Task
Tasks, Subtasks, Processes, Streams, specified in user written XML
T. Johnson
FERMI Data Processing Pipeline
CHEP 2010, Taipei, Taiwan
8/18
Level 1 Processing Task Example
Digitization
Reconstruction
T. Johnson
FERMI Data Processing Pipeline
CHEP 2010, Taipei, Taiwan
9/18
Processing Pipeline Web Interface
•
Pipeline web interface allows
– Many views of data processing, down to log files of individual jobs
– Job submission (but normally done from command line)
– If jobs do fail they can be “rolled back” directly from the web interface
T. Johnson
FERMI Data Processing Pipeline
CHEP 2010, Taipei, Taiwan
10/18
Front End: Activity Plots
Simulation
L1 Reconstruction
L1 Digitization
T. Johnson
FERMI Data Processing Pipeline
CHEP 2010, Taipei, Taiwan
11/18
Monitoring Data Processing
• Pipeline provides web API for incorporating info into other web
applications
T. Johnson
FERMI Data Processing Pipeline
CHEP 2010, Taipei, Taiwan
12/18
Data Catalog Web Interface
Summary of all
files in a group
Meta-data
associated with
folders, groups and
files
Logical tree
allows browsing
for files
Folders and
Groups
T. Johnson
Crawler runs in background and
validates all files and extracts size,
#events, etc
FERMI Data Processing Pipeline
CHEP 2010, Taipei, Taiwan
13/18
Data Catalog Web Interface
Drill down to get
more details
Download manager,
reliable download of
multiple files
T. Johnson
FERMI Data Processing Pipeline
CHEP 2010, Taipei, Taiwan
14/18
Pipeline Performance and Reliability
x108
T. Johnson
FERMI Data Processing Pipeline
CHEP 2010, Taipei, Taiwan
15/18
Technologies Used
•
•
•
Database
– Oracle
• Java Stored Procedures for performance
• Scheduled Server-side Jobs for monitoring, stats-gathering
• Hierarchical queries
Servers and Client Libraries (Pipeline, Data Catalog)
– Java
• Extensive use of threads, concurrency utilities for performance
– Jython interpreter for user scripts
– JMX MxBean Interfaces for monitoring, communication
– XML used for processing-task definitions
– Batch jobs use e-mail for status notification
• Apache/James Email server
Web:
– Apache/Tomcat servers
– JSP for web pages
• DisplayTag for tabular data
• AIDA tag libraries for plotting
• Custom tag libraries expose Pipeline client methods
– Java Servlets
• Serve GraphViz State diagrams
– JMX Interfaces
T. Johnson
FERMI Data Processing Pipeline
CHEP 2010, Taipei, Taiwan
16/18
Conclusion
• Pipeline is in extensive production use:
– Real Data
– Automated and Routine Science Processing
– Large Simulations
– Reprocessing
• Software not coupled specifically to Fermi or SLAC
– Second pipeline now in use for EXO and CDMS
• New job control daemon written for CDMS
– Submits jobs to condor farm (at SMU)
– Possibility to extend this to submit Cloud/Grid jobs in future
– Pipeline being evaluated for use by James Webb Space
Telescope (Hubble 2.0)
• To see pipeline in action:
– http://srs.slac.stanford.edu/Pipeline-II/
– http://glast-ground.slac.stanford.edu/Pipeline-II/
T. Johnson
FERMI Data Processing Pipeline
CHEP 2010, Taipei, Taiwan
17/18
Acknowledgments
Software Development Team:
Daniel Flath
Charlotte Hee
Karen Heidenreich
Claudia Lavalley
Tony Johnson
Max Turri
Beta Testers:
Warren Focke
Tom Glanzman
T. Johnson
FERMI Data Processing Pipeline
CHEP 2010, Taipei, Taiwan
18/18
Extra Slides
3-Tier Architecture
Connection
Pool
Back-End
Components
JMX
MBean
Interface
Connection
Pool
Pipeline
Server
Data
SQL
Scheduler
RMI
Job Control Service
Thread
Pool
Batch
Farms
T. Johnson
Web Browser
Tomcat Server
TNS
Stored
Procedures
RMI*
Oracle
SQL
RDBMS
HTTP
Web
Application
Apache/James
Email
POP3
Middle-Ware
FERMI Data Processing Pipeline
JMX
Line-mode
Client
JMX
JConsole Monitor
Front-End
User Interfaces
CHEP 2010, Taipei, Taiwan
20/18
Front End: Line-mode Clients
• Command line tools for direct or scripted interaction with
middle-ware
– Control Server
• Ping
• Restart
• Shutdown
– Upload Task Definitions
– Manage processing streams
• Create
• Delete
• Cancel
• Retry from failure point
– Query processing history
– Plus Interaction with Data Catalog
T. Johnson
FERMI Data Processing Pipeline
CHEP 2010, Taipei, Taiwan
21/18
Front End: Web Interfaces
• Provides all administrative functions available in a user-friendly
interactive GUI
• Interactive displays show active (and historical) processing
– Filtering by Task, Process, Status(es), Stream-range, Daterange
• Processing Statistics Plots
– Provided by AIDA tag library
– System throughput plots
• Filterable by Task, Date-Range
– Individual process statistics plots
• CPU time (vs Wallclock)
• Pending time
• By Batch Host-type
• Task diagrams generated by GraphViz and image-mapped to
provide links to task element (Sub-tasks, processes) displays
T. Johnson
FERMI Data Processing Pipeline
CHEP 2010, Taipei, Taiwan
22/18
Front End: Task Summary Display
T. Johnson
FERMI Data Processing Pipeline
CHEP 2010, Taipei, Taiwan
23/18
Front End: Process Detail Plots
T. Johnson
FERMI Data Processing Pipeline
CHEP 2010, Taipei, Taiwan
24/18
Front End: Job Detail Display
T. Johnson
FERMI Data Processing Pipeline
CHEP 2010, Taipei, Taiwan
25/18
Middle Tier: Threading
•
Makes extensive use of Java Concurrency library (java.util.concurrent)
•
Scheduler Threads
– Look for work and delegate to Execution pool
• Ready Jobs (Script and Batch)
• Submits work to execution threads
– Handle Email status-messages
• Checks for email
• Submits status transition calculations to execution threads
• Receives confirmation from workers, deletes email
– Reaper
• Searches for ‘lost’ jobs in Batch, updates processing history
accordingly
•
Execution Threads
– Decode email status messages and update process records
– Execute Jython script processes directly
– Submit Batch jobs to Farms
T. Johnson
FERMI Data Processing Pipeline
CHEP 2010, Taipei, Taiwan
26/18