Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Scientific Workflows Based on Dataflow Process Networks (or from Ptolemy to Kepler) (or Workflow Considered Harmful …) Bertram Ludäscher San Diego Supercomputer Center [email protected] NeSCR Dec-3 -2003 Bertram Ludaescher Overview 1. 2. 3. 4. 5. Scientific Workflow (SWF) Examples SWF Requirements & Characteristics Workflow standards considered harmful for SWF!? Dataflow Process Networks (Ptolemy II) Scientific Workflows (Kepler = Ptolemy II + X) NeSCR Dec-3 -2003 Bertram Ludaescher • NSF, NIH, DOE Acknowledgements I • GEOsciences Network (NSF) – www.geongrid.org • Biomedical Informatics Research Network (NIH) – www.nbirn.net • Science Environment for Ecological Knowledge (NSF) – seek.ecoinformatics.org • Scientific Data Management Center (DOE) – sdm.lbl.gov/sdmcenter/ NeSCR Dec-3 -2003 Bertram Ludaescher Acknowledgements II • • • • • • • • • • • • • • • Ilkay Altintas SDM Chad Berkley SEEK Shawn Bowers SEEK Jeffrey Grethe BIRN Christopher H. Brooks Ptolemy II Zhengang Cheng SDM Efrat Jaeger GEON Matt Jones SEEK Edward A. Lee Ptolemy II Kai Lin GEON Bertram Ludaescher BIRN, GEON, SDM, SEEK Stephen Neuendorffer Ptolemy II Mladen Vouk SDM Yang Zhao Ptolemy II … • Coming soon!?: – ROADNet, myGrid, GriPhyN, ... NeSCR Dec-3 -2003 Bertram Ludaescher Ptolemy II Promoter Identification Workflow (PIW) NeSCR Dec-3 -2003 Bertram Ludaescher Source: Matt Coleman (LLNL) Execution Semantics NeSCR Dec-3 -2003 Bertram Ludaescher Promoter Identification Workflow in Ptolemy-II (SSDBM’03) GARP Invasive Species Pipeline Test sample (d) Registered Ecogrid Database EcoGrid Query Species presence & absence points (native range) (a) Registered Ecogrid Database +A1 +A2 +A3 Sample Data Training sample (d) Data Calculation GARP rule set (e) Map Generation Native range prediction map (f) Model quality parameter (g) Integrated layers (native range) (c) Environmental layers (native range) (b) Invasion area prediction map (f) Map Generation Layer Integration Registered Ecogrid Database Environmental layers (invasion area) (b) Layer Integration User Model quality parameter (g) Integrated layers (invasion area) (c) EcoGrid Query Registered Ecogrid Database Validation Validation Archive To Ecogrid Selected prediction maps (h) Generate Metadata Species presence &absence points (invasion area) (a) NeSCR Dec-3 -2003 Bertram Ludaescher Source: NSF SEEK (Deana Pennington et. al, UNM) Rock & Mineral Classification Workflow NeSCR Dec-3 -2003 Bertram Ludaescher A Look Inside Classification Finer granularity Extracted from the mineral composition and this level’s diagram coordinates. Diagrams information and transitions between them. Classifier: Locates the point’s region. SVG to polygons. Displays the point in the diagram for this level. NeSCR Dec-3 -2003 Bertram Ludaescher Source: NIH BIRN (Jeffrey Grethe, UCSD) NeSCR Dec-3 -2003 Bertram Ludaescher SWF Requirements & Characteristics • Scientist friendly "problem solving environment" – WF design – WF execution – WF steering and UI • pause; revise; resume; rollback (cf. SCIRun) – repositories of reusable components – data and WF provenance (virtual data concept) • logging, cache reuse/partial re-derive, reports, … – Conceptual modeling support • complex data (semantics) support • “wiring” support (cf. web service composition) • planning support NeSCR Dec-3 -2003 Bertram Ludaescher SWF Requirements & Characteristics • "Modeling" support – – – – Abstraction, hierarchical modeling Models of Computation (MoC) component interaction; combination of MoCs (cf. CCA) WF multi-grain/granola: powder to bolders (and back) • Boolean (N)AND, (N)OR,… vs. chaining together Grid-apps – Rich data structures and type systems • End user "programming" support – high-level programming constructs • e.g. map/3 for iteration, filter, select, branch, merge, ... – data transformations – legacy tool integration (plug-ins) – data streaming • How to tame (e.g., starve a dataflow; then resume)? Zauberlehrling’s problem NeSCR Dec-3 -2003 Bertram Ludaescher SWF Requirements & Characteristics • Grid-enabling SWFs – transparent use of (remote) resources – big data – big computation requirements – early/late binding of logical to physical resources, … – planning, scheduling, … cf. Chimera, Pegasus, DAGman, Condor(-G) NeSCR Dec-3 -2003 Bertram Ludaescher Scientific Workflows: Some Findings • More dataflow than (business) workflow – but some branching looping, merging, … – not: documents/objects undergoing modifications – instead often: dataset-out = analysis(dataset-in) • Need for “programming extension” – Iterations over lists (foreach); filtering; functional composition; generic & higher-order operations (zip, map(f), …) • Need for abstraction and nested workflows • Need for data transformations (compute/transform alternations) • Need for rich user interaction & workflow steering: – pause / revise / resume – select & branch; e.g., web browser capability at specific steps as part of a coordinated SWF • Need for high-throughput transfers (“grid-enabling”, “streaming”) • Need for persistence of intermediate products data provenance (“virtual data” concept) NeSCR Dec-3 -2003 Bertram Ludaescher A ZOO of Workflow Standards and Systems Source: W.M.P. van der Aalst et al. http://tmitwww.tm.tue.nl/research/patterns/ NeSCR Dec-3 -2003 Bertram Ludaescher Business Workflows • Business Workflows – – – – – show their office automation ancestry documents and “work-tasks” are passed no data streaming, no data-intensive pipelines lots of standards to choose from: WfMC, WSFL, BMPL, BPEL4WS,.. XPDL,… but often no clear execution semantics for constructs as simple as this: Source: Expressiveness and Suitability of Languages for Control Flow Modelling in Workflows, PhD thesis, Bartosz Kiepuszewski, 2002 NeSCR Dec-3 -2003 Bertram Ludaescher On Workflow Standards… http://tmitwww.tm.tue.nl/staff/wvdaalst/Publications/publications.html NeSCR Dec-3 -2003 Bertram Ludaescher Workflow “Standards” Debunked Source: Don’t go with the flow:Web services composition standards exposed,W.M.P. van der Aalst, Trends Controversies, NeSCR Dec-3& -2003 Bertram Ludaescher Jan/Feb 2003 issue of IEEE Intelligent Systems Web Services - Been there done that? Workflow “Standards” Debunked Source: Don’t go with the flow:Web services composition standards exposed,W.M.P. van der Aalst, Trends Controversies, NeSCR Dec-3& -2003 Bertram Ludaescher Jan/Feb 2003 issue of IEEE Intelligent Systems Web Services - Been there done that? But never mind the standards discussion: Many Scientific Workflows are Dataflows! (Check YOUR examples …) NeSCR Dec-3 -2003 Bertram Ludaescher Commercial Workflow/Dataflow Systems NeSCR Dec-3 -2003 Bertram Ludaescher SCIRun: Component-Based Problem Solving Environments for Large-Scale Scientific Computing • • • SCIRun: problem solving environment for interactive construction, debugging, and steering of large-scale scientific computations Component model, based on generalized dataflow programming Source: Steve Parker (cs.utah.edu); SciDAC/SDM collaboration NeSCR Dec-3 -2003 Bertram Ludaescher Workflow and distributed computation grid created with Kensington Discovery Edition from InforSense. NeSCR Dec-3 -2003 Bertram Ludaescher Dataflow Process Networks: Putting Computation Models first! typed i/o ports FIFO actor actor • Synchronous Dataflow Network (SDF) advanced push/pull – Statically schedulable single-threaded dataflow • Can execute multi-threaded, but the firing-sequence is known in advance – Maximally well-behaved, but also limited expressiveness • Process Network (PN) – Multi-threaded dynamically scheduled dataflow – More expressive than SDF (dynamic token rate prevents static scheduling) – Natural streaming model • Other Execution Models (“Domains”) – Implemented through different “Directors” NeSCR Dec-3 -2003 Bertram Ludaescher Dataflow Process Networks and Ptolemy-II see! read! try! NeSCR Dec-3 -2003 Bertram Ludaescher Source: Edward Lee et al. http://ptolemy.eecs.berkeley.edu/ptolemyII/ Why Ptolemy-II? • PTII Objective: – “The focus is on assembly of concurrent components. The key underlying principle in the project is the use of well-defined models of computation that govern the interaction between components. A major problem area being addressed is the use of heterogeneous mixtures of models of computation.” • Data & Process oriented: – Dataflow process networks • Natural Data Streaming Support • End user “WF console” (Vergil GUI) • PRAGMATICS – mature, actively maintained, well-documented – open source system – leverage “sister projects” activities (e.g. SEEK, SDM, BIRN,…) NeSCR Dec-3 -2003 Bertram Ludaescher NeSCR Dec-3 -2003 Bertram Ludaescher Source: Edward Lee et al. http://ptolemy.eecs.berkeley.edu/ptolemyII/ NeSCR Dec-3 -2003 Bertram Ludaescher Source: Edward Lee et al. http://ptolemy.eecs.berkeley.edu/ptolemyII/ Marrying & Divorcing Control- & Dataflow Source: Edward Lee et al. http://ptolemy.eecs.berkeley.edu/ptolemyII/ NeSCR Dec-3 -2003 Bertram Ludaescher Another Goodie: Ptolemy-II Type System NeSCR Dec-3 -2003 Bertram Ludaescher Support for Multiple Workflow Granularities Bolders Plumbing Powder Sand NeSCR Dec-3 -2003 Bertram Ludaescher Abstraction: Sand to Rocks Scientific Workflows = Dataflow Process Networks + X Kepler = Ptolemy-II + X • X=… – Database plug-ins – Legacy application plug-ins (via command line, as web services, …) – Grid extensions: • • • • – – – – Actors as web/grid services 3rd party data transfer, high-throughput data streaming Dealing with thousands of files (cf. astrophysics, astronomy, HEP, … examples) Data and service repositories, discovery Extended type system (structural & semantic extensions) Programming extensions (declarative/FP) and Rich user interactions/workflow steering Rich data transformations (compute/transform alternations) Data provenance • (semi-)automatic meta-data creation NeSCR Dec-3 -2003 Bertram Ludaescher Status update / specific tasks for Kepler $DONE, %ONGOING, *NEW • User interaction, workflow steering – $ Pause/revise/resume – $ BrowserUI actor (browser as a 0-learning display and selection tool) • Distributed execution – $ Dynamically port-specializing WSDL actor – * Dynamically specializing Grid service actor • Port & actor type extensions (SEEK leverage) – * Structural types (XML Schema) – * Semantic types (OWL) incl. unit types w/ automatic conversion • Programming extensions – % Data transformation actors (XSLT, XQuery, Python, Perl,…) – * map, zip, zipWith, …, loop, switch “patterns” • Specialized Data Sources – $ EML (SEEK), – % MS Access (GEON), *JDBC, – *XML, *NetCDF, … NeSCR Dec-3 -2003 Bertram Ludaescher Some specific tasks for Kepler (all NEW) • Design & develop transparent, Grid-enabled PNs: – – – – Communication protocol details Grid-actor extensions and/or Grid-Process Network director (G-PN) Host/Source-location becomes actor parameter • add “active-inline” parameter display for grid-actors (@exec-loc), channels (@transport-protocol), source-actors (@{src-loc|catalog-loc}) • Activity Monitoring – Add “activity status” display (green, yellow, red) to replace PtII animation (needed for concurrently executing PN!) • Registration & Deployment mechanisms – Actor/Data/Workflow repository (=composite actors) – Shows up as (config’able) actor library – OGSA Service Registry approach? (SEEK leverage; UDDI complex & limited says MattJ) • http://www-unix.globus.org/toolkit/draft-ggf-ogsi-gridservice-33_2003-06-27.pdf • Extensions to deal with failures (fault tolerance) NeSCR Dec-3 -2003 Bertram Ludaescher Example: Database actors for Ptolemy II (Kepler-GEON; Efrat Jaeger) NeSCR Dec-3 -2003 Bertram Ludaescher Database Actors • Database Connection actor: • Database Query actor: Database Actors Example Database Actors Example Example: Web service-enabling Ptolemy II (Kepler-SDM; Ilkay Altintas) NeSCR Dec-3 -2003 Bertram Ludaescher A Generic Web Service Actor Configure Configure – select - selectWSDL service url from operation repository NeSCR Dec-3 -2003 Bertram Ludaescher Set Parameters and Commit Specialized Actor Set parameters and commit NeSCR Dec-3 -2003 Bertram Ludaescher Web Service Actor after Instantiation NeSCR Dec-3 -2003 Bertram Ludaescher Composing Third-Party Web Services Output of previous web service User interaction & Transformations NeSCR Dec-3 -2003 Bertram Ludaescher Input of next web service Results of the Execution User I/O via standard brower! Run Window / WF Deployment NeSCR Dec-3 -2003 Bertram Ludaescher Composing Legacy Applications (here: Phylogeny): Shell / Command-Line Actors Source: Alex Borchers, UCSD NeSCR Dec-3 -2003 Bertram Ludaescher Example: Grid-enabling Ptolemy II ( Kepler-SEEK, Chad Berkley Kepler-SDM, Ilkay Altintas, … myGrid?, … …GriPhyN?, … … OGS{I|A}-[DAI] ...) NeSCR Dec-3 -2003 Bertram Ludaescher Transparently Grid-Enabling PTII: Handles Logical token transfer (3) requires get_handle(1,2); then exec_handle(4,5,6,7) for completion. PTII space A 3 4 1 2 Grid space B 7 1. 2. 3. 4. 5. 6. 7. AGA: get_handle GAA: return &X AB: send &X BGB: request &X GBGA: request &X GA GB: send *X GBB: send done(&X) 5 GA NeSCR Dec-3 -2003 Bertram Ludaescher 6 GB Example: &X = “GA.17” *X =<some_huge_file> Transparently Grid-Enabling PTII • Different phases – – – – Register designed WF (could include external validation service) Find suitable grid service hosts for actors Pre-stage execution Execute (w/ provenance) • Interactively steer (pause; revise; resume) • Batch process; re-run parts later – Register/store data products and execution logs • Kepler implementation choices: – Grid-actors (no change of Director necessary!?) and/or – Grid-(PN)-director (also need to change actors!?) – Add grid service host id as actor parameter: A@GA – Similar for data: myDB@GA NeSCR Dec-3 -2003 Bertram Ludaescher “C-z ; bg &” – Detach your WF execution! • Currently in PTII – tight coupling of WF execution and PTII Java client (also Vergil GUI) • To-do for Kepler: – detaching WF console (Vergil) from a Grid-aware execution engine Grid-PN Director! Transport protocol parameter NeSCR Dec-3 -2003 Bertram Ludaescher Data location parameter Host location parameter Semantic Type-enabling Ptolemy II (OWL – here we go… ;-) (Kepler-SEEK; Shawn Bowers) NeSCR Dec-3 -2003 Bertram Ludaescher Semantic Type Extensions • Take concepts and relationships from an ontology to “semantically type” the data-in/out ports • Application: e.g., design support: – smart/semi-automatic wiring, generation of “massaging actors” m1 p3 (normalize) Takes Abundance Count Measurements for Life Stages NeSCR Dec-3 -2003 Bertram Ludaescher p4 Returns Mortality Rate Derived Measurements for Life Stages NeSCR Dec-3 -2003 Bertram Ludaescher NeSCR Dec-3 -2003 Bertram Ludaescher Semantic Types • The semantic type signature – Type expressions over the (OWL) ontology m1 p3 (normalize) p4 SemType m1 :: Observation & itemMeasured.AbundanceCount & hasContext.appliesTo.LifeStageProperty -> DerivedObservation & itemMeasured.MortalityRate & hasContext.appliesTo.LifeStageProperty NeSCR Dec-3 -2003 Bertram Ludaescher Extended Type System (here: OWL Semantic Types) SemType m1 :: Observation & itemMeasured.AbundanceCount & hasContext.appliesTo.LifeStageProperty DerivedObservation & itemMeasured.MortalityRate & hasContext.appliesTo.LifeStageProperty Substructure association: XML raw-data =(X)Query=> object model =link => OWL ontology NeSCR Dec-3 -2003 Bertram Ludaescher Programming Extensions (some lessons from SciDAC/SSDBM demo) NeSCR Dec-3 -2003 Bertram Ludaescher Promoter Identification Workflow in control Ptolemy-II hand-crafted (SSDBM’03) solution; also: forces designed to fit designed to fit sequential execution! hand-crafted Web-service actor No data transformations available NeSCR Dec-3 -2003 Bertram Ludaescher Complex backward control-flow Promoter Identification Workflow in FP genBankG :: GeneId -> GeneSeq genBankP :: PromoterId -> PromoterSeq blast :: GeneSeq -> [PromoterId] promoterRegion :: PromoterSeq -> PromoterRegion transfac :: PromoterRegion -> [TFBS] gpr2str :: (PromoterId, PromoterRegion) -> String d0 d1 d2 d3 d4 d5 d6 d7 d8 d9 = = = = = = = = = = Gid "7" -- start with some gene-id genBankG d0 -- get its gene sequence from GenBank blast d1 -- BLAST to get a list of potential promoters map genBankP d2 -- get list of promoter sequences map promoterRegion d3 -- compute list of promoter regions and ... map transfac d4 -- ... get transcription factor binding sites zip d2 d4 -- create list of pairs promoter-id/region map gpr2str d6 -- pretty print into a list of strings concat d7 -- concat into a single "file" putStr d8 -- output that file NeSCR Dec-3 -2003 Bertram Ludaescher Cleaned up Process Network PIW • Back to purely functional dataflow process network map(f)-style iterators (= also a data streaming model!) Powerful type checking Generic, declarative “programming” constructs Generic data transformation actors • Re-introducing map(f) to Ptolemy-II (was there in PT Classic) no control-flow spaghetti data-intensive apps free concurrent execution free type checking automatic support to go from piw(GeneId) to PIW :=map(piw) over [GeneId] Forward-only, abstractable subworkflow piw(GeneId) NeSCR Dec-3 -2003 Bertram Ludaescher Optimization by Declarative Rewriting I • PIW as a declarative, referentially transparent functional process map(f o optimization via functional rewriting possible g) instead of map(f) o map(g) e.g. map(f o g) = map(f) o map(g) • Details: Combination of map and zip – Technical report &PIW specification in Haskell http://kbi.sdsc.edu/SciDAC-SDM/scidac-tn-map-constructs.pdf NeSCR Dec-3 -2003 Bertram Ludaescher Optimizing II: Streams & Pipelines Source: Real-Time Signal Processing: Dataflow, Visual, and Functional Programming, Hideki John Reekie, University of Technology, Sydney • Clean functional semantics facilitates algebraic workflow (program) transformations (Bird-Meertens); e.g. mapS f • mapS g mapS (f • g) NeSCR Dec-3 -2003 Bertram Ludaescher Summary • Many (most of ours anyways) scientific workflows are dataflows – lots of workflow “standards” (messy and not focused on SWF problems) – should we start a new wave of dataflow standards?? • Importance of clear semantics for – – – – different MoCs (models of computation: PN, SDF, DE, CT, …) component composition across MoCs component interaction Ptolemy II directors • Kepler: – Based on extensible Ptolemy II system – Cross-project activity (SEEK, SDM, Ptolemy II, GEON, BIRN, and counting) – Plug-in / interface with your SWF planner, execution engine, grid-WF tool! NeSCR Dec-3 -2003 Bertram Ludaescher Your Projects & Icons <HERE> NeSCR Dec-3 -2003 Bertram Ludaescher A Note on the Style of these Slides Due to lack of time, most of the following slides are “by reference” only ;-) – …Each speaker was given four minutes to present his paper, as there were so many scheduled -- 198 from 64 different countries. To help expedite the proceedings, all reports had to be distributed and studied beforehand, while the lecturer would speak only in numerals, calling attention in this fashion to the salient paragraphs of his work. ... Stan Hazelton of the U.S. delegation immediately threw the hall into a flurry by emphatically repeating: 4, 6, 11, and therefore 22; 5, 9, hence 22; 3, 7, 2, 11, from which it followed that 22 and only 22!! Someone jumped up, saying yes but 5, and what about 6, 18, or 4 for that matter; Hazelton countered this objection with the crushing retort that, either way, 22. I turned to the number key in his paper and discovered that 22 meant the end of the world… [The Futurological Congress, Stanislaw Lem, translated from the Polish by Michael Kandel, Futura 1977] NeSCR Dec-3 -2003 Bertram Ludaescher