Download InforSense Architecture - All Hands Meeting 2011

Making Workflows Work


Prof. Yike Guo


Dept. of Computing
Imperial College London


InforSense Limited
DiscoveryNet Project

Funding


One of the Eight UK National e-Science Projects (£2.2 M)
Sept 2001 – March 2005

Partners

Achievements



Constructing the World’s First Infrastructure for Building Analytical Services by Scientists
For the First time Discovery Net Realises the Dynamic Construction of Compositional Services
on GRID for Real Time Knowledge Discovery and Decision Making
Outputs







Software Research: DNet platform commercialized by InforSense Ltd (>100 customers)
Total user numbers > 2000
Applications Research: Application out
puts in sensor technology commercialized by deltaDot Ltd
Number papers published: 10 Journal Papers, 30 Conference Papers
8 PhD completed and 50 Master students
Ranked OUTSTANDING at the project final review
Proprietary and Confidential
InforSense Introduction
100+ customers (70% Fortune 200 companies)

2006 3rd fastest growing company in UK (Sunday Times Tech Track)

2007 8th fastest growing venture-based company in UK (Financial Times)

Global footprint with offices in London (HQ and R/D), Boston (USA HQ) and Shanghai (Asia
HQ and Development base)

Global sales with 70% outside of Europe

7 years of delivering products and services to pharmaceutical and Financial industries

Spin out from Imperial College London
Innovation in Embedding Analytics
InforSense Formed
Introduced
‘00 KDE Analytics
Platform
Invented “
‘98 Distributed
Data Mining ”
Proprietary and Confidential
Discovery Net
‘01 Project
Embedding
Analytics Technology
‘053rd fast growing
company in UK
IEEE Supercomputing Award –
,03 Grid based
analytics
First
2004
Enterprise
Deployment
‘06
Embedding
Analytics in
Major Enterprise
Systems
Those who are using our workflow
CAMBRIA
BIOSCIENCES
Proprietary and Confidential
InforSense Workflow Methodology
Portal / Dashboard
Application
Delivery to
End User
Business Process
Administrator
Clinician
Disease Biologist
Rapid Application Deployment
Integrative
Analytics
Workflow
Environment
Automation & Scheduling
Interactive Solution Building
Interactive Knowledge Discovery
Dynamic Data & App Integration
Data
Applications
InforSense
Analytics
Oracle
Pre-processing
3rd party
Analytics
Web services
Biomedical
Informatics tools
Multiple
data sources
Components
Files
Proprietary and Confidential
EMR
Databases
Excel
What is InforSense WF System
Designed for ?

InforSense workflow system is not an application but a framework
to build and deliver applications directly to scientist/business user:
ADMET Browser
Chem-Studio
Proprietary and Confidential
InforSense Generic Workflow Engine
Pipelining
Simulation
& Modelling
Business Process
Managenment
Web Service
Orchestration
ETL
Data/Text Mining
Enterprise
Service Bus
Proprietary and Confidential
Experience of 7 years in WF business

Building workflow is easy !

However,
 Building a USABLE workflow is not easy
 Building a REUSABLE workflow is hard
 Building a REUSABLE workflow applications is very
hard
 Building a REUSABLE workflow application for
EVERYONE is very very hard

Building a function is easy, building an application is
hard, it is even harder if we enable a non-IT person to
build a good reliable application for other people to use
everyday!
Proprietary and Confidential
InforSense Workflow System Development
Native MPI
Condor-G
OGSA-service
Web Service
Resource
Mapping
Workflow Embedding
Web Wrapper
Sun Grid
Engine
Pervasive WF applications
Oralce 10g
Unicore
Workflow Execution
Workflow
Warehousing
Reliable Enterprise Wide
Execution
Workflow Authoring
Composing services
Service
Abstraction
Workflow Management
Collaborative Knowledge Management
Proprietary and Confidential
Workflow Deployment:
Building Reusable WF Applications
Three Tiers of Workflow Framework
Service Orchestration
Embedding Layer
Business Rules
BPEL
Application Layer
Embed in Other Applications
Analytic Service Encapsulation
Publish Services for Display
Analytical Workflow Development
Building Layer
Proprietary and Confidential
Rapid Application Development
InforSense Workflow Building:
Not about another graph notation but
about how to build a meaningful
graph
Current model of workflow authoring/execution




No help provided to user (authoring/execution)
Model is based on expert user who know about services
Model requires user to be trained in a workflow language/system
Interoperability between workflow systems is only at run-time
Proprietary and Confidential
The key the success : End User Oriented Workflow
Construction

Build semi-automatic tools that advise/assist
user in wf authoring

Make use of previous knowledge about
developing workflows

Explicit/Expert knowledge

Implicit knowledge in previous workflows
The aim is to help user, not replace him
Proprietary and Confidential
Guided Workflow Construction

User is presented by high-level descriptions of
predefined task steps

User is guided iteratively in instantiating the task
descriptions using workflow templates

User can retrieve workflows and workflow
templates from repository

Approach supports using workflows from multiple
systems using existing run-time interoperability
mechanisms
Proprietary and Confidential
Workflow Advisor:
InforSense Customer Hubs
Proprietary and Confidential
Extended infrastructure:
Workflow warehousing and mining

Workflow Advisor

Initial implementations of prototype for bio
applications

Workflow Assistant

Abstract component initial prototypes

Workflow Mining

Repository of workflows from Southampton

Workflow Annotations

independent from workflow language

Warehouse

Search and execute web services/Grid services and
workflows

Syntactic and semantic search
Proprietary and Confidential
Extended infrastructure:
Workflow warehouse/registry
Proprietary and Confidential
InforSense Embedding and
Deployment
Workflow output is not a data, but an
application/service
InforSense KDE Deployment Strategies
Deploy workflows to InforSense portal

Deployment features: multi-page, service chain, layout editor

Multi-stage applications: group workflows into stages

Component based deployment

Portlet based deployment

Portlet component: JSR 168 compatible portlet components
Business process workflow

Based on control flow orchestrated workflows and role based deployment
Proprietary and Confidential
Web-based Deployment
Portal Container
allows users to
build dashboards
Each Workflow
generate data for a
dashboard component
Workflow results
viewed in simple
charts - can be linked
to other pages
Proprietary and Confidential
Deployment Features (2)
Define multiple pages
Move to next page
Proprietary and Confidential
Example Application
Chip QC
Analytical stage
Normalise
Analyse
Interpret
Design Experiment
Design Study groups for
transcriptomics portal
Results
QC Step
n
ru
Re
t1
t es
QC t 1
tes
QC t 1
t es
QC t 1
tes
QC t 1
t es
QC t 1
tes
QC t 1
t es
t1
t es
QC
Splice Variance Analysis
Pre-process and Analyse
the results of an Exon Chip
to find differences in splice
variance between control
vs. test populations
Submit to Report>
QC
Gene Expression Profiling
Pre-process and Analyse
the results of a gene
expression analysis to
compare control vs. test
populations
Slide 1
Slide 2
Slide 3
Next Steps
Slide 4
Slide 5
WorkflowSlide
configured
6
to group according
to stage
Proprietary and Confidential
Chip must be rerun
Recommended rerun
Normalisation
services
•RMA
(recommended)
•LiWong
•ETC
Portal look and feel can be
customized by style sheet
Example Application
Chip QC
Normalise
Analyse
Interpret
Design Experiment
Design Study groups for
transcriptomics portal
Gene Expression Profiling
Pre-process and Analyse
the results of a gene
expression analysis to
compare control vs. test
populations
Submit to Report>
Results
Splice Variance Analysis
Pre-process and Analyse
the results of an Exon Chip
to find differences in splice
variance between control
vs. test populations
Next Steps
Analysis services
•Volcano Plot
(recommended)
•PCA
•Dendrogram
Proprietary and Confidential
Example Application
Chip QC
Normalise
Analyse
Interpret
Design Experiment
Design Study groups for
transcriptomics portal
Gene Expression Profiling
Pre-process and Analyse
the results of a gene
expression analysis to
compare control vs. test
populations
Submit to Report>
Results
Save Result to
Report
Splice Variance Analysis
Pre-process and Analyse
the results of an Exon Chip
to find differences in splice
variance between control
vs. test populations
Next Steps
Analysis services
•Select Transcripts
•Filter Data
Proprietary and Confidential
Example Application
Chip QC
Normalise
Analyse
Interpret
Design Experiment
Design Study groups for
transcriptomics portal
Gene Expression Profiling
Pre-process and Analyse
the results of a gene
expression analysis to
compare control vs. test
populations
Submit to Report>
Results
Save Selected
Items to Report
Splice Variance Analysis
Pre-process and Analyse
the results of an Exon Chip
to find differences in splice
variance between control
vs. test populations
Next Steps
Interpretation
services
•Send Data to
Ingenuity
•Send Data to Gene
Go
•Send Data to
•Text Analysis
Proprietary and Confidential
Example Application
Chip QC
Normalise
Analyse
Interpret
Design Experiment
Design Study groups for
transcriptomics portal
Gene Expression Profiling
Pre-process and Analyse
the results of a gene
expression analysis to
compare control vs. test
populations
Submit to Report>
Results
Save to Report
Related Pathway
Splice Variance Analysis
Pre-process and Analyse
the results of an Exon Chip
to find differences in splice
variance between control
vs. test populations
Next Steps
Interpretation
services
•Send Data to Gene
Go
•Text Analysis
Proprietary and Confidential
Example Application
Chip QC
Normalise
Analyse
Interpret
Design Experiment
Design Study groups for
transcriptomics portal
Gene Expression Profiling
Pre-process and Analyse
the results of a gene
expression analysis to
compare control vs. test
populations
Submit to Report>
Select Subset for
Text Analysis
Results
Related Pathway
Splice Variance Analysis
Pre-process and Analyse
the results of an Exon Chip
to find differences in splice
variance between control
vs. test populations
Next Steps
Interpretation
services
•Send Data to Gene
Go
•Text Analysis
Proprietary and Confidential
Business Process Workflow (1)
Business Process Management Development

A Business Process Management (BPM) describes the
orchestration of different tasks to complete a specific
business objective

Business Processes need to orchestrate
 Automated Tasks
 User Tasks
 Exception Handling
 Running Tasks in parallel
 Synchronisation of parallel tasks
Proprietary and Confidential
Business Process Workflow (2)
InforSense Control Flow

InforSense Control Flow for Orchestrating Workflows for
Business Process
Run Task
Initiate Parallel
Tasks
Handle
Exceptions
Proprietary and Confidential
Apply Rules
Synchronize
Parallel Tasks
Business Process Workflow (3)
Orchestra business analytics by control flow
Control Flow Represents a
Business Process
Process
Building
Blocks
Deploy to
Portal
Sub-process 1
Workflow A
Sub-process 2
Workflow B
Workflow C
definition of
linkage/control
and user
interactions
Application
Building
Blocks
services
Proprietary and Confidential
Workflow interoperability
Workflows and business processes (BPEL)
Proprietary and Confidential
Embedding Workflow Analytics into Applications
Process View
Embeddable Analytic
Applications
Analytical Workflows
customer data
Get Value Score
Get Churn
Score
Churn Service
Predictive scores
Lifetime Value Service
Risk data
Risk Service
Risk Assessment
Risk Evaluation
KVM
Acceptable Risk?
No
Deploy New Actions
Yes
Normal Service
Upgrade offer
Model Repository
Business Rules and Model
Proprietary and Confidential
Integrating Analytics with Business Rules:
Adaptive Business Process
Business
Process
Business
operational
data
Proprietary and Confidential
Rule engine
Rule Engine
Enterprise Services Bus
Analytics to
drive
adaptive
processes
Business Portal
Embedding with Applications
InforSense Tools as one item in Windows based application system
Proprietary and Confidential
Proprietary and Confidential
Making Workflow Work
“One of the biggest barriers to achieving productivity and responsiveness is IT – it has
become a bottleneck. Another barrier to achieving the goal is the lack of intelligence that
drives most IT applications. They are just operating as a rapid functional replacement, and
failing to exploit the data which is being generated within other elements of the IT
infrastructure.
A product that could meet that challenge and enable business to generate and
deploy intelligence with speed, accuracy and without the need for specialized
skills would be remarkable.
I believe that InforSense is that remarkable tool.”
-- David Norris, Senior Analyst, Bloor Research
Proprietary and Confidential
Thank You !
Proprietary and Confidential