Download Ideas on the LCG application architecture

Ideas on the LCG Application
Architecture
Application Architecture Blueprint RTAG
12th June 2002
P. Mato / CERN
Application Domains




Event Processing Applications
– Trigger, Simulation,
Reconstruction, Selection programs
Mainly batch oriented.
Interactive for
development & debugging.
Real-time.
– Event/Detector display, data
presentation programs
Mainly interactive
– Calibration and Alignment programs
Batch and interactive
Data Analysis
Detector calibration
Job configuration, submission,
monitoring and control
– Grid awareness
24/10/2001
LCG Application Architecture
Mainly interactive
2
No Disjoint Domains

For example, the LHCb requirements for
interactive analysis:
– Better [than PAW] integration with experiment
framework
» consistent with the analysis batch environment, use
the same toolkits and experiment algorithms/tools
» access the experiment data objects and allow
browsing
» integrated with event display
» allow interactive reconstruction and simulation
24/10/2001
LCG Application Architecture
3
Architectural styles

General categorization of systems [1]
user-centric
focus on the direct visualization
and manipulation of the objects
that define a certain domain
data-centric
focus upon preserving the integrity
of the persistent objects in a
system
computation-centric focus is on the transformation of
objects that are interesting to the
system
[1] G. Booch, “Object Solutions”, Addison-Wesley 1996
24/10/2001
LCG Application Architecture
4
Different style in different domains
• The applications in the
different domains may
have different
emphasis in:
• Interactivity
datacentric
usercentric
User Interface
Interactivity
Scripting
Object store
Database
Data Integrity
• Database
• Computation
Algorithms
Computation
• Elements of all three
are present in all
applications
24/10/2001
computation-centric
LCG Application Architecture
5
Computation-centric: GAUDI


Framework adequate for “all” event processing applications
Algorithms process “event data” with the help of “services”
and using “detector data”.
Application
Manager
Message
Service
JobOptions
Service
Particle Prop.
Service
Other
Services
24/10/2001
Converter
Converter
Converter
Event
Selector
Event Data
Service
Persistency
Service
Data
Files
Transient
Event Store
Algorithm
Algorithm
Algorithm
Detec. Data
Service
Transient
Detector
Store
Persistency
Service
Data
Files
Histogram
Service
Transient
Histogram
Store
Persistency
Service
Data
Files
LCG Application Architecture
6
User-centric: PAW




Provides interactive graphical presentation and
statistical or mathematical analysis, working on objects
familiar to physicists like histograms, event files
(Ntuples), vectors, etc.
Is based on several components of the CERN Program
Library. Easy to use and freely available
No integration within experiments framework
Offline analysis done in 2 separate steps “batch” and
“interactive”, the user decides how much to do in each
step
24/10/2001
LCG Application Architecture
7
The Challenge
Many components are shared among the different
applications in the different domains
 We would like to profit from the best of the
existing solutions (implementations) for each
component but not duplicate everything.

>>> INTEGRATION <<<
24/10/2001
LCG Application Architecture
8
Component interface specifications

Integration technologies
Probably the best solution
but unrealistic in our
environment
– Mainly for distributed computing
– CORBA, DCOM (.NET), Web Services, JavaBeans

Abstract interfaces
Works well within a given
Framework
– C++ “in process” integration
– Robust and efficient. A bit awkward for end-users.

Scripting extension modules
– Ideal for prototyping. Rapid application development
Minimal coupling,
(RAD).
maximum flexibility.
– Dependent of the scripting tool
Lacks performance.
24/10/2001
LCG Application Architecture
9
For user-centric applications
Interactive analysis,
configuration, monitoring
applications, etc.
Software Bus

Define a “bus” specifications equivalent to a
hardware bus (e.g. VME) where modules
implementing different functionality can be
plugged easily.
Grid
Database
GUI
shell
Bus
Analysis
Tools
(Lizard)
24/10/2001
Experiment
Framework
(Gaudi)
LCG Application Architecture
math
math
11
Python as a Software-Bus
Python is an object-oriented scripting language
commonly used in a wide variety of domains
 But, it could also be seen as framework where you
can plug easily “extension modules” in binary form
implemented using other languages.

– Very easy and natural to interface to C++ classes (C++
API)
– Python should only be the “glue” between modules
developed in C++ or other languages
– The interface (API) for Python extension modules is
quite simple and at the same time flexible
24/10/2001
LCG Application Architecture
12
Module Communication

Communication between two extension modules is always
possible using Python primitive types (a)
– Example: I would like to display an histogram of Gaudi

It is more efficient to
communicate using a
reference to an interface Python
(AIDA?) (b)
– The objects in Python
are only references
Both ways are possible!!
24/10/2001
Python
Primitives
(a)
(b)
Plotter
IH1D
Lizard (C++)
LCG Application Architecture
IH1D*
Gaudi (C++)
13
Why Python?

Interpreted
– However it is quite fast (byte code idea from Java)

Dynamically typed
– Not need to declare any variable
 Simple
syntax
– Emphasis by the authors to minimize typing
Shells
 Powerful built-in types and modules
 Ideal for Scripting and Prototyping

24/10/2001
LCG Application Architecture
14
What is Feasible?
Very rich set
specialized
generic modules
LHC modules
Gateways to
other frameworks
EDG API
PVSS
XML
Several GUI
toolkits
Database
GUI
GUI
Python
JPE
rootpython
gaudipython
Java
Classes
Root
Classes
Gaudi
Framework
24/10/2001
LCG Application Architecture
math
math
shell
Very rich set of
Python standard
modules
15
Example
Filling an
Excel
spreadsheet
from a ROOT
ntuple
24/10/2001
# Get the ntuple from the ROOT file
import rootmodule
hfile
= TFile('hsimple.root')
ntuple = gROOT.FindObject('ntuple')
entries = ntuple.GetEntries()
nvar
= ntuple.GetNvar()
tuple
= ntuple.GetArgs()
# Initialize Excel
import win32com.client
excel = win32com.client.Dispatch('Excel.Application')
wbook = excel.Workbooks.Add()
wsheet = wbook.WorkSheets.Add()
wsheet.Name = ntuple.GetTitle()
# Fill Excel sheet
for i in xrange(500) :
ntuple.GetEntry(i)
for j in range(nvar) :
wsheet.Cells(i+1,j+1).value = tuple[j]
# Make Excel sheet visible
excel.Visible = 1
LCG Application Architecture
16
Ideal Interactive Application
Menu
GUI
Canvas
(histograms,
3D graphics,
etc.)
Data
Browser
>>> execfile(‘mymacro’)
>>> for c in cands :
plot(c)
>>>
24/10/2001
Scripting
Window
LCG Application Architecture
17
Where are the problems?

The Software-Bus architecture is not applicable
everywhere
– I would not like to implement the LHCb 2 level trigger
with that !
– Software-Bus + Abstract interfaces is a good
compromise for user-centric applications (fast
prototype)

In addition to “standardize” to the Python
extension API we need more
– AIDA ?, interoperability between Boost/Swig? …

Mixing several GUI toolkits, thread libraries, etc.
may pose serious technical problems.
24/10/2001
LCG Application Architecture
18