Download Sphinx Control Process

Main Sphinx Design Concepts
•
There are two primary design components which comprise Sphinx
• The Database Warehouse
• The Control Process
•
The Database Warehouse is the information depot for all of Sphinx
• All information is stored/retrieved via the database (or possibly multiple
databases)
• There is no direct communication between different internal Sphinx modules
• The API for all internal modules is SQL along with the database schema
•
The Control Process is the main point of activity
• It is a loop which calls different Sphinx modules
• All modules in the control process operate on stateful entities in the database,
changing the state of an entity from one to another state
• Sphinx operates as a Finite State Machine!
• Uses data mining principles in the design:
• The API for all modules is SQL along the database schema
• Modules mine the database for information and then take some action.
More Details
• The components are currently split (packaged) in the following way:
• Sphinx Client
•
•
•
•
•
Clarens Web Service (XML-RPC) + VDT-Client
Sends a Chimera DAX to Sphinx Server
Receives individual “concrete Jobs” from Sphinx Server
Submits “Job” (= “micro DAG”) to Condor-G/DAGMan
“micro DAG” (for now…could use Grid-shell or equivalent in the future)
•
•
•
•
•
•
Job to create environment
Job to move input data locally (if required)
Job to run application
Job to move output data (if required)
Job to register data in RLS
Job to clean-up environment
• Sphinx Server
•
•
•
•
Clarens Web Service (XML-RPC)
Receives a Chimera DAX
Returns individual concrete “Jobs” (=“micro DAG”)
Provides heuristic algorithms and Satisfies constraints for:
•
•
•
•
policies,
input data existence,
required resources
grid weather
Version 1 (current configuration):
Sphinx Server with Sphinx Submission Client
Multi-threaded
Control Process
Data Warehouse
Message Interface
Sphinx Client
Message Interface
DAG Reducer
Job Predictor
Policies
Accounting
Resource Prop.
Resource Hier.
Resource Dist.
Distance
Entity-Name
Entity Hier.
Job-Entity Prop.
Job
DAG
Rel. Data
Rel. Appl.
Job Tracking
Grid Weather
Schedule
Prediction
Message
DAG Predictor
Job Tracker
Job Submitter
Job Admission Control
DAG Admission Control
Condor-G/DAGMan
DAG Data Repl. Planner
VDT Client
Job Execution Planner
Resource
DAG Tracker
Data Management
Information Gatherer
Sphinx Server
indicates current place-holder implementation
Repl. Loc. Serv.
Monitoring Service
VDT Server Site
Version 1: Sphinx Control Process –
Simple Finite State Machine
DAG: unreduced
JOB: unpredicted
no
Await Mesg.
yes
DAG: unpredicted
JOB: unpredicted
no
Reduce DAG
yes
DAG: unpredicted
JOB: unaccepted
no
Predict Job
no
Predict DAG
DAG: unfinished
JOB: unsent
DAG: unfinished
JOB: remove
no
Admit Job
DAG: remove
JOB: remove
yes
Remove DAG
yes
no
Plan Job
no
Submit Job
yes
yes
DAG: unplanned
JOB: unplanned
Plan Dag
yes
yes
DAG: unaccepted
JOB: unplanned
no
yes
yes
DAG: unaccepted
JOB: unaccepted
DAG: unfinished
JOB: unplanned
no
Admit DAG
no
Remove Job
Examples of how planning modules can be moved
inside/outside the Sphinx Server/Client
Multi-threaded
Control Process
Data Warehouse
Message Interface
Sphinx Client
Message Interface
DAG Reducer
Job Predictor
Policies
Accounting
Resource Prop.
Resource Hier.
Resource Dist.
Distance
Entity-Name
Entity Hier.
Job-Entity Prop.
Job
DAG
Rel. Data
Rel. Appl.
Job Tracking
Grid Weather
Schedule
Prediction
Message
DAG Predictor
Job Tracker
Job Submitter
Job Admission Control
DAG Admission Control
Condor-G/DAGMan
DAG Data Repl. Planner
VDT Client
Job Execution Planner
Resource
DAG Tracker
Data Management
Information Gatherer
Sphinx Server
Repl. Loc. Serv.
Monitoring Service
VDT Server Site
Examples of how planning modules can be moved
inside/outside the Sphinx Server/Client
Multi-threaded
Control Process
Data Warehouse
Sphinx Client
Message Interface
Message Interface
DAG Reducer
Job Predictor
Entity Hier.
Job-Entity Prop.
Job
DAG
Rel. Data
Rel. Appl.
Job Tracking
Grid Weather
Schedule
Prediction
Message
DAG Predictor
Job Admission Control
DAG Admission Control
DAG Data Repl. Planner
Job Execution Planner
Job Submitter
Job Tracker
DAG Tracker
Data Management
Information Gatherer
Sphinx Server/VDT Client
Condor-G/DAGMan
Policies
Accounting
Resource Prop.
Resource Hier.
Resource Dist.
Distance
Entity-Name
UI Machine
Resource
Repl. Loc. Serv.
Monitoring Service
VDT Server Site
Examples of how planning modules can be moved
inside/outside the Sphinx Server/Client
Multi-threaded
Control Process
Data Warehouse
Sphinx Client
Message Interface
Message Interface
DAG Reducer
Job Predictor
Entity Hier.
Job-Entity Prop.
Job
DAG
Rel. Data
Rel. Appl.
Job Tracking
Grid Weather
Schedule
Prediction
Message
DAG Predictor
Job Admission Control
DAG Admission Control
DAG Data Repl. Planner
Job Submitter
Job Tracker
DAG Tracker
Data Management
Information Gatherer
Sphinx Server/VDT Client
Condor-G/DAGMan
Policies
Accounting
Resource Prop.
Resource Hier.
Resource Dist.
Distance
Entity-Name
Job Execution Planner
VO Specific
External Planning Service
Resource
Repl. Loc. Serv.
Monitoring Service
VDT Server Site