Download Lecture 7

Process Query Systems
ENGS 112
Lecture 7
Process Query Systems (PQS) vs
Data Base Systems (DBS)
Data Base System
Field
Oriented
Data
Field
Oriented
Query
(eg SQL)
Business
Requirement
Data Sources
Query Responses
Field
Oriented
Data
Process Query System
Business
Requirement
Data Base
System
Process
Oriented
Query
Data Sources
Process
Query
Responses
Process
Query
System
Data Base
System
When do we use “processes”?
• Is there a large ground vehicle convoy moving towards our
position? (Tactical C4ISR)
• Is there an unusual pattern of network and system calls on a
server? (Cyber-security)
• Is there a pattern of unusual document accesses within the
enterprise document control system? (Insider Threat Detection)
• Is there a pattern of unusual transactions? (Homeland Security)
• Is my software operating normally? (Autonomic computing)
• Is the workflow system working normally? (Business Process
Engineering)
IMPORTANT – All are “adversarial” processes, not cooperative so
the observations are not necessarily labeled for easy
identification and association with a process!
How are processes defined or
specified? (viz SQL)
A “state”
State transition
An “observation”
Non-branching process (A then B then C then D then …)
Branching process (A then (B or C or D) then (if B then E or F) or if then …)
How are processes defined or
specified? (viz SQL) cont’d
A “state”
State transition
An “observation”
kinematic state at time t + Dt
kinematic state at time t
F = ma
Laws of motion
Continuous kinematic processes (constrained by physical laws)
p( s(t) = si | s(t-1) = sj ) = pij, p( o(t) = ok | s(t) = sj )
State (observation) at time t is s(t) (o(t),
state space = { si }, observation space = { oj }
Markovian models (eg, source code generators and channel models)
Petri network models
Process descriptions
• Using RDF or DAML ontologies
• Describe states, possible state transitions
and observables using either RDF/DAML
and/or remote objects and methods
• Detection of identifiable “allowed”
processes leaves a residue of anomalies
in the DBS so this can be used for
anomaly-based detection as well as
signature-based detection
Example – vehicle tracking
Time t
Time t+1
Time t+2
Example – vehicle tracking
Time t
Time t+1
Time t+2
Example – vehicle tracking
Time t
Time t+1
Time t+2
Example – vehicle tracking
Time t
Time t+1
Time t+2
Example – vehicle tracking
Time t
Time t+1
Time t+2
Example – vehicle tracking
Time t
Time t+1
Gates
Predicted tracks
Hypotheses
Time t+2
Common Logic
•
•
•
•
•
•
Track-hypothesis initiation
Hypothesis management – pruning
Handling missed detections
Handling track termination
Subscription via gates
Publication of hypothesized tracks
Process Query System algorithms
• Continuous kinematics – Kalman filtering
• Markov processes – Viterbi-like algorithms
• Multiple process disambiguation – multitarget multiple hypothesis tracking (MHT)
algorithms
• Model estimation and optimization Estimation-Maximization (EM) Methods
• Many-to-one (multispectral sensing) and
one-to-many (coincidental evidence)
observation-to-process resolution methods
Generic PQS Operation
1. A user defines a “query” in terms of a process
description
2. The process query is submitted to the PQS
3. The PQS searches the DBS for evidence of
process instances (may be more than 1)
4. The evidence consists of ordinary DBS records
5. The PQS manages the complexity of having
multiple instances of the processes
6. The PQS maintains a pool of the most likely
hypotheses about which processes exist and
what their states are
A PQS Implementation TRAFEN
• TRacking And Fusion ENgine
• Currently in alpha version
• Handles ground vehicle tracking using
acoustic sensors
• Handles ICMP-T3 event analysis for worm
detection
• Uses RDF, DAML and web services for
resource description, discovery and
access
Data mining and Expert Systems
• Data mining
– used to discover unknown patterns in data
– can be used to define processes (front end)
• Expert Systems
– TRAFEN implements a specialized logic for
process discovery
– ES’s do not handle statistical inferences as
needed in this domain
To do list
• Graphical programming tools for processes and
semantic descriptions
• Control/activation of processes and/or sensor
infrastructure – ie. “feedback loops”
• Performance models and analysis (sampling rates,
hypothesis growth, etc)
• Hierarchical process capabilities (ie, identified
processes are observables of higher level
processes, such as convoys are aggregations of
vehicles, etc)
• Truly generic Kalman, HMM, MHT algorithms