Download Lecture-13-Environments - Computation Structures Group

Reactive and Responsive
Intelligent Environments
Kevin Quigley
aire group
MIT AI Lab
LCS
Reactive and Responsive Environments
• We are trying to build pervasive, perceptually enabled humancentered environments
• Such an environment must respond in reasonable ways to
high level requests from its users.
– It should be up to the system to figure out a reasonable way to implement
the request, translating goals to plans that meet the users needs and that
utilize available resources.
– E.g. I might ask to light the room up; the system responds by opening the
drapes.
• Such an environment should react to events in the
environment even when there is no explicit user request.
– E.g. when I walk into my room in the morning, the system should light up
the room.
• Reactions and responses should both be contextually
sensitive.
• Both must show human levels of adaptivity
Kevin Quigley — MIT Artificial Intelligence Lab
Oxygen Alliance Annual Meeting — February 25 – 28, 2003
5 Challenges
1. Providing a practical level of knowledge representation that
enables groups interactions and grounding in the real world of
space and time
2. Providing run-time composable services in a multi-user
environment that make optimal use of the currently available
resources
3. Recovering from equipment failures, information attacks,
misestimates of sensors, etc.
4. Coordinating and fusing information from many sensors and
modalities
5. Capitalizing on and recognizing context (task, location, personal
style & state)
6. Maintaining security and privacy and trading these off against
other goals
Kevin Quigley — MIT Artificial Intelligence Lab
Oxygen Alliance Annual Meeting — February 25 – 28, 2003
Challenge 1:
Grounding in Real-World Semantics
• We want to build applications that service many
individuals and groups of individuals
• These people will move among many physical spaces
• The devices and resources they use change as time
progresses
• The context shifts during interactions
• The relevant information base evolves as well.
• The system is required to respond dynamically
Kevin Quigley — MIT Artificial Intelligence Lab
Oxygen Alliance Annual Meeting — February 25 – 28, 2003
Research Agenda:
Knowledge Representations
• People
•Information nodes
–Topic area, place in ontology,
– Interests, skills,
format
responsibilities, organizational
•Services
role
• Organizations
– Members, structure
• Spaces
– Location
– Subspaces
– Devices and resources
• Resources
Kevin Quigley — MIT Artificial Intelligence Lab
–Methods, parameter bindings,
resource requirements
•Agents
–Capabilities, society, acting on
behalf of whom
•Events
–E.g. Person identification,
motion into a new region of
space, gestures
–Qualitative Changes in any of
the properties in the KR
Oxygen Alliance Annual Meeting — February 25 – 28, 2003
Challenge 2:
Adaptive Resource Management
• In most systems, applications are written in terms of
specific resources
– (e.g. The left projector in Michael’s office, or worse yet, a physical
address).
• This is in conflict with
–
–
–
–
–
–
Portability across physical contexts
Changes in equipment availability across time
Multiple applications demanding similar resources
Need to take advantage of new resources
Need to integrate mobile devices as they migrate into a space
Need to link two or more spaces
• What is required is a more abstract approach to
resources in which no application needs to be tied to a
specific device.
Kevin Quigley — MIT Artificial Intelligence Lab
Oxygen Alliance Annual Meeting — February 25 – 28, 2003
An Example of Service Requests
• When I come into my office in the morning it’s dark
– The lights are out
– The drapes are closed
• I ask the office to light up the room
• It’s a sunny day, it opens the drapes
– If I had asked it to turn on the lights, it wouldn’t have opened the drapes
• It’s a cloudy day, it turns on the lights
Kevin Quigley — MIT Artificial Intelligence Lab
Oxygen Alliance Annual Meeting — February 25 – 28, 2003
Responsive, Goal-Directed Processing
Goals
Service
Mapper
Plans
Resource
Allocator
Resource
Pool
Actions
Service mapping is provided by the resource management component
Kevin Quigley — MIT Artificial Intelligence Lab
Oxygen Alliance Annual Meeting — February 25 – 28, 2003
Services are Dynamically Mapped to Plans
Service
Control
Parameters
User’s
Utility
Function
Each Method Binds the Settings of The
Control Parameters in a Different Way
Abstract
Service
The Binding of
Parameters Has
a Value to the
User
Each Plan Requires
Different Resources
User
Requests A
Service With
Certain
Parameters
Resource1,1
Plan1
Resource1,2
Plan2
Resource1,j
Resource
Cost
Function
Plann
Each Service Can Be
Provided by Several
Plans
The Resources
Used by the Method
Have a Cost
Net Benefit
The System Selects the Plan Which Maximizes Net Benefit
Kevin Quigley — MIT Artificial Intelligence Lab
Oxygen Alliance Annual Meeting — February 25 – 28, 2003
Challenge 3:
Robustness and Recovery From Failures
•
Breakdowns are inevitable
– Resources sometimes fail while being used
– The system acts on sensor data which has uncertainty
•
•
•
The system renders services by translating them into plans
A plan-monitor watches over the execution of a plan.
Each plan step accomplishes sub-goals needed by
succeeding steps
– Each sub-goal has some way of monitoring whether it has been accomplished
– These monitoring steps are also inserted into the plan
•
•
If a sub-goal fails to be accomplished, model-based diagnosis
isolates and characterizes the failure
A recovery is chosen based on the diagnosis
– It might be as simple as “try it again”, we had a network glitch
– It might be “try it again, but with a different selection of resources”
– It might be as complex as “clean up and try a different plan”
Kevin Quigley — MIT Artificial Intelligence Lab
Oxygen Alliance Annual Meeting — February 25 – 28, 2003
The Plan Monitor
Manages Recovery From Failures
I don’t see light on the screen
Plan 1:
• Locate a systems wizard in the E21
• Monitor: check that person is still there
• Turn on the selected projector
• Monitor: check that projector turned on
• Project the message
• Done
• Monitor: check that the person noticed
the message
I need to ask a question
of a systems wizard
I see Sally by the screen
Kevin Quigley — MIT Artificial Intelligence Lab
Plan
Breakdown
Projector-1 must be broken.
We’ll try again, but using
Projector-3.
Oxygen Alliance Annual Meeting — February 25 – 28, 2003
Making the System Responsible for Achieving Its Goals
Service
Request
Diagnostic
Service
Plan-for
Localization &
Characterization
Repair Plan
Selector
alerts
StepB
StepA
Scope of Recovery
Selection of Alternative
Rollback
Designer
requires
achieves
Condition-1
Condition-1
Concrete
Repair Plan
prerequisite
Plan
Monitor
A plan is a partially ordered collection of steps
Each step achieves a subgoal
Some steps establish pre-requisites for others
Kevin Quigley — MIT Artificial Intelligence Lab
Resource
Allocator
Resource Plan
Enactment
Oxygen Alliance Annual Meeting — February 25 – 28, 2003
Self-Adaptive
Goal Directed Processing
Goals
Service
Mapper
Plan
Monitor
Diagnosis
& Recovery
Plans
Resource
Allocator
Resource
Pool
Actions
Kevin Quigley — MIT Artificial Intelligence Lab
Oxygen Alliance Annual Meeting — February 25 – 28, 2003
Challenge 4: Context Awareness
• The context should influence how the system behaves:
–
–
–
–
Task Structure
Location
Emotional State
Personal Style
• Perception can help determine the context
• The system should choose its reactions to events based on
the context
• Perceptual interpretation should be biased by context
– E.g. a person near the White Board, is likely to start drawing
• Estimation of utility should be influenced by context
Kevin Quigley — MIT Artificial Intelligence Lab
Oxygen Alliance Annual Meeting — February 25 – 28, 2003
A Reactive System Responds to Events
If somebody enters the room
Then illuminate the room
Kevin Quigley — MIT Artificial Intelligence Lab
Oxygen Alliance Annual Meeting — February 25 – 28, 2003
Structure of REBA
• A Reaction maps an event to a goal
• Reactions are grouped into behavioral bundles
– Sets of reactions that are always activated as a unit
• The Context is determined by the task within a plan
– As well as location, people present, ...
• Each context activates a set of behavioral bundles
• Contexts have Sub-Contexts, activities that occur within other
activities
– Watching a video within a meeting
• The active reactions of a sub-context override the reactions of
the parent context
Goal2
Sub-context-1
Goal1
Context Stack
Sub-context-2
Reactions
Events
Kevin Quigley — MIT Artificial Intelligence Lab
Oxygen Alliance Annual Meeting — February 25 – 28, 2003
Context Awareness Conditions Reactions
If somebody enters the room
Then illuminate the room
Kevin Quigley — MIT Artificial Intelligence Lab
But not if a movie is being watched
Oxygen Alliance Annual Meeting — February 25 – 28, 2003
Context Sensitivity
Goals
Reactive
Manager
Service
Mapper
Diagnosis
& Recovery
Context
Plan
Monitor
Plans
Resource
Allocator
Resource
Pool
Events
Actions
Reactive Processing is provide by the REBA MetaGlue Component
Kevin Quigley — MIT Artificial Intelligence Lab
Oxygen Alliance Annual Meeting — February 25 – 28, 2003
Challenge 5:
Perceptual Integration
• We want to separate the implementation of perceptual tasks
from the uses to which perception is put
• Some modules advertise the class of “behavioral events”
they are capable of recognizing and signaling
– These events are organized into a taxonomy
– The same event can be signaled by quite different perceptual modules
(e.g. both face and voice recognition can localize a person).
• Other modules register their interest in certain classes of
events
– Requests at a higher level in the taxonomy subsume lower level events
• Modules which receive low-level events may register for and
collate many different classes of events
– They combine these and signal higher-level events
• Modules may request perceptual services when they are
uncertain of their conclusions
Kevin Quigley — MIT Artificial Intelligence Lab
Oxygen Alliance Annual Meeting — February 25 – 28, 2003
A Dynamic Event Bus For Perceptual
Integration
Interested in the Location of Individuals
Signal People Approaching the Whiteboard
White Board
Context Manager
Interested in Face Location
I Signal the Location of Individuals
Face
Recognition
Interested in Body Motion
Signal the Location of Individuals
Interested in Body Motion
I Signal Face Location
Face
Spotter
Voice
Identification
Signal Body Motion
Visual Tracker
• A “Blackboard” System
• Publishers & Subscribers Are “Knowledge Sources”
• Events Are the Blackboard Data Items
• Highly Distributed
• Use of Bayesian Techniques
Kevin Quigley — MIT Artificial Intelligence Lab
Oxygen Alliance Annual Meeting — February 25 – 28, 2003
Blackboards: Using Context
Context:
Discussion of Oil Tanker Crash
Fragility of Environment
It’s not Hard to Wreck a Nice Beach
intent
Wreck
Recognize
A Nice Beach
Speech
Wreck A NICE
Recognize
Rec
R
eck
WreckOg A
Og
Kevin Quigley — MIT Artificial Intelligence Lab
word
NIS
IZE
N
sentence
I Z
Syllable
SP
phonemic
acoustic
Oxygen Alliance Annual Meeting — February 25 – 28, 2003
Integration of Reactive and
Goal Directed Processing
Goals
Reactions
Service
Mapper
Events
Context
Plan
Monitor
Plans
Blackboard
Resource
Allocator
Sensory Systems
Actions
Kevin Quigley — MIT Artificial Intelligence Lab
Diagnosis
& Recovery
Resource
Pool
Oxygen Alliance Annual Meeting — February 25 – 28, 2003
Summary
• Responsive processing dynamically maps goals to plans
– Plans are selected by balancing the benefit to the user against the cost of
resources
– Plan monitoring recovers from plan breakdown
– Access control is handled as part of the cost benefit analysis
• Reactive processing dynamically maps event to goals
– Events are handled within context
– Perceptions are maps to events within context
Kevin Quigley — MIT Artificial Intelligence Lab
Oxygen Alliance Annual Meeting — February 25 – 28, 2003
Reactive and Goal Directed Processing
with Security
Goals
Reactions
Service
Mapper
Events
Context
Kevin Quigley — MIT Artificial Intelligence Lab
Plans
Resource
Allocator
Blackboard
Sensory Systems
Plan
Monitor
Access
Policies
Diagnosis
& Recovery
Resource
Pool
Actions
Oxygen Alliance Annual Meeting — February 25 – 28, 2003
The Metaglue System
Software agents for intelligent spaces
LCS
Computational needs
• Smart Environments
– Distributed
Components
– Dynamic Changes
– Highly Varied
Equipment
– New Modes of
Interaction
– Frequent Failures
– Abundance of
Information
Kevin Quigley — MIT Artificial Intelligence Lab
• Metaglue
– Communication
– Resource
Management
– Customization
– Multi-modal HCI
– Agent Recovery
– Persistent Storage
Oxygen Alliance Annual Meeting — February 25 – 28, 2003
The Metaglue System
• What is it?
– A communication infrastructure for building systems of
distributed software agents
– A software architecture for creating adaptive applications for
Intelligent Environments
• Why for Intelligent Environments?
– dynamic adaptation of the applications based on
* the availability of resources (hardware/software) in
the current system/environment
* security controls of the participants
* preferences of the participants
Kevin Quigley — MIT Artificial Intelligence Lab
Oxygen Alliance Annual Meeting — February 25 – 28, 2003
Metaglue Software Infrastructure
• What do you get with Metaglue?
– Multi-modal Human-Computer Interaction
* Spoken Natural Language
* Perceptual (Vision)
* Direct manipulation (Graphical)
– Persistent storage
– Multiple communication paths
* Remote procedure call (Java RMI)
* Publish/Subscribe message passing
– Configurable setting for asserting preferences
– Start on demand of Agents
– Automatic recovery / handling of direct communication errors
– Resource Management and Service mapping
Kevin Quigley — MIT Artificial Intelligence Lab
Oxygen Alliance Annual Meeting — February 25 – 28, 2003
What Are Software Agents?
Our definition:
• A software agent is any software object capable of
communication by exposing functionality to other agents
running within the network.
Metaglue is a Multi-Agent System where agents perform
individually specialized, (usually) simple tasks but connect in
a web of intercommunication to cooperate on more complex
tasks.
Kevin Quigley — MIT Artificial Intelligence Lab
Oxygen Alliance Annual Meeting — February 25 – 28, 2003
More than Java Objects
• Metaglue Agents can be as complex or simple as necessary
– Larger programs can be used in Java thru JNI or other interface
* ViaVoice, ASSIST (a sketching tool)
– Most Metaglue Agents are simpler and single-purpose
* Projector display or light control
– Complex agents are those that control other simpler agents
* ReBa – Reactive Behavioral System
* SPIe – Self-adaptive Plan-based Intelligent Environment
• Metaglue provides to agents
–
–
–
–
identity and occupation
the entity this agent represents
location and the ability to change locations
intrinsic communication to other agents
Kevin Quigley — MIT Artificial Intelligence Lab
Oxygen Alliance Annual Meeting — February 25 – 28, 2003
What are some of the agents in Metaglue?
• Device Control
– lights, drapes, fans, sensors
– DVD & CD players, MP3
– video multiplexers, projectors
– Cameras, TVs, VCRs, Audio
• Agent Systems Applications
– ReBa – interactive behaviors
– SPIe and Planlet – plan monitoring
– Web info display
• Debugging and Logging
• Data Organization Systems
–
–
–
–
Blackboard agents – data flow
START system – NLP & KB
Presence & Location agents
Newswall – visual data organization
and presentation
Kevin Quigley — MIT Artificial Intelligence Lab
– Agent testing, Simulated devices
– Log & Catalog Monitors
– Notification listeners
• Recreation
– Checkers, Boggle, RPG,
HexaCheckers, Crosswords,
ELIZA clone
Oxygen Alliance Annual Meeting — February 25 – 28, 2003
System Organization - Societies
• Clusters of agents that operate on behalf of a real-world entity
(space or person) are called Societies
• Societies allow the same agent to exist with different
customized attributes.
• Agents can talk to agents in other societies as easily as their
own society
• Societies look like agents when viewed from the outside
– They exposed functionality to higher level resource management through
Hyperglue
Kevin Quigley — MIT Artificial Intelligence Lab
Oxygen Alliance Annual Meeting — February 25 – 28, 2003
System Organization - Catalog
• The Catalog Agent is the central component which knows
about all running agents
Kevin Quigley — MIT Artificial Intelligence Lab
Oxygen Alliance Annual Meeting — February 25 – 28, 2003
System Organization - MetaglueVM
To Catalog
• The Metaglue Virtual Machine (MVM) runs as a base platform for all
other agents in the Metaglue system.
– It handles all registration of the agent with the Catalog Agent
– Provides methods for direct communication (RMI) to other agents on the
current catalog
Kevin Quigley — MIT Artificial Intelligence Lab
Oxygen Alliance Annual Meeting — February 25 – 28, 2003
Calling on Other Agents – agent present
1. A reliesOn call by Agent B will create a chain of events to get the RMI stub
for Agent A
2. The MVM takes the call and passes it to the Catalog Agent
3. The Catalog checks the internal table of agents to see if there is one
matching the requested description
1. If the agent exists, the RMI stub for Agent A is returned
Kevin Quigley — MIT Artificial Intelligence Lab
Oxygen Alliance Annual Meeting — February 25 – 28, 2003
Calling on Other Agents – agent not
present
3. The Catalog Agent checks the internal table of agents to see if there is one
matching the requested description
2. If the agent does not exist, it must be started locally (calling MVM)
3. The new Agent A registers its stub
4. That stub is then returned to Agent B
Kevin Quigley — MIT Artificial Intelligence Lab
Oxygen Alliance Annual Meeting — February 25 – 28, 2003
Automatic Agent Recovery
Failure recovery through proxies
LCS
Without error handling
Kevin Quigley — MIT Artificial Intelligence Lab
Oxygen Alliance Annual Meeting — February 25 – 28, 2003
Without error handling
Kevin Quigley — MIT Artificial Intelligence Lab
Oxygen Alliance Annual Meeting — February 25 – 28, 2003
Without error handling
Kevin Quigley — MIT Artificial Intelligence Lab
Oxygen Alliance Annual Meeting — February 25 – 28, 2003
Without error handling
Kevin Quigley — MIT Artificial Intelligence Lab
Oxygen Alliance Annual Meeting — February 25 – 28, 2003
Error handling with proxy objects
Kevin Quigley — MIT Artificial Intelligence Lab
Oxygen Alliance Annual Meeting — February 25 – 28, 2003
Error handling with proxy objects
Kevin Quigley — MIT Artificial Intelligence Lab
Oxygen Alliance Annual Meeting — February 25 – 28, 2003
Error handling with proxy objects
Kevin Quigley — MIT Artificial Intelligence Lab
Oxygen Alliance Annual Meeting — February 25 – 28, 2003
Statistics on Metaglue
• 10 Tons of fun:
– There are over 450 agents that exist within Metaglue
– Between 50 and 80 agents are running the intelligent room
– You are using more than 10 agents just while running the
X10BasicLightControl
* Test it! Use agentland.util.LogMonitor
• Metaglue has been in development since 1998
• The system is used in several offices and homes including the
office of the AI lab director, Rodney Brooks
• There are 2 full spaces at MIT (a 3rd is coming soon!) and one
space in Australia running Metaglue
– Why not get your own?
Kevin Quigley — MIT Artificial Intelligence Lab
Oxygen Alliance Annual Meeting — February 25 – 28, 2003