Download Soarian™ User Interface

Command and Natural Language

Command and Natural Languages
 Hieroglyphs (a 5000 year old technology) versus icons (the
technology of the day)
 Special languages
• Mathematics
• Music
• Chemistry
• Computers (interactive, user/programming communication)
1
2
Command and Natural Language

Command and Natural Languages
 Goals of a language
• Precision
• Compactness
• Ease of writing and reading
• Completeness
• Speed in learning
• Simplicity in reducing errors
• Ease of retention over time
3
Command and Natural Language
•
Command and Natural Languages
•
Tools that accelerated language development
• Printing Press versus Computer
•
•
•
•
Print Press provided widespread dissemination of written work
Computer provides widespread dissemination of words via networks
Computers provide tools to manipulate language
Language is a tool to manipulate computers
4
Command and Natural Language
•
Command and Natural Languages
•
Computer Languages
•
•
•
•
Modest influences on spoken natural languages
Huge impact on the development of formal written languages
Early computers were designed to perform mathematical computations
Future is to operate on the real world
• Directing robots
• Controlling manufacturing
• Guiding spacecraft
5
Command and Natural Language

Languages that tell computers what to do
Language
Program Size
# of Programmers
Use
FORTRAN
small
mostly single
engineering
COBOL
mostly large
mostly multiple
business
ALGOL
mostly large
mostly multiple
business
PL/1
small to large
mostly multiple
business
Pascal
small to large
single/multiple
academic
BASIC
small to large
single/multiple
business
LISP
small
few
artificial
intelligence
APL
small
few
mathematics
PROLOG
Mostly small
few
artificial
intelligence
ADA
mostly large
mostly multiple
government
C++
mostly large
multiple
business
JAVA
Mostly large
multiple
business, cross
platform
C#
small to large
single/multiple
business, cross
platform
Perl
Small
Small
web scripting
6
Command and Natural Language
•
Command and Natural Languages
•
Scripting Languages
• HyperCard
• Revolution
•
Database Query Languages
• Structured Query Languages (SQL)
• The goal of the user is to create a result, not a
program (3 to 20 lines)
• Boolean operations (AND, OR, NOT)
•
Command Languages
• Ephemeral
• Produce an immediate result on an object
• Example command line input to a browser:
http://www.google.com/
• New command languages have declined as a result of
direct manipulation
• Typical format: Verb followed by a noun with qualifiers
– PRINT MYFILE 3 COPIES
– UPD CAD TN=555-555-5555 BILLTN=302-555-1234
7
Command and Natural Language
•
A UI Design Approach to Command Language Interfaces
 Study the user’s task domain to determine the functionality required of the
application
 Derive a list of task actions and objects
 Represent the items as the low-level interface syntax
• Update Customer Record
• Copy File_A File_B
 Determine the frequency of specific tasks and facilitate the execution of
those tasks
 Determine whether destructive actions are reversible (e.g., delete)
 Determine error messages
 Transition diagram of how specific commands take the user to specific
states
8
Command and Natural Language
•
Transition Diagrams
iNUM
oNewNum
iCAR
oNewCar
iCADb
oNewCAD
iCPRb
oNewCPR
iCAD
iCAD
iUPD
iCADa
Logon
iCPY
iCPRa
oNoCAD
iDEL
iCAD
iCPR
oNoCPR
9
Command and Natural Language
•
UPD CAD=555-555-5555 NUM=444-444-4444 NewNum=333-333-3333
iNUM
oNewNum
iCAR
oNewCar
iCADb
oNewCAD
iCPRb
oNewCPR
iCAD
iCAD
iUPD
iCADa
Logon
iCPY
iCPRa
oNoCAD
iDEL
iCAD
iCPR
oNoCPR
10
Command and Natural Language
•
CPY CPR=555-555-5555 CPR=444-444-4444
iNUM
oNewNum
iCAR
oNewCar
iCADb
oNewCAD
iCPRb
oNewCPR
iCAD
iCAD
iUPD
iCADa
Logon
iCPY
iCPRa
oNoCAD
iDEL
iCAD
iCPR
oNoCPR
11
Command and Natural Language
•
DEL CAD=555-555-5555
iNUM
oNewNum
iCAR
oNewCar
iCADb
oNewCAD
iCPRb
oNewCPR
iCAD
iCAD
iUPD
iCADa
Logon
iCPY
iCPRa
oNoCAD
iDEL
iCAD
iCPR
oNoCPR
12
Command and Natural Language
•
Command Organization
•
Simplest – each command carries out a single task
•
Commands plus arguments
• COPY FILE_A,FILE_B
• DELETE FILE_A
• PRINT FILE_A,FILE_B,FILE_C
•
Delimiters of blanks or commas
•
Keywords
• COPY FROM=FILE_A TO=FILE_B
•
Options
• PRINT/3 PRINTER_LOC_A FILE_A
•
Command to check seat availability on a flight on August 21 from
Washington’s National Airport to LaGuardia Airport departing about 3:00PM
• A0821DCALGA0300PM
• Parsed: A 0821 DCA LGA 0300PM
• Speed
13
Command and Natural Language
•
Command Organization
•
•
Tree Structure
5x3x4 = 60 tasks based on 5 commands and 1 rule
Action
Object
Destination
CREATE
File
File
DISPLAY
Process
Local Printer
REMOVE
Directory
Screen
COPY
Remote Printer
MOVE
Remote Printer
14
Command and Natural Language
•
Command Organization
•
•
Tree Structure
5x3x4 = 60 tasks based on 5 commands and 1 rule
Action
Object
Destination
CREATE
File
File
DISPLAY
Process
Local Printer
REMOVE
Directory
Screen
COPY
Remote Printer
MOVE
Remote Printer
15
Command and Natural Language
•
Command Organization
•
•
Tree Structure
5x3x4 = 60 tasks based on 5 commands and 1 rule
Action
Object
Destination
CREATE
File
File
DISPLAY
Process
Local Printer
REMOVE
Directory
Screen
COPY
Remote Printer
MOVE
Remote Printer
16
Command and Natural Language
•
Command Organization
•
•
Tree Structure
5x3x4 = 60 tasks based on 5 commands and 1 rule
Action
Object
Destination
CREATE
File
File
DISPLAY
Process
Local Printer
REMOVE
Directory
Screen
COPY
Remote Printer
MOVE
Remote Printer
17
Command and Natural Language
•
Natural Language Interaction (NLI)
•
Users communicating in a familiar language (e.g., English) to a computer, and
receiving response from the computer in the same language
•
Potential applications
•
Pay to University of Delaware on 4/18/06 $1525.00
• What is the weather?
• How is traffic to work today?
 Previously direct manipulation interfaces have had more success in the market
18
Command and Natural Language
 Natural Language Queries (NLQ)
 Natural Language front end for databases
 The user types an English-like sentence
 The result is a structured database search
19
Command and Natural Language
 Natural Language Question Answering (NLQA)
 “Who was the first Prime Minister of the European Union?”
 Don’t attempt an exact answer
• Users make incorrect assumptions
– There is a president of the European Union
– There is a European Commission
 Better to return a list of results pages
20
Command and Natural Language
 Text-database searching





Used by Lycos, Google, AltaVista
Stemming (attempt to reduce a word to its stem or root form)
Relevance ranking by word-frequency analysis
Filtering of common words (e.g., “the”, “of”, “in”)
Latent semantic indexing
• Uses a term-document matrix which describes the occurrences of terms in documents
• It is a sparse matrix whose rows correspond to documents and whose columns
correspond to terms, typically stemmed words that appear in the documents.
• A typical example of the weighting of the elements of the matrix:
21
Command and Natural Language
 Natural Language Text Generation (NLTG)
 The computer generates readable reports
• “White-blood-cell count is 12,000”
• “This value exceeds the normal range of 3,000 to 8,000 by 50%”
• “Further examination for systemic infection is recommended”
 Natural Language, Adventure Games and Instructional Systems
 Example user commands
• TAKE ALL OF THE KEYS
• PICK UP THE SWORD
 Instructional software
• Guide students through the learning process
• Algebra and Geometry
• Feedback is provided in conversational language
22
Command and Natural Language
 Applications in mobile devices – Voice Commands
 Call





Recognizes verb
Accepts name or number as input
Requests clarification by providing a list of names
Confirms dialing
Useful for dialing while driving
 Send Text Message
 Recognizes verb
 Accepts name or number as input
 Opens the text composer
 Applications in mobile devices – Voice Read Out
23
Command and Natural Language
 Natural Language is applicable to situations where the user cannot
actually touch the computer or device
 Voice input
 Synthesized voice output
24
Command and Natural Language
•
Natural Language Interaction (NLI) vs. Direct Manipulation
• Users seek predictable responses
• Users get discouraged having to engage in clarification dialogs
• Visually oriented interaction exploits
• The computer’s capacity for rapid display of information
• The human’s ability for rapid visual recognition
25
Command and Natural Language
•
Natural Language Interaction (NLI)
•
Turing Test
• “Computer program whose conversation is indistinguishable from a human’s”
 ELIZA - emulates a Rogerian psychotherapist.
 ELIZA has almost no intelligence whatsoever, only tricks like string substitution
and canned responses based on keywords. The illusion of intelligence works best,
however, if you limit your conversation to talking about yourself and your life.
 http://www.manifestation.com/neurotoys/eliza.php3
 AskJeeves
• http://www.ask.com/
• Invites natural language questions
• Does keyword extraction
• Returns web-page result sets
26
Command and Natural Language
•
Natural Language Interaction (NLI) vs. Direct Manipulation
• Users seek predictable responses
• Users get discouraged having to engage in clarification dialogs
• Visually oriented interaction exploits
• The computer’s capacity for rapid display of information
• The human’s ability for rapid visual recognition
27
Command and Natural Language
•
NLI Future Vision – 2001: A Space Odyssey
 Astronauts set off in a spaceship controlled by HAL 9000, a revolutionary computer
system that is every bit mankind's equal, and perhaps his superior.
 HAL endangers the crew's lives for the sake of the mission, and eventually, kills
almost all of the astronauts in the film. The astronauts have to overcome the
computer.
 HAL behaves in the most human fashion of all of the characters.
• He reaches human intelligence levels,
• Develops human traits of paranoia, jealousy and other emotions.
 Humans act like machines
• Cooly performing their tasks in a mechanical fashion,
• Even under extreme duress.
 HAL, develops an acute emotional crisis because he can not accept evidence of his
own fallibility.
28
Command and Natural Language
•
NLI Future Vision – 2001: A Space Odyssey (1968)
 Check out this video to see the natural communication between the astronauts and
the computer.
 2001 Space Odyssey Video
•
•
•
•
•
•
•
•
Ease of communication
Personal assessment of Dave’s art work
Determining what is a “personal” question
HALs superior statement of human error to computer error
Preparing the crew psychological report
HAL providing a hard copy of a report
Astronauts talking in private about whether to trust HAL
HAL reading lips
http://www.youtube.com/watch?v=Be8Gbqdox68
29
Human Robot Interaction – AI by Spielberg
• AI – by Steven Spielberg
• Global warming lead humans to near to extinction
• Humans create robots known as mechas which can emulate thoughts
and emotions.
• David is designed to resemble a human child and to virtually feel love
for its human owners.
• Humans then become extinct
• Humanoid’s use David’s memories, to reconstruct the Swinton home
• The humanoids also create a clone Monica (his mother). She can only
live for one day.
• David spends the happiest day of his life playing with Monica and
Teddy.
30
Human Robot Interaction – AI by Spielberg
http://www.youtube.com/watch?v=xRhCz0ELrKs
31
Human Robot Interaction
•
More questions and answers
•
•
•
•
The role of facial expressions?
Context?
Personality?
The role of psychology in applying what is known about behavior to
robots?
• Education?
• Living family trees?
http://www.youtube.com/watch?v=IhVu2hxm07E
&feature=related
http://www.youtube.com/watch?v=pk
pWCu1k0ZI
32
Human Robot Interaction
http://www.pbs.org/wgbh/nova/tech/socialrobots.html
33
Human Robot Interaction
• Human-robot interaction (HRI)
• The study of interaction dynamics between humans and robots
• Robots name first used by Capek’s in the play R.U.R.
• http://www.youtube.com/watch?v=mEG7MhltHWk
• Robots roles in today’s society
• factory automation
• service applications
• medical care
• entertainment
• Asimov in I, Robot (1950) used the term ”robotics”
• Three laws of robotics:
1. A robot may not injure a human being or, through inaction, allow a
human being to come to harm.
2. A robot must obey orders given it by human beings except where such
orders would conflict with the First Law.
3. A robot must protect its own existence as long as such protection
does not conflict with the First or Second Law.
4. http://www.youtube.com/watch?v=WJRfWyTpJRQ&list=PL2FA31FC4B4268
06D
From: Human-Robot Interaction - David Feil-Seifer & Maja J Mataric (2009)
34
Human Robot Interaction
• Philip K. Dick’s novel Do Androids Dream of Electric Sheep (1968)
• Created replicants are humanoid robots that look and act like humans
• special tests are devised to determine if an individual is a human or a
replicant
• Test is related to the Turing Test
• Asks probing questions that require human experiences and
capacities in order to answer correctly.
• George Lucas’ Star Wars movies (1977)
• C3P0 and R2D2
• http://www.youtube.com/watch?v=1rorneEGPso
• Star Trek: The Next Generation (1987-1994),
• Data is an android with super-human intelligence but no emotions
• http://www.youtube.com/watch?v=qcqIYccgUdM
35
Human Robot Interaction
• HRI inputs include vision and speech and can…
• Process human-oriented data such as facial expression and gestures
• Understand language between human users and robots
• Apply meaning to the facial and physical expressions and speech
• Real-time perceptive abilities of robots
• Need to perceive, understand, and react to human activity in real-time
• Need to deal with uncertainty in sensing
• Depend on specific lighting conditions
• Can process physiological signals
• Heart rate
• Blood pressure
• Galvanic skin response (the user’s emotional state)
• Mower et al. (predicts when a user is about to quit a
rehabilitation-type task)
• Body pose and movement (sources of information for social
interaction)
• Kapoor and Picard implemented an affect recognition system
• Detects the attention of human users in order to determine if
a user is addressing the robot
• Integrates person tracking, face recognition, sound source
localization, and leg detection.
36
Human Robot Interaction
• Embodiment
• Impact of physical embodiment on social interaction
• Bartneck - the robot was more engaging to the user than a simulated agent
• Anthropomorphism
• Differences in interaction between anthropomorphic and non-anthropomorphic
robots
• Children with autism improve their social interaction skills and imitation
skills when interacting with robots
• https://www.youtube.com/watch?v=Fp8kndY5E84
• Some studies find that humanoid robots are overwhelming and intimidating
• Other studies find that humanoid robots have therapeutic benefit
• Biomimetic, (anthropomorphic form) allow human-like gestures and direct
imitation movements
• Non-biomimetic form preserves the appeal of computers and mechanical
objects.
• Simplicity/Complexity of Robot Design
• Wainer
• The more realistic or complex a robot was, the more watchful it seemed
• Participants were less likely to share personal information with a realistic or
complex robot.
37
Human Robot Interaction
• Other Attributes
• Reeves and Nass
• Users experienced a stronger sense of social presence from the agent
when the voice type and personality matched, than when they did not
• Tapus and Mataric
• When a robot’s expressive personality matched the user’s personality, task
performance was better than when the personalities were mismatched
• Takeuchi
• How cultural norms and customs affect the use of robot systems
• Michio Kaku
http://www.youtube.com/watch?v=hKAkLUd1jFghttps://www.youtube.com/watch?v=JPVOPzYiCeg
38
Human Robot Interaction
• Assistive robots
• Gives aid or support to a human user
• Intended for schools, hospitals, homes
• Socially assistive robotics (SAR)
• PR2 https://www.youtube.com/watch?v=VELzj6Wyg3Y
• Elder care, education, social and cognitive disorders, and rehabilitation
• Post-operative cardiac surgery recovery and Post-stroke rehabilitation
• Rehabilitation
• Motivating rehabilitation through mutual storytelling
• A robot and a user construct a story, which, when acted out, require
the user to perform physical therapy exercises.
• Feeding, brushing teeth, getting in and out of bed, getting into and out of a
wheelchair, and adjusting a bed for maximum comfort
• Envision robots as part of a ubiquitous computing system
• Combines cameras and other sensors in the environment and
computer controlled appliances (such as light switches, doors, and
televisions)
• The Huggable - a robot outfitted with several sensors to detect different
types of touch
• http://www.youtube.com/watch?v=nJXkL7bcQR0
39
Human Robot Interaction
• Paro - an actuated stuffed seal which behaves in response to touch and sound.
• Goal is to provide the benefits of pet-assisted therapy in nursing
homes that cannot support pets
• Studies have shown lowered stress levels in residents interacting with
this robot, as well as an overall increase in the amount of socialization
among residents in the common areas of the same facility.
• http://www.youtube.com/watch?v=N5WEbw7DOMY
• http://www.youtube.com/watch?v=PNw4oicWmWU
• Diagnosis and socialization of children with autism spectrum disorders (ASD).
• Robots can observe children in ways that humans cannot
• Eye-tracking studies have shown remarkable promise when evaluating
children for the purposes of diagnosing ASD.
• Robots are a more comfortable social partner for children with ASD than
people.
• These robots encourage social behavior, such as dancing, singing, and
playing
40
Human Robot Interaction
Educational Robotics
• Goal to develop robot hardware and software to improve education from
elementary school onward
• Traditional robot competitions
• Botball
• FIRST
• RoboCup http://www.youtube.com/watch?v=acrS0kPY-J8
• What need is it filling?
• NAO for special education
• http://www.youtube.com/watch?v=LoqU0igk_qQ
41
Human Robot Interaction
Ethical Issues for HRI
• The technology to support how a robot can make eye contact is comparatively simple
• Evaluating how the person is affected by the robot’s gaze and behavior is difficult.
• Does the person get bored or frustrated?
• Does the user consider the robot helpful and effective?
• Is the robot trusted to perform its intended tasks?
• Challenges
• Unintended uses of the robot
• Unintended situations that might be encountered
• If the person needs emergency attention, what is the robot’s responsibility?
42
Human Robot Interaction
• How safe is the robot itself, and how safe can the robot make life for its user?
• The ability to maneuver about a scene without unwanted contact or collisions.
• Safety also refers to protection (as much as it is possible) of a robot’s user and of the
robot itself.
• CASERO- a robot used in a nursing home
• http://www.youtube.com/watch?v=dx0zxr3D_zU
• Scalability
• How well does a robot perform with users from the general population?
• How many people can be helped by such a robot
• How does speech recognition perform when the speaker has recently suffered a
stroke?
• Can the robot adapt its behavior to different users?
43
Human Robot Interaction
• Social interaction
• The primary means of interaction
• The driving force behind the design
• Autonomy
• Autonomy is not desirable in situations such as medication dispersion and therapy
monitoring
• The user must clearly retain authority
• Rehabilitation should terminate if the user is in pain
• Imitation
•
The Total Turing Test
•
A system communicating in human-like ways (text, speech, facial expressions)
tries to fool a human user into believing it is human
•
A robot’s personality can effect a user’s compliance
•
When exhibiting a serious personality, the robot can provoke a greater degree
of compliance than displaying a playful personality
•
When the robot’s extroversion/introversion personality traits matched the user’s,
task performance was improved
•
The Uncanny Valley theory suggests that as a robot becomes very similar in
appearance to a human, that robot appears less, rather than more, familiar
44
Human Robot Interaction
• Privacy
• In contrast to ubiquitous systems where a user has no idea of when the system may
be watching, robots are tangible and their perception limited and observable.
• A robot can be told to leave when privacy is desired, and the user can observe when
privacy is achieved.
• Because of its synthetic nature, a robot is perceived as less of a privacy invasion than
a person, especially in potentially embarrassing situations.
• Measuring Social Success
• Does the robot successfully achieve the desired social identity?
• When the robot is intended to be playful, do users find the robot playful?
• If the robot is supposed to be a social peer, do users act as if it were a social peer?
• The social identity of the robot (both the personality and the role of the robot) has an
effect on the user’s task performance
45
Human Robot Interaction
• How the Robot Perceive the Person
•
•
•
•
•
•
•
•
•
Emotion recognition
Vocalizations
Speech
Language
Motor acts
Gestures
GSR
Heart Rate
Radio frequency identification (RFID) tags
• Positioned to track and observe children in school hallways to detect when users
were in social range, and who they were interacting with over time to help the
robot determine appropriate social responses
46
Human Robot Interaction
• Nao
• http://www.humarobotics.com/en/robotics-lab/nao-humanoid-robot-plays-connect-4/
47
Human Robot Interaction
 Facebook Facial Recognition
 Security
 Or, hold up a photo
 Mood Recognition
 The API will estimate your mood
automatically, with 5 values: Happy, Sad,
Surprised, Angry, and Neutral.
48
Human Robot Interaction
• Success Relative to Human Caregiver
• How does the robot perform relative to a human performing the same task?
• In rehabilitation tasks, measure functional improvement
• Overall learning measures such as grades, tests, or evaluations can be used.
• Spirometry task robot instructs a cardiac surgery patient to do breathing exercises
compliance with a human was a suitable can be measured.
• A key role of assistive HRI is to provide care where human care is not available.
• Cost/Benefit Analysis
• Rehabilitation tasks where a robot could serve as a therapist
• Giving advice on specific movements
• Motivational coach
• Monitoring progress
• Learning aid, or as a demonstration, showing a user how to do specific exercises.
• Low-cost robot vacuum cleaners
• Some treated the robot as if it were a member of the household, with status
roughly between the vacuum cleaner and a pet.
• Others treated it strictly as a device with a purpose.
• Does the use of the robot change the cost/benefit ratio of providing such care?
• Doe it make such care available where it was not previously possible?
49
Human Robot Interaction
• Impact on Caregivers In some cases, the goal is to make the user’s or caregivers’ job
easier and more manageable.
• By automating the prompting and monitoring of spirometry, which must be performed
ten times per hour for the critical post-surgery period, the robot made it possible for
caregivers to attend to other tasks and provide more individualized services.
• Does the job condition of the caregiver improve as a result of the robot?
• How well does the caregiver work with the robot?
50
Human Robot Interaction
• Existing Quality of Life Measures
• Dementia Care Mapping (DSM)
• Robots to observe and measure socially-sensitive care
• Eye-contact
• Favorable attention
• Does the robot result in a general increase in the quality of life as perceived by the
user?
• Impact on the Role in Community/Society
• When fish tanks were introduced into a nursing home environment to test the effects on
residents, observers found an overall increase in nutrition on the part of the
participating residents
• The introduction of new objects of social interest into an environment can thus change
the dynamics of the community.
51
Human Robot Interaction
• Paro
• Reduction of stress proteins in the urine of the participants.
• Residents were in the common areas longer and socialized more.
• Robovie
• Used a robot to stimulate social interaction among a group of elementary school
students
• The robot was able to elevate a student’s status in the group by giving him/her
special information
http://www.youtube.com/watch
?v=RuZFZay7XWU
• Does the robot increase or decrease the amount of socialization in its user community?
• Are changes in community due to a robot positive or negative?
52
Human Robot Interaction
 LEGO MINDSTORMS EV3
 http://www.youtube.com/watch?v=Unjiv0oBcV0
 LabVIEW
• http://www.youtube.com/watch?v=I8pc8-VcVFo
 Most robotic programming language are proprietary or for industrialized
robots. ROS possibility an exception.
53
Human Robot Interaction
 Snake Robotics
• User control and applications of the technology
• https://www.youtube.com/watch?v=XYiodXEZopk