Download Physical User Interfaces

Physical User Interfaces
What they are and how to build them
Saul Greenberg
University of Calgary
Physical User Interfaces
Special purpose computer-controlled devices that
can be situated in a real-world setting.
• typically designed for particular contexts and uses.
Physical User Interfaces
Foundations for
• Ubiquitous Computing
• Tangible Media
• Context aware computing
Styles of use
•
•
•
•
•
foreground interaction
ambient displays
information appliance
collaborative interaction
physical controls
Ubiquitous Computing
• A less-traveled path I call the invisible; its highest
ideal is to make a computer so imbedded, so fitting,
so natural, that we use it without even thinking
about it.
• Provide hundreds of wireless computing devices per
person per office, of all scales … It is invisible,
everywhere computing that does not live on a
personal device of any sort, but is in the woodwork
everywhere.
Source: Mark Weiser’s UbiqCom web site
Mark Weiser
Xerox PARC
Ubiquitous Computing
Invisible Everywhere Computing
• invisible:
o tiny, embedded, attachable…
• everywhere:
o wireless, dynamically configurable,
remote access, adapting…
• aka Pervasive Computing
Source: Mark Weiser’s UbiqCom web site
Mark Weiser
Xerox PARC
Ubicomp is Situated Computing
Makes use of simple shared
context
•
•
•
•
space
time
proximity
affordances
Source: Mark Weiser’s UbiqCom web site
Participation in the context
• is physical
• is out here with us
• is in many small and large
places, including trivial ones
Ubicomp Technology Trends
Source: Mark Weiser’s UbiqCom web site
Ubicomp Technology Trends
Processors:
• cheap, small, dedicated, microprocessors
Low Power
• small batteries, solar (?)
Wireless
• 802.11, Bluetooth, infrared, mobile telephony, …
Displays
• very small (inches) to very large (walls)
Peripherals
• sensors, actuators, motors, …
Run-time systems
• Linux on a chip, Windows CE, downloadable executables…
Source: Mark Weiser’s UbiqCom web site
ParcTab
Mobile hardware
• infrared
• room-sized cells
• location information
A portable GUI device
•
•
•
•
•
•
small case with belt clip, ½ size of PDAs
touch sensitive 128x64 pixels display
3 finger-operated mechanical buttons (chorded)
piezo-electric speaker
low power needs (~ 1 week between charges )
can be used in either hand by rotating display
Source: Mark Weiser’s UbiqCom web site
Tangible User Interfaces
From ‘painted bits’ to ‘tangible bits’
• genre change
• give physical form to digital information
• physical objects, surfaces, and spaces become
tangible embodiments of digital information.
Hiroshi Ishii
MIT
seamlessly couple the dual worlds of bits and atoms
• input: grasp and manipulate
• output: change of physical properties of object
Source: Tangible Media Group web site
Context-Aware Computing
Context as information
• … characterizes a situation of a person, place or object relevant
to the interaction between a user and an application
o
o
o
o
location
identity
state and activites of people, groups
state of computational and physical objects
Context-aware computing
• uses contextual information to
o selectively present information and services
o automatically execute a service
o attach context information for later retrieval
Source: Dey, Abowd and Salber, HCI Journal 2001
Styles of use
Foreground interactions
Conscious intentional interactions
• graspable objects
• augmented surfaces
• link between physical and virtual actions
• exploit human senses of touch and kinesthesia.
MusicalBottles - controlling sound, light
Interaction
One opens and closes bottles
to control digital contents e.g.,
opening it tells a story
Music bottles
movement and uncorking of
the bottles controls the
different sound tracks and the
patterns of colored light rearprojected onto the table’s
translucent surface.
Source: Tangible Media Group web site
MusicalBottles - controlling sound, light
Source: Tangible Media Group web site
Ishii, MIT
IAMASCOPE
Wooden Mirror - wooden pixels, image
Daniel Rozin, NYU Ishii, MIT
Wooden Mirror - wooden pixels, image
Ishii, MIT
Daniel Rozin, NYU
Triangles
– building a digital story
When triangles connect together,
they trigger digital events.
These events influence the progress
of a non-linear story, or allow users
to organize media elements in order
to create their own story space.
Extracted from Tangible Media Group web site
Triangles
– building a digital story
Extracted from Tangible Media Group web site
Ambient displays
Background information
• communicate digitally-mediated senses of activity and presence
at the periphery of human awareness.
• ambient light sound, airflow, water movement, object motion…
• peripheral displays
Extracted from Mark Weiser’s UbiqCom web site
Extracted from Mark Weiser’s UbiqCom web site
Pinwheels -
ambient digital wind
Extracted from Tangible Media Group web site
Pinwheels -
ambient digital wind
Extracted from Tangible Media Group web site
Information Perculator
bubbles of digital patterns
Heiner, Hudson & Tanaka
Information Perculator
bubbles of digital patterns
Heiner, Hudson & Tanaka
Ambient Light Display
light reflection from water onto
ceiling
Ambient Room
ACM CHI’98, Tangible Media Group
Personal Ambient Display
Small, physical devices worn to
display information to a person
in a subtle, persistent, and
private manner.
Ambient information is displayed
solely through tactile modalities
such as heating and cooling,
movement and vibration, and
change of shape.
Extracted from Tangible Media Group web site
Physical controls
Physically-mediated computer controlled
interactivity
Marble Answering Machine
Incoming voice messages are
physically instantiated as
marbles.
The user can grasp the message
(marble) and drop it into an
indentation in the machine to
play the message.
The user can also place the
marble onto an augmented
telephone, thus dialing the caller
automatically.
Durrell Bishop
TouchCounters
Computational tags track the
usage of physical objects.
TouchCounters sense activity
through magnetic, acceleration,
and infrared sensors, and
indicate their status on bright
LED displays.
TouchCounters can be networked
to a web server that generates
use histograms for each object.
Extracted from Tangible Media Group web site
TouchCounters
Extracted from Tangible Media Group web site
Tagged Objects
ACM CHI’99, Xerox Parc
mediaBlocks
Extracted from Tangible Media Group web site
SenseTable
A system for tracking the positions and
states of multiple objects wirelessly on a
flat surface.
Objects can be equipped with various
controls -- dials or buttons -- which can be
monitored in real-time.
When coupled with a projector, the system
can display information about the objects
on or near the objects themselves.
Extracted from Tangible Media Group web site
HandScape
A vectorizing digital tape
measure for digitizing field
measurements, and visualizing
the volume of the resulting
vectors with computer
graphics.
Using embedded orientationsensing hardware, it captures
relevant vectors on each linear
measurements and transmits
this data wirelessly to a
remote computer in real-time.
Extracted from Tangible Media Group web site
Manipulative User Interfaces
Customizable Physical Interfaces
ACM UIST 2002, Greenberg & Boyle
Collaborative interactions
Physical objects connecting collaborators
• collaborative interaction
• awareness
Adapted from Tangible Media Group web site
PingPong Plus
features a "reactive table" that
incorporates sensing, sound, and
projection technologies.
Projectors display patterns of
light and shadow on the table;
bouncing balls leave images of
rippling water; and the rhythm of
play drives accompanying music
and visuals.
Extracted from Tangible Media Group web site
PingPong Plus
Extracted from Tangible Media Group web site
InTouch – collaborative haptics
Force-feedback technology is
employed to create the illusion
that people, separated by
distance, are interacting with a
shared physical object.
When one of the rollers is
rotated, the corresponding
roller on the other distant
object rotates in the same
way. Two people separated by
distance can then play…
Extracted from Tangible Media Group web site
InTouch – collaborative haptics
Extracted from Tangible Media Group web site
Bench
…two cold steel benches located in
different cities.
When a person sits on one of these
benches, a corresponding position on
the other bench warms, and a bidirectional sound channel is opened.
At the other location, after feeling the
bench for "body heat," another
person can decide to make contact
by sitting near the warmth.
Initially the sound channel is
distorted, but as the second party
lingers, the audio channel clears.
--summarized by Ishii and Ullmer
Anthony Dunne and Fiona Raby at the RCA
Physical but Digital Surrogates
Proximity detector
ultrasonic sensor
Hydra unit
video, camera,
speakers, microphone
Rotating figurine
servo motor
Tippable figurine
light sensors
Physical but Digital Surrogates
Privacy preserving media space
Roomware
i-land
Computer-augmented room elements
• like doors, walls, furniture (e.g. tables and chairs) with integrated information
and communication technology.
From the GMD Darmstadt
web site on I-Land
Roomware
i-land
Dynawall
From the GMD Darmstadt web site on I-Land
Roomware
i-land
CommChair
From the GMD Darmstadt web site on I-Land
Roomware
i-land
ConnecTable
By moving multiple
ConnecTables together, they can be
arranged to form a large display area.
Integrated sensors measure the distance
between the ConnecTables and initiate
the automatic coupling of the displays
From the GMD Darmstadt web site on I-Land
Roomware
i-land
InteracTable
•touch-sensitive plasma-display (PDP) is integrated into the table top
•Border for leaning…
From the GMD Darmstadt web site on I-Land
Roomware
i-land
Phillips –
Intelligent Information Surfaces
Tokens
From the Philips Lime Video CD
Phillips –
Intelligent Information Surfaces
From the Philips Lime Video CD
Designing out of the box
The problem:
• programming / designing with physical devices is hard
o
o
o
o
o
o
o
circuit design (electrical engineering)
microprocessor interface to digital/analog devices
‘wire’ interface (serial, USB, wireless, IR…)
wire protocol
connection/disconnection/intermittent connectivity
software to use devices
maintenance and extensibility
• simple things take a long time to do
• most people don’t bother
Solution 1: Interdisciplinary team
Works, but…
Proximity detector
ultrasonic sensor
• Still takes time
• When one of the team
leaves, knowledge is
lost
• systems hard to
maintain &
extend
Hydra unit
video, camera,
speakers, microphone
Rotating figurine
servo motor
Tippable figurine
light sensors
Solution 2: Hack existing devices
Programmable Embodied
Agents (Kaminsky et al)
• hacked Microsoft Actimates
Solution 2: Hack existing devices
Programmable Embodied
Agents (Kaminsky et al)
•
•
•
Arm position -> quantity
Squeezing hand/leg -> counting
Movement-> task progress
Proxy for other person

squeeze hand, other’s hand goes up)
Event monitoring

Signal document is printing, then complete

Barney Email biff
Solution 3: Phidgets
Physical Widgets
• simple, easy to program devices and component-based software
with well-defined API
• building blocks for physical interfaces
• analogous to GUI widgets
Phidget Examples
PhidgetServo:
• Control 1 or 2 servo motors
PhidgetPowerbar
• Control power state of outlets on a power bar
PhidgetInterfaceKit
• 8 simple input and outputs plus 2 sensors
• A constructor kit
PhidgetProximitySensor
• Returns how close something is to it
PhidgetMotionDetector
• Periodically returns the amount of motion in a space
Related areas
Mobile Computing
Augmented Reality
Context-aware computing
Reactive Environments
Ubiquitous Media
Cooperative Buildings
Digital Inputs - switches
Rocker
Rotary
Toggle
Slide
Push button
Tactile
Push-Pull
Keylock
Analog inputs: Environmental Sensors
Distance
Light
Temperature
Pressure
Motion
rangefinder
proximity
Voltage
Weight
Distance
Analog inputs: Input Sensors
Force
single-turn
Mini-joystick
multi-turn
Capacitive
encoder
Potentiometers
Accelerometer
slider
Analog inputs: Input Sensors
Bend
Force
Camera
Tilt
Custom Input: Identification
RFID Tags and Antenna
Bar Code Scanner
Custom inputs
Camera
Digital Outputs – low power
lamps
LEDs
Lights
Relays
Solenoids
Output: Motors
Position: 0-180o
Servo
Speed
DC Motor
Rotate by steps: +/- xo
Stepper
Output: Displays
Text LCD
Numeric-alpha
Graphics
(not yet)
Approaches: PIC Micro-controller
Single programmable chip computer with:
•
CPU, RAM, ROM, I/O, serial/parallel ports, A/D and D/A converters
Need to know:
• basic circuit design
• basic electronics
o resistor, capacitor,
diodes, transistors…
• low level programming
• PIC hardware details
• …
Powerful, flexible, but
•
•
•
•
learning curve
excessive time in low level details
complex things are hard
serial
e.g. www.voti.nl/swp
Approaches: Basic Stamp
Pre-built boards
• Pic microcontroler
• pre-wired circuits and connectors
• boards designed for different uses
Still need to know
• electronic components
• electronic circuitry
• PBasic language: stamp-specific
instruction set
Still flexible, but
•
•
•
•
learning curve still there
time in low level details
complex things are hard
serial
Approaches: HW as SW components
Phidgets and Making Things
•
•
•
•
dedicated devices
some plug and play electronics
under direct computer control
well-defined component-based software
o Interface via APIs, objects, and/or widgets
Need to know
• conceptual knowledge of device
• high level programming language
• software component documentation
High design flexibility, low electronic flexibility
• very low learning curve
• design by combining and varying
• time in conceptual design, not electronics
Approaches: Programmable Brick
Lego Mindstorms
•
•
•
•
•
robotics (downloadable code)
computer control as well
proprietary RCX microcontroller
infrared communication
3 input ports
www.mastincrosbie.com/mark/
lego/images/grabberarm1.jpg
o light/touch sensors…
• 3 output ports
o motors, lights…
• Lego building blocks
• children’s programming language for downloading but
o well-defined SDK
o 3rd party access from standard languages
Need to know
• SDK / language
e.g. http://www.crynwr.com/lego-robotics/