Download AAC for Elderly People with Hearing, Speech or Memory Loss due

AAC for Elderly People with Hearing,
Speech or Memory Loss due to
Aging
Lawrence Chung, Shinyoung Lim,
Yoosun Chung, Rutvij Y. Mehta, Amit B. Chembra
25th Annual International Technology & Persons with
Disabilities Conference, San Diego, March 2010
© 2010
1
Acknowledgements
•
‘SymbolStix Inc.’
• Students that participated in the prototype implementation
Abdullah Azzouni, Bharat Paturi, Manoj Kumar, Michael Fashola, Nisha Mathew,
Rakesh Sanapala, Rohit Sharma, Sandeep Sistla , Sritej Vepa, Vamshi Bandaru,
Vidya Vijayan , Vinay Kumar
• For their valuable comments & feedback (Research group)
Ebenezer Oladimeji, Sam Supakkul and Tom Hill (a special thanks for the Nexus!)
25th Annual International Technology & Persons with
Disabilities Conference, San Diego, March 2010
© 2010
2
Outline
Introduction
Goal
Motivation
User Needs of AAC Service
Design of a prototype system
Our models – Olive* , H.O.P.E
Features and Demo
Enhancements
Project in a Nutshell
Conclusion
25th Annual International Technology & Persons with
Disabilities Conference, San Diego, March 2010
© 2010
3
Introduction
• ‘Science Blog’ – Reduced Hearing more common
and hearing aids inadequate for elderly
• AAC focused on Autism, rarely on elderly people
• Treatment focused on medical/surgical procedure
not rehabilitative approaches
• Communication barriers of elderly not researched
and existing measures not tailored to their needs
• Current technology – No concept exploration
• Population of the elderly to be significant soon
• Need for an assistive system ever more than before
25th Annual International Technology & Persons with
Disabilities Conference, San Diego, March 2010
© 2010
4
Goal
•
•
•
•
Identify – Communication difficulty is a problem
Understand user needs for an assistive system
Eradicate the need for bulky, costly instruments
Develop a communication system to overcome
Hearing loss, Memory loss, Speech Disorder,
Weak Vision
• Use existing hardware (e.g. Cell phones) - cut cost
• System that can be ‘personalized’ for user needs
• A ‘mobile’, ‘assistive’ and ‘cost-effective’ system
that helps reduce the user’s communication
barriers
25th Annual International Technology & Persons with
Disabilities Conference, San Diego, March 2010
© 2010
A
S
I
S
T
O
B
E
5
Communication difficulty - A problem ?
•
•
•
•
•
•
•
•
Elderly are concerned over perception of others
Lack of communication decreases their self reliance
Due to communication barriers, they limit their interactions
Lose confidence and remain confined
Fear getting embarrassed, so do not communicate
Give up their interests and live a neglected life
Psychological problems may arise
Problems and Behavior can be identified in day to day events
or scenarios
• Such problems are preventable – just have to fill the gaps
25th Annual International Technology & Persons with
Disabilities Conference, San Diego, March 2010
© 2010
6
Motivation (The Why question)
• Scenario (Family Dinner Outing)
- Elderly parents
- Loss of hearing, Speech impairment and Memory loss
- Not ready to publicly accept their age-related disorders
- Do not welcome use of external medical aids
- Human perception, memory and communication are
visual not textual
- We function best when all senses complement each other
- Similar scenarios repeat in other forms every day
25th Annual International Technology & Persons with
Disabilities Conference, San Diego, March 2010
© 2010
7
A visual scenario (Family Dinner Outing)
Son: Mom and Dad, Get
Ready. We are going to
the Chinese restaurant
we went last time for a
family get together.
Old parents
Son
Parents: Sure
Entire family gets in the
car and leave for the
restaurant
25th Annual International Technology & Persons with
Disabilities Conference, San Diego, March 2010
© 2010
8
In the car – On the way
Mom: Son, where are
we going ?
Son: I told you – the
Chinese restaurant
25th Annual International Technology & Persons with
Disabilities Conference, San Diego, March 2010
© 2010
9
In the car – On the way ( A few miles later …)
Mom: Son, where are
we going ?
Son: (In an irritated
tone) I have already told
you – the Chinese
restaurant !!
Son: (Realizing) Mom –
Are you wearing your
hearing aid ?
Mom: (Does not have it
on) Hmm.. Yes..
Ah..Here I have it….
25th Annual International Technology & Persons with
Disabilities Conference, San Diego, March 2010
LOSS OF HEARING !
© 2010
10
In the car – ( Almost near the restaurant …)
Mom: (To Dad) – If
Steve comes and
bows, then
acknowledge him. Last
time he met, you did
not talk nor gave a
smile to him
Dad: (STEVE ??? Has a
clueless expression) OK
MEMORY
LOSS !!
25th Annual International Technology & Persons with
Disabilities Conference, San Diego, March 2010
© 2010
11
Outside the restaurant …
Steve walks and
greets Dad.
Dad fails to recognize
him !!!
Steve – at
present
Mom: (To Dad) – Why
did you not greet
Steve ? I told you to
greet him !!!
Mom: You remember
Steve – our daughter’s
son ! Remember how
cute he was when he
was a small boy !
Dad: (Apparently still
clueless – trying to
recollect) Ah..OK
Dad: (Embarrassed) I
could not recognize
him !
MEMORY
LOSS !!
Steve as a boy
25th Annual International Technology & Persons with
Disabilities Conference, San Diego, March 2010
© 2010
12
Inside the restaurant …
Mom instructs her
grandson to eat
carefully without
spilling
Grandson: Dad – What
did Grandma say? I
cannot understand
what she murmurs….
Grandson
Son: Mom – Are you
wearing your
dentures?
Unclear/Incomprehensible
Speech !!
25th Annual International Technology & Persons with
Disabilities Conference, San Diego, March 2010
© 2010
13
User Needs of AAC Service
• Categories of Difficulties/Communication Barriers for Elderly
- Unclear Utterance , often leading the elderly to repeat and
hard for the listeners to comprehend
- Incomplete/Incomprehensible speech: Speech often times
nonsensical, difficult to express thoughts
- Listening Inability: Hard of hearing especially mobile
communications (Hearing devices – voice-to-text
translation could be introduced)
25th Annual International Technology & Persons with
Disabilities Conference, San Diego, March 2010
© 2010
14
User Needs of AAC Service
- Comprehension Inability: Cannot grasp or understand what
others have to say (Cognitive impairment)
- Communication issues due to memory loss: Cannot
remember names of people, places, activities, topic
- Communication in emergency: Should be able to make an
emergency call (say, 911), especially those having medical
problems or living alone (Voice to text/Voice to image relay)
25th Annual International Technology & Persons with
Disabilities Conference, San Diego, March 2010
© 2010
15
Design of the prototype system
• Target 3 language acquisition mechanisms by icons/photos
1) See what you cannot hear
- Overcome loss of hearing
2) Show what you cannot speak
- Overcome unclear speech (Text-to-speech)
3) See what you cannot remember
- Overcome memory loss
25th Annual International Technology & Persons with
Disabilities Conference, San Diego, March 2010
© 2010
16
Design of the prototype system
• Key technical concepts of the system:
- Vocabulary : To show visual images, for words heard and to
to be spoken, transformation to address complex scenarios
- Navigation: Identify the scenario/context and allow quick
and easy navigation based on activities of daily living
- Learning: Detecting and building patterns of frequent
vocabulary use in daily living – System adaptability.
25th Annual International Technology & Persons with
Disabilities Conference, San Diego, March 2010
© 2010
17
Design of the prototype system
• Development Focus:
Ontology: Philosophical study of existence (source: Wikipedia)
Things, relationships and constraints on them
Focus towards ontology-driven context awareness in User
Interface and services
Vocabulary : 5 basic entities
TEXT, ICONS, SOUND
SPEECH & OBJECT RECOGNITION
Target the important sensory activities of SPEECH,
HEARING, VISION and THOUGHT process
All activities available to the user based on the 5 entities
25th Annual International Technology & Persons with
Disabilities Conference, San Diego, March 2010
© 2010
18
OLIVE – The first attempt
O.L.I.V.E – Older Living Independently
• Started as a independent study project
• AAC research for the elderly – first step
• Implementation on Samsung Instinct
Phone
• Work on ‘Search’ and ‘Teach’ functions
25th Annual International Technology & Persons with
Disabilities Conference, San Diego, March 2010
© 2010
19
HOPE
H.O.P.E – Helping Older People Easily
• Research ongoing since 2009
• Built on Android Platform
G1, Nexus Phone
• Increased the vocabulary from
Search and Teach to the
5 ontological primitives
25th Annual International Technology & Persons with
Disabilities Conference, San Diego, March 2010
© 2010
20
HOPE
H.O.P.E – Helping Older People Easily
• Main categories
- Greetings
- Daily Living Activities
- Emergency situations
- Talk (Build sentences)
- Real World Scenarios
- Active Mind (Games)
25th Annual International Technology & Persons with
Disabilities Conference, San Diego, March 2010
© 2010
21
HOPE
H.O.P.E – Helping Older People Easily
• Menu categories
Speech recognition used in
- Tagging images (assigning a
name to your images for a photo
album – to be extended for all
images in the system)
- Retrieving an image based on
voice/text input
25th Annual International Technology & Persons with
Disabilities Conference, San Diego, March 2010
© 2010
22
HOPE
Search
Greetings
25th Annual International Technology & Persons with
Disabilities Conference, San Diego, March 2010
© 2010
23
HOPE
Playback Feature
Emergency
Daily Living Activities
25th Annual International Technology & Persons with
Disabilities Conference, San Diego, March 2010
© 2010
24
HOPE
All by Yourself ?
Nothing to talk ?
No problem…..
Special added feature :
Active Mind (Memory games)
25th Annual International Technology & Persons with
Disabilities Conference, San Diego, March 2010
© 2010
25
HOPE
H.O.P.E – Helping Older People Easily
• Prominent Points:
- Not feature oriented but scenario oriented
- Use contextual mechanism to minimize
number of clicks
- Speech recognition to make it not only AAC
but also assistive
- Activities revolve around 5 vocabulary entities
- Add on game for Active Mind, can be used when not
actively communicating
25th Annual International Technology & Persons with
Disabilities Conference, San Diego, March 2010
© 2010
26
HOPE (Demo)
25th Annual International Technology & Persons with
Disabilities Conference, San Diego, March 2010
© 2010
27
HOPE
H.O.P.E – Helping Older People Easily
• Enhancements:
- Add more scenarios to reduce effort of the end user in
creating statements
- Extend the tag/recognize feature to associate a name with
any image of the system instead of the photo album
- Add more user configuration options like zooming of icons
- Add more vocabulary primitives along the 5 organizational
entities
- Cater application to a larger target user population
25th Annual International Technology & Persons with
Disabilities Conference, San Diego, March 2010
© 2010
28
HOPE
H.O.P.E – Helping Older People Easily
• Enhancements:
- Explore the ‘Contextual’ improvement
- Use the ‘minimum clicks’ approach
- A sample contextual mechanism for Daily Living Activities
inspired from the vocabulary of the Maya civilization
Revolves around the activity with only relevant contextual
icons surrounding it
Final sentence is the combination of the activity and the
context icons e.g. Take Bath + ( Cold or Hot Water)
25th Annual International Technology & Persons with
Disabilities Conference, San Diego, March 2010
© 2010
29
Project in a Nutshell
•
•
•
•
•
•
Feasibility Study involved identifying communication
barriers and user needs
Developed service requirements and design concepts for
our prototypical system
Still difficult to draw solid tangible requirements from user
needs due to complexity and individuality in communication
barriers
On going project incremental – evolutionary and agile
methodology
Many issues – what to represent, how to represent in the
best effective manner.
Growth depends on more learning, discovery
25th Annual International Technology & Persons with
Disabilities Conference, San Diego, March 2010
© 2010
30
Conclusion
• Population of elderly to grow – so is the need and urgency
for countering communication difficulties
• Issue has been studied, user needs and concerns identified
• Design of the system based on 5 types of vocabulary
• Tiny AAC software implemented – H.O.P.E
• More investigation on contextual vocabulary and
representing core vocabulary
• Develop a ‘minimum clicks – maximum vocabulary’
approach
• Extend system by adding features for a larger audience
• Make a change – Bridge the gap due to communication
barriers
25th Annual International Technology & Persons with
Disabilities Conference, San Diego, March 2010
© 2010
31
Questions ?
Contact:
Rutvij Mehta
Email : [email protected]
25th Annual International Technology & Persons with
Disabilities Conference, San Diego, March 2010
© 2010
32