Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
AAC for Elderly People with Hearing, Speech or Memory Loss due to Aging Lawrence Chung, Shinyoung Lim, Yoosun Chung, Rutvij Y. Mehta, Amit B. Chembra 25th Annual International Technology & Persons with Disabilities Conference, San Diego, March 2010 © 2010 1 Acknowledgements • ‘SymbolStix Inc.’ • Students that participated in the prototype implementation Abdullah Azzouni, Bharat Paturi, Manoj Kumar, Michael Fashola, Nisha Mathew, Rakesh Sanapala, Rohit Sharma, Sandeep Sistla , Sritej Vepa, Vamshi Bandaru, Vidya Vijayan , Vinay Kumar • For their valuable comments & feedback (Research group) Ebenezer Oladimeji, Sam Supakkul and Tom Hill (a special thanks for the Nexus!) 25th Annual International Technology & Persons with Disabilities Conference, San Diego, March 2010 © 2010 2 Outline Introduction Goal Motivation User Needs of AAC Service Design of a prototype system Our models – Olive* , H.O.P.E Features and Demo Enhancements Project in a Nutshell Conclusion 25th Annual International Technology & Persons with Disabilities Conference, San Diego, March 2010 © 2010 3 Introduction • ‘Science Blog’ – Reduced Hearing more common and hearing aids inadequate for elderly • AAC focused on Autism, rarely on elderly people • Treatment focused on medical/surgical procedure not rehabilitative approaches • Communication barriers of elderly not researched and existing measures not tailored to their needs • Current technology – No concept exploration • Population of the elderly to be significant soon • Need for an assistive system ever more than before 25th Annual International Technology & Persons with Disabilities Conference, San Diego, March 2010 © 2010 4 Goal • • • • Identify – Communication difficulty is a problem Understand user needs for an assistive system Eradicate the need for bulky, costly instruments Develop a communication system to overcome Hearing loss, Memory loss, Speech Disorder, Weak Vision • Use existing hardware (e.g. Cell phones) - cut cost • System that can be ‘personalized’ for user needs • A ‘mobile’, ‘assistive’ and ‘cost-effective’ system that helps reduce the user’s communication barriers 25th Annual International Technology & Persons with Disabilities Conference, San Diego, March 2010 © 2010 A S I S T O B E 5 Communication difficulty - A problem ? • • • • • • • • Elderly are concerned over perception of others Lack of communication decreases their self reliance Due to communication barriers, they limit their interactions Lose confidence and remain confined Fear getting embarrassed, so do not communicate Give up their interests and live a neglected life Psychological problems may arise Problems and Behavior can be identified in day to day events or scenarios • Such problems are preventable – just have to fill the gaps 25th Annual International Technology & Persons with Disabilities Conference, San Diego, March 2010 © 2010 6 Motivation (The Why question) • Scenario (Family Dinner Outing) - Elderly parents - Loss of hearing, Speech impairment and Memory loss - Not ready to publicly accept their age-related disorders - Do not welcome use of external medical aids - Human perception, memory and communication are visual not textual - We function best when all senses complement each other - Similar scenarios repeat in other forms every day 25th Annual International Technology & Persons with Disabilities Conference, San Diego, March 2010 © 2010 7 A visual scenario (Family Dinner Outing) Son: Mom and Dad, Get Ready. We are going to the Chinese restaurant we went last time for a family get together. Old parents Son Parents: Sure Entire family gets in the car and leave for the restaurant 25th Annual International Technology & Persons with Disabilities Conference, San Diego, March 2010 © 2010 8 In the car – On the way Mom: Son, where are we going ? Son: I told you – the Chinese restaurant 25th Annual International Technology & Persons with Disabilities Conference, San Diego, March 2010 © 2010 9 In the car – On the way ( A few miles later …) Mom: Son, where are we going ? Son: (In an irritated tone) I have already told you – the Chinese restaurant !! Son: (Realizing) Mom – Are you wearing your hearing aid ? Mom: (Does not have it on) Hmm.. Yes.. Ah..Here I have it…. 25th Annual International Technology & Persons with Disabilities Conference, San Diego, March 2010 LOSS OF HEARING ! © 2010 10 In the car – ( Almost near the restaurant …) Mom: (To Dad) – If Steve comes and bows, then acknowledge him. Last time he met, you did not talk nor gave a smile to him Dad: (STEVE ??? Has a clueless expression) OK MEMORY LOSS !! 25th Annual International Technology & Persons with Disabilities Conference, San Diego, March 2010 © 2010 11 Outside the restaurant … Steve walks and greets Dad. Dad fails to recognize him !!! Steve – at present Mom: (To Dad) – Why did you not greet Steve ? I told you to greet him !!! Mom: You remember Steve – our daughter’s son ! Remember how cute he was when he was a small boy ! Dad: (Apparently still clueless – trying to recollect) Ah..OK Dad: (Embarrassed) I could not recognize him ! MEMORY LOSS !! Steve as a boy 25th Annual International Technology & Persons with Disabilities Conference, San Diego, March 2010 © 2010 12 Inside the restaurant … Mom instructs her grandson to eat carefully without spilling Grandson: Dad – What did Grandma say? I cannot understand what she murmurs…. Grandson Son: Mom – Are you wearing your dentures? Unclear/Incomprehensible Speech !! 25th Annual International Technology & Persons with Disabilities Conference, San Diego, March 2010 © 2010 13 User Needs of AAC Service • Categories of Difficulties/Communication Barriers for Elderly - Unclear Utterance , often leading the elderly to repeat and hard for the listeners to comprehend - Incomplete/Incomprehensible speech: Speech often times nonsensical, difficult to express thoughts - Listening Inability: Hard of hearing especially mobile communications (Hearing devices – voice-to-text translation could be introduced) 25th Annual International Technology & Persons with Disabilities Conference, San Diego, March 2010 © 2010 14 User Needs of AAC Service - Comprehension Inability: Cannot grasp or understand what others have to say (Cognitive impairment) - Communication issues due to memory loss: Cannot remember names of people, places, activities, topic - Communication in emergency: Should be able to make an emergency call (say, 911), especially those having medical problems or living alone (Voice to text/Voice to image relay) 25th Annual International Technology & Persons with Disabilities Conference, San Diego, March 2010 © 2010 15 Design of the prototype system • Target 3 language acquisition mechanisms by icons/photos 1) See what you cannot hear - Overcome loss of hearing 2) Show what you cannot speak - Overcome unclear speech (Text-to-speech) 3) See what you cannot remember - Overcome memory loss 25th Annual International Technology & Persons with Disabilities Conference, San Diego, March 2010 © 2010 16 Design of the prototype system • Key technical concepts of the system: - Vocabulary : To show visual images, for words heard and to to be spoken, transformation to address complex scenarios - Navigation: Identify the scenario/context and allow quick and easy navigation based on activities of daily living - Learning: Detecting and building patterns of frequent vocabulary use in daily living – System adaptability. 25th Annual International Technology & Persons with Disabilities Conference, San Diego, March 2010 © 2010 17 Design of the prototype system • Development Focus: Ontology: Philosophical study of existence (source: Wikipedia) Things, relationships and constraints on them Focus towards ontology-driven context awareness in User Interface and services Vocabulary : 5 basic entities TEXT, ICONS, SOUND SPEECH & OBJECT RECOGNITION Target the important sensory activities of SPEECH, HEARING, VISION and THOUGHT process All activities available to the user based on the 5 entities 25th Annual International Technology & Persons with Disabilities Conference, San Diego, March 2010 © 2010 18 OLIVE – The first attempt O.L.I.V.E – Older Living Independently • Started as a independent study project • AAC research for the elderly – first step • Implementation on Samsung Instinct Phone • Work on ‘Search’ and ‘Teach’ functions 25th Annual International Technology & Persons with Disabilities Conference, San Diego, March 2010 © 2010 19 HOPE H.O.P.E – Helping Older People Easily • Research ongoing since 2009 • Built on Android Platform G1, Nexus Phone • Increased the vocabulary from Search and Teach to the 5 ontological primitives 25th Annual International Technology & Persons with Disabilities Conference, San Diego, March 2010 © 2010 20 HOPE H.O.P.E – Helping Older People Easily • Main categories - Greetings - Daily Living Activities - Emergency situations - Talk (Build sentences) - Real World Scenarios - Active Mind (Games) 25th Annual International Technology & Persons with Disabilities Conference, San Diego, March 2010 © 2010 21 HOPE H.O.P.E – Helping Older People Easily • Menu categories Speech recognition used in - Tagging images (assigning a name to your images for a photo album – to be extended for all images in the system) - Retrieving an image based on voice/text input 25th Annual International Technology & Persons with Disabilities Conference, San Diego, March 2010 © 2010 22 HOPE Search Greetings 25th Annual International Technology & Persons with Disabilities Conference, San Diego, March 2010 © 2010 23 HOPE Playback Feature Emergency Daily Living Activities 25th Annual International Technology & Persons with Disabilities Conference, San Diego, March 2010 © 2010 24 HOPE All by Yourself ? Nothing to talk ? No problem….. Special added feature : Active Mind (Memory games) 25th Annual International Technology & Persons with Disabilities Conference, San Diego, March 2010 © 2010 25 HOPE H.O.P.E – Helping Older People Easily • Prominent Points: - Not feature oriented but scenario oriented - Use contextual mechanism to minimize number of clicks - Speech recognition to make it not only AAC but also assistive - Activities revolve around 5 vocabulary entities - Add on game for Active Mind, can be used when not actively communicating 25th Annual International Technology & Persons with Disabilities Conference, San Diego, March 2010 © 2010 26 HOPE (Demo) 25th Annual International Technology & Persons with Disabilities Conference, San Diego, March 2010 © 2010 27 HOPE H.O.P.E – Helping Older People Easily • Enhancements: - Add more scenarios to reduce effort of the end user in creating statements - Extend the tag/recognize feature to associate a name with any image of the system instead of the photo album - Add more user configuration options like zooming of icons - Add more vocabulary primitives along the 5 organizational entities - Cater application to a larger target user population 25th Annual International Technology & Persons with Disabilities Conference, San Diego, March 2010 © 2010 28 HOPE H.O.P.E – Helping Older People Easily • Enhancements: - Explore the ‘Contextual’ improvement - Use the ‘minimum clicks’ approach - A sample contextual mechanism for Daily Living Activities inspired from the vocabulary of the Maya civilization Revolves around the activity with only relevant contextual icons surrounding it Final sentence is the combination of the activity and the context icons e.g. Take Bath + ( Cold or Hot Water) 25th Annual International Technology & Persons with Disabilities Conference, San Diego, March 2010 © 2010 29 Project in a Nutshell • • • • • • Feasibility Study involved identifying communication barriers and user needs Developed service requirements and design concepts for our prototypical system Still difficult to draw solid tangible requirements from user needs due to complexity and individuality in communication barriers On going project incremental – evolutionary and agile methodology Many issues – what to represent, how to represent in the best effective manner. Growth depends on more learning, discovery 25th Annual International Technology & Persons with Disabilities Conference, San Diego, March 2010 © 2010 30 Conclusion • Population of elderly to grow – so is the need and urgency for countering communication difficulties • Issue has been studied, user needs and concerns identified • Design of the system based on 5 types of vocabulary • Tiny AAC software implemented – H.O.P.E • More investigation on contextual vocabulary and representing core vocabulary • Develop a ‘minimum clicks – maximum vocabulary’ approach • Extend system by adding features for a larger audience • Make a change – Bridge the gap due to communication barriers 25th Annual International Technology & Persons with Disabilities Conference, San Diego, March 2010 © 2010 31 Questions ? Contact: Rutvij Mehta Email : [email protected] 25th Annual International Technology & Persons with Disabilities Conference, San Diego, March 2010 © 2010 32