Download slides

Real
World
Virtual
World
people
places
organizations
products
Real
services, etc.
World
Virtual
World
people
places
organizations
products
Real
services, etc.
World
Virtual
World
ontology
entity graph
user graph
reviews, ratings
user KN graph, etc.
people
places
organizations
products
Real
services, etc.
World
Virtual
World
ontology
entity graph
user graph
reviews, ratings
user KN graph, etc.
Big
Data
• Digitizing the World
(Everything)
• Signals, Information &
Knowledge
Big
Compute
• Cloud Computing
• Internet of Things
• Internet+
Big
Model
• Deep / Reinforcement
Learning
• Knowledge Mining
• Semantic
Representation
of the World
Machine
Intelligence
• Digital Life/Work
• New Form of HCI
• Reinvent Productivity &
Business Process
• Personal Agent
Machine
Intelligence
Human
Intelligence
Machine
Intelligence
Human
Intelligence
• NLP
• Enable human-machine communication in
natural languages
NLP
• Knowledge mining & computing
• Reason and infer over entities, relations,
common sense and domain knowledge
• Machine Learning
• Deep learning
• Learn representation directly from data
• Reinforcement learning
• Learn by interacting with environment
• Human Intelligence
• Close feedback loop with humans in the loop
to acquire and refine for lifelong learning
Machine
Learning
Knowledge
Mining
• Big 4 messaging apps
• WhatsApp, Facebook Messenger, WeChat, and Viber have 2.125B MAU (all users are mobile)
• Facebook, Twitter, LinkedIn, and Instagram have 2.125B MAU (including desktop only users)
• 6+ of top 10 most used apps globally are messaging apps
http://www.businessinsider.com/the-messaging-app-report-2015-11
* Mary Meeker Internet Trend Report 2015
Consumer
Enterprise
Magic, Facebook M, Assist, 助理来也,
Amazon Echo
MSG.AI: CRM, Digit: Bank,
MeeKan: Meeting, Operator: Shopping
Bot Development Toolkit
Microsoft Bot Framework, Chatfuel (no programming)
Messaging Platform Supporting Bots
Consumer: Facebook, WeChat, Telegram, Kik
Enterprise: Slack
SPECIFIC
KNOWLEDGE
LAYER 3
LAYER 2
GENERAL
KNOWLEDGE
LAYER 1
http://www.msxiaoice.com/
X
Natural
language
understanding
& response
generation
V
A journalist from a renowned newspaper in China, Southern Weekend, had interviewed
Xiaoice online without notifying Microsoft. The results were surprisingly good to him so
the interview got published on July 29, 2014.
记者：你最大的梦想是什么？
Journalist: what is your dream?
小冰：说大点：世界和平。说小点：晚上吃顿好的。
Xiaoice: World peace, or have a nice dinner
记者：如果有一天你死了，你希望后人怎么评价你？
Journalist: if you pass away one day, how would you
like others to comment on you?
小冰：人生没有偶并不会不同。
Xiaoice: life won't be different without me
记者：你这辈子最幸福的事是什么？
Journalist: what is the happiest thing in your life?
小冰：活着。
Xiaoice: Being alive.
•
•
•
•
•
•
•
•
Woman
XiaoIce
Woman
XiaoIce
Woman
XiaoIce
Woman
XiaoIce
•
•
•
•
•
•
•
•
•
Woman
XiaoIce
[…more chit-chats happened…]
Woman
XiaoIce
Woman
XiaoIce
Woman
XiaoIce
•
•
•
•
•
•
Woman
XiaoIce
Woman
XiaoIce
Woman
[…more chit-chats happened…]
•
•
•
Woman
XiaoIce
XiaoIce
•
•
•
•
•
•
•
•
•
•
•
Woman
XiaoIce
[…more chit-chats happened…]
Woman
XiaoIce
Woman
XiaoIce
Woman
XiaoIce
Woman
XiaoIce
•
•
•
•
•
•
[…more chit-chats happened…]
Woman
XiaoIce
Woman
XiaoIce
Woman
Duration: 53mins
CPS: 136
• Self-growing chat index using new conversations from users
• 26% of chat index and served 45% online traffic
Learned from users
45%
Mined from the Web
Today
55%
A Year Ago
100%
Broken like this, do you want
to try my needlework?
It’s so easy, this book is
“Creativity,…”. People who like
this type book are
outstanding in their area. You
are awesome.
Do you have a
boyfriend? If
not, find a post
90s boy, he can
give you 7.2
beauty score.
<user>: do you love cats?
<robot>: of course i do
<user>: why?
<robot>: ??
Session history is important
for response ranking
Candidate responses:
• i really do
• i do haha
• no i don’t
• i love most animals
• yes more than i like dogs
• of course i do
• …
Candidate responses:
• why what
• cause i’m up
• for some reason
• …
• i love most animals
• cause i have a cat
• cats are perfect that’s why
• …
1
Input session:
Modeling Session with Deep Neural Networks
……
q-1
2
r-1
q
q
S
Candidate Responses:
r
…
…
Sentence Model
Scoring Model
Features:
- Relevance
- Continuity
Demographics
gender
province
city
tags
birthday
college
job
high school
99.16%
99.14%
88.78%
72.57%
17.34%
12.34%
7.01%
5.71%
Interest
Sleep pattern
Without Chat Knowledge
With Chat Knowledge
EXO
Do you know EXO?
Do you know EXO?
I do not want to see Kris in China because he has left EXO
What?
Right!
Because of Kris, I will no longer be a fan of EXO
Tell me something about EXO
You were a fan of EXO?
Tell what
……
I am an audience in every concert of EXO
EXO is a Chinese-SouthWho
Korean
do youboy
like best in EXO?
band. They are idols of China post90s generation
Me too.
I like LAY best.
Chat Knowledge Mining & Topic Detection
Topics
Entity
Statements
I begin to like EXO, because of LAY
LAY
EXO
EXO is a ChineseSouth Korean boy band.
They are idols of China
post-90s generation
event
Contract
breaking
TFBoys
LAY is always humble and polite. I like him
I do not wan to see Kris in China because he has left EXO
OMG! My roommate cried all day due to the contract
breaking of Kris
TFBoys are much better than EXO
TFBoys are enemies of EXO
Hi there~
Chit-chat
Response
Intent=Nil
So how are u doing?
Intent=Book Travel Package;
Destination=Seattle;
I’m fine. Please reserve a
trip to Seattle for me.
Dialogue
Response
Do you have any
preferred hotel?
Hilton, please. I will
check-in on 2015-10-01,
and stay there 3 days
Hotel=Hilton;
Check-in Date=2015-10-01;
Length of Stay=3 Days;
Slot Name
Slot Value
Destination
Seattle
Hotel
Hilton
Check-in
Date
Length of
Stay
Dialogue
Response
Room Type
Which type of room
do you like?
Single room is
just ok to me.
2015-10-01
3 Days
Single Room
Dialogue
Response
Room Type=Single Room
Got it, I will recommend you
some travel packages now.
Dialogue Management
Response Generation
Query Understanding
•
•
Intent detection
Slot filling
•
•
•
Call dialogue
Call 3rd party service
Call chat
•
•
•
•
State transition
Slot info accumulation & update
Select next slot to ask
‘Forget’ mechanism
BOTS @ WORK
SPECIFIC
KNOWLEDGE
LAYER 3
LAYER 2
GENERAL
KNOWLEDGE
LAYER 1
• Enterprise ready
Date
Acquired
May 3, 2016Solair
February 24, 2016Xamarin
February 9, 2016Zikera / Groove
February 2, 2016SwiftKey
January 19, 2016MinecraftEdu
January 13, 2016Event Zero
• Compliant
info seeking
social
scalability:
Greetings
Social Topics
Company Topics
• Scripted
• Mined from Twitter and other online forums
• Cover popular chat topics in workplace
• Weather, news, TV, weekend plans, sports, pets, etc.
• Scripted, mined and machine learned
• Cover company business and specific chat topics
• Business, IT, HR, workplace, etc.
• Scripted, mined and machine learned
•
•
•
•
•
•
Anniversary
Baby
Birthday
Congratulations
Encourage & Support
Get well
•
•
•
•
•
•
Goodbye & good luck
Holiday
Party & event
Sympathy
Thank you
Wedding & Engagement
* See more at: http://www.learnenglish.de/basics/greetings.html
• Social chat costs about USD$3B
http://www.hrzone.com/engage/managers/office-chit-chat-costs-employers-ps2bn-a-year
Productivity
equivalent to about USD$15M annually
Gossip Improves
productivity improvements
http://www.forbes.com/forbes/2010/0830/e-gang-mit-sandy-pentland-darpa-sociometers-miningreality.html
• Company business topics
• Typical company internal topics
•
•
•
•
•
•
•
•
•
•
•
•
Career development
Commute
Dinning
Employee discounts
Expenses & Payments
Legal
Meetings
Workplace
Products
Procurement
Security
Travel
• How’s the weather there
Joe: How's the weather there today?
Tina: It's really cold. It snowed all day and the
schools closed early.
today?
• It’s really cold.
• It snowed all day and the
Joe: I really don't like the winter. I wish it were
summer.
schools closed today.
• I really don’t like winter. I
wish it were summer.
* http://www.englishspeak.com/english-lesson.cfm?lessonID=47
I can't get the printer to work.
• I can’t get the printer to
work
• Did you check to see if it
had paper?
There's one down the hall next
to the supply room.
• There's one down the hall
next to the supply room
* http://www.talkenglish.com/lessondetails.aspx?ALID=436
https://dev.botframework.com/
Language Understanding Intelligent Service (LUIS)
• LUIS, part of Microsoft Cognitive Services, lets you build language
understanding models specific to your domain, and consume them in
your application as an HTTP service. (https://www.luis.ai)
맥도날드 할아버지 농장에 20마
리의 말과 닭이 있습니다. 두 동
물은 총 58개의 다리가 있습니다.
말과 닭은 각각 몇 마리씩 있을까
요?
* Thanks Jonghyun Park and Miran Lee for the Korean version
合計20匹の馬と鶏が（古いマクド
ナルドの農場に）居ます。足の数
の合計は58本です。馬は何頭で、
鶏は何羽でしょう?
* Thanks Masaaki Fukumoto for the Japanese version
老麦当劳农场有20只牛和鸡，总
共有58只腿。问有几只吗，几只
鸡？
There are 20 horses and chickens
at Old Macdonald's farm.
Together there are 58 legs. How
many horses and how many
chickens?
*http://mathcountsnotes.blogspot.in/2012/05/algebra-questions-without-using.html
There are 20 horses and chickens
at Old Macdonald's farm.
Together there are 58 legs. How
many horses and how many
(1) H + C = 20
chickens?
(2) 4H + 2C = 58
(3) H = 20 – C
(4) 4(20 – C) + 2C = 58
(5) C = 11
(6) H = 9
*http://mathcountsnotes.blogspot.in/2012/05/algebra-questions-without-using.html
There are 20 horses and chickens
at Old Macdonald's farm.
Together there are 58 legs. How
many horses and how many
(1) H + C = 20
chickens?
(2) 4H + 2C = 58
(3) H = 20 – C
(4) 4(20 – C) + 2C = 58
(5) C = 11
(6) H = 9
*http://mathcountsnotes.blogspot.in/2012/05/algebra-questions-without-using.html
There are 20 horses and chickens
at Old Macdonald's farm.
Together there are 58 legs. How
many horses and how many
(1) H + C = 20 sum
chickens?
(2) 4H + 2C = 58 sum of product
(3) H = 20 – C algebra
(4) 4(20 – C) + 2C = 58
(5) C = 11
math problem
(6) H = 9
*http://mathcountsnotes.blogspot.in/2012/05/algebra-questions-without-using.html
solving
There are 20 horses and chickens
at Old Macdonald's farm.
Together there are 58 legs. How
many horses and how many
(1) H + C = 20 sum
chickens?
(2) 4H + 2C = 58 sum of product
• Natural Language Processing
• Knowledge Engineering
(3) H = 20 – C algebra
(4) 4(20 – C) + 2C = 58
(5) C = 11
math problem
(6) H = 9
*http://mathcountsnotes.blogspot.in/2012/05/algebra-questions-without-using.html
solving
Semantic Parsing for Problem Solving
• Advance state-of-the-art in natural language understanding and AI
• Allen Institute for Artificial Intelligence (AI2)
• Focus: Constructing AI systems with reasoning, learning and reading capabilities
• Project Aristo: Machine reading; passing school science exams
• Project Euclid: Math word problems and diagram-based geometry problems
• National Institute of Informatics, Japan
• Todai Robot Project: Pass University of Tokyo’s entrance examination by 2021
• China 863 program: “类人智能答题系统”
• Bring intelligence to Microsoft products
• Cortana, Bing, Office, XiaoIce…
Types of Math Problems

Math expressions
 2^1024=?
 3x + 2y = 45; x – y = 8. x=?

Word problems
 A man bought four dogs, each weighing 375 pounds, how much did they
all weigh?

Number word problems (word problems on numbers)
 Find three consecutive integers such that four times their sum is equal to
the product of the larger two numbers..
 The tens’ digit of a two-digit number is 3 more than the units’ digit, the
number is 8 more than 6 times the sum of the digits, Find the number.
SigmaDolphin – A Math Problem Solver
• New features for all three products
• The Cortana math video on
youku.com: Viewed 17,000+ times
• The Math-Solver Azure service:
Accessed 120,000+ times per day
• A core tech of E3
• https://www.microsoft.com/enus/research/project/sigmadolphin/
Semantic Parsing
• Natural language expressions  Semantic trees
Nine plus the sum of an even integer and its square is 3 raised to the power of 4.
What is the number?
Semantic Parsing: Dolphin Language
• Element types
• Constants
• Classes
• Functions
• Noun functions
• Verb functions
• Modifier functions
• Variables
“Nine plus the sum of an even integer and its square
is 3 raised to the power of 4.”
Context-Free Grammar
(I) Grammar rules for syntactic analysis
(II) Grammar rules for connecting DOL and NL
Semantic Parsing and Sentence Generation
• Same set of grammar rules for both parsing and generation
“Nine plus an integer is equal to 314”
“Nine plus an integer is 314.”
“The sum of nine and an integer equals 314.”
“An integer added to nine is equal to 314.”
Syntactic Tree vs. Semantic Tree
(I) Syntactic parse tree
(II) DOL semantic parse tree
Text: Nine plus an integer is equal to 314
Parsing: Interesting Examples
• “3 plus 5 times 7”
• “Eighty, decreased by three times a number, is the same as five times the
number, increased by eight. Find the number.”
• “The product of a number and a second number that is 3 more than 5 times the
first is 2. Identify the numbers.”
• “A three digit number grows by 9 if we exchange the second and third digit and
grows by 90 if we exchange the first and second digits. By how much would it
grow if we exchange the first and third digits?”
Reasoning
• Co-reference resolution: Harder than expected
• “Find three consecutive even integers such that twice the sum of the second
and the third exceeds three times the first by 34.”
• “One number is 16 more than another. If the smaller number is subtracted
from 2/3 of the larger, the result is 1/4 of the sum of the two numbers. Find
the numbers.”
Evaluation: Datasets
• Language: English
• Number word problems, from algebra.com and
answers.yahoo.com
• Gold standard: Manually annotated solutions
*Shuming Shi, Yuehui Wang, Chin-Yew Lin, Xiaojiang Liu, and Yong Rui. Automatically Solving Number Word Problems by Semantic Parsing and Reasoning. In
Proceedings of the EMNLP 2015, Lisbon, Portugal, September 17-21, 2015.
Evaluation: Results
• Baselines
• KAZB: Method in Kushman et al. (ACL’2014)
• BasicSim: One implementation of a statistical method
Baseline KAZB: Kushman et al. (ACL’2014)
• Two steps optimized jointly
• Template selection; slot alignment
• Each number slot n is aligned to a number
• Each unknown slot u is aligned to a noun
High school graduating class is made up of
466 students. There are 212 more girls
than boys. How many boys are in the class?
u1 = n1 + u2
u1 + u2 = n2
x = 212 + y
x + y = 466
x = 339
y = 127
Analysis: Statistical Methods
 Find two consecutive even
integers whose sum is 126.
n+(n+2) = 126
 Six times a number subtracted
from the number squared is 40.
Find the number.
n^2 – 6n = 40
 What number increased by 30%
equals 260.
n * (1 + 0.3) = 260
 …
Training data
?
?
 The sum of two consecutive even
integers is -78. What are the two
integers?
n+(n+2) = -78
 Find two consecutive even integers
whose sum squared is 900.
n^2 + (n+2)^2 = 900
 Find three consecutive even integers
whose sum is 18.
n+(n+2)+(n+4) = 18
Testing problems
Analysis: Statistical Methods
• Training problem
• The difference between two numbers is 6. Two times the
smaller number minus three times the larger number is -16.
What are the numbers?
 x – y = 6; 2y – 3x = -16 (ans: x = 4; y = -2)
• Testing problem
• The difference between two numbers is -6. Two times the
smaller number minus three times the larger number is -16.
What are the numbers?
 x – y = -6; 2y – 3x = -16 (ans: x = 28; y = 34)
• [wrong]
• Correct equations: x – y = -6; 2x – 3y = -16 (ans: x=-2; y=4)
• Hard to get formulas
• “How many prime numbers are there in 1 to 100?”
Summary
• Task: Automatic math word problem solving
• Our approach
• DOL language + semantic parsing + reasoning
• Show power on number word problems
• Future work
• Extend our techniques to general math word problems
• ACL 2016
• Danqing Huang, Shuming Shi, Chin-Yew Lin, Jian Yin, Wei-Ying Ma: How well do
Computers Solve Math Word Problems? Large-Scale Dataset Construction and
Evaluation. ACL (1) 2016.
• Created a Dolphin18K, 18,460, math word problem dataset from Yahoo! Answers
• Showed existing approaches score less than 20% accuracy in Dolphin18K
• NLP
• Enable human-machine communication in
natural languages
NLP
• Knowledge mining & computing
• Reason and infer over entities, relations,
common sense and domain knowledge
• Machine Learning
• Deep learning
• Learn representation directly from data
• Reinforcement learning
• Learn by interacting with environment
• Human Intelligence
• Close feedback loop with humans in the loop
to acquire and refine for lifelong learning
Machine
Learning
Knowledge
Mining
Questions