Download Semantics-SQL system.

SemanticSQL Redux Matthew Fay and Jodyann Coley
Original Term Project, 6.863, Spring 2010
Matthew Fay
Project Update, Summer 2010
Current Code: http://semanticsql.dnsalias.org/SemanticSQL/SemanticSQL.zip
Live Current Version: http://semanticsql.dnsalias.org/SemanticSQL/
Introduction
A large goal of modern computational studies is to develop intelligent agents that are able to reason.
Intelligent systems must be equipped with tools that allow them to learn about their environments, process the
information, and make decisions. From a human perspective, the root of such tools lies in language. Thus if
we are to be able to enable computational reasoning, we must develop useful computational language
models. Language research can be divided into a number of areas such as syntax, morphology and semantics. We decided to focus on semantics for our project. Semantics allows us to derive meaning from sentences in
order to gather information from natural language descriptions. This project aims to create a semantic parser
of English that is able to extract meaning from sentences and store the information about a scene into a SQL
database. The same parser can then intelligently answer questions phrased in English about the constructed
scene. Previous work in this area was done by Eric Lieberman and Mike Lin. The goal of our work is to
improve upon the parsing methods and then to take it to the next level by incorporating new levels of
understanding. Specifically, we sought to incorporate the following grammar constructs:
1. How an action was taken using adverbs
2. Location of events using prepositional phrases
3. Instrument usage using prepositional phrases
4. Cause and Effect using If-­Then sentence structures
5. Usage of the adverb not for negations
6. Passive sentences
7. Object classes and quantifiers
Additionally, new question structures needed to be incorporated to allow this new information to be used. Finally, we wanted to improve the semantic processing to allow more intelligent question responses such as
through improved indefinite noun usage. For example, if the sentence "John likes the cats" was previously
submitted to the system, then the system should return yes to the question "Does John like a cat".
With the expansions incorporated, the SemanticSQL system can understand sentences such as "John
happily ate icecream with a spork in the park" and then appropriately answer questions such as "How did
John eat". The current version currently is running on a test server and can be found here.
Design and Implementation
The SemanticSQL processor works through the following steps:
1. The English sentence is passed through a grammar parser.
2. The parser pulls the semantic meaning from the sentence using lambda calculus.
3. The semantic event is translated into the appropriate SQL statements.
A simple example of this process is the question "Did John sleep", which would be translated into a SQL
statement resembling "SELECT FROM Event WHERE action=sleep and agent=john". The current
implementation, however, allows more robust object tracking. Whenever a new agent takes an action in the
scene, the system checks to see if it knows of the object using a SELECT statement, then adds the object to
the database using an INSERT statement if it is not found. This allows an object to appropriately be involved
in multiple events. While this makes the system more flexible, it also means that even simple statements such
as "John kissed Mary" requires 3-­5 SQL statements depending on whether or not John and Mary are already
in the scene.
The database currently consists of three tables for semantic knowledge: Events, Objects, and Modifiers.
The table structures are detailed below.
Modifier Table
Mod
The adjective, ex. red, green
Target The ID of the object modified
Object Table
ID
Unique ID Assigned to Object
Name
The name of the object, ex: John, Mary
Type
The type of the object, ex: cat, dogs
Definite Whether the object is definite, indefinite, or modified by a quantifier
Event Table
Action
The action taken by the agent, ex: eat, sleep
Agent
The ID of the object that does the action
Patient
The ID of the direct object
Beneficiary The ID of the recipient of the action, for example who something was given to
Tense
When the event occurred
Loc_Prep
The preposition related to the location, ex: in, under
Loc_Target The ID of the location where the event took place
Instrument The ID of the instrument used in the event
Adverb
How the action was taken
Adjective
Adjectives used with the verb 'to be'
Negation
Whether an event is a truth or a negation
The following is a sample set of sentences that have been annotated with the generated SQL code to show
how the system actually works.
Sentence: Poirot likes the dogs
Step Description
1
2
3
4
SQL
Result
resolving named
object
SELECT id FROM Object WHERE name='Poirot'
not found,
will insert
inserting a new
INSERT INTO Object(`name`,`type`,`definite`) VALUES
named object
('Poirot',NULL,NULL)
resolving unnamed
SELECT id FROM Object WHERE `type`='dogs' AND
not found,
object
Object.definite = '+'
will insert
inserting a new
INSERT INTO Object(`name`,`type`,`definite`) VALUES
unnamed object
(NULL,'dogs','+')
INSERT INTO Event VALUES
('like',184,185,NULL,'present',NULL,NULL,NULL,NULL)
5 adding fact
OK, id = 184
OK, id = 185
OK
SemanticSQL now knows that "Poirot likes the dogs". Similarly, we can say that "Mary likes the cats". After these sentences are submitted to the system, the system generates the following SQL code for the
sentences below.
Sentence: Does Poirot like the dogs
Step Description SQL
resolving
1 named
object
Result
OK,
SELECT id FROM Object WHERE name='poirot'
id =
189
resolving
2 unnamed
SELECT id FROM Object WHERE `type`='dogs' AND Object.definite = '+'
object
yes/no
3
question
OK,
id =
190
SELECT CASE WHEN COUNT(*) > 0 THEN 'Yes' ELSE 'No' END FROM
Event WHERE action='like' AND agent=189 AND patient=190 AND
Yes
tense='present'
Sentence: Does Poirot like the cats
Step Description SQL
Result
resolving
1 named
object
OK,
SELECT id FROM Object WHERE name='poirot'
resolving
2 unnamed
object
yes/no
3
question
id =
189
OK,
SELECT id FROM Object WHERE `type`='cats' AND Object.definite = '+'
id =
194
SELECT CASE WHEN COUNT(*) > 0 THEN 'Yes' ELSE 'No' END FROM
Event WHERE action='like' AND agent=189 AND patient=194 AND
tense='present'
No
SemanticSQL correctly detects that Poirot likes the dogs and is not confused by the addition of the cats to the
scene. The following sections discuss the implementation of the new features and grammar constructs
available in SemanticSQL.
Adverbs
The system allows users to add adverbs to modify verbs, as demonstrated in the sample sequence of
statements in following table. Sentence
System Response
John eats quickly
OK
How does John eat
quickly
John eats gracefully
OK
How does John eat
quickly and gracefully
Locations
Our system allows users to specify the locations of events and ask questions regarding where events
occur. In our design, locations are considered to be features of verbs. The location feature consists of the
preposition of the prepositional phrase (Loc_Prep in the Event Table) and the object of the location
(Loc_Target in the Event Table). The table below illustrates an example sequence of sentences that are
allowed in the system and the system's responses.
Sentence
System Response
John kissed Mary in the park
OK
Where did John kiss Mary
in the park
John happily kissed Mary under the table
OK
Where did John kiss Mary
in the park and under the table
Where did John happily kiss Mary
under the table
Instruments
The system allows users to specify instruments that are used during events. Similar to locations,
instruments are considered as features of verbs. An example sequence is shown in the table below.
Sentence
System Response
John ate icecream with a spoon
OK
What did John eat with
a spoon
John ate the chicken with a fork
OK
What did John eat the chicken with
a fork
What did John eat with
a spoon and a fork
John slept happily in a bed with a pillow
OK
What did John sleep with
a pillow
John slept in the bed with Mary
OK
What did John sleep with
a pillow and Mary
Cause and Effect
Users are able to add new events to the database through if-­then statements. If the first part of the
statement (the cause) is true based on the information in the database, then the second part of the statement
(the effect) is added to the database. Otherwise, "the effect" is not added to the database. Some example
sentences are in the table below. Sentence
System Response
Mary likes John
OK
If Mary likes John then Mary likes the werewolves
Cause was found, Effect added to the database
If Mary likes the cats then Mary likes the dogs
Cause not found, Effect will not be added
John sleeps happily
OK
If John sleeps happily then John likes Mary
Cause was found, Effect added to the database
Negations
Negations can be applied to nearly any sentence structure that exists in the grammar. Some examples
include: John does not like Mary, Who did not eat the icecream, and Poirot did not give the dog to John.
The adverb not is applied on the event level tracking whether an event happened or whether the event
specifically did not occur.
Sentence
System Response
Mary likes John
OK
Mary does not like Poirot
OK
Does Mary like John
Yes
Who does Mary not like
Poirot
Does Mary like Poirot
No
Passive Sentences and the Verb "to be"
The verb "to be" adds an additional layer of complexity into the English grammar. Primarily it allows
numerous new sentence constructions to be included. Passive constructions such as John was driven by
Poirot and Mary was liked by the dogs can now be interpreted correctly and stored in the scene database.
Additionally constructs such as John was happy and Mary is blue were enabled by including an adjective
portion of an event occurrence. A final category included is the addition of the ability to ascribe classes to
proper nouns which will be described more in the next section.
Sentence
System Response
Mary was liked by John
OK
Does John like Mary
Yes
Poirot ate icecream
OK
Was the icecream eaten
Yes
Who was the icecream eaten by
Poirot
Object Classes and Quantifiers
The ability for people to categorize people, places and things is extremely important for our overall
understanding of the world. Incorporated into the SemanticSQL grammar and SQL code is the new
understanding of categories. The most simple of example of this is the following.
Sentence
System Response
John is a person
OK
Is John happy
No
Everyone is happy
OK
Is John happy
Yes
This ability can allow for a great deal of common knowledge understanding to begin to be incorporated
into the SemanticSQL architecture. Quantifiers can also be successfully parsed and stored in the database.
Sentence
System Response
All people like dogs
OK
Some people are happy
OK
Dogs eat some icecream
OK
Some quantifiers can also be extended into definite predictions about the scene that are not explicitly
stated. Here's another more complex example of the power of quantifiers in classification.
Sentence
System Response
John is a person
OK
All people like dogs
OK
Does John like dogs
Yes
Fido is happy
OK
Does John like Fido
No
Fido is a dog
OK
Does John like Fido
Yes
Infrastructure
In addition to the features added to the grammar that have been covered thus far, a number of
improvements were made to the infrastructure of SemanticSQL to improve its robustness. The first change
was the construction of vocabulary tables in the database. The noun, verb, adjective, and adverb lexicons
were moved out of the python server code into the SQL database. By making this change, new vocabulary
can be instantly added to the dictionary without needing to change any of the underlying code. This makes
dealing with new words very convenient. The adverb and adjective tables are simple word lists. The verb
table stores the verb and its morphological forms. The noun table additionally incorporates useful metadata. It stores not only each noun in the lexicon,
but also the associated feature information such as whether the word is plural, a mass noun, or a name. Since
the features are stored as text, they can easily be expanded on to fit the needs of a more powerful grammar
without needing the noun table to be altered. Finally the noun table also stores the plural form of singular
nouns, creating a semantic link between both forms of the noun.
The noun lexicon table enabled us to deal with indefinite noun phrases in a more powerful way. For
example, if you tell the system "Mary likes the dogs", then it will create an entry in the object table for Mary
and another for a definite type "dogs". Later one might ask, "Does Mary like a dog". Intuitively the answer
must be yes. While a simple query would return no, we can use the metadata linking the plural and singular
noun forms to perform a second search. Now it will find that Mary likes more than one dog so she must like
at least one dog, and the query returns yes.
In order to facilitate some of these changes such as the incorporation of object classes, the SQL code
generation it self had to be tweaked significantly. The primary change is that when asking questions that take
classifiers into account, the SQL code is generated to allow more flexibility in the WHERE clauses including
searching for multiple possible ID matches in the same search.
Testing and Analysis
During the course of development, we periodically tested the system using a large set of sentences that
worked on the previous version of SemanticSQL, as well as new sample sentences that incorporated the
features under active development. By doing this, we quickly identified problems that arose from the
incorporation of new grammar, semantic, and SQL features and made sure that all previous constructs
remained viable. Through our careful analysis of the grammar and overall SemanticSQL system, we
discovered some problem areas in doing English to SQL translations.
One area that presents a significant amount of difficulty are indefinite and anonymous objects. Part of
this issue was addressed by incorporating more robust infrastructure, as previously mentioned. However, it
still poses a significant problem. For example, the sentence "Mary likes the dogs" works fine, while "Mary
likes dogs" does not. The problem goes beyond the syntax. Since the SemanticSQL database was designed
to store information about a scene, it does not currently have the constructs to support detailed object
descriptions such as Mary's general likes and dislikes that do not refer to specific objects in the scene, as
demonstrated in the previous example. A temporary fix could have been implemented to tag "dogs" as
indefinite in the failed sentence and add it to the object table. However, it's quite reasonable to assume that
while "Mary likes dogs (in general)", that does not necessarily imply that "Mary likes the dogs (in the
scene)". A possible solution could be to incorporate more detailed object descriptions and the potentially
probabilistic scene understanding. In this case, if "Mary likes dogs", then there would be a high probability
that "Mary likes the dogs".
Another issue that arose as we incorporated an ever growing feature set was the problem of
permutations. Take the sentence "John peacefully ate the fish in the park with a fork" for example. The
same sentence could be expressed in a number of other ways such as "John ate the fish peacefully with a fork
in the park", "John ate the fish peacefully in the park with a fork", "John ate the fish with a fork peacefully in
the park", etc. We have incorporated many of these permutations. However, as we increase the number of
features, it becomes more difficult to use the grammar. A related problem would be detecting whether or not
the prepositions are attached to the verb as we assume, or if it would be more accurate to attach them to an
object that appears earlier in the sentence.
For convenience, the table below summarizes the current ability of the system in dealing with these
issues.
Sentence
System Response
Mary likes the dogs
OK
Mary likes dogs
NOT OK
John peacefully ate the fish in the park with a fork
OK
John ate the fish peacefully with a fork in the park
OK
John ate the fish peacefully in the park with a fork
OK
John ate the fish with a fork peacefully in the park
NOT OK
John ate the fish with a fork in the park peacefully
NOT OK
Future Work
The SemanticSQL system works well within the bounds of the current grammar constructs it supports.
However, developing a translation tool that is able to parse even a small subset of the entire English language
is a daunting task. The sections below discuss a number of ideas that we believe would be useful expansions
to SemanticSQL.
Multiple Prepositional Phrases and Adverbs
The current implementation only allows a single occurrence of location, adverb, and instrument phrases.
One way to improve the system would be to move these properties of events to their own tables. This would
allow a variable number of each to affect events once the proper grammar expansions were incorporated.
This feature would allow sentences such as "John slept under the tree in the park" to be understood, and the
question "Where did John sleep" would return "under the tree in the park".
Symmetric Relations
Currently, the system does not account for symmetric relations. For example, if "John kissed Mary", the
current implementation does not know that "Mary kissed John". Adding symmetric relations would improve
the system, especially when users ask questions based on those relations. For example, the question "Did
Mary kiss John" should return yes if "John kissed Mary" was submitted to the system.
Kimmo
The system could benefit greatly from a closer connection to Kimmo and morphological rules. By
incorporating morphological rules, the grammar could be easily expanded. A major advantage of this would
be the ability to draw features from the vocabulary. For example, instead of storing "the dogs" as an object,
all nouns could be broken down to their core meanings, such as "dog[plural, definite]". This would help with
the indefinite object problem previously mentioned. Ideally, the Kimmo lexicon could be directly connected
to the SQL vocabulary tables that have already been implemented.
If-­Then Expansion
The current implementation of the "If...Then..." cause and effect rules is fairly primitive. The construct
is parsed, acted upon, and then discarded. A more robust system could create a new "Rules" table that could
store causal information. Then whenever an event occurs, the event could be checked against the rules table
to see if any "effects" should be triggered. If one were to incorporate this expansion, he/she would have to
consider whether or not he/she wanted later events to trigger "effects" that were previously entered. Comparisons
An expansion to the project would be to add comparisons to the system. This would consist of adding
quantities, such as "The red ball weighs 10 pounds", in addition to modifiers such as "little"and "big" that
could be used in statements such as "The little dog walked in the park". As such, users would be able to ask
questions such as "Which ball is the biggest" and "Does the red ball weigh more than the blue ball".
Time
Another expansion on this project would be to add the capability of time. For example, it would be a
good feature for users to be able to ask questions such as "When did John go to sleep". Also, this could be
expanded further by relating events based on time, such as "Did Tom go to the store before Mary came
home". Probabilistic Parsing
A final powerful tool to incorporate into the project would be the addition of a probabilistic sentence
parser such as a grammar trained on a large corpus like the Wall Street Journal. For example using the
MXPOST maximum entropy tagger could allow the system to intelligently identify the part-­of-­speech of
previously unknown words and appropriately add them to the dictionary database. Contributions
In this SemanticSQL project we have:
Expanded the English to SQL parser to incorporate new knowledge types: locations, instruments,
negations, and passive phrases.
Added important grammar and semantic features for understanding including the idea of object classes.
Implemented new infrastructure to allow dealing with causal relations, indefinite constructs and an
expanded lexicon.
Identified future areas for development on the English to SQL parser.
Appendix A: Example Sentences
The table below lists a number of sentences that can be used in the SemanticSQL system.
John slept under the tree
John ate the green moldy bread with the werewolf
Poirot sadly gave the cats to Mary
What did John eat
Mary ate the icecream
What did John eat with
Where did John sleep
How did Fido toss the ball
Who gave the cats to Mary
Fido gracefully tossed the ball to Fido in the park with a glove
Who did Poirot give the cats to
Poirot slept
Mary likes John
Where did Bill kill Mary
Jane ate icecream quickly in the park with a spoon If Poirot kissed Mary then Mary kissed Poirot
Appendix B: Technical Requirements and How to Run it
Getting SemanticSQL running can take a bit of work. The system has been tested using the following
programs and libraries:
Python 2.6.5
NLTK 1.2
NLTK-­Contrib 1.4.2
MySQLdb 1.2.3c1 (Python Library)
Apache 2.2.11
MySQL 5.1.36
Once these are installed, Apache needs to be configured to run Python code as a CGI script. The lines below should be present in the http.d file for apache.
Options ExecCGI
AddHandler cgi-­script .py
On linux, Apache can use the Shebang line of the python scripts to find the interpreter.
On windows, the following line can also be added to the http.d file to use the windows registry to find the
interpreter.
ScriptInterpreterSource Registry-­Strict In the main directory of the SemanticSQL project, the ".htaccess" file should contain the following lines, the
PYTHON path should be modified appropriately:
SetEnv PYTHONPATH C:/wamp/www/nlp/semantics/
Options ExecCGI
AddHandler cgi-­script .py
MySQL will need to be setup to allow connections from the Apache server and should have the database
from the nlp.sql (in the source code) available. The index.py file will need to be augmented with the SQL
server information (user, pass, db, etc).
Once this is done, SemanticSQL should be able to be run from the browser by accessing the index.py file.