Download BrainGene: computational creativity algorithm that invents novel

BrainGene:
computational creativity algorithm
that invents novel names
Maciej Pilichowski & Włodzisław Duch
Department of Informatics,
Nicolaus Copernicus University, Toruń, Poland
School of Computer Engineering,
Nanyang Technological University, Singapore
Google: W. Duch
SSCI 2013, Singapore
Most mysterious …
Intuition is relatively easy …
What features of our brain/minds are the most mysterious?
•
•
•
•
Consciousness?
Imagination?
Emotions, feelings?
Thinking?
Masao Ito, neuroscientist,
director of the RIKEN Brain Science Institute,
answered:
Creativity.
Creativity in psychology
MIT Encyclopedia of Cognitive Sciences (2001) has 1100
pages,6 chapters about logics, over 100 references to
logics in the index. Creativity is covered on 1 page
(+1 page about „creative person”).
Intuition: 0, not even mentioned in the index.
In everyday life we use intuition and creativity more often than
logics. It is a necessary part of human-level intelligence.
Computational approach to creativity is slowly getting more
popular, see http://computationalcreativity.net/, ICCC conferences
“Computational Creativity is the study and simulation, by
computational means, of behaviour, natural and artificial, which
would, if observed in humans, be deemed creative.”
Creativity in math
H. Poincare, The Foundations of Science (1908):
• Mathematical intuition and creativity is a discrimination
between promising and useless ideas and their combinations.
• Mathematical thinking may be based on heuristic search among
sufficiently rich representations.
• Math intuition is an interplay between spatial imagination,
abstraction and approximate reasoning, and analytical reasoning
or visual-spatial and linguistic thinking.
This is observed in fMRI neuroimaging (ex. S. Dehaene, 1999).
J. Hadamard, An Essay on The Psychology of Invention in the
Mathematical Field. Dover Publications, 1949, unconscious sources.
Mechanizing inventions
J. Hadamard, An Essay on The Psychology of Invention in the
Mathematical Field. Dover Publications, 1949
G. Altshuller, Algorithm of Invention. Moscowskiy Rabochy, 1969.
S. Savransky, Engineering of Creativity. Introduction to TRIZ
Methodology of Inventive Problem Solving. CRC Press, 2000.
TRIZ: "a problem-solving, analysis and forecasting tool
derived from the study of patterns of invention in the
global patent literature”.
•
•
•
•
SIT (systematic inventive thinking)
ASIT (advanced systematic inventive thinking)
USIT (unified structured inventive thinking)
TRIZICS (systematic application of TRIZ)
Creativity in business
J. Goldenberg, D. Mazursky, S. Solomon,
Templates in creative sparks.
Science, vol. 285, no. 5433, 1999.
J. Goldenberg, D. Mazursky,
Creativity in product innovation.
Cambridge University Press, 2002
Creativity templates: Attribute dependence,
Replacement and Displacement templates.
This approach is used for product design by large companies,
ex. Philips, Ford, Kodak, Coca-Cola, Motorola ...
Some history
Koza, J.R.; Keane, M.A.; Streeter, M.J.; Mydlowec, W.; Yu, J.; &
Lanza, G. (2003). Genetic Programming IV: Routine HumanCompetitive Machine Intelligence, Springer.
GECCO Conference offered “Hummies” cash awards for humancompetitive results that are patentable as new inventions.
Discovery of scientific laws:
J.M. Zytkow, H.A. Simon, A theory of historical discovery. Machine
Learning, 1, 107, 1986.
B. Goertzel, From complexity to creativity. Emergent Patterns and
Self-Organizing Dynamics in Mental, Computational and Physical
Systems. Springer, 1997.
Creativity and the brain
• A. Dijksterhuis, T. Meurs, Where creativity resides:
The generative power of unconscious thought,
Consciousness and Cognition 15(1), 135–146, 2006.
• W. Duch, Computational creativity, in: World Congress on
Computational Intelligence, Vancouver 2006, 1162–1169.
• Duch W, Pilichowski M, Experiments with computational
creativity. Neural Information Processing – Letters and Reviews,
11(4-6), 123-133, 2007.
• W. Duch, Intuition, insight, imagination and creativity.
IEEE Computational Intelligence Magazine, 2(3), 40–52, 2008.
• W. Duch, P. Matykiewicz, J. Pestian, Neurolinguistic approach to
natural language processing with applications to medical text
analysis.” Neural Networks, vol. 21(10), 1500–1510, 2008.
Psychology of creativity
G. Wallas, The art of thought (1926): four-stage Gestalt model of
problem solving.
4 stages: preparation, incubation, illumination and verification.
These stages have been identified in creative problem solving by
individuals and small groups of people; additional stages may be
added: finding or noticing a problem, proposing interesting
questions, frustration period preceding illumination,
communication following verification etc.
Understanding details of such stages and sequences yielding
creative productions is a central issue for creativity research,
but it is not sufficient for computational creativity.
What we need is approximation to creative processes in the brain.
Words in the brain
Psycholinguistic experiments show that most likely categorical,
phonological representations are used, not the acoustic input.
Acoustic signal => phoneme => words => semantic concepts.
Phonological processing precedes semantic by 90 ms (N200 ERPs).
F. Pulvermuller (2003) The Neuroscience of Language. On Brain
Circuits of Words and Serial Order. Cambridge University Press.
Action-perception
networks inferred
from ERP and fMRI
Phonological neighborhood density = the number of words that
sound similar to a target word. Similar patterns of brain activations.
Semantic neighborhood density = the number of words that are
similar in meaning to a target word.
Symbols in the brain
How do words that we hear, see or are thinking of,
activate the brain? Seeing words:
orthography => phonology => articulation => semantics.
Lateral inferotemporal multimodal area (LIMA) reacts to auditory +
visual stimulation, has cross-modal phonemic and lexical links.
Adjacent visual word form area (VWFA) in the left occipitotemporal
sulcus is unimodal. Perhaps auditory word form area also will be
found in the left anterior superior temporal sulcus.
Left hemisphere: precise representations of symbols, including
phonological components.
Right hemisphere probably sees clusters of concepts.
Models of creativity 1
Simplest creativity: invention of novel words,
understanding of neologisms. Go to the lower level …
Look at combinations of phonemes, emergence of morphemes, etc.
Creativity =
space +
priming +
imagination +
filtering.
Space::neural tissue providing space for almost infinite # of
distinguishable activation patterns; in the model trained neural
networks provide search space of internal states that reflect past
experience of the system, constraining possible neural activations;
Models of creativity 2
• Priming: reading/hearing/seeing primes the brain;
priming of neural networks due to the recent history of
activations increases probability of cooperation between
neurons encoding fragments of distributed representations.
• Imagination: some novel words come to my mind.
Many chains of phonemes activate in parallel both words and
non-words representations, depending on the strength of
synaptic connections.
• D. Simonton, “Creative thought as blind-variation and selective
retention: Combinatorial models of exceptional creativity,”
Physics of Life Reviews, vol. 7, p. 156–179, 2010.
• BVSR was invented by D. Campbell (1960), with blind variation
linking partial activations of primed neural circuits into chunks.
Models of creativity 3
• Filtering: the most active chunks, relevant in a given
context, win the competition filtering out less active chunks;
• Solution: results appear as spontaneous thoughts, solutions to
the problem, intentions for the next action, associations,
emotions.
chess board
domino
n
black
white
m
do
o
i
phonological reps
The Task
Given a description of some object (product, internet
site, service, organization, invention) create a
neologism that is easy to remember, word reminding
us of some qualities strongly associated with the object itself.
A real letter from a friend:
I am looking for a word that would capture the following
qualities: portal to new worlds of imagination and creativity,
a place where visitors embark on a journey discovering their inner
selves, awakening the Peter Pan within.
A place where we can travel through time and space (from the
origin to the future and back).
So, its about time, about space, infinite possibilities.
FAST!!! I need it sooooooooooooooooooooooon.
BrainGene preparation
Preparation phase:
• count the number of occurrences of every word
w ∈ W in a large corpus;
• split words into n-grams;
• put occurrence counts in the matrix at the position determined
by n-grams – it will tell us how likely are different n-grams;
• normalize word matrix.
This matrix describes lexical knowledge network, the space for
general word imagery.
BrainGene priming
Priming phase creates primed knowledge network:
• From the description of the product extract initial keywords for
•
•
•
•
•
•
•
phonological priming;
spread the activation to concepts and word-forms that are
strongly related (go, goes, went) … this forms a priming set;
count occurrences of every word w ∈ W in this set;
split words into n-grams;
put occurrence counts in the matrix at positions determined by
n-grams;
add words for negative priming (inhibit some words);
normalize priming word matrix;
add it to the general word matrix to form primed word matrix.
BrainGene filtering
Assumption: priming increases probability
of well-established associations in an additive way.
• Create new words from combinations of the most active ngrams in the final primed word matrix.
• Rank novel words using normalized counts for n-grams
(phonological plausibility).
• Words should not resemble closely any word in the dictionary
(including substrings, potentially offending words in various
languages, use the multilingual.sensegates.com service).
• Add novel words to the result set, retain only highest-ranking
These novel words have overlapping n-grams with priming ones.
BrainGene workflow
Primed knowledge network, rather complex probabilistic model,
including various linguistic peculiarities.
Some experiments
Search for good name for e-book reader (Kindle?):
• Priming set (after some stemming):
Acquir, collect, gather, air, light, lighter, lightest, paper, pocket,
portable, anyplace, anytime, anywhere, cable, detach, global,
globe, go, went, gone, going, goes, goer, journey, move,
moving, network, remote, road, roads, travel, wire, world,
book, data, informati, knowledge, librar, memor, news, word,
words, comfort, easi, easy, gentl, human, natural, personal,
computer, electronic, discover, educat, learn, read, reads,
reading, explor.
• Exclusion list (for inhibition):
aird, airin, airs, bookie, collectic, collectiv, globali, globed,
papere, papering, pocketf, travelog.
Results for novel words
Created word Word count and # domains in Google
• librazone
968
1
• inforizine
--• librable
188
-• bookists
216
-• inforld
30
-• newsests
3
-• memorld
78
1
• goinews
31
-• libravel
972
-• rearnews
8
-• booktion
49
-• newravel
7
-• lighbooks
1
-+ popular infooks, inforion, datnews, infonews, journics
Words: experiments
Portal to new worlds of imagination and creativity…
•
•
•
•
•
•
•
•
•
•
creatival (creativity, portal), used in creatival.com
creativery (creativity, discovery), creativery.com
discoverity = {disc, disco, discover, discovery, verity, creativity)
digventure ={dig, digital, venture, adventure} was new …
imativity (imagination, creativity);
infinitime (infinitive, time)
infinition (infinitive, imagination), already a company name
portravel (portal, travel); sportal (space, sport, portal), taken
timagination (time, imagination); timativity (time, creativity)
tivery (time, discovery); trime (travel, time)
Evaluation
Large number of experiments performed.
In most cases 50-80% of invented names are in use as
company names, site names, or product names.
Ex. for e-books: infoworld, inforion, bookist, boomation,
bookstion, cablects, cablector, dataction, datamation,
datnews, datmation, easnews, educatics, electroad, explobal,
goinmation, gonewsy, infordata, inforld, inforlds, inforion,
infornews, infortion, infonews, inforvel, infravel, newsion,
newstion, papnews, travelation, travelnews, wentnews ...
In February 2013 some of these internet domains were for
sale, for example cablead, easmation, educatnews, infovel,
pocketnews, wortion, with prices exceeding 1000 USD.
http://domomark.com/ service evaluates how good are
domain names, assigning scores in rather naive way.
Insights and brains
Brain solving problems that required insight vs. analytic way. E.M.
Bowden et al., New approaches to demystifying insight TCS 2005.
300 ms before insight a burst of gamma activity was observed in the
right anterior superior temporal gyrus (RH-aSTG) , interpreted by the
authors as „making connections across distantly related information
during comprehension ... that allow them to see connections that
previously eluded them”.
Conclusions & hopes
Creativity is not so mysterious, intuition, insight, creativity
in limited domain can be simulated at the human competence
level, opening a new vista in creativity research and suggesting
new neuroimaging experiments.
Even drastically simplified representations of lexical and semantic
knowledge are useful for applications in word games, query
precisiation, medical applications, common sense.
Brain processes analyzed at different levels are great inspiration!
Fruitful approach: approximations to knowledge reps in brain
networks: memory types/interactions, a priori knowledge,
spreading activation in networks, simulations of real brain
functions, graphs of consistent concepts, ontology-based
enhancements, connection to vector models, etc.
Approximate brain neurodynamics!
Thank
you
for
lending your
ears
...
Google: W. Duch => Papers/presentations/projects