Download View slides

Visiwords
John Tait
Chief Scientific
Officer
Warning
• A few half formed ideas from the world of
image and video indexing which may be of
interest to MT people
• Not original ideas (apart from I think the
link)
• In fact a line of work which derives
originally from MT
The nub
• Unsupervised Clustering of bundles of
features
– Colour, texture from image segments
– Words, phrases from sentences or paragraphs ?
• Associate these bundles with
“translations” by supervised machine
learning
– Categorised images
– Parallel texts
Origins
• “Matching Words and Pictures”: Barnard,
Duygulu, Forsyth, de Freitas, Blei, Jordan.
Journal of Machine Learning Research 3
(2003) 1107-1135
• “Image Classification Using Hybrid Neural
Networks” Tsai, McGarry and Tait.
Proceedings of the 26th ACM SIGIR
Conference on Research and Development
in Information Retrieval (SIGIR 2003),
Toronto, July, 2003. pp 431-432.
More or less general
clusters
General Concepts
Specific Concepts
Visiwords
• Derived from Visiterms
– These feature cluster nodes
– A notion of an area of the “semantic field”
– Remember these are colour, texture etc. for
an area of an image …. No relation to
language … or at least a very deep one
Matching
L1 General Concepts
Examples
L1 Specific Concepts
L2 General Concepts
L2 Specific
Concepts
Fast Forward to 2009
• Better statistical models tuned to the data
• Much Bigger vocabularies of words
categories
• … and lots of other advances
A question
• Is there anything like this current MT
research ?
Concluding remarks
• I’m surprised this worked at all
– Why should image data be coherent and
cohesive ?
• But text is !!!
• Is this a better way to deal with unknown
and changing vocabulary
Some other references
•
•
A Correlation Approach for Automatic Image Annotation Hardoon, D.,
Saunders, C., Szedmak, S. and Shawe-Taylor, J. (2006) A Correlation
Approach for Automatic Image Annotation. In: Second International
Conference on Advanced Data Mining and Applications, ADMA 2006,
August, Xi'an, China.
Kucuktunc, O., Sevil, S. G., Tosun, A. B., Zitouni, H., Duygulu, P., and
Can, F. 2008. Tag Suggestr: Automatic Photo Tag Expansion Using Visual
Information for Photo Sharing Websites. In Proceedings of the 3rd
international Conference on Semantic and Digital Media Technologies:
Semantic Multimedia (Koblenz, Germany, December 03 - 05, 2008). D.
Duke, L. Hardman, A. Hauptmann, D. Paulus, and S. Staab, Eds.
Lecture Notes In Computer Science, vol. 5392. Springer-Verlag, Berlin,
Heidelberg, 61-73. DOI= http://dx.doi.org/10.1007/978-3-540-922353_7