Download Multimedia Databases

Multimedia Databases
Wolf-Tilo Balke
Silviu Homoceanu
Institut für Informationssysteme
Technische Universität Braunschweig
http://www.ifis.cs.tu-bs.de
0. Organizational Issues
• Lecture
– 22.10.2009 – 04.02.2010
– 15:00-17:15 (3 lecture hours with a break)
– Exercises, detours, and home work
discussion integrated into lecture
• 4 Credits
• Exams
– Oral exam
– 50% of exercise points needed
to be eligible for the exam
Multimedia Databases – Wolf-Tilo Balke – Institut für Informationssysteme – TU Braunschweig
2
0. Organizational Issues
• Recommended literature
– Schmitt: Ähnlichkeitssuche in
Multimedia-Datenbanken,
Oldenbourg, 2005
– Steinmetz: Multimedia-Technologie:
Grundlagen, Komponenten und
Systeme, Springer, 1999
Relational Database Systems 1 – Wolf-Tilo Balke – Institut für Informationssysteme – TU Braunschweig
3
0. Organizational Issues
– Castelli/Bergman: Image Databases,
Wiley, 2002
– Khoshafian/Baker: Multimedia and
Imaging Databases, Morgan
Kaufmann, 1996
– Sometimes: original papers (on our Web page)
Relational Database Systems 1 – Wolf-Tilo Balke – Institut für Informationssysteme – TU Braunschweig
4
0. Organizational Issues
• Course Web page
– http://www.ifis.cs.tu-bs.de/teaching/ws-0910/mmdb
– Contains slides, excercises,
related papers and a video
of the lecture
– Any questions? Just drop
us an email…
Relational Database Systems 1 – Wolf-Tilo Balke – Institut für Informationssysteme – TU Braunschweig
5
1. Introduction
1. Introduction
1.1 What are multimedia databases?
1.2 Multimedia database applications
1.3 Evaluation of retrieval techniques
Multimedia Databases– Wolf-Tilo Balke – Institut für Informationssysteme – TU Braunschweig
6
1.1 Multimedia Databases
• What are multimedia databases (MMDB)?
– Databases + multimedia = MMDB
• Key words: databases and multimedia
• We already know databases, so what is
multimedia?
Multimedia Databases– Wolf-Tilo Balke – Institut für Informationssysteme – TU Braunschweig
7
1.1 Basic Definitions
• Multimedia
– The concept of multimedia expresses the
integration of different digital media types
– The integration is usually performed in a document
– Basic media types are text, image, vector graphics,
audio and video
Multimedia Databases– Wolf-Tilo Balke – Institut für Informationssysteme – TU Braunschweig
8
1.1 Data Types
• Text
– Text data, Spreadsheets, E-Mail, …
• Image
– Photos (Bitmaps),Vector graphics, CAD, …
• Audio
– Speech- and music records, annotations, wave files,
MIDI, MP3, …
• Video
– Dynamical image record, frame-sequences,
MPEG, AVI, …
Multimedia Databases– Wolf-Tilo Balke – Institut für Informationssysteme – TU Braunschweig
9
1.1 Documents
• Earliest definition of information retrieval:
“Documents are logically interdependent
digitally encoded texts“
• Extension to multimedia documents allows the
additional integration of other media types as
images, audio or video
Multimedia Databases– Wolf-Tilo Balke – Institut für Informationssysteme – TU Braunschweig
10
1.1 Documents
• Document types
– Media objects are documents which are of
only one type (not necessarily text)
– Multimedia objects are general documents
which allow an arbitrary combination of
different types
• Multimedia data is transferred through the
use of a medium
Multimedia Databases– Wolf-Tilo Balke – Institut für Informationssysteme – TU Braunschweig
11
1.1 Basic Definitions
• Medium
– A medium is a carrier of information in a
communication connection
– It is independent of the transported information
– The used medium can also be
changed during information
transfer
Multimedia Databases– Wolf-Tilo Balke – Institut für Informationssysteme – TU Braunschweig
12
1.1 Medium Example
• Book
– Communication between author
and reader
– Independent from content
– Hierarchically built on text
and images
– Reading out loud represents
medium change to sound/audio
Multimedia Databases– Wolf-Tilo Balke – Institut für Informationssysteme – TU Braunschweig
13
1.1 Medium Classification
• Based on receiver type
– Visual/optical medium
– Acoustic mediums
– Haptical medium – through tactile senses
– Olfactory medium – through smell
– Gustatory medium – through taste
• Based on time
– Dynamic
– Static
Multimedia Databases– Wolf-Tilo Balke – Institut für Informationssysteme – TU Braunschweig
14
1.1 Multimedia Databases
• We now have seen…
– What multimedia is
– And how it is transported (through some medium)
• But… why do we need databases?
– Most important operations of databases are data
storage and data retrieval
Multimedia Databases– Wolf-Tilo Balke – Institut für Informationssysteme – TU Braunschweig
15
1.1 Multimedia Databases
• Persistent storage of multimedia data, e.g.:
– Text documents
– Vector graphics, CAD
– Images, audio, video
• Content-based retrieval
– Efficient content based search
– Standardization of meta-data (e. g., MPEG-7, MPEG-21)
Multimedia Databases– Wolf-Tilo Balke – Institut für Informationssysteme – TU Braunschweig
16
1.1 Multimedia Databases
• Stand-alone vs. database storage model?
– Special retrieval functionality as well as corresponding
optimization can be provided in both cases…
– But in the second case we also get the general
advantages of databases
•
•
•
•
•
Declarative query language
Orthogonal combination of the query functionality
Query optimization, Index structures
Transaction management, recovery
...
Multimedia Databases– Wolf-Tilo Balke – Institut für Informationssysteme – TU Braunschweig
17
1.1 Historical Overview
1960
Retrieval procedures for text documents
(Information Retrieval)
1970
Relational Databases and SQL
1980
Presence of multimedia objects intensifies
1990
SQL-92 introduces BLOBs
First Multimedia-Databases
2000
Multimedia Databases– Wolf-Tilo Balke – Institut für Informationssysteme – TU Braunschweig
18
1.1 Commercial Systems
• Relational Databases use the data type BLOB
(binary large object)
– Uninterpreted data
– Retrieval through metadata like e.g., file name, size,
author, …
• Object-relational extensions feature
enhanced retrieval functionality
– Semantic search
– IBM DB2 Extenders, Oracle Cartridges, …
– Integration in DB through UDFs, UDTs, Stored
Procedures, …
Multimedia Databases– Wolf-Tilo Balke – Institut für Informationssysteme – TU Braunschweig
19
1.1 Requirements
• Requirements for multimedia databases
(Christodoulakis, 1985)
– Classical database functionality
– Maintenance of unformatted data
– Consideration of
special storage and
presentation devices
Multimedia Databases– Wolf-Tilo Balke – Institut für Informationssysteme – TU Braunschweig
20
1.1 Requirements
• To comply with these requirements the following
aspects need to be considered
– Software architecture – new or extension of
existing databases?
– Content addressing – identification of the objects
through content-based features
– Performance – improvements using indexes,
optimization, etc.
Multimedia Databases– Wolf-Tilo Balke – Institut für Informationssysteme – TU Braunschweig
21
1.1 Requirements
– User interface – how should the user interact with
the system? Separate structure from content!
– Information extraction – (automatic) generation
of content-based features
– Storage devices – very large storage capacity,
redundancy control and compression
– Information retrieval – integration of some
extended search functionality
Multimedia Databases– Wolf-Tilo Balke – Institut für Informationssysteme – TU Braunschweig
22
1.1 Retrieval
• Retrieval means the choice between data
objects which can be based on…
– a SELECT condition (exact match)
– or a defined similarity connection
(best match)
• Retrieval may cover the delivery
of the results to the user, too
Multimedia Databases– Wolf-Tilo Balke – Institut für Informationssysteme – TU Braunschweig
23
1.1 Retrieval
• Closer look at the search functionality
– „Semantic“ search functionality
– Orthogonal integration of classical and extended
functionality
– Search does not directly access the media objects
– Extraction, normalization and indexing of contentbased features
– Meaningful similarity/distance measures
Multimedia Databases– Wolf-Tilo Balke – Institut für Informationssysteme – TU Braunschweig
24
1.1 Content-based Retrieval
• “Retrieve all images showing a sunset !”
• What exactly do these images have in common?
Multimedia Databases– Wolf-Tilo Balke – Institut für Informationssysteme – TU Braunschweig
25
1.1 Schematic View
• Usually 2 main steps
– Example: image databases
Image
collection
Digitization
Image analysis
and feature
extraction
Image
database
Creating the database
Querying the database
Image
query
Digitization
Image analysis
and feature
extraction
Similarity
search
Search result
Multimedia Databases– Wolf-Tilo Balke – Institut für Informationssysteme – TU Braunschweig
26
1.1 Detailed View
Query
Result
3. Query
preparation
5. Result preparation
Query plan & feature values
Result data
4. Similarity computation & query processing
Feature values
Raw data
2. Extraction
of features
Raw & relational data
MM-Database
1. Insert into the
database
MM-Objects + relational data
Multimedia Databases– Wolf-Tilo Balke – Institut für Informationssysteme – TU Braunschweig
27
1.1 More Detailed View
Result
Query preparation
Normalization
Segmentation
Feature extraction
Optimization
Feature values
Query plan
Similarity computation
Relevance feedback
Query
Result preparation
Medium transformation
Format transformation
Result data
Query processing
Feature values
MM Database
MM-Database
Feature index
BLOBs/CLOBs
Relational DB
Metadata
Profile
Structure data
Feature extraction
Pre-processing
Feature recognition
Feature preparation
Decomposition
Normalization
Segmentation
MM-Objects
Relational data
Multimedia Databases– Wolf-Tilo Balke – Institut für Informationssysteme – TU Braunschweig
28
1.2 Applications
• Lots of multimedia content on the Web
– Social networking e.g., Facebook, MySpace, Hi5, Orkut,
etc.
– Photo sharing e.g., Flickr, Photobucket, Imeem, Picasa,
etc.
– Video sharing e.g.,YouTube, Megavideo, Metacafe,
blip.tv, Liveleak, etc.
Multimedia Databases– Wolf-Tilo Balke – Institut für Informationssysteme – TU Braunschweig
29
1.2 Applications
• Cameras are everywhere
– In London “there are at least
500,000 cameras in the city, and
one study showed that in a single
day a person could expect to be filmed 300 times”
Multimedia Databases– Wolf-Tilo Balke – Institut für Informationssysteme – TU Braunschweig
30
1.2 Applications
• Picasa face recognition
Multimedia Databases– Wolf-Tilo Balke – Institut für Informationssysteme – TU Braunschweig
31
1.2 Applications
• Picasa, face recognition example
Multimedia Databases– Wolf-Tilo Balke – Institut für Informationssysteme – TU Braunschweig
32
1.2 Applications
• Picasa, learning
phase
Multimedia Databases– Wolf-Tilo Balke – Institut für Informationssysteme – TU Braunschweig
33
1.2 Applications
• Picasa example
Multimedia Databases– Wolf-Tilo Balke – Institut für Informationssysteme – TU Braunschweig
34
1.2 Sample Scenario
• Consider a police investigation of
a large-scale drug operation
• Possible generated data:
– Video data captured by surveillance cameras
– Audio data captured
– Image data consisting of still photographs taken by
investigators
– Structured relational data
containing background
information
– Geographic information
system data
Multimedia Databases– Wolf-Tilo Balke – Institut für Informationssysteme – TU Braunschweig
35
1.2 Sample Scenario
• Possible queries
– Image query by example:
police officer has a photograph
and wants to find the identity
of the person in the picture
• Query: “retrieve all images from the image library in which
the person appearing in the (currently displayed)
photograph appears”
– Image query by keywords: police officer wants to
examine pictures of “Tony Soprano”
• Query: “retrieve all images from the image library in which
‘Tony Soprano’ appears"
Multimedia Databases– Wolf-Tilo Balke – Institut für Informationssysteme – TU Braunschweig
36
1.2 Sample Scenario
– Video Query:
• Query: “Find all video segments in which
Jerry appears”
• By examining the answer of the above query,
the police officer hopes to find other people
who have previously interacted with the victim
Multimedia Databases– Wolf-Tilo Balke – Institut für Informationssysteme – TU Braunschweig
37
1.2 Sample Scenario
– Heterogeneous Multimedia Query:
• Find all individuals who have been photographed with “Tony
Soprano” and who have been convicted of attempted
murder in New Jersey and who have recently had electronic
fund transfers made into their bank accounts from ABC
Corp.
Multimedia Databases– Wolf-Tilo Balke – Institut für Informationssysteme – TU Braunschweig
38
1.2 Characteristics
• Basic difference
– Static
• High number of search queries (read access), few modifications of the data
state
– Dynamic
• Often modifications of the data state
– Active
• Database functionality lead to application operations
– Passive
• Database reacts only at requests from outside
– Standard search
• Queries are answered through the use of metadata e.g., Google-image
search
– Retrieval functionality
• Content based search on the multimedia repository e.g., Picasa face
recognition
Multimedia Databases– Wolf-Tilo Balke – Institut für Informationssysteme – TU Braunschweig
39
1.2 Example
• Passive static retrieval
– Art historical use case
Multimedia Databases– Wolf-Tilo Balke – Institut für Informationssysteme – TU Braunschweig
40
1.2 Example
– Possible hit in a multimedia database
Multimedia Databases– Wolf-Tilo Balke – Institut für Informationssysteme – TU Braunschweig
41
1.2 Example
• Active dynamic retrieval
– Wetter warning through evaluation of satellite photos
Extraction
Typhoon-Warning
for the Philippines
Multimedia Databases– Wolf-Tilo Balke – Institut für Informationssysteme – TU Braunschweig
42
1.2 Example
• Standard search
– Queries are answered through the use of metadata
e.g., Google-image search
Multimedia Databases– Wolf-Tilo Balke – Institut für Informationssysteme – TU Braunschweig
43
1.2 Example
• Retrieval functionality
– Content based e.g., Picasa face recognition
Multimedia Databases– Wolf-Tilo Balke – Institut für Informationssysteme – TU Braunschweig
44
1.3 Towards Evaluation
• Basic evaluation of retrieval techniques
– Efficiency of the system
• Efficient utilization of system resources
• Scalable also over big collections
– Effectivity of the retrieval process
• High quality of the result
• Meaningful usage of the system
– Weighting is application specific
Multimedia Databases– Wolf-Tilo Balke – Institut für Informationssysteme – TU Braunschweig
45
1.3 Evaluating Efficiency
• Characteristic values to measure
efficiency are e.g.:
– Memory usage
– CPU-time
– Number of I/O-Operations
– Response time
• Depends on the (Hardware-) environment
• Goal: the system should be efficient enough!
Multimedia Databases– Wolf-Tilo Balke – Institut für Informationssysteme – TU Braunschweig
46
1.3 Evaluating Effectivity
• Measuring effectivity is more difficult and always
depending on the query
• Goal: define some query-dependent evaluation
measures!
– Objective quality metrics
– Result evaluation based on the query
– Independent from used querying interface
and retrieval procedure
– Leads to comparability of different systems/algorithms
Multimedia Databases– Wolf-Tilo Balke – Institut für Informationssysteme – TU Braunschweig
47
1.3 Evaluating Effectivity
• Effectivity can be measured regarding some
explicit query
– Main focus on evaluating the behavior of the system
with respect to a query
– Relevance of the result set
• But effectivity also needs to consider implicit
information needs
– Main focus on evaluating the usefulness, usability and
user friendliness of the system
Multimedia Databases– Wolf-Tilo Balke – Institut für Informationssysteme – TU Braunschweig
48
1.3 Relevance
• To evaluate a retrieval system over some query,
each document will be classified binary as
relevant or irrelevant with respect to the
query
– This classification is performed by “experts”
– The response of the system to the query will be
compared to this manual classification
• Compare the obtained response with the “ideal” result
Multimedia Databases– Wolf-Tilo Balke – Institut für Informationssysteme – TU Braunschweig
49
1.3 Usefulness (Pertinence)
• Subjective measure estimating to what degree
the information need of the user is satisfied
– Difficult to measure (empirical studies)
– Questionable instrument for comparing
procedures/systems
• Attention: useful documents can be irrelevant
when considering the query (serendipity)
• In this lecture:
explicit query evaluation measures only!
Multimedia Databases– Wolf-Tilo Balke – Institut für Informationssysteme – TU Braunschweig
50
1.3 Involved Sets
collection
searched for
(= relevant)
found
(= query result)
Multimedia Databases– Wolf-Tilo Balke – Institut für Informationssysteme – TU Braunschweig
51
1.3 False Positives
• Irrelevant documents, classified as relevant by the
system
– False alarms, false drops, …
• Needlessly increase the result set
• Usually inevitable (ambiguity)
• Can be easily eliminated by the user
Multimedia Databases– Wolf-Tilo Balke – Institut für Informationssysteme – TU Braunschweig
52
1.3 False Negatives
• Relevant documents classified by the system as
irrelevant
– False dismissals
• Dangerous, since they can not be detected easily
by the user
– Does the collection contain “better” documents?
– False positives are usually not as bad as false negatives
Multimedia Databases– Wolf-Tilo Balke – Institut für Informationssysteme – TU Braunschweig
53
1.3 Remaining Sets
• Correct positives (correct alarms)
– All documents correctly classified by the
system as relevant
• Correct negatives (correct dismissals)
– All documents correctly classified by the system as
irrelevant
• All sets are disjunctive and their reunion is the
entire document collection
Multimedia Databases– Wolf-Tilo Balke – Institut für Informationssysteme – TU Braunschweig
54
1.3 Overview
Systemevaluation
relevant
irrelevant
relevant
ca
fd
irrelevant
fa
cd
Userevaluation
Multimedia Databases– Wolf-Tilo Balke – Institut für Informationssysteme – TU Braunschweig
55
1.3 Interpretation
• {Relevant results} = fd + ca
• {Retrieved results} = ca + fa
collection
fd
ca
searched for
fa
found
cd
Multimedia Databases– Wolf-Tilo Balke – Institut für Informationssysteme – TU Braunschweig
56
1.3 Precision
• Precision measures the ratio of correctly
returned documents relative to all returned
documents
– P = ca / (ca + fa)
• Value between [0, 1]
(1 representing the best value)
• High number of false alarms mean
worse results
Multimedia Databases– Wolf-Tilo Balke – Institut für Informationssysteme – TU Braunschweig
57
1.3 Recall
• Recall measures the ratio of correctly returned
documents relative to all relevant documents
– R = ca / (ca + fd)
• Value between [0, 1]
(1 representing the best value)
• High number of false drops mean
worse results
Multimedia Databases– Wolf-Tilo Balke – Institut für Informationssysteme – TU Braunschweig
58
1.3 Precision-Recall Analysis
• Both measures only make sense, if considered at
the same time
– E.g., get perfect recall by simply returning all
documents, but then the precision is extremely low…
• Can be balanced by tuning the system
– E.g., smaller result sets lead to better precision rates
at the cost of recall
• Usually the average precision-recall of more
queries is considered (macro evaluation)
Multimedia Databases– Wolf-Tilo Balke – Institut für Informationssysteme – TU Braunschweig
59
1.3 Actual Evaluation
• Alarms (returned elements) can be easily
divided in ca and fa
– Precision is easy to calculate
• Dismissals (not returned elements) are not so
trivial to divide in cd und fd, because the entire
collection has to be classified
– Recall is difficult to calculate
• Standardized Benchmarks
– Provided connections and queries
– Annotated result sets
Multimedia Databases– Wolf-Tilo Balke – Institut für Informationssysteme – TU Braunschweig
60
1.3 TREC
• Text REtrieval Conference
• De-Facto-Standard since 1992
• Establish average precision for 11 fixed recall
points (0; 0,1; 0,2; …; 1) according to defined
procedures (trec_eval)
• Different tracks, extended also for video data,
´Web retrieval and Question Answering
• Other initiatives: e.g., CLEF (cross-language
retrieval) or INEX (XML-Documents)
Multimedia Databases– Wolf-Tilo Balke – Institut für Informationssysteme – TU Braunschweig
61
1.3 Example
Query
fa ca fd cd
P
R
Q1
8
2
6
4
0,2
0,25
Q2
2
8
2
8
0,8
0,8
0,5
0,525
Average
Multimedia Databases– Wolf-Tilo Balke – Institut für Informationssysteme – TU Braunschweig
62
1.3 Representation
• Precision-Recall-Graphs
Average precision of
the system 3 at a
recall-level of 0,2
System 1
Which system is the
best?
System 2
System 3
What is more
important: recall or
precision?
Multimedia Databases– Wolf-Tilo Balke – Institut für Informationssysteme – TU Braunschweig
63
Next lecture
•
•
•
•
Retrieval of images by color
Introduction to color spaces
Color histograms
Matching
Multimedia Databases – Wolf-Tilo Balke – Institut für Informationssysteme – TU Braunschweig
64