Download Efficient Relevance Feedback for Content

Efficient Relevance Feedback for
Content-Based Image Retrieval
by Mining User Navigation Patterns
Ja-Hwung Su, Wei-Jyun Huang, Philip S. Yu, Fellow, IEEE,
and Vincent S. Tseng, Member, IEEE Abstract
IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING,
VOL. 23, NO. 3, MARCH 2011
1
OUTLINE
• INTRODUCTION
• RELATED WORK
– Query Reweighting
– Query Point Movement
– Query Expansion
• PROPOSED APPROACH
– Offline Knowledge Discovery
– Online Image Search
• EMPIRICAL EVALUATIONS
– Experimental Data
– Experimental Results
• CONCLUSION
2
INTRODUCTION
• Multimedia contents are growing explosively and the need for
multimedia retrieval is occurring more and more frequently in our
daily life.
• Typically, in the development of an image requisition system,
semantic image retrieval relies heavily on the related captions.
• Unfortunately, this kind of textual-based image retrieval always
suffers from two problems:
– high-priced manual annotation
– inappropriate automated annotation
3
INTRODUCTION
• High-priced manual annotation cost is prohibitive in coping with a
large-scale data set.
• Inappropriate automated annotation yields the distorted results for
semantic image retrieval.
• Content-Based Image Retrieval（CBIR） is to present an image
conceptually ,with a set of low-level visual features such as color,
texture, and shape.
4
INTRODUCTION
• These conventional approaches for image retrieval（CBIR） are
based on the computation of the similarity between the user’s query
and images via a query by example (QBE) system.
• But it is very difficult to optimize the retrieval quality of CBIR
within only one query process.
• The hidden problem is that the extracted visual features are too
diverse to capture the concept of the user’s query.
5
INTRODUCTION
•
To solve such problems, in the QBE system, the users can pick up
some preferred images to refine the image explorations iteratively.
• The feedback procedure, called Relevance Feedback (RF), repeats
until the user is satisfied with the retrieval results.
• Although a number of RF studies have been made on interactive
CBIR, they still incur some common problems：
– redundant browsing
– exploration convergence
6
INTRODUCTION
• First, in terms of redundant browsing, most existing RF methods
focus on how to earn the user’s satisfaction in one query process.
• Existing methods refine the query again and again by analyzing the
specific relevant images picked up by the users.
• Especially for the compound and complex images, the users might
go through a long series of feedbacks to obtain the desired images
using current RF approaches.
7
INTRODUCTION
• Fig. 1 illustrates the problem of exploration convergence.
Fig. 1. Motivating example for the problem of exploration convergence.
8
INTRODUCTION
• The involved problem, so-called visual diversity, is shown in
Fig. 2.
Fig. 2. Example of visual diversity.
9
RELATED WORK
Query Reweighting
• The notion behind QR is that, if the ith feature fi exists in positive
examples frequently, the system assigns the higher degree to fi .
• The diverse visual features extremely limit the effort of image
retrieval. Fig. 4 illustrates this limitation that although the search
area is continuously updated by reweighting the features, some
targets could be lost.
10
RELATED WORK
Query Reweighting
Fig. 3. Relevance feedback with generalized QR technique
11
RELATED WORK
Query Point Movement
• Another solution for enhancing the accuracy of image retrieval is
moving the query point toward the contour of the user’s preference
in feature space.
• QPM regards multiple positive examples as a new query point at
each feedback.
nr
nir
Rj
IRj
Qi  Qi  1      
j 1 nr
j 1 nir
(1)
• A specific measuring function indeed cannot cover all target groups
with various visual contents.
12
RELATED WORK
Query EXpansion
• Because QR and QPM cannot elevate the quality of RF, QEX has
been another hot technique in the solution space of RF recently.
• That is, straightforward search strategies, such as QR and QPM,
cannot completely cover the user’s interest spreading in the broad
feature space. Diverse results for the same concept are difficult to
obtain.
• For this reason, the modified version of MARS groups the similar
relevant points into several clusters, and selects good representative
points from these clusters to construct the multipoint query.
13
PROPOSED APPROACH
• As elaborated above, the critical issue of RF can be chiefly
summarized thus: how to achieve effective and efficient image
retrieval.
•
To deal with this issue, we describe how our proposed approach
NPRF integrates the discovered navigation patterns and three
RF techniques to achieve efficient and effective exploration of
images.
14
PROPOSED APPROACH
Fig. 4. Workflow of NPRF.
15
Offline Knowledge Discovery
Data Transformation
• If all positive images are considered for navigation pattern mining,
too many items make the frequent itemsets (navigation patterns)
hard to find. Also, the mining cost is expensive.
• The aim of data transformation is to generate Query Point
Dictionary (QPD) to reduce the kinds of items on the transaction
list.
• In this phase, the transformed log table is first generated by the
logged query sessions containing query session id, iteration number,
positive images, and visual query point number.
16
Offline Knowledge Discovery
Fig. 5.The query point dictionary of the proposed approach
17
Offline Knowledge Discovery
Fig. 6.The entity-relationship data model for partitioning the log data.
18
Offline Knowledge Discovery
Navigation Patterns Mining
• This stage focuses on the discovery of relations among the users’
browsing behaviors on RF.
• Basically, the frequent patterns mined from the user logs are
regarded as the useful browsing paths to optimize the search
direction on RF.
• In our NPRF approach, the users’ common interests can be
represented by the discovered frequent patterns (also called frequent
itemsets).
19
Offline Knowledge Discovery
Navigation Patterns Mining
• In this phase, the Apriori-like algorithm is performed to exploit
navigation patterns using the transformed data.
• The task for establishing the navigation model can be decomposed
into two steps:
– Step 1: Construction of the navigation transaction table
To exploit valuable navigation patterns, all query sessions in the
transformed log table are collected as the navigation-transaction
table.
– Step 2: Generation of navigation patterns
This operation concentrates on mining valuable navigation patterns to
facilitate online image retrieval.
20
Offline Knowledge Discovery
TABLE 1
Example of Navigation-Transaction Table
TABLE 2
Example of Navigation Patterns
21
Offline Knowledge Discovery
Pattern Indexing
• In this stage, we describe how to build the navigation pattern tree
with the discovered navigation patterns.
• A tree contains a number of navigation paths, and each node of the
paths stands for an item consisting of several visual query points.
Fig. 9. Example of navigation pattern trees.
22
Algorithm NPRFSearch
Query point generation
• The basic idea of this operation is to find the images not only with
the specific similarity function.
qpnew  {F 1, F 2...Fb}
where
1 i  b
(2)
Fi  { f 1, f 2,.... fd}
and
ft


1 x  k , f t x Fi
ft x
k
23
Algorithm NPRFSearch
Feature reweighting
• Consider a set of positive examples G  {g1 , g 2 ,...., g k } found by the
preceding query point qpold . Given that a set of { 1,  2,...., b} is
referred {F 1, F 2,...., Fb} . The new weight of the i th feature Fi is
b
defined as
 y 1y
i
wi 

b

b
z 1
y 1
y
z
(3)
where

d
k
 
x 1
j 1
( f jx  f jqpold ) 2
d
and 1  i  b
24
Algorithm NPRFSearch
Query expansion
• To keep an eye on the problem of exploration convergence, the
attempt of this stage is to cover all possible results by the relevant
patterns discovered.
• As a result, a set of positive query seeds is selected to be the start of
potential search paths.
• If the seed owns the maximum number of negative examples or
owns no positive example, it will be tokenized as a bad manner.
25
Algorithm NPRFSearch
Observation and discussion
• We adopt DFS-based search to hierarchically find the desired
images from query point level to image level.
• The experimental results show that a vertical search is better than a
horizontal search in this caseour proposed method.
• In other words, a query point represents a topic in the user’s
mind, containing a set of similar positive images.
26
EMPIRICAL EVALUATIONS
Observation and discussion
• Two major criteria, namely precision and coverage, are used to
measure the related experimental evaluations.
precision 
Coverage 
correct
100%
retrieved
ac _ correct
relevant
(4)
100%
(5)
27
EMPIRICAL EVALUATIONS
TABLE 3
The Experimental Data Sets
28
EMPIRICAL EVALUATIONS
Experimental Data
TABLE 4
The Experimental Parameter Settings
29
EMPIRICAL EVALUATIONS
Experimental Data
Fig. 13. The average precisions of different cl for
data set 3.
Fig. 14. The average precisions of different
numbers of log-transaction
for data set 3.
30
EMPIRICAL EVALUATIONS
Experimental Results
Fig. 15. The average precisions
of different minsup for data set 3.
Fig. 16. The average precisions
of different s for data set 3.
31
EMPIRICAL EVALUATIONS
Experimental Results
Fig. 17. The precisions of different approaches Fig. 18. The coverage of different approaches
for data set 7.
for data set 7.
32
EMPIRICAL EVALUATIONS
Experimental Results
TABLE 5
The Minimum Number of Feedbacks for Different Approaches to
Reach the Specific Precision 80 Percent
33
EMPIRICAL EVALUATIONS
Experimental Results
TABLE 6
The Precisions for Different Amounts of Data by NPRF
34
EMPIRICAL EVALUATIONS
Experimental Results
TABLE 7
The Execution Time for Different Amounts of Data by NPRF
35
EMPIRICAL EVALUATIONS
Experimental Results
Fig. 19. The resulting example for NPRF.
36
EMPIRICAL EVALUATIONS
Experimental Results
Fig. 20. The resulting example for QR.
Fig. 21. The resulting example for QPM.
37
CONCLUSION
• In summary, the main feature of NPRF is to efficiently optimize the
retrieval quality of interactive CBIR.
38