Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Efficient Relevance Feedback for Content-Based Image Retrieval by Mining User Navigation Patterns Ja-Hwung Su, Wei-Jyun Huang, Philip S. Yu, Fellow, IEEE, and Vincent S. Tseng, Member, IEEE Abstract IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING, VOL. 23, NO. 3, MARCH 2011 1 OUTLINE • INTRODUCTION • RELATED WORK – Query Reweighting – Query Point Movement – Query Expansion • PROPOSED APPROACH – Offline Knowledge Discovery – Online Image Search • EMPIRICAL EVALUATIONS – Experimental Data – Experimental Results • CONCLUSION 2 INTRODUCTION • Multimedia contents are growing explosively and the need for multimedia retrieval is occurring more and more frequently in our daily life. • Typically, in the development of an image requisition system, semantic image retrieval relies heavily on the related captions. • Unfortunately, this kind of textual-based image retrieval always suffers from two problems: – high-priced manual annotation – inappropriate automated annotation 3 INTRODUCTION • High-priced manual annotation cost is prohibitive in coping with a large-scale data set. • Inappropriate automated annotation yields the distorted results for semantic image retrieval. • Content-Based Image Retrieval(CBIR) is to present an image conceptually ,with a set of low-level visual features such as color, texture, and shape. 4 INTRODUCTION • These conventional approaches for image retrieval(CBIR) are based on the computation of the similarity between the user’s query and images via a query by example (QBE) system. • But it is very difficult to optimize the retrieval quality of CBIR within only one query process. • The hidden problem is that the extracted visual features are too diverse to capture the concept of the user’s query. 5 INTRODUCTION • To solve such problems, in the QBE system, the users can pick up some preferred images to refine the image explorations iteratively. • The feedback procedure, called Relevance Feedback (RF), repeats until the user is satisfied with the retrieval results. • Although a number of RF studies have been made on interactive CBIR, they still incur some common problems: – redundant browsing – exploration convergence 6 INTRODUCTION • First, in terms of redundant browsing, most existing RF methods focus on how to earn the user’s satisfaction in one query process. • Existing methods refine the query again and again by analyzing the specific relevant images picked up by the users. • Especially for the compound and complex images, the users might go through a long series of feedbacks to obtain the desired images using current RF approaches. 7 INTRODUCTION • Fig. 1 illustrates the problem of exploration convergence. Fig. 1. Motivating example for the problem of exploration convergence. 8 INTRODUCTION • The involved problem, so-called visual diversity, is shown in Fig. 2. Fig. 2. Example of visual diversity. 9 RELATED WORK Query Reweighting • The notion behind QR is that, if the ith feature fi exists in positive examples frequently, the system assigns the higher degree to fi . • The diverse visual features extremely limit the effort of image retrieval. Fig. 4 illustrates this limitation that although the search area is continuously updated by reweighting the features, some targets could be lost. 10 RELATED WORK Query Reweighting Fig. 3. Relevance feedback with generalized QR technique 11 RELATED WORK Query Point Movement • Another solution for enhancing the accuracy of image retrieval is moving the query point toward the contour of the user’s preference in feature space. • QPM regards multiple positive examples as a new query point at each feedback. nr nir Rj IRj Qi Qi 1 j 1 nr j 1 nir (1) • A specific measuring function indeed cannot cover all target groups with various visual contents. 12 RELATED WORK Query EXpansion • Because QR and QPM cannot elevate the quality of RF, QEX has been another hot technique in the solution space of RF recently. • That is, straightforward search strategies, such as QR and QPM, cannot completely cover the user’s interest spreading in the broad feature space. Diverse results for the same concept are difficult to obtain. • For this reason, the modified version of MARS groups the similar relevant points into several clusters, and selects good representative points from these clusters to construct the multipoint query. 13 PROPOSED APPROACH • As elaborated above, the critical issue of RF can be chiefly summarized thus: how to achieve effective and efficient image retrieval. • To deal with this issue, we describe how our proposed approach NPRF integrates the discovered navigation patterns and three RF techniques to achieve efficient and effective exploration of images. 14 PROPOSED APPROACH Fig. 4. Workflow of NPRF. 15 Offline Knowledge Discovery Data Transformation • If all positive images are considered for navigation pattern mining, too many items make the frequent itemsets (navigation patterns) hard to find. Also, the mining cost is expensive. • The aim of data transformation is to generate Query Point Dictionary (QPD) to reduce the kinds of items on the transaction list. • In this phase, the transformed log table is first generated by the logged query sessions containing query session id, iteration number, positive images, and visual query point number. 16 Offline Knowledge Discovery Fig. 5.The query point dictionary of the proposed approach 17 Offline Knowledge Discovery Fig. 6.The entity-relationship data model for partitioning the log data. 18 Offline Knowledge Discovery Navigation Patterns Mining • This stage focuses on the discovery of relations among the users’ browsing behaviors on RF. • Basically, the frequent patterns mined from the user logs are regarded as the useful browsing paths to optimize the search direction on RF. • In our NPRF approach, the users’ common interests can be represented by the discovered frequent patterns (also called frequent itemsets). 19 Offline Knowledge Discovery Navigation Patterns Mining • In this phase, the Apriori-like algorithm is performed to exploit navigation patterns using the transformed data. • The task for establishing the navigation model can be decomposed into two steps: – Step 1: Construction of the navigation transaction table To exploit valuable navigation patterns, all query sessions in the transformed log table are collected as the navigation-transaction table. – Step 2: Generation of navigation patterns This operation concentrates on mining valuable navigation patterns to facilitate online image retrieval. 20 Offline Knowledge Discovery TABLE 1 Example of Navigation-Transaction Table TABLE 2 Example of Navigation Patterns 21 Offline Knowledge Discovery Pattern Indexing • In this stage, we describe how to build the navigation pattern tree with the discovered navigation patterns. • A tree contains a number of navigation paths, and each node of the paths stands for an item consisting of several visual query points. Fig. 9. Example of navigation pattern trees. 22 Algorithm NPRFSearch Query point generation • The basic idea of this operation is to find the images not only with the specific similarity function. qpnew {F 1, F 2...Fb} where 1 i b (2) Fi { f 1, f 2,.... fd} and ft 1 x k , f t x Fi ft x k 23 Algorithm NPRFSearch Feature reweighting • Consider a set of positive examples G {g1 , g 2 ,...., g k } found by the preceding query point qpold . Given that a set of { 1, 2,...., b} is referred {F 1, F 2,...., Fb} . The new weight of the i th feature Fi is b defined as y 1y i wi b b z 1 y 1 y z (3) where d k x 1 j 1 ( f jx f jqpold ) 2 d and 1 i b 24 Algorithm NPRFSearch Query expansion • To keep an eye on the problem of exploration convergence, the attempt of this stage is to cover all possible results by the relevant patterns discovered. • As a result, a set of positive query seeds is selected to be the start of potential search paths. • If the seed owns the maximum number of negative examples or owns no positive example, it will be tokenized as a bad manner. 25 Algorithm NPRFSearch Observation and discussion • We adopt DFS-based search to hierarchically find the desired images from query point level to image level. • The experimental results show that a vertical search is better than a horizontal search in this caseour proposed method. • In other words, a query point represents a topic in the user’s mind, containing a set of similar positive images. 26 EMPIRICAL EVALUATIONS Observation and discussion • Two major criteria, namely precision and coverage, are used to measure the related experimental evaluations. precision Coverage correct 100% retrieved ac _ correct relevant (4) 100% (5) 27 EMPIRICAL EVALUATIONS TABLE 3 The Experimental Data Sets 28 EMPIRICAL EVALUATIONS Experimental Data TABLE 4 The Experimental Parameter Settings 29 EMPIRICAL EVALUATIONS Experimental Data Fig. 13. The average precisions of different cl for data set 3. Fig. 14. The average precisions of different numbers of log-transaction for data set 3. 30 EMPIRICAL EVALUATIONS Experimental Results Fig. 15. The average precisions of different minsup for data set 3. Fig. 16. The average precisions of different s for data set 3. 31 EMPIRICAL EVALUATIONS Experimental Results Fig. 17. The precisions of different approaches Fig. 18. The coverage of different approaches for data set 7. for data set 7. 32 EMPIRICAL EVALUATIONS Experimental Results TABLE 5 The Minimum Number of Feedbacks for Different Approaches to Reach the Specific Precision 80 Percent 33 EMPIRICAL EVALUATIONS Experimental Results TABLE 6 The Precisions for Different Amounts of Data by NPRF 34 EMPIRICAL EVALUATIONS Experimental Results TABLE 7 The Execution Time for Different Amounts of Data by NPRF 35 EMPIRICAL EVALUATIONS Experimental Results Fig. 19. The resulting example for NPRF. 36 EMPIRICAL EVALUATIONS Experimental Results Fig. 20. The resulting example for QR. Fig. 21. The resulting example for QPM. 37 CONCLUSION • In summary, the main feature of NPRF is to efficiently optimize the retrieval quality of interactive CBIR. 38