AN INTELLIGENT SPARSE VISUAL FEATURE
DESCRIPTORS FOR CONTENT-BASED IMAGE RETRIEVAL
Introduction
• With the rapid development of digital cameras and social
media, image libraries have exponentially grown.
• Billions of images in the form of personal collections,
medical collection, enterprise collection, archives & Internet.
• Unlabeled collection of digital images.
• Visual information retrieval.
Research area
• Major Research Area
• Content-based image retrieval
• Research Pockets in Major Research Area
• Feature extraction
• Clustering (i.e. K-means, K-means++, DBScan, etc. )
• Image classification (i.e. Decision trees, KNN,SVM, ANN, DCNN etc.)
• Deep learning
• Relevance feedback
• Similarity measure (i.e. Manhattan distance, Euclidean distance etc.)
Problem statement
• Large image and video databases like Flicker, ImageNet, ImageCLEF
and youtube.
• High computational cost.
• Issue of Semantic gap;
which addresses:
• the lack of output verification.
• neighborhood similarity avoidance.
• Retrieving image by text (i.e. image annotation) such as:
• Too much responsibility on the end-user.
• Large volumes of databases.
• Valid only for one language.
• Lost of spatial information.
Problem statement (Cont.)
Cloud
Sky
Mountain
Mountain Forest
Water
Water People
Forest Lake Street
Sky Cloud Cloud
Mountain
Mountain
Peoples
Beach Water Water Beach
• Figure: Semantic gap-Corel Images of different categories (Mountains and Beach) with close visual appearance.
Problem statement (Cont.)
• Figure: Result of image retrieval using text-based image annotation.
BoVW based image representation
Selected standard image benchmarks
Image Benchmark Name Total Classes Total Images
Corel-1000 10 1000
Corel-1500 15 1500
Caltech-256 256 30607
Fifteen Scene 15 4485
Ground Truth 22 1109
Research objectives
The objectives of proposed research are;
1. Provide an efficient image representation for feature
extraction and image retrieval.
2. Avoid image segmentation for CBIR systems.
3. Provide rotation-invariant and scale-invariant features.
4. Reduce the off-line and real-time feature computation times.
5. Procedures for adding the spatial information to the inverted
index of the standard BoVW model.
6. Reduction of semantic gap between the high-level semantic
concepts and low-level features of the image.
7. Automatic image annotation (AIA) for CBIR.
Proposed approaches for CBIR
We have proposed three approaches for effective content-based image
retrieval that are as follows;
1. CBIR-based on a combination of local and global histograms of
visual words.
2. CBIR-based on rectangular spatial histograms of visual words.
3. CBIR and semantic automatic image annotation based on weighted
average of spatial triangular histograms of visual words.
First approach: CBIR-based on a combination of local
and global histograms of visual words
Figure: Image (on left hand side) is presenting the procedure for computation of global histogram, while
image (on right hand side) is presenting the procedure for the extraction of local histogram.
First approach: CBIR-based on a combination of local
and global histograms of visual words
Figure: Global and local regions of images from four different semantic classes of Corel-A image benchmark.
Figure: Image (a) is presenting the procedure for computation of global histogram, while image (b) is
presenting the procedure for the extraction of local histogram.
First approach: contributions
1. Representation of the image as a combination of local and
global histograms of visual words in order to enhance the
performance of CBIR.
2. Procedure for adding the spatial information to the inverted
index of the standard BoVW model.
3. Reduction of semantic gap between the high-level semantic
concepts and low-level features of the image.
Block diagram: CBIR-based on a combination of local
and global histograms of visual words
Experimental study and results
Figure: MAP as a function of vocabulary size on Corel-1000 image benchmark.
Precision & Recall on Corel-1000 image benchmark
PR curve measure on Corel-1000 image benchmark
Result of top-20 image retrieval on Corel-1000 image
benchmark
Figure: Retrieval result shows reduction of semantic gap for the semantic class “Buses”.
Result of top-20 image retrieval on Corel-1000 image
benchmark
Figure: Retrieval result shows reduction of semantic gap for the semantic class “Beach”.
Performance on Caltech-256 image benchmark
Performance on Ground truth image benchmark
Computational cost (time in seconds)
Second approach: CBIR-based on rectangular
spatial histograms of visual words
Figure: Corel images of different classes with a semantic rectangular relationship.
Figure: Image representation in the form of rectangular spatial histograms of visual words.
Second approach: contributions
1. Addition of spatial information to the inverted index of the
standard BoVW model.
2. Image representation in the form of rectangular spatial histograms
of visual words.
3. Reduction of semantic gap between high-level concepts and low-
level features of an image.
4. Automatic image annotation based on the classification scores.
Second approach: CBIR-based on rectangular spatial
histograms of visual words
MAP on Corel-1000 image benchmark
Figure: MAP of the proposed approach obtained from Corel-A image dataset with step size of 5.
Precision on Corel-1000 image benchmark
Image retrieval on Corel-1000 image benchmark
Figure: Top-20 image retrievals shows reduction of semantic gap for the semantic class “Elephants” of
Corel-A image dataset.
Result of automatic image annotation
Figure: Automatic image annotation result for the semantic class "Horses" of Corel-A image dataset.
Result of automatic image annotation
Figure: Automatic image annotation result for the semantic class "Mountains" of Corel-A image dataset.
Performance measure on the Ground Truth image
benchmark
Computational cost (time in seconds)
Table: Comparison of average CPU time (in seconds) required to extract feature descriptor.
Table: Computational cost (in seconds) of the proposed approach.
Third approach: CBIR and AIA based on weighted
average of spatial triangular histograms
(a): Rule of third based approach (b) Spatial pyramid matching based approach
Ssss
(c) Proposed WATH based approach (d) Proposed WATH based approach
Third approach: CBIR and AIA based on weighted
average of spatial triangular histograms
The contributions of this research article are as follows:
1. The addition of spatial information to the inverted index of BoVW-
based image representation.
2. The image representation on the basis of weighted average of
histograms of triangular regions that resolve the problem of over
fitting on dictionary of larger sizes.
3. Reduction of semantic gap issue between low-level image features
and high-level image concepts.
Block diagram: CBIR and AIA based on weighted
average of spatial triangular histograms
MAP on Corel-1000 image benchmark
Table: MAP for top 20 image retrievals by using Corel-1000 image benchmark.
Precision & Recall on Corel-1000 image benchmark
Image retrieval on Corel-1000 image benchmark
Table: Top-20 image retrievals shows reduction of semantic gap for the semantic class “Dinosaurs” of Corel-A
image dataset.
Performance evaluation on the Corel-1500 image
benchmark
Comparative analysis of the proposed approaches
Table: Comparative analysis of proposed approaches in terms of computational
complexity (time in seconds).
Table: Comparative analysis of proposed approaches in terms of MAP.
Comparative analysis of the proposed approaches
• Figure: Precision-recall curve of proposed approaches for the Corel-1000 image
benchmark on pixel stride of 5.
Thanks