SAVITRIBAI PHULE PUNE UNIVERSITY
A PRELIMINARY PROJECT REPORT ON
Automatic Image Annotation Using Weekly
Labeled Data
SUBMITTED TOWARDS THE
PARTIAL FULFILLMENT OF THE REQUIREMENTS OF
BACHELOR OF ENGINEERING(INFORMATION
TECHNOLOGY)
Submitted BY
[Link] more [B120138555]
[Link] kukureja [B120138550]
[Link] yadav [B120138521]
Under The Guidance of
[Link]
DEPARTMENT OF INFORMATION
TECHNOLOGY
[Link] Institute of Engineering Education and
Research, Nashik.
1
[Link] Institute of Engineering Education and Research,Nashik
DEPARTMENT OF INFORMATION TECHNOLOGY
CERTIFICATE
This is to certify that the Project Entitled
Automatic Image Annotation Using Weekly Labeled
Data
Submitted by
[Link] more [B120138555]
[Link] kukureja [B120138550]
[Link] yadav [B120138521]
is a bonafide work carried out by Students under the supervision of Prof.
and it is submitted towards the partial fulfillment of the requirement of Bach-
elor of Engineering (INFORMATION TECHNOLOGY) Project.
Prof. Prof.T
Internal Guide H.O.D
Dept. of Information tech. Dept. of Information tech.
2
[Link] Institute of Engineering Education and Research 2016-2017
0
1
Abstract
The unique feature of our proposed method is in its almost ex-
clusive reliance on a single modality visual data for annotating images. Ad-
ditionally, we do not utilize any of the provided training data, but instead
create our own similarly-sized training set. By exploiting the latest research
in deep learning and computer vision, we are able to test the applicability of
these techniques to a problem of extremely noisy [Link] preprocessing
the content and comparing with image annotations,we use text preprocess-
ing and semantic analysis using [Link] to finduseful tokens from search
query.
We are able to obtain state-of-the art results on an inherently multi-modal
problem thereby demonstrating that computer vision can also be a primary
classification modality instead of relying primarily on text to determine con-
text prior to image [Link] Scope of Project is to assign tagging of
new images using available Dataset and tags.
Image retrieval is more effective using this technique. Context and meaning
based tags of image can be more useful.
2
Acknowledgments
We would like to take this opportunity to express our profound gratitude to
all those who have provided timely guidance and helping hand in our Project
Phase-I. We want to thank the department of Information Technology of
K.K.W.I.E.E.R, Nashik for giving us opportunity to represent this re-
port as part of Final Year Project to do the necessary research work and to
use the departmental resources. We extend our sincere thanks to respected
Prof. Dr. K. N. Nandurkar(Principal) of [Link] College of
Engineering, Prof. Dr. P. D. Bhamre(HOD) of IT Department and
all the teachers of the department who encouraged us for the project work.
We deeply thank our guide Prof. [Link] whose guidance, stimu-
lating suggestions and encouragement helped us in the project related work.
We even want to thank our department colleagues for all their help, support,
interest and valuable hints.
3
Contents
1 Introduction 9
1.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
1.1.1 Advantages . . . . . . . . . . . . . . . . . . . . . . . . 9
1.2 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
1.3 Organization of the report . . . . . . . . . . . . . . . . . . . . 10
2 Literature Survey 11
2.1 Related work done . . . . . . . . . . . . . . . . . . . . . . . . 11
2.2 Existing Systems . . . . . . . . . . . . . . . . . . . . . . . . . 11
3 Problem Statement 13
3.1 Problem Statement . . . . . . . . . . . . . . . . . . . . . . . . 13
3.2 Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
3.3 Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
3.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
4 Project Requirement 15
4.1 Proposed System . . . . . . . . . . . . . . . . . . . . . . . . . 15
4.2 Hardware Requirement . . . . . . . . . . . . . . . . . . . . . . 15
4.3 Software Tools / Technologies to be used . . . . . . . . . . . 15
4.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
5 System Analysis and Design 16
5.1 Proposed Architecture . . . . . . . . . . . . . . . . . . . . . . 16
5.2 Algorithms / techniques used . . . . . . . . . . . . . . . . . . 17
5.2.1 Techniques Used . . . . . . . . . . . . . . . . . . . . . 17
5.3 High level design of the project (UML/DFD/ERD) . . . . . . 18
5.3.1 Data Flow Diagram . . . . . . . . . . . . . . . . . . . . 18
5.3.2 Usecase Diagram . . . . . . . . . . . . . . . . . . . . . 19
5.3.3 Activity Diagram . . . . . . . . . . . . . . . . . . . . . 20
5.3.4 Sequence Diagram . . . . . . . . . . . . . . . . . . . . 23
4
5.3.5 Collaboration Diagram . . . . . . . . . . . . . . . . . . 24
5.3.6 Component Diagram . . . . . . . . . . . . . . . . . . . 25
5.3.7 Deployment Diagram . . . . . . . . . . . . . . . . . . . 27
5.3.8 Package Diagram . . . . . . . . . . . . . . . . . . . . . 28
5.4 Modules Split-up (with GUI design) . . . . . . . . . . . . . . . 29
5.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
6 Project Plan 30
6.1 System Implementation Plan . . . . . . . . . . . . . . . . . . . 30
6.2 PERT chart . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
7 Conclusion 33
8 References 34
5
List of Figures
5.1 Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
5.2 Data Flow Diagram . . . . . . . . . . . . . . . . . . . . . . . . 18
5.3 use case . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
5.4 Activity Diagram . . . . . . . . . . . . . . . . . . . . . . . . . 21
5.5 Activity Diagram . . . . . . . . . . . . . . . . . . . . . . . . . 22
5.6 Sequence Diagram . . . . . . . . . . . . . . . . . . . . . . . . 23
5.7 Collaboration Diagram . . . . . . . . . . . . . . . . . . . . . . 24
5.8 Component Diagram . . . . . . . . . . . . . . . . . . . . . . . 26
5.9 Deployment Diagram . . . . . . . . . . . . . . . . . . . . . . . 27
5.10 Packege Diagram . . . . . . . . . . . . . . . . . . . . . . . . . 28
6.1 Year Planner . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
6
List of Tables
6.1 Project Plan . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
7
0
8
Chapter 1
Introduction
1.1 Introduction
Automatic image annotation (also known as automatic image tagging
) is the process by which a computer system automatically assigns metadata
in the form of captioning or keywords to a digital image. In this project
including data collection, classifier training and post-processing tweaks. We
also discuss the challenges posed due to the fact that test data is annotated
via crowd-sourcing which adds another source of label noise to ground-truth”
data. Finally, we present our results on SCIA along with proposals for future
research to improve the automatic annotation capabilities of techniques in
this field. This application of computer vision techniques is used in image
retrieval systems to organize and locate images of interest from a database.
This system to find holiday locations to visit by using text description and
in many areas where text based image search is required.
1.1.1 Advantages
1. Project is to assign tagging of new images using availableDataset and
tags.
2. Image retrieval is more effective using this technique.
3. Context and meaning based tags of image can be more useful.
1.2 Motivation
By preprocessing the content and comparing with image annotations,
we use text preprocessing and semantic analysis using [Link] to find
useful tokens from search query. Project is to assign tagging of new images
9
using available Dataset and tags. Image retrieval is more effective using this
technique.
1.3 Organization of the report
The following are the chapter-wise contents of the report:
Chapter 1: Introduction:Consists of introduction to our system and the
motivation.
Chapter 2: Related works and existing systems:Consists of brief information
about existing systems.
Chapter 3: Problem statement: Consists of problem statement and a brief
idea about the objectives and scope.
Chapter 4: Project Requirements:Consists of the information about the pro-
posed system ie the methodology and the software and hardware require-
ments of our project.
Chapter 5: System analysis and design:Consists of system architecture, algo-
rithms and techniques to be used, high level diagrams i.e. usecase,sequence
and activity diagram of our system.
Chapter 6: project plan: Consists of the project phases schedule.
10
Chapter 2
Literature Survey
2.1 Related work done
Image Tagging or annotation allocation by system is widely accepted
area by researchers and it is because of its requirements like image content
identification and this content identification differs to according the context.
[Link]:Everingham, M., Van Gool, L., Williams, C.K.I., Winn, J.,
Zisserman
Paper TitlesThe PASCAL Visual Object Classes Challenge 2011 (VOC2011)
Results. [Link] [Link]/challenges/VOC/voc2011/workshop/[Link]
Technique Used The PASCAL Visual Object Classes
[Link]:Fan, R.E., Chang, K.W., Hsieh, C.J., Wang, X.R., Lin, C.J.:
Liblinear
Paper TitlesA library for large linear classication. The Journal of Ma-
chine Learning Research 9, 1871-1874 (2008)
Technique Used A library for large linear classication
2.2 Existing Systems
In existing System, The unique feature of sytem is in its almost ex-
clusive reliance on a single modality visual data for annotating images. Ad-
ditionally, we do not utilize any of the provided training data, but instead
11
create our own similarly-sized training set. By exploiting the latest research
in deep learning and computer vision, we are able to test the applicability
of these techniques to a problem of extremely noisy learning. The training
and test sets are identical. Therefore, a method that is able to exploit the
noisy training data (e.g. via data cleaning) could, in theory, benefit from
potentially over fitting the training data.
12
Chapter 3
Problem Statement
3.1 Problem Statement
A fundamental assumption of traditional recognition techniques is that the
data available for training are accurately labelled. Given the scale and diver-
sity of web data, it takes considerable annotation effort to reduce label noise
to acceptable levels. In this work, we propose a novel method to work around
this issue by utilizing approximate apriori estimates of the mislabelling prob-
abilities to design a noise-aware learning framework.
3.2 Scope
1. The Scope of Project is to assign tagging of new images using available
Dataset and tags.
2. Image retrieval is more effective using this technique.
3. Context and meaning based tags of image can be more useful.
4. Find relevant images and associated information is required in many areas
like criminal identification, place identification etc
3.3 Objectives
1. To Process Dataset of images and assigning correct tags to Input Concept
. 2. To perform Image content identification
3. To Process Image Retrieval
4. To perform Image Based data Search
13
3.4 Summary
Thus this chapter consists of problem statement and a brief idea about the
objectives and scope of real time attendance management system.
14
Chapter 4
Project Requirement
4.1 Proposed System
4.2 Hardware Requirement
• Processer : Intel i3
• Storage : 1GB.
• Monitor : 15 VGA Colour.
4.3 Software Tools / Technologies to be used
• Operating System : Windows 7/8
• Front End : JSP
• Server : Apache Tomcat
• Back End : MySQL
• Netbeans 8.01
• JDK 1.8
4.4 Summary
Consists of the information about the proposed system ie the methodology
and the software and hardware requirements of our project.
15
Chapter 5
System Analysis and Design
5.1 Proposed Architecture
Figure 5.1: Architecture
16
Dataset Reader:-Dataset Reader will contain Images. Tags will be as-
signed to the images. Pre-Processing:-Assign the tags obtained from step1
to the Images in the Data. Indexing 1:-A Cluster will be made containing the
Images with same [Link]: If tag-1 is Mountain in an image and several
other images also contain mountain then they will be clustered together in a
table to make searching easy. Input Concept:-Description will be given as in-
put. Pre-Processing:-Tokens will be separated from the description and Tags
will be Assigned. Technique used for Assigning Tags: Text Mining Semantic
Analysis will: Tokenize Remove Unrelated Words Index Comparison:-Tags
obtained will be compared with our Datasets Tags. Output will be generated
as an Image.
5.2 Algorithms / techniques used
In this Project a image annotation mechanism using web search is proposed,
here we find description of image which we preprocess to find annotations of
image and then we can process input as a text to find images matching to
given content. By preprocessing the content and comparing with image an-
notations, we use text preprocessing and semantic analysis using [Link]
to find useful tokens from search query.
Steps: [Link] Reader will contain Images. Tags will be assigned to
the images.
[Link]-Processing. Assign the tags obtained from step1 to the Images in the
Data.
3. Indexing. A Cluster will be made containing the Images with same tags.
Example: If tag-1 is Mountain in an image and several other images also
contain mountain then they will be clustered together in a table to make
searching easy. [Link] Concept:
Description will be given as input. [Link]-Processing:
Tokens will be separated from the description and Tags will be Assigned.
Technique used for Assigning Tags: Text Mining Semantic Analysis will: To-
kenize Remove Unrelated Words [Link] Comparison:
Tags obtained will be compared with our Datasets Tags. [Link] will be
generated as an Image.
5.2.1 Techniques Used
Image semantic Analysis. Tokenize. Grammar correction. [Link]
Term comparison. Indexing.
17
5.3 High level design of the project (UML/DFD/ERD)
5.3.1 Data Flow Diagram
A data flow diagram (DFD) illustrates how data is processed by a system in
terms of inputs and outputs. As its name indicates its focus is on the flow of
information, where data comes from, where it goes and how it gets stored.
Figure 5.2: Data Flow Diagram
18
5.3.2 Usecase Diagram
Use case diagrams are used to gather the requirements of a system includ-
ing internal and external influences. These requirements are mostly design
requirements. So when a system is analyzed to gather its functionalities use
cases are prepared and actors are identified. Use cases are meant for specifi-
cation of the interaction between the system and end users of the system. It
offers a detailed description of how the system is used.
Figure 5.3: use case
19
5.3.3 Activity Diagram
Activity is used for showing behavior and it results in action. It is a single
step within an activity and is the component where data manipulation occurs
in a system archetype. The basic purposes of activity diagrams are similar
to other four diagrams. It captures the dynamic behaviour of the system.
Other four diagrams are used to show the message flow from one object to
another but activity diagram is used to show message flow from one activity
to another.
Activity is a particular operation of the system. Activity diagrams are not
only used for visualizing dynamic nature of a system but they are also used
to construct the executable system by using forward and reverse engineering
techniques. The only missing thing in activity diagram is the message part.
20
Figure 5.4: Activity Diagram
21
Figure 5.5: Activity Diagram
22
5.3.4 Sequence Diagram
The Sequence Diagram models the collaboration of objects based on a time
sequence. It shows how the objects interact with others in a particular sce-
nario of a use case. With the advanced visual modeling capability, you can
create complex sequence diagram in few clicks. Besides, Visual Paradigm can
generate sequence diagram from the flow of events which you have defined
in the use case description
Figure 5.6: Sequence Diagram
23
5.3.5 Collaboration Diagram
It is also called as communication diagram or interaction diagram, is an illus-
tration of the relationships and interaction among software objects in UML.
A collaboration diagram resembles a flowchart that portrays the roles, func-
tionality and behavior of individual objects as well as the overall operation of
the system in real time. Objects are shown as rectangles with naming labels
inside. These labels are preceded by colons and may be underlined. The re-
lationships between the objects are shown as lines connecting the rectangles.
The messages between objects are shown as arrows connecting the relevant
rectangles along with labels that define the message sequencing.
[Link]
Figure 5.7: Collaboration Diagram
24
5.3.6 Component Diagram
It depicts how components are wired together to form larger component and
software system. Component diagram is a special kind of diagram in UML.
The purpose is also different from all other diagrams discussed so far. It does
not describe the functionality of the system but it describes the components
used to make those functionalities.
So from that point component diagrams are used to visualize the physical
components in a system. These components are libraries, packages, files etc.
25
Figure 5.8: Component Diagram
26
5.3.7 Deployment Diagram
Deployment diagrams are used to visualize the topology of the physical com-
ponents of a system where the software components are deployed.
So deployment diagrams are used to describe the static deployment view
of a system. Deployment diagrams consist of nodes and their relationships.
Figure 5.9: Deployment Diagram
27
5.3.8 Package Diagram
Package diagram is UML structure diagram which shows packages and de-
pendencies between the packages. Model diagrams allow to show different
views of a system, for example, as multi-layered (aka multi-tiered) application
- multi-layered application model.
Figure 5.10: Packege Diagram
28
5.4 Modules Split-up (with GUI design)
1. Image Dataset Processor
Image Dataset Process images with search and finds related text.
2. Annotation Allocator
Annotation allocation to Images
3. Input Query Preprocessor
Read Input Query and Convert to Tokens
4. Query Comparator
Compare Query Tokens with Annotations
5. Result Analysis
Shows List of images matching to query.
5.5 Summary
Thus this chapter consists of system architecture, algorithms and techniques
to be used, high level diagrams ie usecase,sequence and activity diagram of
our system, and information about login,authentication,server,administration
and report generation module. Also the GUI designs are shown in this chap-
ter
29
Chapter 6
Project Plan
6.1 System Implementation Plan
Table 6.1: Project Plan
Sr. No Task Start Date End Date Status
1. Introduction and Problem defination 27/9/2016 1/10/2016. Completed
2. Literature Survey 2/10/2016 15/10/2016 Completed
3. System Requirement Gathering 16/10/2016 20/10/2016 Completed
4. Feasibility Study 20/10/2016 28/10/2016 Completed
5. System Analysis 29/10/2016 8/11/2016 Completed
6. System Design 10/11/2016 18/11/2016 Completed
7. Conclusion 19/11/2016 28/11/2016 Completed
6.2 PERT chart
Year planner chart is also called as time line chart. Time line chart is use for
proper scheduling of project. A timeline chart can be developing for entire
project. Alternately separate time line chart is developing for each function.
A time line chart enables what task will be conducted at given point of
time. The task set that is selected is collection of software engineering work
task, milestone and deliverable that must be to complete a particular project.
In timeline chart all project task are listed in left-hand column.
30
The horizontal bar indicates duration of each task. When multiple bars
occur at the same time, task concurrency is implied. The diamond indicates
milestones. Time line chart help to determine how each task is initiated in
addition it gives the understanding of priority and criticality of each task.
When force with serve deadline pressures, a project scheduling technique
called Time-Boxing is used.
Time line plan is use for correct setting up of project. A timeline chart
can be rising for entire project. Alternately divide time line chart is on the
rise for each function. Year Planner enables you to create a structured and
refined Year Plan of your project. It is designed to help you identify the
project’s goals and outlines the analysis, strategies and planning required for
these goals to be achieved.
The Year Planner allows you to create a yearly plan for your project.
Here you are able to outline what you want your project to achieve during
the year ahead.
Therefore, you are able to stay focused on the big picture and will not get
lost in the detailed day-to-day planning. It also reduces risks and increases
the chances of success for your project.
31
Figure 6.1: Year Planner
32
Chapter 7
Conclusion
In this work, a novel technique is presented to learn a classifier by incorpo-
rating the noise inherently present in the data labels. we perform learning
on this as a log-likelihood optimization problem. We evaluate our proposed
technique on multiple datasets and modalities and demonstrate its effective-
ness. We also perform robustly at high levels of noise and achieve better
or similar performance , often with much lower computational requirements,
than other state-of-the-art methods. We plan to explore the use of larger
datasets, noise estimation techniques and deeper learning architectures in
the future.
33
Chapter 8
References
1. Open biometrics, [Link]
2. Everingham, M., Van Gool, L., Williams, C.K.I., Winn, J., Zisserman,
A.: The PASCAL Visual Object Classes Challenge 2011 (VOC2011) Results.
[Link] [Link]/challenges/VOC/voc2011/workshop/[Link]
3. Fan, R.E., Chang, K.W., Hsieh, C.J., Wang, X.R., Lin, C.J.: Liblin-
ear: A library for large linear classication. The Journal of Machine Learning
Research 9, 1871 1874 (2008)
4. Ferrari, V., Marin-Jimenez, M., Zisserman, A.: Progressive search
space reduction for human pose estimation. In: Computer Vision and Pat-
tern Recognition, 2008. CVPR 2008. IEEE Conference on. pp. 18. IEEE
(2008)
5. Jiang, Y.G., Wang, J., Chang, S.F., Ngo, C.W.: Domain adaptive
semantic diffusion for large scale context-based video annotation. In: Com-
puter Vision, 2009 IEEE 12th International Conference on. pp. 14201427.
IEEE (2009)
34