PROJECT REPORT

on

IMAGE CAPTION GENERATOR

Submitted for the Partial Fulfillment of the

Requirements for the Award of the degree of

BACHELOR OF TECHNOLOGY
in
INFORMATION TECHNOLOGY
(2026)

By

MONIKA SHARMA (2301410139011)

PANKAJ PANDEY (2301410139012)
ROHIT YADAV (2301410139015)
SAKSHAM KAPOOR (2301410139016)

Under the guidance

of

MR. RAJESH KUMAR SHARMA

Sagar Institute of Technology and Management (SITM), Barabanki

Affiliated to

Dr. APJ ABDUL KALAM TECHNICAL UNIVERSITY, LUCKNOW

 COMPUTER SCIENCE & ENGINEERING

SAGAR INSTITUTE OF TECHNOLOGY AND

MANAGEMENT (SITM)
OF ENGINEERING AND MANAGEMENT

CERTIFICATE
Certified that the project entitled “Image
Caption Generator” submitted by
MONIKA SHARMA (2301410139011)
PANKAJ PANDEY (2301410139012)
ROHIT YADAV (2301410139015)
SAKSHAM KAPOOR (2301410139016)
in the partial fulfillment of the requirements for the award of the degree of Bachelor of Technology
(Computer Science and Engineering) of Dr. APJ Abdul Kalam Technical University (Uttar Pradesh,
Lucknow), is a record of students’ own work carried under our supervision and guidance. The
project report embodies results of original work and studies carried out by students and the contents
do not forms the basis for the award of any other degree to the candidate or to anybody else.

MR. RAJESH KUMAR SHARMA MR. RAJESH KUMAR SHARMA

Assistant Professor Head of Department
(Project Guide)
 DEPARTMENT OF INFORMATION TECHNOLOGY
SAGAR INSTITUTE OF TECHNOLOGY AND
MANAGEMENT, BARABANKI

DECLARATION

I/We hereby declare that the project entitled “Image Caption Generator” submitted by me/us in
the partial fulfilment of the requirements for the award of the degree of Bachelor of Technology
(Computer Science and Management) of Dr APJ Abdul Kalam Technical University (Uttar Pradesh,
Lucknow) is a record of my/our work carried under the supervision and guidance of MR. RAJESH
KUMAR SHARMA.

To the best of my/our knowledge, this project has not been submitted to Dr. APJ Abdul Kalam
Technical University (Uttar Pradesh, Lucknow) or any other University or Institute for the award of
any degree.

MONIKA SHARMA (2301410139011)

PANKAJ PANDEY (2301410139012)
ROHIT YADAV (2301410139015)
SAKSHAM KAPOOR (2301410139016)
 DEPARTMENT OF INFORMATION TECHNOLOGY
OF

SAGAR INSTITUTE OF TECHNOLOGY AND

MANAGEMENT (SITM)

ACKNOWLEDGEMENT

We take this opportunity to express our profound gratitude and deep regards to our guide MR.
RAJESH KUMAR SHARMA for their exemplary guidance, monitoring and constant encouragement.
The blessing, help and guidance given by them time to time shall carry us a long way in the journey
of life on which we are about to embark.
We also take this opportunity to express a deep sense of gratitude to IT Department, SITM,
Barabanki for their cordial support, valuable information and guidance, which helped us in this
task through various stages.
We are obliged to staff members of IT Department, SITM, Barabanki, for the valuable
information provided by them in their respective fields. We are grateful for their cooperation. We
would like to express my special gratitude and thanks to them for giving us such attention and time.
Last but definitely not least, we would like to thank our mother, father, family member and friends
for the constant encouragement and constant support they showed us throughout our entire period as
a college student which helped me to keep going and never give up.

MONIKA SHARMA (2301410139011)

PANKAJ PANDEY (2301410139012)
 ROHIT YADAV (2301410139015)
SAKSHAM KAPOOR (2301410139016)

PREFACE
In this project, we employ Convolutional Neural Networks (CNN) and Long Short-Term Memory
(LSTM) networks to interpret textual descriptions for images. With the ongoing progress in deep
learning techniques, having ample datasets and computational power is pivotal in crafting models
capable of generating captions for images. This Python-driven endeavor aims to implement such
models, utilizing deep learning methodologies like CNNs and Recurrent Neural Networks (RNNs).

The image caption generation process is intricate, integrating natural language processing and
computer vision principles to grasp the context of an image and express it in English. This review
paper thoroughly investigates fundamental concepts pertaining to image captioning and prevalent
methodologies. Our discourse covers the use of essential tools such as the Keras library, numpy, and
Jupyter notebooks in developing this project. Furthermore, we explore the employment of the
flickr_dataset and CNN for image classification tasks.

INTRODUCTION: This chapter gives our problem definition along with the aims and objectives.
This part also includes a section on objectives, and project analysis which gives information about
our project.

PROBLEM DEFINITION: This chapter explains the problem that we are working on and ways to
address the problem and make it a feasible project.

LITERATURE REVIEW: This chapter explains the form of making important summaries from
these sources that are of relevance from the entire work.

SYSTEM DESIGN: In the proposed Image Caption Generator model, a convolutional neural
network generates a dense feature vector, serving as input for LSTM-based caption generation.
MODULE DESCRIPTION: This section is used to describe the different modules and algorithms
that we use in this project along with their theory and overview.

CONCLUSION: This section covers the various inferences that were drawn after the completion
of the entire project.

FUTURE SCOPE: This section gives the future enhancements that can be made to the project
idea and its implementation.

REFERENCES: This section lists all the sources we have used in our project so that readers can
easily find what we have cited.
 ABSTRACT
In this project, we utilise Convolutional Neural Networks (CNN) and Long Short-Term Memory
(LSTM) networks to discern the textual description for an image. With the continuous advancement
of deep learning techniques, the availability of extensive datasets and computational resources plays
a crucial role in constructing models capable of generating image captions. This Python-based
project aims to implement such models, leveraging deep learning methodologies like CNN and
Recurrent Neural Networks (RNN).

The image caption generator is a multifaceted process that incorporates both natural language
processing and computer vision principles to comprehend the context of an image and articulate it in
English. This survey paper meticulously explores fundamental concepts related to image captioning
and prevalent approaches. Our discussion encompasses the application of key tools such as the
Keras library, numpy, and Jupyter notebooks in the development of this project. Additionally, we
delve into the utilization of the flickr_dataset and CNN for image classification purposes.

KEYWORDS: generate captions, deep learning techniques, concepts of image captioning.

 TABLE OF CONTENTS

CERTIFICATE ii

DECLARATION iii

ACKNOWLEDGEMENT iv

PREFACE v- vi

ABSTRACT vii

TABLE OF CONTENTS viii-ix

1. INTRODUCTION (1)

2. PROBLEM DEFINITION (2 – 7)

2.1 Problem Identification 2

2.1.1 Vanishing Gradient Problem 2

2.1.2 Training as RNN is Difficult 3

2.2 Proposed Methodology 4

3. LITERATURE REVIEW (8 – 15)

3.1 Image Captioning Models 8

3.2 DL Captions Models 10

3.3 Supervised Learning vs Others DL 11

3.4 Dense Captioning vs Captions for the Whole Scene 12

3.5 EN-DC Architecture vs Compositional Architecture 13

4. HARDWARE & SOFTWARE REQUIREMENTS (16 – 17)

5. SYSTEM DESIGN (18 – 21)
5.1 System Architecture 18
5.2 Proposed Image Caption Generator 20
6. MODULE DESCRIPTION (22 – 26)

6.1 Model Overview 22

6.2 Recurrent Neural Network 23

6.3 Convolutional Neural Network 24

6.4 Long Short-Term Memory 25

6.5 VGG16 26

7. CODING PART (27 – 60)

7.1 What is an Image Caption Generator? 27

7.2 Image Caption Generator with CNN – Python-Based Project 27

7.3 The Dataset of the Python-Based Project 27

7.4 Pre-requisites 27

7.5 Image Caption Generator – Python-Based Project 28

7.6 Project File Structure 28

7.7 Building the Python-Based Project 29

8. RESULT AND ANALYSIS (61 – 63)

8.1 Applications 61

8.2 Advantages 61

8.3 Limitations 62

8.4 Result Overview 63

9. CONCLUSION (64)

10. FUTURE SCOPE (65)

LIST OF FIGURES x

ABBREVIATIONS xi

REFERENCES xii - xiii

 CHAPTER - 1

INTRODUCTION

Every day, we encounter a large number of images from various sources such as the Internet,
news articles, document diagrams and advertisements. These sources contain images that
viewers would have to interpret themselves. Most images do not have a description, but the
human can largely understand them without their detailed captions. However, a machine needs to
interpret some form of image captions if humans need automatic image captions from it.
Image captioning is important for many reasons. Captions for every image on the internet can
lead to faster and descriptively accurate image searches and indexing.

Ever since researchers started working on object recognition in images, it became clear that only
providing the names of the objects recognized does not make such a good impression as a full
human-like description. As long as machines do not think, talk, and behave like humans, natural
language descriptions will remain a challenge to be solved.
Image captioning has various applications in various fields such as biomedicine, commerce, web
searching and military etc. Social media like Instagram , Facebook etc can generate captions
automatically from images.

Creating captions for images is a crucial endeavour within the realms of both Computer Vision
and Natural Language Processing. Replicating the human capacity to articulate image descriptions
using machines represents a significant milestone in the field of Artificial Intelligence. The
primary obstacle in this endeavor lies in comprehending the relationships between objects within
an image and conveying them in a natural language such as English.

Traditionally, computer systems have relied on predefined templates for crafting textual
descriptions for images. However, this method lacks the necessary diversity required to produce
linguistically rich text descriptions. This limitation has been overcome with the enhanced efficacy
of neural networks. Many cutting-edge models employ neural networks to generate captions,
taking the image as input and predicting the subsequent lexical units in the resulting sentence.
 CHAPTER - 2

PROBLEM DEFINITION

2.1 PROBLEM IDENTIFICATION

Despite the achievements demonstrated by numerous systems utilizing Recurrent Neural
Networks (RNNs), several unresolved issues persist. Two particular challenges stand out as
prominent concerns for most RNN-based systems:

1. The Vanishing Gradient Issue.

2. The arduous nature of training RNNs.

A recurrent neural network represents a deep learning algorithm crafted to handle a diverse range
of complex computational tasks, including tasks like object classification and speech recognition.
RNNs are tailored to process sequences of events that unfold over time, with the understanding of
each event informed by preceding events.

In an ideal scenario, we would prefer to employ deeper RNN architectures, as they could exhibit
longer memory capabilities and enhanced performance. These attributes could find practical
applications in various real-world scenarios such as stock prediction and improved speech
recognition. Nevertheless, despite their potential, RNNs are rarely deployed in practical, real-
world situations due to the persistent challenge of the vanishing gradient problem.

2.1.1 VANISHING GRADIENT PROBLEM

The vanishing gradient problem is a significant hurdle in the performance of Recurrent Neural
Networks (RNNs). RNNs are designed to process sequential data, but they suffer from a
fundamental limitation: a restricted memory capacity. Their hidden states can only retain
information from a limited number of preceding time steps, making them well-suited for short
sequences but ineffective at capturing long-term dependencies.

This issue becomes particularly pronounced when training RNNs on deep sequences, and it is
known as the vanishing gradient problem. During backpropagation, the gradients of the loss
function diminish exponentially as they traverse the layers of the network. This leads to extremely
small gradients, hindering effective parameter updates and causing the network to learn slowly or
not at all.
The vanishing gradient problem arises due to improper configurations of network parameters and
hyperparameters. It is more likely to occur in deep networks with poorly chosen weights and
activation functions. As a result, traditional RNNs struggle to maintain memory and lose
important information when dealing with extended sequences.

Addressing the vanishing gradient problem is crucial for improving RNN performance. Strategies
such as using alternative architectures like Long Short-Term Memory (LSTM) and Gated
Recurrent Unit (GRU) or careful parameter tuning can mitigate this issue, enabling RNNs to excel
in tasks involving sequences with long-term dependencies.

2.1.2 TRAINING AN RNN IS DIFFICULT

Training artificial recurrent neural networks presents a considerable challenge due to their inherent
complexity. Unlike feedforward neural networks, which have a unidirectional signal flow from
input to output layers, recurrent neural networks (RNNs) introduce intricate signal dynamics
involving feedback loops and bidirectional signal propagation. These characteristics are closely tied
to the memory capabilities of RNNs, which are essential for tasks like natural language processing.

In natural language processing, RNNs need to retain contextual information and remember inputs.
For instance, when analyzing or predicting a word within a sentence, the network must consider a
fixed window of surrounding words and maintain awareness of their contextual relationships. This
necessity leads to the inclusion of hidden loops and feedback mechanisms within the RNN
architecture, but these complexities hinder the training process.

One significant challenge linked to RNN training is the exploding and vanishing gradient problem.
As RNNs process data through multiple time steps, the network's weight updates can either grow
exponentially (exploding) or shrink to negligible values (vanishing). This phenomenon occurs
during backward passes and is particularly pronounced in long sequences with numerous time steps,
making it difficult to ensure stable learning.

While weight initialization techniques can alleviate this issue to some extent, the fundamental
complexities associated with RNN design persist. These intricacies challenge our ability to
effectively manage and scale these advanced neural network architectures, highlighting the inherent
trade-offs between complexity and predictability.

PAGE | 3
 Image Caption Generator

2.2 PROBLEM METHODOLOGY

The main aim of this project is to get a little bit of knowledge of deep learning techniques. We use
two techniques mainly CNN and LSTM for image classification.

So, to make our image caption generator model, we will be merging these architectures. It is also
called a CNN-RNN model.
 CNN is used for extracting features from the image. We will use the pre-trained model
Xception.
 LSTM will use the information from CNN to help generate a description of the image.

PROPOSED WORK
1. Task - The task is to build a system that will take an image input in the form of a dimensional
array and generate an output consisting of a sentence that describes the image and is
syntactically and grammatically correct.
2. Corpus - We have used the Flickr 8K dataset as the corpus. The dataset consists of 8000
images and for every image, there are 5 captions. The 5 captions for a single image helps in
understanding all the various possible scenarios. The dataset has a predefined training dataset
Flickr_8k.[Link] (6,000images), development dataset Flickr_8k.[Link] (1,000
images), and test dataset Flickr_8k.[Link] (1,000 images). The Images are opted from
six varied Flickr groups and do not contain any well-known personality or places. However,
they are manually selected to show a variety of scenes.
3. Preprocessing - Data preprocessing is done in two parts, the images and the corresponding
captions are cleaned and pre-processed separately. Image preprocessing is done by feeding the
input data to the Xception application of the Keras API running on top of TensorFlow.
Xception is pre-trained on ImageNet. This helped us train the images faster with the help of
Transfer learning. The descriptions are cleaned using the tokenizer class in Keras, this will
vectorize the text corpus and is stored in a separate dictionary. Then each word of the
vocabulary is mapped with a unique index value.
4. Model - Deep learning carries out the machine learning process using an artificial neural
network that is composed of several levels arranged in a hierarchy. The model is based on
deep networks where the flow of information starts from the initial level, where the model
learns something simple and then the output of which is passed to layer two of the network
and input is combined into something that is a bit more complex and passes it on to the third
level.
This process continues as each level in the network produces something more complex from the
input it received from the ascendant level.

Figure 2.1 Glimpse of the Flickr8k Image Dataset

CONVOLUTIONAL NEURAL NETWORK

A Convolutional Neural Network (ConvNet/CNN) is a Deep Learning algorithm which can take in
an input image, assign importance (learnable weights and biases) to various aspects/objects in the
image and be able to differentiate one from the [Link] pre-processing required in a ConvNet is
much lower as compared to other classification algorithms.

Convolutional Neural networks are specialized deep neural networks which can process the data
that has input shape like a 2D matrix. Images are easily represented as a 2D matrix and CNN is very
useful in working with images.

It scans images from left to right and top to bottom to pull out important features from the image
and combines the feature to classify images. It can handle the images that have been translated,
rotated, scaled and changes in perspective.

LONG SHORT TERM MEMORY

LSTM stands for Long short term memory, they are a type of RNN (recurrent neural network)
which is well suited for sequence prediction problems.
Based on the previous text, we can predict what the next word will be. It has proven itself effective
from the traditional RNN by overcoming the limitations of RNN which had short term memory.
LSTM can carry out relevant information throughout the processing of inputs and with a forget gate,
it discards non-relevant information.

Figure. 2.2 Model Image Caption Generator

LSTMs are designed to overcome the vanishing gradient problem and allow them to retain
information for longer periods compared to traditional RNNs. LSTMs can maintain a constant error,
which allows them to continue learning over numerous time-steps and backpropagate through time
and layers.

LSTMs use gated cells to store information outside the regular flow of the RNN. With these cells, the
network can manipulate the information in many ways, including storing information in the cells and
reading from them. The cells are individually capable of making decisions regarding the information
and can execute these decisions by opening or closing the gates. The ability to retain information for a
long period of time gives LSTM the edge over traditional RNNs in these tasks.

The chain-like architecture of LSTM allows it to contain information for longer time periods, solving
challenging tasks that traditional RNNs struggle to or simply cannot solve.
The three major parts of the LSTM include:
Forget gate - removes information that is no longer necessary for the completion of the task. This step
is essential to optimizing the performance of the network
Input gate - responsible for adding information to the cells
Output gate - selects and outputs necessary information

IT DEPARTMENT, SITM, BARABANKI PAGE | 6

 Image Caption Generator

Figure. 2.3 Forget Gate, Input Gate, Output Gate

The CNN LSTM architecture involves using Convolutional Neural Network (CNN) layers for feature
extraction on input data combined with LSTMs to support sequence prediction. This architecture was
originally referred to as a Long-term Recurrent Convolutional Network or LRCN model, although
we will use the more generic name “CNN LSTM” to refer to LSTMs that use a CNN as a front end
in this lesson. This architecture is used for the task of generating textual descriptions of images. Key
is the use of a CNN that is pre-trained on a challenging image classification task that is re-purposed
as a feature extractor for the caption generating problem.

IT DEPARTMENT, SITM, BARABANKI PAGE | 7

 Image Caption Generator

CHAPTER - 3

LITERATURE REVIEW

Image captioning has recently gathered a lot of attention specifically in the natural language
domain. There is a pressing need for context-based natural language description of images,
however, this may seem a bit farfetched but recent developments in fields like neural networks,
computer vision and natural language processing have paved the way for accurately describing
images i.e. representing their visually grounded meaning. We are leveraging state-of-the-art
techniques like Convolutional Neural Network (CNN), Recurrent Neural Network (RNN) and
appropriate datasets of images and their human perceived description to achieve the same. We
demonstrate that our alignment model produces results in retrieval experiments on datasets such as
Flicker.

3.1 IMAGE CAPTIONING METHODS

There are various Image Captioning Techniques some are rarely used in present but it is necessary
to take an overview of those technologies before proceeding ahead. The main categories of existing
image captioning methods they include template-based image captioning, retrieval-based image
captioning, and novel caption generation. Novel caption generation-based image caption methods
mostly use visual space and deep machine learning-based techniques. Captions can also be
generated from multimodal space. Deep learning-based image captioning methods can also be
categorized on learning techniques: Supervised learning, Reinforcement learning, and Unsupervised
learning. We group reinforcement learning and unsupervised learning into Other Deep Learning.
Usually captions are generated for a whole scene in the image. However, captions can also be
generated for different regions of an image (Dense captioning). Image captioning methods can use
either simple Encoder-Decoder architecture or Compositional architecture. There are methods that
use attention mechanism, semantic concept, and different styles in image descriptions. Some
methods can also generate description for unseen objects. We group them into one category as
“Others". Most of the image captioning methods use LSTM as language model. However, there are
a number of methods that use other language models such as CNN and RNN. Therefore, we include
a language model-based category as “LSTM vs. Others".

IT DEPARTMENT, SITM, BARABANKI PAGE | 8

 Image Caption Generator

3.1.1 TEMPLATE-BASED APPROACHES

Template-based approaches have fixed templates with a number of blank slots to generate captions.
In these approaches, different objects, attributes, actions are detected first and then the blank spaces
in the templates are filled. For example, Farhadi et al. use a triplet of scene elements to fill the
template slots for generating image captions. Li et al. extract the phrases related to detected objects,
attributes and their relationships for this purpose. A Conditional Random Field (CRF) is adopted by
Kulkarni et al. to infer the objects, attributes, and prepsitions before filling in the gaps. Template-
based methods can generate grammatically correct captions. However, templates are predefined and
cannot generate variable-length captions. Moreover, later on, parsing based language models have
been introduced in image captioning which are more powerful than fixed template-based methods.
Therefore, in this paper, we do not focus on these template based methods.

3.1.2 RETRIEVAL-BASED APPROACHES

Captions can be retrieved from visual space and multimodal space. In retrieval-based approaches,
captions are retrieved from a set of existing captions. Retrieval based methods first find the visually
similar images with their captions from the training data set. These captions are called candidate
captions. The captions for the query image are selected from these captions pool. These methods
produce general and syntactically correct captions. However, they cannot generate image specific
and semantically correct captions.

3.1.3 NOVEL CAPTION GENERATION

Novel image captions are captions that are generated by the model from a combination of the image
features and a language model instead of matching to an existing captions. Generating novel image
captions solves both of the problems of using existing captions and as such is a much more
interesting and useful problem. Novel captions can be generated from both visual space and
multimodal space. A general approach of this category is to analyze the visual content of the image
first and then generate image captions from the visual content using a language model These
methods can generate new captions for each image that are semantically more accurate than
previous approaches. Most 13 novel caption generation methods use deep machine learning based
techniques. Therefore, deep learning based novel image caption generating methods are our main
focus in this literature.

IT DEPARTMENT, SITM, BARABANKI PAGE | 9

 Image Caption Generator

Figure. 3.1 NOVEL CAPTION GENERATION

3.2 DEEP LEARNING-BASED IMAGE CAPTIONING METHODS

We draw an overall taxonomy in Figure 1 for deep learning-based image captioning methods. We
discuss their similarities and dissimilarities by grouping them into visual space vs. multimodal
space, dense captioning vs. captions for the whole scene, Supervised learning vs. Other deep
learning, Encoder-Decoder architecture vs. Compositional architecture, and one „Others‟ group that
contains Attention-Based, Semantic Concept-Based, Stylized captions, and Novel Object-Based
captioning. We also create a category named LSTM vs. Others. A brief overview of the deep
learning-based image captioning methods is shown in table. It contains the name of the image
captioning methods, the type of deep neural networks used to encode image information, and the
language models used in describing the information.

3.2.1 VISUAL SPACE VS. MULTIMODAL SPACE

Deep learning-based image captioning methods can generate captions from both visual space and
multimodal space. Understandably image captioning datasets have the corresponding captions as 14
text. In the visual space-based methods, the image features and the corresponding captions are
independently passed to the language decoder. In contrast, in a multimodal space case, a shared
multimodal space is learned from the images and the corresponding caption-text. This multimodal
representation is then passed to the language decoder.

IT DEPARTMENT, SITM, BARABANKI PAGE | 10

 Image Caption Generator

VISUAL SPACE
Bulk of the image captioning methods use visual space for generating captions. In the visual space-
based methods, the image features and the corresponding captions are independently passed to the
language decoder.

MULTIMODAL SPACE
The architecture of a typical multimodal space-based method contains a language Encoder part, a
vision part, a multimodal space part, and a language decoder part. A general diagram of multimodal
space-based image captioning methods is shown in Figure 2. The vision part uses a deep
convolutional neural network as a feature extractor to extract the image features. The language
encoder part extracts the word features and learns a dense feature embedding for each word. It then
forwards the semantic temporal context to the recurrent layers. The multimodal space part maps the
image features into a common space with the word features.

Figure. 3.2 A block diagram of multimodal space-based image captioning.

3.3 SUPERVISED LEARNING VS. OTHER DEEP LEARNING

In supervised learning, training data come with desired output called label. Unsupervised learning, on the
other hand, deals with unla techniques. Reinforcement learning is another type of machine learning approach
where the aims of an agent are to discover data and/or labels through exploration and a reward signal. A
number of image captioning methods use reinforcement learning and GAN based approaches. These methods
sit in the category of “Other Deep Learning". beled data. Generative Adversarial Networks (GANs) are a
type of unsupervised learning

IT DEPARTMENT, SITM, BARABANKI PAGE | 11

 Image Caption Generator

3.3.1 SUPERVISED LEARNING-BASED IMAGE CAPTIONING

Supervised learning-based networks have successfully been used for many years in image classification ,
object detection and attribute learning . This progress makes researchers interested in using them in
automatic image captioning .In this paper, we have identified a large number of supervised learning-based
image captioning methods. We classify them into different categories: (i) Encoder-Decoder Architecture, (ii)
Compositional Architecture, (iii) Attentionbased, (iv) Semantic concept-based, (v) Stylized captions, (vi)
Novel object-based, and (vii) Dense image captioning.

3.3.2 OTHER DEEP LEARNING-BASED IMAGE CAPTIONING

In our day to day life, data are increasing with unlabeled data because it is often impractical to accurately
annotate data. Therefore, recently, researchers are focusing more on reinforcement learning and unsupervised
learning-based techniques for image captioning.

3.4 DENSE CAPTIONING VS. CAPTIONS FOR THE WHOLE SCENE

In dense captioning, captions are generated for each region of the scene. Other methods generate captions for
the whole scene.

3.4.1 DENSE CAPTIONING

The previous image captioning methods can generate only one caption for the whole image. They use
different regions of the image to obtain information of various objects. However, these methods do not
generate region wise captions. Johnson et al. [62] proposed an image captioning 16 method called DenseCap.
This method localizes all the salient regions of an image and then it generates descriptions for those regions.

A typical method of this category has the following steps:

1. Region proposals are generated for the different regions of the given image.

2. CNN is used to obtain the region-based image features.

3. The outputs of Step 2 are used by a language model to generate captions for every region. A block
diagram of a typical dense captioning method is given in Figure [Link]

IT DEPARTMENT, SITM, BARABANKI PAGE | 12

 Image Caption Generator

3.4.2 CAPTIONS FOR THE WHOLE SCENE

Encoder-Decoder architecture, Compositional architecture, attention-based, semantic concept-based, stylized
captions, Novel object-based image captioning, and other deep learning networks-based image captioning
methods generate single or multiple captions for the whole scene

Figure. 3.3 A block diagram of simple Encoder-Decoder architecture-based

image captioning.

3.5 ENCODER-DECODER ARCHITECTURE VS. COMPOSITIONAL

ARCHITECTURE

Some methods use just simple vanilla encoder and decoder to generate captions. However, other
methods use multiple networks for it.

3.5.1 ENCODER-DECODER ARCHITECTURE-BASED IMAGE CAPTIONING

The neural network-based image captioning methods work as just simple end to end manner. These
methods are very similar to the encoder-decoder framework-based neural machine translation [131].
In this network, global image features are extracted from the hidden activations of CNN and then
fed them into an LSTM to generate a sequence of words. A typical method of this category has the
following general steps:

1. A vanilla CNN is used to obtain the scene type, to detect the objects and their relationships.

2. The output of Step 1 is used by a language model to convert them into words, combined phrases
that produce an image captions.

A simple block diagram of this category is given in Figure [Link].

IT DEPARTMENT, SITM, BARABANKI PAGE | 13

 Image Caption Generator

3.5.2 COMPOSITIONAL ARCHITECTURE-BASED IMAGE CAPTIONING

Compositional architecture-based methods are composed of several independent functional building
blocks: First, a CNN is used to extract the semantic concepts from the image. Then a language
model is used to generate a set of candidate captions. In generating the final caption, these candidate
captions are re-ranked using a deep multimodal similarity model. A typical method of this category
maintains the following steps:

1. Image features are obtained using a CNN.

2. Visual concepts (e.g. attributes) are obtained from visual features.

3. Multiple captions are generated by a language model using the information of Step 1 and Step 2.

4. The generated captions are re-ranked using a deep multimodal similarity model to select high
quality image captions.

A common block diagram of compositional network-based image captioning methods is given in

Figure [Link].

Figure. 3.4 A block diagram of a compositional network-based captioning

3.6 LSTM VS. OTHERS

Image captioning intersects computer vision and natural language processing (NLP) research. NLP
tasks, in general, can be formulated as sequence-to-sequence learning. Several neural language
models such as neural probabilistic language models, log-bilinear models, skip-gram models, and
recurrent neural networks (RNNs) have been proposed for learning sequence-to-sequence tasks.
RNNs have widely been used in various sequence learning tasks.
IT DEPARTMENT, SITM, BARABANKI PAGE | 14

Image Caption Generator

However, traditional RNNs suffer from vanishing and exploding gradient problems and cannot
adequately handle long-term temporal dependencies. LSTM networks are a type of RNN that has
special units in addition to standard units. LSTM units use a memory cell that can maintain
information in memory for long periods of time. In recent years, LSTM-based models have
dominantly been used in sequence to sequence learning tasks. Another network, Gated Recurrent
Unit (GRU) has a similar structure to LSTM but it does not use separate memory cells and uses
fewer gates to control the flow of information. However, LSTMs ignore the underlying hierarchical
structure of a sentence. They also require significant storage due to long-term dependencies through
a memory cell. In contrast, CNNs can learn the internal hierarchical structure of the sentences and
they are faster in processing than LSTMs.

Therefore, recently, convolutional architectures are used in other sequence-to-sequence tasks, e.g.,
conditional image generation and machine translation. Inspired by the above success of CNNs in
sequence learning tasks, Gu proposed a CNN language model-based image captioning method. This
method uses a language-CNN for statistical language modelling. However, the method cannot
model the dynamic temporal behaviour of the language model only using a language-CNN. It
combines a recurrent network with the languageCNN to model the temporal dependencies properly.
Aneja proposed a convolutional architecture for the task of image captioning. They use a
feedforward network without any recurrent function. The architecture of the method has four
components: (i) input embedding layer (ii) image embedding layer (iii) convolutional module, and
(iv) output embedding layer. It also uses an attention mechanism to leverage spatial image features.
They evaluate their architecture on the challenging MSCOCO dataset and shows comparable
performance to an LSTM based method on standard metrics.
IT DEPARTMENT, SITM, BARABANKI PAGE | 15
 Image Caption Generator

CHAPTER - 4

HARDWARE AND SOFTWARE REQUIREMENTS

4.1 Software Requirement

Recommended System Requirements to train the model.

 A good CPU and a GPU with at least 8GB of memory

 At least 8GB of RAM

 Active internet connection so that Keras can download inceptionv3/vgg16

model weights

Some required libraries for Python along with their version numbers
used while making & testing of this project.

 Tensor-flow: Tensor-Flow is an end-to-end open source platform for machine learning. Tensor-
flow is developed by Google and has integrated the most common units in deep learning
frameworks. It supports many up-to-date networks such as CNN and RNN with different
settings. Tensor-flow is designed for remarkable flexibility, portability, and high efficiency of
equipped hardware.

 PyTorch: PyTorch is a Python-based scientific computing package that serves two purposes: as
a replacement for NumPy to use the power of GPUs and as a deep learning research platform
that provides maximum flexibility and speed.

 Keras: Keras is a high-level neural network API, written in Python and capable of running on
top of Tensor-flow, CNTK, or Theano. It was developed with a focus on enabling fast
experimentation. Being able to go from idea to result with the least possible delay is the key to
doing good research. Keras allows for easy and fast prototyping (through user friend lines
modularity, and extensibility).Keras supports both convolutional networks and recurrent
networks, as well as a combination of both.

IT DEPARTMENT, SITM, BARABANKI PAGE | 16

Image Caption Generator

4.2 Hardware Requirement
The science and methodology behind deep learning have been in existence for decades. In recent
years, however, there has been a significant acceleration in the utilization of deep learning due to an
increasing abundance of digital data and the involvement of powerful hardware.
 GPU- Compared to the CPU, the performance of matrix multiplication on the Graphics
Processing Unit is significantly better. With GPU computing resources, all the deep learning
tools mentioned achieve much higher speedup when compared to their CPU-only versions
GPUs have become the platform of choice for training large, complex Neural Network based
systems because of their ability to accelerate the systems.
 TPU- Tensor Processing Unit (Domain-Specific Architecture) is a custom chip that has been
deployed in Google data centres since 2015. DNNs are dominated by tensors, so the architects
created instructions that operate on tensors of data rather than one data element per instruction.
To reduce the time of deployment, TPU was designed to be a coprocessor on the PCI Express
I/O bus rather than be tightly integrated with a CPU, allowing it to plug into existing servers just
as a GPU does. The goal was to run whole inference models in the TPU to reduce I/O between
the TPU and the host CPU. Minimalism is a virtue of domain-specific processors.

Fig. 4.1 The Required Platform for Implementation

IT DEPARTMENT, SITM, BARABANKI PAGE | 17

 Image Caption Generator

CHAPTER - 5

IMAGE CAPTION SYSTEM DESIGN

5.1 SYSTEM ARCHITECTURE

Figure 5.1 Proposed Model of Image Caption Generator

The proposed model of Image Caption Generator is as shown in the above figure 1. Here in this
model, input image is given & then A convolutional neural network is used to create a dense feature
vector as shown in figure. This dense vector, also called an embedding, this vector can be used as
input into other algorithms, and its generates [11] suitable caption for given image as output. For an
image caption generaor, this embedding becomes a representation of the image and is used as the
initial state of the LSTM for generating meaningfull captions, for the image.

Sytem Architecture of our system is shown below in Figure2.

This is our proposed system architecture will look like .

Figure 5.2 System Architecture of Image CaptionGenerator

IT DEPARTMENT, SITM, BARABANKI PAGE | 18

 Image Caption Generator

Algorithms
1) Convolutional Neural Network: Convolutional Neural networks [10] are specialized deep neural
networks that processes the data that has input shape like a 2D matrix. CNN works well with images
and are easily represented as a 2D matrix. Image classification and identification can be easily done
using CNN. It can determine whether an image is a bird, a plane or Superman, etc. [7]. Important
features of an image can be extracted by scanning the image from left to right and top to bottom and
finally the features arecombined together to classify images. It can deal with the images that have
been translated, rotated, scaled and changes in perspective.

2) Long Short Term Memory: LSTM are type of RNN (recurrent neural network) [2] which is well
suited for sequence prediction problems. We can predict what the next words will be based on the
previous text. It has shown itself effective from the traditional RNN by overcoming the limitations
of RNN. LSTM can carry out relevant information throughout the processing, it discards non-
relevant information [16].

3) Data Exploration: For the image caption generator, we have used the Flickr_8K dataset. There
are also other [6] big datasetslike Flickr_30K and MSCOCO dataset but it can take weeks for
systems having only CPU support just to train the network, so we used a small Flickr8k
[Link] a huge dataset helps in developing a better model.

Fig 5.3 Overview of Image Caption Generator Model

IT DEPARTMENT, SITM, BARABANKI PAGE | 19

Image Caption Generator

5.2 PROPOSED IMAGE CAPTION GENERATOR

Here we have shown the DFD’s of our system (i.e. Data FlowDiagrams). DFD’s provide us the basic
overview of the whole Image Caption Generator System or process being analysedor modelled.

Fig. 5.4 DFD Diagram

The Figure 5.2.2 shows State Chart Diagram of the system. First user will browse the site. Then he
will upload the image, CNN will identify the objects present in the image then LSTM will start
preparing captions considering the objects present in the image using [8] Training Dataset, which
comprises of Image Data set and Text Data Set, after the training a suitable caption will be
generated and displayed to the user.
 IT DEPARTMENT, SITM, BARABANKI PAGE | 20

Image Caption Generator

Figure 5.5 State Chart Diagram of Steps taken by the System.

The proposed system of Image Caption Generator has the capabilities to Generate Captions for the Images,
provided during the Training purpose & also for the New images as well. Our Model takes an Image
asInputand by analyzing theimage it detects objects present in an image and create a caption which describes
the image well enough for any machine to understand what an image is trying to say. [15]

IT DEPARTMENT, SITM, BARABANKI PAGE | 21

 Image Caption Generator

CHAPTER - 6

MODULE DESCRIPTION

6.1 Model Overview

The model is trained to maximise the probability of p(S|I), where S is the sequence of words
produced by the model and each word St. is produced using a lexicon that was created using the
training dataset. A deep vision convolutional neural network (CNN) is fed the input image I, which
aids in object detection in the image. As shown in Figure 3.4, the Language Generating. Recurrent
Neural Network (RNN) receives the picture encodings and uses them to create a relevant phrase for
the image. The model can be compared to a language translation RNN model, where the goal is to
maximise the p(T|S), where T is the translation to the sentence S. However, in our model the
encoder RNN which helps in transforming an input sentence to a fixed length vector is replaced by a
CNN encoder. Recent research has shown that the CNN can easily transform an input image to a
vector

Figure 6.1 Methodology Flow

We employ the pre-trained model VGG16 for the picture classification task. The next section
discusses the models' specifics. The pre-trained VGG16 is then followed by an LSTM network.
Language creation is accomplished using the LSTM network. Since a current token depends on the
preceding tokens for a sentence to have sense, LSTM networks are different from typical Neural
Networks in that they take this into consideration. We use a VGG16 pretrained model to the task of
classifying images.

IT DEPARTMENT, SITM, BARABANKI PAGE | 22

 Image Caption Generator

The next section discusses the models' specifics. The VGG16 is followed by an LSTM network,
which stands for long short-term memory. Language creation is accomplished using the LSTM
network. Traditional Neural Networks are different from LSTM since a current token depends on
the previous tokens for a sentence to be meaningful and LSTM networks take this factor into
account.

Figure 6.2 VGG16 Model

The model that existed prior to the project has two distinct input streams, one for picture features
and the other for captions from the preprocessed input. To represent the picture features in a
different dimension, the image features are transferred through a dense, completely linked layer.
The embedding layer is carried through after the input captions. After merging these two input
streams, an LSTM layer receives them as inputs. The caption embedding serves as the LSTM's
input while the image serves as its initial state.

6.2 Recurrent Neural Network

A feed forward neural network with an internal memory is known as a recurrent neural network.
The result of the current input depends on the previous computation, making RNNs recurrent in
nature because they carry out the same function for every data input. The output is created, copied,
and then delivered back into the recurrent network. It takes into account both the current input and
the output that it has learned from the prior input when making a decision

IT DEPARTMENT, SITM, BARABANKI PAGE | 23

 Image Caption Generator

Figure 6.3 RNN

One type of artificial neural network called a recurrent neural network (RNN) has connections
between nodes that create a directed graph along a temporal sequence. It can display temporal
dynamic behaviour as a result of this. RNNs, which are derived from feed-forward neural networks,
can process input sequences of varying length by using their internal state (memory).

6.3 Convolutional Neural Network

Convolutional neural networks (CNN, or ConvNet) are a class of deep neural networks used most
frequently to analyse visual vision in deep learning. Based on the shared-weight convolution
kernels' translation invariance properties and shift invariance architecture, they are often referred to
as shift invariant or space invariant artificial neural networks (SIANN).

Figure 6.4 CNN

IT DEPARTMENT, SITM, BARABANKI PAGE | 24

 Image Caption Generator

They can be used in a variety of fields, including image and video recognition, recommender
systems, image classification, image segmentation, and medical image analysis. They can also be
used in brain-computer interfaces, natural language processing, and financial time series. Multilayer
perceptrons are modified into CNNs. Fully linked networks, or multilayer perceptrons, are those in
which all of the neurons in one layer are connected to all of the neurons in the following layer.
These networks are vulnerable to overfitting data because of their "fullyconnectedness." Changing
the weights when the loss function is decreased and randomly trimming connections are common
methods of regularisation.

6.4 Long Short Term Memory

Long Short-Term Memory (LSTM) networks, Figure 6.4.1, are a type of recurrent neural network
capable of learning order dependence in sequence prediction problems. This is a behavior required
in complex problem domains like machine translation, speech recognition, and more.

Figure 6.5 LSTM Inner Model

A challenging area of deep learning is LSTMs. Understanding LSTMs and how concepts like
bidirectional and sequence-to-sequence relate to the field can be challenging. LSTM has feedback
connections in contrast to traditional feed forward neural networks. It can analyse whole data
sequences in addition to single data points (like photos) (such as speech or video).

IT DEPARTMENT, SITM, BARABANKI PAGE | 25

 Image Caption Generator

For instance, LSTM can be used for tasks like linked, unsegmented handwriting identification,
speech recognition, and network traffic anomaly detection, or IDSs (intrusion detection systems). A
cell, an input gate, an output gate, and a forget gate make up a typical LSTM unit. The three gates
control the information flow, and the cell retains values across arbitrary time intervals.

6.5 VGG16
It is regarded as one of the best vision model architectures created to date. The most distinctive
feature of VGG16 is that it prioritised having convolution layers of 3x3 filters with a stride 1 and
always utilised the same padding and maxpool layer of 2x2 filters with a stride 2, as shown in
Figure 3.9. Throughout the entire architecture, convolution and max pool layers are arranged in the
same manner. Two FC (completely connected layers) are present at the very end, followed by a
softmax for output. The 16 in VGG16 denotes the fact that there are 16 layers with weights. This
network has over 138 million parameters, making it a sizable network.

Figure 6.6 VGG-16

In their publication "Very Deep Convolutional Networks for Large-Scale Image Recognition," K.
Simonyan and A. Zisserman from the University of Oxford introduced the convolutional neural
network model known as VGG16. In the top five tests, the model performs 92.7% accurately in
ImageNet, a dataset of over 14 million images divided into 1000 classes. It was a well-known model
that was submitted to ILSVRC-2014. By sequentially substituting several 33 kernel-sized filters for
AlexNet's big kernel-sized filters (11 and 5, respectively, in the first and second convolutional
layers), it improves upon AlexNet. Using NVIDIA Titan Black GPUs, VGG16 underwent weeks of
training.

IT DEPARTMENT, SITM, BARABANKI PAGE | 26

 Image Caption Generator

CHAPTER - 7

CODE IMPLEMENTATION

7.1 What is an Image Caption Generator?

Image caption generator is a task that involves computer vision and natural language processing
concepts to recognize the context of an image and describe it in a natural language like English.

7.2 Image Caption Generator with CNN – About the Python-based Project
The objective of our project is to learn the concepts of a CNN and LSTM model and build a
working model of Image caption generator by implementing CNN with LSTM. In this Python
project, we will be implementing the caption generator using CNN (Convolutional Neural
Networks) and LSTM (Long short term memory). The image features will be extracted from
Xception which is a CNN model trained on the imagenet dataset and then we feed the features into
the LSTM model which will be responsible for generating the image captions.

7.3 The Dataset of Python based Project

For the image caption generator, we will be using the Flickr_8K dataset. There are also other big
datasets like Flickr_30K and MSCOCO dataset but it can take weeks just to train the network so we
will be using a small Flickr8k dataset. The advantage of a huge dataset is that we can build better
models.
 Flicker8k_Dataset
 Flickr_8k_text

The Flickr_8k_text folder contains file [Link] which is the main file of our dataset that
contains image name and their respective captions separated by newline(“\n”).

7.4 Pre-requisites
This project requires good knowledge of Deep learning, Python, working on Jupyter notebooks,
Keras library, Numpy, and Natural language processing.

Make sure you have installed all the following necessary libraries:

IT DEPARTMENT, SITM, BARABANKI PAGE | 27

 Image Caption Generator

 pip install tensorflow

 keras
 pillow
 numpy
 tqdm

7.5 Image Caption Generator – Python-based Project

7.5.1 What is CNN?
Convolutional Neural networks are specialized deep neural networks that can process the data that has
an input shape like a 2D matrix. Images are easily represented as a 2D matrix and CNN is very useful
in working with images. CNN is used for image classifications and identifying if an image is a bird, a
plane Superman, etc. It scans images from left to right and top to bottom to pull out important features
from the image and combines the features to classify images. It can handle the images that have been
translated, rotated, scaled and changed in perspective.

7.5.2 What is LSTM?

LSTM stands for Long short-term memory, they are a type of RNN (recurrent neural network) which is
well suited for sequence prediction problems. Based on the previous text, we can predict what the next
word will be. It has proven itself effective from the traditional RNN by overcoming the limitations of
RNN which has short-term memory. LSTM can carry out relevant information throughout the
processing of inputs and with a forget gate, it discards non-relevant information.

7.5.3 Image Caption Generator Model

So, to make our image caption generator model, we will be merging these architectures. It is also called
a CNN-RNN model. CNN is used for extracting features from the image. We will use the pre-trained
model Xception. LSTM will use the information from CNN to help generate a description of the
image.

7.6 Project File Structure

Downloaded from dataset:

 Flicker8k_Dataset – Dataset folder which contains 8091 images.

 Flickr_8k_text – Dataset folder which contains text files and captions of images.

The below files will be created by us while making the project –

IT DEPARTMENT, SITM, BARABANKI PAGE | 28

 Image Caption Generator

 Models – It will contain our trained models.

 [Link] – This text file contains all image names and their captions after preprocessing.

 Features.p – Pickle object that contains an image and its feature vector extracted from the
Xception pre-trained CNN model.

 Tokenizer.p – Contains tokens mapped with an index value.

 [Link] – Visual representation of dimensions of our project.

 Testing_caption_generator.py – Python file for generating a caption of any image.

 [Link]– Jupyter notebook in which we train and build our image caption
generator.

7.7 Building the Python-based Project

Let’s start by initializing the Google Colab Notebook server by opening Google Colab in the browser. It will
open up the interactive Python notebook where you can run your code. Create a Python3 notebook and name it
[Link].

CODE -

1. MOUNT GOOGLE DRIVE

from [Link] import drive
[Link]('/content/drive')

2. INSTALL ALL NECESSARY PACKAGES

!pip install tensorflow keras pillow numpy tqdm

3. IMPORT ALL NECESSARY PACKAGES

import string
import numpy as np
from PIL import Image
import os
from pickle import dump, load
import numpy as np
from [Link] import Xception, preprocess_input

IT DEPARTMENT, SITM, BARABANKI PAGE | 29

 Image Caption Generator

from [Link] import load_img, img_to_array

from [Link] import Tokenizer
from [Link] import pad_sequences
from [Link] import to_categorical
import tensorflow as tf
from datetime import datetime
# from [Link] import to_categorical
from [Link] import concatenate as add
from [Link] import Model, load_model
from [Link] import Input, Dense, LSTM, Embedding, Dropout

# small library for seeing the progress of loops.

from tqdm import tqdm_notebook as tqdm
tqdm().pandas()

OUTPUT -
<ipython-input-3-1ab10fb22dc8>:22: TqdmDeprecationWarning: This function will be removed in
tqdm==5.0.0
Please use [Link] instead of tqdm.tqdm_notebook
tqdm().pandas()
0/? [00:04<?, ?it/s]

4. GETTING AND PERFORMING DATA CLEANING

The main text file which contains all image captions is [Link] in our Flickr_8k_text folder.
Have a look at the file –
The format of our file is image and caption separated by a new line (“\n”).
Each image has 5 captions and we can see that #(0 to 5)number is assigned for each caption.
We will define 5 functions:

load_doc( filename ) – For loading the document file and reading the contents inside the file into a
string.

all_img_captions( filename ) – This function will create a descriptions dictionary that maps images
with a list of 5 captions. The description dictionary will look something like this:

IT DEPARTMENT, SITM, BARABANKI PAGE | 30

 Image Caption Generator

cleaning_text( descriptions) – This function takes all descriptions and performs data cleaning. This is
an important step when we work with textual data, according to our goal, we decide what type of
cleaning we want to perform on the text. In our case, we will be removing punctuations, converting all
text to lowercase and removing words that contain numbers. So, a caption like “A man riding on a
three-wheeled wheelchair” will be transformed into “man riding on three wheeled wheelchair”

text_vocabulary( descriptions ) – This is a simple function that will separate all the unique words
and create the vocabulary from all the descriptions.

save_descriptions( descriptions, filename ) – This function will create a list of all the descriptions
that have been preprocessed and store them in a file. We will create a [Link] file to store all
the captions. It will look something like this:

CODE -
# Loading a text file into memory
def load_doc(filename):
# Opening the file as read only
file = open(filename, 'r')
text = [Link]()
[Link]()
return text

# get all imgs with their captions

def all_img_captions(filename):
file = load_doc(filename)
captions = [Link]('\n')
descriptions ={}
for caption in captions[:-1]:
img, caption = [Link]('\t')
if img[:-2] not in descriptions:
descriptions[img[:-2]] = [ caption ]
else:
descriptions[img[:-2]].append(caption)
return descriptions

IT DEPARTMENT, SITM, BARABANKI PAGE |31

 Image Caption Generator

#Data cleaning- lower casing, removing puntuations and words containing numbers
def cleaning_text(captions):
table = [Link]('','',[Link])
for img,caps in [Link]():
for i,img_caption in enumerate(caps):
img_caption.replace("-"," ")
desc = img_caption.split()
#converts to lowercase
desc = [[Link]() for word in desc]
#remove punctuation from each token
desc = [[Link](table) for word in desc]
#remove hanging 's and a
desc = [word for word in desc if(len(word)>1)]
#remove tokens with numbers in them
desc = [word for word in desc if([Link]())]
#convert back to string
img_caption = ' '.join(desc)
captions[img][i]= img_caption
return captions

def text_vocabulary(descriptions):
# build vocabulary of all unique words
vocab = set()
for key in [Link]():
[[Link]([Link]()) for d in descriptions[key]]
return vocab

#All descriptions in one file

def save_descriptions(descriptions, filename):
lines = list()
for key, desc_list in [Link]():
for desc in desc_list:
[Link](key + '\t' + desc )
data = "\n".join(lines)
file = open(filename,"w")

IT DEPARTMENT, SITM, BARABANKI PAGE | 32

 Image Caption Generator

[Link](data)
[Link]()
# Set these path according to project folder in you system
dataset_text = "/content/drive/MyDrive/ML/Flickr8k_text"
dataset_images = "/content/drive/MyDrive/ML/Flicker8k_Dataset"

#we prepare our text data

filename = dataset_text + "/" + "[Link]"
#loading the file that contains all data
#mapping them into descriptions dictionary img to 5 captions
descriptions = all_img_captions(filename)
print("Length of descriptions =" , len(descriptions))

#cleaning the descriptions

clean_descriptions = cleaning_text(descriptions)
#building vocabulary
vocabulary = text_vocabulary(clean_descriptions)
print("Length of vocabulary = ", len(vocabulary))

#saving each description to file

save_descriptions(clean_descriptions, "/content/drive/MyDrive/ML/[Link]")

OUTPUT –
Length of descriptions = 8092
Length of vocabulary = 8763

5. EXTRACTING THE FEATURE VECTOR FROM IMAGES

This technique is also called transfer learning, we don’t have to do everything on our own, we use the
pre-trained model that have been already trained on large datasets and extract the features from these
models and use them for our tasks. We are using the Xception model which has been trained on
imagenet dataset that had 1000 different classes to classify. We can directly import this model from the
[Link] . Make sure you are connected to the internet as the weights get automatically
downloaded.
Since the Xception model was originally built for imagenet, we will do little changes for integrating

IT DEPARTMENT, SITM, BARABANKI PAGE | 33

 Image Caption Generator

our model. One thing to notice is that the Xception model takes 299*299*3 image size as input. We
will remove the last classification layer and get the 2048 feature vector. model =
Xception( include_top=False, pooling=’avg’ ) The function extract_features() will extract features for
all images and we will map image names with their respective feature array. Then we will dump the
features dictionary into a “features.p” pickle file.

CODE -
def extract_features(directory):
model = Xception( include_top=False, pooling='avg' )
features = {}
for img in tqdm([Link](directory)):
filename = directory + "/" + img
image = [Link](filename)
image = [Link]((299,299))
image = np.expand_dims(image, axis=0)
#image = preprocess_input(image)
image = image/127.5
image = image - 1.0

feature = [Link](image)
features[img] = feature
return features
#2048 feature vector
features = extract_features(dataset_images)
dump(features, open("/content/drive/MyDrive/ML/features.p","wb"))

OUTPUT –

IT DEPARTMENT, SITM, BARABANKI PAGE | 34

Image Caption Generator

This process can take a lot of time depending on your system. I am using an Nvidia 1050 GPU for
training purposes so it took me around 7 minutes to perform this task. However, if you are using CPU
then this process might take 1-2 hours. You can comment out the code and directly load the features
from our pickle file.

6. LOAD THE FEATURES FILE FOR LATER USE

features = load(open("features.p","rb"))

7. LOADING DATASET FOR TRAINING MODEL

In our Flickr_8k_test folder, we have Flickr_8k.[Link] file that contains a list of 6000 image
names that we will use for training.

For loading the training dataset, we need more functions:

load_photos( filename ) – This will load the text file in a string and will return the list of image
names.

load_clean_descriptions( filename, photos ) – This function will create a dictionary that contains
captions for each photo from the list of photos. We also append the and identifier for each caption. We
need this so that our LSTM model can identify the starting and ending of the caption.

load_features(photos) – This function will give us the dictionary for image names and their feature
vector which we have previously extracted from the Xception model.

CODE -
#load the data
def load_photos(filename):
file = load_doc(filename)
photos = [Link]("\n")[:-1]
return photos
def load_clean_descriptions(filename, photos):
#loading clean_descriptions
file = load_doc(filename)
descriptions = {}
for line in [Link]("\n"):

IT DEPARTMENT, SITM, BARABANKI PAGE | 35

Image Caption Generator

 words = [Link]()
if len(words)<1 :
continue
image, image_caption = words[0], words[1:]
if image in photos:
if image not in descriptions:
descriptions[image] = []
desc = '<start> ' + " ".join(image_caption) + ' <end>'
descriptions[image].append(desc)
return descriptions
def load_features(photos):
#loading all features
all_features = load(open("/content/drive/MyDrive/ML/features.p","rb"))
#selecting only needed features
features = {k:all_features[k] for k in photos}
return features
filename = dataset_text + "/" + "Flickr_8k.[Link]"

#train = loading_data(filename)
train_imgs = load_photos(filename)
train_descriptions = load_clean_descriptions("/content/drive/MyDrive/ML/[Link]",
train_imgs)
train_features = load_features(train_imgs)

8. TOKENIZING THE VOCABULARY

Computers don’t understand English words, for computers, we will have to represent them with
numbers. So, we will map each word of the vocabulary with a unique index value. Keras library
provides us with the tokenizer function that we will use to create tokens from our vocabulary and save
them to a “tokenizer.p” pickle file.

CODE –
#converting dictionary to clean list of descriptions
def dict_to_list(descriptions):
all_desc = []
for key in [Link]():

IT DEPARTMENT, SITM, BARABANKI PAGE | 36

Image Caption Generator

[all_desc.append(d) for d in descriptions[key]]
return all_desc

#creating tokenizer class

#this will vectorise text corpus
#each integer will represent token in dictionary

from [Link] import Tokenizer

def create_tokenizer(descriptions):
desc_list = dict_to_list(descriptions)
tokenizer = Tokenizer()
tokenizer.fit_on_texts(desc_list)
return tokenizer

# give each word an index, and store that into tokenizer.p pickle file
tokenizer = create_tokenizer(train_descriptions)
dump(tokenizer, open('/content/drive/MyDrive/ML/tokenizer.p', 'wb'))
vocab_size = len(tokenizer.word_index) + 1
vocab_size

OUTPUT –
7577

9. CALCULATE THE MAXIMUM LENGTH OF DESCRIPTIONS

We calculate the maximum length of the descriptions. This is important for deciding the model
structure parameters. Max_length of the description is 32.

CODE –
#calculate maximum length of descriptions
def max_length(descriptions):
desc_list = dict_to_list(descriptions)
return max(len([Link]()) for d in desc_list)
max_length = max_length(descriptions)
max_length

IT DEPARTMENT, SITM, BARABANKI PAGE | 37

Image Caption Generator

OUTPUT –
32

10. CREATE DATA GENERATOR

Let us first see how the input and output of our model will look like. To make this task into a
supervised learning task, we have to provide input and output to the model for training. We have to
train our model on 6000 images and each image will contain 2048 length feature vector and caption is
also represented as numbers. This amount of data for 6000 images is not possible to hold into memory
so we will be using a generator method that will yield batches. The generator will yield the input and
output sequence.

For example: The input to our model is [x1, x2] and the output will be y, where x1 is the 2048 feature
vector of that image, x2 is the input text sequence and y is the output text sequence that the model has
to predict.

x1(feature vector) x2(Text sequence) y(word to predict)

feature start, two
feature start, two dogs
feature start, two, dogs drink
feature start, two, dogs, drink water
feature start, two, dogs, drink, water end

CODE –
#data generator, used by model.fit_generator()
def data_generator(descriptions, features, tokenizer, max_length):
while 1:
for key, description_list in [Link]():
#retrieve photo features
feature = features[key][0]
input_image, input_sequence, output_word = create_sequences(tokenizer, max_length,
description_list, feature)
yield ([input_image, input_sequence], output_word)
def create_sequences(tokenizer, max_length, desc_list, feature):
X1, X2, y = list(), list(), list()
# walk through each description for the image

IT DEPARTMENT, SITM, BARABANKI PAGE | 38

Image Caption Generator

for desc in desc_list:
seq = tokenizer.texts_to_sequences([desc])[0]
# split one sequence into multiple X,y pairs
for i in range(1, len(seq)):
# split into input and output pair
in_seq, out_seq = seq[:i], seq[i]
# pad input sequence
in_seq = pad_sequences([in_seq], maxlen=max_length)[0]
# encode output sequence
out_seq = to_categorical([out_seq], num_classes=vocab_size)[0]
# store
[Link](feature)
[Link](in_seq)
[Link](out_seq)
return [Link](X1), [Link](X2), [Link](y)
#You can check the shape of the input and output for your model
[a,b],c = next(data_generator(train_descriptions, features, tokenizer, max_length))
[Link], [Link], [Link]
#((47, 2048), (47, 32), (47, 7577))

OUTPUT –
((47, 2048), (47, 32), (47, 7577))

10. DEFINING THE CNN – RNN MODEL

To define the structure of the model, we will be using the Keras Model from Functional API. It will
consist of three major parts:
Feature Extractor – The feature extracted from the image has a size of 2048, with a dense layer, we
will reduce the dimensions to 256 nodes.

Sequence Processor – An embedding layer will handle the textual input, followed by the LSTM layer.

Decoder – By merging the output from the above two layers, we will process by the dense layer to
make the final prediction. The final layer will contain the number of nodes equal to our vocabulary
size.

IT DEPARTMENT, SITM, BARABANKI PAGE | 39

Image Caption Generator

CODE -
from [Link] import plot_model
# define the captioning model
def define_model(vocab_size, max_length):
# features from the CNN model squeezed from 2048 to 256 nodes
inputs1 = Input(shape=(2048,))
fe1 = Dropout(0.5)(inputs1)
fe2 = Dense(256, activation='relu')(fe1)

# LSTM sequence model

inputs2 = Input(shape=(max_length,))
se1 = Embedding(vocab_size, 256, mask_zero=True)(inputs2)
se2 = Dropout(0.5)(se1)
se3 = LSTM(256)(se2)
# Merging both models
decoder1 = add([fe2, se3])
decoder2 = Dense(256, activation='relu')(decoder1)
outputs = Dense(vocab_size, activation='softmax')(decoder2)
# tie it together [image, seq] [word]
model = Model(inputs=[inputs1, inputs2], outputs=outputs)
[Link](loss='categorical_crossentropy', optimizer='adam')
# summarize model
print([Link]())
plot_model(model, to_file='/content/drive/MyDrive/ML/[Link]', show_shapes=True)
return model

This code defines a neural network model for image captioning using a combination of CNN and
LSTM layers. The `define_model` function constructs the architecture, taking vocabulary size and
maximum sequence length as inputs. The CNN path processes image features extracted by a pre-
trained model, applying dropout for regularization and a dense layer for feature reduction. The LSTM
path handles textual sequences, employing embedding and dropout layers for sequence processing.
These paths are merged, and a decoder consisting of dense layers produces the final output, predicting
the next word in the caption. The model is compiled with categorical cross-entropy loss and the Adam
optimizer. Additionally, it prints a summary of the model architecture and generates a visual
representation of the model's structure as a PNG image for further analysis and understanding.

IT DEPARTMENT, SITM, BARABANKI PAGE | 40

Image Caption Generator

A visual representation of the final model is given below -

11. TRAINING THE MODEL

To train the model, we will be using the 6000 training images by generating the input and output
sequences in batches and fitting them to the model using model.fit_generator() method. We also save
the model to our models folder. This will take some time depending on your system capability.

CODE –
# train our model
retrain = True
print('Dataset: ', len(train_imgs))
print('Descriptions: train=', len(train_descriptions))
print('Photos: train=', len(train_features))
print('Vocabulary Size:', vocab_size)

IT DEPARTMENT, SITM, BARABANKI PAGE | 41

Image Caption Generator

print('Description Length: ', max_length)

model = define_model(vocab_size, max_length)
print(model,'model')
epochs = 8
steps = len(train_descriptions)
# making a directory models to save our models
[Link]("/content/drive/MyDrive/ML/models", exist_ok=True)
checkpoint_filepath =
"/content/drive/MyDrive/ML/models/model_time_"+[Link]().strftime("%Y%m%d_%H%M
%S") + "epoch{epoch:02d}loss{loss:.2f}.hdf5"
# creating callback to save model
model_checkpoint_callback = [Link](
filepath=checkpoint_filepath,
monitor='loss',
mode='min',
save_best_only=True)
generator = data_generator(train_descriptions, train_features, tokenizer, max_length)
if retrain:
model_path = input("enter the model path that you want to retrain ")
model = [Link].load_model(model_pat
learning_rate = learning_rate = float(input("enter the learning rate you want to set(default is 0.001) "))
[Link](loss='categorical_crossentropy',
optimizer=[Link](learning_rate=learning_rate))
[Link](generator, epochs=epochs, steps_per_epoch=steps, verbose=1,
callbacks=model_checkpoint_callback)
#[Link]("/content/drive/MyDrive/ML/models/model_" + str(i) + ".h5")

This code segment is responsible for training a neural network model for image captioning. It first
checks if retraining is required and then prints information about the dataset. The model architecture is
defined using the `define_model` function, and its summary is printed. It then initializes training
parameters like epochs and steps. The code creates a directory to save trained models and sets up a
callback to save the best-performing model during training. Next, it prompts the user to input the path
of the model to retrain and the learning rate. After loading the model, it compiles it with the specified
loss function and optimizer. Finally, it fits the model to the data using a generator, specifying the
number of epochs and steps for training, and saving the best model checkpoints.

IT DEPARTMENT, SITM, BARABANKI PAGE | 42

Image Caption Generator

OUTPUT –

IT DEPARTMENT, SITM, BARABANKI PAGE | 43

 Image Caption Generator

12. TEST THE MODEL USING THE FOLLOWING IMAGE AND CODE
The model has been trained, now, we will make a separate file testing_caption_generator.py which will
load the model and generate predictions. The predictions contain the max length of index values so we
will use the same tokenizer.p pickle file to get the words from their index values.

CODE -
from PIL import Image
img = [Link]('/content/drive/MyDrive/ML/Flicker8k_Dataset/96973080_783e375945.jpg')
# imagePath = '/content/drive/MyDrive/ML/Flicker8k_Dataset/3738685861_8dfff28760.jpg'
# img = [Link](imagePath)
Img

OUTPUT -

CODE -
!python3 '/content/drive/MyDrive/ML/testing_caption_generator.py' -i
'/content/drive/MyDrive/ML/Flicker8k_Dataset/96973080_783e375945.jpg'
# !python3 '/content/drive/MyDrive/ML/testing_caption_generator.py' -i
'/content/drive/MyDrive/ML/Flicker8k_Dataset/3738685861_8dfff28760.jpg'

13. STRUCTURE OF TESTING CAPTION GENERATOR FILE

CODE –
from [Link] import Tokenizer

IT DEPARTMENT, SITM, BARABANKI PAGE | 44

Image Caption Generator

from [Link] import pad_sequences
from [Link] import Xception
from [Link] import load_model
from pickle import load
import numpy as np
from PIL import Image
import [Link] as plt
import argparse
ap = [Link]()
ap.add_argument('-i', '--image', required=True, help="Image Path")
args = vars(ap.parse_args())
img_path = args['image']
# img_path = '/content/drive/MyDrive/ML/Flicker8k_Dataset/111537222_07e56d5a30.jpg'
def extract_features(filename, model):
try:
image = [Link](filename)
except:
print("ERROR: Couldn't open image! Make sure the image path and extension is correct")
image = [Link]((299,299))
image = [Link](image)
# for images that has 4 channels, we convert them into 3 channels
if [Link][2] == 4:
image = image[..., :3]
image = np.expand_dims(image, axis=0)
image = image/127.5
image = image - 1.0
feature = [Link](image)
return feature
def word_for_id(integer, tokenizer):
for word, index in tokenizer.word_index.items():
if index == integer:
return word
return None
def generate_desc(model, tokenizer, photo, max_length):
in_text = 'start'

IT DEPARTMENT, SITM, BARABANKI PAGE | 45

Image Caption Generator

for i in range(max_length):
sequence = tokenizer.texts_to_sequences([in_text])[0]
sequence = pad_sequences([sequence], maxlen=max_length)
pred = [Link]([photo,sequence], verbose=0)
pred = [Link](pred)
word = word_for_id(pred, tokenizer)
if word is None:
break
in_text += ' ' + word
if word == 'end':
break
return in_text
#path = 'Flicker8k_Dataset/111537222_07e56d5a30.jpg'
max_length = 32
tokenizer = load(open("/content/drive/MyDrive/ML/tokenizer.p","rb"))
model = load_model('/content/drive/MyDrive/ML/models/model_9.h5')
xception_model = Xception(include_top=False, pooling="avg")
photo = extract_features(img_path, xception_model)
img = [Link](img_path)
description = generate_desc(model, tokenizer, photo, max_length)
print("\n\n")
print(description)
[Link](img)

OUTPUT -

IT DEPARTMENT, SITM, BARABANKI PAGE | 46

Image Caption Generator

14. WHOLE GOOGLE COLAB CODE
from [Link] import drive
[Link]('/content/drive')

!pip install tensorflow keras pillow numpy tqdm

import string
import numpy as np
from PIL import Image
import os
from pickle import dump, load
import numpy as np
from [Link] import Xception, preprocess_input
from [Link] import load_img, img_to_array
from [Link] import Tokenizer
from [Link] import pad_sequences
from [Link] import to_categorical
import tensorflow as tf
from datetime import datetime
# from [Link] import to_categorical
from [Link] import concatenate as add
from [Link] import Model, load_model
from [Link] import Input, Dense, LSTM, Embedding, Dropout
# small library for seeing the progress of loops.
from tqdm import tqdm_notebook as tqdm
tqdm().pandas()
# Loading a text file into memory
def load_doc(filename):
# Opening the file as read only
file = open(filename, 'r')
text = [Link]()
[Link]()
return text
# get all imgs with their captions

IT DEPARTMENT, SITM, BARABANKI PAGE | 47

Image Caption Generator

def all_img_captions(filename):
file = load_doc(filename)
captions = [Link]('\n')
descriptions ={}
for caption in captions[:-1]:
img, caption = [Link]('\t')
if img[:-2] not in descriptions:
descriptions[img[:-2]] = [ caption ]
else:
descriptions[img[:-2]].append(caption)
return descriptions

#Data cleaning- lower casing, removing puntuations and words containing numbers
def cleaning_text(captions):
table = [Link]('','',[Link])
for img,caps in [Link]():
for i,img_caption in enumerate(caps):
img_caption.replace("-"," ")
desc = img_caption.split()
#converts to lowercase
desc = [[Link]() for word in desc]
#remove punctuation from each token
desc = [[Link](table) for word in desc]
#remove hanging 's and a
desc = [word for word in desc if(len(word)>1)]
#remove tokens with numbers in them
desc = [word for word in desc if([Link]())]
#convert back to string
img_caption = ' '.join(desc)
captions[img][i]= img_caption
return captions

def text_vocabulary(descriptions):
# build vocabulary of all unique words
vocab = set()

IT DEPARTMENT, SITM, BARABANKI PAGE | 48

Image Caption Generator

 for key in [Link]():
[[Link]([Link]()) for d in descriptions[key]]
return vocab

#All descriptions in one file

def save_descriptions(descriptions, filename):
lines = list()
for key, desc_list in [Link]():
for desc in desc_list:
[Link](key + '\t' + desc )
data = "\n".join(lines)
file = open(filename,"w")
[Link](data)
[Link]()

# Set these path according to project folder in you system

dataset_text = "/content/drive/MyDrive/ML/Flickr8k_text"
dataset_images = "/content/drive/MyDrive/ML/Flicker8k_Dataset"

#we prepare our text data

filename = dataset_text + "/" + "[Link]"
#loading the file that contains all data
#mapping them into descriptions dictionary img to 5 captions
descriptions = all_img_captions(filename)
print("Length of descriptions =" , len(descriptions))

#cleaning the descriptions

clean_descriptions = cleaning_text(descriptions)

#building vocabulary
vocabulary = text_vocabulary(clean_descriptions)
print("Length of vocabulary = ", len(vocabulary))

#saving each description to file

save_descriptions(clean_descriptions, "/content/drive/MyDrive/ML/[Link]")

IT DEPARTMENT, SITM, BARABANKI PAGE | 49

Image Caption Generator

def extract_features(directory):
model = Xception( include_top=False, pooling='avg' )
features = {}
for img in tqdm([Link](directory)):
filename = directory + "/" + img
image = [Link](filename)
image = [Link]((299,299))
image = np.expand_dims(image, axis=0)
#image = preprocess_input(image)
image = image/127.5
image = image - 1.0
feature = [Link](image)
features[img] = feature
return features

#2048 feature vector

features = extract_features(dataset_images)
dump(features, open("/content/drive/MyDrive/ML/features.p","wb"))

features = load(open("/content/drive/MyDrive/ML/features.p","rb"))

#load the data

def load_photos(filename):
file = load_doc(filename)
photos = [Link]("\n")[:-1]
return photos

def load_clean_descriptions(filename, photos):

#loading clean_descriptions
file = load_doc(filename)
descriptions = {}
for line in [Link]("\n"):
words = [Link]()
if len(words)<1 :
continue

IT DEPARTMENT, SITM, BARABANKI PAGE | 50

Image Caption Generator

 image, image_caption = words[0], words[1:]
if image in photos:
if image not in descriptions:
descriptions[image] = []
desc = '<start> ' + " ".join(image_caption) + ' <end>'
descriptions[image].append(desc)
return descriptions

def load_features(photos):
#loading all features
all_features = load(open("/content/drive/MyDrive/ML/features.p","rb"))
#selecting only needed features
features = {k:all_features[k] for k in photos}
return features
filename = dataset_text + "/" + "Flickr_8k.[Link]"

#train = loading_data(filename)
train_imgs = load_photos(filename)
train_descriptions = load_clean_descriptions("/content/drive/MyDrive/ML/[Link]",
train_imgs)
train_features = load_features(train_imgs)

#converting dictionary to clean list of descriptions

def dict_to_list(descriptions):
all_desc = []
for key in [Link]():
[all_desc.append(d) for d in descriptions[key]]
return all_desc

#this will vectorise text corpus

#each integer will represent token in dictionary

from [Link] import Tokenizer

def create_tokenizer(descriptions):

IT DEPARTMENT, SITM, BARABANKI PAGE | 51

Image Caption Generator

 desc_list = dict_to_list(descriptions)
tokenizer = Tokenizer()
tokenizer.fit_on_texts(desc_list)
return tokenizer

# give each word an index, and store that into tokenizer.p pickle file
tokenizer = create_tokenizer(train_descriptions)
dump(tokenizer, open('/content/drive/MyDrive/ML/tokenizer.p', 'wb'))
vocab_size = len(tokenizer.word_index) + 1
vocab_size

#calculate maximum length of descriptions

def max_length(descriptions):
desc_list = dict_to_list(descriptions)
return max(len([Link]()) for d in desc_list)
max_length = max_length(descriptions)
max_length

#create input-output sequence pairs from the image description.

#data generator, used by model.fit_generator()
def data_generator(descriptions, features, tokenizer, max_length):
while 1:
for key, description_list in [Link]():
#retrieve photo features
feature = features[key][0]
input_image, input_sequence, output_word = create_sequences(tokenizer, max_length,
description_list, feature)
yield ([input_image, input_sequence], output_word)
def create_sequences(tokenizer, max_length, desc_list, feature):
X1, X2, y = list(), list(), list()
# walk through each description for the image
for desc in desc_list:
# encode the sequence
seq = tokenizer.texts_to_sequences([desc])[0]

IT DEPARTMENT, SITM, BARABANKI PAGE | 52

Image Caption Generator

 # split one sequence into multiple X,y pairs
for i in range(1, len(seq)):
# split into input and output pair
in_seq, out_seq = seq[:i], seq[i]
# pad input sequence
in_seq = pad_sequences([in_seq], maxlen=max_length)[0]
# encode output sequence
out_seq = to_categorical([out_seq], num_classes=vocab_size)[0]
# store
[Link](feature)
[Link](in_seq)
[Link](out_seq)
return [Link](X1), [Link](X2), [Link](y)

#You can check the shape of the input and output for your model
[a,b],c = next(data_generator(train_descriptions, features, tokenizer, max_length))
[Link], [Link], [Link]
#((47, 2048), (47, 32), (47, 7577))

from [Link] import plot_model

# define the captioning model

def define_model(vocab_size, max_length):

# features from the CNN model squeezed from 2048 to 256 nodes
inputs1 = Input(shape=(2048,))
fe1 = Dropout(0.5)(inputs1)
fe2 = Dense(256, activation='relu')(fe1)

# LSTM sequence model

inputs2 = Input(shape=(max_length,))
se1 = Embedding(vocab_size, 256, mask_zero=True)(inputs2)
se2 = Dropout(0.5)(se1)
se3 = LSTM(256)(se2)

IT DEPARTMENT, SITM, BARABANKI PAGE | 53

Image Caption Generator

 # Merging both models
decoder1 = add([fe2, se3])
decoder2 = Dense(256, activation='relu')(decoder1)
outputs = Dense(vocab_size, activation='softmax')(decoder2)

# tie it together [image, seq] [word]

model = Model(inputs=[inputs1, inputs2], outputs=outputs)
[Link](loss='categorical_crossentropy', optimizer='adam')

# summarize model
print([Link]())
plot_model(model, to_file='/content/drive/MyDrive/ML/[Link]', show_shapes=True)
return model

# train our model

retrain = True
print('Dataset: ', len(train_imgs))
print('Descriptions: train=', len(train_descriptions))
print('Photos: train=', len(train_features))
print('Vocabulary Size:', vocab_size)
print('Description Length: ', max_length)

model = define_model(vocab_size, max_length)

print(model,'model')
epochs = 8
steps = len(train_descriptions)
# making a directory models to save our models
[Link]("/content/drive/MyDrive/ML/models", exist_ok=True)

checkpoint_filepath =
"/content/drive/MyDrive/ML/models/model_time_"+[Link]().strftime("%Y%m%d_%H%M
%S") + "epoch{epoch:02d}loss{loss:.2f}.hdf5"
# creating callback to save model
model_checkpoint_callback = [Link](
filepath=checkpoint_filepath,

IT DEPARTMENT, SITM, BARABANKI PAGE | 54

Image Caption Generator

 monitor='loss',
mode='min',
save_best_only=True)

generator = data_generator(train_descriptions, train_features, tokenizer, max_length)

if retrain:
model_path = input("enter the model path that you want to retrain ")
model = [Link].load_model(model_path)
learning_rate = learning_rate = float(input("enter the learning rate you want to set(default is 0.001) "))
[Link](loss='categorical_crossentropy',
optimizer=[Link](learning_rate=learning_rate))

[Link](generator, epochs=epochs, steps_per_epoch=steps, verbose=1,

callbacks=model_checkpoint_callback)
#[Link]("/content/drive/MyDrive/ML/models/model_" + str(i) + ".h5")

from PIL import Image

img = [Link]('/content/drive/MyDrive/ML/Flicker8k_Dataset/96973080_783e375945.jpg')
# imagePath = '/content/drive/MyDrive/ML/Flicker8k_Dataset/3738685861_8dfff28760.jpg'
# img = [Link](imagePath)
img

!python3 '/content/drive/MyDrive/ML/testing_caption_generator.py' -i
'/content/drive/MyDrive/ML/Flicker8k_Dataset/96973080_783e375945.jpg'
# !python3 '/content/drive/MyDrive/ML/testing_caption_generator.py' -i
'/content/drive/MyDrive/ML/Flicker8k_Dataset/3738685861_8dfff28760.jpg'

15. FRONT – END PART

This code is an Image Captioning application built using Streamlit, a Python library for creating web
applications. The application allows users to upload an image, upon which it generates a textual
description of the image content. The image processing and caption generation are powered by pre-
trained deep-learning models. This application harnesses the power of deep learning and Streamlit to
create an intuitive tool for automatically describing the content of uploaded images, offering both
functionality and ease of use to its users.

IT DEPARTMENT, SITM, BARABANKI PAGE | 55

Image Caption Generator

Here's a breakdown of the code:
Imports: Necessary libraries such as Streamlit, Keras for deep learning, PIL for image processing,
matplotlib for visualization, and Google Translate API for text translation are imported.

Feature Extraction: The extract_features function uses a pre-trained Xception model to extract
features from the uploaded image.

Caption Generation: The generate_desc function generates a caption for the image using a pre-trained
language model. It utilizes a tokenizer to convert text into sequences and a deep learning model trained
on image-caption pairs to predict the next word in the caption sequence.

Translation: There's a commented-out section for translating captions into different languages using
Google Translate.

Streamlit App: The main application interface is created using Streamlit. It includes a title, file
uploader for image selection, and a button to trigger caption generation. Upon button click, the
uploaded image is displayed along with the generated caption.

Overall, the code integrates deep learning models for image feature extraction and text generation into
a user-friendly web application using Streamlit.

CODE -
import streamlit as st
from [Link] import Tokenizer
from [Link] import pad_sequences
from [Link] import Xception
from [Link] import load_model
from pickle import load
import numpy as np
from PIL import Image
import [Link] as plt
from googletrans import Translator
# Function to extract features
def extract_features(image, model):

IT DEPARTMENT, SITM, BARABANKI PAGE | 56

Image Caption Generator

 image = [Link]((299, 299))
image = [Link](image)
# for images that have 4 channels, convert them into 3 channels
if [Link][2] == 4:
image = image[..., :3]
image = np.expand_dims(image, axis=0)
image = image / 127.5
image = image - 1.0
feature = [Link](image)
return feature

# Function to generate caption

def word_for_id(integer, tokenizer):
for word, index in tokenizer.word_index.items():
if index == integer:
return word
return None

def generate_desc(model, tokenizer, photo, max_length):

in_text = 'start'
for i in range(max_length):
sequence = tokenizer.texts_to_sequences([in_text])[0]
sequence = pad_sequences([sequence], maxlen=max_length)
pred = [Link]([photo, sequence], verbose=0)
pred = [Link](pred)
word = word_for_id(pred, tokenizer)
if word is None:
break
in_text += ' ' + word
if word == 'end':
break
return in_text
# Load pre-trained models and tokenizer
max_length = 32

IT DEPARTMENT, SITM, BARABANKI PAGE | 57

Image Caption Generator

tokenizer = load(open("C:/Users/yashv/Downloads/Imp Files for Frontend/tokenizer.p", "rb"))
model = load_model("C:/Users/yashv/Downloads/Imp Files for
Frontend/model_time_20240315_142904epoch07loss2.59.hdf5")
xception_model = Xception(include_top=False, pooling="avg")

def translate_text(text, target_lang):

translator = Translator()
translated_text = [Link](text, dest=target_lang).text
return translated_text

# Streamlit app
[Link]("Image Captioning App")
uploaded_file = st.file_uploader("Choose an image...", type="jpg")

if [Link]("Generate Captions") and uploaded_file is not None:

# Read uploaded image
image = [Link](uploaded_file)
[Link](image, caption="Uploaded Image", use_column_width=True)

# Convert image to numpy array

img = [Link](image)

# Extract features only if button is clicked

photo = extract_features(image, xception_model)

# Generate caption
captions = generate_desc(model, tokenizer, photo, max_length)

# Display caption
[Link]("Generated Captions:")
[Link](captions)

# translated_caption = translate_text(captions[6:len(captions)-4], "hi")

IT DEPARTMENT, SITM, BARABANKI PAGE | 58

Image Caption Generator

 # [Link]("Translated Captions in Hindi:")
# [Link](translated_caption)
# translated_caption = translate_text(captions[6:len(captions)-4], "ja")
# [Link]("Translated Captions in Japanese:")
# [Link](translated_caption)

OUTPUT –

IT DEPARTMENT, SITM, BARABANKI PAGE | 59

 Image Caption Generator

IT DEPARTMENT, SITM, BARABANKI PAGE | 60

 Image Caption Generator

CHAPTER - 8

RESULT AND ANALYSIS

8.1 APPLICATIONS
Image caption generators have a wide range of applications across various industries and domains.
Here are some of the key applications of image caption generators:

1. Accessibility: Image caption generators can make digital content more accessible to individuals
with visual impairments. By providing textual descriptions of images, these tools enable people
who use screen readers to understand the content of images shared online.

2. Content Indexing and Search: Image captions help in indexing and searching for images in
large databases or on the internet. This is particularly useful in e-commerce, stock photography,
and image libraries.

3. Social Media: Social media platforms often use image captioning to improve user experience.
Captions can be autogenerated to give context to images shared on platforms like Instagram
and Twitter.

4. Automated Video Description: Image caption generators can be extended to generate

descriptions for individual frames of videos. This is beneficial for video content accessibility.

5. Medical Imaging: In the field of healthcare, image caption generators can assist in describing
medical images like X-rays, MRIs, and CT scans. This aids in medical diagnosis and record-
keeping.

8.2 ADVANTAGES
Image caption generators offer several advantages across different domains and applications, Here
are some key advantages:
1. Accessibility: Image caption generators make digital content more accessible to individuals
with visual impairments by providing textual descriptions of images, enabling screen readers to
convey the content accurately.
2. Improved User Experience: They enhance the user experience by providing context and
information about images, making it easier for users to understand and engage with visual
content.
3. Content Indexing and Search: Image captions help in indexing and searching for images in
large databases or on the internet, improving the discoverability of visual content.
IT DEPARTMENT, SITM, BARABANKI PAGE | 61

Image Caption Generator

4. Automation: Image caption generators automate the process of adding descriptions to images,
saving time and effort compared to manual captioning.
5. Scalability: They can handle a large volume of images quickly and consistently, making them
suitable for applications with extensive image libraries or real-time content generation.

8.3 LIMITATIONS
The neural image caption generator gives a useful framework for learning to map from images to
human-level image captions. By training on large numbers of image-caption pairs, the model learns
to capture relevant semantic information from visual features. However, with a static image,
embedding our caption generator will focus on features of our images useful for image classification
and not necessarily features useful for caption generation. To improve the amount of task-relevant
information contained in each feature, we can train the image embedding model (the VGG-16
network used to encode features) as a piece of the caption generation model, allowing us to fine-
tune the image encoder to better fit the role of generating captions. Also, if we actually look closely
at the captions generated, we notice that they are rather mundane and commonplace. Take this
possible image-caption pair for instance:

Figure 8.1 The above picture depicts clear limitation of the model because it rely most on the
training dataset

This is most certainly a “giraffe standing next to a tree.” However, if we look at other pictures, we
will likely notice that it generates a caption of “a giraffe next to a tree” for any picture with a giraffe
because giraffes in the training set often appear near trees.
IT DEPARTMENT, SITM, BARABANKI PAGE | 62

Image Caption Generator

Image caption generators, while valuable, also have several limitations and challenges that can
impact their performance and effectiveness. These limitations include:

1. Accuracy and Quality: Image caption generators may produce inaccurate or low-quality
captions, especially when dealing with complex or uncommon images. They can misinterpret
the context or objects in the image, leading to incorrect descriptions.

2. Lack of Context: Caption generators may struggle to capture nuanced context or subtle details
in images, resulting in descriptions that miss important elements or fail to convey the full
meaning.

3. Ambiguity Handling: Dealing with ambiguous images or situations can be challenging for
caption generators. They may provide generic or vague descriptions when faced with unclear
visual content.

4. Limited Creativity: Image caption generators often produce factual, descriptive captions but
may lack creativity or storytelling abilities, making their output less engaging for creative or
artistic purposes.

5. Overfitting: Some caption generators can overfit to the training data, meaning they may
perform well on common images but struggle with novel or unusual images not represented in
their training data.

8.4 RESULT OVERVIEW

The image caption generator represents a sophisticated fusion of Convolutional Neural Networks
(CNN) for feature extraction and Long Short-Term Memory (LSTM) networks for language
modelling. CNNs excel in discerning intricate patterns within images, enabling the model to
accurately capture relevant visual features. These features serve as a rich foundation for the LSTM
to construct coherent and contextually appropriate captions. The LSTM's ability to comprehend
sequential data ensures that the generated captions maintain syntactic and semantic coherence.

Furthermore, the frontend interface, developed using the Streamlit library, provides an intuitive
platform for users to interact with the model seamlessly. Throughout the training process, the
Flickr8k dataset was utilized, offering a diverse array of images paired with corresponding captions
for robust model training. The integration of these components forms a comprehensive image
captioning system capable of accurately describing the content of images in natural language, with
potential applications ranging from accessibility aids for the visually impaired to enhancing image
indexing and retrieval systems.
IT DEPARTMENT, SITM, BARABANKI PAGE | 63
 Image Caption Generator

CHAPTER - 9

CONCLUSION

In conclusion, this project has successfully developed an Image Caption Generator employing
Convolutional Neural Networks (CNN) for feature extraction and Long Short-Term Memory
(LSTM) networks for language modelling. Utilizing the Streamlit library for the frontend interface
and the Flickr8k dataset for training purposes, the project has achieved its objectives of creating a
robust system capable of generating coherent and contextually relevant captions for a wide variety
of images.

The primary objective of this project was to bridge the semantic gap between images and natural
language, a challenge crucial for advancing computer vision and human-computer interaction. By
harnessing the power of CNNs to extract meaningful visual features and LSTM networks to
generate fluent captions, the project has made significant strides towards achieving this goal.

Furthermore, the relevance of this project for future use in the captioning world cannot be
overstated. As the demand for automated image description systems continues to grow across
various domains, including accessibility aids, content indexing, and multimedia retrieval, the
capabilities demonstrated by this Image Caption Generator hold immense promise. Its accuracy,
versatility, and user-friendly interface make it a valuable tool with potential applications in diverse
real-world scenarios, paving the way for further advancements in the field of computer vision and
natural language processing.

In the evolving landscape of multimedia content creation and consumption, the importance of
automated image captioning systems continues to grow. This project's contribution lies not only in
its technical advancements but also in its practical relevance and potential impact. By providing a
seamless interface for generating descriptive captions from images, the system empowers users
across various domains, including social media, e-commerce, education, and assistive technologies.
Thus, this project stands at the forefront of innovation in the captioning world, poised to make
meaningful contributions to how we interact with and understand visual content in the digital age.

IT DEPARTMENT, SITM, BARABANKI PAGE | 64

 Image Caption Generator

CHAPTER - 10

FUTURE SCOPE

1. Social Media - Reference and Keyword Modification: Enable users to modify generated
captions to suit specific social media contexts, incorporating relevant hashtags and references to
increase engagement and reach. By facilitating easy modification of captions, the image caption
generator becomes a versatile tool for crafting compelling narratives tailored to the dynamic
landscape of social media platforms, fostering greater audience engagement and interaction.

2. Medical Diagnosis: Extend the application to medical imaging, providing descriptive captions
to assist healthcare professionals in interpreting diagnostic images accurately and efficiently.
By providing supplementary textual information, the generator augments the diagnostic
process, potentially reducing interpretation errors, improving workflow efficiency, and
ultimately enhancing patient care outcomes.

3. Feature for Website When Image Does Not Load: Develop a fallback mechanism to
generate captions for images that fail to load on websites, ensuring seamless user experience
and reducing the need for manual backend updates. This feature not only enhances accessibility
for all users but also saves time and effort for website administrators who would otherwise need
to manually update alt text or descriptions for each image.

4. Enhanced Accessibility Features: Implementing support for generating descriptive captions

for visually impaired users, aiding in accessibility and inclusivity across digital platforms. This
feature could include integration with screen readers and voice assistants to deliver captions
audibly.

5. Integration with Augmented Reality (AR) Applications: Enabling real-time image

captioning within AR environments, enhancing user experiences in navigation, education, and
entertainment realms. This integration could provide contextual information about objects or
scenes viewed through AR devices, enriching the user's understanding and interaction with the
physical world.

IT DEPARTMENT, SITM, BARABANKI PAGE | 65

 Image Caption Generator

Appendix- A
LIST OF FIGURES
Figure No. Description Page No.

2.1 Glimpse of Flickr8k Image Dataset 5

2.2 Model Image Caption Generator 6

2.3 Forget, Input, Output Gate 7

3.1 Novel Captions Generation 10

3.2 Diagram of Multimodal Space-Based Captioning 11

3.3 Diagram of Encoder-Decoder Architecture 13

3.4 Diagram of Compositional Network Captioning 14

4.1 Required Platform for Implementation 17

5.1 Proposed Model of Image Caption Generator 18

5.2 System Architecture of Image Caption Generator 18

5.3 Overview of Image Caption Generator 19

5.4 DFD Diagram 20

5.5 State Chart Diagram of Steps Taken by System 21

6.1 Methodology Flow 22

6.2 VGG16 Model 23

6.3 Recurrent Neural Network 24

6.4 Convolutional Neural Network 24

6.5 Long Short-Term Memory Inner Model 25

6.6 VGG-16 26

8.1 Limitation of Model 62

IT DEPARTMENT, SITM, BARABANKI x

 Image Caption Generator

Appendix- B
ABBREVIATIONS

RNN Recurrent Neural Network

CNN Convolutional Neural Network
LSTM Long Short-Term Memory
GPU Graphics Processing Unit
TPU Tensor Processing Unit

IT DEPARTMENT, SITM, BARABANKI xi

 Image Caption Generator

Appendix- C
REFERENCES
[1] S. ALBAWI and T. A. MOHAMMED, "Understanding of a Convolutional Neural Network," in
ICET, Antalya,2017.

[2] S. Hochreiter, "LONG SHORT-TERM MEMORY," Neural Computation, December 1997.

[3] O. Vinyals, A. Toshev, S. Bengio and D. Erhan, "A Neural Image Caption Generator," CVPR
2015 Open Access Repository, vol. Xiv, 17 November 2014.

[4] D. S. Whitehead, L. Huang, H. and S.-F. Chang, "Entity- aware Image Caption Generation," in
Empirical Methods in Natural Language Processing, Brussels, 2018.

[5] C. Elamri and T. Planque, "Automated Neural Image Caption Generator for Visually Impaired
People,"California, 2016.

[6] G. Ding, M. Chen, S. Zhao, H. Chen, J. Han and Q. Liu, "Neural Image Caption Generation with
Weighted Training and Reference," Cognitive Computation, 08 August 2018

[7] J. Chen, W. Dong and M. Li, "Image Caption GeneratorBased On Deep Neural Networks,"
March 2018.

[8] S. Bai and S. An, "A Survey on Automatic Image CaptionGeneration," Neurocomputing, 13
April 2018.

[9] R. Staniute and D. Sesok, "A Systematic Literature Review on Image Captioning," Applied
Sciences, vol. 9,no. 10, 16 March 2019.

[10] J. Hessel, N. Savva and M. J. Wilber, "Image Representations and New Domains in Neural
Image Captioning," ACL Anthology, vol. Proceedings of the Fourth Workshop on Vision and
Language, p. 29–39, 18September 2015.

[11] M. Z. Hossain, F. SOHEL, M. F. SHIRATUDDIN and H. LAGA, "A Comprehensive Survey

of Deep Learning for Image Captioning," ACM Journals, vol. 51, no. 6, 14 Octomber 2018.

[12] A. Farhadi, M. Hejrati, M. A. Sadeghi and P. Young, "Every PictureTells a Story: Generating
SentencesfromImages," in ACM Digital Library, 2010.

[13] S. Yan, F. wu, J. Smith and W. Lu, "ImageCaptioning via aHierarchical Attention Mechanism
and Policy GradientOptimization," LATEX CLASS FILES, vol. 14, 11 January2019.

IT DEPARTMENT, SITM, BARABANKI xii

 Image Caption Generator

[14] A. Karpathy and L. Fei-Fei, "Deep Visual-Semantic Alignments for Generating Image
Descriptions," CVPR 2015 Paper, December 2014.

[15] K. Simonyan and A. Zisserman, "VERY DEEP CONVOLUTIONAL NETWORKS FOR

LARGE-SCALE IMAGE RECOGNITION," in ICLR, 2015.

[16] J. Donahue, L. A. Hendricks and M. Rohrbach, "Long- term Recurrent Convolutional

Networks for Visual Recognition and Description," CVPR 2015, vol. 14, 31 May 2016.

IT DEPARTMENT, SITM, BARABANKI xiii