Module-1

Introducing Deep Learning: Biological and Machine Vision:

Biological Vision,
Machine Vision: The Neocognitron, LeNet-5, The Traditional Machine Learning Approach,
ImageNet and the ILSVRC, AlexNet, TensorFlow Playground.
Human and Machine Language: Deep Learning for Natural Language Processing: Deep
Learning Networks Learn Representations Automatically, Natural Language Processing, A
Brief History of Deep Learning for NLP,
Computational Representations of Language: One-Hot Representations of Words, Word
Vectors, Word-Vector Arithmetic, word2viz, Localist Versus Distributed Representations,
Elements of Natural Human Language.
Biological Vision
Definition
Biological vision is the process by which living organisms perceive and understand visual
information from their surroundings using the eyes and brain.

Working of Biological Vision

1. Light Reception
Light from objects enters the eye and falls on the retina.
The retina contains photoreceptor cells that convert light into electrical signals.
2. Signal Transmission
These electrical signals are sent to the brain through the optic nerve.
3. Primary Visual Cortex (V1)
The signals first reach the primary visual cortex (V1) in the brain.
This area is responsible for initial processing of visual information.
4. Simple Cells
Neurons called simple cells respond to basic visual features such as straight edges
and lines at specific orientations (horizontal, vertical, slanted).
5. Complex Cells
Output from simple cells is combined by complex cells.
Complex cells detect more complicated patterns like corners, curves, and shapes.
6. Hierarchical Processing
Visual information passes through multiple layers of neurons.
At each level, the information becomes more detailed and meaningful.
7. Object Recognition
Higher areas of the visual cortex recognize complete objects such as animals,
humans, faces, and everyday objects.
Some regions are specialized for color, motion, and face detection.
Machine Vision
Definition
Machine vision is a field of computer science that enables machines to see, analyze, and
understand images or visual data, similar to how humans use their vision.

Inspiration from Biological Vision

Machine vision is inspired by the biological visual system of humans and animals.
Research on how the human brain processes visual information, especially the hierarchical
structure of the visual cortex, has influenced modern machine vision techniques.

Historical Background
Early studies of vision include:
• Development of vision in early organisms like trilobites.
• Research by Hubel and Wiesel (1959), which showed that the brain processes visual
information in layers, starting from simple features to complex patterns.
These studies laid the foundation for machine-based visual systems.

Approaches in Machine Vision

Machine vision evolved through two main approaches:
1. Traditional Machine Learning Approach
• Uses manually designed features such as edges, corners, and textures.
• Feature extraction and classification are done as separate steps.
• Performance depends heavily on human expertise.
• Less effective for complex and large-scale image data.
2. Deep Learning Approach
• Inspired by the layered structure of the biological visual cortex.
• Uses neural networks with multiple layers to automatically learn features from raw
images.
• Early layers detect simple features (edges, lines), while deeper layers detect complex
objects.
• More accurate, scalable, and powerful compared to traditional methods.

Importance of Deep Learning in Machine Vision

Deep learning has revolutionized machine vision by:
 • Reducing manual feature engineering
• Handling large datasets efficiently
• Achieving high accuracy in tasks like image classification, object detection, and face
recognition

Differences between Biological Vision and Machine Vision

Biological Vision Machine Vision

Biological vision is the natural process by which

Machine vision is an artificial system that
humans and animals see and understand the
enables machines to interpret visual data.
environment.

It uses eyes, retina, optic nerves, and the brain It uses cameras, sensors, and computer
for visual processing. algorithms to process images.

Visual processing is carried out by neurons in Visual processing is carried out using
the visual cortex. algorithms and neural networks.

Features are either manually designed

Feature detection happens naturally through
(traditional ML) or automatically learned
simple and complex cells.
(deep learning).

It is highly efficient and adaptive to changes in It requires large datasets and training to
lighting and environment. handle variations accurately.

Learns continuously through experience without Needs training using labeled data and
explicit programming. predefined models.

Can easily recognize objects with very little Usually requires large amounts of data for
data. good performance.

Served as the inspiration for modern deep Is an application of computer science

learning–based vision systems. inspired by biological vision.

The Neocognitron
Introduction
The Neocognitron is an early and important model in machine vision. It was proposed in the
late 1970s by Japanese engineer Kunihiko Fukushima. This model was inspired by how
the human visual system processes visual information.

Inspiration from Biological Vision

The Neocognitron was directly inspired by the research of Hubel and Wiesel, who
discovered simple cells and complex cells in the primary visual cortex.
• Simple cells detect basic features like edges and line orientations.
• Complex cells combine simple features to detect more complex shapes.
Fukushima clearly mentioned Hubel and Wiesel’s work in his research and used the same
terminology of simple and complex cells.

Architecture of the Neocognitron

1. Hierarchical Structure
o The Neocognitron consists of multiple layers of artificial neurons.
o These layers are arranged in a hierarchical manner, similar to the biological
visual cortex.
2. Simple Cell Layers
o The first layers contain simple-cell-like neurons.
o They detect basic features such as edges and line orientations.
3. Complex Cell Layers
o The next layers contain complex-cell-like neurons.
o They combine information from simple cells to recognize shapes and
patterns.
4. Increasing Abstraction
o As data moves deeper into the network, the representation becomes more
complex and abstract.
o Deeper layers can recognize complete objects rather than simple lines.

Importance of the Neocognitron

• It was one of the first deep neural network models for vision.
• It introduced the idea of layered feature learning, which is the foundation of modern
Convolutional Neural Networks (CNNs).
• It showed that machines can learn visual patterns automatically without manually
designing features.
LeNet-5
Introduction
LeNet-5 is an early and influential Convolutional Neural Network (CNN) proposed by
Yann LeCun in 1998.
It was designed mainly for handwritten digit recognition and became one of the first
successful deep learning models in machine vision.

Motivation and Background

LeNet-5 was inspired by:
• The biological visual system (simple and complex cells discovered by Hubel and
Wiesel)
• The Neocognitron model proposed by Kunihiko Fukushima
Unlike earlier models, LeNet-5 was more accurate, efficient, and trainable due to better
data, faster computers, and the use of backpropagation.

Architecture of LeNet-5
LeNet-5 follows a hierarchical layered structure:
1. Input Layer
o Takes a grayscale image of a handwritten digit (for example, 32×32 pixels).
2. Convolution Layers
o Extract simple features such as edges and lines.
o Similar to simple cells in the human visual cortex.
3. Subsampling (Pooling) Layers
o Reduce the size of feature maps.
o Make the model more robust to small shifts in the image.
4. Deeper Convolution Layers
o Combine simple features into more complex patterns like curves and shapes.
5. Fully Connected Layers
o Use the extracted features to perform classification.
6. Output Layer
o Produces probability scores for digits 0 to 9.
o The digit with the highest probability is selected as the output.
Learning Method
LeNet-5 uses backpropagation to adjust weights across all layers.
This allows the network to learn features automatically instead of relying on manual feature
engineering.

Applications and Importance

• Used by the United States Postal Service (USPS) to read ZIP codes automatically.
• First practical demonstration of CNNs for real-world problems.
• Laid the foundation for modern deep learning models like AlexNet.
• Showed the power of end-to-end learning from raw images.

The figure shows the hierarchical structure of LeNet-5. The leftmost block represents the
input image. The next layers extract simple features such as edges using convolution
operations. As the data moves to deeper layers, the features become more complex and
abstract. Finally, the fully connected layers produce probability outputs for digits 0 to 9,
allowing correct classification of handwritten digits.

ImageNet and the ILSVRC

Introduction
ImageNet is a large-scale, publicly available image dataset created to support research in
machine vision and deep learning. It played a major role in advancing modern image
recognition systems.

ImageNet Dataset
1. Creator
ImageNet was developed by Fei-Fei Li and her research team in 2009.
2. Size of the Dataset
o Contains about 14 million labeled images
 o Images are organized into around 22,000 categories
3. Types of Categories
The categories include a wide variety of real-world objects such as animals, vehicles,
food items, and natural objects.
This diversity makes ImageNet suitable for training powerful vision models.
4. Importance of ImageNet
o Provided much more data compared to earlier datasets like handwritten digits
used in LeNet-5
o Enabled deep neural networks to learn complex visual features
o Became a standard benchmark dataset in computer vision research

ImageNet Large Scale Visual Recognition Challenge (ILSVRC)

1. Definition
ILSVRC is an annual open competition started in 2010 using a subset of the
ImageNet dataset.
2. Dataset Used in ILSVRC
o Around 1.4 million images
o Spread across 1,000 categories
3. Purpose of the Challenge
o To evaluate and compare the performance of different machine vision
algorithms
o To push the development of more accurate image recognition models
4. Special Feature
Many categories include similar objects, such as different breeds of dogs, which
tests the model’s ability to distinguish subtle differences between images.

Significance of ImageNet and ILSVRC

• Encouraged the development of deep learning models like AlexNet
• Shifted machine vision from traditional feature engineering to deep learning
• Established a standard benchmark for measuring progress in visual recognition
AlexNet
Introduction
AlexNet is a deep convolutional neural network proposed in 2012 by Alex Krizhevsky and
Ilya Sutskever, under the guidance of Geoffrey Hinton.
It achieved a breakthrough performance in the ImageNet Large Scale Visual Recognition
Challenge (ILSVRC) and marked the beginning of the deep learning revolution in machine
vision.
Background and Importance
Before AlexNet, most ILSVRC participants used traditional machine learning with manual
feature engineering.
AlexNet was the first deep learning model to significantly outperform all traditional
approaches, reducing the image classification error by a large margin.

Architecture of AlexNet
AlexNet follows a hierarchical CNN architecture similar to LeNet-5 but is much deeper and
more powerful.
1. Input Layer
o Takes a color image as input.
2. Convolutional Layers (CONV)
o Extract low-level features such as edges and textures in early layers.
o Deeper convolution layers extract complex patterns like shapes and object
parts.
3. Activation Function (ReLU)
o Uses Rectified Linear Unit (ReLU) to speed up training and improve
performance.
4. Pooling Layers
o Reduce spatial dimensions and make the model robust to small variations.
5. Fully Connected Layers (FC)
o Combine extracted features to perform final classification.
6. Output Layer
o Produces probability scores for different object categories (e.g., cat, dog,
horse, flower).

Key Innovations in AlexNet

• Use of large ImageNet dataset (1.4 million images)
• Training on GPUs, which greatly increased speed
• Introduction of ReLU activation
• Use of dropout to reduce overfitting
• Deeper architecture compared to earlier models
Performance in ILSVRC
• AlexNet won ILSVRC 2012 by a huge margin.
• After 2012, all top-performing ILSVRC models were deep learning based.
• By 2015, machine accuracy surpassed human accuracy on ImageNet.

The figure shows the hierarchical structure of AlexNet. The input image is processed through
multiple convolutional layers to extract simple features like edges in early layers and
complex features in deeper layers. The extracted features are passed to fully connected
layers, which output probabilities for different object classes. This hierarchical feature
learning is similar to the biological visual system.

Deep Learning Networks Learn Representations Automatically

Introduction
Deep learning is a subfield of machine learning in which artificial neurons are arranged into
multiple layers to form deep neural networks. These networks are capable of learning useful
representations directly from raw data.

Machine Learning and Representation Learning

Machine learning techniques can be broadly divided into traditional machine learning and
representation learning. Representation learning refers to methods that automatically learn
features from data. Deep learning is the most popular and powerful approach within
representation learning.
The terms feature and representation are often used interchangeably, as both refer to
meaningful information extracted from raw data.
Traditional Machine Learning Approach
In traditional machine learning, raw data such as text, images, or speech is first converted
into features using manual feature engineering. These features are then provided to
algorithms like linear regression, decision trees, random forests, or support vector machines.
This approach depends heavily on human expertise. Feature design is time-consuming,
highly specialized, and often needs repeated refinement.

Deep Learning Approach

In deep learning, raw data is directly fed into a neural network. The network learns features
automatically during training:
• The first layer learns simple patterns
• Each subsequent layer learns more complex and abstract patterns
This layered learning process removes the need for manual feature engineering and
improves adaptability to new data.

Explanation of Figure 2.1

Figure 2.1 shows a Venn diagram that explains the relationship between machine learning
techniques.
• The outer region represents Machine Learning.
• One section shows Traditional Machine Learning, which includes algorithms like
regression, decision trees, random forests, and support vector machines.
• Another section represents Representation Learning, which focuses on automatic
feature learning.
• Deep Learning (Artificial Neural Networks) is shown as a subset within
representation learning.
The figure clearly highlights that deep learning differs from traditional ML by learning features
automatically instead of relying on handcrafted features.

Advantages of Representation Learning

• Reduces dependency on domain experts
• Eliminates manual feature engineering
• Learns features quickly during training
• Adapts easily to changes in data and tasks
Natural Language Processing (NLP)
Introduction
Natural Language Processing (NLP) is a field of study that focuses on enabling machines to
understand, process, and generate human language. It deals with naturally spoken or written
language such as text documents, emails, messages, and speech.

Fields Involved in NLP

NLP combines knowledge from multiple disciplines:
• Computer Science – for algorithms and data processing
• Artificial Intelligence – for intelligent decision-making
• Linguistics – for understanding language structure and meaning
Because of this, NLP is considered an interdisciplinary field.

What NLP Does

NLP allows machines to automatically perform tasks involving human language, such as:
• Understanding text or speech
• Extracting useful information
• Assisting humans in language-related tasks
Languages like programming code or spreadsheet symbols are not considered natural
language.
Applications of NLP
1. Document Classification
Automatically classifies documents such as emails, tweets, or reviews into categories
like spam, urgency level, or sentiment.
2. Machine Translation
Translates text from one language to another, for example English to German or
Mandarin.
3. Search Engines
Helps in auto-completion of queries and predicting what users are searching for.
4. Speech Recognition
Converts spoken language into text and performs actions using voice commands, as
in virtual assistants.
5. Chatbots
Enables systems to conduct conversations with users, especially for customer
support and service applications.

Complexity Levels in NLP

• Simple NLP tasks: spell checking, synonym suggestions, keyword search
• Intermediate tasks: document classification, sentiment analysis, named entity
extraction
• Complex tasks: machine translation, question answering, and conversational
chatbots
Deep learning is mainly used for intermediate and complex NLP tasks.

Explanation of Figure 2.2

Figure 2.2 shows a Venn diagram illustrating the position of NLP.
It shows that NLP lies at the intersection of Computer Science, Linguistics, and Artificial
Intelligence. This indicates that NLP requires computational techniques, language
understanding, and intelligent reasoning to process human language effectively.
Computational Representations of Language
Introduction
For a machine or deep learning model to process human language, the language must be
converted into a numerical form. Computers cannot understand text directly, so words and
sentences are represented using numbers, usually in the form of vectors or matrices.
Two common techniques for representing language numerically are:
• One-hot encoding
• Word vectors
This section mainly focuses on one-hot representation of words.

One-Hot Representation of Words

Definition
One-hot encoding is a traditional method used to represent words numerically in natural
language processing. In this method, each word is represented as a binary vector containing
only 0s and a single 1.

Working of One-Hot Encoding

• Each unique word in the entire corpus is assigned a unique row.
• Each word position in a sentence is represented as a column.
• If a word appears at a particular position, the corresponding cell is marked as 1.
• All other cells in that column are marked as 0.
If a corpus contains:
• 100 unique words → the matrix has 100 rows
• 1,000 unique words → the matrix has 1,000 rows
Thus, the size of the matrix depends on the vocabulary size.

Characteristics of One-Hot Encoding

• Values are binary (0 or 1)
• Each column has only one value equal to 1
• The matrix is sparse, meaning most values are 0
• Words are treated as independent, with no information about similarity or meaning
Explanation of Figure 2.4
Figure 2.4 illustrates one-hot encoding using the sentence “The bat sat on the cat”.
Each unique word in the sentence corresponds to a row, and each word position
corresponds to a column. A value of 1 indicates the presence of a word at a specific position,
while 0 indicates absence. For example, the word “the” appears in multiple positions and is
marked accordingly with 1s in its row.

Advantages of One-Hot Encoding

• Simple and easy to understand
• Can be directly used as input to machine learning and deep learning models

Limitations of One-Hot Encoding

• Produces very large and sparse matrices for large vocabularies
• Does not capture relationships or similarity between words
• Inefficient for complex natural language tasks

Word Vectors
Introduction
Word vectors are a numerical representation of words used in Natural Language Processing
(NLP). They are an advanced alternative to one-hot encoding and are designed to capture
not only the position of words but also their meaning and relationships.
Word vectors are also known as word embeddings or vector-space embeddings.

Need for Word Vectors

One-hot encoding represents words using large, sparse binary vectors and does not capture
any semantic relationship between words. For example, words like king and queen are
treated as completely unrelated.
Word vectors overcome this limitation by representing words as dense numerical vectors
that preserve semantic similarity.

Concept of Vector Space

In word vector representation:
• Each word is mapped to a point in a multi-dimensional vector space
• Words with similar meanings appear close to each other
• Each word is represented by a vector of real numbers
Initially, word vectors are assigned randomly, but during training, their positions are adjusted
based on how words appear together in text.

Learning Word Vectors Using Context

Word vectors are learned by analyzing the context of words in a corpus.
• A target word is selected
• Nearby words within a fixed window are treated as context words
• The model learns to predict the target word using its context (or vice versa)
Words that appear in similar contexts gradually move closer in vector space, following the
idea:
“You shall know a word by the company it keeps.”

Popular Word Vector Techniques

1. Word2Vec
o Learns word vectors by predicting words from their context
o Efficient and widely used
2. GloVe (Global Vectors)
o Uses global word co-occurrence statistics
o Captures both local and global context information

Explanation of Figure 2.6

Figure 2.6 shows a three-dimensional representation of word meaning in vector space.
Each word is represented by coordinates along the x, y, and z axes. Words with related
meanings (such as programming languages, days of the week, or gender relationships like
king–queen) appear close together, demonstrating how semantic relationships are preserved
in word vectors.
Advantages of Word Vectors
• Dense and information-rich representation
• Capture semantic similarity between words
• Reduce dimensionality compared to one-hot encoding
• Well suited for deep learning–based NLP models

Word-Vector Arithmetic
Introduction
Word-vector arithmetic is a powerful property of word embeddings in Natural Language
Processing (NLP). Because word vectors store meaning as positions in a vector space,
mathematical operations on these vectors can capture meaningful relationships between
words.

Meaning Representation in Vector Space

In word embeddings, each word is represented as a point in a multi-dimensional vector
space.
The direction and distance between word vectors represent semantic relationships such
as:
• Gender (man ↔ woman)
• Country–capital relationships (France ↔ Paris)
• Organization–product relationships
These relationships are preserved consistently across the vector space.
Concept of Word-Vector Arithmetic:
Since relationships are encoded as vector movements, we can perform arithmetic operations
on word vectors.
The general idea is:
• Subtract a relationship from one word
• Add another related concept
• Obtain a new word that matches the combined meaning
This works because similar semantic relationships correspond to similar vector directions.
Word vectors allow meaningful arithmetic operations that capture relationships between
words.
• Example 1 (Gender):
Vking−Vman+Vwoman≈Vqueen
Here, the vector difference between "king" and "man" represents masculinity. Adding
"woman" shifts it to the female counterpart, resulting in "queen."
• Example 2 (Company-Product):
𝑉𝑀𝑖𝑐𝑟𝑜𝑠𝑜𝑓𝑡 − 𝑉𝑊𝑖𝑛𝑑𝑜𝑤𝑠 + 𝑉𝐴𝑛𝑑𝑟𝑜𝑖𝑑 ≈ 𝑉𝐺𝑜𝑜𝑔𝑙𝑒
The vector difference between a company and its product can be applied to another product
to identify the corresponding company.
• Example 3 (Country-Capital):
VParis−VFrance+VItaly≈VRome
This shows that relationships like gender, capital-country, and company-product are
captured as directions and distances in vector space.

Figure 2.7 shows examples of word-vector arithmetic.

It demonstrates how vector operations can capture relationships such as:
• king − man + woman = queen
• bezos − amazon + tesla = musk
• windows − microsoft + google = android
These examples show that word vectors preserve meaningful relationships and allow
arithmetic operations to produce logically related words.
Importance in NLP Applications
• Helps models understand semantic relationships
• Useful in tasks like analogy solving, question answering, and information retrieval
• Provides a strong foundation for deep learning models in NLP
Word2Viz
Introduction
Word2Viz is an interactive visualization tool used to explore word vectors (word
embeddings). It helps in developing an intuitive understanding of how words are
represented in vector space and how their meanings relate to one another in Natural
Language Processing (NLP).

Purpose of Word2Viz
Word vectors usually exist in high-dimensional spaces (for example, 50 or more
dimensions), which are difficult for humans to visualize. Word2Viz simplifies this by reducing
high-dimensional word vectors into two dimensions, allowing users to visually inspect
relationships between words.

Working of Word2Viz
• Word2Viz uses a pre-trained word-vector model built from a very large natural
language corpus.
• The vectors are compressed into a 2D space and plotted on an x–y coordinate
system.
• Axes are defined using meaningful word pairs (for example, she–he or woman–
queen).
• Words are placed on the graph based on their semantic similarity and relationships.

Exploring Relationships Using Word2Viz

Word2Viz allows users to explore different semantic relationships by selecting views such
as:
• Gender analogies (e.g., princess–prince, duchess–duke)
• Adjective analogies (e.g., small–smaller, nice–nicer)
• Number representations
• Verb tenses
• Job-related terms
Words with similar meanings or roles appear close to each other in the plot.
Explanation of Figure 2.8
Figure 2.8 shows the default screen of the Word2Viz tool.
The x-axis represents gender using the reference words she and he, while the y-axis
represents social status using woman and queen. Words are plotted based on their
meanings. Female-related words appear toward one side of the axis and male-related words
toward the other, while more royal or higher-status words appear higher on the graph.

Detecting Bias Using Word2Viz

Word2Viz can also reveal biases present in word vectors.
By exploring views such as jobs or verb tenses, it becomes clear that word embeddings may
reflect gender or societal biases that exist in the training data. Identifying and reducing such
biases is an important research area in NLP.

Importance of Word2Viz
• Helps understand how word vectors encode meaning
• Useful for analyzing strengths and weaknesses of embeddings
• Assists in detecting bias in NLP models
• Acts as both an educational and analytical tool
Elements of Natural Human Language
Introduction
Natural human language is not made up of words alone. Words themselves are built from
smaller language units, and words combine to form more complex structures. These
elements range from basic sound units to highly abstract meaning. Understanding these
elements is important for building effective Natural Language Processing (NLP) systems.

Hierarchy of Language Elements

Natural language elements can be arranged from simple to complex, where each element
acts as a building block for the next level. As we move upward, the abstractness and
complexity increase, making them harder to model computationally.
The main elements are:
• Phonemes
• Morphemes
• Words
• Syntax
• Semantics

Phonology (Phonemes)
Phonology deals with how language sounds when spoken.
• Phonemes are the smallest sound units in a language.
• Traditional ML represents speech by mapping audio segments to known phonemes.
• Deep learning automatically learns phoneme representations from audio and
encodes them as vectors.
(Although this book focuses on text, these techniques also apply to speech data.)

Morphology (Morphemes)
Morphology studies the structure of words.
• Morphemes are the smallest units of meaning.
• Example: out + go + ing = outgoing
Traditional ML identifies morphemes using predefined lists.
Deep learning learns morphemes automatically and represents them as vectors, which can
be combined to form word vectors.

Words
Words are formed from phonemes (speech) or morphemes (text).
In NLP, words are the most commonly used unit because:
• They are easy to define and tokenize
• They are well studied
• Word vectors are efficient and accurate
Traditional ML uses one-hot encoding, which is sparse and rigid.
Deep learning uses word vectors, which are dense, flexible, and capture meaning.
Syntax
Syntax refers to how words are arranged into phrases and sentences.
It defines grammatical structure.
• Traditional ML assigns phrases to fixed grammatical categories.
• Deep learning represents words and phrases as vectors and combines them
hierarchically using neural networks.

Semantics
Semantics deals with the meaning of sentences.
Meaning depends on words, syntax, and broader context.
• Traditional ML struggles with semantics because it cannot handle ambiguity well.
• Deep learning represents meaning using vectors and combines phrase-level vectors
to infer semantic meaning.
Semantics is the most abstract and complex element of language.

Explanation of Figure 2.9

Figure 2.9 shows the relationship between elements of natural human language.
Phonemes and morphemes form words, words combine into syntax, and syntax leads to
semantics. The figure also shows that as we move from left to right, the elements become
more abstract and complex, making them harder to model in NLP systems.