Unit-5
CASE STUDY AND APPLICATIONS
ImageNet & Detection, Audio WaveNet
1. ImageNet
What is ImageNet?
ImageNet is a massive visual dataset used for training and benchmarking computer vision models.
• 14+ million images
• 20,000+ categories (WordNet synsets)
• Widely used for:
o Classification
o Localization
o Object Detection
ImageNet Challenge Tasks (ILSVRC)
1. Classification
Predict the label of the dominant object.
Input Image → Neural Network → “Dog”
2. Localization
Predict the label + bounding box.
+-------------------------+
| (dog) |
| +---------+ |
| | dog | |
| | box | |
| +---------+ |
+-------------------------+
3. Object Detection
Predict multiple bounding boxes + labels.
+-------------------------------+
| person [box] |
| +---------+ |
| | person | |
| +---------+ |
| dog [box] |
| +-----------+ |
| | dog | |
| +-----------+ |
+-------------------------------+
Why ImageNet Was Important
• Enabled the deep learning revolution in 2012 (AlexNet).
• Drove rapid architecture improvements:
o AlexNet (2012)
o VGG (2014)
o GoogLeNet (2014)
o ResNet (2015)
Typical CNN Pipeline on ImageNet
Image → Convolution → ReLU → Pooling → … → Fully Connected → Softmax → Class
2. Object Detection
Object detection = What + Where
Outputs:
• Class of each object
• Bounding box (x, y, width, height)
Two Main Families of Detectors
A. Two-Stage Detectors (Region-based)
Examples: R-CNN → Fast R-CNN → Faster R-CNN
Pipeline Diagram
Image
↓
Region Proposal Network (RPN)
↓ proposals
RoI Pooling
Classifier + Bounding Box Regressor
Final Labels + Refined Boxes
Characteristics
• High accuracy
• Slower
• Used for applications needing precision (medical imaging, robotics)
B. One-Stage Detectors (Direct)
Examples: YOLO, SSD, RetinaNet
Pipeline Diagram
Image
Single CNN Network (no RPN)
Grid Predictions (class + box for each grid cell)
Final Detections
Characteristics
• Real-time
• Slightly less accurate (but modern YOLO versions are very strong)
• Used for:
o Self-driving
o Real-time camera analytics
o Drones
YOLO Prediction Diagram
Image → Divided into grid cells (e.g., 13×13)
Each cell predicts:
- bounding boxes
- confidence scores
- class probabilities
Example grid:
+---+---+---+
| | | |
+---+---+---+
| |X| |
+---+---+---+
| | | |
+---+---+---+
Cell “X” predicts an object.
Evaluation Metric: mAP (mean Average Precision)
mAP measures how well a detector:
• finds objects (recall)
• labels correctly (precision)
3. Audio WaveNet
WaveNet is a deep generative model for raw audio created by DeepMind.
It produces speech sample-by-sample, leading to extremely natural sound.
Key Concept: Autoregressive Audio Generation
x[t] depends on (x[t-1], x[t-2], x[t-3], ...)
WaveNet learns:
P(x[t] | x[t-1], x[t-2], ..., x[0])
It predicts the next audio sample based on previous samples.
WaveNet Architecture Diagram (Simplified)
Input Audio Samples
[ Causal Convolutions ]
[ Dilated Convolution Blocks ]
Softmax Output → Next Sample Prediction
Dilated Causal Convolution (key innovation)
Normal conv:
o-o-o-o
Small receptive field.
Dilated conv (rate = 2):
o-·-o-·-o-·-o
Spacing increases → receptive field grows exponentially without huge layers.
Why WaveNet Was Revolutionary
• Produced very natural speech compared to older parametric vocoders.
• Could mimic:
o human voices
o accents
o emotional tone
o musical instruments
• Led to production TTS systems (e.g., Google Assistant).
Challenges
• Autoregressive = slow inference
• Needs many samples/second (16k–48k Hz)
Solutions:
• Parallel WaveNet
• WaveGlow
• MelGAN / HiFiGAN (non-autoregressive)
Summary Table
Topic Key Idea Example Models Notes
ImageNet Large dataset for vision tasks AlexNet, ResNet Drove CNN research
One-stage = fast, two-stage =
Detection What + Where YOLO, Faster R-CNN
accurate
Sample-by-sample audio WaveNet, Parallel
WaveNet State-of-art natural speech
generation WaveNet
Natural Language Processing: Word2Vec, Joint Detection
1. Natural Language Processing (NLP)
NLP is a field of AI focused on enabling machines to understand, generate, and reason about human
language.
Main Goals
• Text understanding
• Text generation
• Information extraction
• Language translation
• Question answering
NLP Pipeline (Simplified Diagram)
Input Text
Tokenization
Text Representation (Word2Vec, embeddings)
Modeling (RNN, Transformer, etc.)
Output (translation, summary, sentiment, etc.)
2. Word2Vec
Word2Vec is a method for learning dense vector embeddings of words so that words with similar
meanings are close in vector space.
Developed by Mikolov et al. (Google, 2013).
Why Word2Vec?
Before Word2Vec, text was represented using:
• One-hot vectors → sparse, huge, no semantic meaning
• Bag-of-Words → loses order; still sparse
• TF-IDF → weights important words but no meaning/relationships
Word2Vec:
• Dense vectors
• Captures semantic relationships
• Computationally efficient
Word2Vec Embedding Space (Conceptual Diagram)
king - man + woman ≈ queen
Vector space:
king ●
\ queen ●
man ● \
woman ●
Two Word2Vec Architectures
1. CBOW (Continuous Bag of Words)
Predicts the target word from surrounding context words.
Context: ["the", ___, "sat"]
Predict → "cat"
CBOW Diagram
context words
w(t-2), w(t-1), w(t+1), w(t+2)
Average/Combine
Neural Network
Predict w(t)
• Fast
• Good for frequent words
2. Skip-Gram
Predicts context words from a single center word.
Target: "cat"
Predict → ["the", "sat", "on", "mat"]
Skip-Gram Diagram
target word w(t)
Neural Network
predicted context words:
w(t-2), w(t-1), w(t+1), w(t+2)
• Better for rare words
• More commonly used
Training Techniques
Negative Sampling
Instead of predicting the entire vocabulary, the model:
• Chooses real context words = positive examples
• Randomly samples unrelated words = negative examples
Greatly reduces computation.
Hierarchical Softmax
Uses a binary tree to reduce softmax cost from O(V) → O(log V).
3. Joint Detection (in NLP)
“Joint detection” refers to models that detect multiple linguistic elements simultaneously, instead
of separately.
Most commonly refers to:
✔ Joint Named Entity Recognition (NER) + Relation Extraction (RE)
✔ Joint Intent Detection + Slot Filling (in conversational AI)
✔ Joint Event Detection + Argument Extraction
The idea:
Joint models perform both tasks together, sharing features, improving accuracy.
Why Joint Detection?
Traditional pipeline:
NER → relation extraction
Problem: Errors propagate from the first stage into the next.
Joint detection:
NER + RE (single model)
• Reduces error propagation
• Allows shared context
• Uses Transformers or structured prediction
Example: Joint NER + Relation Extraction
Goal: Identify entities and how they relate.
Input sentence:
“Alice founded OpenAI in 2015.”
Outputs:
• Entities:
o Alice → PERSON
o OpenAI → ORGANIZATION
o 2015 → DATE
• Relation:
o (Alice, founded, OpenAI)
Joint Detection Model Diagram
Sentence
Embedding Layer (Word2Vec, BERT, etc.)
↓
Shared Encoder (Transformer / BiLSTM)
↓ ↓
NER Head Relation Head
(entity tags) (entity pairs + relations)
The model:
• Learns entities
• Learns relations
• Uses shared features
Common Joint Detection Approaches
1. Multi-Task Learning
Shares encoder, multiple task-specific heads.
2. Span-based Models
Predict entity spans + relations jointly.
3. Graph Neural Network (GNN) Approaches
Represent tokens/entities as nodes → detect relations on graph.
4. Transformer-based Joint Models
Like BERT, RoBERTa, DeBERTa:
• Sequence labeling for NER
• Classification for relations
Summary Table
Topic Purpose Key Techniques Notes
Convert words → dense CBOW, Skip-Gram, Negative Creates semantic
Word2Vec
vectors Sampling space
Tokenization, embeddings,
NLP Process human language Wide application area
Transformers
Joint Predict multiple linguistic MTL, span-based, transformer Reduces error
Detection tasks together models propagation
Applications: Bioinformatics, Face Recognition
1. Bioinformatics
Bioinformatics applies machine learning, statistics, and computational modeling to understand
biological data.
Common data types:
• DNA/RNA sequences
• Protein sequences & structures
• Gene expression data
• Medical images (MRI, microscopy)
• Biological networks
1.1 Key ML Tasks in Bioinformatics
A. Sequence Analysis (DNA, RNA, Protein)
Tasks
• Gene prediction
• Promoter detection
• Motif discovery
• Protein family classification
ML/Deep Learning Methods
• CNNs → detect motifs
• RNNs/LSTMs → sequence modeling
• Transformers → long-range dependencies
Sequence Modeling Diagram:
DNA Sequence: A C G T T A C G
Neural Network Encoder
Predicted Function / Class
B. Protein Structure Prediction
Key breakthrough: AlphaFold2
Goals
• Predict 3D protein structure from amino acid sequence
• Understand folding, binding, and interactions
Techniques
• Transformers
• Attention networks
• Graph neural networks (GNNs)
C. Genomics & Personalized Medicine
ML Helps With:
• Predicting disease risk (e.g., cancer susceptibility)
• Identifying biomarkers from gene expression
• Drug response prediction
Pipeline Diagram:
Patient Genetic Data
ML Model (e.g., Random Forest, Transformer)
Risk Score / Treatment Recommendation
D. Drug Discovery
ML accelerates:
• Molecular property prediction
• Protein-ligand binding affinity
• Virtual screening
• De novo molecular design (using generative models)
Models:
• Graph Neural Networks (GNNs)
• Variational Autoencoders (VAEs)
• Generative Adversarial Networks (GANs)
E. Medical Imaging in Bioinformatics
Used for:
• Tumor classification
• Cell segmentation
• MRI/CT scan analysis
Techniques:
• CNNs
• U-Net models
• Vision Transformers (ViT)
Summary of ML in Bioinformatics
Task Method Example
DNA/Protein classification CNN, RNN, Transformers Motif detection
Protein structure Attention, GNN AlphaFold
Drug discovery GNN, VAEs Molecule generation
Medical imaging U-Net Tumor segmentation
Personalized medicine Gradient Boosting, DNN Risk prediction
2. Face Recognition
Face recognition = identifying or verifying a person from an image.
(Note: I will describe the technology without identifying real individuals.)
2.1 Steps in Face Recognition
A. Face Detection
Locate the face in an image.
Models:
• Haar cascades
• HOG + SVM
• Modern: MTCNN, RetinaFace
Diagram:
Input Image
Face Detector
Bounding Box around Face
B. Face Alignment
Normalize face orientation (rotate, scale, align landmarks).
Landmarks:
• Eyes
• Nose
• Mouth
Diagram:
Detected Face
Detect keypoints → Align to template
C. Feature Extraction
Deep neural networks generate embedding vectors that represent face features.
Key models:
• DeepFace
• FaceNet
• VGGFace
• ArcFace
Embedding Space Diagram (conceptual)
Embedding Dimension = 128 or 512
Faces of same person → cluster together
Faces of different people → far apart
●●● (Person A)
●●● (Person B)
●●● (Person C)
D. Face Matching / Verification
Compare embeddings using:
• Cosine similarity
• Euclidean distance
If distance < threshold → same person
Else → different people
2.2 Applications of Face Recognition
✔ Security & Surveillance
• Access control (buildings, devices)
• Monitoring restricted zones
✔ Authentication
• Phone unlock
• Airport identity verification
• Banking apps
✔ Smart Devices
• Personalized home assistants
• Smart retail (VIP identification, emotion recognition)
✔ Healthcare
• Patient identification
• Monitoring elderly / infants
✔ Forensics
• Identifying missing persons
• Crime investigation (with legal and ethical constraints)
2.3 Concerns and Ethics
• Privacy
• Bias (racial, gender, age)
• Surveillance misuse
• Consent requirements
Responsible ML is essential.
Summary Table
Application Technique Example
Bioinformatics Sequence models, GNNs Gene prediction, drug discovery
Face Recognition CNNs, embeddings, similarity Authentication, security
Scene Understanding & Gathering Image Captions (Review and Discussion)
1. Scene Understanding
Scene Understanding is the process by which a model interprets the high-level meaning of an image.
It goes far beyond simply identifying objects.
1.1 What Does Scene Understanding Include?
A. Object Detection
• Finding objects + their locations (bounding boxes)
B. Semantic Segmentation
• Labeling each pixel with a class
(e.g., road, sky, car)
car car road road
road road sky sky
C. Instance Segmentation
• Distinguish individual objects
(e.g., person #1 vs person #2)
D. Scene Classification
• Assign one overall label:
“kitchen”, “street”, “beach”, “forest”, etc.
E. Spatial Relationships
• Understanding how objects interact:
“person sitting on chair”, “cup on table”
F. 3D Understanding
• Depth estimation
• Layout prediction
• 3D object reconstruction
1.2 Why Scene Understanding Matters
Scene understanding supports higher-level AI tasks:
• Autonomous driving
• Robotics environment understanding
• Visual question answering (VQA)
• Augmented reality
• Smart surveillance
• Human–AI interaction
It moves from low-level perception → high-level semantic reasoning.
**2. Image Captioning
(Generating Descriptions from Images)**
Image captioning is a task where a model outputs a sentence describing the image.
It is essentially:
Vision Model + Language Model = Caption
2.1 Traditional Architecture (CNN + RNN)
Earlier captioning models used:
• A CNN encoder (ResNet, Inception)
• An RNN decoder (LSTM, GRU)
Diagram
Image → CNN → Feature Vector → LSTM Decoder → "A dog running in a park."
2.2 Attention-based Captioning
Attention helps the model “look” at relevant parts of the image while generating words.
Diagram (Conceptual)
Image → Features → Attention Map → Word Output
“dog” ← focus on dog
“running” ← focus on legs
“grass” ← focus on background
2.3 Transformer & Vision-Language Models
Modern models replace CNN+RNN with Transformers:
• ViT (Vision Transformer)
• BLIP, BLIP-2
• CLIP
• GPT-Vision variants
• PaLI, Flamingo
These models generate captions using joint vision–language attention.
3. Relationship Between Scene Understanding & Captioning
Image captioning depends on scene understanding.
Example:
Image:
Person riding a bike in a park.
Scene Understanding:
• Objects: person, bike, grass, trees
• Actions: riding
• Context: outdoors, daytime
Caption:
“A person is riding a bicycle through a sunny park.”
The caption is a natural-language summary of the scene understanding process.
3.1 Captioning Needs More Than Detection
Object Detection:
• “person”, “bicycle”
Captioning:
• “A person is riding a bicycle on a sunny afternoon.”
Captioning adds:
• Context
• Relationships
• Actions
• Grammar
• Style
It is vision + natural language reasoning.
4. Challenges & Research Questions
A. Understanding High-level Concepts
• Actions (“playing”, “running”)
• Intentions (“preparing to hit the ball”)
• Emotions (“a happy crowd”)
B. Commonsense Reasoning
A kitchen scene implies cooking; a classroom implies learning.
C. Ambiguity
A model may misinterpret objects or relationships.
D. Biases
Captioning models may inherit:
• gender bias
• cultural bias
• object/context bias
E. Evaluation Metrics Are Imperfect
Common metrics:
• BLEU
• METEOR
• CIDEr
But they compare generated text to human references, not true meaning.
5. Applications
Scene Understanding
• Autonomous vehicles
• Robotics navigation
• Video surveillance
• Augmented reality
• Scene retrieval & indexing
Image Captioning
• Accessibility tools (e.g., alt-text for blind users)
• Content search
• Social media auto-captioning
• Human–AI communication
• Visual assistants
Summary (Short Version)
Scene Understanding:
Interpreting objects, actions, relationships, layout, and semantics of an image.
Image Captioning:
Generating natural language descriptions that express the understanding of a scene.
Connection:
Captioning = scene understanding + language generation.