Perceptron Limitations : Minsky and
Papert highlighted the limitations of
the perceptron in their book
Almost lead to the abandonment of
Perceptrons, particularly its inability to
connectionist AI Winter
solve non-linear problems like XOR.
McCulloch Pitts First generation Multilayer Perceptrons
Neuron perceptron Ivakhnenko et. al
McCulloch (neuroscientist) the perceptron may eventually be
The backpropagation algorithm,
and Pitts able to learn, make decisions, and
rediscovered by Rumelhart, Hinton, and
(logician) proposed a highly translate languages” -Frank
Williams in 1986, became a key method
simplified model Rosenblatt
for training multi-layer neural networks,
of the neuron (1943)
overcoming earlier limitations.
Breakthroughs in Deep Learning
(2010–2015)
2012: CNN
Convolutional Neural Networks (CNNs) were first introduced in the 1980s.
LeCun's LeNet-5 (1998) is one of the earliest and most famous CNN architectures.
AlexNet and ImageNet
• AlexNet marked a revolutionary moment in deep learning when it won the
ImageNet Large Scale Visual Recognition Challenge (ILSVRC) in 2012. This
achievement highlighted the power of deep convolutional neural networks (CNNs)
for image classification tasks.
• Layers: AlexNet consisted of 8 layers: 5 convolutional layers and 3 fully connected
layers.
• ReLU Activation: ReLU (Rectified Linear Unit) was used instead of traditional
activation functions (like sigmoid or tanh), solving the vanishing gradient problem
and speeding up training.
• Dropout Regularization: Introduced dropout to prevent overfitting during training.
CNN Architectures
Network Error Layers
AlexNet[ 16.0% 8
ZFNet 11.2% 8
VGGNet 7.3% 19
GoogLeNet 6.7% 22
Object Detection and Recognition
(Post-2012)
• R-CNN (Regions with CNN Features, 2014):
• Proposed by Ross Girshick, uses Selective Search and SVM Computationally expensive .
• Fast R-CNN and Faster R-CNN (2015):
• Improvements over R-CNN with better speed and accuracy.
• Faster R-CNN introduced the Region Proposal Network (RPN), which replaced traditional methods for
generating region proposals, making object detection more efficient.
• YOLO (You Only Look Once, 2016):
• Developed by Joseph Redmon, YOLO framed object detection as a single regression problem.
• Achieved real-time detection speeds with good accuracy by predicting bounding boxes and class
probabilities in one pass.
Applications:
Autonomous Vehicles: Object detection is critical for identifying pedestrians, vehicles, and traffic signs.
Healthcare: Detecting tumors or abnormalities in medical imaging.
Surveillance: Recognizing individuals or activities in security systems.
Augmented Reality (AR): Enabling applications to interact with real-world objects.
Breakthroughs in Deep Learning (2010–
2015)- Recurrent Neural Networks (RNNs)
• RNNs became a critical innovation for handling sequential data, such as time series,
speech, and text, where the order of data points matters.
• RNNs were prone to the vanishing gradient problem, which made it difficult to
learn long-term dependencies.
• Long Short-Term Memory (LSTM) networks, introduced by Hochreiter and
Schmidhuber, solved the vanishing gradient problem by introducing memory cells
and gates (input, forget, and output gates) to control the flow of information.
• LSTMs demonstrated the importance of memory and context in neural networks,
paving the way for more advanced models like GRUs (Gated Recurrent Units) and
Transformers.
Generative Models:
• Generative Models emerged as a powerful class of deep learning techniques and
Introduced by Ian Goodfellow in 2014, focusing on generating new data samples
that resemble the input data distribution.
• Autoencoders and Variational Autoencoders (VAEs) emerged for unsupervised
learning tasks.
• Autoencoders
Autoencoders compress (encode) and reconstruct (decode) input data to learn efficient
representations.
1. Image Denoising: Removes noise from images by learning clean patterns.
2. Anomaly Detection: Identifies outliers by measuring reconstruction errors.
Variational Autoencoders (VAEs)
Learn a probabilistic distribution of the latent space for generative tasks. Specifically
designed for generating new, realistic samples.
1. Synthetic Data Generation: Creates realistic data for training.
2. Medical Imaging: Generates new medical scans for research and diagnostics.
Generative Adversarial Networks (GANs)
GANs, introduced by Ian Goodfellow in 2014, revolutionized generative modeling by
creating realistic data through adversarial training.
Architecture:
Generator: Synthesizes fake data.
Discriminator: Differentiates between real and fake data.
Both networks improve iteratively, resulting in high-quality data synthesis.
Applications:
• Image Synthesis: Creating realistic images (e.g., DeepFake technology).
• Style Transfer: Adapting one image's style to another.
• 3D Modeling: Generating realistic 3D assets for gaming and design.
GANs transformed generative modeling, enabling applications in creative industries,
simulation, and more.
Year Milestone Key Contributions Impact
1943 McCulloch-Pitts Model First computational model of artificial Laid the theoretical foundation for neural
neurons. networks.
1950s Perceptron Introduced by Frank Rosenblatt; capable of First practical implementation of artificial
learning weights and biases. neural networks.
1980s Backpropagation Algorithm Rediscovered by Rumelhart, Hinton, and Solved key training issues for deeper
Williams; enabled multi-layer networks. networks.
1990s LSTM (Long Short-Term Introduced by Hochreiter and Schmidhuber; Enabled sequential data processing for
Memory) solved vanishing gradient problem in RNNs. tasks like speech and text.
2012 AlexNet Deep CNN won the ImageNet competition; Revolutionized computer vision and
used ReLU, dropout, and GPUs for training. popularized deep learning.
2014 GANs (Generative Adversarial Introduced by Ian Goodfellow; adversarial Enabled applications in image synthesis,
Networks) training for generating realistic samples. style transfer, and more.
2015 ResNet Introduced skip connections to solve Enabled training of ultra-deep networks
vanishing gradient problem in very deep effectively.
networks.
2016 YOLO (You Only Look Once) Real-time object detection in a single pass. Made object detection fast and efficient
for real-world applications.
2017 Transformers Introduced in 'Attention Is All You Need'; Basis for models like BERT, GPT, and Vision
revolutionized NLP. Transformers (ViT).
2020 GPT-3 A massive Transformer-based language Achieved state-of-the-art performance in
model. natural language understanding and
generation.
2020s Multimodal Models (DALL-E, Combined vision and language tasks. Enabled applications like text-to-image
CLIP) generation and enhanced AI
understanding.
Applications
• Computer vision (face recognition, self-driving cars, surveillance)
• Speech recognition (voice assistants)
• Natural language processing (chatbots, translation)
• Healthcare (medical imaging, diagnosis)
• Recommendation systems