Autoencoder
Autoencoder – Semester Exam Notes
1. Definition:
An Autoencoder is a neural network trained using unsupervised learning to copy its input to
its output.
It learns a compact internal representation (embedding) that captures the most important
information about the input data.
2. Structure:
• Encoder: Maps input ( x ) to a hidden code ( h = f(x) ).
• Decoder: Reconstructs input ( r = g(h) ) from the code.
Together, they minimize the difference between input ( x ) and reconstructed output ( r ).
3. Objective:
To learn an efficient low-dimensional representation of input data while minimizing
reconstruction error.
4. Manifold Learning:
The embeddings represent a low-dimensional manifold inside the high-dimensional input
space, capturing data patterns and structure.
5. Key Characteristics:
• Autoencoders are data-specific (trained on one data type).
• They perform lossy compression (output not perfectly identical).
• Learn useful latent features rather than copying input exactly.
Types of Autoencoders (with Short Definitions)
1. Undercomplete Autoencoder:
Hidden layer smaller than input; forces network to learn compressed, meaningful
features.
2. Sparse Autoencoder:
Uses sparsity constraints so only a few neurons activate — helps capture essential
features.
3. Denoising Autoencoder:
Trained to reconstruct clean data from noisy input — improves robustness.
4. Contractive Autoencoder:
Adds penalty on encoder’s sensitivity to small input changes — makes representation
stable.
5. Variational Autoencoder (VAE):
Learns probabilistic latent variables and is used in data generation and generative
models.
Applications of Autoencoders
1. Dimensionality Reduction:
Non-linear alternative to PCA, compresses data while preserving key information.
2. Image Denoising:
Removes noise from images, reconstructing clean outputs.
3. Anomaly Detection:
Identifies unusual data based on high reconstruction error.
4. Feature Extraction:
Learns latent representations useful for classification or clustering tasks.
6. Learning Insight:
Autoencoders cannot copy data perfectly; they are restricted to learn the most important
aspects of input data.
7. Stochastic Variants:
Some autoencoders use probabilistic mappings like
( p_{encoder}(h|x) ) and ( p_{decoder}(x|h) ) for flexibility in learning.
8. Historical Context:
Autoencoders have been part of neural network research for decades and are now central to
deep generative models such as Variational Autoencoders (VAEs).
9. Summary Table:
Aspect Description
Learning Type Unsupervised
Goal Reconstruct input data
Components Encoder + Decoder
Output Approximation of input
Application Domains Image, NLP, Anomaly Detection, Compression
10. Final Note:
Autoencoders bridge the gap between representation learning and generative modeling,
forming the foundation for modern deep learning architectures like VAEs and GANs.
Would you like me to make this into a well-formatted one-page PDF for printing or quick
revision?