Comprehensive Guide to Neural Networks with Python
Neural Networks (NNs) are the foundational technology behind modern Artificial Intelligence.
In this guide, we move beyond simple logic gates to Computer Vision, using the famous MNIST
dataset to recognize handwritten digits (0-9).
1. Core Fundamentals
What is a Neuron?
A neuron is a mathematical function that takes inputs, multiplies them by weights, adds a bias,
and passes the result through an activation function.
Weights (w): Determine the importance of each input.
Bias (b): An extra parameter that allows the output to be shifted, helping the model fit the
data better.
Activation Function (sigma): A non-linear function (like Sigmoid or ReLU) that decides
if a neuron should "fire."
Sigmoid: avoid
TanH: avoid
ReLu: mostly used in hidden layers
Output layer: classification problem: softmax AF
Regression problem: linear AF
The Formula:
n
output =∑ mx+ b
i=0
Architecture of a Network
Deep learning model: the input layer input dimension must match the dimension of the
dataset: 28x28
Hidden layers:
Funnel like architecture: 2 hidden layers:
1st hidden layer: 512 neurons 2nd hidden layer: 256 neurons
Output layer: 10
the input layer input dimension must match the dimension of the dataset: 28x28
Hidden layers:
Funnel like architecture: 3 hidden layers:
1st hidden layer: 128 neurons 2nd hidden layer: 64 neurons. 32 neurons
Output layer: 10
1. Input Layer: The entry point for your data. For MNIST, this is a flattened array of 784
values (28x28 pixel images).
2. Hidden Layers: One or more layers where the network learns complex representations
(e.g., detecting edges or loops).
3. Output Layer: The final layer that produces the prediction (e.g., a probability for each
digit from 0 to 9).
2. How the Network Learns
Learning happens through two main phases:
Phase 1: Forward Propagation
Data flows from the input layer to the output layer. The network makes a prediction based on its
current weights and biases.
Phase 2: Backpropagation & Optimization
1. Loss Function: We calculate how far off the prediction was from the actual target (e.g.,
Sparse Categorical Crossentropy for digit classification).
2. Optimizer: A tool (like Adam or SGD) that calculates the "gradient" (slope) of the error
and adjusts weights to minimize that error.
3. Python Implementation (using TensorFlow/Keras)
While building from scratch is great for learning math, TensorFlow is used in professional
environments because it handles complex calculus and hardware acceleration automatically.
import tensorflow as tf
import numpy as np
# 1. Load and Preprocess the Data
# The MNIST dataset contains 70,000 grayscale images of digits 0-9
mnist = [Link]
(x_train, y_train), (x_test, y_test) = mnist.load_data()
# Normalize pixel values (0-255) to a range between 0 and 1
x_train, x_test = x_train / 255.0, x_test / 255.0
# 2. Define the Model Architecture
model = [Link]([
# Flatten 28x28 images into 1D vectors of 784 pixels
[Link](input_shape=(28, 28)),
# Hidden layer with 128 neurons and ReLU activation
[Link](units=128, activation='relu'),
# Dropout layer to prevent overfitting
[Link](0.2),
# Output layer: 10 neurons (one for each digit) with Softmax
[Link](units=10, activation='softmax')
])
[Link]()
# 3. Compile the Model
[Link](
optimizer='adam',
loss='sparse_categorical_crossentropy',
metrics=['accuracy']
)
# 4. Train the Model
print("Starting training...")
[Link](x_train, y_train, epochs=5)
# 5. Evaluate the Model
print("\nEvaluating on test data:")
[Link](x_test, y_test, verbose=2)
4. Summary of Key Concepts
Term Description
Epoch One full pass of the entire training dataset through the network.
Learning
A small constant that determines how large the steps are during weight updates.
Rate
Dense Layer A layer where every neuron is connected to every neuron in the previous layer.
(Rectified Linear Unit) The standard activation for hidden layers: $f(x) = \max(0,
ReLU
x)$.
Softmax Turns the output layer into probabilities that sum to 1.0 (100%).
A technique where randomly selected neurons are ignored during training to
Dropout
reduce overfitting.