A Comprehensive PyTorch Tutorial
1. Introduction to PyTorch
PyTorch is an open-source deep learning library developed by Meta AI (formerly Facebook AI
Research). It provides:
Dynamic computation graphs, allowing models to be changed on the fly.
GPU acceleration for fast computations.
Integration with Python tools like NumPy, scikit-learn, and matplotlib.
PyTorch is widely used in research and production environments. It is flexible and intuitive,
making it ideal for both beginners and experienced developers.
2. Installation and Setup
Install PyTorch with pip. Choose the command based on your system and whether you want
GPU support:
pip install torch torchvision torchaudio
Verify installation:
import torch
print(torch.__version__)
print("CUDA Available:", [Link].is_available())
Note: If CUDA is available, PyTorch can run computations on your GPU, significantly speeding
up training.
3. Working with Tensors
Tensors are multi-dimensional arrays, the core data structure in PyTorch. They are similar to
NumPy arrays but can leverage GPU acceleration.
import torch
x = [Link]([[1, 2], [3, 4]])
y = [Link](2, 2)
print("Tensor x:", x)
print("Random Tensor y:", y)
print("Sum:", x + y)
Tip: Always use [Link]() to create tensors with random values for initializing model
parameters.
4. Tensor Operations and Broadcasting
PyTorch supports element-wise operations and broadcasting, which automatically expands
dimensions of tensors during arithmetic.
a = [Link](4).reshape(2, 2)
b = [Link]([1, 2])
print(a + b)
# In-place operations (modify tensor directly)
a.add_(5)
print(a)
Warning: In-place operations change the original tensor. Use them carefully during model
training.
5. GPU Acceleration with CUDA
GPUs allow massively parallel computations. You can move tensors and models to GPU:
device = [Link]("cuda" if [Link].is_available() else "cpu")
x = [Link](1000, 1000).to(device)
y = [Link](1000, 1000).to(device)
z = [Link](x, y)
print("Matrix multiplication on:", [Link])
Note: Moving tensors between CPU and GPU frequently can slow down training.
6. Autograd and Computational Graphs
Autograd automatically computes gradients for backpropagation.
x = [Link](2, 2, requires_grad=True)
y = x + 2
z = y * y * 3
out = [Link]()
[Link]()
print([Link])
Gradients are stored in [Link]. Every operation is tracked to form a computation graph.
7. Building Neural Networks with [Link]
PyTorch’s nn module simplifies network construction.
import [Link] as nn
class Net([Link]):
def __init__(self):
super(Net, self).__init__()
self.fc1 = [Link](28*28, 128)
self.fc2 = [Link](128, 10)
def forward(self, x):
x = [Link](self.fc1([Link](-1, 28*28)))
return torch.log_softmax(self.fc2(x), dim=1)
Tip: Always define the forward pass; PyTorch automatically constructs the computation graph.
8. Training a Neural Network (MNIST Example)
Train a feed-forward network on MNIST.
from torchvision import datasets, transforms
from torch import optim
transform = [Link]([
[Link](),
[Link]((0.5,), (0.5,))
])
trainset = [Link]('./data', download=True, train=True,
transform=transform)
trainloader = [Link](trainset, batch_size=64,
shuffle=True)
model = Net().to(device)
criterion = [Link]()
optimizer = [Link]([Link](), lr=0.001)
for epoch in range(3):
running_loss = 0
for images, labels in trainloader:
images, labels = [Link](device), [Link](device)
optimizer.zero_grad()
log_ps = model(images)
loss = criterion(log_ps, labels)
[Link]()
[Link]()
running_loss += [Link]()
print(f'Epoch {epoch+1} - Loss: {running_loss/len(trainloader):.4f}')
Tip: Always set [Link]() before training and [Link]() before evaluation.
9. Using Datasets and DataLoaders
DataLoader handles batching, shuffling, and parallel data loading.
testset = [Link]('./data', download=True, train=False,
transform=transform)
testloader = [Link](testset, batch_size=64,
shuffle=False)
10. Model Evaluation
Evaluate your model on test data:
[Link]()
correct, total = 0, 0
with torch.no_grad():
for images, labels in testloader:
images, labels = [Link](device), [Link](device)
outputs = model(images)
_, predicted = [Link](outputs, 1)
total += [Link](0)
correct += (predicted == labels).sum().item()
print("Test Accuracy:", 100 * correct / total)
Save and load models:
[Link](model.state_dict(), "mnist_model.pth")
loaded_model = Net()
loaded_model.load_state_dict([Link]("mnist_model.pth"))
11. TensorBoard Visualization
Visualize training progress:
from [Link] import SummaryWriter
writer = SummaryWriter("runs/mnist_example")
sample_images, _ = next(iter(trainloader))
writer.add_images("MNIST Images", sample_images)
[Link]()
Tip: Monitor loss curves and sample predictions to debug training issues.
12. Advanced Topics and Best Practices
1. Transfer Learning: Fine-tune pretrained models from [Link].
2. Mixed Precision Training: Use [Link] for faster and memory-efficient
training.
3. Model Deployment: Convert models to TorchScript with [Link].
4. Hyperparameter Tuning: Tools like Optuna or Ray Tune.
Best Practices:
Normalize data.
Use GPU whenever possible.
Save checkpoints during long training runs.
Track experiments using TensorBoard or Weights & Biases.
13. Summary and Further Learning Resources
PyTorch offers flexibility, speed, and usability. With this tutorial, you now understand:
Tensors and operations
Autograd and computational graphs
Neural network building and training
GPU acceleration
Model evaluation and saving
Resources:
Official Tutorials: [Link]
Deep Learning with PyTorch (Manning)
PyTorch Lightning for structured training