# [ Deep Learning Using PyTorch ] [ cheatsheet ]

Tensor Creation and Manipulation

● Create a tensor from a list: tensor = [Link]([1, 2, 3])

● Create a tensor of zeros: tensor = [Link](shape)
● Create a tensor of ones: tensor = [Link](shape)
● Create a tensor with random values: tensor = [Link](shape)
● Create a tensor with normally distributed random values: tensor =
[Link](shape)
● Create a tensor with a range of values: tensor = [Link](start, end,
step)
● Create a tensor with evenly spaced values: tensor = [Link](start,
end, steps)
● Reshape a tensor: tensor = [Link](new_shape)
● Transpose a tensor: tensor = [Link](dim1, dim2)
● Flatten a tensor: tensor = [Link]()
● Concatenate tensors along a dimension: tensor = [Link]([tensor1,
tensor2], dim)
● Stack tensors along a new dimension: tensor = [Link]([tensor1,
tensor2], dim)
● Squeeze a tensor (remove dimensions of size 1): tensor =
[Link]()
● Unsqueeze a tensor (add a dimension of size 1): tensor =
[Link](dim)
● Permute the dimensions of a tensor: tensor = [Link](dims)

Tensor Operations

● Addition: result = tensor1 + tensor2

● Subtraction: result = tensor1 - tensor2
● Multiplication (element-wise): result = tensor1 * tensor2
● Division (element-wise): result = tensor1 / tensor2
● Matrix multiplication: result = [Link](tensor2)
● Exponential: result = [Link](tensor)
● Logarithm: result = [Link](tensor)
● Square root: result = [Link](tensor)
● Sine: result = [Link](tensor)
● Cosine: result = [Link](tensor)

By: Waleed Mousa

 ● Tangent: result = [Link](tensor)
● Sigmoid: result = [Link](tensor)
● ReLU: result = [Link](tensor)
● Tanh: result = [Link](tensor)
● Softmax: result = [Link](tensor, dim)

Neural Network Layers

● Linear layer: layer = [Link](in_features, out_features)

● Convolutional layer: layer = nn.Conv2d(in_channels, out_channels,
kernel_size, stride, padding)
● Transposed convolutional layer: layer = nn.ConvTranspose2d(in_channels,
out_channels, kernel_size, stride, padding)
● Max pooling layer: layer = nn.MaxPool2d(kernel_size, stride, padding)
● Average pooling layer: layer = nn.AvgPool2d(kernel_size, stride, padding)
● Batch normalization layer: layer = nn.BatchNorm2d(num_features)
● Dropout layer: layer = [Link](p)
● Recurrent layer (RNN): layer = [Link](input_size, hidden_size,
num_layers)
● Long Short-Term Memory layer (LSTM): layer = [Link](input_size,
hidden_size, num_layers)
● Gated Recurrent Unit layer (GRU): layer = [Link](input_size, hidden_size,
num_layers)
● Embedding layer: layer = [Link](num_embeddings, embedding_dim)

Loss Functions

● Mean Squared Error (MSE) loss: loss_fn = [Link]()

● Cross-Entropy loss: loss_fn = [Link]()
● Binary Cross-Entropy loss: loss_fn = [Link]()
● Negative Log-Likelihood loss: loss_fn = [Link]()
● Kullback-Leibler Divergence loss: loss_fn = [Link]()
● Margin Ranking loss: loss_fn = [Link]()
● Triplet Margin loss: loss_fn = [Link]()
● Cosine Embedding loss: loss_fn = [Link]()
● Hinge Embedding loss: loss_fn = [Link]()

Optimization Algorithms

By: Waleed Mousa

 ● Stochastic Gradient Descent (SGD): optimizer =
[Link]([Link](), lr)
● Adam: optimizer = [Link]([Link](), lr)
● RMSprop: optimizer = [Link]([Link](), lr)
● Adagrad: optimizer = [Link]([Link](), lr)
● Adadelta: optimizer = [Link]([Link](), lr)
● Adamax: optimizer = [Link]([Link](), lr)
● Sparse Adam: optimizer = [Link]([Link](), lr)
● LBFGS: optimizer = [Link]([Link](), lr)

Learning Rate Schedulers

● Step LR: scheduler = [Link].lr_scheduler.StepLR(optimizer,

step_size, gamma)
● Multi-Step LR: scheduler =
[Link].lr_scheduler.MultiStepLR(optimizer, milestones, gamma)
● Exponential LR: scheduler =
[Link].lr_scheduler.ExponentialLR(optimizer, gamma)
● Cosine Annealing LR: scheduler =
[Link].lr_scheduler.CosineAnnealingLR(optimizer, T_max)
● Reduce LR on Plateau: scheduler =
[Link].lr_scheduler.ReduceLROnPlateau(optimizer, mode, factor,
patience)
● Cyclic LR: scheduler = [Link].lr_scheduler.CyclicLR(optimizer,
base_lr, max_lr, step_size_up)

Model Training and Evaluation

● Move model to device: model = [Link](device)

● Set model to training mode: [Link]()
● Set model to evaluation mode: [Link]()
● Forward pass: outputs = model(inputs)
● Compute loss: loss = loss_fn(outputs, targets)
● Backward pass: [Link]()
● Update model parameters: [Link]()
● Zero gradients: optimizer.zero_grad()
● Get model parameters: parameters = [Link]()
● Get model state dictionary: state_dict = model.state_dict()
● Load model state dictionary: model.load_state_dict(state_dict)
● Save model checkpoint: [Link](model.state_dict(), '[Link]')

By: Waleed Mousa

 ● Load model checkpoint:
model.load_state_dict([Link]('[Link]'))

Data Loading and Processing

● Create a dataset: dataset = [Link](inputs,

targets)
● Create a data loader: data_loader = [Link](dataset,
batch_size, shuffle)
● Iterate over data loader: for batch in data_loader: inputs, targets =
batch
● Normalize data: data = (data - [Link]()) / [Link]()
● Resize images: images = [Link](images, size)
● Random crop images: images =
[Link](size)(images)
● Random horizontal flip images: images =
[Link]()(images)
● Convert images to tensors: images =
[Link]()(images)
● Normalize images: images = [Link](mean,
std)(images)

Pretrained Models

● Load a pretrained model: model =

[Link].resnet18(pretrained=True)
● Freeze model weights: for param in [Link]():
param.requires_grad = False
● Replace the last layer of a pretrained model: [Link] = [Link](512,
num_classes)
● Extract features from a pretrained model: features = model(inputs)

Model Evaluation Metrics

● Accuracy: accuracy = (predicted == targets).float().mean()

● Precision: precision = [Link](predicted * targets) /
[Link](predicted)
● Recall: recall = [Link](predicted * targets) / [Link](targets)
● F1 score: f1_score = 2 * (precision * recall) / (precision + recall)
● Mean Absolute Error (MAE): mae = [Link](predicted - targets).mean()

By: Waleed Mousa

 ● Mean Squared Error (MSE): mse = [Link](predicted - targets).mean()
● Root Mean Squared Error (RMSE): rmse = [Link](mse)
● Intersection over Union (IoU): iou = [Link](predicted * targets) /
[Link]((predicted + targets) > 0)
● Area Under the ROC Curve (AUC): auc =
[Link](predicted, targets)
● Average Precision (AP): ap =
[Link].average_precision(predicted, targets)
● Confusion Matrix: cm =
[Link].confusion_matrix(predicted, targets)

Model Visualization

● Visualize the model architecture: print(model)

● Visualize the model graph: torchviz.make_dot(outputs,
params=dict(model.named_parameters())).render('model_graph',
format='png')
● Visualize the model summary: [Link](model, input_size)

Transfer Learning

● Freeze the weights of the feature extractor: for param in

[Link](): param.requires_grad = False
● Fine-tune the last layer: for param in [Link]():
param.requires_grad = True
● Load a pretrained model and replace the last layer: model =
[Link].resnet18(pretrained=True); [Link] = [Link](512,
num_classes)

Adversarial Attacks

● Fast Gradient Sign Method (FGSM) attack: perturbed_inputs = inputs +

epsilon * [Link]([Link])
● Projected Gradient Descent (PGD) attack: for _ in range(num_steps):
perturbed_inputs = [Link](perturbed_inputs + alpha *
[Link](perturbed_inputs.grad), min=inputs-epsilon,
max=inputs+epsilon)
● Carlini & Wagner (C&W) attack: adversarial_inputs = [Link](inputs +
perturbations, min=0, max=1)

By: Waleed Mousa

Model Pruning

● Prune model weights: pruned_model =

[Link].random_unstructured(model, name='weight',
amount=pruning_ratio)
● Prune model biases: pruned_model =
[Link].l1_unstructured(model, name='bias',
amount=pruning_ratio)
● Prune model layers: pruned_model =
[Link].ln_structured(model, name='conv',
amount=pruning_ratio, n=2, dim=0)

Model Quantization

● Quantize model weights: quantized_model =

[Link].quantize_dynamic(model, {[Link]},
dtype=torch.qint8)
● Quantize model activations: quantized_model =
[Link].quantize_dynamic(model, {[Link]},
dtype=torch.quint8)
● Convert model to quantized version: quantized_model =
[Link](model)

Distributed Training

● Initialize distributed training:

[Link].init_process_group(backend='nccl',
init_method='tcp://localhost:23456', rank=[Link],
world_size=args.world_size)
● Wrap model with DistributedDataParallel: model =
[Link](model,
device_ids=[args.local_rank])
● Synchronize gradients across devices: [Link]()
● Reduce gradients across devices: [Link].all_reduce(tensor,
op=[Link])

Model Interpretability

● Compute gradients w.r.t. inputs: gradients = [Link](outputs,

inputs, grad_outputs=torch.ones_like(outputs))

By: Waleed Mousa

 ● Compute saliency maps: saliency_maps = [Link](gradients).max(dim=1,
keepdim=True)[0]
● Compute guided backpropagation: guided_backprop = [Link](gradients,
min=0)
● Compute class activation maps (CAM): cam = [Link](features *
[Link](num_classes, -1), dim=1).view(batch_size, -1)
● Compute Grad-CAM: grad_cam = [Link](features *
[Link](batch_size, num_channels, -1), dim=2).view(batch_size,
num_channels, 1, 1)

Model Debugging

● Print model gradients: for name, param in model.named_parameters(): if

param.requires_grad: print(name, [Link])
● Print model activations: for name, module in model.named_modules(): if
isinstance(module, [Link]): module.register_forward_hook(lambda module,
input, output: print(name, output))
● Print model parameters: for name, param in model.named_parameters():
print(name, param)
● Print model buffers: for name, buffer in model.named_buffers():
print(name, buffer)
● Set anomaly detection for debugging:
[Link].set_detect_anomaly(True)

By: Waleed Mousa