PyTorch
Differentiable programming
CS114 - Shayan Sadigh - [Link]
PyTorch vs. Numpy
1. PyTorch can run code on GPUs, which we saw last lab.
CS114 - Shayan Sadigh - [Link]
PyTorch vs. Numpy
1. PyTorch can run code on GPUs, which we saw last lab.
2. Another difference with numpy - you can use PyTorch to train neural
networks.
a. More in general, PyTorch is capable differentiable programming.
CS114 - Shayan Sadigh - [Link]
Two types of data
- This will be a bit abstract.
CS114 - Shayan Sadigh - [Link]
Two types of data
- This will be a bit abstract.
- We can imagine computers as handling two types of information:
CS114 - Shayan Sadigh - [Link]
Two types of data
- This will be a bit abstract.
- We can imagine computers as handling two types of information:
- Information that describes data.
CS114 - Shayan Sadigh - [Link]
Two types of data
- This will be a bit abstract.
- We can imagine computers as handling two types of information:
- Information that describes data.
- Information that describes computation.
CS114 - Shayan Sadigh - [Link]
x, y, and z describe data.
CS114 - Shayan Sadigh - [Link]
add_2 and do_stuff describe computation.
CS114 - Shayan Sadigh - [Link]
General idea
CS114 - Shayan Sadigh - [Link]
General idea
CS114 - Shayan Sadigh - [Link]
General idea
CS114 - Shayan Sadigh - [Link]
General idea
Some mathematicians:
All computer programs can be boiled down to information describing
computation being applied to information describing state. (See
Lambda Calculus)
CS114 - Shayan Sadigh - [Link]
In the world of tensors
- We could also classify tensors as one of two types.
CS114 - Shayan Sadigh - [Link]
In the world of tensors
- We could also classify tensors as one of two types.
- Tensors that represent data.
CS114 - Shayan Sadigh - [Link]
In the world of tensors
- We could also classify tensors as one of two types.
- Tensors that represent data.
- A rank 2 grayscale image tensor (100 by 100 pixels).
CS114 - Shayan Sadigh - [Link]
CS114 - Shayan Sadigh - [Link]
Width (16)
Height
(22)
CS114 - Shayan Sadigh - [Link]
Width (16)
This is a rank 2 tensor
Height with shape (22, 16)
(22)
CS114 - Shayan Sadigh - [Link]
In the world of tensors
- We could also classify tensors as one of two types.
- Tensors that represent data.
- A rank 2 grayscale image tensor (100 by 100 pixels).
- A rank 3 color image tensor (3 RGB values, 100 * 100 pixels)
CS114 - Shayan Sadigh - [Link]
(Rank 3)
[Link]
[Link]
CS114 - Shayan Sadigh - [Link]
(Rank 3)
[Link]
[Link]
CS114 - Shayan Sadigh - [Link]
[Link]
g-images/
CS114 - Shayan Sadigh - [Link]
CS114 - Shayan Sadigh - [Link]
CS114 - Shayan Sadigh - [Link]
CS114 - Shayan Sadigh - [Link]
CS114 - Shayan Sadigh - [Link]
CS114 - Shayan Sadigh - [Link]
[Link]
g-images/
CS114 - Shayan Sadigh - [Link]
CS114 - Shayan Sadigh - [Link]
CS114 - Shayan Sadigh - [Link]
5 columns
3 color
channels
6 rows
CS114 - Shayan Sadigh - [Link]
5 columns
3 color
channels
6 rows
This is a rank 3 tensor of shape (6x5x3)
CS114 - Shayan Sadigh - [Link]
In the world of tensors
- We could also classify tensors as one of two types.
- Tensors that represent data.
- A rank 2 grayscale image tensor (100 by 100 pixels).
- A rank 3 color image tensor (3 RGB values, 100 * 100 pixels)
- A rank 4 batch of color images.
CS114 - Shayan Sadigh - [Link]
[Link]
set-7ff62631766a
A single rank 3 color image as
before.
CS114 - Shayan Sadigh - [Link]
[Link]
set-7ff62631766a
CS114 - Shayan Sadigh - [Link]
[Link]
set-7ff62631766a
Image 3
Image 2
Image 1
A batch of 3 color
images.
CS114 - Shayan Sadigh - [Link]
[Link]
set-7ff62631766a
Image 3
Image 2
Image 1
This is a rank 4 tensor of shape:
(3, 3, 28, 28)
(batch_size, colors, rows, columns)
CS114 - Shayan Sadigh - [Link]
In the world of tensors
- We could also classify tensors as one of two types.
- Tensors that represent data.
- A rank 2 grayscale image tensor (100 by 100 pixels).
- A rank 3 color image tensor (3 RGB values, 100 * 100 pixels)
- A rank 4 batch of color images.
- Tensors that represent computation .
CS114 - Shayan Sadigh - [Link]
In the world of tensors
- We could also classify tensors as one of two types.
- Tensors that represent data.
- A rank 2 grayscale image tensor (100 by 100 pixels).
- A rank 3 color image tensor (3 RGB values, 100 * 100 pixels)
- A rank 4 batch of color images.
- Tensors that represent computation .
- A rank 2 100*100 tensor representing a single matrix / neural net layer.
CS114 - Shayan Sadigh - [Link]
CS114 - Shayan Sadigh - [Link]
CS114 - Shayan Sadigh - [Link]
CS114 - Shayan Sadigh - [Link]
[Link] creates a tensor of the given shape (784,
10) prefilled with random values from the standard
normal distribution.
CS114 - Shayan Sadigh - [Link]
We can imagine computation as a function that takes
784 inputs, and produces 10 outputs.
CS114 - Shayan Sadigh - [Link]
Right now, we can imagine this as completely
random function. It performs a completely random
computation to 784 inputs and essentially returns 10
random values.
CS114 - Shayan Sadigh - [Link]
The @ operator performs matrix multiplication.
CS114 - Shayan Sadigh - [Link]
CS114 - Shayan Sadigh - [Link]
The input was a flattened size 784 tensor.
The computation was a matrix of shape (784, 10)
Applying the computation on the tensor produced a size 10 output.
CS114 - Shayan Sadigh - [Link]
From a purely linear algebra perspective, we’re just multiplying a vector by a
matrix.
But in differentiable programming, we assign meaning to the tensors.
CS114 - Shayan Sadigh - [Link]
Tensor multiplication
- What if we didn’t flatten the image?
CS114 - Shayan Sadigh - [Link]
Tensor multiplication
- What if we didn’t flatten the image?
If we tried this:
Shape(28 x 28) @ Shape(784, 10)
CS114 - Shayan Sadigh - [Link]
Tensor multiplication
- What if we didn’t flatten the image?
If we tried this:
Shape(28 x 28) @ Shape(784, 10)
(Left side represents an image, right side represents a function)
CS114 - Shayan Sadigh - [Link]
Tensor multiplication
- What if we didn’t flatten the image?
If we tried this:
Shape(28 x 28) @ Shape(784, 10)
This would throw an error.
CS114 - Shayan Sadigh - [Link]
Tensor multiplication
- What if we didn’t flatten the image?
If we tried this:
Shape(28 x 28) @ Shape(784, 10)
This would throw an error.
Rule of thumb with matrices:
The length of the dimensions “touching”
each other need to match.
CS114 - Shayan Sadigh - [Link]
Tensor multiplication
- What if we didn’t flatten the image?
If we tried this:
Shape(28 x 28) @ Shape(28, 10)
This would work.
We change our function to
take 28 inputs.
CS114 - Shayan Sadigh - [Link]
Tensor multiplication
- What if we didn’t flatten the image?
If we tried this:
Shape(28 x 28) @ Shape(28, 10)
But don’t we have
784 values on this
side in total?
CS114 - Shayan Sadigh - [Link]
Tensor multiplication
- What if we didn’t flatten the image?
If we tried this:
Shape(28 x 28) @ Shape(28, 10)
But don’t we have How does this
784 values on this multiplication not throw
side in total? an error when the
function only takes 28
inputs?
CS114 - Shayan Sadigh - [Link]
Tensor multiplication
Shape(28 x 28) @ Shape(28, 10)
Actually, you can imagine the left side as 28
different length 28 vectors.
CS114 - Shayan Sadigh - [Link]
Tensor multiplication
Shape(28 x 28) @ Shape(28, 10)
Actually, you can imagine the left side as 28
different length 28 vectors.
Each of the vectors is passed through the
function on the right individually.
The output shape would be 28x10
CS114 - Shayan Sadigh - [Link]
Tensor multiplication
Create a list of 28 different
length 28 vectors.
CS114 - Shayan Sadigh - [Link]
Tensor multiplication
Initialize the output.
We’re going to fill in z with the
result.
CS114 - Shayan Sadigh - [Link]
Tensor multiplication
CS114 - Shayan Sadigh - [Link]
Tensor multiplication
CS114 - Shayan Sadigh - [Link]
Tensor multiplication
This code we just
described:
CS114 - Shayan Sadigh - [Link]
Tensor multiplication
This code we just
described:
Does the same exact
thing as this:
CS114 - Shayan Sadigh - [Link]
Flattening vs. no flattening
- If you flatten the image first so the input size is:
A (784) rank 1 tensor
A (784, 10) function produces a single output vector of size (10)
CS114 - Shayan Sadigh - [Link]
Flattening vs. no flattening
- If you don’t flatten the image so the input size is:
A (28, 28) rank 2 tensor
A (28, 10) function produces 28 output vectors of size 10, so the final output
shape is (28, 10).
Let’s say shape is (5, 4)
-> [[0, 0, 0, 0], [0, 0, 0, 0], [0, 0, 0, 0], [0, 0, 0, 0], [0, 0, 0, 0] ]
CS114 - Shayan Sadigh - [Link]
Flattening vs. no flattening
- In other words, with flattening, the function produces a single output based on
every pixel in the image at once.
CS114 - Shayan Sadigh - [Link]
Flattening vs. no flattening
- In other words, with flattening, the function produces a single output based on
every pixel in the image at once.
- Without flattening, the function produces 28 outputs, one for each row in the
image.
CS114 - Shayan Sadigh - [Link]
Function that takes size 9 inputs
Produces size 4 outputs.
Input (E.G. flattened
image) of size 9.
CS114 - Shayan Sadigh - [Link]
Function that takes size 9 inputs
Produces size 4 outputs.
Input (E.G. flattened
image) of size 9.
CS114 - Shayan Sadigh - [Link]
Function that takes size 9 inputs
Produces size 4 outputs.
Input (E.G. flattened
image) of size 9.
CS114 - Shayan Sadigh - [Link]
Function that takes size 9 inputs
Produces size 4 outputs.
Input (E.G. flattened
image) of size 9.
CS114 - Shayan Sadigh - [Link]
Function that takes size 9 inputs
Produces size 4 outputs.
Input (E.G. flattened
image) of size 9.
CS114 - Shayan Sadigh - [Link]
Function that takes size 9 inputs
Produces size 4 outputs.
Input (E.G. flattened Single size 4
image) of size 9. output.
CS114 - Shayan Sadigh - [Link]
What if we don’t flatten?
CS114 - Shayan Sadigh - [Link]
Function that takes size 3 inputs
Produces size 4 outputs.
Input (an image) of
shape (3, 3).
What if we don’t flatten?
CS114 - Shayan Sadigh - [Link]
Function that takes size 3 inputs
Produces size 4 outputs.
Input (an image) of
shape (3, 3).
CS114 - Shayan Sadigh - [Link]
Function that takes size 3 inputs
Produces size 4 outputs.
Input (an image) of
shape (3, 3).
CS114 - Shayan Sadigh - [Link]
Function that takes size 3 inputs
Produces size 4 outputs.
Input (an image) of
shape (3, 3).
CS114 - Shayan Sadigh - [Link]
Function that takes size 3 inputs
Produces size 4 outputs.
Input (an image) of
shape (3, 3).
CS114 - Shayan Sadigh - [Link]
Function that takes size 3 inputs
Produces size 4 outputs.
Input (an image) of
shape (3, 3).
CS114 - Shayan Sadigh - [Link]
Function that takes size 3 inputs
Produces size 4 outputs.
Input (an image) of
shape (3, 3).
CS114 - Shayan Sadigh - [Link]
Function that takes size 3 inputs
Produces size 4 outputs.
Input (an image) of
shape (3, 3).
CS114 - Shayan Sadigh - [Link]
Function that takes size 3 inputs
Produces size 4 outputs.
Input (an image) of
shape (3, 3).
A size 4 output produced for each
row in the input.
Final output size: (3, 4)
CS114 - Shayan Sadigh - [Link]
Patches
- If we flatten, we’re asking the program to process the entire image at once,
but humans also don’t look at the world this way.
- E.G. if you shift the image 1 pixel, the entire input to the function changes completely. Not
“shift invariant”.
CS114 - Shayan Sadigh - [Link]
Patches
- If we flatten, we’re asking the program to process the entire image at once,
but humans also don’t look at the world this way.
- E.G. if you shift the image 1 pixel, the entire input to the function changes completely. Not
“shift invariant”.
- If we don’t flatten, we’re asking the program to process the image row by
row, but humans don’t look at the world this way.
- Same shift invariance issue, but just horizontal shifts.
CS114 - Shayan Sadigh - [Link]
Convolution
- A convolution performs matrix multiplications not row-by-row, and not on the
whole image at once, but on patches of the image at a time.
CS114 - Shayan Sadigh - [Link]
Convolution
- A convolution performs matrix multiplications not row-by-row, and not on the
whole image at once, but on patches of the image at a time.
- This is closer to how humans see the world, and math shows that the bias
enforced by looking at patches at a time helps functions learn better.
CS114 - Shayan Sadigh - [Link]
Convolution
- A convolution performs matrix multiplications not row-by-row, and not on the
whole image at once, but on patches of the image at a time.
- This is closer to how humans see the world, and math shows that the bias
enforced by looking at patches at a time helps functions learn better.
- How to perform convolution / enforce this bias?
- This is done by reshaping the image first to give it a new structure.
CS114 - Shayan Sadigh - [Link]
Convolution
Output
Function
(9x1)
Input
CS114 - Shayan Sadigh - [Link]
Convolution
Output
Graphically it looks
like function is 3x3 but
implementation-wise it Function
flattens the patches (9x1)
first.
Rank 2 (2, 3) Input
[[1, 2, 3], [4, 5, 6]]
Flatten to rank 1 (6):
[1, 2, 3, 4, 5, 6]
CS114 - Shayan Sadigh - [Link]
Convolution
The white border is “padding” we
need to add to make things align.
CS114 - Shayan Sadigh - [Link]
Convolution in PyTorch
CS114 - Shayan Sadigh - [Link]
Convolution in PyTorch
1 pixel of padding to each
side of the image.
CS114 - Shayan Sadigh - [Link]
Convolution in PyTorch
28x28 image is
padded to 30x30
CS114 - Shayan Sadigh - [Link]
Convolution in PyTorch
CS114 - Shayan Sadigh - [Link]
Convolution in PyTorch
CS114 - Shayan Sadigh - [Link]
Convolution in PyTorch
CS114 - Shayan Sadigh - [Link]
Convolution in PyTorch
CS114 - Shayan Sadigh - [Link]
Convolution in PyTorch
CS114 - Shayan Sadigh - [Link]
Convolution in PyTorch
CS114 - Shayan Sadigh - [Link]
Convolution in PyTorch
CS114 - Shayan Sadigh - [Link]
CS114 - Shayan Sadigh - [Link]
Using only tensor reshaping and matrix
multiplication, we’ve implemented most of a
convolution layer.
CS114 - Shayan Sadigh - [Link]
CS114 - Shayan Sadigh - [Link]
Why not use for-loops?
- Why not have a list of tensors and iterate over them in a for-loop?
- Why construct everything as high-dimensional tensors and do everything at
once with a matrix multiply?
CS114 - Shayan Sadigh - [Link]
Why not use for-loops?
- Why not have a list of tensors and iterate over them in a for-loop?
- Why construct everything as high-dimensional tensors and do everything at
once with a matrix multiply?
- Every time you perform a tensor operation you make a GPU call.
- Called executing a GPU kernel.
CS114 - Shayan Sadigh - [Link]
Why not use for-loops?
- If you can formulate everything as a big tensor operation, you only need to
execute a single GPU kernel for that one operation.
CS114 - Shayan Sadigh - [Link]
Why not use for-loops?
- If you can formulate everything as a big tensor operation, you only need to
execute a single GPU kernel for that one operation.
CS114 - Shayan Sadigh - [Link]
Why not use for-loops?
- If you can formulate everything as a big tensor operation, you only need to
execute a single GPU kernel for that one operation.
CS114 - Shayan Sadigh - [Link]
Why not use for-loops?
- If you can formulate everything as a big tensor operation, you only need to
execute a single GPU kernel for that one operation.
Both snippets of code
to the same thing.
But the second invokes
the GPU only once.
CS114 - Shayan Sadigh - [Link]
GPUs
- The more data you can give to the GPU at once, the more benefit you get
from parallelization.
CS114 - Shayan Sadigh - [Link]
GPUs
- The more data you can give to the GPU at once, the more benefit you get
from parallelization.
- Ideally if you replace a size 10 for-loop with a 10x larger GPU call, you should
get a 10x speedup.
- In practice the speedup differs due to:
- Cost of transferring data to GPU
- GPUs don’t have infinite threads
- GPUs don’t have infinite memory.
CS114 - Shayan Sadigh - [Link]
Training a function
- Earlier we talked about how some tensors represent functions, others
represent state/data (such as from a dataset).
CS114 - Shayan Sadigh - [Link]
Training a function
- Earlier we talked about how some tensors represent functions, others
represent state/data (such as from a dataset).
- How do we find the right numbers to put in the tensors that represent
functions?
CS114 - Shayan Sadigh - [Link]
Training a function
- Earlier we talked about how some tensors represent functions, others
represent state/data (such as from a dataset).
- How do we find the right numbers to put in the tensors that represent
functions?
- So far we’ve been creating a random function tensor with [Link]().
CS114 - Shayan Sadigh - [Link]
Training a function
- If we have a label / target output we want the function to produce, we can use
gradient descent to bring the function closer to producing the target.
CS114 - Shayan Sadigh - [Link]
-1 means take all remaining dimensions
and combine them into a single
dimension.
So size (28, 28) size is converted to (784)
tensor.
CS114 - Shayan Sadigh - [Link]
Torch can convert your scalar
value (E.G. 6) to a one-hot
encoded version.
E.G. to [0, 0, 0, 0, 0, 0, 1, 0, 0, 0]
CS114 - Shayan Sadigh - [Link]
Set requires_grad = True on
tensors that represent functions
/ computation.
CS114 - Shayan Sadigh - [Link]
Need to repeatedly
perform gradient descent
steps until converges to
your target.
CS114 - Shayan Sadigh - [Link]
CS114 - Shayan Sadigh - [Link]
Computes euclidean
distance between output
and target.
CS114 - Shayan Sadigh - [Link]
These three lines
take the learning
step.
CS114 - Shayan Sadigh - [Link]
CS114 - Shayan Sadigh - [Link]
One-hot embedding
- Let’s say our targets are digits 0-9 (like with MNIST images).
- It’s common practice to encode the value as a one-embedding.
Number 1 -> [0, 1, 0, 0, 0, 0, 0, 0, 0, 0]
Number 3 -> [0, 0, 0, 1, 0, 0, 0, 0, 0, 0]
Number 9 -> [0, 0, 0, 0, 0, 0, 0, 0, 0, 1]
CS114 - Shayan Sadigh - [Link]
Why one-hot-embedding?
Number 1 -> [0, 1, 0, 0, 0, 0, 0, 0, 0, 0]
- You can imagine this as:
- 0% chance of number 0,
- 100% chance of number 1,
- 0% chance of number 2, etc
- If you outputted the value directly, we wouldn’t have a way of telling how
“confident” the function is of it’s output.
- The details are for another day.
CS114 - Shayan Sadigh - [Link]
Lab (Tensors and gradient descent practice)
- Use the resources linked in the comments to complete the lab.
- Micrograd slides could he helpful.
- There are questions about indexing and slicing tensors. See numpy
resources: # [Link]
- These slides will also help.
- The slides may be especially helpful for the last couple questions.
CS114 - Shayan Sadigh - [Link]
Lab due date
- This lab is due before class next Thursday (11/16 at 5
PM).
- NOT SATURDAY.
CS114 - Shayan Sadigh - [Link]
GPU
You don’t need to use the CUDA device in this lab since the tensor operations are
small enough to be computed fast on CPU.
No need to call .to(device) anywhere, is on CPU by default.
CS114 - Shayan Sadigh - [Link]