Video Classification - Project

TA Minh-Duc Bui

1
Year 2023
 Outline

1. Overview of Video Data

2. RWF2000 Dataset for Violence Detection Task

3. Models for Video Classification Task

4. Optional: Self-Supervised Learning for Video Data

2
 Outline

1. Overview of Video Data

2. RWF2000 Dataset for Violence Detection Task

3. Models for Video Classification Task

4. Optional: Self-Supervised Learning for Video Data

3
 Video Classification Overview

Video = 2D + Time

A video is a sequence of images

4D tensor:
Tx3xHxW

4
 Video Classification Overview

Swimming
Running
Jumping
Standing
Input video:
Tx3xHxW

5
 Video Classification Overview

Dog
images: recognize objects Cat
Fish
Bird

Swimming
videos: recognize actions Running
Jumping
Standing

6
 Video Classification Overview

Videos are big!

Videos are ~30 frames per second (fps)

Size of uncompressed video

(3 bytes per pixel):

•SD (640 x 480): ~1.5 GB per minute

•HD (1920 x 1080): ~10 GB per minute

Input video:
Tx3xHxW Solution: train on short clips: low fps and low
spatial resolution
e.g. T = 16, H=W=112
(3.2 seconds at 5 FPS, 588 KB)

7
 Video Classification Overview

Original video: long, high FPS

Training: Train model to classify short clips with low FPS

Testing: Run model on different clips, average predictions

8
EECS 498-007 / 598-005: Deep Learning for Computer Vision
 Outline

1. Overview of Video Data

2. RWF2000 Dataset for Violence Detection Task

3. Models for Video Classification Task

4. Optional: Self-Supervised Learning for Video Data

9
 RWF2000 - Violence Detection

Collected raw surveillance videos from

YouTube, sliced them into clips within 5s at 30
fps, and labeled each clip as Violent or Non-
Violent.
Finally, we have 2000 clips and 300,000 frames Challenges
as a new data set for detecting real-world violent
behavior under a surveillance camera. Since all the videos are captured by surveillance
cameras in public places, many may need better
imaging quality due to dark environments, fast
movement of objects, lighting blur, etc.

10
Cheng, Ming, Kunjing Cai, and Ming Li. "RWF-2000: an open large scale video database for violence detection." 2020 25th International Conference on Pattern Recognition (ICPR). IEEE, 2021.
 RWF2000 - Violence Detection

Only part of the person appears in the picture Crowds and chaos
Challenges

Transient Action Low resolution

11
Cheng, Ming, Kunjing Cai, and Ming Li. "RWF-2000: an open large scale video database for violence detection." 2020 25th International Conference on Pattern Recognition (ICPR). IEEE, 2021.
Create Video Dataset

12
Create Video Dataset

13
 Outline

1. Overview of Video Data

2. RWF2000 Dataset for Violence Detection Task

3. Models for Video Classification Task

4. Optional: Self-Supervised Learning for Video Data

14
 From 2D to 3D CNNs

What is this?
2D or 3D CNN?

15
[Link]
 Video Classification: Single-Frame CNN

• Simple idea: train normal 2D CNN to classify video frames

Run 2D CNN on each
independently! (Average predicted probs at test-time)
frame, and feed to MLP
• Often a solid baseline for video classification.

Average Logits

MLP MLP MLP MLP

Problem: pooling is not
aware of the temporal order!
VGG
ResNet CNN CNN CNN CNN
...

16
 Video Classification: Single-Frame CNN

Average Logits

MLP MLP MLP MLP

VGG
ResNet CNN CNN CNN CNN
...

17
 Video Classification: Single-Frame CNN

#params Best Val Accuracy

Single-Frame CNN 11.5M 72.8

18
 Video Classification: Late Fusion

Intuition: Get a high-level appearance Run 2D CNN on each frame,

of each frame, and combine them concatenate features, and feed to MLP

MLP

Average Logits Problem: Hard to compare

low-level motion between
frames
VGG
ResNet CNN CNN CNN CNN
...

19
 Video Classification: Late Fusion

MLP

Average Logits

VGG
ResNet CNN CNN CNN CNN
...

20
 Video Classification: Late Fusion

#params Best Val Accuracy

Single-Frame CNN 11.5M 72.8

Late Fusion 11.5M 73.0

21
 Video Classification: Early Fusion

MLP
Intuition: Get a high-level appearance
of each frame and combine them

VGG
2D CNN ResNet
First 2D convolution collapses all ...
temporal information:
• Input: 3T x H x W
• Output: D x H x W

Problem: Hard to compare low-level

motion between frames

Input: 3T x H x W
22
 Video Classification: Early Fusion

MLP

VGG
2D CNN ResNet
...

Input: 3T x H x W

23
 Video Classification: Early Fusion

#params Best Val Accuracy

Single-Frame CNN 11.5M 72.8

Late Fusion 11.5M 73.0

Early Fusion 11.6M 76.3

24
 Video Classification: CNN-LSTM

• Intuition: Get a high-level appearance

of each frame, and combine them Run 2D CNN on each frame,
• Recurrent Neural Networks are well concatenate features, and feed to LSTM
suited for processing sequences.

MLP

LSTM LSTM LSTM LSTM

Problem: RNNs are sequential
and cannot be parallelized.
VGG
ResNet CNN CNN CNN CNN
...

25
 Video Classification: CNN-LSTM

MLP

LSTM LSTM LSTM LSTM

VGG
CNN CNN CNN CNN
ResNet
...
 Video Classification: CNN-LSTM

#params Best Val Accuracy

Single-Frame CNN 11.5M 72.8

Late Fusion 11.5M 73.0

Early Fusion 11.6M 76.3

CNN-LSTM 15.3M 63.0

CNN-Mamba 16.4M 83.0

27
 Video Classification: 3D CNN

MLP
Use 3D Convs to explicitly extract
features from the input video

3D CNN

Computional expensive

Input: T x 3 x H x W

28
 Video Classification: 3D CNN

A step back: Depthwise Separable

Convolution (in MobileNet)

C
... Standard Kernel

H
W ←𝑁→
1 Depthwise Kernel
H ...

W ←𝐶→ Depthwise Separable

Convolution

C Pointwise Kernel
...

1
1 ←𝑁→
 Video Classification: 3D CNN

Depthwise Separable
Convolution

30
[Link]
 Video Classification: 3D CNN

S3D Model Sep-Conv 1x3x3 1x3x3

7x7x7 Conv Sep-Conv
Max-Pool Max-Pool Sep-Inc
1x1x1 3x3x3
Stride 2 Stride 1,2,2 Stride 1,2,2

3x3x3
Sep-Inc Sep-Inc Sep-Inc Sep-Inc Max-Pool Sep-Inc
Stride 2,2,2

3x3x3 3x3x3
Conv
Sep-Inc Max-Pool Sep-Inc Sep-Inc Max-Pool
1x1x1
Stride 2,2,2 Stride 2,2,2

Sep-Inc Concat prediction

Conv Conv Conv

Conv 3x1x1 Conv
3x1x1 3x1x1
1x1x1 1x1x1
Sep-Conv
3x3x3 = Conv
Conv
1x3x3
Conv
1x3x3

1x3x3 Conv Conv 3x3x3

1x1x1 1x1x1 Max-
Pool

Previous Layer
 Video Classification: 3D CNN

Sep-Conv block in S3D

Conv
8x1x1
Sep-Conv
8x3x3 = Conv
1x3x3

32
[Link]
 Video Classification: S3D

#params Best Val Accuracy

Single-Frame CNN 11.5M 72.8

Late Fusion 11.5M 73.0

Early Fusion 11.6M 76.3

CNN-LSTM 15.3M 63.0

CNN-Mamba 16.4M 83.0

3D CNN 9.1M 78.8

33
 Video Classification: Video ViT (ViViT)

A step back: Vision Transformer (ViT)

Transformer Encoder
Class MLP
Bird Head Lx +
Cat
Dog MLP
... Transformer Encoder
Norm

Patch + Position +
Embedding 0 1 2 3 4 5 6 7 8 9
*Extra learnable Multi-Head
[class] embedding
Linear Projection of Flattened Patches Attention

Norm

Embedded
Patches
34
 Video Classification: Video ViT (ViViT)

What are tokens in images?

224 16
16

Linear
224 Projection 196
tokens

35
 Video Classification: Video ViT (ViViT)

What are tokens in videos?

Uniform frame sampling Tubelet embedding

• uniformly sample 𝑛! frames from the input

• embed each 2D frame independently using the • fuse spatio-temporal information during tokenisation
same method as ViT • extract non-overlapping, spatio-temporal “tubes” from the
• concatenate all these tokens together input volume

36
Arnab, Anurag, et al. "Vivit: A video vision transformer." Proceedings of the IEEE/CVF international conference on computer vision. 2021.
Video Classification: Video ViT (ViViT)

MLP Head

0 Transformer Encoder
cls

Lx +
1
MLP
2
Norm

3
Embed to +
tokens
... Multi-Head
Attention
N
Norm

37
 Video Classification: Video ViT (ViViT)

Factorised Encoder Factorised Self-Attention Factorised Dot-Product

MLP Head
Temporal Temporal Fuse

Transformer Encoder Temporal Spatial Spatial Temporal

0

1
Lx +
... ... ...
MLP
2 Fuse
Norm Spatial Temporal

Embed to 3 + Spatial Temporal

Spatial Spatial
tokens
... Multi-Head
Attention
N
Norm

38
Arnab, Anurag, et al. "Vivit: A video vision transformer." Proceedings of the IEEE/CVF international conference on computer vision. 2021.
 Video Classification: Video ViT (ViViT)

Factorised Encoder
MLP
Temporal Transformer Encoder
Head

Token Embedding
Temporal +
0
c
l
s
1 2 ... T

Spatial Spatial Spatial

Transformer Transformer Transformer
Encoder Encoder Encoder
Token Embedding
...
Positional +

c c c
0 l 1 ... N 0 l 1 ... N 0 l 1 ... N
s s s

Embed to tokens

39
 Video Classification: Video ViT (ViViT)

#params Best Val Accuracy

Single-Frame CNN 11.5M 72.8

Late Fusion 11.5M 73.0

Early Fusion 11.6M 76.3

CNN-LSTM 15.3M 63.0

CNN-Mamba 16.4M 83.0

3D CNN 9.1M 78.8

Video ViT (ViViT) 86.5M 84.8

40
41
 Quiz Time!!!
1. Đâu là một cách xử lý video có kích thước
lớn?

A. Sử dụng toàn bộ frame trong video.

B. Chia video thành nhiều clip ngắn và train 3. Trong early fusion, các frame được kết hợp như
model. thế nào trước khi đưa vào model?
C. Bỏ qua các frame không quan trọng.
D. Sử dụng video với tốc độ khung hình cao. A. Tất cả các frames được nén lại thành một
frame.
B. Các frames được giữ nguyên và xử lý độc lập.
2. Đâu là đặc điểm của data trong video C. Các frames được đưa vào LSTM như là các
classification? input độc lập.
A. Video là chuỗi của các frame theo không D. Các frame được kết hợp để tạo thành tensor
gian. có kích thước (3*T x H x W).
B. Mỗi frame thường có kích thước nhỏ.
C. Video là chuỗi của các frame theo thời
gian.
D. Video không chứa dữ liệu âm thanh.

42
 Quiz Time!!!

5. Trong VideoDataset, hàm uniform_sample dùng

để làm gì?

4. Single-frame model hoạt động dựa trên A. Để sắp xếp các frames theo thứ tự số.
nguyên tắc nào? B. Để chuyển đổi các frame sang RGB.
C. Để lấy mẫu đều các frame từ danh sách.
A. Tổng hợp các feature vector từ mỗi frame. D. Để kết nối với GPU cho việc training nhanh hơn.
B. Dùng 2D model để predict trên mỗi frame
và tổng hợp kết quả.
C. Xử lý từng frame với một mạng LSTM.
D. Kết hợp tất cả frames thành một tensor 3D.

43
 Outline

1. Overview of Video Data

2. RWF2000 Dataset for Violence Detection Task

3. Models for Video Classification Task

4. Optional: Self-Supervised Learning for Video Data

44
 Optional
Self-Supervised Learning: VideoMAE

A step back: Mask Autoencoder (MAE)

45
He, K., Chen, X., Xie, S., Li, Y., Dollár, P., & Girshick, R. (2021). Masked Autoencoders Are Scalable Vision Learners. ArXiv. /abs/2111.06377
 Optional
Self-Supervised Learning: VideoMAE

A step back: Mask Autoencoder (MAE)

46
He, K., Chen, X., Xie, S., Li, Y., Dollár, P., & Girshick, R. (2021). Masked Autoencoders Are Scalable Vision Learners. ArXiv. /abs/2111.06377
 Optional
Self-Supervised Learning: VideoMAE

VideoMAE

47
Tong, Z., Song, Y., Wang, J., & Wang, L. (2022). VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training. ArXiv. /abs/2203.12602
 Optional
Self-Supervised Learning: VideoMAE

VideoMAE

Reduce information leakage

48
Tong, Z., Song, Y., Wang, J., & Wang, L. (2022). VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training. ArXiv. /abs/2203.12602
 Optional
Self-Supervised Learning: VideoMAE

Comparison on Something-Something dataset

49
Tong, Z., Song, Y., Wang, J., & Wang, L. (2022). VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training. ArXiv. /abs/2203.12602
 Video Classification: Video ViT (ViViT)

#params Best Val Accuracy

Single-Frame CNN 11.5M 72.8
Late Fusion 11.5M 73.0
Early Fusion 11.6M 76.3
CNN-LSTM 15.3M 63.0
CNN-Mamba 16.4M 83.0
3D CNN 9.1M 78.8
Video ViT (ViViT) 86.5M 84.8
VideoMAE 86.2M 91.3

50
 Summary

Video = 2D + Time Videos are big! Tokens in Video Data

Solution: train on short clips:

low fps and low spatial
resolution

Tx3xHxW

51
Thanks!
Any questions?

52