Scribd
Upload
41 views1 page

Swin Transformer: Key Concepts Explained

The document discusses key concepts of the Swin Transformer model, including hierarchical feature maps built by progressively merging and downsampling spatial resolution, and the use of shifted window attention. The Swin Transformer block contains two sub-units, each with normalization, attention, normalization, and MLP layers, with the first using window MSA and the second using shifted window MSA. Swin Transformer introduces techniques like patch merging and shifted windows to address issues in the original Vision Transformer model.

Uploaded by

Rahul Jaiswal
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as TXT, PDF, TXT or read online on Scribd
41 views1 page

Swin Transformer: Key Concepts Explained

The document discusses key concepts of the Swin Transformer model, including hierarchical feature maps built by progressively merging and downsampling spatial resolution, and the use of shifted window attention. The Swin Transformer block contains two sub-units, each with normalization, attention, normalization, and MLP layers, with the first using window MSA and the second using shifted window MSA. Swin Transformer introduces techniques like patch merging and shifted windows to address issues in the original Vision Transformer model.

Uploaded by

Rahul Jaiswal
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as TXT, PDF, TXT or read online on Scribd

Overfitting

Underfitting
weighted average of two model
Swin Transfer

The Swin Transformer introduced two key concepts to address the issues faced by the
original ViT — hierarchical feature maps and shifted window attention. In fact, the
name of Swin Transformer comes from “Shifted window Transformer”.

The ‘Patch Merging’ block and the ‘Swin Transformer Block’ are the two key building
blocks in Swin Transformer.

The first significant deviation from ViT is that Swin Transformer builds
‘hierarchical feature maps’.

the spatial resolution of these hierarchical feature maps is identical to those in


ResNet. This was done intentionally, so that Swin Transformers can conveniently
replace the ResNet backbone networks in existing methods for vision tasks.

Hierarchical feature maps allow the Swin Transformer to be applied in areas where
fine-grained prediction is required, such as in semantic segmentation.

Hierarchical feature maps are built by progressively merging and downsampling the
spatial resolution of the feature maps. In convolutional neural networks such as
ResNet, downsampling of feature maps is done using the convolution operation.

The convolution-free downsampling technique used in Swin Transformer is known as


patch merging.

The 'patch’ refers to the smallest unit in a feature map. In other words, in a
14x14 feature map, there are 14x14=196 patches.

The Swin Transformer block consists of two sub-units. Each sub-unit consists of a
normalization layer, followed by an attention module, followed by another
normalization layer and a MLP layer. The first sub-unit uses a Window MSA (W-MSA)
module while the second sub-unit uses a Shifted Window MSA (SW-MSA) module.

Common questions

Powered by AI

The Swin Transformer consists of two primary building blocks: the 'Patch Merging' block and the 'Swin Transformer Block'. The Patch Merging block is responsible for hierarchical feature map creation, which is crucial for tasks requiring fine-grained predictions, like semantic segmentation. This block handles downsampling of feature maps without convolution, a method known as patch merging, preserving spatial resolution across layers similar to ResNet. The Swin Transformer Block, containing sub-units that feature normalization layers and multi-layer perceptron (MLP) layers, enhances attention mechanisms using Window Multi-Head Self-Attention (W-MSA) and Shifted Window Multi-Head Self-Attention (SW-MSA) modules. This structure allows the Swin Transformer to replace convolutional networks such as ResNet efficiently, improving its applicability in various vision tasks by maintaining computational feasibility and accuracy .

You might also like