Pixelnet- Real Example: Semantic Segmentation of Street
Scenes
Let’s consider a real-world example: segmenting each pixel of a street image into classes like
road, car, pedestrian, building, etc
How PixelNet works step by step, using semantic segmentation of a street scene (e.g.,
identifying roads, cars, people, buildings):
1. Input Image: A 512×512 street scene photo is fed into PixelNet.
2. Pixel Sampling: Instead of all 262,144 pixels, the model randomly samples a subset
(e.g., 2,000) to train efficiently.
3. Feature Extraction: A CNN (like VGG-16) extracts hierarchical feature maps (edges in
shallow layers, objects in deeper layers).
4. Hypercolumn Construction: For each sampled pixel, PixelNet gathers corresponding
activations from multiple CNN layers and concatenates them into a single vector.
5. Dense Representation: This hypercolumn combines texture (e.g., road edges) and
semantic context (e.g., presence of a car).
6. MLP Classifier: The hypercolumn is fed into a small MLP, which predicts the pixel’s
class (road, car, pedestrian, building, sky, etc.).
7. Parallel Processing: Thousands of hypercolumns are processed simultaneously for fast
training.
Parallel Prediction:
All 2,000 hypercolumns are sent together into the MLP classifier.
Instead of classifying pixel 1, then pixel 2, and so on, the MLP
predicts the labels for all 2,000 pixels in a single forward pass.
Example Output:
Pixel at (100, 200) → predicted as road
Pixel at (210, 180) → predicted as car
Pixel at (330, 400) → predicted as pedestrian
Pixel at (400, 100) → predicted as building
… and this happens for all 2,000 pixels simultaneously.
8. Loss Calculation: Pixel-wise cross-entropy compares predictions with manually labeled
ground truth segmentation maps.
9. Backpropagation: The model adjusts both CNN and MLP weights to improve accuracy
over time.
10. Dense Prediction: Once trained, PixelNet predicts a class for every pixel in a new street
scene, producing a complete segmentation map.
Final output: roads are labeled gray, cars red, pedestrians green, and sky blue — all at
pixel-level precision.
Detailed Explanation
Input
An RGB street scene image (1024×512 px).
Step 1: Feature Extraction
Use a backbone like VGG-16 to get intermediate feature maps from conv1, conv3, conv5, etc.
Step 2: Hypercolumn for Each Pixel
For a pixel (x, y):
Extract the corresponding feature vector from each chosen layer (by bilinear interpolation
if needed),
Concatenate them into one long vector — this is the hypercolumn for that pixel.
Step 3: Pixel Sampling
Instead of using all ~500k pixels:
Randomly sample, say, 2000 pixels per image per iteration.
These sampled pixels are representative and help the network generalize better.
Step 4: MLP Classification
Feed each hypercolumn into an MLP:
Hidden layers → non-linear transformations
Output layer → softmax over the number of classes (e.g., 19 classes for Cityscapes)
Step 5: Training
Train end-to-end using cross-entropy loss for sampled pixels. The CNN backbone is fine-tuned
jointly with the MLP.
Step 6: Prediction
At test time, the model computes hypercolumns for all pixels (not just samples) and predicts
class labels for each pixel to produce a dense segmentation map.
Suppose your CNN has 3 layers chosen for hypercolumns:
conv1: size 512×512, stride 1 → shallow
conv3: size 128×128, stride 4 → mid
conv5: size 32×32, stride 16 → deep
conv1 → 512×512, stride 1 → Shallow layer
Input: 512×512 image
Stride 1: The convolution slides one pixel at a time → output retains the same spatial
size.
Feature Type:
o Captures low-level / shallow features like edges, textures, corners, and color
patterns.
o High spatial resolution → maintains fine details (important for pixel-level tasks).
Why it matters for PixelNet:
o These features help classify fine boundaries and small objects.
conv3 → 128×128, stride 4 → Mid-level layer
As we go deeper, pooling and strides gradually downsample the feature maps.
After a few convolution + pooling layers, the resolution shrinks to 128×128 (i.e., 4×
smaller than input), because:
o Stride 4 = total downsampling factor → input size / 4 = 512 / 4 = 128.
Feature Type:
o Captures mid-level features such as object parts, shapes, and basic structures.
o Less spatial detail, but more semantic context.
Why it matters for PixelNet:
o Adds contextual understanding while still preserving some location information.
conv5 → 32×32, stride 16 → Deep layer
After more pooling, the resolution drops to 32×32 (i.e., 16× smaller than input).
o 512 / 16 = 32 → explains the stride of 16.
Feature Type:
o Captures deep, high-level semantic features — entire objects, categories, or scene
structure.
o Coarse spatial resolution but strong semantic meaning.
Why it matters for PixelNet:
o These deep features provide global context for each pixel, helping disambiguate
local appearances.
How PixelNet Uses These Layers
PixelNet builds hypercolumns for each sampled pixel by:
1. Taking the feature vector from conv1, conv3, and conv5 (shallow, mid, deep).
2. Interpolating them back to the original pixel’s (x, y) position.
3. Concatenating them into one long vector → rich in both fine detail (conv1) and semantic
context (conv5).
-----
For pixel (x=100, y=200) in the original image:
conv1 → just take feature at (100, 200) directly.
conv3 → corresponding coordinate = (100/4=25, 200/4=50).
→ bilinear interpolate between neighboring cells.
conv5 → corresponding coordinate = (100/16=6.25, 200/16=12.5).
→ bilinear interpolate in 32×32 grid.
Then:
Feature vector from conv1: length 64
Feature vector from conv3: length 128
Feature vector from conv5: length 256
Concatenate → Hypercolumn for pixel (100,200): length = 64+128+256 = 448
This 448-dimensional vector is the pixel’s descriptor, containing:
Edges & textures (from conv1),
Mid-level patterns (from conv3),
Semantic context (from conv5).
Step 3: Pixel Sampling Strategy- More details
Main idea:
Instead of training on every pixel in the image (which is computationally expensive and
redundant), PixelNet randomly samples a subset of pixels from each image in every training
iteration.
This is crucial because:
Dense pixel prediction tasks have hundreds of thousands of pixels per image (e.g.,
512×512 = 262,144 pixels),
But many neighboring pixels are highly correlated — they carry similar information,
Training on all pixels slows down learning and wastes GPU memory.
Why Sampling Matters
Problem if you use all pixels PixelNet's Sampling Fix
Huge computational cost per batch Only process a few thousand pixels per image
Strong spatial correlation — adjacent pixels Random sampling picks diverse pixels from
look similar across the image
Independent samples → faster, more stable
Redundant gradients → slower convergence
training
Memory bottleneck for hypercolumns Sampling keeps memory usage manageable
How Sampling Works
Suppose you’re training on 4 images per batch, each 512×512: i.e the number of pixels in each
dimension
Note:
Term Meaning
Pixel Intensity Range The value stored at each pixel (e.g., brightness level or RGB value).
(0–255) Used in low-level image representation.
Image Resolution (e.g., The number of pixels in the image along width and height. Used to
512×512) describe the size of the image.
Total pixels per batch: 4 × 512 × 512 = 1,048,576 pixels
PixelNet samples: say, N = 2000 pixels per image
→ Total pixels per batch = 4 × 2000 = 8,000 pixels
For each sampled pixel:
1. Compute its hypercolumn vector (Step 2),
2. Feed it into the MLP classifier,
3. Compute the per-pixel loss (e.g., cross-entropy),
4. Backpropagate to update both the CNN backbone and MLP.
Sampling Strategies
PixelNet primarily uses uniform random sampling, but other strategies can be adapted depending
on the task:
Strategy Description Use case
Pick random pixels uniformly from the whole
Uniform random General dense prediction
image
Stratified sampling Sample more pixels from rare classes or edges Class imbalance problems
Sample near boundaries to improve boundary Segmentation with thin
Edge-focused
accuracy edges
Strategy Description Use case
Hard example
Sample pixels with high loss Advanced training stage
mining
The original PixelNet paper shows that even uniform random sampling gives excellent results
and is simpler.
Example (Cityscapes Semantic Segmentation)
Imagine a 1024×512 street image:
Road pixels dominate → ~60% of pixels
Building ~30%
Pedestrians, cars, signs → tiny % of pixels
If you train on all pixels, the gradient is dominated by road and building, leading to class
imbalance.
If you sample 2000 pixels randomly, you’ll:
Still get many road/building pixels (because they’re common),
But also get some from rarer classes, improving representation,
Reduce computation dramatically.
Use of PixelNet vs DenseNet
Aspect PixelNet DenseNet
Designed for pixel-level prediction Designed as a feature extraction architecture
Primary
tasks (e.g., semantic segmentation, for image classification, segmentation, and
Use
depth estimation, edge detection). other high-level vision tasks.
Acts as a dense prediction head that
Role in Acts as a backbone network to extract
uses hypercolumns + MLP to classify
Pipeline hierarchical features from the entire image.
or regress each pixel.
Efficient pixel-wise learning through Strong feature reuse and gradient flow →
Strength sampling and multi-scale context → excels at representation learning and global
excels at dense labeling. image understanding.
Image classification, object detection
Street scene segmentation, medical
Typical backbones, or feature extractor for
image labeling, surface normal
Use Case segmentation models (e.g., DenseNet-
prediction.
UNet).
Why PixelNet is Useful
Memory Efficient — avoids processing all pixels during training.
Better Generalization — pixel sampling reduces correlation among training examples.
Flexible — works for multiple dense prediction tasks: segmentation, edge detection,
surface normals, etc.
Can Use Pretrained CNNs — integrates well with VGG, ResNet, etc.