1

Understanding the In-Camera

Image Processing Pipeline for
Computer Vision

IEEE CVPR 2016

June 26, 2016

Michael S. Brown
National University of Singapore
(York University, Canada)
 Tutorial schedule
• Part 1 (General Part)
– Motivation
8.30am-10.25am
– Review of color/color spaces
– Overview of camera imaging pipeline
Coffee Break
• Part 2 (Specific Part) 10.25am – 11am

– Modeling the in-camera color pipeline

– Photo-refinishing
• Part 3 (Wrap Up)
11.00am-12.30pm
– The good, the bad, and the ugly of
commodity cameras and computer vision
research
– Concluding remarks 2
 3

Motivation for
this tutorial?
Shifting landscape of cameras

4
 DSLR/Mirrorless

Point & Shoot

Film
Market Analysis by
Mayflower Concepts

5
Mobile Phone
Cameras

Market Analysis by
Mayflower Concepts

6
OR ≈

1 50+

7
Image capture is mainstream

8
Not always a good thing . . .

9
Imaging for more than photography

Many applications are not

necessarily interested in the actual
photo.

10
Scientist’s view of photography

11
Photo by Uwe Hermann
Scientist’s view of photography

12
 Camera = light-measuring device
Illumination source
(radiance)

Output (digital) image

Imaging System

Internal Image Plane

Scene Element

Simple models assume an image is a “quantitative measurement” of scene radiance.

13
Figure from Digital Image Processing, Gonzales/Woods
Image = radiant-energy measurement
Illumination source
(radiance)

Output (digital) image

Imaging System

Internal Image Plane

Scene Element

Simple models assume an image is a “quantitative measurement” of scene radiance.

14
Figure from Digital Image Processing, Gonzales/Woods
 Assumption used in many places
Shape-from-shading

• Shape from shading

• HDR Imaging
• Image Matching
• Color constancy
• Etc . . .

From Lu et al, CVPR’10

Image matching HDR imaging

15
From Jon Mooser, CGIT Lab, USC From O’Reilly’s digital media forum
Camera = light-measuring device?

16
 Light-measuring device?
Samsung Galaxy S6 edge HTC One M9

LG G4

Google Camera App

17
All settings the same
 Onboard processing (photo finishing)
“Secret recipe” of a camera

Photographs taken from three different cameras with the same aperture, shutter speed,
white-balance , ISO, and picture style.
18
 Modern photography pipeline
Scene Radiance

Pre-Camera
Lens Filter In-Camera
Lens CCD response (RAW)
Shutter CCD Demosaicing (RAW)
Aperture
“Photo-finishing Processing”

Starting point:
reality (in radiance)

Final output

Camera Output: sRGB

Post-Processing Even if we stopped here,

Ending point: Touch-up the original CCD response
better than reality (in RGB) Hist equalization potentially has had many
Spatial warping levels of processing.
Etc . . .
19
 Digital cameras
• Digital cameras are far from being light-
measuring devices
• They are designed to produce visually pleasing
photographs
• There is a great deal of processing (photo-
finishing) happening on the camera

The goal of this tutorial is to discuss

common processing steps that take place
onboard consumer cameras
20
 This tutorial will examine
Scene Radiance

Pre-Camera
Lens Filter In-Camera
Lens CCD response (RAW)
Shutter CCD Demosaicing (RAW)
Aperture
“Photo-finishing Processing”

Starting point:
reality (in radiance)

What is (potentially) Camera Output: sRGB

happening on a camera.*

*Camera pipelines are almost always proprietary, so knowing exactly what steps are performed for a
particular make/model is not possible. This tutorial examines the most common steps likely to be found
21
on most cameras.
 Tutorial schedule
• Part 1 (General Part)
– Motivation
– Review of color/color spaces
– Overview of camera imaging pipeline
• Part 2* (Specific Part)
– Modeling the in-camera color pipeline
– Photo-refinishing
• Part 3 (Wrap Up)
– The good, the bad, and the ugly of commodity cameras
and computer vision research
– Concluding Remarks
22
* Mainly involves shameless self-promotion of my group’s work.
 23

“Crash Course” on
Color & Color Spaces
 Color
Def Color (noun): The property possessed by an
object of producing different sensations on the eye
as a result of the way it reflects or emits light.
Oxford Dictionary

Def Colour (noun): The word color spelt with the

letter ‘u’ and pronounced with a non-American
accent.

24
 Color is perceptual
• Color is not a primary physical property
• Red, Green, Blue, Pink, Orange, Atomic Tangerine,
Baby Pink, etc. . .
– Adjectives we assign to “color sensations”

Which is “True Blue”?

25
[Link]
 Subjective terms to describe color
Hue
Name of the color
(yellow, red, blue, green, . . . )

Value/Lightness/Brightness
How light or dark a color is. Hue

Value
Saturation/Chroma/Color Purity
How “strong” or “pure” a color is.
Chroma

Image from Benjamin Salley

A page from a Munsell Student Color Set 26
Where do “color sensations” come from?

A very small range of electromagnetic radiation

Generally wavelengths
from 390 to 700nm are
visible to most individuals

27
 “White light” through a prism

“white light”

Refracted light

Prism
Spectral “colors”
Relative Power

450nm 600nm 650nm

Light is separated into “monochromatic” light at different wave lengths. 28

 Sensations?
• Our eye has three receptors (cone cells) that respond
to visible light and give the sensation of color

Short, Medium,
Long Cones Retina

Optical nerve

Eye
29
 Cones and rods
• We have additional light sensitive cells called rods
that are not responsible for color
• Cones are most concentrated on the fovea

30
 Spectral power distribution (SPD)

We rarely see monochromatic light in real world scenes. Instead,

objects reflect a wide range of wavelengths. This can be described
by a spectral power distribution (SPD) shown above. The SPD plot
shows the relative amount of each wavelength reflected over the
visible spectrum. 31
 SPD relation to color is not unique
• Due to the accumulation effect of the cones, two
different SPDs can be perceived as the same color
Lettuce SPD

Lettuce SPD
stimulating
S=0.2, M=0.8,
L=0.8
SPD of “real lettuce”

Green Ink SPD

Result in the same
Green ink SPD
color “sensation”.
stimulating
S=0.2, M=0.8,
L=0.8

SPD of ink in a “picture of lettuce” 32

 Tristimulus color theory
• Even before cone cells were discovered, it was
empirically found that only three distinct
colors (primaries) could be mixed to produce
other colors
• Thomas Young (1803), Hermann von
Helmholtz (1852), Hermann Grassman (1853),
James Maxwell (1856) all explored the theory
of trichromacy for human vision

33
 Tristimulus color theory
Grassman’s Law states that a source color can be matched by a linear combination
of three independent “primaries”.
Three primaries and the
weights (R1, G1, B1) of each
= R1 + G1 + B1 primary needed to match the
source light #1 perceived
Source light 1 color.

Same three primaries and the

= R2 + G2 + B2
weights (R2, G2, B2) of each
primary needed to match the
source light #2 perceived
Source light 2 color

If we combined source
lights 1 & 2 to get The amount of each primary needed to match the new source
a new source light 3
light 3 will be the sum of the weights that matched lights 1 & 2.

= (R1+ R2) + (G1+ G2 ) + (B1+B2)

Source light 3 34
 Radiometry vs. photometry
• Radiometry
– Quantitative measurements of radiant energy
– Often shown as spectral power distributions (SPD)
– Measures either light coming from a source (radiance) or light falling on a
surface (irradiance)
• Photometry/ colorimetry
– Quantitative measurement of perceived radiant energy based on human’s
sensitivity to light
– Perceived in terms of “brightness” (photometry) and color (colorimetry)

Object

Photometry/
Radiometry
colorimetry
Tomato’s SPD

Perception of the
Wavelength (λ) tomato’s SPD
35
 Quantifying color
• We still need a way to quantify color &
brightness
• SPDs go through a “black box” (human visual
system) and are perceived as color
• The only way to quantify the “black box” is to
perform a human study

Tomato’s SPD “Black box” Ripe Red

Wavelength (λ)
36
 Experiments for photometry
Relative Power

450nm 600nm 650nm

Reference bright light
with fixed radiant power.

(Alternating between source and reference @ 17Hz)

Chromatic source light at Alternate between the source light and

a particular wavelength and reference light 17 times per second (17 hz).
adjustable radiant power. A flicker will be noticeable unless the two
lights have the same perceived “brightness”.
+ The viewer adjusts the radiant power of the
Viewer gradually
increases source chromatic light until the flicker disappears
radiant power (i.e. the lights fuse into a constant color).
The amount of radiant power needed for this
fusion to happen is recorded.

Repeat this flicker fusion test for each wave-

length in the source light. This allows method
The “flicker photometry” can be used to determine the perceived
experiment for photopic “brightness” of each wavelength. 37
sensitivity.
 CIE* (1924) Photopic luminosity function

_
The Luminosity Function (written as y(λ) or V(λ)) shows the eye’s sensitivity to radiant energy
into luminous energy (or perceived radiant energy) based on human experiments
(flicker fusion test).
*International Commission on Illumination (CIE comes from the French name Commission
internationale de l'éclairage) was a body established in 1913 as an authority on light,
illumination and color . . CIE is still active today -- [Link] 38
 Colorimetry
• Based on tristimulus color theory, colorimetry
attempts to quantify all visible colors in terms
of a standard set of primaries

= R1 + G1 + B1

Target color Three fixed primaries

39
 CIE RGB color matching
+/- Red (700nm)

+/-
Green (546nm)

Test color Matched Color

+/- Blue (435nm)
Human subjects
matched test colors 2◦ FoV
by add or subtracting
three primaries.
Same?
Field of view was 2-degrees
(where color cones are most
concentrated)

“Standard Observer”
(Willing participant with no eye disease)
Experiments carried out by 40
W. David Wright (Imperial College) and John Guild (National Physical Laboratory, London) – Late 1920s
 CIE RGB color matching
Primary is added to the test color!
+
+/- Red (700nm)

+/-
Green (546nm)

Test color Matched Color

+/- Blue (435nm)

2◦ FoV
For some test colors, no
mix of the primaries could
give a match! For these cases,
the subjects were ask to add Same?
primaries to the test color
to make the match.

This was treated as a negative “Standard Observer”

value of the primary added to (Willing participant with no eye disease)
the test color. 41
 CIE RGB results

Plots are of the mixing

coefficients of each
primary needed to
produce the corresponding
monochromatic light at
that wavelength.

Note that these functions

have been scaled such
that area of each curve
is equal.

CIE RGB 2-degree Standard Observer

(based on Wright/Guild’s data)

42
CIE RGB results

Negative points, the primary used did not

span the full range of perceptual color.
43
 CIE 1931 - XYZ
• In 1931, the CIE met and approved defining a new canonical
basis, termed XYZ that would be derived from Wright-Guild’s
CIE RGB data
• Properties desired in this conversion:
– White point defined at X=1/3, Y=1/3, Z=1/3
– Y would be the luminosity function (V(λ))
– Quite a bit of freedom in selecting these XYZ basis
– In the end, the adopted transform was:

𝑋 0.4887180 0.3106803 0.2006017 𝑅

𝑌 = 0.1762044 0.8129847 0.0108109 𝐺
𝑍 0.0000000 0.0102048 0.9897952 𝐵
CIE RGB
Nice article see: Fairman et al “How the CIE 1931 Color-Matching Functions Were Derived from Wright–
44
Guild Data”, Color Research & Application, 1997
 CIE XYZ

_ _ _
This shows the mixing coefficients x(λ), y(λ), z(λ) for the CIE 1931 2-degree standard
observer XYZ basis computed from the CIE RGB data. Coefficients are all now positive.
Note that the basis XYZ are not physical SPD like in CIE RGB, but linear combinations
defined by the matrix on the previous slide.
45
 CIE XYZ 3D plot

3D plot of the CIE XYZ matching functions

against the XYZ axis. Note that scaling of
46
this plot is not uniform.
 What does it mean?
• We now have a canonical color space to describe SPDs
• Given an SPD, I(λ), we can find its mapping into the CIE XYZ
space
780 780 780

𝑋= 𝐼(λ)𝑥(λ) ⅆλ 𝑌= 𝐼(λ)𝑦 (λ) ⅆλ 𝑍= 𝐼(λ)𝑧(λ) ⅆλ

380 380 380

• Given two SPDs, if their CIE XYZ values are equal, then they
are considered the same perceived color, i.e.
– I1 (λ), I2 (λ) → (X1, Y1, Z1) = (X2, Y2, Z2) [ perceived as the same color ]

• So . . we can quantitatively describe color!

47
 SPD to CIE XYZ example
CIE XYZ Values
SPD1
SPD1
X=0.2841
Y=0.2989
From their CIE XYZ
Z=0.3254 mappings, we can
SPD2
determine
SPD2 that these two
SPDs will be
X=0.2841 perceived as the
Two SPDs Y=0.2989 same color (even
Z=0.3254 without needing
to see the color!)

Thanks CIE XYZ!

Radiometric Colorimetric
CIE XYZ gives a way to go from radiometric to colorimetric.
Imbedded is also the photometric measurement in the Y value. 48
 What does it mean?
• CIE XYZ space is also considered “device
independent” – the XYZ values are not specific
to any device
• Devices (e.g. cameras, flatbed, scanners,
printers, displays) can find mappings of their
device specific values to corresponding CIE
XYZ values. This provides a canonical space to
match between devices (at least in theory)

49
Luminance-chromaticity space (CIE xyY)

• CIE XYZ describes a color in terms of linear

combination of three primaries (XYZ)
• Sometimes it is useful to discuss color in terms
of luminance (perceived brightness) and
chromaticity (we can think of as the hue-
saturation combined)
• CIE xyY space is used for this purpose

50
 Deriving CIE xyY

Project the CIE XYZ values onto the X=1, Y=1, Z=1 plane.

51
Figure from Ponce and Forsyth
 Projection plot
A bit hard to visualize after seeing the
CIE XYZ 3D plot, but we do obtain
the shape on the previous slide from
the projection onto X=1, Y=1, Z=1

52
 Projection plot
A bit hard to visualize after seeing the
CIE XYZ 3D plot, but we do obtain
the shape on the previous slide from
the projection onto X=1, Y=1, Z=1

53
 CIE x-y chromaticity diagram

This gives us the familiar

horseshoe shape of
visible colors as 2D plot.
Note the axis are x & y.

Point “E” represents

where X=Y=Z have equal
energy (X=0.33, Y=0.33, Z=0.33) E
CIE XYZ “white point”

In the 1930s, CIE had a bad habit of over using the variables X, Y. Note that x, y are chromaticity coordinates,
54
_ _
x, y (with the bar above) are the matching functions, and X, Y are the imaginary SPDs of CIE XYZ.
 CIE x-y chromaticity diagram
Moving along the outside
of the diagram gives
the hues.
Moving way from
the white point
represents more saturation.

55
 CIE xyY
• Generally when we use CIE xyY, we only look at the
(x,y) values on the 2D diagram of the CIE x-y
chromaticity chart
• However, the Y value (the same Y from CIE XYZ)
represents the perceived brightness of the color
• With values (x,y,Y) we can reconstruct back to XYZ

𝑌 𝑌
𝑋= 𝑥 𝑍 = (1 − 𝑥 − 𝑦)
𝑦 𝑦

56
 Fast forward 80+ years
• CIE 1931 XYZ, CIE 1931 xyY (2-degree standard
observer) color spaces have stood the test of
time
• Many other studies have followed (most
notably - CIE 1965 XYZ 10-degree standard
observer), . . .
• But in the literature (and in this tutorial) you’ll
find CIE 1931 XYZ color space making an
appearance often

57
 What is perhaps most amazing?
• 80+ years of CIE XYZ is all down to the
experiments by the “standard observers”

• How many standard observers were used?

100, 500, 1000?

A Standard Observer

58
CIE XYZ is based on 17 Standard Observers

10 by Wright, 7 by Guild

“The Standard Observers” 59

 A caution on CIE x-y chromaticity
From Mark D. Fairchild book: “Color Appearance
Models”

“The use of chromaticity diagrams should be

avoided in most circumstances, particularly when
the phenomena being investigated are highly
dependent on the three-dimensional nature of color.
For example, the display and comparison of the
color gamuts of imaging devices in chromaticity
diagrams is misleading to the point of being almost
completely erroneous.”

60
Re-quoted from the Mark Meyer’s “Deconstructing Chromaticity” article.
We are done with color, right?
Almost . . .

61
 An object’s SPD
• In a real scene, an object’s SPD is a combination of the
its reflectance properties and scene illumination

Our earlier example

ignored illumination
Illuminant 1 SPD Illuminant 2 SPD Illuminant 3 SPD
(we could assume it was pure
white light).

Tomato SPD
.* .* .*
Wavelength (λ)

Instead, think of this

of how the object
reflects different
λ λ
wavelengths λ

62
 Color constancy
• Our visual system is able to compensate for
the illumination
Illuminant 1 SPD Illuminant 2 SPD Illuminant 3 SPD

X X X

λ λ
λ

Looks the same!

63
Color constancy/chromatic adaptation
• Color constancy, or chromatic adaptation, is the
ability of the human visual system to adapt to scene
illumination
• This ability is not perfect, but it works fairly well
• Image sensors do not have this ability (it must be
performed as a processing step, i.e. “white balance”)
• Note: Our eyes do not adjust to the lighting in the
photograph -- we adjust to the viewing conditions of
the scene we are viewing the photograph in!

64
 Color constancy and illuminants
• To understand color constancy, we have to
consider SPDs of different illuminants

65
Figure from Ponce and Forsyth
 Color temperature
• Illuminants are often described by their color temperature
• This is associated with theoretical “blackbody radiators” that produce SPDs
based on a given temperature (expressed in kelvins)
• We often map light sources (both real and synthetic) to their closest color
temperature (esp in Photography/Video production)

Freeware app by, Fu-Kwun Hwang

to generate blackbody SPDs
at different temperatures, showing
the range in the visible spectrum.

66
Color temperature

Typical description of
color temperature used
in photography & lighting
sources.

67
From B&H Photo
Plotted in CIE x-y chromaticity

Plot of color CIE xy

locations of SPDs based
on color temperature.

68
Man made illuminants SPDs

69
Figure from Ponce and Forsyth
 CIE standard illuminants
• CIE established several “synthetic” SPDs that serve as proxies for
common real illuminants
• Illuminant A
– tungsten-filament lighting (i.e. a standard light-bulb)
• Illuminant B
– noon sunlight
• Illuminant C
– average daylight
• Illuminant D series
– represent natural daylight at various color temps (5000K, 5500K,
6500K), generally denoted as D50, D55, D65
• Illuminant E
– idea equal-energy illuminant with constant SPD
– does not represent any real light source, but similar to D55
• Illuminant F series
– emulates a variety of fluorescents lamps (12 in total)
70
 CIE standard illuminants

SPDs for CIE standard illuminant A, B, C SPDs for CIE standard illuminant D50, D55, D65

SPDs for CIE standard illuminant E SPDs for CIE standard illuminants F2, F8, F11

71
D, E, and F series images from [Link]
 White point
• A white point is a CIE XYZ or CIE xyY value of
an ideal “white target” or “white reference”
• This is essentially an illuminants SPD in terms
of CIE XYZ/CIE xyY
– We can assume the white reference is reflecting
the illuminant
• The idea of chromatic adaptation is to make
white points the same between scenes

72
White points in CIE x-y chromaticity

CIE Illuminants
A, B, C, D65, E in terms of CIE x-y
A
CIE x , y
A 0.44757 , 0.40745
C
B 0.34842 , 0.35161
C 0.31006 , 0.31616
B D65 0.31271 , 0.32902
E E 0.33333 , 0.33333
D65

73
 Color constancy (at its simplest)
• (Johannes) Von Kries transform
• Compensate for each channel corresponding
to the L, M, S cone response

𝐿2 1/𝐿1𝑤 0 0 𝐿1
𝑀2 = 0 1/𝑀1𝑤 0 𝑀1
𝑆2 0 0 1/𝑆1𝑤 𝑆1

L2, M2, S2 is the new LMS L1w, M1w, S1w is the LMS L1, M1, S1 are the input
response with the illuminant response to “white” under LMS space under an
divided “out”. In this case this illuminant illuminant.
74
white is equal to [1,1,1]
 Illuminant to illuminant mapping
• More appropriate would be to map to another illuminant’s LMS
response (e.g. in the desired viewing condition)
• (LMS)1 under an illuminant with white-response (L1w, M1w, S1w)
• (LMS)2 under an illuminant with white-response (L2w, M2w, S2w )

𝐿2 𝐿2𝑤 /𝐿1𝑤 0 0 𝐿1
𝑀2 = 0 𝑀2𝑤 /𝑀1𝑤 0 𝑀1
𝑆2 0 0 𝑆2𝑤 /𝑆1𝑤 𝑆1

L1w, M1w, S1w is the LMS

L2, M2, S2 is the new LMS L1, M1, S1 are the input
response to “white” the
response with the illuminant LMS space under an
input illuminant, L2w, M2w,
divided “out” and scaled to illuminant.
S2w response to “white”
LMS2 illuminant 75
of output illuminant
 Example

Simulation of different “white points” by photographing a “white” object under different illumination.

76
Images courtesy of Sharon Albert (Weizmann Institute)
 Example
Here, we have mapped the
two input images to one below
to mimic chromatic adaptation.
Input The “white” part of the cup is
shown before and after to help
show that the illumination falling
on white appears similar after
the “chromatic adaptation”.

Adapted to
“target”
illuminant

Before After Before After Target Illumination

77
Now we are finally done with color?
Almost (really) . . .

78
 CIE XYZ and RGB
• While CIE XYZ is a canonical color space,
images/devices rarely work directly with XYZ
• XYZ are not real primaries
• RGB primaries dominate the industry
• We are all familiar with the RGB color cube

But by now, you should realize

that Red, Green, Blue have no
quantitative meaning. We need to
know their corresponding SPDs or
CIE XYZ values

79
Device specific RGB values

The RGB values span a subspace

of CIE-XYZ to define the devices
gamut.
G
If you have RGB values, they are
specific to a particular device .

Color Gamut

80
 Trouble with RGB
Device 1
Device 2
Device 3

B RGB values have no meaning

if the primaries between devices
are not the same! This is a
huge problem for color reproduction
81
from one device to the next.
Standard RGB (sRGB)
In 1996, Microsoft and HP
defined a set of “standard”
RGB primaries.
R=CIE xyY (0.64, 0.33, 0.2126)
G G=CIE xyY (0.30, 0.60, 0.7153)
B=CIE xyY (0.15, 0.06, 0.0721)

This was considered an RGB

space achievable by most
R devices at the time.

White point was set to the D65

illuminant. This is an important
thing to note. It means sRGB
has built in the assumed
viewing condition (6500K daylight).
82
 CIE XYZ to sRGB conversion
Matrix conversion:
𝑅 3.2404542 −1.5371385 0.4985314 𝑋
𝐺 = −0.9692660 1.8760108 0.0415560 𝑌
𝐵 0.0556434 −0.2040259 1.0572252 𝑍
Linearized sRGB (D65) CIE XYZ

• R=G=B=1 is defined as illuminant D65 is in CIE XYZ*

• This is the linear-sRGB space
• sRGB also specifies a gamma correction of the
values
83
 sRGB gamma curve

Output sRGB 0-255 (byte)

This is a close approximation of

the actual sRGB gamma

Input linear-sRGB (range 0-1)

Actual formula is a bit complicated, but effectively this is gamma (I’ = 255*I(1/2.2)), where I’ is
the output intensity and I is the linear sRGB ranged 0-1, with a small linear transfer for
linearized sRGB values close to 0 (not shown in this plot). 84
 Gamma justification
• Gamma encoding is used in hardware to compensate for the non-linear
characteristics of cathode ray tubes (CRT).
• The application of the gamma=1/(2.2) will be undone by the CRT’s non-
linear responsible to voltage to beam intensity to produce the desired
result

Gamma to
compensate
for CRT

85
Before (linear sRGB) & after (sRGB)

Final sRGB
Linear sRGB

Linear sRGB Final sRGB

86
Standardization isn’t new - NTSC/PAL
0.9
Primaries of sRGB
0.8 Primaries of NTSC
CIE XYZ 1

0.7
0.8 Gamma
NTSC encoding
0.6
(2.2)-1
sRGB 0.6
0.5

0.4 0.4

2.2
0.3
0.2
Gamma
0.2
decoding
0
0 0.2 0.4 0.6 0.8 1
0.1

87
 CIE XYZ ↔ NTSC/sRGB
(know your color space!)
Linear-sRGB back to XYZ

Linear-NTSC back to XYZ

It is important to
known which color space
your image is in.

88
CIE XYZ: The mother of color spaces
grand
CIE XYZ
ProPhoto

Adobe RGB
L*ab sRGB NTSC

YUV HSV HSI YIQ

89
 CIE XYZ: The mother of color spaces
grand
CIE XYZ
ProPhoto

Adobe RGB
L*ab sRGB NTSC

YUV HSV HSI YIQ

But, be cautious. All Ys are not equal. The YUV and YIQ are defined from the gamma
encoded RGB values. Technically they are not CIE Y luminance, but “luma”.
They should be written with a Y’ to distinguish them. 90
Standard color spaces are great

sRGB -> CIE XYZ -> Display (ICC Profile)

Camera
sRGB Image

91
 Benefits of sRGB
• Like CIE XYZ, sRGB is a device independent
color space (often called an output color
space)
• If you have two pixels with the same sRGB
values, they will have the same CIE XYZ value,
which means “in theory” they will appear as
the same perceived value
• Does this happen in practice?

92
 See for yourself

sRGB values that are the same will be perceived as the same on the screen. However,
from “real scene SPD” to final sRGB, cameras are clearly doing different things.
93
 Congratulations!

You
26 June 2016

“Crash Course on Color”

94
 Crash course on color is over!
• A lot of information to absorb
• Understanding colorimetry is required to
understand imaging devices
• CIE XYZ and CIE illuminants will make many
appearances in color imaging/processing
discussions

95
 Tutorial schedule
• Part 1 (General Part)
– Motivation
– Review of color/color spaces
– Overview of camera imaging pipeline
• Part 2 (Specific Part)
– Modeling the in-camera color pipeline
– Photo-refinishing
• Part 3 (Wrap Up)
– The good, the bad, and the ugly of commodity cameras
and computer vision research
– Concluding Remarks
96
 Overview of the
Camera Imaging Pipeline

97
 Pipeline for sRGB (JPEG)
Gain Control
White
A/D Converter
Balance
Possible LUT
AFE – Analog Front End
Sensor with color filter array
Sensor related processing
(CCD/CMOS)

Color Space Noise

Tone CFA
Transform + Reduction/
Reproduction Demoasicing
Color Preferences Sharpening

JPEG Save to
Exif File Info storage
Compression

98
* Note that steps can be optional (e.g. noise reduction) or applied in slightly different order.
 Pipeline for RAW
Gain Control
Lossless
A/D Converter
Compression
Possible LUT
AFE – Analog Front End
Sensor with color filter array
Sensor related processing
(CCD/CMOS)

Additional steps could be “estimated” but not TIFF encoding

applied for RAW*. For example, white (meta data)
point/illuminant estimation, which is saved in the
meta-data but not applied to the RAW. Some
cameras even save the full “Exif” data with RAW so
the RAW can be rendered as if it was fully processed Save to
as an sRGB/JPEG. storage

99
*Technically speaking, RAW shouldn’t be all caps, it is not an acronym.
Imaging sensor A/D
Gain
etc
White
Balance

• We will discuss the basics of the imaging

sensor at a high level
– Could have a whole tutorial just on sensors
• The assumption is after this step, the digital
values coming off the sensor are linear with
respect to the amount of falling on the pixel
over a given amount of time
• Will discuss some common steps to make this
happen

100
 Imaging sensor A/D
Gain
etc
White
Balance

Basic imaging
sensor design
Microlens
Light to help increase
light collection
on the sensor

Color filters place over

the sensor. This forms
Color Filter Color Filter Color Filter a Color Filter Array (CFA)
Sensor Sensor Sensor also called a “Bayer Pattern”
after inventor Bryce Bayer.

Photodiode Silicon/Circuitry
(Photon comes in,
electron pops out)

101
 CCD & CMOS

CCD CMOS
CCD (charge coupled device) has a CMOS (complementary metal oxide
different readout technology to convert semiconductor) converts charge to voltage at
charge to voltage and buffer it for output. the pixel site. This allows faster readouts, but
The plus side is there is more space on less space for the photo sensor per pixel.
the pixel for the photo sensor.

102
Figures from Dave Litwiller, “CCD vs. CMOS”, Photonics Spectra, Jan 2001
 Camera RGB sensitivity
• The color filter array (CFA) on the camera
filters the light into three primaries

Plotted from camera sensitivity database by Dr. Jinwei Gu from Rochester Institute of Technology (RIT)
103
[Link]
 Black light subtraction
• Sensor values for pixels with “no light” should be
zero
• But, often, this is not the case for various reasons
– Cross talk on the sensor, etc. .
– This can also change with sensor temperature
• This can be corrected by capturing a set of pixels
that do not see light
• Place a dark-shield around sensor
• Subtract the level from the “black” pixels

104
 Optical black (OB)
Sensor area(s)
capturing
“optical black”

105
Black light capturing areas (likely exaggerated) from Sony US Patent US8227734B2 (Filed 2008) .
 Signal amplification (gain)
• Imaging sensor signal is amplified
• Amplification to assist A/D
– Need to get the voltage to the range required to
the desired digital output
• This gain could also be used to accommodate
camera ISO settings
– Unclear if all cameras do this here or use a simple
post-processing gain to the RAW for ISO settings
– DSLR cameras RAW is modified by the ISO setting

106
 Defective pixel mask
• CCD/CMOS have pixels that are defective
• Dead pixel masks are pre-calibrated at the factory
– Using “dark current” calibration
– Take an image with no light
– Record locations reporting values to make “mask”
• Bad pixels in the mask are interpolated
• This process seems to happen before RAW is saved
– If you see dead pixels in RAW, these are generally new
dead pixels that have appeared after leaving the factory

107
 Example

Identifying “dead pixels” After interpolation

108
Image courtesy of Lu Zheng (CityU Hong Kong) and Moshe Ben-Ezra (MIT)
 Nonlinear response correction
• Some image sensor (generally CMOS) often
have a non-linear response to different
amounts of irradiance
• A non-linear adjustment or look up table (LUT)
interpolation can be used to correct this

109
 Other possible distortion correction

• Sensor readout could have spatial distortion

for various reasons, e.g. sensor cross-talk
• For point-and-shoot/mobile cameras with
fixed lens, vignetting correction for lens
distortion could be applied
• Such corrections can be applied using a LUT or
polynomial function (in the case of vignetting)

110
Ex: Flat field correction

Black level
subtraction and
linearization
111
Ex: Flat field correction
(non-uniform gain)

112
 At this stage A/D
Gain
etc

• We now have a reading from the sensor that is

linear with respect to light coming in
• Defective pixels have been interpolated
• Potential distortion has been reduced

Gain Control

Irradiance
Light IN A/D Converter
Possible LUT
AFE – Analog Front End
Sensor with Color Filter Array
Sensor related processing
(CCD/CMOS)
Digital Value
113
 RAW – stop here A/D Lossless
Compression
Gain
etc
TIFF
encoding
(meta data)

• If saving in RAW, we can stop here. Save to

storage

• Convert to TIFF + metadata, save to media

• RAW generally represents gained + linearized
sensor response before CFA demosaicing and
white-balance correction
• We like to think of RAW as “unprocessed”
sensor response
Important: RAW image color space will be camera-specific
114
 White balance A/D
Gain
etc
White
Balance

CFA

• White balance is intended to mimics

demos-
acing

chromatic adaptation of the eye

• Users can manually set the white balance
– Camera specific white-balance matrices are used
selected illuminant (see next slides)
– This is often stored in the Exif metadata
• Otherwise auto white balance (AWB) is
performed

115
 WB manual settings

Typical mapping of WB icons to related color temperate.

White-balance matrix is often stored in the exif file.

116
Image from [Link]
 Examples of manual WB matrices
Nikon D7000
Sunny Incandescent Shade
2.0273 0 0  1.3047 0 0 
2.4922 0 0 
 0  0 0 
 1.0000 0   1.0000  0
 1.0000 0 
 0  0 0 2.2148
0 1.3906  0 0 1.1367

Canon 1D
Daylight Tungsten Shade
2.0938 0 0 
1.4511 0 0  2.4628 0 0 
 0 0   0
 1.0000
 1.0000 0   0
 1.0000 0 
 0 0 1.5020  0 0 2.3487  0 0 1.2275

Sony A57K
Daylight Tungsten Shade
2.6836 0 0  1.6523 0 0 
3.1953 0 0 
 0 0   0 0   0
 1.0000  1.0000
 1.0000 0 
 0 0 1.5586  0 0 2.7422  0 0 1.2891

117
 Auto white balance (AWB)
• If manual white balance is not used, then an
AWB algorithm is performed
• This is not entirely the same as chromatic
adaptation, because it doesn’t have a target
illuminant, instead AWB (as the name implies)
attempts to make what is assumed to be
white map to “pure white”
• Next slides introduce two well known
methods: “Gray World” and “White Patch”

118
 AWB: Gray world algorithm
• This methods assumes that average reflectance of a scene is
achromatic (i.e. gray)
– Gray is just the white point not at its brightest, so it serves as an
estimate of the illuminant
– This means that image average should have equal energy, i.e. R=G=B
• Based on this assumption, the algorithm adjusts the input
average to be gray as follows:

First, estimate the average response:

1 1 1
𝑅𝑎𝑣𝑔 = 𝑅𝑠𝑒𝑛𝑠𝑜𝑟(r) 𝐺𝑎𝑣𝑔 = 𝐺𝑠𝑒𝑛𝑠𝑜𝑟(g) 𝐵𝑎𝑣𝑔 = 𝐵𝑠𝑒𝑛𝑠𝑜𝑟(b)
𝑁𝑟 𝑁𝑔 𝑁𝑏
r = red pixels values, g=green pixels values, b =blue pixels values
Nr = # of red pixels, Ng = # of green pixels, Nb = # blue pixels

Note: # of pixel per channel may be different if white balance is applied to the RAW image before
demosaicing. Some pipelines may also transform into another colorspace, e.g. LMS, to perform the white-
119
balance procedure.
 AWB: Gray world algorithm
• Based on averages, white balance can be
expressed as a matrix as:

𝑅′ 𝐺𝑎𝑣𝑔 /𝑅𝑎𝑣𝑔 0 0 𝑅
𝐺′ = 0 1 0 𝐺
𝐵′ 0 0 𝐺𝑎𝑣𝑔 /𝐵𝑎𝑣𝑔 𝐵

White-balanced Sensor RGB

sensor RGB
Matrix scales each channel by its average and
then normalizes to the green channel average.

Note: some (perhaps most) pipelines may also transform into another colorspace, e.g. LMS, to perform
120
the
white-balance procedure.
 AWB: White patch algorithm
• This methods assumes that highlights represent specular
reflections of the illuminant
– This means that maximum R, G, B values are a good estimate of the
white point
• Based on this assumption, the algorithm works as follows:

𝑅𝑚𝑎𝑥 = 𝑚𝑎𝑥(𝑅𝑠𝑒𝑛𝑠𝑜𝑟(r)) 𝐺𝑚𝑎𝑥 = 𝑚𝑎𝑥( 𝐺𝑠𝑒𝑛𝑠𝑜𝑟(g)) 𝐵𝑚𝑎𝑥 = 𝑚𝑎𝑥( 𝐵𝑠𝑒𝑛𝑠𝑜𝑟(b))

r = red pixels values, g=green pixels values, b =blue pixels values

121
 AWB: White patch algorithm
• Based on RGB max, white balance can be
expressed as a matrix as:

𝑅′ 𝐺𝑚𝑎𝑥 /𝑅𝑚𝑎𝑥 0 0 𝑅
𝐺′ = 0 1 0 𝐺
𝐵′ 0 0 𝐺𝑚𝑎𝑥 /𝐵𝑚𝑎𝑥 𝐵

White-balanced Sensor RGB

sensor RGB
Matrix scales each channel by its maximum value and
then normalizes to the green channel’s maximum.

122
 AWB example

Input Gray World White Patch

123
 Better AWB methods
• Gray world and white patch are very basic algorithms
– These both tend to fail when the image is dominated by
large regions of a single color (e.g. a sky image)
• There are many improved versions
• Most improvements focus on how to perform these
white point estimation more robustly
• Note: AWB matrix values are often stored in an
images Exif file

124
 CFA demosaicing White
Balance
CFA
demos-
acing

Noise

• Color filter array/Bayer pattern placed Reduction

Sharpening

over pixel sensors

• We want an RGB value at each pixel, so we
need to perform interpolation

B B B B
R G R G G G G G
R R R R
R R R B
G B G B R R R G
R R R R
R R R B
R G R G R R R G
R R R R

Sensor RGB layout Desired output with RGB per

pixel.
125
 Simple interpolation
G2

G4 ? G6
G8
B1 G2 B3
R5 G5 = G2 + G4 + G6 + G8
G4 R5 G6
B7 G8 B9 G5 ? 4
B5 ?

Simply interpolate B1 B3 B5 = B1 + B3 + B7 + B9
based on neighbor
values.
? 4
B7 B9

126
 Simple “edge aware” interpolation
Case 1
B1 G2 B3 If (|G2–G8) && |(G4-G8)| both < Thres) G2

= G4 G6
G4 R5 G6 G5 = G2 + G4 + G6 + G8
G8
B7 G8 B9 4 All about the same.

elseif ( |G2-G8| > Thres)

Case 2
R5 G2
G2 G5 = G4 + G6
G4 ? G6 G5 ? 2
G4

G8
G6

G8 B5 ? else G2 and G8 differ

G5 = G2 + G8 G2

G4 G6
2
G8

G4 and G6 differ
127
 Demosaicing
• These examples are simple algorithms
• Cameras almost certainly use more complex
and proprietary algorithms
• Demosaicing can be combined with additional
processing
– Highlight clipping
– Sharpening
– Noise reduction

128
Noise Reduction (NR) CFA
demos-
acing
Noise
Reduction
Sharpening

• All sensors inherently have noise Color

Transform
• Some of this is mitigated in the sensor circuitry
(discussed previously)
• Some cameras apply NR after A/D conversion
• A couple simple methods are presented here
• For high-end cameras, it is likely that cameras apply
slightly different strategies depending on ISO settings,
e.g. high ISO will result in more noise, so a more
aggressive NR could be used
• Examples given are more likely on lower-end point-
and-shoot cameras

129
 NR – Rank order statistics
• Sliding window median filter
• Sort all pixels in a 3x3 (or larger) widow about
center pixel by value
• Select median (i.e. pixel #5 in rank)

10 11 10 10 11 10
9 5 10 [5,8,9,9,10,10,10,10,11] 9 10 10
10 8 9 rank #5 10 8 9
input output
Median filter is nice because is preserves edges.

130
 NR – Rank order statistics
• Sliding window despeckle filter
• Sort all pixels in a 3x3 (or larger) widow about center
pixel by value
• If center pixel maps to extreme (rank 1 or 9) and is
significantly different than closest neighbor, take
neighboring ranked pixel value

10 11 10 10 11 10
9 19 10 [8,9,9,10,10,10,10,11,19] 9 11 10
10 8 9 10 8 9
rank #9
input output
Center pixel mapped to an extreme.
|Rank 9 (19) – Rank 8 (11)| > threshold
Replace with Rank 8 131
 NR + Sharpening
• Another strategy is to apply a blur and add the detail
back in “content regions”, or even boost content to
perform sharpening
Sketch of the procedure here:
NR+Sharpened
Output
Com-
input I B(I) Output
bined
blur input
reduces noise
but blurs edges
For values with high-response
assume this is “content” and
- |I-B(I)| add it back to the blur version
(can even give a slight boost = sharpening).
Subtract I-B If low response, don’t do anything, then
(high pass filter) this region will have NR applied. 132
Input NR
133
Color space manipulation Noise
Reduction
Sharpening
Color
Transform

Tone
• We are now at one of the most crucial Reproduc-
tion

steps in the pipeline

• Up until this point, the “RGB” values have
been related to the camera’s RGB sensitivity,
i.e. sensor RGB color space
• We now need to map these values to the
output color space (sRGB)

134
 Color space transform
• Color Correction Matrix (CCM)
– Transforms sensor native RGB values into some
canonical colorspace (e.g. CIE XYZ) that will
eventual be transform to the final sRGB
colorspace
• It is important that the white-balance has
been performed correctly

135
 Color transform conceptually

These LUT used to RAW sensor RGB values Tone-scale and sRGB
correct sensor RAW come in, and sRGB values gamma are applied
come out

Top figure from: Parulski & Spaulding, Chapter 12 “Color image processing for digital camera”, 136
Digital Color Imaging Handbook, 2003
 Matrix decompositions
Focusing on the matrix part only.
Conceptually we could breakdown
the color transform in to a series
of transforms. It may desirable
to perform AWB in a different
colorspace, e.g. LMS, then to
sRGB (using CIE XYZ as an intermediate
colorspace to try things together)

Matrix 1 Matrix 2 Matrix 3 Matrix 4 Matrix 5

LMS to
White-
Sensor RGB to CIE XYZ to CIE-XYZ CIE-XYZ (D65)
CIE XYZ X LMS X balance X D65 white X to sRGB
To ideal white
point

First, convert CIE XYZ to

LMS to CIE-XYZ D65
sensor RGB to LMS
CIE-XYZ (w/ to linearized
CIE (1931) XYZ
optional white sRGB
This is from Parulski & Spaulding, however, it is more common to point) 137
perform white-balance first – more on this later.
 Tone-mapping Color
Transform
Tone
Reproduc-
tion

• Non-linear mapping of RGB tones

JPEG
Compress
-ion

• Applied to achieve some preferred tone-

reproduction
– This is not sRGB gamma
– This is to make the images look nice
• To some degree this mimics the nonlinearity in
film (known as the Hurter-Driffield Curves)
• Each camera has its own unique tone-
mapping (possibly multiple ones)

138
 Examples

(RAW or linearize sRGB)

Here are several examples of tone-mappings for a white range of cameras.

The x-axis is the input “RAW” (or linearized-sRGB), the y-axis is the output
After tone mapping.
139
From Grossberg & Nayar (CVPR’03)
 Tone-mapped examples

Input

140
 Note about tone-mapping
• It is worth noting, that up until this stage, our
color values (either RAW or linear sRGB) are
related to incoming light in a linear fashion
• After this step, that relationship is broken
• Unlike the sRGB gamma (which is known), the
tone-mapping is propriety and can only be
found by a calibration procedure

141
 JPEG compression Tone
Reproduc-
tion
JPEG
Compress
-ion

• Joint Photographic Experts Group (JPEG) Exif

metadata

• Lossy compression strategy based on the 2D

Discrete Cosine Transformation (DCT)
• The by far the most widely adopted standard
for image storage

142
 JPEG approach
f(x,y) - 128
(normalize between –128 to 127)

Quantize DCT coefficients

8x8 DCT via
(Forward DCT) Quantization “Table”
on each block
C’(u,v) = round(C(u,v)/T(u,v))

T(u,v)
Take original image and
Break it into 8x8 blocks Differential 0
coding DC component

Huffman
JPEG bitstream
Encode RLE
AC Vector

“Zig-zag” Order Coefficients

JPEG applies almost every compression trick known. 143

1) Transform coding, 2) psychovisual (loss), 3) Run-length-encoding (RLE), 4) Difference coding, and Huffman.
 JPEG quality
• The amount of quantization applied on the DCT
coefficients amounts to a “quality” factor
– More quantization = better compression (smaller file size)
– More quantization = lower quality
• Cameras generally allow a range that you can select

Image from [Link] 144

 Exif metadata JPEG
Compress
-ion
Exif
metadata

Save to
• Exchangeable image file format (Exif) flash
memory

• Created by the Japan Electronics and Information Technology

Industries Association (JEITA)
• Associates meta data with images
– Date/time
– Camera settings (basic)
• Image size, aperture, shutter speed, focal length, ISO speed, metering
mode (how exposure was estimated)
– Additional info (from in some Exif files)
• White-balance settings, even matrix coefficients of white-balnace
• Picture style (e.g. landscape, vivid, standard, portrait)
• Output color space (e.g. sRGB, Adobe RGB, RAW)
• GPS info
• More . . . 145
 Saved to storage. We are done!
Gain Control
White
A/D Converter
Balance
Possible LUT
AFE – Analog Front End
Sensor with color filter array
Sensor related processing
(CCD/CMOS)

Color Space Noise

Tone CFA
Transform + Reduction/
Reproduction Demoasicing
Color Preferences Sharpening

JPEG Save to
Exif File Info storage
Compression

146
* Note that steps can be optional (e.g. noise reduction) or applied in slightly different order.
 Pipeline comments
• Again, important to stress that the exact steps
mentioned in these notes only serve as a
guide of common steps that take place
• For different camera makes/models, these
could be performed in different order (e.g.
white-balance after demosaicing) and in
different ways (e.g. combining sharpening
with demosaicing)

147
 Color manipulation
Gain Control
White
A/D Converter
Balance
Possible LUT
AFE – Analog Front End
Sensor with Color Filter Array
Sensor related processing
(CCD/CMOS)

Color Space Noise

Tone CFA
Transform + Reduction/
Reproduction Demoasicing
Color Preferences Sharpening

These two steps combined is often referred to as “color

rendering” or “color processing/interpolation”. What
actually happens here is very much specific to individual
cameras, and even settings on the camera.
148
 ICC and color profiles
• International Color Consortium (ICC)
– In charge of developing several ISO standards for color
management
• Promote the use of ICC profiles
• ICC profiles are intended for device manufacturers to
describe how their respective color spaces (e.g. sensor
RGB) map to canonical color spaces called Profile
Connection Spaces (PCS)
• PCS are similar to linking all devices to CIE XYZ, but are
more flexible allowing for additional spaces to be
defined (beyond CIE XYZ)

149
From the ICC –
ISO 22028

In this case, “sensor characterization” is

related to the Color Adaptation Matrix
described earlier.

We can see tone-mapping is not explicitly

denoted, instead it is grouped with “color
render”.

150
 From the ICC This describes
a basic digital
ISO 22028 camera pipeline
in more detail.

RGB values
linked to the
device are
considered
“scene referred”.

We can see a more detailed After the color

breakdown of color transform to
rendering. sRGB they are
denoted as
“output referred”
color encodings.

151
 Tone mapping/color rendering
• From the ICC ISO standards, we see this part is
more complex than just matrix + tone-map
• In fact, it can often involve 3D LUTs

3D LUT
(Color
prefer-
ences)

152
 Camera image processors
• Steps applied after the sensor output are generally performed
by an “image processor” on the camera
• Different cameras use different processing boards/software
• High-end cameras and associated processors
– Nikon – Expeed
– Fuji – Real photo engine
– Canon – DIGIC
– Sony – BIONZ
– ...
• Mobile-devices
– Qualcomm - Snapdragon
– Nvidia - Tegra

153
Example: Expeed block diagram

154
[Link]
Example: Sony Exmor + BIONZ

Not a lot of detail . . .

155
From Sony’s website
 Tutorial schedule
• Part 1 (General Part)
– Motivation
– Review of color/color spaces
– Overview of camera imaging pipeline
• Part 2* (Specific Part)
– Modeling the in-camera color pipeline
– Photo-refinishing
• Part 3 (Wrap Up)
– The good, the bad, and the ugly of commodity cameras
and computer vision research
– Concluding Remarks
156
* Mainly involves shameless self-promotion of my group’s work.
 Part 2:
Modeling the onboard
camera color processing
pipeline
 Part 2 Acknowledgements

Prof. Seon Joo Kim Dr. Hai-ting Lin

(Yonsei) (NUS/U. Delaware)

Prof. Sabine Süsstrunk [Link]

StevenLin
Lin Dr. Zheng Lu
(EPFL) (MSR-Asia)
(MSR-Asia) (NUS/City U Hong Kong)

S. J. Kim et al “A New In-Camera Imaging Model for Color Computer Vision and its
158
Application”, IEEE Transactions on Pattern Analysis and Machine Intelligence, Dec 2012
 Standard color spaces are great

?
“The real scene” Camera
sRGB Image
sRGB -> CIE XYZ -> Display (ICC Profile)

159
 Ideal (simple) camera pipeline
Sensor Image (RAW) sRGB Output
White Color Space
Transform (CST) 1
balance 0.8
3.24 −1.53 0.49 0.6
−0.96 1.87 0.04
0.4
0.05 −0.20 1.05
0.2

0
CIE XYZ 0 0.2 0.4 0.6 0.8 1

CIE XYZ Linear-sRGB

Camera-Speficic
White-balanced CST maps from camera Apply sRGB
R/G/B Sensitivity camera-specific color space to CIE XYZ. encoding gamma.
Functions
raw-RGB Then we go to linear-sRGB.

STEP 1 STEP 2 STEP 3

160
161
 The world isn’t ideal
Samsung Galaxy S6 edge HTC One M9 LG G4

162
 Tone-curve is camera-specific
Sensor Image (RAW)
White Color Space
Transform (CST) 1
balance 0.8
3.24 −1.53 0.49 0.6
−0.96 1.87 0.04
0.4
0.05 −0.20 1.05
0.2

0
CIE XYZ 0 0.2 0.4 0.6 0.8 1
f
CIE XYZ Linear-sRGB

163
 Tone-curve is camera-specific
Sensor Image (RAW)
White Color Space
Transform (CST) 1
balance 0.8
3.24 −1.53 0.49 0.6
−0.96 1.87 0.04
0.4
0.05 −0.20 1.05
0.2

0
CIE XYZ 0 0.2 0.4 0.6 0.8 1
f
CIE XYZ Linear-sRGB

Mann and Picard, SPIE’95

Debevec and Malik, SIG’97
We have known for a long time that Mitsunaga and Nayar, CVPR’99
Farid, TIP’01
nobody uses the sRGB encoding gamma Grossberg and Nayar, TPAMI’03
Grossberg and Nayar, TPAMI’04
Lin et al, CVPR’04
…
Large body of research to Manders et al, ICIP’04
estimate these camera-specific Pal et al, CVPR’04
Lin et al, ICCV’05
tone-curves f. All assume this is Kim and Pollefeys, TPAMI’08
Chakrabarti et al, BMVC’09
fixed per camera. 164
Manufacturers give us a hint

From Canon’s user manual

165
Nikon D7000 - RAW

166
NEUTRAL

167
STANDARD

168
PORTRAIT

169
LANDSCAPE

170
VIVID

171
Camera = light-measuring device

“All models are wrong, but some are useful;

how wrong can they be and still be useful.”

George Box
Professor Emeritus of Statistics
172
U. Wisconsin
 Early work
“Radiometric Calibration”

E*k (RAW) f (x)

E = amount of light (irradiance) Sensor response Final
k = time of integration (exposure) Output

Unknown f (tone-map) is the camera’s non-

linear transform of RAW to “intensity”.
173
Debevec and Malik [SIG’97]
 Accepted model

 Erx   erx   Erx   irx   f r (erx ) 

E  e  E  i   f (e ) 
=
 gx 
 Ebx 
 gx  =
 ebx 
T  gx 
 Ebx 
 gx 
ibx 
 g gx 
 f b (ebx ) 

(RAW) T is a 3x3 matrix i is the sRGB

output and f is
(small e are white-balanced
a non-linear
RAW)
function
174
Prior work
Mann and Picard, SPIE’95

Fixed property of the camera

Debevec and Malik, SIG’97
Mitsunaga and Nayar, CVPR’99
Farid, TIP’01
Grossberg and Nayar, TPAMI’03
Grossberg and Nayar, TPAMI’04
Lin et al, CVPR’04
…
Manders et al, ICIP’04
Pal et al, CVPR’04
Lin et al, ICCV’05
Kim and Pollefeys, TPAMI’08
Chakrabarti et al, BMVC’09

175
 Prior work
Mann and Picard, SPIE’95

Fixed property of the camera

Debevec and Malik, SIG’97
Mitsunaga and Nayar, CVPR’99
Farid, TIP’01
Grossberg and Nayar, TPAMI’03
Grossberg and Nayar, TPAMI’04
Lin et al, CVPR’04
…
Manders et al, ICIP’04
Pal et al, CVPR’04
Lin et al, ICCV’05
Kim and Pollefeys, TPAMI’08
Chakrabarti et al, BMVC’09

Chakrabarti et al conclusions:
 RAW is meaningful . . . .

 But, requires a 24 parameter model that is scene-dependent 176

to accurately go back from sRGB to RAW.
 Scene dependent. . .

Tone curve, f , is computed based on scene content.

This makes it almost impossible to pre-compute.
177
 Accepted model

Is processing scene dependent?

Or is this
 i  model
rx  f (e )  not
r rx e good
 enough?
rxE  rx
i   f (e )  e  E 
 gx  =  g gx 
ibx   f (ebx ) 
 gx  =
 ebx 
T  gx 
 Ebx 

178
 Our experiment: data collection
• More than 10,000 images from 33 cameras from DSLRs to point-and-shoots

• Images of color charts under indoor / outdoor (cloudy)

• Images are taken at all possible shutter speeds, at multiple aperture,

and white balance settings. JPEG / RAW both captured if possible.

* Special shooting features such as lighting optimizer are turned off

179
Lin et al, ICCV 2011; Kim et al, TPAMI 2012
 Data collection
More than 10,000 images from 33 cameras

Different Cameras

Different Lightings sRGB

Different WBs

Different Exposures RAW

180
 Data collection
More than 10,000 images from 33 cameras

Different Cameras

Different Lightings sRGB

Different WBs

Different Exposures RAW

181
 Checked if f is fixed or scene dependent
• How?
• Plot the brightness transfer function (BTF)
– Plot points from image pairs of different scenes
– Each pair has the same ratio, τ , of exposure change

Scene 1 Scene 2 Scene 3

Exposure τki Exposure τki Exposure τkj

Exposure ki Exposure ki Exposure kj

182
 Checked if f is fixed or scene dependent
• How?
• Plot the brightness transfer function (BTF)
– Plot points from image pairs of different scenes
– Each pair has the same ratio, τ , of exposure change

Scene 1 Scene 2 Scene 3

Exposure 2s Exposure 1s Exposure 0.5s

Exposure 1s Exposure 0.5s Exposure 0.25s

183
 Checked if f is fixed or scene dependent
• How?
• Plot the brightness transfer function (BTF)
– Plot points from image pairs of different scenes
– Each pair has the same ratio, τ , of exposure change

Scene 1 Scene 2 Scene 3

Exposure 2s Exposure 1s Exposure 0.5s

Exposure 1s Exposure 0.5s Exposure 0.25s

184
Scene 1

BTF
ki
Exposure ki
Exposure kj

Exposure ki (intensity)
kj

Exposure kj (intensity)

Linear function looks like this. . .

185
Scene 1

BTF
ki
Exposure ki
Exposure kj

Exposure ki (intensity)
Scene 2
kj

Exposure ki
Exposure kj
Scene 3 Exposure kj (intensity)

Linear function looks like this. . .

Exposure ki
Exposure kj 186
Scene 1

BTF ki

Exposure ki
Exposure kj

Exposure ki (intensity)
kj
Scene 2

Exposure ki
Exposure kj
Scene 3 Exposure kj (intensity)

Non-linear BTF looks like this . .

Exposure ki
Exposure kj 187
Scene 1

BTF
Exposure ki
Exposure kj

Exposure ki (intensity)
Scene 2

Exposure ki
Exposure kj
Scene 3 Exposure kj (intensity)

Scene dependent non-linear BTF

looks like this . .
Exposure ki
Exposure kj 188
For the most part . . . it was ok

189
For the most part . . . it was ok
outliers

190
 Where are the outliners?

Outliers were not scene

dependent.
Outliers were color dependent.

191
Gamut mapping

Gamut mapping is necessary

because the gamut of the camera’s
camera’s gamut
color space is different from the
gamut of sRGB.
sRGB gamut

Gamut mapping is a natural

mechanism to support scene
modes, such as vivid model, portrait
mode, landscape mode, etc.

192
 Proposed a new model

RAW sRGB
Tw

Ts
White balance (Tw) Gamut mapping Tone mapping
Camera RAW to sRGB (Ts) (h) (f) sRGB
(RAW) (JPEG)

193
 Proposed a new model

RAW sRGB
Tw

Ts
White balance (Tw) Gamut mapping Tone mapping
Camera RAW to sRGB (Ts) (h) (f) sRGB
(RAW) (JPEG)

irx    Erx  
Introduce h, a 3D function
i   f (h  T E  )
that takes in input RGB and
 gx    gx  
maps it to a new RGB value
ibx   E  
  bx  
194
 sRGB Image to RAW
 f 1 (irx )    erx     Erx  
 1     
 f (igx )  h  egx   h  T  E gx  
 f 1 (ibx ) 
=  e   =  E  
    bx     bx  

Based on several sRGB-RAW pairs,

• f-1 & T-1 are computed using less saturated points
• h-1 is computed with scatter point interpolation via radial basis func.

Gamut mapping modeled using

radial basis function

Gamut Mapping
(h)
195
 196

Canon EOS1Ds Mark III Picture Styles

Ts White Balance S Standard P Portrait L Landscape …
…
Tw1 Tw2 Tw3 Tw4 h1, f1 h2, f2 h3, f3

RAW sRGB
Tw

Ts
White balance (Tw) Gamut mapping Tone mapping
Camera RAW to sRGB (Ts) (h) (f) sRGB
(RAW) (JPEG)
 197

Canon EOS1Ds Mark III Picture Styles

Ts White Balance S Standard P Portrait L Landscape …
…
Tw1 Tw2 Tw3 Tw4 h1, f1 h2, f2 h3, f3

RAW sRGB
Tw

Ts
White balance (Tw) Gamut mapping Tone mapping
Camera RAW to sRGB (Ts) (h) (f) sRGB
(RAW) (JPEG)
 Gamut mapping

Mapping is represented as a displacement map of the

camera’s original RGB value to its sRGB location.

The plots above are displacement map of the function h, projected

into a 2D plane. This is intended to help visualize how much deformation
to the color is taking place. E.g. we can see in “Landscape style”, the
green and blue colors are more manipulated for the Canon EOS 1Ds.
198
Experiments : Mapping sRGB back to RAW

input sRGB image ground truth RAW

Canon EOS1D

199
Experiments : Mapping sRGB back to RAW

input sRGB image estimated RAW

Canon EOS1D

200
 Experiments : Mapping sRGB back to RAW

Our model (f, T, h) old model (f, T)

We cannot handle
fully saturated points. Canon EOS1D

201
Experiments : Mapping sRGB back to RAW

input sRGB image ground truth RAW

Canon EOS550D

202
Experiments : Mapping sRGB back to RAW

input sRGB image estimated RAW

Canon EOS550D

203
Experiments : Mapping sRGB back to RAW

new model (f, T, h) old model (f, T)

Canon EOS1D

204
Experiments : Mapping sRGB back to RAW

input sRGB image ground truth RAW

Sony A200
205
Experiments : Mapping sRGB back to RAW

input sRGB image estimated RAW

Sony A200
206
Experiments : Mapping sRGB back to RAW

Our model (f, T, h) old model (f, T)

Sony A200

207
Application: Photo Refinishing

208
 209

Canon EOS1Ds Mark III Picture Styles

Ts White Balance S Standard P Portrait L Landscape …
…
Tw1 Tw2 Tw3 Tw4 h1, f1 h2, f2 h3, f3

RAW sRGB
Tw

Ts
White Balance (Tw) Gamut Mapping Tone Mapping
Camera RAW to sRGB (Ts) (h) (f) sRGB
(RAW) (JPEG)
 210

Canon EOS1Ds Mark III Picture Styles

Ts White Balance S Standard P Portrait L Landscape …
…
Tw1 Tw2 Tw3 Tw4 h1, f1 h2, f2 h3, f3

RAW sRGB
Tw

Ts
White Balance (Tw) Gamut Mapping Tone Mapping
Camera RAW to sRGB (Ts) (h) (f) sRGB
(RAW) (JPEG)
 211

Canon EOS1Ds Mark III Picture Styles

Ts White Balance S Standard P Portrait L Landscape …
…
Tw1 Tw2 Tw3 Tw4 h1, f1 h2, f2 h3, f3

RAW sRGB
Tw

Ts
White Balance (Tw) Gamut Mapping Tone Mapping
Camera RAW to sRGB (Ts) (h) (f) sRGB
(RAW) (JPEG)
 212

Canon EOS1Ds Mark III Picture Styles

Ts White Balance S Standard P Portrait L Landscape …
…
Tw1 Tw2 Tw3 Tw4 h1, f1 h2, f2 h3, f3

RAW sRGB
Tw

Ts
White Balance (Tw) Gamut Mapping Tone Mapping
Camera RAW to sRGB (Ts) (h) (f) sRGB
(RAW) (JPEG)
What if you took a photo
with the wrong settings?
 213

Canon EOS1Ds Mark III Picture Styles

Ts White Balance S Standard P Portrait L Landscape …
…
Tw1 Tw2 Tw3 Tw4 h1, f1 h2, f2 h3, f3

RAW sRGB
Tw

Ts
White Balance (Tw) Gamut Mapping Tone Mapping
Camera RAW to sRGB (Ts) (h) (f) sRGB
(RAW) (JPEG)
 214

Canon EOS1Ds Mark III Picture Styles

Ts White Balance S Standard P Portrait L Landscape …
…
Tw1 Tw2 Tw3 Tw4 h1, f1 h2, f2 h3, f3

RAW sRGB
Tw -1

Ts -1

White Balance (Tw -1) Gamut Mapping Tone Mapping

Camera RAW to sRGB (Ts -1) (h -1) ( f -1) sRGB
(RAW) (JPEG)
 215

Canon EOS1Ds Mark III Picture Styles

Ts White Balance S Standard P Portrait L Landscape …
…
Tw1 Tw2 Tw3 Tw4 h1, f1 h2, f2 h3, f3

RAW sRGB
Tw

Ts

Camera sRGB
Tw -1
(RAW) (JPEG)

Ts -1

White Balance (Tw -1) Gamut Mapping Tone Mapping

RAW to sRGB (T -1) (h -1) ( f -1)
 Result - Canon EOS 1Ds Mark III

Input: cloudy WB + landscape style

216
 Result - Canon EOS 1Ds Mark III

Ground truth: fluorescent WB + standard style

217
Result - Canon EOS 1Ds Mark III

Photoshop result
218
Result - Canon EOS 1Ds Mark III

Refinished result
219
 Result - Canon EOS 1Ds Mark III

Ground truth: fluorescent WB + standard style

220
 Result - Canon EOS 1Ds Mark III

Input Ground truth

Photoshop Our refinished result

221
 Result - Canon EOS 1Ds Mark III

Input: tungsten WB + standard style

222
 Result - Canon EOS 1Ds Mark III

Ground truth: daylight WB + standard style

223
Result - Canon EOS 1Ds Mark III

Photoshop result
224
Result - Canon EOS 1Ds Mark III

Our refinished result

225
 Result - Canon EOS 1Ds Mark III

Ground truth: daylight WB + standard style

226
 Result - Canon EOS 1Ds Mark III

Input Ground truth

Photoshop Our refinished result

227
 Result – Nikon D200

Input: tungsten WB + standard style

228
 Result – Nikon D200

Ground truth: daylight WB + standard style

229
Result – Nikon D200

Photoshop result
230
Result – Nikon D200

Refinished result
231
 Result – Nikon D200

Ground truth: daylight WB + standard style

232
 Result – Nikon D200

Input Ground truth

Photoshop Photo refinish

233
 Result - Sony α200

Input: tungsten WB + standard style

234
 Result - Sony α200

Ground truth: daylight WB + standard style

235
Result - Sony α200

Photoshop result
236
Result - Sony α200

Our refinished result

237
 Result - Sony α200

Ground truth: daylight WB + standard style

238
 Result - Sony α200

Input Ground truth

Photoshop Our Refinished Result

239
Remember these guys?

240
241
 Aside: Probabilistic Approach

The mapping of the function, h, is not one-to-one. Chakrabarti et al [TPAMI 2014] has a
nice paper on a probabilistic approach for this inverse mapping problem.

A. Chakrabarti, Y. Xiong, B. Sun, T. Darrell, D. Scharstein, T. Zickler, and K. Saenko,

"Modeling Radiometric Uncertainty for Vision with Tone-mapped Color Images," 242
IEEE Transactions on Pattern Analysis and Machine Intelligence, 2014
 Aside: Probabilistic Approach

A. Chakrabarti, Y. Xiong, B. Sun, T. Darrell, D. Scharstein, T. Zickler, and K. Saenko,

"Modeling Radiometric Uncertainty for Vision with Tone-mapped Color Images," 243
IEEE Transactions on Pattern Analysis and Machine Intelligence, 2014
 Tutorial schedule
• Part 1 (General Part)
– Motivation
– Review of color/color spaces
– Overview of camera imaging pipeline
• Part 2 (Specific Part)
– Modeling the in-camera color pipeline
– Photo-refinishing
• Part 3 (Wrap Up)
– The good, the bad, and the ugly of commodity cameras
and computer vision research
– Concluding Remarks
244
 245

Part 3:
Wrap Up
 Image Credit: Timothy Anderson

The Good, The Bad, and the Ugly - 1966

Western Movie directed by Sergio Leone
One of the most influential “Western” films and one of the “Great Films of All Times” 246
247
 Color on cameras is not standard
• We have already discussed the “bad” in part 2
• No camera outputs true “sRGB” with respect
to the incoming light
• Each camera applies its own color rendering
• Calibration of the camera is required
• While we have good calibration methods, such
calibration needs to be done for each camera,
and for many different settings (not practical)

248
 No standard for RAW color space
• RAW is good because it is linear
• However, RAW is a scene-referred color space,
specific to the sensor
• This mean RAW RGB values from image on
difference cameras of the same scene will be
different

249
 Example – RAW is not standard

Example
Top: RAW images from three cameras, all of the same scene.
Bot: Error plots showing the pixel-wise L2 difference between camera pairs

250
251
 Problems in academic research
1. Lack of understanding of color on cameras
and relationship to “real” color spaces

2. Research and results performed “out of

context” of the camera processing pipeline

3. Lack of ability to emulate full camera

pipelines

252
 Ugly Example – Color spaces
• Recall our color space transforms
• Camera images are saved in sRGB

Linear-sRGB back to XYZ

Linear-NTSC back to XYZ

253
Ugly examples

254
Ugly examples

255
Ugly examples

256
 Analyzing ugly
• One of the most common examples in the
academic literature – obtaining the luminance
(CIE Y) channel from an sRGB image

sRGB CIE Y

• Goal is to often purported to find the imaged

scene’s “brightness” (i.e. CIE Y)

257
Analyzing sRGB to luminance (Y)

1) Perform experiments to see just how

accurate sRGB to luminance conversion

2) Examine the common mistakes made in

the academic literature

258
 Experimental setup
Generate “true CIE XYZ” images (ground truth)
SPD1

𝑆(x, 𝜆) Chromaticity
y
SPD2 (Y:38, x:0.24, y:0.61)

400 500 600 700

𝐶𝑐 (𝜆) Z CIE XYZ
2.5
X X
2 Y
Z
Y
1.5

0.5

0
400 500 600 700
SPD2
Color Matching Functions
CIE XYZ Y 0 SPD1 (Y:50, x:0.56, y:0.37) 𝑥
400 500 600 700
(Luminance)
Scene (Spectral Power Distribution) Image under CIE XYZ color space

259
 Test on two types of inputs
SPD1
𝑆(x, 𝜆)

400 500 600 700

SPD2

400 500 600 700

Color rendition chart Full scenes

260
 Generate synthetic camera images
Sensor Image (RAW)

White Color Space CIE XYZ Tone

balance Transform (CST) Linear-sRGB Curve

Simulate different Generate images with Generate images

cameras using different white balance with camera-specific
sensitivity functions settings on camera tone-curves

261
Three common mistakes in the literature
(when attempting to map a sRGB camera image to CIE – Y)

• Assumption that white balance is correct

• Using the wrong equations (and calling it CIE Y)

• Not applying the correct linearization step

262
 1. White balance assumption

Sensor Image (RAW)

White Color Space CIE XYZ Tone

balance Transform (CST) Linear-sRGB Curve

There is an implicit assumption

that the white balance was
estimated correctly

263
 2. Wrong Equations
NTSC is used instead of sRGB (often under the guise of YUV/YIQ)

∗
Correct eq (𝑌 = 0.2126𝑅 + 0.7152𝐺 + 0.0722𝐺)
Average of RGB is used

Hue, Saturation, Value (HSV) - Value is taken to equate to scene Luminance

* YUV/YIQ are actually defined with these weighted coefficients applied on the gamma-encoded
RGB. So, the entire equation is an incorrect interpretation of Y. 264
 3. Linearization is incorrect
Sensor Image (RAW)
White- Color Space
Transform (CST) 1
balance 0.8
3.24 −1.53 0.49 0.6
−0.96 1.87 0.04
0.4
0.05 −0.20 1.05
0.2

0
CIE XYZ 0 0.2 0.4 0.6 0.8 1

CIE XYZ Linear-sRGB Camera-specific

tone-curve

0.8

Apply inverse 0.6

0.4

equations 0.2

0
0 0.2 0.4 0.6 0.8 1

Wrong tone-curve
(sRGB gamma)

265
 3. Linearization is not applied
Sensor Image (RAW)
White- Color Space
Transform (CST) 1
balance 0.8
3.24 −1.53 0.49 0.6
−0.96 1.87 0.04
0.4
0.05 −0.20 1.05
0.2

0
CIE XYZ 0 0.2 0.4 0.6 0.8 1

CIE XYZ Linear-sRGB Camera-specific

tone-curve

0.8

Apply inverse 0.6

0.4

equations 0.2

0
0 0.2 0.4 0.6 0.8 1

Wrong tone-curve
(sRGB gamma)

266
 3. No linearization – “Luma”

Ignoring linearization has a name. It is called Luma instead of Luminance.

Variable Y’ is used to distinguish it from Luminance Y (e.g. YIQ, YUV)

‘ ‘ ‘ ‘

‘ ‘ ‘ ‘

‘ ‘ ‘ ‘

267
How ugly is it?

268
 White balance not correct

CIE – Y

Nikon camera – white balance is wrong, but we have ideal CST, sRGB gamma

Overall error less than 1.5%

269
 Wrong tone-curve (right equation)
Camera tone-curves Ground Truth Y Result with using sRGB gamma Error Map

Errors for some colors can be over 25-40%270

Wrong equations (right tone-curve)

271
Wrong equations (wrong tone-curve)

Commonly found
in academic papers 272
Wrong equations (no linearization)

All very common

in academic papers
273
 Ugly analysis
• White balance assumption violation not too serious

• Incorrect tone-curve is significant (25-40% error)

• No tone-curve (Luma) is the worse (over 40% error)

• HSV is the worst of the “wrong” equations,

but not worse than using the incorrect
tone-curve

274
 Why bother with CIE-Y?

(a) sRGB image (b) Y of YIQ (c) RGB to Gray*

SIFT

(d) SIFT features on (b) (e) SIFT features on (c)

Canny

(f) Canny edges on (b) (g) Canny edges on (c)

275
*C. Lu, et al “Contrast preserving decolorization with perception-based quality metrics”, IJCV, 2014
More examples . .
IJCV’14

276
 Another ugly problem . . .
• Applying operations in the “wrong” context . . .

• E.g. applying white balance on sRGB images . . .

• Or, denoising an sRGB image. . .

• Or, deblurring an sRGB image . . .

277
 Camera processing pipeline
Gain Control
A/D Converter White Balance
Possible LUT
Sensor with color filter array
(CCD/CMOS)

Color Space Noise

CFA
Tone Curve Transform + Color Reduction/
Demoasicing
Preferences Sharpening

JPEG Save to
Exif File Info storage
Compression

sRGB
278
Classic white balance results out-of-context

“Subjective” white balance results shown

(images are in camera-raw color space)
These subjective results have absolutely no visual meaning.
The camera-raw is not a standard color space!

From Cheng et al CVPR 2015 279

 Consider image deblurring

Assumption is: linear color space (RAW or linear-sRGB)

Reality: image has been run through the pipeline and some non-linear tone-curve f

280
 Gamma/tone-curve effect on blur
linear input non-linear input

Tai Y.-W. , Chen X., Kim S., Kim S. J, Li F., Yang J., Yu J., Matsushita Y., Brown M. S. (2013) "Nonlinear Camera
Response Functions and Image Deblurring: Theoretical Analysis and Practice", IEEE Transactions on Pattern Analysis
and Machine Intelligence (TPAMI), 35(10), Oct 2013 281
 RAW vs sRGB deblurring

Input with blur Ground Truth Deblur on RAW Deblur on sRGB

282
 State of affairs (The Ugly)
• Many researchers don’t understand camera color

• Attempt to treat sRGB images as true scene

measurements (ideal sRGB images)
– Often use wrong equations, forget linearization, etc. .
– Results can be erroneous by up to 50%
– Why even attempt CIE Y?

• Research is often applied without understanding

the context of the color processing pipeline

283
284
285
From [Link]
 Adobe Digital Negative (DNG)
• Public raw-camera image file specification

• Open source SDK for processing DNG to sRGB

• After almost 10 years, this is becoming

mainstream

287
 Android Camera2 API

• Allows access to many of the onboard

camera procedures

• Access to camera characteristics (CST, white

balance)

• Can capture RAW images in DNG format

288
 DNG SDK “access” to the pipeline
Demosaicing
Black Light Lens Correction
Reading RAW
Subtraction
Image
Linearization
1 2 3 4

Exposure Curve
White Balance
Application Hue/Sat Map
Noise Reduction
Application Color Space
7 6 5 Transform

Tone Curve Final Gamma Curve

Look Table Application Color Space Application
Application Conversion
8 9 10 11

289
sRGB
 Matlab platform for color pipeline
manipulation
User has access to the image at each step in
the color processing pipeline.

Parameters for each step can be modified.

All using Matlab calls.

Camera color pipeline platform

Matlab

Camera 2 API Adobe DNG SDK

Camera raw + camera parameters (CST, WB,

lens correction, etc)
290
 Matlab platform for color pipeline
manipulation
Black Light
Reading RAW Lens Demosaicing
Subtraction
Image Correction
Linearization

Exposure White Balance

Hue/Sat Map Noise
Curve Color Space
Application Reduction
Application Transform

Tone Curve Final Gamma Curve

Look Table
Application Color Space Application
Application
Conversion

Camera color pipeline platform

Matlab Output Image

Camera 2 API Adobe DNG SDK

Camera raw + camera parameters (CST, WB,

lens correction, etc)
291
 Matlab camera platform / Adobe DNG
Demosaicing
Black Light Lens Correction
Reading RAW
Subtraction
Image
Linearization
1 2 3 4

Exposure Curve
White Balance
Application Hue/Sat Map
Noise Reduction
Application Color Space
7 6 5 Transform

Tone Curve Final Gamma Curve

Look Table Application Color Space Application
Application Conversion
8 9 10 11

292
Reading RAW Image

293
 Matlab camera platform / Adobe DNG
Demosaicing
Black Light Lens Correction
Reading RAW
Subtraction
Image
Linearization
1 2 3 4

Exposure Curve
White Balance
Application Hue/Sat Map
Noise Reduction
Application Color Space
7 6 5 Transform

Tone Curve Final Gamma Curve

Look Table Application Color Space Application
Application Conversion
8 9 10 11

294
Black light subtraction + linearization

295
 Matlab camera platform / Adobe DNG
Demosaicing
Black Light Lens Correction
Reading RAW
Subtraction
Image
Linearization
1 2 3 4

Exposure Curve
White Balance
Application Hue/Sat Map
Noise Reduction
Application Color Space
7 6 5 Transform

Tone Curve Final Gamma Curve

Look Table Application Color Space Application
Application Conversion
8 9 10 11

296
Lens Correction
(non-uniform gain)

297
 Matlab camera platform / Adobe DNG
Demosaicing
Black Light Lens Correction
Reading RAW
Subtraction
Image
Linearization
1 2 3 4

Exposure Curve
White Balance
Application Hue/Sat Map
Noise Reduction
Application Color Space
7 6 5 Transform

Tone Curve Final Gamma Curve

Look Table Application Color Space Application
Application Conversion
8 9 10 11

298
Demosaic

299
 Matlab camera platform / Adobe DNG
Demosaicing
Black Light Lens Correction
Reading RAW
Subtraction
Image
Linearization
1 2 3 4

Exposure Curve
White Balance
Application Hue/Sat Map
Noise Reduction
Application Color Space
7 6 5 Transform

Tone Curve Final Gamma Curve

Look Table Application Color Space Application
Application Conversion
8 9 10 11

300
White balance + color space transform

301
 Matlab camera platform / Adobe DNG
Demosaicing
Black Light Lens Correction
Reading RAW
Subtraction
Image
Linearization
1 2 3 4

Exposure Curve
White Balance
Application Hue/Sat Map
Noise Reduction
Application Color Space
7 6 5 Transform

Tone Curve Final Gamma Curve

Look Table Application Color Space Application
Application Conversion
8 9 10 11

302
Hue/Sat map application

303
 Matlab camera platform / Adobe DNG
Demosaicing
Black Light Lens Correction
Reading RAW
Subtraction
Image
Linearization
1 2 3 4

Exposure Curve
White Balance
Application Hue/Sat Map
Noise Reduction
Application Color Space
7 6 5 Transform

Tone Curve Final Gamma Curve

Look Table Application Color Space Application
Application Conversion
8 9 10 11

304
Exposure curve application

EV+1

305
 Matlab camera platform / Adobe DNG
Demosaicing
Black Light Lens Correction
Reading RAW
Subtraction
Image
Linearization
1 2 3 4

Exposure Curve
White Balance
Application Hue/Sat Map
Noise Reduction
Application Color Space
7 6 5 Transform

Tone Curve Final Gamma Curve

Look Table Application Color Space Application
Application Conversion
8 9 10 11

306
3D lookup table application

307
 Matlab camera platform / Adobe DNG
Demosaicing
Black Light Lens Correction
Reading RAW
Subtraction
Image
Linearization
1 2 3 4

Exposure Curve
White Balance
Application Hue/Sat Map
Noise Reduction
Application Color Space
7 6 5 Transform

Tone Curve Final Gamma Curve

Look Table Application Color Space Application
Application Conversion
8 9 10 11

308
Tone curve application

309
 Matlab camera platform / Adobe DNG
Demosaicing
Black Light Lens Correction
Reading RAW
Subtraction
Image
Linearization
1 2 3 4

Exposure Curve
White Balance
Application Hue/Sat Map
Noise Reduction
Application Color Space
7 6 5 Transform

Tone Curve Final Gamma Curve

Look Table Application Color Space Application
Application Conversion
8 9 10 11

310
Final color space conversion
(CIE XYZ -> linear sRGB)

311
 Matlab camera platform / Adobe DNG
Demosaicing
Black Light Lens Correction
Reading RAW
Subtraction
Image
Linearization
1 2 3 4

Exposure Curve
White Balance
Application Hue/Sat Map
Noise Reduction
Application Color Space
7 6 5 Transform

Tone Curve Final Gamma Curve

Look Table Application Color Space Application
Application Conversion
8 9 10 11

312
sRGB gamma curve application

313
All together now

Camera RAW

314
All together now

Black level
subtraction and
linearization
315
All together now

Lens correction
316
(non-uniform gain)
All together now

Demosaicing
317
+ Noise Reduction
All together now

White balance
318
+ color space transform (CIE XYZ)
All together now

Hue-saturation
319
adjustment
All together now

Exposure
320
Compensation
All together now

Tone-Curve 321
All together now

sRGB color space

322
All together now

sRGB gamma 323

Using the platform - Examples

• White balance “in context”

• Camera colorimetry

• Real sRGB outputs

324
 “White balance” in context
Demosaicing
Black Light Lens Correction
Reading RAW
Subtraction
Image
Linearization
1 2 3 4

Exposure Curve
White Balance
Application Hue/Sat Map
Noise Reduction
Application Color Space
7 6 5 Transform

Tone Curve Final Gamma Curve

Look Table Application Color Space Application
Application Conversion
8 9 10 11

325
sRGB
 White balance example

Input Raw White balance method 1 White balance method 2

This is how subjective WB results would have been presented before.

Results through the “full pipeline”

326
 Colorimetry example
Demosaicing
Black Light Lens Correction
Reading RAW
Subtraction
Image
Linearization
1 2 3 4

Exposure Curve
White Balance
Application Hue/Sat Map
Noise Reduction
Application Color Space
7 6 5 Transform

Tone Curve Final Gamma Curve

Look Table Application Color Space Application
Application Conversion
8 9 10 11

327
sRGB
 Example - colorimetry
Sensor Image (RAW)
White Color Space
Transform (CST) 1
balance 0.8
3.24 −1.53 0.49 0.6
−0.96 1.87 0.04
0.4
0.05 −0.20 1.05
0.2

0
CIE XYZ 0 0.2 0.4 0.6 0.8 1

CIE XYZ Linear-sRGB 3D RGB Warp

h f

Can compute our own CST.

Derive a mapping from the camera
raw color space to the CIE XYZ space.
This is the CST in the pipeline.

328
Calibrate using X-Rite software

329
 Four Android cameras
Samsung S6-Edge HTC M9 Motorola Nexus 6 LG-G4

Goal is that the colors for all the cameras are the same.

330
 Color chart patches
RAW– (i.e. without any color transformation)

331
 Color chart patches
RAW + white balanced and CST from the camera

332
Color chart patches
X-Rite profile calibration (CST)

333
Color chart patches
Our CST estimation

334
Based on a method by Bastani and Funt, SPIE 2014
Applied to other materials

81 Paint chips

335
 Additional materials
RAW– (i.e. without any color transformation)

336
 Additional materials
RAW (white balanced and CTM of camera)

337
Additional materials
X-rite Profile

338
Additional materials
Our CST estimation

339
 True sRGB Example
Demosaicing
Black Light Lens Correction
Reading RAW
Subtraction
Image
Linearization
1 2 3 4

Exposure Curve
White Balance
Application Hue/Sat Map
Noise Reduction
Application Color Space
7 6 5 Conversion

Tone Curve Final Gamma Curve

Look Table Application Color Space Application
Application Conversion
8 9 10 11

340
sRGB
True sRGB vs. camera RGB

“True sRGB” Camera sRGB

341
True sRGB vs. camera RGB

“True sRGB” Camera sRGB

342
 State of affairs (The Good)
• DNG allows access to processing pipeline
parameters

• Android Camera2 API now supports DNG

capture

• Developing a Matlab platform for researcher

working on low-level vision

343
 Concluding remarks
• Your camera performs a great deal of processing
• RAW
– Represents relatively unprocessed sensor response
– Does not represent CIE XYZ
– Transform to CIE XYZ is not readily available
– There is no standard RAW!
• sRGB
– Is defined with respect to CIE XYZ
– However, incoming radiant energy (SPDs) to sRGB is
not standardize (otherwise all camera images would
look the same)
344
 Concluding remarks
• Our experiments found
– Mapping from RAW to sRGB changes for different
picture styles (portrait, landscape, vivid, standard,
etc. . )
– Picture styles can be modeled with a 3D function
+ tone-map
– White-balance (so far) seems independent of
picture styles

345
Good, Bad, Ugly Summary
• Cameras are black boxes
• sRGB is not standard across cameras
• Makes it hard to develop apps

• Researchers need more education

• Lack of understanding
• Research being performed “out of context”

• Situation is (slowly) changing

• DNG + Android
• Platforms to support research are coming
346
Embrace the in-camera processing
• Onboard photofinishing make our photos look
great
• Embrace the technology!

347
 Thank you for attending

Li Yu Hakki Karaimer Rang Nguyen Cheng Dongliang

Special thanks to my students and colleagues for contributing images, code,

and materials to this tutorial.

348
 Understanding Color Perception

Hermann Grassmann Johannes von Kries W. David Wright

James Clerk Maxwell Hermann von Helmholtz Thomas Young 349

 Digital Cameras

Bryce E. Bayer Willard Boyle and George Smith Eric R. Fossum

(Nobel Prize for inventing the CCD) (Invented CMOS)
Photo: Reuters

350
 And of course

“The Standard Observers”

351
 References/additional materials
Books
∙ G. Sharma, Digital Color Imaging Handbook, CRC Press, 2003
∙ M. Fairchild, Color Appearance Models, Wiley, 2005
∙ D. Forsyth & J. Ponce, Computer Vision: A modern approach, Prentice Hall, 2011
∙ R. Lukac, Single-Sensor Imaging: Methods and Applications for Digital Cameras, CRC Press, 2008
∙ C. Poynton, Digital Video and HD Algorithms and Interfaces, Morgan Kaufmann, 2012
Articles/Conference Papers
∙ R. Ramantha et al “Color Image Processing Pipeline: a general survey of digital still cameras”, IEEE Signal
Processing Magazine, Jan 2005
∙ H. Fairman et al “How the CIE 1931 Color-Matching Functions Were Derived from Wright–Guild Data”, Color
Research & Application, Feb 1997
∙ G. Meyer, “Tutorial on Color Science”, The Visual Computer, 1986
∙ S. J. Kim et al “A New In-Camera Imaging Model for Color Computer Vision and its Application”, IEEE
Transactions on Pattern Analysis and Machine Intelligence, Dec 2012
∙ E. Garcia, M. Gupta , “Building accurate and smooth ICC profiles by lattice regression”, Color and Imaging
Conference, 2009
∙ A. Chakrabarti et al. "Modeling Radiometric Uncertainty for Vision with Tone-mapped Color Images," IEEE
Transactions on Pattern Analysis and Machine Intelligence, 2014
∙ Cheng D. L., Price B., Cohen S., Brown M. S. (2015) “Beyond White: Ground Truth Colors for Color Constancy
Correction”, IEEE International Conference on Computer Vision, Dec 2015
∙ Tai Y.-W. , Chen X., Kim S., Kim S. J, Li F., Yang J., Yu J., Matsushita Y., Brown M. S. (2013) "Nonlinear Camera
Response Functions and Image Deblurring: Theoretical Analysis and Practice", IEEE Transactions on Pattern
Analysis and Machine Intelligence (TPAMI), 35(10), Oct 2013
 References/additional materials
Websites/Online Resources
ICC – ISO 22028 (2012) Document
Photo Tech Edu Series Jan 2007 (Google)
Available on youtube
Richard Lyon: Digital Camera Image Processing Pipelines
[Link]
Mark Meyer Photography
Professional photography blog with nice tie-ins to color from a
photographers point of view
[Link]
Doug Kerr postings
A large collection of self-published articles on various aspects of imaging in
an “accessible language” to most readers

353
 Acknowledgements
This work was supported in part by generous
gifts from:
• Adobe Research
• Google Research (Faculty Award 2015)

354