Example-Based Video Color Grading

Nicolas Bonneel1∗ Kalyan Sunkavalli2 Sylvain Paris2 Hanspeter Pfister1

1 2
Harvard University Adobe

(a) Input segmented video (b) Model segmented video “Transformers” (c) Our result

Figure 1: Color grading is the process of adjusting the color and tonal balance of a movie to achieve a specific look. This is a critical step of
the movie editing pipeline. However, even with dedicated software, it remains a painstaking task that can be done only by skilled artists. We
propose a new model-based approach that automatically transfers the look of a professionally edited sequence to another video. To produce
sophisticated effects like the contrasted orange-teal look of this example, we use a user-provided foreground-background segmentation. This
allows us to process the input sequence (a) to reproduce the characteristic visual style of “Transformers” (b) to convey a similar tense mood (c).
Our approach produces results that are free from artifacts and temporally coherent, as can be seen in the companion video. Video credits:
Mike Lerner (input), “Transformers”(2007) c Paramount Pictures (model).

Abstract 1 Introduction
In most professional cinema productions, the color palette of the The color palette used in a movie often plays a critical role in estab-
movie is painstakingly adjusted by a team of skilled colorists – lishing its visual look. It can be used to locate a movie in place and
through a process referred to as color grading – to achieve a certain time – for example, the Coen Brothers’ 2000 film, O’ Brother, Where
visual look. The time and expertise required to grade a video makes Art Thou? uses a sepia-tinted color scheme to evoke its setting of
it difficult for amateurs to manipulate the colors of their own video rural Mississippi during the time of the Great Depression1 . In other
clips. In this work, we present a method that allows a user to transfer instances, the color scheme is manipulated to evoke certain emotions
the color palette of a model video clip to their own video sequence. or reinforce a certain mood (as demonstrated by Jean-Pierre Jeunet’s
We estimate a per-frame color transform that maps the color dis- use of rich, warm colors to reinforce the vibrant, happy mood of his
tributions in the input video sequence to that of the model video 2001 film, Amélie). Over time, certain looks have come to represent
clip. Applying this transformation naively leads to artifacts such as entire genres of movies – Film Noir’s use of low-key lighting and
bleeding and flickering. Instead, we propose a novel differential- contrast between light and shadows is one such iconic visual style.
geometry-based scheme that interpolates these transformations in a
manner that minimizes their curvature, similarly to curvature flows. This relationship between visual styles and the process of story-
In addition, we automatically determine a set of keyframes that best telling [Oldenborg 2006] makes color management a critical part
represent this interpolated transformation curve, and can be used of film production. The visual style of a movie is often carefully
subsequently, to manually refine the color grade. We show how our devised by the cinematographer, and executed by a team of skilled
method can successfully transfer color palettes between videos for a colorists who manipulate the colors of the movie footage – through a
range of visual styles and a number of input video clips. process known as color grading – to match his or her vision. While
in the past color grading was done using photo-chemical processing,
most modern post-production pipelines digitize the movie footage
CR Categories: I.4.3 [Computing Methodologies]: Image Process-
and use a combination of hardware and software tools to digitally
ing and Computer Vision—Enhancement;
color grade the movie [Selan 2012]. Today, color grading tools are
even part of popular video processing software such as After Effects
Keywords: color grading, color transfer, video, visual look and Final Cut Pro.
Links: DL PDF However, in spite of the range of tools available today, color grading
is still a tedious process that requires a skill level and time budget
∗ e-mail:nbonneel@[Link]
that puts it out of the reach of amateur video enthusiasts. The goal
of our work is to make it possible for amateur users to apply popular
color grading styles to their own home videos with minimal user
interaction. We achieve this using an example-based approach; users
are asked to specify a model video (or image) that represents the
color grading style they like, and our technique transfers the color
palette of this model video to their clip. This approach offers two
advantages; first, it allows users to specify the visual style they would
like in an intuitive manner, and second, it allows us to leverage the
1 The
short documentary Painting with Pixels: O’ Brother, Where Art
Thou? offers a fascinating perspective into this process.
(a) first frame
applied to all
(b) one transform
per frame
(c) our result

Figure 2: (a) We evaluate the temporal consistency of several color transfer options on four frames. The first three frames are nearly
consecutive and the last one is more distant. Computing the color transform once at the beginning and applying it to the entire sequence yields
results that degrade as time passes (e.g., the bluish face on the last frame). (b) Evaluating the transform at each frame produces temporally
inconsistent results when the content changes (e.g., when the man appears in the second frame). (c) In comparison, our result is stable and
does not drift. Video credits: Greg Henkel.

skill and time that went into the grading of the model video clip. rameters stored at a few sparse keyframes that can be easily edited by
artists, thereby preserving their full artistic control over the output.
By posing video color grading as the problem of transferring color
distributions between video sequences, we can leverage an exten-
sive body of work on color matching in images [Reinhard et al. 2 Previous work
2001; Pitié et al. 2005]. However, these methods cannot be eas-
ily extended to video sequences – applying color matching naively Color transfer for images Matching colors in images has attracted
to every frame of video sequences leads to artifacts such as color much attention since the pioneering work of Reinhard et al. [2001].
bleeding and temporal incoherence (see Fig. 2). Instead, in our These methods have either tried to match the entire color distribu-
work, we estimate per-frame color transformations, and filter these tion of the model image [Pitié et al. 2005] or focused on low order
transformations using a novel temporal filtering technique based on statistics so as to avoid artifacts such as overly high contrasts [Rein-
new approximate curvature flow and differential geometry principles. hard et al. 2001; Pitié and Kokaram 2007]. Accounting for spatial
This filtering technique considers the sequence of transformations as knowledge has shown to improve results when the content of the two
a high-dimensional curve in a manifold, and replaces pieces of this images differs, by transferring colors locally between corresponding
curve that vary at a high frequency by a geodesic interpolation. This segments or using user specified strokes [Reinhard et al. 2001; Tai
allows us to preserve the video content faithfully while simultane- et al. 2005; An and Pellacini 2010; Pouli and Reinhard 2011], or
ously handling temporal consistency issues at both short scales (for spatially matching [HaCohen et al. 2011] or aligning [Kagarlitsky
example, high-frequency flickering) and longer scales (for example, et al. 2009] the images. Color transfer is important to a number of
changes in the scene or lighting). In addition, our filtering technique applications; it has been used to mimic the style of a specific cam-
also allows us to automatically determine a small set of keyframes era [Wang et al. 2011], to convey semantic concepts [Murray et al.
that can be used to further artistically manipulate the color palette. 2011], and to increase the realism of composite images [Xue et al.
As we show in our results, our technique is able to handle a number 2012; Johnson et al. 2011]. In addition to color, other aspects of
of visual styles (including Film Noir, bleach bypass, orange-teal) appearance also effect the visual look of an image. Bae et al. [2006]
and a wide range of input video sequences. transfer tonal balance and detail across images using a non linear
two-scale decomposition.
Contributions We describe a technique to transfer the look of pro-
fessionally color graded footage onto amateur videos. We rely on a In our work, we adapt features of these previous techniques to cap-
robust, low-dimensional representation of the visual style of each ture a large range of color palettes. We use a color transfer model
frame that can handle a wide variety of looks and contents. The core inspired by color grading software used by artists, e.g., Da Vinci
of our technique is based on a new filtering approach to temporal Resolve and SpeedGrade. It consists of a nonlinear luminance curve
consistency. We explain how to extend the concept of curvature-flow and affine remapping of the chrominance values. These transforma-
smoothing often used on 3D meshes to higher-dimensional function tions are applied in three luminance bands and for each user-defined
spaces. In our context, we apply this smoothing scheme in the space segment of the video.
of color transformations and we demonstrate that it successfully
produces temporally consistent results without degrading the video Temporally consistent video processing Extending image-based
content. Further, our final result is defined by a small number of pa- algorithms to video sequences is non-trivial, and is especially diffi-
 Input Per-frame color matching Temporal smoothing of transformations

T3
T2
Keyframe
Keyframe
T’1 T’2 T’3 T’4
T1 T2 T3 T4

Model Result
Representative model frames

Figure 3: Our color grading method works in two stages. In the first stage, we match each frame of the input video to one image out of a set of
representative model video frames. We estimate color transformations between these matching frames. In the second stage, we filter these
transformations using a novel approximate curvature flow technique. The basic idea behind this technique is to treat the set of transformations
as a curve in high-dimensional space and detect points of low curvature (i.e., keyframes). Interpolating the color transformations at these
keyframes produces a temporally coherent set of transformations, that when applied to the input video, results in a high-quality color graded
result. Video credits: “Transformers” (2007) c Paramount Pictures (model).

cult for the kind of videos we are targeting – clips captured by am- palette of M to I to create the color graded result O. Optionally,
ateurs under largely uncontrolled conditions, and characterized by a segmentation of the two videos into foreground and background
complex scenes, arbitrary motion, and changes in illumination and regions can be used for improved matching results. There are two
camera parameters. If these effects are not accounted for properly, simple approaches to this problem that seem viable at the outset.
the color graded results often exhibit artifacts such as high-frequency One is to estimate a single global color transfer function T that
flickering or low-frequency color drifts (Fig. 2). best matches the colors of the input video to those of the output
video, and apply that transfer function to every input video frame,
Paris [2008] analyzed an extension of Gaussian convolution to video i.e., Ot = T (It ). The other approach is to estimate a color transfer
streams, and used it to devise temporally-consistent video filter- function Tt for every input video frame It that maps its colors
ing techniques (including edge-preserving smoothing). Lang et to those of a corresponding model video frame, and apply it to
al. [2012] used efficient edge preserving filtering techniques to esti- produce the result, i.e., Ot = Tt (It ) . While these approaches
mate temporally coherent optical flow and used it to make graphics would work for some videos, they do not generalize well to the
applications such as colorization temporally coherent. Farbman and range of videos that we would like to handle – videos captured with
Lischinski [2011] stabilize tonal fluctuations in videos by using opti- arbitrary scene content, complex motion, and changes in lighting and
cal flow estimated at sparse locations to propagate tonal differences camera settings that lead to changes in pixel values over time. This is
across frames. Oskam et al. [2012] employ spatio-temporally consis- illustrated in Fig. 2, where using a single global color transform leads
tent radial basis functions to color balance sequences for augmented to drift in the colors over time (due to changes in the pose and the
reality. However, all these methods directly filter the pixel colors and camera exposure and white balance of the input video over time). On
would generate overly smooth results, which would be undesirable the other hand, color-matching every frame leads to high-frequency
in our context. We address this issue by working in the space of flickering (due to sudden changes in scene content).
color transforms. Our scheme preserves the frame details while
ensuring temporal smoothness.
Instead, we propose a novel two-stage approach, shown in Fig. 3,
Kiser et al. [2012] devise a temporally stable tone mapping system that is designed to handle these color inconsistencies. In the first step,
that relies on a leaky integrator to smooth out the parameters esti- we compute per-frame transformations Tt (Sec. 4) that match the
mated at every frame. Similarly, Chang et al. [2007] use anisotropic color palette of each input video frame to a corresponding represen-
diffusion to ensure that subtle color transformations are temporally tative model frame (Sec. 4.2). By estimating color transformations
coherent. In our context, low-pass filtering the color transformations at every frame, we capture the changes in the scene content of both
computed at every frame might smooth out noisy transformations, the input and model videos. Representative model frames summa-
but does so at the cost of corrupting the transformations at neigh- rize the important aspects of the content in the model video, and
boring frames, i.e., large variations in color distributions over short are chosen automatically so as to span the appearance space of
periods of time become smaller variations that are extended over the model. However, applying these transformations naively to the
a longer period. In our work, we aim at explicitly detecting and input video leads to temporal artifacts. To avoid this, in the sec-
removing the outlier color transformations; this ensures that the ond step (Sec. 5), we filter the per-frame transformations to make
results are temporally consistent and also accurately model the true them temporally coherent. We achieve this filtering through a novel
scene content. differential-geometry analysis of the series of color transformations
T as a curve in a higher-dimensional transform space. We show that
points at which this curve has a high curvature directly correspond
3 Overview to instants where directly applying the transformations Tt to the
input video It would produce artifacts such as temporal flickering.
Fig. 3 shows an overview of our process. Given a user-specified Instead, we detect points on this curve that have a low curvature
model video M and an input clip I, we seek to transfer the color (called keyframes) and interpolate the transformations at these points
 luminance histogram operator, respectively applied to I and M. The
generalized inverse [Villani 2003] (p.73) of H is used to account for
non-invertible cumulative distribution functions. To prevent extreme
histogram stretching in the cases where the two frames are widely
Input frame different or are noisy (as in the case of low-quality video input), we
regularize the transfer function by convolving it with a boundary-
Shadows Midtones Highlights preserving Gaussian kernel of standard deviation σ = 10%.
Luminance nonlinear transform Chrominance affine transforms
Model frame
Chrominance Our chrominance model is inspired by standard color
grading tools that let users control the color transform separately for
the shadows, midtones, and highlights. To allow for this, we separate
each frame into three luminance bands with an equal number of
Figure 4: Our set of transformations consists of a nonlinear lu- pixels. To avoid discontinuities, we make these bands overlap and
minance function and a set of affine chrominance transformations. smoothly transition from one to the other. We achieve this by making
These transformations can be computed for each segment of a seg- bands overlap by 10%, and setting pixels weights to 1 inside their
mented video if a segmentation is available. Video credits: “Trans- corresponding band with a cubic fallof outside. Finally, the per-pixel
formers” (2007) c Paramount Pictures (model). weights across all the bands are normalized to 1.
For each band, we model chrominance properties by the weighted
mean and covariance [Galassi et al. 2011] of the a and b chrominance
to build a set of temporally coherent color transformations T0 . By channels – a 2x1 vector and a 2x2 matrix, respectively. Given these
temporally smoothing out these color transformations, we filter out two properties, we estimate an affine transform from the model
high-frequency noise while still adapting to low-frequency changes video frame to the input video frame. The translation component
in the input video. Performing this analysis in a space of color trans- (a 2x1 vector) of this transform corresponds to the difference of
formations rather than at the pixel levels makes it computationally the means. The mapping between the covariances (a 2x2 scaling
efficient since we only filter a small number of numerical values. matrix) is computed in closed form using the technique of Pitié
Applying the interpolated transformations T0t to the input video and Kokaram [2007]. The advantage of this representation is that
frames It produces the final color graded output video frames Ot it is low-dimensional enough not to overly dependent on the scene
(Fig. 2, bottom). content, while still being flexible enough to capture a wide range of
effects, including color filters that are represented by the translation
component and sophisticated saturation variations represented by a
4 Single-frame Color Matching full 2x2 linear transformation in the ab plane.
Our method first computes per-frame color transformations Tt to Since the bands have smooth transitions, a pixel can be affected by
match each input frame It to a desired image or model frame Mt . In more than one chrominance transforms. To handle this, we cross-
this section, we first describe how we model color transformations fade the individual matrices using the pixel weights for each band to
and then describe how we pair each input frame with a model frame obtain the final pixel’s transformation matrix. Fig. 5 illustrates the
and actually estimate a per-frame color transform. advantage of using our color transfer model.

4.1 Color Transfer Model Foreground-background segmentation Some color grades rely
on a specific treatment of the foreground and the background. For
We seek to transfer the color distribution of a model video frame instance, Fig. 1b shows an example of a orange-teal look that em-
Mt to an input video frame It . A color transfer method that is phasizes the tension of the scene. Professional movie makers can
too flexible such as full color histogram matching would be highly achieve this effect by carefully setting up the shoot (props, costumes,
dependent on the footage content, e.g., for an outdoor scene, the lighting) and fine-tuning it in post-processing using global adjust-
proportion of sky between the input and the model would need to be ments. However, amateur videos are taken under uncontrolled light-
exactly the same for the transfer to be successful. Such sensitivity ing environments and global edits cannot produce such foreground-
is undesirable in our context since we are unlikely to get such good vs-background effects (Fig. 6). We enable these visual styles by
input-model matches. On the other hand, a transfer model without letting users specify a segmentation of the scene into foreground and
enough flexibility such as matching only the mean color would not background, and estimating the color transforms for these regions
be expressive enough to handle the visual looks that we are interested separately. In practice, we use the Snapcut tool [Bai et al. 2009] to
in. We strike a balance by first transferring the luminance values create a mask for the foreground. To reduce segmentation artifacts,
using a smooth remapping curve and then matching the chrominance we further refine this binary mask to produce an alpha matte. For
values using three affine transforms, one for each of shadows, mid- each frame, we erode and dilate the provided mask to determine
tones, and highlights (see Fig. 4). In cases where a segmentation an area of uncertainty around edges. This produces a tri-map that
into foreground and background objects is provided, we do this is used to build a soft mask for each frame [Levin et al. 2008].
transformation for foreground and background, respectively, for a We then process these masks with the edge-aware spatio-temporal
total of six transformations per frame. We finalize the per-frame filter of Lang et al. [Lang et al. 2012]. This creates a temporally
transfer with an edge-aware smoothing filter that ensures clean edges consistent matte that we use to adjust the luminance band weights
with no halos. For the color space, we follow the recommendation multiplicatively. We apply the color transform estimation technique
of Reinhard and Pouli [2011] and work in CIE-Lab and the D65 described above to construct luminance and chrominance transforms
illuminant. We describe the details of this model below. separately for the foreground and background.

Luminance We represent the tonal aspect of a frame by its intensity Spatial filtering Although refining the segmentation greatly con-
histogram. To transfer the tonal properties of the model onto the tributes to having clean edges, halos may still appear in some cases
input, we use standard histogram
matching, i.e., we use the transfer (Fig. 7, left). We follow an approach inspired by Rabin et al. [2010]
function T` = H −1 (I) H(M) , where H denotes the cumulative and Pouli and Reinhard [2011]. We compute the difference between
 (a) Model video frame (b) Input video frame (c) Naive histogram (d) Our color transfer
“Transformers” “Amélie” matching model

Figure 5: We would like to transfer the color palette of the model video frame (a) to an input video frame (b). (c) The simplest way to do this
would be to apply histogram matching in each color channel (and the foreground and background) independently. While this might match the
color styles, it often produces artifacts (note, the color artifacts on Amélie’s face). (d) Our color transfer model is able to produce an equivalent
result without any artifacts. Video credits: “Transformers”(2007) c Paramount Pictures (model), “Amélie”(2001) c Miramax Films (input).

(a) Model video frame (b) Input video frame (c) Color transfer (d) Color transfer
“D.O.A” without segmentation with segmentation

Figure 6: Some color styles have spatially-varying characteristics that cannot be replicated with global color adjustments. (a) For e.g., in this
example of the Film Noir style, the face is bright, while the background is darker, but the input frame (b) does not match this. Transferring this
look to the input video frame using global color adjustments does not replicate this style. (d) By using a user-specified segmentation (shown in
a and b), we can produce a rendering that is more faithful to this style. Video credits: “D.O.A”(1950) (model), Tilke Judd (input).

the original and transformed images, apply an edge-aware smooth- Ferradans et al. [2012] that relies on the means, µ, and covariances,
ing filter to this difference, and add this filtered difference back to Σ, of the color distributions to estimate the distance d(p, q)2 between
the original image. However, we observed color bleeding when we frames tp and tq as:
processed all three color channels and found that processing only
1 1 1
the luminance produced better results. In practice, we chose the Do-
X
tr(Σkp + Σkq ) − 2(Σkp 2 Σkq Σkp 2 ) 2 + kµkp − µkq k2 , (1)
main Transform filter [Gastal and Oliveira 2011] over the bilateral k
filter [Pouli and Reinhard 2011] and the Yaroslavsky filter [Rabin
et al. 2010] because of its speed. where the summation is over the three luminance bands. We remove
outliers by ignoring medoids with less than 30 samples, which repre-
4.2 Representative model frames sent less than a second at 30fps. Given the representative frames, we
match each input frame to the representative frames using the same
metric, and estimate the corresponding color transformation. This
Our approach supports several options for specifying the model data. process is illustrated in Fig. 8, where using an arbitrary model video
Users can provide a single model image, or for more control, specify frame produces a bad color transfer, but using our automatically
model images for several hand-picked input video frames. That said, estimated match improves the quality of the result substantially.
we believe that most amateur users would find it burdensome to
painstakingly specify these correspondences. Instead, we present a
technique that only requires users to provide an input and a model 5 Differential Color Transform Filtering
video, and automatically pairs each input video frame with a model
video frame. After the previous stage, we have a color transform Tt for each
frame. Each of these transformations has been computed in isolation
First, to avoid doing a computationally expensive exhaustive search and applying them directly to the video frames produces a temporally
among all the model frames for each input frame, we summarize inconsistent result. For instance, in Fig. 2, the colors of the wall
the model sequence with a few representative frames whose color and of the person in the foreground change when a passer-by briefly
distribution is representative of that of the entire video. We perform walks across the background, even though this is an momentary
this clustering using the K-medoids clustering technique of Park and event that ideally should not affect the result. One could apply
Jun [2009]. We chose K-medoids over standard K-means because a spatio-temporal filter to the video pixels to average this out but
medoids are constrained to be one of the input samples, which fulfills this would produce a smooth result in which high-frequency details
our goal of selecting frames from a video sequence. In practice, we are lost. We address this issue by temporally filtering the color
run the K-medoids algorithm on the frames of the model video to transforms applied to video frames. This ensures that both the
produce one representative model frame for every 30 frames of the temporal coherence observed in the input sequence and the spatial
model video, i.e., one for every second of a model video sampled at details in the video frames are preserved. However, this poses a
30fps. We refer to the original K-medoids article for the details of major challenge: the space of color transforms is high-dimensional,
the algorithm. To compare frames, we use the metric proposed by and extending the notion of a filter to this domain is nontrivial. An
 (a) Input video frame (b) Color transfer without (c) Color transfer with
spatial filtering spatial filtering

Figure 7: Applying color transformations to the foreground and background independently can lead to artifacts. (b) This is illustrated for this
example, where applying the Film Noir style (Fig. 6a) to the input video frame (a) produces haloing artifacts at segment boundaries. (c) Using
an edge aware filter to smooth the difference improves the quality of the result. Video credits: Markus Wills (input).

(a) Input video frame (b) Bad model frame (c) Color transfer with (d) Good model frame (e) Color transfer with
“The Dark Knight” bad model match “The Dark Knight” good model match

Figure 8: To ensure that the per-frame color transforms computed are meaningful, it is important that input video frames are matched to
meaningful model video frames. When an input video frame (a) is color graded with a bad model frame (b), the results are often bad (c).
Instead, in our work, we pre-cluster the model video into a small number of representative model frames and match each video frame to the
most similar representative frame (d). This matching produces a meaningful color graded result (e). Video credits: Akash Viren Prasad (input),
“The Dark Knight” (2008) c Warner Bros. (model).

Tb and a C++ implementation. For a detailed presentation of differential

geometry please see books such as do Carmo [1992] and Amari and
Nagaoka [2000].
Tb(Ib)
Ta=T’a T’b
Tc=T’c A curvature-based filter Our filter is inspired by the curvature flow
filter defined on surfaces in 3D [Ilmanen 1995]. The observation
behind this work is that adding noise to an otherwise smooth surface
generates points with a high local curvature. The curvature-flow filter
moves these points in a direction that reduces their local curvature,
which effectively smooths the surface. In our context, we analyze
the color transforms, T estimated in the previous section, as a 1D
T’a(Ia) T’b(Ib) T’c(Ic) curve in the space of color transforms. Rapid changes of colors
resulting from sudden changes in the scene (for e.g., when the
Figure 9: The per-frame color transformations Tt can be analyzed passer-by enters the frame in Fig. 2) correspond to frames where the
as points on a curve in a high-dimensional space (blue curve, green estimated transforms vary sharply, and consequently the curvature
dots). Points on this curve with a high curvature correspond to trans- of the transformation curve is high at these points. Conversely,
formations that cause temporal artifacts in the output video (for e.g., frames with a low curvature correspond to time instants when the
the background in the top right image brightens when a person walks transforms are stable making these frames good candidates to be
across the back). Our color transform filtering technique detects keyframes for estimating a smooth color transformation curve. Our
keyframes (purple circles) by sampling regions of low curvature and approach builds upon this observation. First, we extend the notion of
computes a smooth transform curve by interpolating the original curvature to the color transforms previously computed. Second, we
transformations at the keyframes (red dashed curve, green dots). find points of low curvature and select them as keyframes. Finally,
Applying the interpolated transforms to the input frames produces a we interpolate the color transforms at the detected keyframes to
temporally consistent result (bottom middle). compute a temporally smooth set of color transformations for the
entire sequence; this effectively replaces the high-curvature points
that lie between the keyframes with transforms interpolated from
the neighboring keyframes. Fig. 9 illustrates our approach.
important contribution of this work is to introduce a smoothing
filter that operates on color transforms. Our approach relies on
existing tools developed in the field of high-dimensional differential 5.1 Estimating the curvature
geometry. A comprehensive review of this field is beyond the scope
of this paper. Below, we provide an intuitive presentation of the We would like to define a notion of curvature for the 1D color
concepts involved as well as a short technical discussion. We refer transformation curve, T, in the space of color transformations, and
the reader to the supplemental material for additional information this requires tools from differential geometry. Readers might be
more familiar with the definition of curvature in 2D or 3D; while the vector for each subspace independently.
exact form of the analysis in the higher-dimensional color transform
space differs from the corresponding analysis in 2D, the high-level Each sub-component of the color transforms can be represented by
intuition remains the same. The curvature of the color transform a matrix (for e.g., the luminance scaling and translations are 1 × 1
curve is defined as the rate of change of the tangent to the curve. In matrices, the chrominance translations are 2 × 1 matrices, and the
order to compute this quantity, we need to define the gradient of the chrominance rotations are 2 × 2 symmetric matrices). In the rest of
color transform curve and a metric to estimate how fast the gradient this section, we consider the case of a single sub-component that is
varies. represented by a single matrix, and derive the form for its covariant
derivative vector. In addition, the subpace of the luminance and
A simple approach to implement our strategy is to work in a Eu- chrominance translations is Euclidean, and the covariant derivative
clidean space. This implies using a Euclidean metric, defining curva- corresponds exactly to the standard second derivative of each com-
ture as the magnitude of the second derivative of the color transform ponent of the translation vector with respect to time. As such, we
curve with respect to time, ||T̈||, and using linear interpolation to will focus on the luminance scaling and the chrominance rotation
interpolate the color transforms between keyframes. However, linear sub-components of the color transformations.
interpolation does not accurately model the rotations of the color We first express the components of our matrix in a particular basis of
wheel that are commonly used for color grading and looses correla- dimension d that gives us a parametrization of the space of transfor-
tions between color channels. Therefore, we use a Euclidean space mations. For instance, for the 2 × 2 symmetric chrominance rotation
only for the translational components of the color transformations. and scaling matrices, a natural basis consists of the basis matrices
We instead handle rotations and scalings via a Wasserstein space that {x1 = [ 10 00 ] , x2 = [ 01 10 ] , x3 = [ 00 01 ]}. Any 2 × 2 symmetric
accounts for their properties [Takatsu 2011] more accurately; for matrix can be decomposed as the weighted sum of these d = 3 basis
e.g., interpolation is modeled as a mass transport problem that blends matrices. We denote the ith components of the matrix Tt in this
rotations and scalings appropriately. A challenge when working in basis as T i (i = 1 . . . d).
such a non-Euclidean space is that standard formulas for curvature
need to be adapted. While the tangent to the curve is still defined as Obtaining the covariant derivative then requires a set of real co-
the first derivative Ṫ, the definition of its curvature is slightly more efficients called the Christoffel symbols, Γik,` , of the Wasserstein
involved since it needs to account for the curvature of the space metric that account for the space curvature [Amari and Nagaoka
itself. That said, while the exact definitions are different, the high- 2000], where i, k and ` range from 1 to the dimension d of the space
level intuition about curvature and interpolation remains the same; (d = 1, 2 or 3), defined in Eq. 3 and 4. Using these Christoffel sym-
interpolating transforms along a geodesic in this high-dimensional bols, we can compute the ith component of the covariant derivative
space produces a “flat trajectory”, i.e., a zero curvature path in the ∇Ṫ Ṫ as:
space of transforms generates a temporally coherent output that is X i
visually pleasing. ∇iṪ Ṫ = T̈ i + Γk,` Ṫ k Ṫ ` , i = 1 . . . d (2)
k,`
One notion of curvature of a curve is captured by the rate of change
of tangent vectors along the curve, and is evaluated using the covari- where Ṫ i and T̈ i are the first and second derivatives of the ith com-
ant derivative, ∇Ṫ Ṫ [do Carmo 1992] that generalizes the notion ponent of the transformations T in the basis xk w.r.t. time and are
of a directional derivative in Euclidean spaces to manifolds. Stan- approximated with standard finite differences.
dard curvature flow techniques operate on a modified version of the
covariant derivative – the second fundamental form [Ilmanen 1995] In the supplemental material, we show that for the 1D luminance
– but this does not account for all the variations we are interested scaling:
1
in. For instance, a non-constant speed variation along a geodesic Γ11,1 = − , (3)
results in a vanishing second fundamental form, and hence, zero 2s
curvature. Instead, we compute the curvature at time instant t, Kt , as where s is the luminance scaling at the current point on the curve.
the magnitude of the covariant derivative vector at that time instant,
In the 3D space in which the chrominance rotations and scalings are
i.e., Kt = ||∇Ṫ Ṫ||. represented, the Christoffel symbols are expressed by:
To compute the curvature, we need to define the covariant derivative 3
vector for the color transformation space. As described in Sec. 4.1, 1 X im ∂gmk ∂gm` ∂gk`
Γik,` = g ( + − ), i, k, ` = 1 . . . d,
our color transformation model consists of a non-linear luminance 2 m=1 ∂x` ∂xk ∂xm
mapping, and three rotations/scalings and translations for the chromi- (4)
nances. A meaningful metric in the infinite-dimensional space of where the partial derivatives can be seen as derivatives along the basis
nonlinear luminance transforms is out of reach. Instead, we approx- matrices xi . The coefficients gij are determined using Takatsu’s met-
imate the 1D luminance mapping by a 1D affine transformation ric [2011] – analogous to the Euclidean dot product of two vectors
(i.e., a scale and a translation) only for the purpose of estimating – as gij = g(mi , mj ) = tr (mi Tt mj ) where tr denotes the stan-
the transform curvature. As a result, the color transform space is dard matrix trace, Tt the current transformation, and mi (resp. mj )
a finite-dimensional space of dimension 17 (or 34 when using a is a 2 × 2 matrix that solves the equation xi = T mi Tt + T Tt mi .
segmentation) that is a Cartesian product of the multiple subspaces Once the coefficients gij have been computed, they define a 3 × 3
corresponding to each transform component. These 17 dimensions matrix: inverting this matrix produces a new matrix whose coeffi-
consist of two 1D spaces (luminance scaling and translation), three cients are conventionally denoted by g ij , as used in Eqn.4. More
2D spaces (chrominance translations for shadows, midtones and details on the terms introduced, and a closed form derivation of these
highlights), and three 3D spaces (one 2 × 2 symmetric matrix repre- coefficients and directional derivatives are provided in supplemental
senting the chrominance rotation and scaling for shadows, midtones, material, as well as in a sample C++ code.
and highlights). We use the property that the squared curvature
of the color transform curve in the high-dimensional space can be The curvature at time t, Kt , is finally computed as the square root
computed as the sum of the squared curvature for each individual of the sum of the squared norms of the covariant derivatives in
component in its appropriate subspace [do Carmo 1992]. This allows the individual subspaces. The squared norm in one subspace is
us to define the appropriate formulas for the covariant derivative computed as g(∇Ṫ Ṫ, ∇Ṫ Ṫ).
5.2 Approximate curvature flow

Having computed a curvature value, Kt , for each frame of the input

video, we use this information to drive a decimation process leading
to smoother transformations over time. Our method can be regarded
as an approximate curvature flow. Curvature flow is a mathematical
method that advects a manifold locally with a speed proportional to
its local curvature, resulting in a smoother manifold, and is often
used to smooth 2D surfaces using the surface mean curvature (the
trace of its second fundamental form). We mimic this behavior with (a) Input (b) Model (c) Result
an approximate, computationally simpler, method, that allows for
keyframe detection. We reduce the total curvature K of our curve Figure 12: Failure cases. (Top) The black and white initial frame
T by detecting segments of high curvature and replacing them by makes the covariance matrices singular; no color information can
(smoother) geodesics. To achieve this, we subsample the curve be recovered from the model frame apart from the mean. (Bottom)
by retaining transformations only in regions of low curvature, and Color grading using a non-relevant model frame produces a non-
interpolating the transformations in-between. meaningful result. Video credits: “Saving Private Ryan” (1998)
c Paramount Pictures, “The Dark Knight” (2008) c Warner Bros.
We first generate a set of r keyframes in areas of low curvature of the
transform curve by sampling the probability function corresponding
to the inverse of the curvature, i.e.,
X produces artifacts even when scene content is varying at low fre-
p(t) = Ki /Kt . (5) quencies. While our method does not always produce satisfactory
i results (see Sec. 7), we show a variety of successful transfers on a
wide range of input and model sequences. In particular, our method
This can done using inversion sampling [Devroye 1986], i.e., by is able to color grade videos that have already been stylized (see
generating a uniform set of samples and warping these samples using Fig. 11). We call this process “color re-grading”. Despite the strong
the cumulative density function of the probability function p(t). This stylization of these input videos, our method is able to capture the
produces more samples in areas of low curvature. We further locally color grade of the model video.
optimize these samples by moving them to the location of lowest
curvature in their neighborhood (15 frames of either side of the All our results used a single set of parameters and required little
current sample). We use one sample for every 30 frames of the input human intervention. Although a fine adjustment of our parameters
video. This typically results in r = 5 for a 5 second video at 30fps. can yield improved results, a fixed parameter set benefits amateur
users.
Once the keyframes have been detected, we interpolate the individ-
ual sub-components of T between each keyframe to produce the
desired smooth color transforms T0 . In particular, the symmetric 7 Limitations and future work
positive definite matrices representing the chrominance scalings are
interpolated using a closed-form interpolation that appropriately
follows geodesics in this Wasserstein space [Ferradans et al. 2012]. The semi-automatic segmentation [Bai et al. 2009] that extracts fore-
The non-linear luminance mappings are linearly interpolated result- ground and background – when necessary – still requires significant
ing in a displacement interpolation of the transformed luminance user input. While out of the scope of our paper, our method would
histograms [Bonneel et al. 2011]. Finally, the interpolated color benefit from future work in fully automated video segmentation and
transformations, T0 , are applied to the input video frames to pro- matting. Similarly, our method would benefit from a multi-label
duce the color graded output video frames. video segmentation, that would allow for better capturing spatial
variations in colors.
6 Results When either the chrominance matrices of the input videos are rank
deficient or luminance histograms are concentrated on a few bins,
We demonstrate color transfer results on a wide range of input our method can lead to visible artifacts. We show this behavior when
and model sequences. We show a subset of our results in Fig.10. trying to match a grayscale sequence to a colored one in Fig. 12 (first
However, temporal consistency can not be evaluated on still images row). Such transfer cannot be achieved by our method: we refer the
and we encourage interested readers to refer to the supplemental reader to video colorization techniques [Yatziv and Sapiro 2006] for
material and video for additional video color transfers results. such applications. In addition, if the content of the model sequence
A typical 4-second (108 frames) HD segmented sequence can be differs significantly from the input sequence, our method lacks the
matched in approximately 3 minutes on an 8-core machine. This semantics that would be needed to yield a meaningful result. For
consists in 1min 30s for colorspace conversions, 20 seconds for the instance, Fig. 12 (second row) shows that matching the white joker’s
per-frame matching (first step), 6 milliseconds for the smoothing face to a neutral face without makeup results in color variations
step and 21 seconds for the spatial luminance smoothing. Obtaining on the resulting face that are not what would be expected from an
a spatio-temporally consistent matte from the user-provided binary appropriate color grade. In this case, the resulting sequence does not
segmentation requires 1.5 additional minutes using our unoptimized succeed in capturing the mood of the model.
implementation of Lang et al. [2012] and 3 minutes for the authors’
implementation of Levin et al. [2008]. Our curvature computation can be used to locally optimize user-
defined keyframes. While the benefits of a local optimization re-
While hand-tuned naive methods occasionally work, this is not mains to be demonstrated, it could lead to better user interfaces for
always the case. For instance, Fig 2 shows that per-frame color processing video sequences. We also believe that our approximate
matching produces artifacts when scene content is changing at high curvature flow filtering technique could be used for a number of
frequencies while a single-frame color matching applied to all frames other applications in video processing including tonal stabilization.
 (a) Input Video Frame (b) Color graded to “D.O.A” (c) Color graded to “300” (d) Color graded to “Saving
(Film Noir) (sepia tones) Private Ryan” (bleach-bypass)
c Warner Bros c Paramount Pictures

Figure 10: Our color grading method can successfully transfer a variety of color styles to a range of input video sequences. Here we
demonstrate this for three such styles – sepia tones, Film Noir, and bleach-bypass – characterized by the films “300”, “D.O.A”, and “Saving
Private Ryan” respectively. In addition to capturing the color palette of the model videos, our color graded results are temporally consistent.
Please refer to the accompanying video and supplemental material to evaluate the temporal consistency and to see more results. Video credits:
Tilke Judd (third row), “Tears of Steel”/The Blender Foundation (fourth row), ““300” (2006) c Warner Bros., “Saving Private Ryan” (1998)
c Paramount Pictures.
 (a) Color graded input (b) Re-graded to “D.O.A” (c) Re-graded to “300” (d) Re-graded to “Amélie”
video frame

Figure 11: Our technique can take stylized video clips and change their visual look. We call this process “color re-grading”. Here we show
results for re-grading (top to bottom) “Transformers” and “Saving Private Ryan” to the styles of the films (left to right) “D.O.A.”, “300”,
and “Amélie”. Our technique can handle transitions between subtle color grades (for e.g., “Saving Private Ryan”’to “300”) as well as more
dramatic color grades (for e.g., “Transformers” to “Amélie”). Please see the accompanying video and supplemental material for more results.
Video credits: “Saving Private Ryan” (1998) c Paramount Pictures, “Transformers” (2007) c Paramount Pictures, “300” (2006) c Warner
Bros., “Amélie”(2001) c Miramax Films.

8 Conclusion References

Color grading is a fundamental color management process that is A MARI , S., AND NAGAOKA , H. 2000. Methods of Information
used to alter or enhance the color palette of a movie to give it a Geometry, vol. 191 of Translations of Mathematical monographs.
characteristic visual look. We have presented a semi-automatic tool Oxford University Press.
that allows even casual users to color grade amateur video clips A N , X., AND P ELLACINI , F. 2010. User-controllable color transfer.
using existing example videos or images. Our tool is able to match Computer Graphics Forum 29, 2, 263–271.
the color palette of the model videos while ensuring that the output
color graded results are temporally consistent. To achieve tractable BAE , S., PARIS , S., AND D URAND , F. 2006. Two-scale tone
and principled temporal consistency, we have proposed an original management for photographic look. ACM Trans. on Graphics
technique that casts the temporal filtering as curve simplification (Proc. of ACM SIGGRAPH 2006) 25, 3, 637 – 645.
problem using differential geometry techniques. While we inherit
the limitations discussed in Sec. 7 from statistically-based image BAI , X., WANG , J., S IMONS , D., AND S APIRO , G. 2009. Video
color transfer techniques, we show that reasonable results can be SnapCut: robust video object cutout using localized classifiers.
obtained on a wide range of videos. We believe that our method In ACM Trans. on Graphics (Proc. of ACM SIGGRAPH 2009),
has the potential to benefit the increasingly large number of casual 70:1–70:11.
videos shared by amateurs on the internet while paving the way to a B ONNEEL , N., VAN DE PANNE , M., PARIS , S., AND H EIDRICH ,
differential geometry analysis of colors in videos. W. 2011. Displacement interpolation using lagrangian mass
transport. In ACM Trans. on Graphics (Proc. of ACM SIGGRAPH
Asia 2011), 158:1–158:12.
Acknowledgements
C HANG , Y., S AITO , S., AND NAKAJIMA , M. 2007. Example-
We thank Harvard Professor Siu-Cheong Lau for useful advice on based color transformation of image and video using basic color
curvature computation. We also thank PavelM/5pm, former contrib- categories. Image Processing, IEEE Trans. on 16, 2, 329–336.
utor on [Link], for detailed clarifications on Takatsu’s D EVROYE , L. 1986. Non-Uniform Random Variate Generation.
paper. We thank the authors of all the videos used to demonstrate the Springer-Verlag. Section 2.2. Inversion by numerical solution of
techniques in this paper. We also thank the SIGGRAPH reviewers F(X) = U.
for their helpful comments. This work was partially supported by
NSF CGV-1111415. DO C ARMO , M. P. 1992. Riemannian Geometry. Springer, Jan.
FARBMAN , Z., AND L ISCHINSKI , D. 2011. Tonal stabilization transfer. In 4th European Conference on Visual Media Produc-
of video. ACM Trans. on Graphics (Proc. of ACM SIGGRAPH tion, 2007. IETCVMP, 1 –9.
2011) 30, 4, 89:1 – 89:9.
P ITI É , F., KOKARAM , A. C., AND DAHYOT, R. 2005. N-
F ERRADANS , S., X IA , G.-S., P EYR É , G., AND AUJOL , J.-F. 2012. dimensional probablility density function transfer and its appli-
Optimal transport mixing of gaussian texture models. Tech. rep., cation to colour transfer. In Proceedings of the Tenth IEEE Int.
Preprint Hal-00662720. Conf. on Computer Vision - Vol. 2, ICCV ’05, 1434–1439.
G ALASSI , M., DAVIS , J., G OUGH , B., J UNGMAN , G., B OOTH , P OULI , T., AND R EINHARD , E. 2011. Progressive color transfer
M., AND ROSSI , F., 2011. GNU scientific library - reference for images of arbitrary dynamic range. Computers & Graphics
manual, version 1.15, sec. 21.7 weighted samples. 35, 1 (Feb.), 67–80.

G ASTAL , E. S. L., AND O LIVEIRA , M. M. 2011. Domain trans- R ABIN , J., D ELON , J., AND G OUSSEAU , Y. 2010. Regularization
form for edge-aware image and video processing. In ACM Trans. of transportation maps for color and contrast transfer. In 2010
on Graphics (Proc. of ACM SIGGRAPH 2011), 69:1–69:12. 17th IEEE Int. Conf. on Image Processing (ICIP), 1933 –1936.

H AC OHEN , Y., S HECHTMAN , E., G OLDMAN , D. B., AND R EINHARD , E., AND P OULI , T. 2011. Colour spaces for colour
L ISCHINSKI , D. 2011. Non-rigid dense correspondence with ap- transfer. In Proceedings of the Third international conference on
plications for image enhancement. ACM Transactions on Graph- Computational color imaging, Springer-Verlag, Berlin, Heidel-
ics (Proceedings of ACM SIGGRAPH 2011) 30, 4, 70:1–70:9. berg, CCIW’11, 1–15.

I LMANEN , T. 1995. Lectures on mean curvature flow and related R EINHARD , E., A SHIKHMIN , M., G OOCH , B., AND S HIRLEY,
equations. In Lecture Notes, ICTP, Trieste. P. 2001. Color transfer between images. IEEE Comput. Graph.
Appl. 21, 5 (Sept.), 34–41.
J OHNSON , M. K., DALE , K., AVIDAN , S., P FISTER , H., F REE -
S ELAN , J. 2012. Cinematic color: From your monitor to the big
MAN , W. T., AND M ATUSIK , W. 2011. CG2Real: improving
screen. Visual Effects Soc. Tech. Committee White Paper (Oct.).
the realism of computer generated images using a large collection
of photographs. IEEE Trans. on Visualization and Computer TAI , Y.-W., J IA , J., AND TANG , C.-K. 2005. Local color transfer
Graphics 17, 9 (Sept.), 1273–1285. via probabilistic segmentation by expectation-maximization. In
Computer Vision and Pattern Recognition, 2005. CVPR 2005.
K AGARLITSKY, S., M OSES , Y., AND H EL -O R , Y. 2009. Piecewise-
IEEE Computer Society Conference on, vol. 1, 747 – 754 vol. 1.
consistent color mappings of images acquired under various con-
ditions. In Computer Vision, 2009 IEEE 12th International Con- TAKATSU , A. 2011. Wasserstein geometry of gaussian measures.
ference on, 2311–2318. Osaka Journal of Mathematics 48, 4 (Dec.), 1005–1026. Mathe-
matical Reviews number (MathSciNet): MR2648273.
K ISER , C., R EINHARD , E., T OCCI , M., AND T OCCI , N. 2012.
Real time automated tone mapping system for HDR video. In V ILLANI , C. 2003. Topics in Optimal Transportation. Graduate
Proc. of the IEEE Int. Conference on Image Processing, IEEE. Studies in Mathematics Series. Amer Mathematical Society.
L ANG , M., WANG , O., AYDIN , T., S MOLIC , A., AND G ROSS , M. WANG , B., Y U , Y., AND X U , Y.-Q. 2011. Example-based image
2012. Practical temporal consistency for image-based graphics color and tone style enhancement. In ACM Trans. on Graphics
applications. ACM Trans. on Graphics (Proc. of ACM SIGGRAPH (Proc. of ACM SIGGRAPH 2011), 64:1–64:12.
2012) 31, 4 (July), 34:1–34:8.
X UE , S., AGARWALA , A., D ORSEY, J., AND RUSHMEIER , H. E.
L EVIN , A., L ISCHINSKI , D., AND W EISS , Y. 2008. A closed- 2012. Understanding and improving the realism of image com-
form solution to natural image matting. IEEE Trans. on Pattern posites. ACM Trans. on Graphics (Proc. of ACM SIGGRAPH
Analysis and Machine Intelligence 30, 2 (Feb.), 228 –242. 2012) 31, 4, 84:1–84:10.

M URRAY, N., S KAFF , S., M ARCHESOTTI , L., AND P ERRONNIN , YATZIV, L., AND S APIRO , G. 2006. Fast image and video col-
F. 2011. Towards automatic concept transfer. In Proc. of the orization using chrominance blending. IEEE Trans. on Image
Eurographics Symposium on Non-Photorealistic Animation and Processing 15, 5 (May), 1120 –1129.
Rendering, ACM, NPAR ’11, 167–176.
O LDENBORG , M. 2006. A comparison between techniques for color
grading in games. PhD thesis, University of Skövde, School of
Humanities and Informatics.
O SKAM , T., H ORNUNG , A., S UMNER , R. W., AND G ROSS , M.
2012. Fast and stable color balancing for images and augmented
reality. In 2nd Int. Conf. on 3D Imaging, Modeling, Processing,
Visualization and Transmission (3DIMPVT), 49 –56.
PARIS , S. 2008. Edge-preserving smoothing and mean-shift segmen-
tation of video streams. In Proc. of the 10th European Conference
on Computer Vision, ECCV ’08, 460–473.
PARK , H.-S., AND J UN , C.-H. 2009. A simple and fast algorithm
for K-medoids clustering. Expert Systems with Applications 36,
2, Part 2 (Mar.), 3336–3341.
P ITI É , F., AND KOKARAM , A. C. 2007. The linear monge-
kantorovitch linear colour mapping for example-based colour