Learning to See in The Dark

Convert shorter exposure images to longer exposure images

Anita Verma, Aniket Gupta, Ajay Maurya, Abhinav Mishra
Dr. A.I.T.H. Awadhpuri Kanpur

(b) our result from

Figure 1.(a)Camera output with ISO raw data of (a)
380,900

Abstract: Low light imaging is a challenge of cameras but it also does not resolve the low SNR because of low
because of low photon counts and low signal to noise photon counts.[1]. We can increase SNR by many
ratio. Generally all camera supports low light imaging physical means like opening the aperture, extending the
but not all images result are good because they mostly exposure time and with the help of flash. But everyone
uses traditional pipeline. And generally short exposure has its own shortcoming. For example when we
images suffers from noise, while long exposure images increase the exposure time then blur will also increases
may induce blur and is often impractical. There are due to camera shake or object motion.
many different types of techniques like denoising, In the last several years, there have been significant
deblurring and image enhancement which are already improvements in modern digital cameras like its
proposed but all these techniques have a certain limits resolution, shutter speed, aperture or f-stop and
that they cannot give better result in low light. In this sensitivity. Regardless of these improvements, image
paper we have proposed a learning based pipeline quality in low-light conditions is still limited [2]. There
which uses Artificial intelligence to train his neurons are many types of noises present in a low light images
to improve the learning, which can see in extremely which is shown in fig 1. The problem of fast imaging in
low lighting. In this paper we are introducing a dataset low light is already known to computational
of raw short-exposure low light images to correspond photography community, but remains open. There are
with images of long exposure time. various proposed techniques for denoising, deblurring,
and enhancement of low-light images .The assumption
Keywords: SNR, CNN, CAN, Exposure time, U-net,
of all above proposed techniques is that the images are
Deep neural network
take in dim environment but in contrast to this we are
taking images in extremely low light like moon light
[Link] and with short exposure time. Here in case of
extremely low light and short exposure time the
traditional pipeline not work correctly.
Whenever we take images, we will definitely get the
noise but in case of low lighting it becomes more We propose a new image processing pipeline which
challenging and complex. To increase the brightness will process the image with extremely low light and
we uses high ISO but it also increases noise. To short exposure time which is based on data driven
improve it we applied Histogram Processing or scaling approach. . Mainly, we are using neural network to

XXX-X-XXXX-XXXX-X/XX/$XX.00 ©20XX IEEE

train the neurons to learn the deep image processing for Contrast Stretching: contrast stretching is a method of
extremely low light raw images data which includes improving an image by stretching the range of intensity
color transformation and color filling, noise reduction, values to spread histogram.
demosaicing and image enhancement.
All the existing methods are working on synthetic
dataset or without ground truth, But in our proposed
processing pipeline we are taking the raw dataset
Because raw dataset have more information in
comparison to synthetic dataset, which are captured by
Sony 7S ii and Fuji film X-T2. The resolution for Sony
is 4240X2832 and for Fuji film images it is
6000X4000. Therefore, we took a new dataset of raw
images captured with fast exposure in lowlight (moon
light, street light) environment. For every low-light
image we have a corresponding long-exposure high-
quality reference image. We systematically analyze key
elements of the pipeline and implements in our project.

Figure 3. Histogram equalization.

Point Processing: In point processing we started

proceeding images from a single pixel. In this T is a
1*1 operator and the new value of f(x, y) depends on
operator T and existing f(x, y).

Figure [Link] of Noise with their static probabilities.[2]

Image denoising: The problem of addressing from a
noisy image environment to a noise free image called
image denoising. There are any techniques like total
variation [36], sparse coding [9, 28], wavelet-domain
2. Related Work processing [33] including several different methods of
There are many techniques for image denoising and WDP, nuclear norm minimization [12], and 3D
enhancement of image. We are giving a short review of transform-domain filtering (BM3D) [7]. These
those existing techniques. techniques are used for image denoising in low-level
vision. All the above methods are basically based on
Image Enhancement: the image preparatory such as sparsity, low rank and
According to Math Works, Image enhancement is the smoothness.
process of modifying digital images so that the result Examples of wavelet-domain processing : BLS-
are more clear for display or further analysis. GSM in which we will transfer image signals to an
Following are methods of image enhancement alternative domain which the can be more easily
separated from the noise. Other method is dictionary-
Histogram Equalization: based method, which uses a sparse linear combination
Histogram equalization is a technique of adjusting of atoms including KSVD [3] for the approximating the
image intensities to magnify contrast using image noisy patch. We have currently available only one
[Link] method generally increases global best method for denoising which based on the iterative
contrast of images. algorithm that learns a dictionary on the noisy image at
hand, NLSC [4] in terms of quality of the results, but
Median and min/max filtering Image subtraction we get one drawback which is it demands long
Frequency domain method computation times. EPLL [5] and BM3D [6] are
examples of last method which based on global image
Image smoothing statistics or other image properties, such as self-
Neighbourhood Averaging similarities. Researchers including convolutional
networks [17, 41], trainable nonlinear reaction exposure image. The images were collected by two cameras
diffusion (TNRD) [6], stacked sparse denoising auto- (top and bottom). From left to right: ratio of exposure times
encoders (SSDA) [39, 1] all of these are based on deep between input and reference images, filter array, exposure
networks for denoising. All the existing methods have time of input image, and number of images in each condition
been evaluated on either without ground truth or
synthetic data, such as images with added photo
electronic (photon noise, thermal noise), Gaussian, Salt
&Paper noise. When we trained our pipeline at certain
noise levels, these data-driven methods will be able to
compete with state-of-the art classic techniques such as
BM3D and sparse coding. A very careful recent
evaluation with ground data (real data) found that
BM3D outperforms more recent techniques on raw-
images (real images) [32]. Joint denoising and
demosaicing uses the deep networks [15, 10] but these
methods have been evaluated only for synthetic Bayer
patterns and synthetic noise, rather than raw images
collected in extreme low-light conditions.
Figure [Link] dark indoor environment, with the help
of convolutional network we took images(extreme low-
light). The illuminance at the camera is < 0.1 lux. The
Sony α7S II sensor is exposed for 1/25 second.
(a)Image produced by the camera with ISO 380,900.
The image suffers from noise and colour bias. (b)
Image produced by our convolutional network applied
to the raw sensor data from (a).
Figure 4:Example of noise image dataset
Noisy image datasets:
Most of the available methods are evaluated on
synthetic data, such as clean images with added salt See-in-the-Dark Dataset:
&pepper or Gaussian noise. Here, we proposed The
RENOIR dataset [2] to canon denoising with real noisy The See-in-the-Dark (SID) dataset contains 5094 raw
images. Bursts of images have been used to reduce short exposure images, each with a corresponding long-
noise in low-light conditions [24] and available datasets exposure reference image. we keep multiple short-
do not contain reliable ground truth data and till now exposure images correspond to the same long-exposure
we did research on only synthetic data. According to reference image. For example: we apply the burst
literature [32], image pairs in the RENOIR dataset denoising methods on collected sequences of short-
exhibit spatial misalignment. We used Google HDR+ exposure images. Every image contains real imaging
dataset [14] in traditional pipeline for extreme low-light artefacts and useful for training and testing and image
or street light or moon light images. Aim of Darmstadt in the sequence is counted as a distinct low-light image,
Noise Dataset (DND) [32] address the need for real data in Here we take 424 distinct long- exposure reference
the denoising community with raw low light images and images in SID. Both indoor and outdoor images present
parallel ground truth. We therefore collect such a dataset in data set. Those images generally captured at night,
to support systematic reproducible research in this area. under moonlight or street called as outdoor. At the
camera, illuminance in the outdoor scenes is generally
Sonyα7S II Filter array Exposure # images between 0.2 lux and 5 lux. But indoor images are even
time (s) more darker than the outdoor. In indoor, images were
x-350 Bayer 1/10, 1/30 1230 captured in closed rooms with regular lights turned off
.In case of the indoor, illuminance at the camera scenes
x-210 Bayer 1/25 960 is generally between 0.03 lux and 0.3 lux. The
x-180 Bayer 1/10 560 exposure time for the input images (raw images) was
set between 1/40 and 1/20 seconds. The corresponding
Fujifilm X- Filter array Exposure # images reference (ground truth) images were captured with 100
T2 time (s) to 300 times longer exposure: i.e., 10 to 30 seconds.
x-350 X-Trans 1/30 699 For the reference images, exposure times are
necessarily long. Each and every scenes (images) in the
x-210 X-Trans 1/25 504 dataset are static. The dataset is summarized in Table 1.
Table 1: The See-in-the-Dark (SID) dataset contains 5094 Figure 2 a small sample is given. Here approximately
raw short-exposure images, each with a reference long- 10%data set taken randomly from validation set and
 2. A corresponding normal lighting scene or long-
exposure picture of the same scene as target.
The neural net is trained on a dataset containing
5094 raw short-exposure images and their
corresponding long-exposure images.
So if you want to train the network, you will have to
first click a photograph under normal lighting
conditions which will be used as a target variable to
Figure (a)&(b) obtain error by the network.
Next, you will have to click a low exposure
photograph of the same scene so that it looks dark. This
validation set and 20% randomly chosen from test sets. will be given as an input to the network while training.
There might be possibilities that long-exposure
reference images may still contain some noise, but the The pair of these two photographs will produce an
sagacity quality is very high for these images to serve (input, output) pair for the network upon which it will
as raw data(ground truth).So our target is aim to be trained to be used on low-light test images.
produce good images in low-light conditions or moon
light or street light, rather than exhaustively removing
all noise .

Network structure:
For this paper , we used U-net network architecture. u-
net is convolutional network architecture and used for
fast and precise segmentation of images(raw images).
Till now it has outperformed the prior best method .

Figure 6(a): Image Using BM3D Denoising+ Scaling

Figure 5.U-net architecture (example for 32x32 pixels in

the lowest resolution). Each blue box corresponds to a multi-
channel feature map. The number of channels is denoted on
top of the box. The x-y-size is provided at the lower left edge
of the box. White boxes represent copied feature maps. The Figure 6(b): Image After Processing Through CNN
arrows denote the different operations.
How is the CNN trained?
The CNN is trained on two sets of images.
1. A dimly lit (almost dark) scene or short-
exposure picture as an input.
Using the L1 loss and Adam optimizer [21], we train
the networks. Our aim is we train one network for each
camera. Under training, raw data of the short-exposed
image and the ground truth is the corresponding long- TRAINING:
exposure image in sRGB space (processed by libraw, a
raw image processing library) is the input to the 1. The dataset is split into training data and test
network. For both training and testing, the data.(1865 IMAGES)
amplification ratio is set to be the exposure difference 2. These raw images are splitted into color
between the input and reference images (e.g., x150, arrays.
x240, or x300) .For every iteration, we randomly crop 3. A Random input patches of size 512*512 are
or pick up a 512×512 patch for training and apply selected and then the augmentation is done.
random flipping and rotation for data augmentation. 4. These augmentation patches of images are sent
After 2000 epochs, the learning rate is reduced 10-2 as input into neural network and trained for
from the initial set to 10-4. Our Training proceeds near 400 epochs.
to 4000 epochs. 5. The network architecture used for training is
U-Net.
condition Sony fuji
Our default pipeline 24.44/0.676 25.52/0.562 6. The loss function used is mean squared error(
U-net → CAN 26.76/0.687 24.98/.0763 MSE). – 1/N ((output_ image – label_
Raw → sRGB 17.65/0.544 25.20/0.567
image)**2).
L1 → SSIM loss 28.64/0.817 23.05/0.577
28.47/0.784 22.54/0.547 7. Optimization Function: Adam’s optimizer
L1 → L2 loss
25.49/0.976 - with learning rate as 0.000.
Packed → Masked
- 19.88/0.645
X-Trans 3 × 3 → 6 × 6
Stretched references 17.97/0.459 18.87/0.4567 TESTING:

The results obtained from the trained model are

overwhelming. The time taken by the model to
produce an output of an image is about 4-5 seconds on
the system on which I trained my model (32GB RAM,
4GB Graphic card).
PREPROCESSING:
STRUCTURE OF PIPELINE:
1. GETTING RAW IMAGES: The input for
the neural network is sony RAW image. The RAW We take the raw data from an imaging sensor, so
images are Bayer RAW images. according to traditional image processing pipeline , it
2. SEPARATING IMAGE INTO will applies a sequence of modules and those modules
COLORED ARRAYS: The Bayer array is are like demosaicing ,denoising ,sharpening, white
spilt into its corresponding red, green, and balance, gamma correction ,color space conversion
blue image arrays. Black levels are subtracted and other modules. But all the available modules are
from the images and the Pixels are normalized often tuned for specific cameras. According to jiang et
in between 0 and 1. al. [18],he proposed that how we can use a large
3. TAKING PARTS OF THE IMAGE collection of learned (L3) filters and local ,linear to
INSTEAD OF WHOLE IMAGE: The approximate the complex non-linear pipelines which is
dimensions of the images are very found in modern consumer imaging systems. But still
large(approx. 3k*3k), so instead of taking the here neither the L3 pipelines nor traditional pipeline
entire image into the neural network at once, successfully deal with the fast low light imaging or
the input is divided into patches of length moon light imaging, even they are not capable to
512*512 or 1024*1024. handle the extremely low SNR. According to Hasinoff
4. DATA AUGMENTATION: The process et al.[14] described a burst imaging pipe line for
in which new training data is created by smartphones cameras. Method of hasinoff can produce
rotation ,translating and shifting the already good results by blending and aligning multiple images,
existing data. but it introduces certain level of complexity. But here
DATA AUGMENTATION is done to reduce we propose to use end-to-end learning for those
over fitting of the model. In this project ,data images who taken under fast low-light images.
augmentation is done by translating and So here we train a fully convolutional network
transposing the input matrix. FCN[22,25] to perform the entire image processing
pipeline. Rather than taking jpeg pictures into
traditional pipeline, we operate on raw sensor data and
give this raw data into our training based pipeline and
then we get the real result.

● ● ●

Ffff f
F
● ● ●

Figure 7(a) : In first, we take raw data and send it to traditional pipe. Now it perform the white
balance, Demosaic, Denoise, Sharpen, color space conversion, Gamma correctionon this raw
data and then it get result. In second, we take raw data and send it to L3. Now it perform the
pixel categorization, learning linear Transformations and weighted summation on raw data and
then it get result. In third, we take raw data and send it to Brust. Now it perform the Align
Merge, white balance, Demosaic, Denoise, local tone map and next this raw data and then it get
result.

Figure 7(b): Our result or traditional pipeline.

Result:we take a picture using a camera(sony 7S
II) under no low light or not under moon light
photography (low ISO), it will look like this-:

Figure 8(a): Traditional pipeline +scaling

Here we uses an end to end trained (our trained by
artificial intelligence) fully-Convolutional Network
which provide to us how the use of a dataset of raw
If we capture the same picture with a low light (moon short-exposure night-time(moon light or street light)
light or street light) camera like that in flagship images, with corresponding long-exposure reference
smartphones, the result would be like what is shown images. This provide to us better results from extreme
below. Notice here we use the scaling filters like scenarios like night photography very easy and
BM3D and lower ISO. So we are able to see how efficient as compared to deblurring techniques and
grainy this picture is? traditional denoising.

SOME OTHER MORE EXAMPLES:

Figure 9: this image captured at night time (too low light)

by the Fujifilm X-T2 camera with ISO 800, aperture is set
Figure 8(b) Scaling + BM3D Denoising to f/7.1, and exposure time is 1/30 second. The illuminance
of this camera is set to approximately 1 lux. (b) Processing
Now the next is our result by fully-convolutional the raw data (ground truth ) by a traditional pipeline does
network. It take the first image and processes it to not more effectively handle the color bias and noise in the
obtain the image below (Yes! I am not kidding.) data. (c) Our result obtained using our trained pipeline for
the same raw data.
References: [20] J. Kim, J. K. Lee, and K. M. Lee. Accurate image
[1] F. Agostinelli, M. R. Anderson, and H. Lee. superresolution using very deep convolutional networks. In
Adaptive multicolumn deep neural networks with CVPR, 2016. 4
application to robust image denoising. In NIPS, 2013. [21] D. P. Kingma and J. Ba. Adam: A method for
stochastic optimization. In ICLR, 2015. 5
[2] J. Anaya and A. Barbu. RENOIR – A dataset for
[22] Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E.
real lowlight image noise reduction. arXiv:1409.8230, Howard, W. Hubbard, and L. D. Jackel. Backpropagation
2014. 2 applied to handwritten zip code recognition. Neural
[3] H. C. Burger, C. J. Schuler, and S. Harmeling. Computation, 1(4), 1989. 4
Image denoising: Can plain neural networks compete with [23] C. Liu and W. T. Freeman. A high-quality video
BM3D? In CVPR, 2012. 2 denoising algorithm based on reliable motion estimation. In
[4] Q. Chen and V. Koltun. Photographic image synthesis ECCV, 2010. 2
with cascaded refinement networks. In ICCV, 2017. 6 [24] Z. Liu, L. Yuan, X. Tang, M. Uyttendaele, and J. Sun.
[5] Q. Chen, J. Xu, and V. Koltun. Fast image processing Fast burst images denoising. ACM Transactions on
with fully-convolutional networks. In ICCV, 2017. 4, 7 Graphics, 33(6), 2014. 1, 2, 5
[6] Y. Chen and T. Pock. Trainable nonlinear reaction [25] J. Long, E. Shelhamer, and T. Darrell. Fully
diffusion: A flexible framework for fast and effective image convolutional networks for semantic segmentation. In
restoration. IEEE Transactions on Pattern Analysis and CVPR, 2015
Machine Intelligence, 39(6), 2017. 2
[7] K. Dabov, A. Foi, V. Katkovnik, and K. Egiazarian.
Image denoising by sparse 3-D transform-domain
collaborative filtering. IEEE Transactions on Image
Processing, 16(8), 2007. 2, 5
[8] X. Dong, G. Wang, Y. Pang, W. Li, J. Wen, W. Meng,
and Y. Lu. Fast efficient algorithm for enhancement of low
lighting video. In IEEE International Conference on
Multimedia and Expo, 2011. 2
[9] M. Elad and M. Aharon. Image denoising via sparse and
redundant representations over learned dictionaries. IEEE
Transactions on Image Processing, 15(12), 2006. 2
[10] M. Gharbi, G. Chaurasia, S. Paris, and F. Durand. Deep
joint demosaicking and denoising. ACM Transactions on
Graphics, 35(6), 2016. 2, 7
[11] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D.
Warde-Farley, S. Ozair, A. Courville, and Y. Bengio.
Generative adversarial nets. In NIPS, 2014. 7
[12] S. Gu, L. Zhang, W. Zuo, and X. Feng. Weighted
nuclear norm minimization with application to image
denoising. In CVPR, 2014. 2
[13] X. Guo, Y. Li, and H. Ling. LIME: Low-light image
enhancement via illumination map estimation. IEEE
Transactions on Image Processing, 26(2), 2017. 2
[14] S. W. Hasinoff, D. Sharlet, R. Geiss, A. Adams, J. T.
Barron, F. Kainz, J. Chen, and M. Levoy. Burst
photography for high dynamic range and low-light imaging
on mobile cameras. ACM Transactions on Graphics, 35(6),
2016. 1, 2, 4, 5
[15] K. Hirakawa and T. W. Parks. Joint demosaicing and
denoising. IEEE Transactions on Image Processing, 15(8),
2006. 2
[16] Z. Hu, S. Cho, J. Wang, and M.-H. Yang. Deblurring
lowlight images with light streaks. In CVPR, 2014. 1
[17] V. Jain and H. S. Seung. Natural image denoising with
convolutional networks. In NIPS, 2008. 2
[18] H. Jiang, Q. Tian, J. E. Farrell, and B. A.
Wandell. Learning the image processing pipeline.
IEEE Transactions on Image Processing, 26(10), 2017.
[19] N. Joshi and M. F. Cohen. Seeing Mt. Rainier:
Lucky imaging for multi-image denoising, sharpening,
and haze removal. In ICCP, 2010. 2