1

TABLE OF CONTENTS

Topic Pg

1 Introduction 2

2 Abstract 3

3 Problem Statement 3

4 Objectives 4

5 Motivation 4

6 Methodology 5

7 Experiment

7.1 Image processing and Recoloring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

7.2 Simulation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

7.3 Recoloring Algorithm. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

7.4 Recoloring using GAN. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

8 Result and Discussion 12

9 Conclusion 13

10 Limitations 14

11 Future Scope 15

12 Team 16

13 References 17
 2

1. INTRODUCTION

1.1 Introduction to the topic

Among the many tasks the eye can perform, one of the eye’s key tasks is to aid in the
determination of the many variations of color. Color perception is one of the essential
functions of human vision. Color is a powerful feature that eases communication among
humans. The colors are detected based on the absorption of combinations of wavelengths
through the retina which allows the brain to perceive the different variations of color. The
cone cells present in the retina are responsible for identifying colours. Color blindness can
happen when one or more of the color cone cells are absent or not working. A person with
color vision deficiency (CVD) cannot distinguish one or more colors from red, blue, green, and
yellow.

CVD has different types and degrees of severity depending on the type and the number
of affected photoreceptor cells of the eye. It can be classified into three types namely,
anomalous trichromacy (Protanomaly, Deuteranomaly, and Tritanomaly), dichromacy
(Protanopia, Deuteranopia, and Tritanopia), and monochromacy. People with anomalous
trichromacy are ‘weak’ in perceiving a specific color whereas dichromats are a specific color
blind. People with monochromacy either perceive all colors in grayscale shades.
 3

Image Recoloring and GAN for CVD

2. ABSTRACT

People with color vision deficiency (CVD) cannot distinguish between colors due to the

damage of color reception nerves. In this work, I have presented the working of simulation

of images (color blind view) and implemented image enhancement approach that increases

the contrast and brightness of the image to assist colorblind people to identify the colors they

are not able to distinguish naturally. Used pix-pix GAN for implementing the recoloring

algorithm to produce new datasets. The image recoloring algorithm is based on equalizing

certain components in the image color space. The experimental results show that my

approach is useful for the recognition and separation of the CVD confusing colors in natural

scene images

3. PROBLEM STATEMENT

GAN for CVD

Creation of a data set for CVD since the data sets for CVD are very limited and setting up 2-3
GANs for recoloring images for the colorblind in an optimized manner. The optimization
function would involve identification of the confusing colors, identification of the
replacement color, and color compensation in optimized time.
 4

Image Recoloring and GAN for CVD

4. OBJECTIVE

My main oobjectives in this project are, to implement an existing recoloring algorithm

and try to produce better results with the same. To use general adversarial networks to

apply the recoloring algorithm so that it becomes easy for creating large, new datasets

and better recolored images.

5. MOTIVATION

As much as 5% of the population suffer from color vision deficiency, most of them being

men. Since there is no cure for this so far, the best thing that can be done is to help color

blind people in distinguishing the colors better and contributing to further research in this

field. Color vision deficiency can affect a person’s life well beyond simply being unable to

see specific colors. This issue becomes even more pertinent due to the increasing

availability of multimedia content in the computational environment. Aware of this

problem, a number of CVD research works came up in the literature in the past two

decades to improve color perception. Since the research had only begun recently the

algorithms are yet to be efficient enough and there is a lack of datasets in the field.
 5

Image Recoloring and GAN for CVD

6. METHODOLOGY

In brief, for proceeding with the project, initially, a detest of flowers containing 20,000

images was obtained from the website of the university of oxford. We made a new set of

1000 images which mostly contained red and green components since red blindness was

considered an example all throughout the project. Using this smaller dataset, we produced a

set of simulated images (protanope’s view). I have applied a slightly changed version of Huang

et al’s method to the simulated images and finally got the recolored images in which the

contrast and brightness were improved. The dataset of original images of flowers and the

final recolored images will be used to train the GAN that results in the production of better-

recolored images and also creates new examples from the existing ones.
 6

Image Recoloring and GAN for CVD

7. EXPERIMENT

7.1 Image processing and Recoloring

Image processing is the process of performing certain operations to get some useful
information from it. There are five main types of image processing:

 Visualization

 Recognition

 Sharpening and restoration

 Pattern recognition

 Retrieval

Image recoloring/enhancement can be considered as a part of the image processing

techniques. Image enhancement is the process of bringing out and highlighting certain
features of interest in an image that has been obscured. This process for color vision
deficiency mainly involves working with the intensity (contrast, brightness etc.) of the image
so that the colour-blind people can better distinguish between different colors.
 7

Image Recoloring and GAN for CVD

7.2 Simulation

Before getting into the implementation part of the recolouring algorithm we should first
be able to know how the color blind people view the images. This simulation is a very
important aspect to understand. For example, consider a person with red blindness
(protanopia) [hereafter read blindness will be considered for the examples] below is the
original image of a flower.

Original image

The following images shows how a red blind person would see the flower:

‘Coblis’(Online simulator) Coded

 8

Image Recoloring and GAN for CVD

In the simulation algorithm, initially, the RGB color space can be converted to the XYZ
color space using a transformation matrix MsRGB. This can then be converted from the XYZ
color space to the LMS color space using the Hunt-Pointer-Estevez transformation
matrix MHPE. In short, MsRGB and MHPE can be multiplied to calculate one transformation
matrix T for converting colors from the Linear RGB color space to the LMS color space.
Protanopes are missing L cones so lc=0. This modified color in LMS color space can then be
converted back to Linear RGB using T−1 and applying gamma correction.

7.3 Recoloring Algorithm

In the past two decades, a significant number of recoloring algorithms for CVD people
have been proposed. Recoloring methods must satisfy three perceptual requirements

• contrast enhancement

• color consistency

• color naturalness

in order not to disturb that much the perceptual learning of CVD people. Though these are
the important conditions, not all the methods satisfy these three properties. In fact, there
has been no method that satisfies all three conditions together. A specific recoloring method
is suitable only for a certain type of color vision deficiency.

The Recoloring algorithms can be broadly classified into the following categories based
on the type of color space used in the algorithm:

 LMS based
 9

Image Recoloring and GAN for CVD

►

 RGB based

 HSx (HSV, HIS, HSL) based

 CIE based

HSx color models (HSI, HSL, and HSV) are known as mind representations of colors; color
components are the hue (H), saturation (S), and brightness (i.e., intensity (I), lightness (L), or
value (V)). In the case of protanopy, the dichromat condition leads to a significant reduction
of the image contrast because of which a HSx-based algorithm can give better results.

Protanopes mainly confuse reddish and greenish colors. Thus, most HSx-based
methods focus on discriminating these confusing hues.

Huang et al.’s method-

This method applies to both anomalous trichromacy and dichromacy. It is a compensation

recoloring method by remapping the hue components in the HSV color space. Therefore, the
recoloring pipeline is as follows:

(R, G, B) → (H, S, V ) → (H’ , S,V ) → (R’ ,G’ , B’ ).

The remapping of the hue components in the HSV color space is performed using a histogram
transformation, which has the effect of enhancing the hue contrast. Histogram Equalization
is an image processing technique that adjusts the contrast of an image by using its histogram.
It stretches out the intensity range of the image.

The slight changes made to The Huang et al’s method are, the initial image was converted
into a BGR color space from the RGB color space before converting it into HSV color space.
Equalized ‘V’ component along with ‘H’ component for better brightness. Below are the
 10

Image Recoloring and GAN for CVD

results obtained from applying the recoloring algorithm on the simulated (protanope’s vies)
image.

Protanope's view [1] Recolored image [2]

Protanope’s view of the recolored image [3]

Image [1] is how a red blind image views the images of a red flower and finally after applying
the recoloring algorithm, the view of the color-blind person changes to the image [3]. Hence
it can be concluded the ultimate goal of the algorithm was achieved which was to increase
the contrast and also the brightness of the image.
 11

Image Recoloring and GAN for CVD

7.4 Recoloring using GAN

A generative adversarial network is a type of learning that involves automatically

discovering and learning the regularities or patterns in input data in such a way that the
model can be used to generate or output new examples that plausibly could have been drawn
from the original dataset. The GAN model architecture involves two sub-models:

 generator model: for generating new examples

 discriminator model: classifying whether generated examples are real, from the
domain, or fake, generated by the generator model.

The Cycle GAN used in the current project is a technique that involves the automatic
training of image-to-image translation models without paired examples. The models are
trained in an unsupervised manner using a collection of images from the source and target
domain that do not need to be related in any way.
Using the GAN architecture for recoloring images has been proven to produce
efficient results. First, a CVD simulation module is introduced and integrated with recoloring
algorithms. A color-to-gray module is added to eliminate the distortion. Multiple GANs i.e.,
pix2pix-GAN, Cycle-GAN, Bicycle-GAN are used for colorblind data conversion. These
datasets provide some help for color-blind image recoloring.
As shown in the following, the first row shows the original images, and the second
row shows the GAN generated recolored images.
 12

Image Recoloring and GAN for CVD

8. RESULTS AND DISCUSSION

An algorithm has been used to compare the recolored image produced from the recoloring

algorithm and the recolored image produced using GAN, assuming they have the same

dimensions and compares the two images pixel by pixel. This model is sensitive to a lot of

factors (operating system, GPU, drivers, …) and works best when testing is done on the

exact same machine with the exact same configuration. As shown below, we applied pixel

by pixel algorithm to Recoloring Algorithm generated images and GAN generated recoloring
 13

Image Recoloring and GAN for CVD

images, we got 35,847 pixel-wise differences for the first set of images and 39,132 pixel-

wise differences for the second set of images. The pixel difference can be decreased if we

increase the number of epochs for a given GAN algorithm, as we get better images.

9. CONCLUSION

For proceeding with this project, initially, a detest of flowers containing 20,000 images
was obtained from the website of the university of oxford. I made a new set of 1000 images
which mostly contained red and green components since red blindness was considered as an
example all throughout the project. Using this smaller dataset, I produced a set of simulated
images (protanope’s view). I have applied a slightly changed version of Huang et al’s method
to the simulated images and finally got the recolored images in which the contrast and
brightness were improved. The dataset of original images of flowers and the final recolored
images will be used to train the GAN that results in the production of better-recolored images
and also creates new examples from the existing ones.
 14

Image Recoloring and GAN for CVD

Hence it can be concluded that as expected, the images produced had high contrast
and improved brightness after applying the recoloring algorithm. Though the hue contrast
got better the color naturalness may not be held for all images, because the hue rotation may
result in an entirely different hue. In some cases, the color consistency may not be preserved
too, because the remapping of a given hue depends on the frequency (i.e., number of times)
of each hue in the image. The research on using general adversarial networks for color vision
deficiency had only begun as recently as 2 years back. Multiple GANs are being evaluated
against recoloring and it has been proven that using any GAN architecture can provide us with
better results.

10. LIMITATIONS

Research around this area about recoloring algorithms for the color-blind has only started in

the recent times and it still has not been extended extensively into algorithms for recoloring

videos and dynamic images. There are a smaller number of datasets available for researchers

to review any past research work or to test their algorithm in large numbers. There's no direct

tool to convert a large set of images into recolored images directly. Yet, there are no

recoloring algorithms that can be used for all types of color-blindness. Moreover, in the

available algorithms often there is a trade-off between preserving the naturalness and

contrast of the original image when it is recolored.

 15

Image Recoloring and GAN for CVD

11. FUTURE SCOPE

This research field on recoloring algorithms to aid people with color vision deficiency is still

in its early stage and hence has a larger scope in the near future. One of the main extensions

that I would like to work on is, applying recoloring algorithms to videos and not-still images

as most of the current algorithms only support recoloring of still images. Also, instead of just

adjusting and increasing the contrast in still images, I will try working with recoloring

algorithms that preserver the naturalness and intensity of the original image to produce

better results. Building a frontend tool where the user can upload desired set of original

images and obtain recolored images of the same, combined with newer and better recolored

images will be very useful for researchers who want to check the working of their algorithm

and also this leads to an increase in the number of datasets making it easy for reviewing the

past data in research.

 16

Image Recoloring and GAN for CVD

TEAM MEMBERS

1. Mohana Sravya Appalaneni

(Indian Institute of Information Technology, Pune)

2. Madan Eknath Dahiphale

(Government College of Engineering, Aurangabad)

3. Nishant Chaki

(Amity University of Kolkata)

4. Ashutosh Kosti

(Gyan Ganga Institute of Technology and Science)

5. Himanshu Bundela

(Gokhale Education Society's R. H. Sapat College of Engineering, Management Studies and

Research)
 17

Image Recoloring and GAN for CVD

REFERENCES

Recoloring Algorithms for Colorblind People: A Survey, ACM Computing Surveys,

2019-Madalena Ribeiro, Abel J. P. GOMES
Enhancing Color Representation for the Color Vision Impaired, Dans Workshop on
Computer Vision Applications for the Visually Impaired, 2008 - Jia-Bin Huang, Sih-Ying
Wu, and Chu-Song Chen
Improving Discrimination in Color Vision Deficiency by Image Re-Coloring, National
Library of Medicine, 2019 - Huei-Yung Lin, Li-Qi Chen, and Min-Liang Wang
Color vision deficiency datasets & recoloring evaluation using GANs, Springer, 2020 -
Hongsheng Li, Liang Zhang, Xiangdong Zhang, Meili Zhang, Guangming Zhu, Peiyi
Shen, Ping Li, Mohammed and Bennamoun, Syed Afaq Ali Shah
Recoloring Image for Color Vision Deficiency by GANs, IEEE 2019 - Xiangdong Zhang,
Meili Zhang, Liang Zhang, Peiyi Shen, Guangming Zhu, Ping Li
Coblis — Color Blindness Simulator
[Link]
[Link]
[Link]
[Link]
Google Scholar