1

23EC921-PYTHON PROGRAMMING FOR IMAGE

AND VIDEO PROCESSING

MODULE-1: PYTHON AND IMAGING

[Link] Name of the Topic Page Number

1 Image, Image Acquisition 2

2 Color Images and Video 8

3 Python libraries for Image 12

processing (NumPy, SciPy, scikit-image, PIL (Pillow),

OpenCV, scikit-learn, Matplotlib)

MODULE-1: PYTHON AND IMAGING

 2

Image, Image Acquisition:

What is an image and how it is stored on a computer

Conceptually, an image in its simplest form (single-channel; for example, binary or

mono-chrome, grayscale or black and white images) is a two-dimensional
function f(x,y) that maps a coordinate-pair to an integer/real value, which is related to the
intensity/color of the point. Each point is called a pixel or pel (picture element). An
image can have multiple channels too (for example, colored RGB images, where a color
can be represented using three channels—red, green, and blue). For a
colored RGB image, each pixel at the (x,y) coordinate can be represented by a three-
tuple (rx,y, gx,y, bx,y).

In order to be able to process it on a computer, an image f(x,y) needs to be digitalized

both spatially and in amplitude. Digitization of the spatial coordinates (x,y) is
called image sampling. Amplitude digitization is called gray-level quantization. In a
computer, a pixel value corresponding to a channel is generally represented as an integer
value between (0-255) or a floating-point value between (0-1). An image is stored as a
file, and there can be many different types (formats) of files. Each file generally has some
metadata and some data that can be extracted as multi-dimensional arrays (for example,
2-D arrays for binary or gray-level images and 3D arrays for RGB and YUV colored
images). The following figure shows how the image data is stored as matrices for
different types of image. As shown, for a grayscale image, a matrix (2-D array) of width x
height suffices to store the image, whereas an RGB image requires a 3-D array of a
dimension of width x height x 3:

MODULE-1: PYTHON AND IMAGING

 3

MODULE-1: PYTHON AND IMAGING

 4

The next figure shows example binary, grayscale, and RGB images:

In this book, we shall focus on processing image data and will use Python libraries to
extract the data from images for us, as well as run different algorithms for different image
processing tasks on the image data. Sample images are taken from the internet, from the
Berkeley Segmentation Dataset and Benchmark
([Link]
dataset/[Link]), and the USC-SIPI Image Database ([Link]
and many of them are standard images used for image processing.

What is image processing?

Image processing refers to the automatic processing, manipulation, analysis, and

interpretation of images using algorithms and codes on a computer. It has applications
in many disciplines and fields in science and technology such as television, photography,
robotics, remote sensing, medical diagnosis, and industrial inspection. Social networking
sites such as Facebook and Instagram, which we have got used to in our daily lives and
where we upload tons of images every day, are typical examples of the industries that
need to use/innovate many image processing algorithms to process the images we upload.

In this book, we are going to use a few Python packages to process an image. First, we
shall use a bunch of libraries to do classical image processing: right from extracting
image data, transforming the data with some algorithms using library functions to pre-
process, enhance, restore, represent (with descriptors), segment, classify, and

MODULE-1: PYTHON AND IMAGING

 5

detect and recognize (objects) to analyze, understand, and interpret the data better. Next,
we shall use another bunch of libraries to do image processing based on deep learning, a
technology that has became very popular in the last few years.

Some applications of image processing

Some typical applications of image processing include medical/biological fields (for

example, X-rays and CT scans), computational photography (Photoshop),
fingerprint authentication, face recognition, and so on.

The image processing pipeline

The following steps describe the basic steps in the image processing pipeline:

1. Acquisition and storage: The image needs to be captured (using a camera, for
example) and stored on some device (such as a hard disk) as a file (for example, a
JPEG file).

2. Load into memory and save to disk: The image needs to be read from the disk
into memory and stored using some data structure (for example, numpy ndarray),
and the data structure needs to be serialized into an image file later, possibly after
running some algorithms on the image.

3. Manipulation, enhancement, and restoration: We need to run some pre-

processing algorithms to do the following:

1. Run a few transformations on the image (sampling and manipulation; for

example, grayscale conversion)
2. Enhance the quality of the image (filtering; for example, deblurring)
3. Restore the image from noise degradation

MODULE-1: PYTHON AND IMAGING

 6

4. Segmentation: The image needs to be segmented in order to extract the objects

of interest.

5. Information extraction/representation: The image needs to be represented in

some alternative form; for example, one of the following:

1. Some hand-crafted feature-descriptor can be computed (for example, HOG

descriptors, with classical image processing) from the image
2. Some features can be automatically learned from the image (for example,
the weights and bias values learned in the hidden layers of a neural net with
deep learning)
3. The image is going to be represented using that alternative representation

6. Image understanding/interpretation: This representation will be used to

understand the image better with the following:

1. Image classification (for example, whether an image contains a human

object or not)
2. Object recognition (for example, finding the location of the car objects in
an image with a bounding box)

The following diagram describes the different steps in image processing:

MODULE-1: PYTHON AND IMAGING

 7

The next figure represents different modules that we are going to use for different image
processing tasks:

MODULE-1: PYTHON AND IMAGING

 8

Apart from these libraries, we are going to use the following:

 [Link] and opencv for different image processing tasks

 scikit-learn for classical machine learning
 tensorflow and keras for deep learning

Color Images and Video

Introduction
Digital images and videos are widely used in multimedia, medical imaging, surveillance,
autonomous vehicles, entertainment, and computer vision applications. While grayscale images
contain only intensity information, color images provide richer visual details by representing
colors through different color models. A video is a sequence of images (frames) displayed
rapidly to create the perception of motion.

MODULE-1: PYTHON AND IMAGING

 9

Python provides powerful libraries such as OpenCV, NumPy, Pillow (PIL), and Matplotlib for
color image and video processing.

Color Images
A color image is composed of multiple color channels. Each pixel stores color information
instead of a single intensity value.

Characteristics

 Contains three or more color components.

 Provides detailed visual information.
 Used in image analysis, object detection, and pattern recognition.

Representation

A color image can be represented as:

where:

 R = Red component
 G = Green component
 B = Blue component

RGB Color Model

RGB (Red, Green, Blue) is the most commonly used color model in digital imaging.

Features

 Additive color model.

 Each pixel contains three values (R, G, B).
 Each channel ranges from 0 to 255 in an 8-bit image.

MODULE-1: PYTHON AND IMAGING

 10

Color Examples
Color R G B

Red 255 0 0

Green 0 255 0

Blue 0 0 255

White 255 255 255

Black 0 0 0

Python Example
The objective of this program is to read an image using OpenCV, convert it from BGR color format to
RGB color format, and display it using Matplotlib.

import cv2
import [Link] as plt

img = [Link]('[Link]')
img_rgb = [Link](img, cv2.COLOR_BGR2RGB)

[Link](img_rgb)
[Link]("RGB Image")
[Link]()

Other Color Models

1. HSV Color Model

HSV stands for:

 H → Hue
 S → Saturation
 V → Value (Brightness)

MODULE-1: PYTHON AND IMAGING

 11

Advantages

 Separates color information from brightness.

 Useful for color segmentation and object tracking.

Conversion
hsv = [Link](img, cv2.COLOR_BGR2HSV)

2. Grayscale Conversion

A grayscale image contains only intensity information.

Formula
Gray=0.299R+0.587G+0.114BGray = 0.299R + 0.587G + 0.114BGray=0.299R+0.587G+0.114B

Python Example
gray = [Link](img, cv2.COLOR_BGR2GRAY)

3. YCbCr Color Model

Used in video compression and television broadcasting.

Where:

 Y = Luminance
 Cb = Blue chrominance
 Cr = Red chrominance

Applications

 JPEG compression
 MPEG video coding

Introduction to Video Processing

A video is a collection of frames displayed continuously at a specific frame rate.

MODULE-1: PYTHON AND IMAGING

 12

Components of Video

1. Frames
2. Frame Rate (FPS)
3. Resolution
4. Codec
5. Duration

Relationship

Video Formats
Common video formats include:

Format Extension
MP4 .mp4
AVI .avi
MOV .mov
MKV .mkv
WMV .wmv

Python libraries for Image processing

Setting up different image processing libraries in Python

The next few paragraphs describe to install different image processing libraries and set up
the environment for writing codes to process images using classical image processing
techniques in Python. In the last few chapters of this book, we will need to use a different
setup when we use deep-learning-based methods.

Installing pip

We are going to use the pip (or pip3) tool to install the libraries, so—if it isn't already
installed. we need to install pip first. As mentioned here

MODULE-1: PYTHON AND IMAGING

 13

([Link] pip is already installed

if we are using Python 3 >=3.4 downloaded from [Link], or if we are working in
a Virtual Environment
([Link]
environments) created
by virtualenv ([Link] or pyvenv (https://
[Link]/key_projects/#venv). We just need to make sure
to upgrade pip ([Link] How to
install pip for different OSes or platforms can be
found here: [Link]
python-3.

Installing some image processing libraries in Python

In Python, there are many libraries that we can use for image processing. The ones we are
going to use are: NumPy, SciPy, scikit-image, PIL (Pillow), OpenCV, scikit-
learn, SimpleITK, and Matplotlib.

The matplotlib library will primarily be used for display purposes, whereas numpy will
be used for storing an image. The scikit-learn library will be used for building machine-
learning models for image processing, and scipy will be used mainly for image
enhancements. The scikit-image, mahotas, and opencv libraries will be used for different
image processing algorithms.

The following code block shows how the libraries that we are going to use can be
downloaded and installed with pip from a Python prompt (interactive mode):

>>> pip install numpy

>>> pip install scipy
>>> pip install scikit-image
>>> pip install scikit-learn
>>> pip install pillow
>>> pip install SimpleITK
>>> pip install opencv-python
>>> pip install matplotlib

MODULE-1: PYTHON AND IMAGING

 14

Installation
Library Description
Command
A fundamental Python library for numerical computing. It
pip install
NumPy numpy provides support for large, multi-dimensional arrays and matrices,
along with mathematical functions for efficient computations.
A scientific computing library built on NumPy. It provides
pip install
SciPy scipy modules for optimization, integration, interpolation, signal
processing, statistics, and linear algebra.
An open-source image processing library. It provides algorithms
pip install
scikit-image scikit-image for image filtering, segmentation, feature extraction, edge
detection, image restoration, and image enhancement.
A machine learning library that provides tools for classification,
pip install
scikit-learn scikit-learn regression, clustering, dimensionality reduction, model evaluation,
and data preprocessing.
A Python Imaging Library used for opening, creating, editing,
Pillow pip install
pillow resizing, cropping, rotating, and saving image files in various
(PIL)
formats.
A simplified interface to the Insight Segmentation and
pip install Registration Toolkit (ITK). It is widely used for medical image
SimpleITK SimpleITK
processing, including image registration, segmentation, filtering,
and visualization.
An open-source computer vision library used for image
pip install
OpenCV opencv-python processing, object detection, face recognition, video analysis,
feature detection, and image transformations.
A plotting and visualization library used to create graphs, charts,
pip install
Matplotlib matplotlib histograms, scatter plots, and display images for data analysis and
scientific visualization.

There may be some additional installation instructions, depending on the OS platform

you are going to use. We suggest the reader goes through the documentation sites for
each of the libraries to get detailed platform-specific installation instructions for each
library. For example, for the scikit-image library, detailed installation instructions for
different OS platforms can be found here: [Link]

MODULE-1: PYTHON AND IMAGING

 15

Also, the reader should be familiar with websites such as stackoverflow to resolve
platform-dependent installation issues for different libraries.

Finally, we can verify whether a library is properly installed or not by importing it from
the Python prompt. If the library is imported successfully (no error message is thrown),
then we don't have any installation issue. We can print the version of the library installed
by printing it to the console.

The following code block shows the versions for the scikit-image and PIL Python
libraries:

>>> import skimage, PIL, numpy

>>> print(skimage.__version__)
# 0.14.0
>>> PIL.__version__
# 5.1.0
>>> numpy.__version__
# 1.14.5

Installing Jupyter Notebook

We are going to use Jupyter notebooks to write our Python code. So, we need to
install the jupyter package first from a Python prompt with >>> pip install jupyter, and
then launch the Jupyter Notebook app in the browser using >>> jupyter notebook. From
there, we can create new Python notebooks and choose a kernel. If we use Anaconda, we
do not need to install Jupyter explicitly; the latest Anaconda distribution comes with
Jupyter.

More about running Jupyter notebooks can be found at [Link]

[Link]/en/latest/[Link].

We can even install a Python package from inside a notebook cell; for example, we
can install scipy with the !pip install scipy command.

MODULE-1: PYTHON AND IMAGING

 16

For more information on installing Jupyter, please refer

to [Link]

MODULE-1: PYTHON AND IMAGING