Lane Line Detection using Computer Vision

Overview

When we drive, we use our eyes to decide where to go. The lines on the road
that show us where the lanes are act as our constant reference for where to
steer the vehicle. Naturally, one of the first things we would like to do in
developing a self-driving car is to automatically detect lane lines using an
algorithm.

In this project you will detect lane lines in images using Python and OpenCV.
OpenCV means "Open-Source Computer Vision", which is a package that has
many useful tools for analyzing images.

The tools we have are color selection, region of interest selection,

grayscaling, Gaussian smoothing, Canny Edge Detection and Hough
Tranform line detection. Our goal is piece together a pipeline to
detect the line segments in the image, then average/extrapolate
them and draw them onto the image for display. Once we have a
working pipeline, we will try it out on the video stream.

Source: Udacity

add Codeadd Markdown

Get the required Images and Videos

Credits: Udacity Self-Driving Car Nanodegree GitHub Repository

add Codeadd Markdown

!git clone [Link]

add Codeadd Markdown

from distutils.dir_util import copy_tree

import shutil

copy_tree("./CarND-LaneLines-P1/test_images", "./test_images")

copy_tree("./CarND-LaneLines-P1/test_videos", "./test_videos")

[Link]('./CarND-LaneLines-P1', ignore_errors=False, onerror=None)

add Codeadd Markdown

1. Color Selection

First let us select some colors. For Instance: Lane Lines are usually White in
color and we know the RGB value of White is (255,255,255). Here we will
define a color threshold in the variables red_threshold, green_threshold,
and blue_threshold and populate rgb_threshold with these values. This
vector contains the minimum values for red, green, and blue (R,G,B) that I
will allow in my selection.

add Codeadd Markdown

import [Link] as plt

import [Link] as mpimg

import numpy as np

# Read in the image

image = [Link]('test_images/[Link]')

# Grab the x and y size and make a copy of the image

ysize = [Link][0]

xsize = [Link][1]

color_select = [Link](image)
# Define color selection criteria

###### MODIFY THESE VARIABLES TO MAKE YOUR COLOR SELECTION

red_threshold = 200

green_threshold = 200

blue_threshold = 200

######

rgb_threshold = [red_threshold, green_threshold, blue_threshold]

# Do a boolean or with the "|" character to identify

# pixels below the thresholds

thresholds = (image[:,:,0] < rgb_threshold[0]) \

| (image[:,:,1] < rgb_threshold[1]) \

| (image[:,:,2] < rgb_threshold[2])

color_select[thresholds] = [0,0,0]

# Display the image

[Link](image)

[Link]("Input Image")

[Link]()

[Link](color_select)

[Link]("Color Selected Image")

[Link]()

# Uncomment the following code if you are running the code locally and
wish to save the image

# [Link]("[Link]", color_select)
add Codeadd Markdown

In the above output we can clearly see the lane lines

add Codeadd Markdown

2. Region Masking

I'll assume that the front facing camera that took the image is mounted in a
fixed position on the car, such that the lane lines will always appear in the
same general region of the image. Next, I'll take advantage of this by adding
a criterion to only consider pixels for color selection in the region where we
expect to find the lane lines.

Check out the code below. The variables left_bottom, right_bottom,

and apex represent the vertices of a triangular region that I would like to
retain for my color selection, while masking everything else out. Here I'm
using a triangular mask to illustrate the simplest case, but we can use a
quadrilateral, and in principle, we could use any polygon.

add Codeadd Markdown

import [Link] as plt

# Define color selection criteria

# MODIFY THESE VARIABLES TO MAKE YOUR COLOR SELECTION

red_threshold = 200

green_threshold = 200

blue_threshold = 200

rgb_threshold = [red_threshold, green_threshold, blue_threshold]

# Define the vertices of a triangular mask.

# Keep in mind the origin (x=0, y=0) is in the upper left

# MODIFY THESE VALUES TO ISOLATE THE REGION

# WHERE THE LANE LINES ARE IN THE IMAGE

left_bottom = [100, 539]

right_bottom = [950, 539]

apex = [480, 290]

# Perform a linear fit (y=Ax+B) to each of the three sides of the triangle

# [Link] returns the coefficients [A, B] of the fit

fit_left = [Link]((left_bottom[0], apex[0]), (left_bottom[1], apex[1]), 1)

fit_right = [Link]((right_bottom[0], apex[0]), (right_bottom[1], apex[1]),

1)

fit_bottom = [Link]((left_bottom[0], right_bottom[0]), (left_bottom[1],

right_bottom[1]), 1)

# Mask pixels below the threshold

color_thresholds = (image[:,:,0] < rgb_threshold[0]) | \

(image[:,:,1] < rgb_threshold[1]) | \

(image[:,:,2] < rgb_threshold[2])

# Find the region inside the lines

XX, YY = [Link]([Link](0, xsize), [Link](0, ysize))

region_thresholds = (YY > (XX*fit_left[0] + fit_left[1])) & \

(YY > (XX*fit_right[0] + fit_right[1])) & \

(YY < (XX*fit_bottom[0] + fit_bottom[1]))

# Mask color and region selection

color_select[color_thresholds | ~region_thresholds] = [0, 0, 0]

# Color pixels red where both color and region selections met

line_image[~color_thresholds & region_thresholds] = [9, 255, 0]

# Display the image and show region and color selections

[Link](image)

x = [left_bottom[0], right_bottom[0], apex[0], left_bottom[0]]

y = [left_bottom[1], right_bottom[1], apex[1], left_bottom[1]]

[Link](x, y, 'r--', lw=4)

[Link]("Region Of Interest")

[Link]()

[Link](color_select)

[Link]("Color Selection in the Triangular Region")

[Link]()

[Link](line_image)

[Link]("Region Masked Image [Lane Lines in Green]")

[Link]()

add Codeadd Markdown

Yeah! We've successfully detected the Lane Lines (Really ?)

Check the below test.

add Codeadd Markdown

import [Link] as plt

# Grab the x and y size and make a copy of the image

ysize = [Link][0]

xsize = [Link][1]

color_select = [Link](image)

line_image = [Link](image)

# Define color selection criteria

# MODIFY THESE VARIABLES TO MAKE YOUR COLOR SELECTION

red_threshold = 200

green_threshold = 200

blue_threshold = 200

rgb_threshold = [red_threshold, green_threshold, blue_threshold]

# Define the vertices of a triangular mask.

# Keep in mind the origin (x=0, y=0) is in the upper left

# MODIFY THESE VALUES TO ISOLATE THE REGION

# WHERE THE LANE LINES ARE IN THE IMAGE

left_bottom = [100, 539]

right_bottom = [950, 539]

apex = [480, 290]

# Perform a linear fit (y=Ax+B) to each of the three sides of the triangle

# [Link] returns the coefficients [A, B] of the fit

fit_left = [Link]((left_bottom[0], apex[0]), (left_bottom[1], apex[1]), 1)

fit_right = [Link]((right_bottom[0], apex[0]), (right_bottom[1], apex[1]),

1)

fit_bottom = [Link]((left_bottom[0], right_bottom[0]), (left_bottom[1],

right_bottom[1]), 1)

# Mask pixels below the threshold

color_thresholds = (image[:,:,0] < rgb_threshold[0]) | \

(image[:,:,1] < rgb_threshold[1]) | \

(image[:,:,2] < rgb_threshold[2])

# Find the region inside the lines

XX, YY = [Link]([Link](0, xsize), [Link](0, ysize))

region_thresholds = (YY > (XX*fit_left[0] + fit_left[1])) & \

(YY > (XX*fit_right[0] + fit_right[1])) & \

(YY < (XX*fit_bottom[0] + fit_bottom[1]))

# Mask color and region selection

color_select[color_thresholds | ~region_thresholds] = [0, 0, 0]

# Color pixels red where both color and region selections met

line_image[~color_thresholds & region_thresholds] = [9, 255, 0]

# Display the image and show region and color selections

[Link](image)

x = [left_bottom[0], right_bottom[0], apex[0], left_bottom[0]]

y = [left_bottom[1], right_bottom[1], apex[1], left_bottom[1]]

[Link](x, y, 'r--', lw=4)

[Link]("Region Of Interest")

[Link]()

[Link](color_select)

[Link]("Color Selection")

[Link]()

[Link](line_image)

[Link]("Output Image")

[Link]()

add Codeadd Markdown

Opps! We got a yellow line [undetected] here. Let's fix this up.

add Codeadd Markdown

3. Canny Edge Detection

Now we are applying Canny to the gray-scaled image and our output will be
another image called edges. low_threshold and high_threshold are
your thresholds for edge detection.

The algorithm will first detect strong edge (strong gradient) pixels above
the high_threshold, and reject pixels below the low_threshold. Next,
pixels with values between the low_threshold and high_threshold will be
included as long as they are connected to strong edges. The output edges
is a binary image with white pixels tracing out the detected edges
and black everywhere else. See the OpenCV Canny Docs for more details.

What would make sense as a reasonable range for these

parameters? In our case, converting to grayscale has left us with an 8-bit
image, so each pixel can take 2^8 = 256 possible values. Hence, the pixel
values range from 0 to 255.

This range implies that derivatives (essentially, the value differences from
pixel to pixel) will be on the scale of tens or hundreds. So, a reasonable
range for your threshold parameters would also be in the tens to hundreds.

As far as a ratio of low_threshold to high_threshold, John Canny himself

recommended a low to high ratio of 1:2 or 1:3.

We'll also include Gaussian smoothing, before running Canny, which

is essentially a way of suppressing noise and spurious gradients by
averaging (check out the OpenCV docs for GaussianBlur).
[Link]() actually applies Gaussian smoothing internally, but we
include it here because you can get a different result by applying
further smoothing (and it's not a changeable parameter within
[Link]()!).**

You can choose the kernel_size for Gaussian smoothing to be any odd
number. A larger kernel_size implies averaging, or smoothing, over a
larger area.

add Codeadd Markdown

# Do all the relevant imports

import [Link] as plt

import [Link] as mpimg

import numpy as np
import cv2

# Read in the image and convert to grayscale

# Note: in the previous example we were reading a .jpg

# Here we read a .png and convert to 0,255 bytescale

image = [Link]('test_images/[Link]')

gray = [Link](image,cv2.COLOR_RGB2GRAY)

# Define a kernel size for Gaussian smoothing / blurring

kernel_size = 5 # Must be an odd number (3, 5, 7...)

blur_gray = [Link](gray,(kernel_size, kernel_size),0)

# Define our parameters for Canny and run it

low_threshold = 180

high_threshold = 240

edges = [Link](blur_gray, low_threshold, high_threshold)

# Display the image

[Link](edges, cmap='Greys_r')

[Link]("Canny Edge Detection Image")

[Link]()

add Codeadd Markdown

4. Hough Transform and detecting Lane Lines

Hough Transform

In image space, a line is plotted as x vs. y, but in 1962, Paul Hough devised a
method for representing lines in parameter space, which we will call “Hough
space” in his honor.
In Hough space, I can represent my "x vs. y" line as a point in "m vs. b"
instead. The Hough Transform is just the conversion from image space to
Hough space. So, the characterization of a line in image space will be a
single point at the position (m, b) in Hough space. References Click Here

add Codeadd Markdown

# Read in and grayscale the image

image = [Link]('test_images/[Link]')

gray = [Link](image,cv2.COLOR_RGB2GRAY)

# Define a kernel size and apply Gaussian smoothing

kernel_size = 5

blur_gray = [Link](gray,(kernel_size, kernel_size),0)

# Define our parameters for Canny and apply

low_threshold = 180

high_threshold = 240

edges = [Link](blur_gray, low_threshold, high_threshold)

# Next we'll create a masked edges image using [Link]()

mask = np.zeros_like(edges)

ignore_mask_color = 255

# This time we are defining a four sided polygon to mask

imshape = [Link]

vertices = [Link]([[(0,imshape[0]),(450, 290), (490, 290),

(imshape[1],imshape[0])]], dtype=np.int32)

[Link](mask, vertices, ignore_mask_color)

masked_edges = cv2.bitwise_and(edges, mask)

# Define the Hough transform parameters

# Make a blank the same size as our image to draw on

rho = 1 # distance resolution in pixels of the Hough grid

theta = [Link]/180 # angular resolution in radians of the Hough grid

threshold = 2 # minimum number of votes (intersections in Hough grid

cell)

min_line_length = 4 #minimum number of pixels making up a line

max_line_gap = 5 # maximum gap in pixels between connectable line

segments

line_image = [Link](image)*0 # creating a blank to draw lines on

# Run Hough on edge detected image

# Output "lines" is an array containing endpoints of detected line segments

lines = [Link](masked_edges, rho, theta, threshold, [Link]([]),

min_line_length, max_line_gap)

# Iterate over the output "lines" and draw lines on a blank image

for line in lines:

for x1,y1,x2,y2 in line:

[Link](line_image,(x1,y1),(x2,y2),(255,0,0),10)

# Create a "color" binary image to combine with line image

color_edges = [Link]((edges, edges, edges))

# Draw the lines on the edge image

lines_edges = [Link](color_edges, 0.8, line_image, 1, 0)

lines_edges = [Link](lines_edges,vertices, True, (0,0,255), 10)

[Link](image)

[Link]("Input Image")

[Link]()

[Link](lines_edges)

[Link]("Colored Lane line [In RED] and Region of Interest [In Blue]")

[Link]()

add Codeadd Markdown

Congratulations! We've successfully detected Lane Lines from different

images using OpenCV. (No Deep Learning :-))

add Codeadd Markdown

5. Let's Make a Lane Detection Pipeline

1. Gray Scale

2. Gaussian Smoothing

3. Canny Edge Detection

4. Region Masking

5. Hough Transform

6. Draw Lines [Mark Lane Lines with different Color]

add Codeadd Markdown

import math

#print(left_line, right_line)

for slope, intercept in [left_line, right_line]:

#getting complete height of image in y1

rows, cols = [Link][:2]

 y1= int(rows) #[Link][0]

#taking y2 upto 60% of actual height or 60% of y1

y2= int(rows*0.6) #int(0.6*y1)

#we know that equation of line is y=mx +c so we can write it x=(y-c)/m

x1=int((y1-intercept)/slope)

x2=int((y2-intercept)/slope)

poly_vertices.append((x1, y1))

poly_vertices.append((x2, y2))

draw_lines(img, [Link]([[[x1,y1,x2,y2]]]))

poly_vertices = [poly_vertices[i] for i in order]

[Link](img, pts = [Link]([poly_vertices],'int32'), color = (0,255,0))

return [Link](image,0.7,img,0.4,0.)

#[Link](img,[Link]([poly_vertices],'int32'), True, (0,0,255), 10)

#print(poly_vertices)

def hough_lines(img, rho, theta, threshold, min_line_len, max_line_gap):

"""

`img` should be the output of a Canny transform.

Returns an image with hough lines drawn.

"""

lines = [Link](img, rho, theta, threshold, [Link]([]),

minLineLength=min_line_len, maxLineGap=max_line_gap)
 line_img = [Link](([Link][0], [Link][1], 3), dtype=np.uint8)

#draw_lines(line_img, lines)

line_img = slope_lines(line_img,lines)

return line_img

# Python 3 has support for cool math symbols.

def weighted_img(img, initial_img, α=0.1, β=1., γ=0.):

"""

`img` is the output of the hough_lines(), An image with lines drawn on it.

Should be a blank image (all black) with lines drawn on it.

`initial_img` should be the image before any processing.

The result image is computed as follows:

initial_img * α + img * β + γ

NOTE: initial_img and img must be the same shape!

"""

lines_edges = [Link](initial_img, α, img, β, γ)

#lines_edges = [Link](lines_edges,get_vertices(img), True,

(0,0,255), 10)

return lines_edges

def get_vertices(image):

rows, cols = [Link][:2]

bottom_left = [cols*0.15, rows]

top_left = [cols*0.45, rows*0.6]

 bottom_right = [cols*0.95, rows]

top_right = [cols*0.55, rows*0.6]

ver = [Link]([[bottom_left, top_left, top_right, bottom_right]],

dtype=np.int32)

return ver

add Codeadd Markdown

# Lane finding Pipeline

def lane_finding_pipeline(image):

#Grayscale

gray_img = grayscale(image)

#Gaussian Smoothing

smoothed_img = gaussian_blur(img = gray_img, kernel_size = 5)

#Canny Edge Detection

canny_img = canny(img = smoothed_img, low_threshold = 180,

high_threshold = 240)

#Masked Image Within a Polygon

masked_img = region_of_interest(img = canny_img, vertices =

get_vertices(image))

#Hough Transform Lines

houghed_lines = hough_lines(img = masked_img, rho = 1, theta =

[Link]/180, threshold = 20, min_line_len = 20, max_line_gap = 180)

#Draw lines on edges

output = weighted_img(img = houghed_lines, initial_img = image, α=0.8,

β=1., γ=0.)

return output
add Codeadd Markdown

Test our Algorithm Pipeline with different Images

add Codeadd Markdown

for image_path in list([Link]('./test_images')):

fig = [Link](figsize=(20, 10))

image = [Link](f'./test_images/{image_path}')

ax = fig.add_subplot(1, 2, 1,xticks=[], yticks=[])

[Link](image)

ax.set_title("Input Image")

ax = fig.add_subplot(1, 2, 2,xticks=[], yticks=[])

[Link](lane_finding_pipeline(image))

ax.set_title("Output Image [Lane Line Detected]")

[Link]()

add Codeadd Markdown

Test on Eastern Delhi Expressway

add Codeadd Markdown

fig = [Link](figsize=(20, 10))

image = [Link]('../input/test-img/Eastern Delhi [Link]')

image = [Link](image, cv2.COLOR_BGRA2RGB)

image *= 255/[Link]()

image = [Link](np.uint8)

ax = fig.add_subplot(1, 2, 1,xticks=[], yticks=[])

[Link](image)

ax.set_title("Input Image")

ax = fig.add_subplot(1, 2, 2,xticks=[], yticks=[])

[Link](lane_finding_pipeline(image))

ax.set_title("Output Image [Lane Line Detected]")

[Link]()

add Codeadd Markdown

6. Let's try with Video Stream [Yes! Real-time Lane Line Detection]

add Codeadd Markdown

Install moviepy package

add Codeadd Markdown

!pip install moviepy

add Codeadd Markdown

# Import everything needed to edit/save/watch video clips

from [Link] import VideoFileClip

from [Link] import HTML

add Codeadd Markdown

# !wget -O "test_videos/jaipurHighway.mp4" "[Link]

[Link]/videoplayback?
expire=1567196271&ei=DzBpXcq8Bs6k1gKvvp_IBg&ip=[Link]&i
d=o-AJgxIMoUEuKDGJuLvFDwCoVO-YFXJIv-63q28E5_-
tuR&itag=137&aitags=133,134,135,136,137,160,242,243,244,247,248,271,
278&source=youtube&requiressl=yes&mime=video/
mp4&gir=yes&clen=290548152&dur=605.233&lmt=1530245281069433&f
vip=6&keepalive=yes&c=WEB&sparams=expire,ei,ip,id,aitags,source,requir
essl,mime,gir,clen,dur,lmt&sig=ALgxI2wwRAIgK8XLsUt8Wep20jakCHAkIuLXv
ss_9xmrjYo1bLouuKQCIDkKYeQn5jmmtnOe1lUV2IxL0dFYnwu5Gs51cATjK38c
&title=&redirect_counter=1&rm=sn-
5hnes77e&req_id=3a264db9667ba3ee&cms_redirect=yes&ipbypass=yes&
mip=2409:4062:2396:e5fe:4403:883f:7067:3510&mm=31&mn=sn-gwpa-
ccpe&ms=au&mt=1567175087&mv=m&mvi=5&pl=36&lsparams=ipbypass
,mip,mm,mn,ms,mv,mvi,pl&lsig=AHylml4wRQIgEyH_i_cWEcUkxdxJmn23UlcB
FXEDgElDaFsMN2WPB8sCIQD5fWqZ19kCQTPncHL3uhVfmYkLGu9jDq7ZJ3Tljk
8b_w=="

add Codeadd Markdown

Test with Video Clip 1 [Solid White Lane Lines]

add Codeadd Markdown

white_output = './solidWhiteRight.mp4'

## To speed up the testing process you may want to try your pipeline on a
shorter subclip of the video

## To do so add .subclip(start_second,end_second) to the end of the line

below

## Where start_second and end_second are integer values representing the

start and end of the subclip

## You may also uncomment the following line for a subclip of the first 5
seconds

#clip1 = VideoFileClip("test_videos/jaipurHighway.mp4").subclip(50,60)

clip1 = VideoFileClip("test_videos/solidWhiteRight.mp4")

white_clip = clip1.fl_image(lane_finding_pipeline) #NOTE: this function

expects color images!!

%time white_clip.write_videofile(white_output, audio=False)

add Codeadd Markdown

Output Video

add Codeadd Markdown

HTML("""

<video width="960" height="500" controls>

<source src="{0}">

</video>

""".format(white_output))

add Codeadd Markdown

Test with Video Clip 2 [With Yellow Lane Lines]

add Codeadd Markdown

yellow_output = './solidYellowLeft.mp4'

## To speed up the testing process you may want to try your pipeline on a
shorter subclip of the video
## To do so add .subclip(start_second,end_second) to the end of the line
below

## Where start_second and end_second are integer values representing the

start and end of the subclip

## You may also uncomment the following line for a subclip of the first 5
seconds

##clip2 = VideoFileClip('test_videos/solidYellowLeft.mp4').subclip(0,5)

clip2 = VideoFileClip('test_videos/solidYellowLeft.mp4')

yellow_clip = clip2.fl_image(lane_finding_pipeline)

%time yellow_clip.write_videofile(yellow_output, audio=False)

add Codeadd Markdown

Output

add Codeadd Markdown

HTML("""

<video width="960" height="500" controls>

<source src="{0}">

</video>

""".format(yellow_output))

add Codeadd Markdown

Shortcomings

1. Hough Transform is fit for Straight Lines only but in reality curved
lane lines exists where this will fail.
2. There are many roads which don't have lane markings where this
will fail.

add Codeadd Markdown

References:

1. Udacity Self-Driving Car Nanodegree

2. Finding Lane Lines — Simple Pipeline For Lane Detection

add Codeadd Markdown

Thank you for your time. Feel free to drop your comments and
suggestions for further improvements and discussions