Pothole Detection Using Machine Learning Models
Pothole Detection Using Machine Learning Models
ARTICLEINFO ABSTRACT
Potholes are damage caused to the ground by the formation of water and
Article History:
wear and tear over time. According to statistical data, bad road conditions
Accepted: 10 March 2024 account for about one- third of the total road accidents which has been
Published: 28 March 2024 increasing exponentially. Potholes have become so common that it has
become second nature for people to learn how to spot and avoid them,
which causes further accidents. The need of the hour is to build a
Publication Issue : dependable pothole detection system to accurately detect potholes and
Volume 11, Issue 2 warn the drivers and government officials in advance. The process to build
March-April-2024 such a system is divided into two steps i.e. collection of data and pothole
identification. The first step is achieved by taking the data from already
Page Number : available data sets on the Internet. The other step includes labeling the
94-105 potholes in the data set which is usually done manually. This paper focuses
mainly on Visual-based techniques to identify the best detection method
by comparing popular Machine Learning models and algorithms. The
obtained data set is trained using various transfer learning techniques like
You Only Look Once (YOLO)[1] and Single Shot Detector (SSD) [1]. Apart
from transfer learning, this paper also focuses on some proposed
techniques using Convolutional Neural Net- works (CNN) and
classification algorithms like Support Vector Machine (SVM)[21] to
identify and localize potholes. The actual size of potholes is calculated
using morphological operations, which is a just a straightforward
technique to analyze figures using set theory. To analyze every model and
find the best model, each model is trained on different sizes of data sets
and the obtained result is validated and examined by considering different
aspects like speed and accuracy in mind.
Index Terms : YOLO, CNN, Potholes, Data, Machine Learning,
Regression, Dilation, Erosion, Closing.
Copyright: © the author(s), publisher and licensee Technoscience Academy. This is an open-access article distributed under the
terms of the Creative Commons Attribution Non-Commercial License, which permits unrestricted non-commercial use,
94
distribution, and reproduction in any medium, provided the original work is properly cited
Rahul Dhingra et al Int J Sci Res Sci Eng Technol, March-April-2024, 11 (2) : 94-105
The number of occurrences of potholes increases This research topic holds an important place in the
rapidly in extreme weather such as snowfall and heavy com- munity of computer vision and hence there has
rain which eventually impacts traffic safety and road been a lot of contributions being made in this research
damage. Due to these bad road conditions, the number area. Below, is the literature overview which has been
of accidents has been increased in the last few decades. used in our research to estimate the pros and cons of
Automatic pothole identification methods are being the techniques been applied in the past by fellow
studied for efficient road repair and pavement research scholars. As proposed in [2], the novel
maintenance and have been proved to be an efficient approach of Support or State Vector Machines have
method to solve this problem. There are many been used for the detection of the potholes. The
solutions available with the advancement in both histograms of the pothole images were first created and
machine and deep learning methodologies. Some of the a boundary region of the image was then pulled out.
main solutions are divided into categories such as Once the image was extracted, SVM kernel algorithm
Vibration based strategy, 3D reconstruction based was then applied to locate the appropriate pothole
Strategy and Vision based Strategy. The vibration image. Using this algorithm the target set of images
based strategies are mathematical oriented and tend to were accurately located.[12]
use accelerometers to find a potential pothole by
estimating the force that could be applied on it. Furthermore, as in another research work [3]. Deep
Moreover, the 3d based strategy, uses some advanced learn- ing approaches and algorithms were used to
laser scanner devices to reconstruct and plot the all complete the same task. CNNs or as known as
new 3D images for the potholes. Furthermore, the Convolutional Neural Net- works were used for the
Vision based strategies are a bit expensive as they classification task. This algorithm proved to be better
utilize the real time images being captured from than previous approaches as this model was not at all
cameras. Using high definition cameras adds up to the affected by the noise and other redundant data which
overall cost which acts as one of the disadvantages for arise due to improper shadows and problems of
this technique. Apart from this the main advantage of illuminations in the images when they are
this method that distinguish it from other strategies is captured.[21]
that one doesn’t need to go over a pothole in order to
collect the data associated with that[6]. The efforts to use deep learning frameworks were
further done by Hiroya Maeda [4], as the research team
We decided to go with Visual based techniques as it created a software framework that was able to detect
was the cheaper option among the three, and there the issue of potholes using their respective mobile
were several data sets available [Link] current devices. Convolutional Neural Network algorithm was
Visual based techniques only deal with the detection used in this approach. The team made huge efforts to
and identification of the pothole on the street. As of collect this massive data and attempt was being made
now, there are not any research work being completed to estimate the speed as well as the accuracy with
that considers both the depth as well as the size of the which the detection was made possible by the system
pothole. These statistics are very important features developed.
and hence we included them in our research plan
which was done in addition to the detection of the
pothole data.
International Journal of Scientific Research in Science, Engineering and Technology | [Link] | Vol 11 | Issue 2 95
Rahul Dhingra et al Int J Sci Res Sci Eng Technol, March-April-2024, 11 (2) : 94-105
Apart from the traditional approach to use the CNNs III. DATA STATISTICS, PRE-PROCESSING AND
in the deep learning framework, some research ANALYSIS
scholars tried their hands on the Binary Classification
techniques in an attempt to predict whether the road We found a data set online that was created by the
image that was consideration belong to the pothole Elec- trical and Electronic Department, Stellenbosch
category or the flat normal road[17]. However, this University in 2015. The data set was collected by
framework required the feature extraction to be done clicking pictures on smartphones by setting it up on
for the images in advance before it was being entered the dashboard of a car. The entire data set consisted of
into the classification system. two different parts, one was a simple data set and the
Many research scholars also, created new networks other was more complex. Due to CPU constraints, We
and framework, such as Cracknet [6]. The unique decided to work on the simpler data set. Apart from
feature for this new neural network framework as that this data set, some pictures from Google Images and
the widely used pooling layer was not included in this various other available data sets were combined with
approach. The results were great from this approach this data set to make it more accurate and
and this new proposed system was very accurate and [Link], the appropriate measures were taken
efficient enough to detect the potholes as well as the to combine these data sets into one larger data set[17].
cracks along with the uneven surface patches on the To find the accuracy of our Machine Learning models,
images of roads[09]. we need a test data set. The final is divided into two
As the size of the image dataset increased, the demand parts i.e. training data set and test data set. These
for the automation picked up and many research folders are then subdivided into following 2 subfolders
scholars started using cheap sensors [7] along with 1) Positive- This data set contains images of roads that
traditional deep neural networks with CNN models to contain potholes 2) Negative - This data set contains
process automation of the crack hole detection. This images of roads that do not contain potholes
research work introduces a new approach which Since the class labels are important for the machine
eliminated the need to perform feature extraction and learning models, hence the image labels are to be
the feature learning was done in an automated pipeline. created either manually or by automated processes.
Additionally, A. Tedeschi [8] and his team also Rectangular boxes called the bounding boxes around
attempted to develop solution based out for Android the images were to be created manually for all the
mobile devices. This model was real time based and training data images. However, to accomplish this task
could detect the cracks and holes on the fly. manually, it posed to be a tedious procedure and
required the need for the tool. An open source widely
used tool, Labellmg was used to accomplish this class
International Journal of Scientific Research in Science, Engineering and Technology | [Link] | Vol 11 | Issue 2 96
Rahul Dhingra et al Int J Sci Res Sci Eng Technol, March-April-2024, 11 (2) : 94-105
labelling task for the potholes detection. A line was A. You Only Look Once (YOLO) Algorithm
dragged along the the required area of the pothole and
The single network, based neural frameworks are very
it made the tedious task quite simpler.[18] Once, the
fast and performance oriented algorithms. You look
drawing was completed for each image, an XML format
Only once, or YOLO is one such algorithm that
based file was created per image. The file had the
exploits the vectors being setup using bounding boxes
margin coordinates for the bounding area- that is both
of the images. This algorithm is particularly better
the bottom left as well as the top left. Once the XML
since they use single networks, not like the other
based files were created they were fed to the machine
algorithms that use a set of multi layered networks.
learning based model to perform the detection and
Using this approach, firstly the image is divided into
prediction task.
a huge grid with the size of SXS. in this new image,
each cell has an individual responsibility to find and
predict N boundary based boxes and hence find it’s
probability of it being a pothole. During analysis, it
was found that most of these boxes computed a very
low probability. A user defined threshold limit was
being setup and the boundary boxes were removed if
their probability was lower than this limit[9].
Once the elimination process is completed the rest
Fig. 1: Constructing of bounding boxes using LabelImg tool of the boxes that passes the probability test were then
moved towards the non max suppression mechanism.
IV. MACHINE LEARNING MODELS
The aim was to remove the duplicate and similar boxes.
YOLO is considered as the fastest algorithm by many
There has been a lot of research done in this area, and
research scholars and they consider the many benefits
it wasn’t hard to find object detection algorithms
of this framework, which made this an apt algorithm
online. After a little research, We decided to go with
You Only Look Once (YOLO) and Single Shot for this use case problem. With YOLO, the major
highlight was that it didn’t require any complex
Detector (SSD) since they were the fastest among all.
structure or pipeline unlike other frameworks and
Apart from transfer learning, We experimented with
creates a simple solution since the object detection is
the algorithm that we made using Convolutional
considered a regression problem.
Neural Networks (CNN) and Support VectorMachine
(SVM), which we will discuss later in this paper. During the processing stage for this algorithm, new
imageswere required and then these new images set to
make new predictions. The YOLO algorithm scans the
overall image and treats as a complete item package
before it computes the results for the predictions. Due
to this approach, the YOLO algorithm is considered to
be the fastest and most widely used algorithm in the
computer vision and object detection community[4].
When YOLO, is compared with the other relative
frame- works like Sliding window, it is considered
better in terms of accuracy as it doesn’t consider the
Fig. 2: Statictics scanning if the entire image during the training,
International Journal of Scientific Research in Science, Engineering and Technology | [Link] | Vol 11 | Issue 2 97
Rahul Dhingra et al Int J Sci Res Sci Eng Technol, March-April-2024, 11 (2) : 94-105
International Journal of Scientific Research in Science, Engineering and Technology | [Link] | Vol 11 | Issue 2 98
Rahul Dhingra et al Int J Sci Res Sci Eng Technol, March-April-2024, 11 (2) : 94-105
further divided into a cluster of small grids with balance out the contribution of the LL that is
equalsizes being assigned[17]. Location Loss Function parameter[4]
SSD has the major job of matching the class labels with Fig. 9: SSD results
the default boxes as rectangles that are dashed. A
match is reached when IOU value is more than 0.5 and Pros
each number being considered is mapped to a feature • This algorithm proves to be fastest running one
map. A VGG- when
• it was compared with the other three popular
models in the same setting and the same image
data set[9].
• The prediction results that are computed are very
accu- rate and the performance is great too since
the features that are extracted are being taken out
Fig. 8: Accuracy vs Dataset Size for SSD model of the pool of features with different scales.
Cons
16 architecture has the benefit of being performance • Small objects that being considered for the
drive architecture for classification tasks and is mostly analysis are very tedious for the detection and it
used in the problems associated with the transfer also bears out on very high cost for the
learning, hence SSD is used with this architecture for computation.
our analysis. Also, As per the figure shared above, SSD • When using SSD algorithm, performing the data
architecture does not use the fully connected layers aug- mentation is one of the key requirements and
when the VGG model is used[16]. In order to decrease is an important process[8].
the size of the inputs in some of the consecutive neural
network layers, some auxiliary CNN layers were added. C. Histogram of Oriented Gradients (HOG) with
These layers were able to pull out various features in Support Vector Machine:
many different scaling dimensions.
• To define the measure of the objectness of the The most important feature to differentiate objects in
bounding box being considered, a parameter an image is their size. Histogram of Oriented Gradients
called Confidence Loss or CL is used. (HOG) is an algorithm that is used to extract the main
• To measure the difference between the actual as features from an image to make the final algorithm
well as the predicted bounding box objectness faster and easy to process. Every input image contains
being predicted, a parameter measure called different objects and colors and a histogram is created
Location Loss or LL was used. LF = Confidence for each orientation of the picture. It is the
Loss + (Alpha) * (Location Loss) where the (Alpha) directional change in the color and the intensity of the
is used as a balance parameter and is used to image. It is explained in the steps below:
International Journal of Scientific Research in Science, Engineering and Technology | [Link] | Vol 11 | Issue 2 99
Rahul Dhingra et al Int J Sci Res Sci Eng Technol, March-April-2024, 11 (2) : 94-105
International Journal of Scientific Research in Science, Engineering and Technology | [Link] | Vol 11 | Issue 2 100
Rahul Dhingra et al Int J Sci Res Sci Eng Technol, March-April-2024, 11 (2) : 94-105
International Journal of Scientific Research in Science, Engineering and Technology | [Link] | Vol 11 | Issue 2 101
Rahul Dhingra et al Int J Sci Res Sci Eng Technol, March-April-2024, 11 (2) : 94-105
To remove the noise in the converted image, dilation depth. To find the best morphological operation, we
was used which is explained below. experimented with a lot of kernel sizes and operations
• Edge Detection are the set of processes to identify and got the best results after using a 9x9 kernel using a
points in the image where the change in the single iteration. After applying these operation, we still
brightness is sharp and is not continuous. These did not have the exact size of the potholes and just had
points are put together into a set of lines called the size of potholes in pixels which clearly wasn’t going
edges. to solve the problem. To solve this issue, multiple
• Dilation process removes the extra unwanted regression can be used by passing the size of the
edges from the gray scale image. potholes in pixels and distance of the object from the
• Erosion shrinks the objects in the gray scale image. bottom of the image as inputs and get the final actual
• Thresholds were used to track the edges and if the size of the pothole.
value is low, this shows that there are more edges.
• In order to not destroy the original shape of the
image, Closing is used to increase the boundary of
the bright regions.
After the coordinates prediction, the image is
converted to a black and white image and it is clearly Fig. 19: Original Image
visible that the depth
VII. COMPARISON
Although YOLO and SSD have a lot in common,
we needed to find a single model that would best suit
the project. SSD, being the model with most number of
Fig. 18: Erosion process
layers has the advantage of the best accuracy which
also makes it the slowest among all. If we only talk
of the potholes can be defined by the maximum
about the accuracy, CNN is clealy out of the
number of continuous black pixels in the vertical
competition. The question arises, YOLO or SSD? Since
direction. Since an image is nothing but an array, a
all the work is needed to be done in real time, we
nested for loop was used to find the depth of the
need a model with faster processing power whichis
pothole. There were a lot refinements that were
why YOLO would outperform in that area As shown
needed to be made before finally calculating the
in
International Journal of Scientific Research in Science, Engineering and Technology | [Link] | Vol 11 | Issue 2 102
Rahul Dhingra et al Int J Sci Res Sci Eng Technol, March-April-2024, 11 (2) : 94-105
The difference in accuracy increases as the object sizes [1]. Anon, (2019). [online] Available at:
becomes smaller. For the smaller objects, the [Link] facts/ [Accessed
performance of the YOLO is the best and next comes 13 Mar. 2019].
SSD and then CNN. The table below shows the mean [2]. J. Lin, Y. Liu, ”Potholes detection based on SVM
average precision , frames per second and the GPU in the pavement distress image”, Appl. Bus. Eng.
time needed by different models. It is clear that the Sci, pp. 544-547, Aug. 2010.
training time needed for the SDD model is the highest. [3]. YoungJin Cha, Wooram Choi, Oral Bykztrk,
VIII. CONCLUSION AND LEARNINGS ”Deep LearningBased Crack Damage Detection
Using Convolutional Neural Networks”, 2017.
We decided to train the models using YOLO, SSD,
[4]. Hiroya Maeda, Yoshihide Sekimoto, Toshikazu
HOG and CNNs as they were the most reliable and
Seto, Takehiro Kashiyama, Hiroshi Omata, Road
robust object detection algorithms out there. we
Damage Detection Using Deep Neural Networks
worked with two already built models which were
with Images Captured Through a Smartphone, 4-
YOLO and SSD and created two of our own models
6-1 Komaba, Tokyo, Japan:University of Tokyo.
using HOG and CNNs. Out of all the models, the best
[5]. Justin Bray, Brijesh Verma, Xue Li, Wade He, ”A
suited model to solve this particular problemturned out
Neural Network based Technique for Automatic
to be YOLO with 82% accuracy. we noticed that as we
Classification of Road Cracks”, 2006
increased the data to train the model, the accuracy
International Joint Conference on Neural
increased which clearly indicates that the all the
Networks Sheraton Vancou- ver Wall Centre
models were data starved. There is possibility the
Hotel, July 16-21, 2006.
increase the accuracy even further. Due to the limited
[6]. Allen Zhang, Kelvin C. P. Wang, Baoxian Li,
size of the dataset, we were able to achieve the
Enhui Yang, Xianxing Dai, Yi Peng, Yue Fei,
accuracy of 82%.
Yang Liu, Joshua Q. Li, Cheng Chen, ”Automated
Pixel - Level Pavement Crack Detection on 3D
International Journal of Scientific Research in Science, Engineering and Technology | [Link] | Vol 11 | Issue 2 103
Rahul Dhingra et al Int J Sci Res Sci Eng Technol, March-April-2024, 11 (2) : 94-105
Asphalt Surfaces Using a Deep-Learning [14]. X. Yu, E. Salari, “Pavement pothole detection
Network”, Computer-Aided Civil and and severity measure- ment using laser imaging”,
Infrastructure Engineering, vol. 00, pp. 1-15, 2011 IEEE INTERNATIONAL CONFER- ENCE
2017. ON ELECTRO/INFORMATION
[7]. Lei Zhang, Fan Yang, Yimin Daniel Zhang, Ying TECHNOLOGY
Julie Zhu, ”Road Crack Detection Using Deep [15]. Yaqi Li, Christos Papachristou, Daniel Weyer,
Convolutional Neural Network”. “Road Pothole Detec- tion System Based on
[8]. A. Tedeschi, F. Benedetto, ”A real-time Stereo Vision”, NAECON 2018 - IEEE National
automatic pavement crack and pothole Aerospace and Electronics Conference
recognition system for mobile Android-based [16]. Alfandino Rasyid, Mochammad Rifki Ulil
devices”, Advanced Engineering Informatics, Albaab, Muhammad Fajrul Falah, Yohanes
vol. 32, pp. 11-25, 2017. Yohanie Fridelin Panduman, Alviansyah Arman
[9]. A. Sachdeva and A. Sachdeva, “YOLO - ’You Yusuf, Dwi Kurnia Basuki, Anang Tjahjono,
only look once’ for Object Detection explained,” Rizqi Putri Nourma Budiarti, Sritrusta
Medium, 26-Mar-2017. [Online]. Available: Sukaridhoto, Firman Yudianto, Hendro
[Link] Wicaksono, “Pothole Visual Detection using
o-you-only- look-once-for-object-detection- Machine Learning Method integrated with
explained-6f80ea7aaa1e. [Accessed: 13-Mar- Internet of Thing Video Streaming Platform”,
2019]. IEEE, 2017
[10]. M. Hollemans, Real-time object detection with [17]. Vinay Rishiwal, Hamshan Khan, “Automatic
YOLO. [On- line]. Available: pothole and speed breaker detection using
[Link] android system”,2016 39th International Con-
with- HYPERLINK vention on Information and Communication
"[Link] Technology, Electronics and Microelectronics
with-%20yolo/" HYPERLINK (MIPRO)
"[Link] [18]. Ya-Wen Hsu, Jau-Woei Perng, Zong-Han Wu,
with-%20yolo/"yolo/. [Accessed: 13-Mar-2019]. “Design and imple- mentation of an intelligent
[11]. Liu, Wei, Anguelov, Dragomir, Erhan, Dumitru, road detection system with multisensor
Scott, Cheng- Yang, Berg, A. C., and Reed, “SSD: integration
Single Shot MultiBox Detector,” [Link], 29- [19]. Sandeep Venkatesh, E. Abhiram, S.
Dec-2016. [Online]. Available: Rajarajeswari, K. M. Sunil Kumar, Shreyas
[Link] [Accessed: 13- Balakuntala, Nitin Jagadish, “An intelligent
Mar-2019]. system to detect, avoid and maintain potholes: A
[12]. Byeong-ho Kang , Su-il Choi, “Pothole detection graph theoretic approach” 2014 Seventh
system using 2D LiDAR and camera”, 2017 Ninth International Conference on Mobile Computing
International Conference on Ubiq- uitous and and Ubiquitous Networking (ICMU)
Future Networks (ICUFN) [20]. Sumit Srivastava, Ayush Sharma, Harsh Balot,
[13]. Sudarshan S. Rode , Shonil Vijay , Prakhar Goyal, “Analysis and Improve- ments on Current
Purushottam Kulkarni, Kavi Arya, “Pothole Pothole Detection Techniques” 2018
Detection and Warning System: Infras- tructure International Conference on Smart Computing
Support and System Design”, 2009 International and Electronic Enterprise (ICSCEE)
Conference on Electronic Computer Technology
International Journal of Scientific Research in Science, Engineering and Technology | [Link] | Vol 11 | Issue 2 104
Rahul Dhingra et al Int J Sci Res Sci Eng Technol, March-April-2024, 11 (2) : 94-105
International Journal of Scientific Research in Science, Engineering and Technology | [Link] | Vol 11 | Issue 2 105