DISSERTATION ON ELECTRICAL AND COMPUTER ENGINEERING 1

A Perception Pipeline for an Autonomous Formula Student Vehicle

Diogo Laranjeiro Amaro Serrão Morgado
Instituto Superior Técnico, Lisboa
[Link]@[Link]

Abstract—Autonomous vehicles need to accomplish all the to ensure the safe and reliable operation of the vehicle by
tasks a human would while driving a common vehicle, which being the source of information used by the path-planning
includes being able to perceive and comprehend the environment and decision-making algorithms that dictate what the vehicle
around the vehicle, in real-time and using a combination of
high-tech distance sensors and cameras. Following the trends should do or where it should go.
of the automotive industry world, Formula Student, one of the A crucial aspect for an autonomous vehicle to reach its full
Europe’s most established educational engineering competition, autonomous capabilities is the ability to operate the vehicle
introduced the Driverless class, which challenges students to build close to its limits of handling. When it comes to new technolo-
a high-performance autonomous race car. In such competitions, gies in the automotive industry, racing has often played a key
the tracks through which the autonomous car should navigate are
unknown and delimited by different colored cones. This thesis role in fuelling innovation and pushing cars to the limit of what
presents a perception pipeline for a Formula Student car that is possible. Autonomous racing has proven to be an essential
exploits the best features of each perception sensor to identify the platform to develop, test and validate new technologies under
cones and, using custom Convolutional Neural Networks, classify challenging conditions. It provides a unique opportunity to
their color. Furthermore, due to the simplicity of the classifiers test autonomous driving software such as redundant percep-
used and in order to mitigate possible misclassifications, it was
developed a tracking algorithm that chooses the cone’s color tion pipelines, failure detection algorithms and control in
by taking into account the color of previous observations. An challenging conditions. Competitions, by themselves, allow
ablation study was conducted on the methods used to identify researchers and developers to test the applicability of different
the cones and, both the classifiers and the tracking module, solutions and their robustness. Racing competitions, in par-
were evaluated individually. Results showed that the proposed ticular, present additional challenges related to computational
perception pipeline is able to accurately detect and classify the
cones in real-time, even when running the pipeline on CPU, speeds, power consumption, and sensing.
something that proved not to be possible using other deep
learning object detectors. A. Formula Student Competition
Index Terms—Cone detection, sensor fusion, classification, Formula Student1 is the Europe’s most established edu-
perception, convolutional neural networks, tracking, autonomous cational automotive engineering competition that challenges
driving, Formula Student.
university students from worldwide top universities to design,
manufacture, build and test electric and combustion race cars
I. I NTRODUCTION following a strict set of rules [12] that prioritize safety. These
students are then challenged to race their cars and compete
The increased safety in addition to an improved comfort
against other teams in international competitions. Backed
promised by autonomous vehicles leads the automotive indus-
by the automotive industry and by high-profile engineers,
try to use a big portion of their investments for the develop-
Formula Student Germany (FSG), one of the most reputable
ment of autonomous driving. When taking the human out of
competition organizers, introduced, in a constant effort to
the equation, the vehicle must be able to respond to unknown
follow the trends of the automotive industry world, a new
situations in order to ensure a reliable system. Therefore, it
competition class, the Formula Student Driverless (FSD) class.
needs to accomplish all the tasks a human would while driving
This is a new competition class that challenges students to
a common vehicle. This includes perceiving the environment,
build an high-performance autonomous race car. In this class,
estimating its position, predicting what other road users will
the autonomous student-developed prototypes must be able to
do, planning a trajectory accordingly and giving the correct
compete in different events fully autonomously without any
acceleration and steering input to follow that trajectory. The
human interaction and without prior knowledge of the track’s
investments made by the industry led us to the current state-of-
layout.
the-art in autonomous driving, with several companies already
In the FSD class, the track boundaries are delimited by
having their autonomous vehicle prototypes being tested on
blue cones on the left and yellow cones on the right, small
real-life scenarios without any human interference.
orange cones delimit stopping zones and big orange cones
A key part when dealing with any kind of autonomous mark timekeeping zones. The shape, dimensions and color
driving problem is being able to perceive and comprehend pattern of these cones is regulated by the FSG rules [12].
the environment around the vehicle, in real-time and using Since the layout of the track is unknown, in order to safely
a combination of high-tech distance sensors and cameras, navigate through the unknown environment the perception
combined with state-of-the-art perception algorithms. This is pipeline implemented in the prototype must be able to identify
still a very challenging task in autonomous vehicles due to
the extremely low acceptable error rate, as it is a crucial task 1 [Link] Accessible on July 30, 2021.
DISSERTATION ON ELECTRICAL AND COMPUTER ENGINEERING 2

the cones’ position and color using the data retrieved from the type of camera used, these implementations start by detecting
available perception sensors. Having detected the cones that the cones and estimating their correspondent color, either by
delimit the track, it is then necessary to compute a valid path using deep learning and state-of-the-art object detectors or
between the track boundaries, something that is done by the using more classical computer vision techniques. Regarding
path planning pipeline. Finally, in order to navigate through the deep learning techniques, some teams [14], [19] chose
the track using the computed path one needs to determine a to use well-known object detectors such as YOLOv2 [23],
speed target and a steering input, which is performed by the YOLOv3 [24] and their correspondent lightweight versions,
control pipeline. YOLOv2-tiny [22] and YOLOv3-tiny [1], respectively, while
others [28] opted to use custom object detectors, based on the
B. Motivation YOLOv3 architecture, to detect the cones, and a custom 7-
layer CNN to estimate their correspondent color. By training
Formula Student Técnico2 (FST) is the team that represents
these real-time and powerful object detectors to detect cones
Portugal and Instituto Superior Técnico (IST) in international
from four different classes (blue, yellow, orange and big
Formula Student competitions. FST has been developing pro-
orange), the teams are able to accurately get bounding boxes
totypes for this competition since 2001, with ten prototypes
around the detected cones, along with the confidence scores
developed so far and proven results in the most prestigious
for each detection.
competitions of Europe. In 2019 the team, with more than
In order to estimate the 3D position of the detected cones,
60 members from different engineering courses, decided to
there is a consensus that, for monocular setups, using a key-
embrace a new challenge: empower the previous electric
point regression alongside the Perspective-n-Point (PnP) [11]
prototype (FST09e) with autonomous driving capabilities and
algorithm is the best approach. The known cone’s shape and
compete in the FSD class in the summer of 2020. Initially
size are exploited to perform a keypoint regression and find
composed of only eight students, the autonomous systems
specific feature points, on each of the detected bounding boxes,
team successfully developed, in less than one year, a complete
that match their 3D correspondences, whose locations can
autonomous pipeline [13], comprising perception, estimation
be measured from a reference frame. A classical computer
and control pipelines.
vision approach [19] was initially explored in order to extract
Unfortunately, due to the COVID-19 pandemic, the formula
these keypoints. First, the RGB bounding box of the detected
student competitions that FST was going to compete in, were
cones is converted to the LAB color space and the ”b”
canceled. Because of this, an opportunity for extensively test-
channel, which represents the color axis from blue to yellow,
ing the developed algorithms emerged. As this was the team’s
is extracted. Then, an Otsu threshold [21] is used to obtain a
first year to develop an autonomous car, it would be expected
binary image where, performing contour fitting, three vertices
to encounter problems during this testing phase, which turned
from the top region of the cone can be identified. These three
out to be the case. Even though the different pipelines were
vertices are then extended, using the known dimensions of the
extensively tested in simulation and using synthetic data, one
cone, to obtain further four more keypoints. This approach
pipeline that was notoriously hard to test on simulation was the
has however shown not to be robust in edge cases, which led
perception pipeline. Due to this, several problems related to the
the teams to resort to deep learning methods. Here, custom
perception pipeline and its computational expensiveness arose
CNNs [9], [28] with a ResNet-based architecture [15] were
during the testing phase. In order for the car to be competitive,
developed to detect ”corner-like” features on the input image
it needs to be able to navigate through the track at a very fast
– the bounding boxes of the previously detected cones. The
pace, which means that an accurate, robust and fast perception
detected keypoints are used as 2D points on the image to
pipeline that takes advantage of all the available sensors must
make correspondences with the respective points on the 3D
be developed.
model of a cone. Using the correspondences and the camera
intrinsics, PnP is used to estimate the 3D position of every
II. R ELATED W ORK detected cone. As for stereo camera setups, the 3D position of
In this chapter, a literature review is performed on the the detected cones can be estimated using stereo matching
existing methods used in the perception pipelines implemented algorithms that compute the points’ depth by getting the
by Formula Student Driverless (FSD) teams. Depending on disparity map between the two images.
the sensors available, different approaches can be considered: Regarding the more classical computer vision techniques,
those that solely rely on either cameras (one or more), de- two methods were tested by Zeilinger et al. [31] in order
scribed in Sec. II-A, or LiDARs, described in Sec. II-B and to detect the cones using a stereo camera. The first one
those that fuse the information of both sensors, described in performs a block-matching algorithm [3] on a depth image,
Sec. II-C. which allows to extract and remove 3D planes. Since the track
floor is geometrically known, it can be pre-segmented, making
A. Camera Only Approaches cone proposals appear as isolated objects in the image. These
These methods are highly dependent on the number and type proposals are then further evaluated using classical image
of cameras used (monocular vs. stereo), as depth estimation processing steps on the estimated location in the image plane,
on stereo cameras can be easily obtained. Regardless of the similar to the approach taken in [30]. The second method
is able to detect cones in both images separately by using
2 [Link] Accessible on July 30, 2021. an algorithm that computes the disparity, i.e. , the distance
DISSERTATION ON ELECTRICAL AND COMPUTER ENGINEERING 3

between two corresponding points in the left and right images C. LiDAR and Camera Approaches
of the stereo camera, the z-depth, and, consequently, the 3D These implementations [13], [29] start by identifying the
position of the cone. In both methods, the color of the cones cones using methods similar to those described in Sec. II-B,
is heuristically estimated by evaluating the RGB pixel values i.e. , methods that only rely on the LiDAR. First, the raw
of the cone’s centroid, which should be within a white or a point cloud is filtered using either a box region to remove
black stripe, and moving downwards until another color (blue, points that are outside it or a pass-through filter that performs
yellow, or orange) is found. a simple filtering along any specified axis, removing points
that lay outside the defined range for each axis. Then, the
B. LiDAR Only Approaches
ground points are removed from the point cloud using Random
These implementations [2], [4], [14] have in common their Sample Consensus (RANSAC) [8] with the assumption that
two first LiDAR processing steps. Firstly, the ground plane the ground plane is flat. Finally, Euclidean clustering is used
is removed using an adaptive ground removal algorithm [16] to cluster the point cloud into groups and to compute their
that adapts to changes in the inclination of the ground using correspondent centroids, which are then projected onto the
a regression-based approach. This algorithm divides the Li- image plane.
DAR’s FoV into angular segments and it splits each segment Tian et al. , in [29], create a Region of Interest (ROI) around
into radial bins from which a line is then regressed using the projected centroids and, from these ROI boxes, gradient
the lowermost points of all the bins in a segment. Finally, features are extracted using Histogram of Oriented Gradients
all the points that are within a threshold distance to this line (HOG) [7]. Furthermore, using the extracted HOG features,
are classified as ground points and are removed. The second a Support Vector Machine (SVM) [6] classifier is trained to
LiDAR processing step consists of clustering the point cloud. distinguish cones from other clustered objects. The RGB color
Here, the Euclidean Clustering method is used to group the space of the ROI patches is then converted to the HSV color
point cloud into clusters by performing clustering extraction in space and the color of the cones is estimated by extracting the
a Euclidean sense, i.e. , by computing the Euclidean distance main color present on the ROI.
between each of the 3D points of the point cloud and, using In [13], the FST Lisboa team proposes using YOLOv3-
heuristic methods, evaluate whether that point corresponds to tiny [1], a lightweight version of the 2D real-time object
the cluster or not. Having the cones clustered out, Agarwal et detector YOLOv3, to classify and identify the location of
al. [2] and Gosala et al. [14] apply some heuristic-based the cones on the camera image. Then, for each projected
filters to estimate the likelihood of the clusters being cones, centroid, it is evaluated if it falls inside any of the bounding
before going for the cones’ color estimation step. First, the boxes that surround the detected objects. Those who do,
dimensions of a bounding box created around the cluster receive the color associated with the class of the correspondent
and the elements of the covariance matrix are used to filter bounding box. Although this approach combines the best
candidates. Then, using the dimensions of the cones and the features of each sensor by using the position estimated by
vertical and horizontal angular resolutions of the LiDAR, a the LiDAR and the color estimated using the camera, it is
rule-based filter is used to check whether the number of points a very calibration dependent approach, in the sense that a
in that cluster is in accordance with the expected number single projected point is used to evaluate if it matches any
of points in a cone at that distance. Only the clusters that bounding box and, since the further the detected cones are,
successfully go through these filters are considered to be cones the smaller the bounding boxes will be, this matching task
and are forwarded to the color estimation step. becomes more demanding and less forgiving for further away
To estimate the color of the validated cones, all the teams cones. Moreover, this approach involves using the full image
chose to use different deep learning methods. These teams to detect the cones, something that is very computationally
exploit the fact that the cones used in Formula Student can be expensive, even running, on GPU, a lightweight version of a
distinguishable not only by their color pattern but also by their real-time object detector like YOLOv3-tiny.
intensity pattern. Yellow cones have a yellow-black-yellow The perception pipeline proposed in this work fits inside this
pattern whereas blue cones have a blue-white-blue pattern, last approach category, in the sense that it fuses the information
which results in differing LiDAR intensity patterns. Gosala et between the two sensors to identify the cones and classify their
al. [14] exploits the differing LiDAR intensity patterns of color. However, the proposed perception pipeline differs by
the cones to develop a custom CNN whose input image not using heavy object detectors to detect the different colored
corresponds to an image created by mapping the 3D bounding cones in the full camera image. Instead, the cones are detected
boxes of the validated clusters and whose pixel values store the using the LiDAR information and are then projected onto the
intensities of points in the point cloud. Similarly, Agarwal et camera image plane, where 2D bounding boxes are created
al. [2] use binary classification techniques by developing around the cones. These 2D bounding boxes are cropped
a shallow 1D CNN that exploits the mean intensity and from the original image and, using custom lightweight CNNs,
number of points per ring in the validated clusters. Following the resultant image patches are classified, something that is
a different route, and inspired by the different colors of cones significantly less computationally expensive.
that distinguish the left and right boundary of the track,
Chen et al. [4] chose to use a custom CNN to distinguish III. P ROPOSED P IPELINE
the geometric distribution of cones and, consequently, find the This section describes the vehicle setup, the proposed per-
color of the validated clusters. ception pipeline and how it helps solve the problems identified
DISSERTATION ON ELECTRICAL AND COMPUTER ENGINEERING 4

described in Sec. III-D, where it will be assessed which

proposals actually represent cones and with what color. These
cone proposals are generated through a multi-step point cloud
processing that includes a pass-through filter to trim the
LiDAR Field of View (FoV), a ground removal algorithm to
remove points that belong to the ground, a clustering algorithm
(a) Side view. (b) Top View. to extract and group the point cloud into clusters, a cone
Fig. 1: Position and Field of View (FoV) of LiDAR and reconstruction algorithm to retrieve points incorrectly removed
Camera. during the ground removal process and a final cone validation
step to remove any remaining outliers.
1) Pass-through Filter: As described in Sec. III-A, the
while using the perception pipeline previously implemented by LiDAR used in the FST autonomous prototype has a range
the FST team. The vehicle setup is described in Sec. III-A. of around 100 meters and a horizontal FoV of 360 degrees
In Sec. III-B it is described the process of identifying the with a 30-degree vertical FoV angle. Due to specifications
cones that delimit the Formula Student tracks using only the and positioning of the LiDAR, most of the points of the
LiDAR sensor. Then, in Sec. III-C, it is detailed how the generated point cloud lay outside the Region Of Interest (ROI).
sensors can be fused to help the cone’s color classification task Therefore, in order to remove these points from the point cloud
described in Sec. III-D. Finally, in Sec. III-E, it is described and avoid using them on future point cloud processing steps,
the implementation of a tracking algorithm that allows to which would increase their total execution time, a pass-through
mitigate misclassifications by taking into account previous filter, from the Point Cloud Library (PCL) [26], is applied
observations. to the raw point cloud generated by the LiDAR. This pass-
through filter performs a simple filtering along any specified
A. Vehicle Setup axis, removing points that lay outside the defined range for
each axis. Thus, applying this pass-through filter on all three
The vehicle chosen to be transformed to compete in the
axes allows not only to remove all the points that are behind
Formula Student Driverless (FSD) class was the last prototype
the vehicle, but also points that are much higher than the height
developed by the Formula Student Técnico (FST) team, the
of a cone used in Formula Student competitions and points
FST09e. This was the ninth prototype developed by the team
that are much farther, widthwise, than the width of a Formula
and it is currently the most successful and reliable prototype
Student track.
developed and built by the team. In order to meet the set of
2) Ground Removal: Although most of the points that
challenges imposed by the FSD competition, the prototype
lay outside the ROI were removed in the previous LiDAR
was equipped with a set of sensors that replaces the driver’s
processing step, there are still several points in the ROI that
ability to perceive the environment. The first sensor added to
were generated by the reflection of the LiDAR laser beams
the car was a Velodyne VLP-16 LiDAR sensor, which allows
on the ground. Since these points can be considered noisy
to accurately get the distance to obstacles that are within a
data, as they mostly belong to the ground plane and do not
100-meter range and inside a 360 degrees horizontal Field
represent cones, the same principle of removing these points
of View (FoV). The LiDAR was positioned on top of the
to avoid using them in LiDAR processing steps is followed.
front wing of the car as it corresponds to the position that
To accomplish this, an implementation of the iterative Random
maximizes the number of point returns per cone, allowing
Sample Consensus (RANSAC) [11] method, from the sample
to detect further away cones. The second perception sensor
consensus module of the PCL library, was used. This imple-
added to the prototype was a Lucid Triton TRI032S camera,
mentation allows to choose a mathematical model to be used
responsible for collecting rich and colorful information of the
as the mathematical model whose parameters will be estimated
environment in the form of an RGB image, complementing
by the RANSAC method. RANSAC is a resampling technique
the sparse information, in the form of a point cloud, retrieved
that generates candidate solutions by using the minimum
by the LiDAR. This camera offers a useful 65 degrees of FoV
number of observations (data points) required to estimate the
with a 2048 × 1536 resolution and a frame rate of 30 FPS.
underlying model parameters. In this case, the points to be
The camera is positioned on the main hoop of the car, close
removed belong mostly to the ground, which in the Formula
to where the eyes of the pilot would be, as this positioning
Student context can be approximated to a plane. Thus, the
allows to reduce the occlusion among cones and to still be
iterative RANSAC method was implemented using the PCL
able to perceive cones that are placed one behind the other (in
sample consensus plane model, which defines a plane model
line of sight). The positioning of these sensors as well as a
for 3D plane segmentation. This allows identifying the ground
representation of their FoV are represented in Fig. 1.
plane and removing it from the point cloud.
3) Euclidean Clustering: After the previous two LiDAR
B. Cones Detection processing steps, one gets an unorganized point cloud where
This section describes the process of identifying and gener- all of the 3D points are contained in the ROI, the majority
ating cone proposals, and their corresponding position, using of them belonging to cones. However, despite knowing that
only the point cloud generated by the LiDAR. The generated most of these 3D points belong to cones, one does not know
cone proposals will later be used in the classification step to which cone each of these points belongs, i.e. , at this
DISSERTATION ON ELECTRICAL AND COMPUTER ENGINEERING 5

point, there can be either as many cones as the number of if they are actually cones and with which color. Since these
3D points in the point cloud or a single cone to which every cone proposals may be outside the Field of View (FoV) of the
3D point belongs. To address this problem, the Euclidean camera, which commonly happens in tight corners or when
Distance Clustering method [27], is used, which performs the cone proposals are very close to the car, two different
cluster extraction in a Euclidean sense, i.e. , it computes the Convolutional Neural Networks (CNNs) are used to classify
Euclidean distance between each one of the 3D points of the proposals: one that resorts to the camera image to classify
the point cloud and it groups them into clusters based on the image patches of the proposals and one that resorts to
predefined parameters such as the minimum and the maximum the point cloud intensity to generate grayscale images of the
number of points that a cluster can contain in order to be cone proposals. The choice of the CNN to be used in the
considered valid, the maximum distance between two points classification task is based on whether the camera is able to see
so that they can be considered part of the same cluster, and the the cone proposal or not. To determine this, both sensors are
spatial tolerance for new cluster candidates. These parameters fused by fitting a 3D bounding box, with similar dimensions to
can be tuned taking into consideration the dimensions of a those of a cone used in Formula Student competitions, around
cone used in Formula Student competitions and the expected each 3D cone proposal’s centroid and then projecting it onto
distance between cones. the image plane. Considering that on an image it is possible
4) Cone Reconstruction: During the ground removal pro- to define a bounding box using only its top-left and bottom-
cess, it was noticed that the bottom-most layer of points right points, only the two correspondent 3D points need to be
hitting a cone was often being removed, as it was being projected onto the camera’s image plane. To compute these
associated with the ground plane. Given the small number two points for each 3D centroid P = (X, Y, Z) ∈ R3 , two
of LiDAR layers hitting cones resulting from the not so fine new points, P1 and P2 , are created. These points are created
LiDAR vertical resolution, which is an hardware drawback, by keeping the centroid’s X coordinate, which represents the
these layers can act as invaluable information, especially in depth of the cone proposal’s centroid, and by taking into
the cone classification step presented in Sec. III-D1. Thus, account the cone’s dimensions, as follows:
an extra step in the LiDAR processing is done in order to
P1 = (X, Y + w/2, Z + h/2), (1)
retrieve these so valuable layers of 3D points. To retrieve these
points, a cylindrical area, with similar dimensions to those of P2 = (X, Y − w/2, Z − h/2), (2)
a Formula Student cone, is created around each cluster using where w and h represent the width and height of a cone,
its corresponding centroid. The cylindrical areas, which are respectively. Here, the first point, P1 , represents the top-
an approximation to the geometrical shape of a cone, are then left point of the bounding box while the second point, P2 ,
used to assess if any of the 3D points from the removed ground represents the bottom-right point of the bounding box.
point cloud is contained inside it. Those that are contained These two points can be mapped to points in pixel coordi-
inside the created cylindrical area are then added back to nates on the image plane using the 3×4 perspective projection
the filtered point cloud and associated with its corresponding matrix P , as follows:
cluster.    
5) Cone Validation: Even though in Formula Student com-
  X X
u   Y  Y 
petitions the tracks are in a substantially controlled envi- v  ∼ K R|T   = P   , (3)
Z  Z 
ronment when compared to urban traffic, there can still be 1
obstacles or objects nearby that are not cones and therefore 1 1
 
not part of the track. Despite not being part of the track, with P = K R|T  , where K represents the camera’s intrinsic
these obstacles can be present in the LiDAR FoV and in matrix and R|T represents the extrinsic matrix. The camera’s
the desired ROI, which may cause them to be considered as intrinsic matrix K transforms 3D camera coordinates into 2D
clusters while not being cones. To avoid using these outlier homogeneous image coordinates using a perspective projection
clusters in the classification task, one last LiDAR processing modeled by the ideal pinhole camera model and is parameter-
step is done. In this step, and similarly to what was done in ized as  
Sec. III-B4, a cylindrical area with similar dimensions to those fx 0 cx
of a Formula Student cone, is fitted around each of the cone K =  0 fy cy  , (4)
proposal’s centroid and it is assessed if all the points of the 0 0 1
proposal’s cluster are contained inside the respective cylinder. where fx and fy represent the camera’s focal length, i.e. , the
At this point, only the clusters whose all points fall inside distance between the camera’s pinhole and the image plane,
the correspondent cylindrical area are considered and used to and cx and cy represent the principal point offset, which is
generate cone proposals to later be classified and assigned a the location of the principal pointrelative to the image plane
color. origin. The extrinsic matrix R|T , given by
 
  r11 r12 r13 t1
C. Sensor Fusion R|T =  r21 r22 r23 t2  , (5)
The clusters validated in the last step of the LiDAR, which r31 r32 r33 t3
may not all be cones, are considered to be cone proposals. is a 3 × 4 matrix that takes the form of a rigid transformation
These cone proposals need to be classified in order to validate by concatenating a 3 × 3 rotation matrix R, whose columns
DISSERTATION ON ELECTRICAL AND COMPUTER ENGINEERING 6

represent the directions of the LiDAR axes in camera coor-

dinates, and the 3 × 1 translation column-vector T , which
can be interpreted as the position of the LiDAR origin in
camera coordinates. The extrinsic matrix describes how the
LiDAR is transformed relative to the camera, i.e. , it allows
to map points from the 3D LiDAR coordinate system to the
3D camera coordinate system.
Using Eq. (3) to project the two points from each of
(a) Blue cone. (b) Yellow cone. (c) Unknown.
the 3D boundig boxes, it is possible to create 2D bounding
boxes around the cone proposals in the image plane. By Fig. 2: Representation of the input image of the LiDAR CNN
simply checking if the entire 2D bounding boxes fit inside for each class.
the camera image, it can be concluded which cone proposals
fall inside the camera’s FoV and which don’t. For those
who do, the corresponding bounding box is cropped from intensity pattern being similar to the intensity pattern of the
the original camera image, resulting in an image patch with blue cones. The orange cones can be distinguished by their
the cone proposal. This image patch corresponds to the input orange-white-orange pattern which, in the point cloud, is also
image of the CNN further described in Sec. III-D2. On the translated to a low-high-low intensity pattern. Furthermore,
other hand, the cone proposals that are outside the camera’s these cones are traditionally placed on straights and therefore
FoV go through a preparation process that converts their 3D can be seen by the camera.
point cloud clusters into grayscale images so that these cone This intensity gradient pattern is then exploited to estimate
proposals can be classified using only information given by the cone proposals’ color using a CNN with a similar archi-
the LiDAR, as further described in Sec. III-D1. tecture to the one proposed by Kabzan et al. [19]. The input
of this CNN corresponds to a 32×32 grayscale image with
the intensity points of the cone proposal and the output cor-
D. Cones Classification responds to the probability of the proposal being blue, yellow
As previously mentioned, two CNNs are used to classify and unknown. The backbone of the CNN, which performs the
the color of the identified cone proposals. In Sec. III-D2 it feature extraction, consists of four convolutional layers with
will be described the CNN that resorts to the camera image a max-pooling layer between them, which not only reduces
to classify the image patches of the proposals that fall inside the number of parameters in the model by downsampling the
the camera’s FoV. Then, in Sec. III-D1 it will be described feature map but it also makes feature detection more robust
the CNN that resorts to the point cloud intensity to generate to object orientation and scale changes. In order to introduce
grayscale images of the cone proposals that are outside the non-linearity into the model, the Rectified Linear Unit (ReLU)
camera’s FoV. activation function is used after each convolutional layer and
1) Cone’s Color Classification with LiDAR: In tight corners after each of the five fully connected layers that compose the
it is common for the camera not to be able to see the cones classifier. The output of the network is normalized to a prob-
placed on the outer side of the turn, due to the camera being ability distribution over the three classes using Softmax. Fur-
rigidly mounted to the car and pointing forward. This means thermore, the aforementioned CNN uses a cross-entropy loss
that the cones placed on the inner side of the curve will not be function that increases as the predicted probability diverges
able to be classified using the camera image. In such situations, from the actual label, penalizing and adjusting the model
and due to the nature of the path planner algorithm developed weights based on how far the predicted probability is from the
by the FST Lisboa team, the resulting trajectory may not be actual expected value. It is also less computationally expensive
representative of a good trajectory for the car to follow, as when computing the network’s gradients, which, consequently,
only one side of track is visible. converges faster to the optimal value. Moreover, while training
According to the rules imposed by the Formula Student the model, Dropout [17] - a regularization technique that
competitions [12], the different cones used in Formula Student randomly “deactivates” neurons in the neural network - and
Driverless must have always the same dimensions, shape and batch-normalization [18] - a technique that normalizes the
color pattern. This means that a yellow cone will always be distributions of the hidden layer’s inputs - are used to prevent
distinguishable by its yellow-black-yellow pattern and a blue complex co-adaptations between neurons, control overfitting
cone by its blue-white-blue pattern. Since the intensity of the and improve the generalization of the network.
LiDAR points correspond to the returned strength of the laser The input images, similar to those represented in Fig. 2 are
pulse that generated the point, which is directly influenced by created by mapping every point of the cone proposal’s cluster
the object’s reflectivity, each cone can also be distinguished in to a 32 × 32 image whose pixel values store the intensity
the point cloud according to its intensity pattern. Therefore, of the LiDAR beam in that point. These points are mapped
since black reflects less than yellow and white reflects more into a 32 × 32 matrix, using a linear interpolation, which
than blue, a yellow cone can be described, from top to bottom leaves a boundary of 3 points around the cone. In order to
and in terms of intensity, as having a high-low-high pattern, enhance the intensity pattern of the cones, the pixel values
while a blue cone can be described as having a low-high-low are scaled using a linear scale factor, benefiting high-intensity
pattern. The orange cones are not considered here due to their values and penalizing low ones. This network has been trained
DISSERTATION ON ELECTRICAL AND COMPUTER ENGINEERING 7

E. Tracking
Given the simplicity of the CNNs described in Sec. III-D1
and in Sec. III-D2, which allow to classify and associate a
color to the cone proposals generated through the LiDAR
processing described in Sec. III-B, a Nearest Neighbor [5]
based tracking algorithm was developed in order to mitigate
possible misclassifications returned by either CNNs. By taking
into account previous observations, it is possible to keep track
(a) (b) of the colors associated with each cone proposal and thus
Fig. 3: 2D bounding boxes created using the 3D-2D projection, decide the cone’s color based on the number of times each
represented in (a), and an example of an image patch with the color was associated with the proposal.
cone proposal, represented in (b). This tracking algorithm starts by fusing the information
returned by both neural networks, the cone detections with
an associated position and color. Once all the cone detections
are fused, a different ID is assigned to each of them and,
with a custom dataset of over 19 000 images with the mapped simultaneously, the cone’s color is registered. Having assigned
intensity values of blue and yellow driverless Formula Student the first IDs, the Nearest Neighbor based tracking algorithm
cones and noisy identified clusters to represent the unknown is used to associate the next set of cone detections to the
class. already existing and tracked cone detections. So, for each
2) Cone’s Color Classification with Camera: The classifi- previously tracked cone detection, it is searched for the closest
cation of the remaining cone proposals, whose projected 2D cone detection, within the new set of cone detections, and the
bounding boxes fit inside the camera image, is performed using tracked cone’s position is updated and the color is registered,
a CNN with an architecture similar to the one described in both using the information of the closest cone detection. It
Sec. III-D1. However, this CNN takes as input a 32 × 32 RGB should be noticed that, to avoid miss associating new cone
image that corresponds to the image patch of the projected detections to a tracked cone that, although is the closest one,
bounding box that surrounds the cone proposal, as shown in it is too far away, a searching radius is used taking into account
Fig. 3. Furthermore, the architecture of the network described the minimum distance between cones defined by the Formula
in this section has one less convolutional layer in the feature Student competitions rules. The cone detections that were not
extraction stage, a change that allowed to achieve better associated with any previously tracked cone are considered
computational times while maintaining an acceptable accuracy to be new cones and an ID is assigned to them, allowing
and reliability, as demonstrated in Sec. IV. them to be used as tracked cones in the next iteration. On
the other hand, the previously tracked cones that were not
This CNN consists of three convolutional layers that per- associated with any new cone detection are registered as being
form feature extraction and, similarly to the CNN described missed. By keeping note of the number of times a tracked
in Sec. III-D1, after each convolutional layer a max-pooling cone was missed, it is possible to remove not only outliers
layer is used to downsample the feature maps after using that stopped being identified but also cones that have already
Rectified Linear Unit (ReLU) as the activation function to left the LiDAR FoV, e.g. , cones that have already been passed
introduce non-linearity to the model. As for the classifier, it by the car and are now behind it.
consists of five fully connected layers, where to the output Furthermore, the cone’s color is decided by searching for the
of each one is also applied the ReLU activation function. color that was most associated with that cone ID. For example,
The output of the network is normalized to a probability a cone tracked with the ID 56 that was classified eight times as
distribution over the five classes using Softmax. Additionally, being blue, if the last color classification happens to classify
while training the model, dropout and batch normalization the cone as a yellow cone, the cone will still go through as
were also used to randomly “deactivate” some neurons in being a blue cone, because it has already been previously
the neural network and to normalize the distributions of classified as being blue many more times. The implemented
the hidden layer’s inputs, preventing complex co-adaptations tracking algorithm has proven to be a powerful filter, as it helps
between neurons, controlling overfitting and improving the to smooth out cone color estimation and make the pipeline
generalization of the network. This network has been trained more robust to misclassifications, something that significantly
with a custom dataset of over 100 000 images of the four helps the path planning algorithms in a previously unknown
classes of Formula Student Driverless cones and background track.
images to represent the unknown class. The dataset was
originally only composed of images from FSOCO [10] - a
collaboration between Formula Student teams that aims to IV. E XPERIMENTAL R ESULTS
accelerate the development of camera-based solutions in the In this chapter, the overall results of the proposed perception
context of Formula Student Driverless - but has since been pipeline will be evaluated in terms of performance and compu-
enlarged with more images gathered from runs on different tation cost. First, the overall results of the Convolutional Neu-
testing sites, with different weather/light conditions and with ral Network (CNN) that solely relies on the information from
different cameras. LiDAR are presented in Sec. IV-A. Similarly, in Sec. IV-B,
DISSERTATION ON ELECTRICAL AND COMPUTER ENGINEERING 8

it is presented the overall results of the CNN that relies on TABLE I: Results of the trained LiDAR CNN model on the
the created image patches with the cone proposals. Then, in test dataset.
Sec. IV-C, an ablation study on the LiDAR processing methods Method Class Precision (%) ↑ Recall (%) ↑ F1 (%) ↑ Accuracy (%) ↑
Blue Cone 93.34 85.98 89.51
With Cone
is conducted in order to justify the design choices made on Reconstruction
Yellow Cone 94.38 84.02 88.90 87.77
Unknown 76.51 94.93 84.73
the LiDAR pipeline. Finally, in Sec. IV-D, it is performed a Without Cone
Blue Cone 90.08 73.03 80.66
Yellow Cone 89.14 69.01 77.79 77.17
comparison of the overall perception pipeline results with and Reconstruction
Unknown 60.87 92.99 73.58
without the tracking module.

A. LiDAR Classification performed. This is especially noticeable for further away cones
which, due to the not so fine vertical resolution of the LiDAR,
The model of the LiDAR CNN was trained on an NVIDIA lack on the number of beams that hit them. For these cones, the
GeForce GTX 1070 for 100 epochs with a batch size of 128 missing layer of points breaks the intensity pattern used as the
and using the Adam optimizer [20] with a learning rate starting base for this CNN, turning the intensity pattern of the cones
on 0.0001. Resorting to a learning rate scheduler, the learning into low-high and high-low intensity patterns, respectively for
rate is dynamically reduced by a factor of 0.1 whenever the a blue cone and a yellow cone. These intensity patterns are
validation loss stops decreasing for more than 10 epochs, much less distinguishable, which justifies the increase of false
which indicates that the learning has stagnated. positives for the unknown class, evidenced by its 15% lower
First, to validate the importance of the using the LiDAR precision.
CNN, a study was conducted on the number of cones that
it helps classifying. For this, the full perception pipeline was
B. Camera Classification
run over raw data gathered from a lap performed around a
Formula Student Driverless track with and without using the The camera CNN model was trained on an NVIDIA
LiDAR CNN. The results of this study showed that using the GeForce GTX 1070 for 250 epochs with a batch size of 64 and
LiDAR CNN allows identifying 16 more cones in a track with using the Stochastic Gradient Descent (SGD) optimizer [25]
a total of 93 cones, which represents an increase of 17.2% with a learning rate starting on 0.00001. Similarly to the
in the number of identified cones. From Fig. 4 it is possible train performed on the LiDAR CNN model, a learning rate
to conclude that the LiDAR CNN is essential to identify the scheduler was used to dynamically reduce the learning rate by
cones placed in the inner side of the curves, where mostly a factor of 0.1 whenever the validation loss stops decreasing
cones from the opposite class can be seen by the camera. for more than 10 epochs.
As mentioned in Sec. III-D2, the architecture of the feature
extractor of this CNN was reduced from four convolutional
layers to only three. This design choice was based on a
study conducted on the camera CNN performance in terms
of classification and computation costs. The results obtained
from this study are represented in Table II and Table III.

TABLE II: Average inference time of the camera CNN on

GPU and CPU for both architectures.
Average Inference Time (ms) ↓
Architecture GPU CPU
4 Convolutional Layers 1.9 4.5
Fig. 4: Number of cones classified without using the LiDAR 3 Convolutional Layers 1.6 2.3
CNN.
TABLE III: Results of the trained camera CNN model on the
Then, in order to validate the importance of the cone re- test dataset.
construction step detailed in Sec. III-B4, which retrieves cone
Architecture Class Precision (%) ↑ Recall (%) ↑ F1 Score (%) ↑ Accuracy (%) ↑
points that were incorrectly removed in the ground removal Blue Cone
Yellow Cone
96.32
97.25
96.00
94.99
96.16
96.11
4 Convolutional
step, a comparison between the LiDAR CNN results with and Layers
Orange Cone
Big Orange Cone
89.48
91.00
90.01
87.00
89.74
88.96
93.58

without performing this reconstruction step was conducted. Unknown

Blue Cone
90.69
94.84
95.99
94.00
93.26
94.42
To perform this comparison, a new test dataset was created 3 Convolutional
Yellow Cone
Orange Cone
95.37
83.23
91.99
86.00
93.65
84.59 90.40
Layers
with the same cone images used in the splitted test dataset but Big Orange Cone
Unknown
86.53
87.26
82.00
93.84
84.20
90.43
without performing cone reconstruction. The results obtained
from the trained model on both test datasets are detailed in Regarding the inference times of the CNN, it is possible
Table I. to conclude that the architecture that resorts only to 3 con-
Observing Table I, one can conclude that the overall re- volutional layers is faster on both GPU and CPU. Although
sults obtained when cone reconstruction is not performed is the decrease in inference time is less noticeable when the
around 10% worse across all performance metrics. This can model is run on GPU, decreasing only 0.3 ms, on CPU the
be explained by the missing bottom-most layer of points in the inference time decreases by almost 50%, from 4.5 ms to
input images of the LiDAR CNN when no reconstruction is 2.3 ms. Furthermore, from Table III, it can be observed that
DISSERTATION ON ELECTRICAL AND COMPUTER ENGINEERING 9

both models achieve around 90% on almost all performance TABLE V: Analysis of the average execution time per iteration
metrics for almost all classes. It is also possible to observe after removing certain LiDAR processing steps.
that the expected loss of performance between the two models Average Execution Time per Iteration (ms)
Total Execution
Euclidean Full LiDAR LiDAR CNN Camera CNN
is not very accentuated, decreasing only around 3% on all Method Method
Clustering Pipeline
GPU CPU GPU CPU
Time (ms)
GPU CPU
performance metrics, except for the Orange and Big Orange Full LiDAR
Pipeline
4.13 0.33 4.13 3.6 5.18 8.38 12.05 16.11 21.36
Without
Cone classes, when the model with three convolutional layers Pass-through Filter
1.65 6.55 37.23 70.48 101.32 281.9 405.24 389.61 543.79
Without
is used. The bigger performance decrease in these two classes Ground Removal
2.16 35.71 37.30 2.34 3.36 5.42 7.80 45.06 48.46
and
can be justified by the fact that these two cones are only Cone Reconstruction
Without
distinguishable by the extra white stripe on the big orange Cone Validation
0.02 0.33 4.14 4.58 6.58 10.69 15.36 19.41 26.08

cones and, since a deeper CNN is able to extract deeper

features, the model with four convolutional layers can extract
deeper features and thus correctly classify these classes more The second method removed from the LiDAR processing
often. Furthermore, the lower performance on the orange and was the ground removal and, consequently, the cone recon-
big orange cones in both models can also be explained by struction, as it is no longer necessary. Removing the ground
the class distribution of the training dataset, as these cones removal step from the LiDAR pipeline decreases the number
are around 10% less represented. This small decrease in the of identified cone proposals and, consequently, the number
classification performance of the camera CNN allied to the of classifications performed by both classification CNNs.
possibility of being able to run the CNN on CPU, justifies the Although these decreases cause a reduction on the execution
choice of decreasing the feature extractor architecture of the time per iteration of the classification task, which is explained
camera CNN. by the reduced number of cone proposals to be evaluated, this
reduction in the number of cones means that less cones are
being identified. Since the ground is not being removed, the
C. LiDAR Processing Ablation Study
euclidean clustering algorithm tends to associate the ground
To justify the need of all the LiDAR processing steps points to cones that are closer to them, which can cause the
described in Sec. III-B, an ablation study was conducted on group of points to exceed the defined maximum number of
the implemented methods presented in the LiDAR pipeline. points that a cone can have and therefore not being considered
The results from this ablation study are depicted in Table IV. as clusters. Furthermore, by removing the ground removal step,
the number of points in the point cloud significantly increases,
TABLE IV: Number of cone proposals and correspondent even more than when the pass-through filter is removed.
distribution per iteration for each CNN after removing certain As previously explained, this causes a significant increase
LiDAR processing steps. in the average execution time per iteration of the euclidean
Number of
Avg. Number of Cone Proposals clustering algorithm (over 35 ms) and, consequently in total
Method Cones on Track per Iteration
Cone Proposals
LiDAR CNN Camera CNN execution time of the full perception pipeline. Therefore, since
Full LiDAR Pipeline 147 168 2.25 5.24
Without Pass-through Filter 147 4 937 44.05 176.19 removing this step decreases the number of cones identified
Without Ground Removal
and Cone Reconstruction
147 109 1.46 3.39 while increasing the total execution time, it is essential to
Without Cone Validation 147 214 2.86 6.68 maintain it on the LiDAR processing.
Finally, it was assessed if it was worth performing the cone
Initially, the pass-through filter used to trim the LiDAR validation step that allows to remove most of the outliers
FoV was removed, significantly increasing the number of cone from the cone proposals identified. Removing this method,
proposals identified and, consequently, the number of cone slightly increases the number of cone proposals identified and,
proposals sent to both classification CNNs. Removing the consequently, the number of classifications performed by the
pass-through filter, which, per iteration, only takes an average classification CNNs. As this method is performed over the
of 1.65 ms to trim the LiDAR FoV, causes an increase of over already clustered cone proposals, the average execution time
6 ms in the euclidean clustering average execution time. Since per iteration of the euclidean clustering remains unchanged.
the euclidean clustering method is responsible for grouping the This extra step only takes an average of 0.02 ms and it is the
point cloud into clusters, the greater is the LiDAR FoV, the only extra delay added to the LiDAR pipeline, i.e. , it does
greater will be the number of points in the point cloud, which not cause any further delay on the LiDAR pipeline. However,
explains this increase in the execution time of the algorithm with this small delay comes a decrease of over 3 ms on the
and in the number of cone proposals identified. Furthermore, total execution time of the perception pipeline, as the number
these increases in the number of points in the point cloud of classifications performed by both the LiDAR CNN and the
and in the number of cone proposals identified, cause the camera CNN are reduced.
subsequent LiDAR processing and classification steps to also
significantly increase their execution times per iteration, as
there are more information to be processed and more cone D. Tracking
proposals to be classified. Consequently, the average execution In order to test the influence of the developed tracking
time per iteration of the full perception pipeline increases by algorithm, the proposed perception pipeline was run over raw
over 2300%, from which can be concluded that the pass- data from an acceleration track in a controlled environment,
through filter is essential in the proposed LiDAR pipeline. where the position and color of the cones is known. The
DISSERTATION ON ELECTRICAL AND COMPUTER ENGINEERING 10

number of color corrections made by the tracking algorithm [4] T. Chen, Z. Li, Y. He, Z. Xu, Z. Yan, and H. Li. From perception to
is depicted in Table VI, from where it is possible to conclude control: an autonomous driving system for a formula student driverless
car, 2019. 3
that 92% of the misclassifications got its color corrected by the [5] K. Choeychuen, P. Kumhom, and K. Chamnongthai. An Efficient
tracking algorithm. Most of the remaining misclassifications Implementation of the Nearest Neighbor Based Visual Objects Tracking.
belong to cases where the cone was associated with the In 2006 International Symposium on Intelligent Signal Processing and
Communications, pages 574–577, 2006. 7
Unknown class several times at the beginning, which can [6] C. Cortes and V. Vapnik. Support Vector Networks. Machine Learning,
happen for further cones if the camera-LiDAR calibration is 20:273–297, 1995. 3
not perfect. For these cones, only when the respective color [7] N. Dalal and B. Triggs. Histograms of oriented gradients for human
detection. In 2005 IEEE Computer Society Conference on Computer
surpasses the unknown classifications, the color is corrected. Vision and Pattern Recognition (CVPR’05), volume 1, pages 886–893,
2005. 3
TABLE VI: Number of misclassifications per class and number [8] K. Derpanis. Overview of the RANSAC Algorithm, 2005. 3
of color corrections. [9] A. Dhall. Real-time 3D Pose Estimation with a Monocular Camera
Using Deep Learning and Object Priors On an Autonomous Racecar,
Number of Number of Color 2018. 2
Class [10] D. Dodel, M. Schötz, and N. Vödisch. FSOCO: The Formula Student
Misclassifications Corrections
Blue 92 84 Objects in Context Dataset, 2020. 7
Yellow 114 106 [11] M. Fischler and R. Bolles. Random sample consensus: a paradigm
Orange 70 65 for model fitting with applications to image analysis and automated
Big Orange 39 36 cartography. Commun. ACM, 24:381–395, 1981. 2, 4
Total 315 291 [12] Formula Student Germany. FS Rules 2020, 2020. 1, 6
[13] Formula Student Técnico Lisboa. Autonomous Design Report - FST10d,
2020. 2, 3
[14] N. Gosala, A. Buhler, M. Prajapat, C. Ehmke, M. Gupta, R. Sivanesan,
V. C ONCLUSION A. Gawel, M. Pfeiffer, M. Burki, I. Sa, and et al. Redundant Perception
and State Estimation for Reliable Autonomous Racing. 2019 Interna-
The perception pipeline proposed in this thesis fuses the best tional Conference on Robotics and Automation (ICRA), May 2019. 2,
features from the two available perception sensors (a camera 3
and a LiDAR) by combining the cone’s position estimated [15] K. He, X. Zhang, S. Ren, and J. Sun. Deep Residual Learning for Image
Recognition, 2015. 2
by the LiDAR pipeline, which processes the raw LiDAR data [16] M. Himmelsbach, F. Hundelshausen, and H. Wuensche. Fast segmenta-
to generate cone proposals, with the cone’s color likelihood tion of 3D point clouds for ground vehicles. In 2010 IEEE Intelligent
estimated by either the camera CNN, a custom CNN that Vehicles Symposium, pages 560–565, 2010. 3
[17] G. Hinton, N. Srivastava, A. Krizhevsky, I. Sutskever, and R. Salakhut-
classifies the image patches of the correspondent generated dinov. Improving neural networks by preventing co-adaptation of feature
cone proposals, or the LiDAR CNN, a custom CNN that detectors, 2012. 6
classifies the color of the cone proposals’ cluster resorting [18] S. Ioffe and C. Szegedy. Batch Normalization: Accelerating Deep
Network Training by Reducing Internal Covariate Shift, 2015. 6
to the intensity pattern that distinguishes the different cones. [19] J. Kabzan, M. de la Iglesia Valls, V. Reijgwart, H. F. C. Hendrikx,
Furthermore, given the simplicity of the custom CNNs used, C. Ehmke, M. Prajapat, A. Bühler, N. Gosala, M. Gupta, R. Sivanesan,
a Nearest Neighbor based tracking algorithm was developed A. Dhall, E. Chisari, N. Karnchanachari, S. Brits, M. Dangel, I. Sa,
R. Dubé, A. Gawel, M. Pfeiffer, A. Liniger, J. Lygeros, and R. Siegwart.
to correct the estimated cone’s color taking into account the AMZ Driverless: The Full Autonomous Racing System, 2019. 2, 6
estimated color of previous observations. [20] D. Kingma and J. Ba. Adam: A Method for Stochastic Optimization,
The results show that the proposed perception pipeline is 2017. 8
[21] N. Otsu. A Threshold Selection Method from Gray-Level Histograms.
able to accurately detect and classify the cones even when IEEE Transactions on Systems, Man, and Cybernetics, 9(1):62–66, 1979.
running the pipeline on CPU, something that can bring benefits 2
on the Efficiency event of the Formula Student Competitions. [22] J. Pedoeem and R. Huang. YOLO-LITE: A Real-Time Object Detection
Algorithm Optimized for Non-GPU Computers, 2018. 2
Furthermore, they also show the importance of the developed [23] J. Redmon and A. Farhadi. YOLO9000: Better, Faster, Stronger, 2016.
LiDAR CNN in order classify the detected cones that the 2
camera is not able to see, which most of times correspond [24] J. Redmon and A. Farhadi. Yolov3: An incremental improvement, 2018.
2
to the cones places in the inner side of the curves. Moreover [25] S. Ruder. An overview of gradient descent optimization algorithms,
the developed tracking algorithm has shown to help correcting 2017. 8
the color of misclassificated cones, avoiding the propagation [26] B. Rusu and S. Cousins. 3D is here: Point Cloud Library (PCL). In 2011
IEEE International Conference on Robotics and Automation, pages 1–4,
of those misclassification to the path planning algorithms that 2011. 4
follow. Further work can extend the taken approach to detect [27] R. Rusu. Semantic 3D Object Maps for Everyday Manipulation in
and classify other objects in other environments by relaxing Human Living Environments. KI - Künstliche Intelligenz, 24:345–348,
2010. 5
the assumptions made for the Formula Student context. [28] K. Strobel, S. Zhu, R. Chang, and S. Koppula. Accurate, Low-Latency
Visual Perception for Autonomous Racing:Challenges, Mechanisms, and
R EFERENCES Practical Solutions, 2020. 2
[29] H. Tian, J. Ni, and J. Hu. Autonomous Driving System Design for
[1] P. Adarsh, P. Rathi, and M. Kumar. Yolo v3-tiny: Object detection Formula Student Driverless Racecar. In 2018 IEEE Intelligent Vehicles
and recognition using one stage improved model. In 2020 6th Interna- Symposium (IV), pages 1–6, 2018. 3
tional Conference on Advanced Computing and Communication Systems [30] H. Yong and X. Jianru. Real-time traffic cone detection for autonomous
(ICACCS), pages 687–694, 2020. 2, 3 vehicle. In 2015 34th Chinese Control Conference (CCC), pages 3718–
[2] A. Agarwal, K. Brandes, R. Chang, K. Doherty, M. Kabir, K. Strobel, 3722, 2015. 2
N. Stathas, C. Trap, A. Wang, and L. Kulik. Robust, High Performance [31] M. Zeilinger, R. Hauk, M. Bader, and A. Hofmann. Design of an
Software Design for the DUT18D Autonomous Racecar, 2018. 3 Autonomous Race Car for the Formula Student Driverless (FSD), 2017.
[3] A. Barjatya. Block matching algorithms for motion estimation. IEEE 2
Transactions Evolution Computation, 8:225–239, 01 2004. 2