Title of the project: Fusion of Visual SLAM and LiDAR SLAM

for enhanced environment perception in Autonomous

system
Submitted by: Mohammad Iftikhar

University of Queensland
 1. State of the art.
SLAM has recently attracted a lot of attention due to the important
applications in autopilot vehicles field this technology has registered great
improvement in the recent years. Specific to the use of cameras as the main
sensors, there has been much work done particular approaches include ORB-
SLAM, LSD-SLAM and SVO. However, the existing visual SLAM systems have
drawbacks associated with the lighting conditions, optical texture and motion
blur. On the other hand, there are LiDAR SLAM which uses LIDAR sensors and
is characterised by high accuracy with minimal interference from
environmental elements. LiDAR SLAM is primarily based on the scan-
matching approaches such as Iterative Closest Point (ICP)[1]. As there are
drawbacks to the use of individual sensors, researchers have turned their
attention towards a combination of visual and LiDAR SLAM system that has
been proved to provide high accuracy, better robustness, and flexibility in
most environments.
2. Description of the project.
2.1 Literature review:
Navigation is an important function of autonomous vehicles in the industry.[2]
However, due to the lack of Google positioning signal (GPS-Signals) in a
closed environment, traditional technology may fail. Although the current
deployment method often has problems in providing a stable solution for
centimeter level indoor localization.[3] Indoor autonomous vehicles have to
construct their own navigation system by installing additional navigation
sensors. A navigation technology called SLAM has been studied in order to
meet these requirement.[4]
SLAM refers to simultaneous localization and mapping was first studied as a
research in 1980s [5] and after that great efforts has been put to solve the
localization and mapping problem as single problem rather than treating it as
two independent ones [6]. SLAM feasibility was first theoretically proven in [7],
where the estimated map was found to converge monotonically to a relative
map.
SLAM algorithm can be divided into the visual SLAM and LiDAR-based SLAM.
Lot of work has been done in the field of visual SLAM in the past two
decades.[8] Many researcher utilizing stereo cameras to implement visual
SLAM after inspiring from human vision. Meanwhile, Monocular camera was
also used to construct effective SLAM. In [9], a popular visual SLAM built on
oriented fast and rotated brief (ORB) feature descriptor was proposed, which
gave real time performance without GPU’s. There are other visual SLAM
methods such as LSD-SLAM and SVO that give good solutions. But due to the
use of visual sensors they are suffered. The Monocular and stereo would
suffer from light and optical texture in the environment. Second the image
that we get as result of visual sensors get blurred when we are moving in fast
speed. These weakness often lead to limitation of visual SLAM in industry.[10]
On the other hand, LiDAR sensor can give high precision measurement
regardless of the light intensity and ambient conditions. Also LiDAR solve
problem of blurry image during high speed conditions. As a result, LiDAR

1|Page
 SLAM can often provide more robust localization performance in indoor
environments.[11] LiDAR SLAM system mainly rely on the scan-matching
technique, which is built on a popular point registration algorithm (Iterative
closet point algorithm or ICP-algorithm.[1]
The concept of elevation mapping has been successfully applied to terrain
mapping in mining applications. A recent study by Bettens [12] demonstrated
a terrain mapping method using two excavator-mounted 3D LiDARs,
maintaining an elevation map of the bench over a 27-hour operating period.
However, this method has limitations, as material added to the workspace
from a face collapse or material movement cannot be updated in the terrain
map. In contrast, our proposed method uses a per-cell height Kalman filter
update, similar to the approach used by D'Adamo [13], and incorporates LiDAR
odometry and SLAM techniques to provide a more robust and accurate terrain
mapping solution, enabling the creation of a detailed and accurate terrain map
that can be updated in real-time, particularly important in dynamic
environments such as mining.
2.2 Research Problem:
Despite the advancement in SLAM technology, autonomous vehicles still face
challenges in achieving accurate and robust localization and mapping,
particularly in complex and dynamic environment. Visual SLAM technologies
suffer from limitations such as sensitivity to lighting condition and motion
blur, while LiDAR SLAM techniques are often affected by multipath and
occlusion issues. These challenges are as follows To overcome these
challenges, the following research objectives are proposed The first research
goal of this research is to study the synergy between visual and LiDAR SLAM
techniques and their impact on the localization and mapping of autonomous
vehicles.
2.3 Objectives:
1. Development of a Robust and Versatile SLAM System: Develop and improve
a SLAM framework that combines Visual SLAM and LiDAR SLAM for
enhancing the robot’s performance in various situations and light conditions.
It is necessary that the system can be easily optimized for different types of
situation for instance when the movement is from indoors to outdoors,
variation in light conditions, and variability of the terrain.
2. SLAM application optimization: As it has been outlined above, the SLAM
system processes two type of data streams – visual and LiDAR to generate a
map of the environment in real time. The following include achieving the best
apriori computational latency time, achieving the best throughput and
ensuring reliability in solutions.
3. Advanced Sensor Fusion Techniques: enhance and increase visual and LiDAR
integration that will address the advantages and limitations of the two
sensors. The four mentioned aspects will have to address sensor spatial
placement and type, temporal data frequency, and effective decision making
in various operational environments.

2|Page
 2.4 Research Methodology:
This research proposal is on an improved LiDAR Odometry hammer that will
be especially useful in constructing machinery which includes excavators and
other related tools. The localization, mapping, and terrain mapping services
will be achieved with the use of data obtained from the LiDAR sensors
mounted on the machinery. Different from [12], where the author proposed
the process of constructing and updating the terrain map based on hidden
markov models, this paper instead proposes a deep learning technique which
is RNNs to enhance the efficiency and stability of the mapping process.

The study will both qualitative and quantity research design to accomplish the
stated aims. The implementation of this algorithm shall have to incorporate
various simulations which are as follows: MATLAB simulation, GAZEBO
simulation. The proposed algorithm will be trained and tested using synthetic
data and as mentioned before, the data will be simulated using the simulators.
The effectiveness of the proposed algorithm will be assessed in terms of
accuracy, precision, recall, as well as the time taken to execute it.

The above algorithm will be experimentally performed through incorporation

of LiDAR sensors on construction equipments including the excavators and
data collection in different construction spaces and contexts. The qualitative
data gathered will be analyzed by using descriptive statistics, while
quantitative data shall be analyzed by adopting inferential statistics, and the
findings will presented in tabular form, graphs, and figures. The findings will
be then utilised to assess the efficiency of the developed algorithm and nests
for its refinement. Different kinds of constructions like excavation, grading,
and paving will be performed to show hysterical and dynamic characteristics
of the algorithm.
Comparison with Existing Methods:
It has been seen that Markov-based HMM methods are highly effective in
different types of localization and mapping for the autonomous vehicles. Such
approach has a number of drawbacks:
•AFP Markov Property It means that the HMM methods require that the state
of the system at a particular time depends on the system at the previous time
only. However, as shall be elucidated in real-life dynamism, it may be a
questionable assumption.
•Lack of Reliability with Non-Linearity: The assumption of HMM is linear
Gaussian in nature and thus cannot handle non- linear relationships between
the state and the observations.
• Noise and Occlusion: It has been found that the performance of HMM
methods is greatly influenced by noise and occlusion of the data.

Nevertheless, using RNN, particularly LSTM networks, has the following

benefits:
•Capability to capture Non-linearity: RNN has the capability to understand the

3|Page
 non-linearity of different variables that is an advantage than using in advanced
and unsteady network applications.
• Noise and Occlusions: Using RNNs, a calibration effect is possible to get
some form of noise and occlusions and therefore improving the accuracy of
the estimates.
•Data modeling: RNNs possess the ability to model long-term dependencies in
the information, which is greatly significant in identifying locations and
mapping of self-employable vehicles.
Therefore, it is believed that the proposed RNN-based methodology will
outperform and be more reliable than Markov-based HMM prevailing in the
present research area.

3. Expected results.
The following are presumed benefits that the proposed system will be useful
in developing: The localization accuracy, robust mapping, real-time capability,
and versatility to different surfaces. The integration would lead to better and
efficient mapping, particularly, dynamics objects and environment changes .
Besides, the proposed optimized algorithm shall be able to produce near real-
time results with a view of making the system effective in real-world
scenarios.
Based on the finding of this study, this work has the potentials of being
adopted into a wide range of applications such as in autonomic cars, robots,
drones, mapping and surveying among other uses. These enhancements
mean also that these parts can allow AVs to locate themselves and map them
more effectively in the world around them in dense environments. In the
following, the verbalised approach is proposed to be implemented in the
robotics and drones to improve their navigation and mapping facilities in
different terrains. The high performance can also be used in other surveys
and mapping, including construction, mining, and environment inspections.

Lead-time for implementation.

Action Plans

[Link] Tasks Time interval

1 Literature review and background Research March 2025- Jun
2025
2 LiDAR and Visual sensors selection and July 2025- Sep 2025
integration
3 Data Collection and Optimization of Oct 2025- Dec 2025

4|Page
 individual Algorithms
4 Development of Machine learning base Jan 2026- April 2026
algorithm to fuse both Models
5 Simulation and testing May 2026 – Jun 2026
6 Experimental Implementation July 2026- Sep 2026
7 Data analysis and evaluation Oct 2026 – Feb 2027
8 Result interpretation and Documentation March 2027 – Jun
2027
7 Research paper writing and publishing July 2027 – Oct 2027
6 Thesis submission Oct 2027-Jan 2028

References

[1] J. Behley and C.

Stachniss, «
“Efficient surfel-
based slam
using 3d laser
range data in
urban
environments,”
in,»
Proceedings of
the Robotics:
Science and
Systems (RSS),
2018, 2018.

[2] W. Z. W. T. Y.
H. a. Q. Z. L.
Chen, «Deep
integration: A
multi-label
architecture for
road scene
recognition,»
IEE
Transactions
on Image
Processing,
Vol. %1 di %2
vol. 28, no. 10, ,

5|Page
 pp. pp.
4883–4898,,
2019.

[3] J.-N. H. G. O. a.
J. P. K.-H. Lee,
«“Ground-
movingplatform
-based human
tracking using
visual SLAM
and
constrained
multiple
kernels,”,» IEEE
Transactions
on Intelligent
Transportation
Systems, , Vol.
%1 di %2vol.
17, no. 12, , pp.
pp.
3602–3612,,
2016..

[4] M. Y. R. Y. a. C.
W. H. Fang,
«Ground-
texture-based
localization for
intelligent
vehicles,,» IEEE
Transactions
on Intelligent
Transportation
Systems,, Vol.
%1 di %2 vol.
10, no. 3,, pp.
pp. 463–468, ,
2009..

[5] H. D. Whyte,
«Simultaneous
localisation and
mapping
(slam): Part I
the essential
algorithms,»
Robotics and

6|Page
 Automation
Magazine,
2006.

[6] L. C. H. C. Y. L.
D. S. J. N. I. R.
a. J. J. L. C.
Cadena, «
“Past, present,
and future of
simultaneous
localization and
mapping:
Toward the
robust-
perception
age,”,» IEEE
Transactions
on Robotics, ,
Vol. %1 di
%2vol. 32, no.
6,, pp. 1309-
1332, 2016.

[7] P. N. S. C. H. F.
D.-W. a. M. C.
M. G.
Dissanayake,
«A solution to
the
simultaneous
localization and
map building
(slam)
problem,,» IEEE
Transactions
on robotics and
automation, ,
Vol. %1 di
%217, no. 3, , p.
229–241, 2001.

[8] D. Z. L. P. R. Y.
P. L. a. W. Y. H.
Zhou,
«StructSLAM:
Visual SLAM
with building
structure lines,»

7|Page
 IEEE
Transactions
on Vehicular
Technology, ,
vol. 64, p.
1364–1375,
2015.

[9] J. M. M. M. a. J.
D. T. R. Mur-
Artal, «Orb-
SLAM: a
versatile and
accurate
monocular
slam system,”,»
IEEE
Transactions
on Robotics,
vol. 31, pp.
1147-1163,
2015.

[10] J. R.-A. a. J. M.
R.-M. J.
Fuentes-
Pacheco, «
Visual
simultaneous
localization and
mapping: a
survey,»
Artificial
Intelligence
Review, , vol.
43, pp. 55-81,
2015.

[11] H. C. H. D. J. G.
G. X. J. Q. a. T.
Y. K. Ji, «Cpfg-
slam: a robust
simultaneous
localization and
mapping based
on lidar in off
road
environment,»
IEEE Intelligent

8|Page
 Vehicles
Symposium
(IV), pp. 650-
655, 2018.

[12] J. J. ,. P. P. R.
M. Vedant
Bhandari,
«Onboard
Sensor-Driven
Terrain
Mapping for
Autonomous
Mining,» School
of Mechanical
and Mining
Engineering, ,
The University
of Queensland,
2023.

[13] X. Y. a. M. H,
«LiDAR-based
SLAM for
robotic
mapping: state
of the art and
new frontier,» in
Intelligent
Connected
Vehicle Center,
Tsinghua
Automotive
Research
Institute, ,
Suzhou, china,
2023.

9|Page