Reinforcement learning-based fuzzy

controller for autonomous guided vehicle

path tracking
Abstract
Automated guided vehicles (AGVs) play a critical role in connecting the entire production line. A
fully automated AGV must perform four functions, namely simultaneous localization and
mapping (SLAM), positioning, routing, and path tracking. In the present study, Hector SLAM,
adaptive Monte Carlo localization, and Anytime Repairing A* were used to perform SLAM,
localization, and path planning functions, respectively. For path tracking, a fuzzy proximal policy
optimization (FPPO) controller was created by applying fuzzy control theory and incorporating
reinforcement learning to improve the accuracy of the fuzzy controller’s output. Currently,
extensive experience is required to manually design fuzzy rules and membership functions; an
inappropriate design can lead to low control precision and poor dynamic system quality. The
experimental results in both virtual and real environments demonstrated that the FPPO
controller reduced both maximum and mean path tracking errors to a considerably greater
extent than did a conventional fuzzy controller. In the virtual environment, the average tracking
error for the circular trajectory decreased from 0.05 to 0.02 m, the U-shaped trajectory error
decreased from 0.02 to 0.01 m, and the right-angle trajectory error decreased from 0.02 to
0.01 m, highlighting the FPPO controller’s high precision and stability. Similarly, in a real
environment, the average tracking error for the circular trajectory decreased from 0.05 to
0.02 m, the U-shaped trajectory error decreased from 0.03 to 0.01 m, and the right-angle
trajectory error decreased from 0.02 to 0.01 m. These results indicate that the FPPO controller
exhibits exceptional adaptability and reliability across various path types. The FPPO controller
overcomes this shortcoming by integrating reinforcement learning to optimize the fuzzy control;
the method also provides a self-learning ability to the AGV. By comparison with a conventional
fuzzy controller, the FPPO controller was demonstrated to improve the AGV’s path tracking
ability.

Introduction
In response to sub-replacement fertility and rising personnel costs, enterprises have
endeavored to find ways to reduce personnel costs and improve productivity. In particular, they
have focused on transitioning from conventional manual production to automation. On an
automated production line, material processing comprises a minimal proportion of the overall
processing procedure; material transportation and waiting for processing are a much larger
fraction. According to [1], workpiece processing time is only 5 % of duration of the entire
procedure; the remaining 95 % is attributable to transportation and temporary storage. An
effective and time-efficient transportation system is conducive to reducing material processing
time and improving the production efficiency and overall competitiveness of an enterprise.
However, several challenges and limitations remain in existing research. For example, although
a study [1] in 1989 indicated that the workpiece processing time accounted for only 5 % of the
entire procedure, this finding might have been derived from outdated technologies, and the
control system adopted in the study might not effectively address the complexity and scale of
contemporary automated manufacturing. Similarly, although the finite-time control algorithm
proposed in [2] was based on a strong theoretical foundation, its practical application in
resource-constrained systems may encounter difficulties. These challenges suggest that the
adaptability and robustness of existing technologies in real-world environments still require
considerable improvement.

Many algorithms have been developed for path tracking control. One study [3] proposed a
behavior-based collaborative tracking policy to enable a dual robotic dolphin system to
accomplish object tracking tasks. Specifically, a behavior-based approach was incorporated in
the centralized architecture of the system to enable high-level decision-making. Another study
[2] employed an observer-based finite-time control method that involved designing a finite-time
extended-state observer to revert a system to previously measured linear and angular velocities
while estimating its total disturbance. A finite-time distributed path maneuvering guidance law
was then designed on the basis of the containment control method and artificial potential
functions. Finally, an antidisturbance control method was created on the basis of the finite-time
extended-state observer and the nonlinear tracking differentiators. A hybrid control law was
proposed by [4] for determining the constraints of the forward and angular velocities of an
unmanned aerial vehicle. Specifically, a hybrid control framework based on an invariant set was
employed to enable coordinated path tracking for a vehicle. The method ensured that path
tracking and coordination errors converged and met the constraints on the forward and angular
velocities. Moreover, existing control strategies may insufficiently account for adaptability and
robustness under dynamic environmental conditions. For instance, the control strategy
proposed in [4] might perform inadequately under extreme conditions, such as those involving
gusts or turbulence. Similarly, the nonsingular terminal sliding mode control presented in [5] still
has room for improvement in shock absorption and system performance enhancement. The root
of these issues lies in the heavy reliance of most studies on simulation testing, with a lack of
validation in real environments. Unpredictable factors in real-world settings can degrade
performance, particularly in situations involving dynamic changes or uncertain disturbances. In
this paper, we further expand the discussion of machine learning techniques applied to
simultaneous localization and mapping (SLAM), positioning, and path planning. Machine
learning methods incorporating physical information, such as physics-informed machine
learning (PIML), have achieved considerable improvements in accuracy and reliability [6], [7].
Reinforcement learning has also shown remarkable potential in dynamic planning for
transportation systems [8]. Furthermore, studies in logistics and transportation have confirmed
that path optimization algorithms such as Anytime Repairing A play a critical role in efficient
material handling [9]. Furthermore, the development of neural network-based SLAM provides an
alternative approach to conventional algorithms, warranting further exploration [10]. An
enhanced path tracking control system was adopted for a semiautomatic trailer docking system
for trucks [11]. The system comprised an improved predictive path-following controller that could
perform adaptive look-ahead for forward path tracking. Reverse path tracking was achieved by
cascading controllers with reverse predictive path-following control and gain-scheduled linear
quadratic control. A Serret–Frenet frame transformation framework [12] was adopted in the
feedback dominance-based adaptive back-stepping controller of a tracking error model. The
back-stepping controller was used to compensate for nonlinear disturbances, and the
disturbance resulting of the model was approximated using an adaptive algorithm. Another
study adopted the least-squares support vector machine model to design a path tracking
controller for an automated guided vehicle (AGV), reducing its calculation load [13]. The
modeling errors and biases of the least-squares support vector machine model were defined as
distractors. To diminish the effect of these distractors, an H-infinity controller was designed to
enable the AGV to track reference paths, and this controller was optimized with the grey wolf
optimizer. The stability condition was converted from bilinear matrix inequality to linear matrix
inequality (LMI). Finally, an LMI toolbox was applied to solve the problems associated with LMI.
One study [14] proposed a computationally efficient path-following control policy to stabilize yaw
motions for electric AGVs. To meet the stability constraints for sideslip angle, yaw rate, turning
position error, and Iyapunov stability, nonlinear model predictive control was applied. To reduce
the computational burden, a continuation/generalized minimal residual algorithm was used.
Dead-zone penalty functions were utilized to address inequality constraints and ensure the
smoothness of the solution, and numerical algorithms were applied to obtain satisfactory initial
values. One study [5] proposed an AGV path tracking control policy that incorporated
nonsingular terminal sliding mode (NTSM) control and active disturbance rejection control
(ADRC). The strategy of this NTSM–ADRC controller involved simplifying the complex path
tracking problem into a yaw angle tracking problem by establishing an expected yaw angle
function. The controller estimated and compensated for the unestablished dynamic model and
external disturbance of the system in real time by using an extended-state observer. To improve
the response of the controller, an exponential approximation law was incorporated into the
NTSM for nonlinear error feedback control. The improved controller calculated simple two-
degree-of-freedom linear vehicle dynamics to enhance the performance of the AGV in various
driving conditions.

AGV routing can be achieved through fixed routing or virtual routing. Fixed routing involves
preplanning the path of the AGV and providing it with physical markers to guide its movement.
Although fixed routing is simple, reliable, accurate, and stable, the markers must be moved if
the path is changed. Virtual routing involves installing a map in the AGV’s computer system and
setting a path on the map or using a routing algorithm to calculate a path; this method does not
require physical markers. Such a virtual route is easier to maintain than a fixed route because, if
changes in the path configuration are required, the path can be updated on the AGV’s computer
instead of by replacing physical markers. However, virtual routing is less stable and has larger
errors than fixed routing does. This study focused on addressing the large errors of virtual
routing by incorporating fuzzy theory [15], [16], [17] with reinforcement learning [18], [19] to
develop a path tracking controller. The method was intended improve the performance of AGV
path tracking.

To address the deficiencies in precision and robustness observed in existing systems, we

proposed an innovative path-tracking controller based on fuzzy control theory and reinforcement
learning algorithms, termed the Fuzzy Proximal Policy Optimization (FPPO) controller. This
controller was trained in a virtual environment, the physics engine of which was established in
Gazebo. The outputs of the fuzzy controller were optimized using proximal policy optimization
(PPO). Compared with conventional fuzzy controllers, the FPPO controller demonstrated
superior performance in terms of both maximum and mean errors, highlighting its advantages in
AGV path-tracking tasks. Furthermore, the system exhibited excellent adaptability and fault
tolerance, offering a novel solution for addressing complex path types and hardware conditions.

In this study, a realistic simulation of a factory environment with a physics engine was created
using Gazebo and Rviz. This simulated environment enabled maps to be quickly and
conveniently made, the training of a model, and the testing controller performance. An
experiment was then conducted to verify the performance of the controller to ensure the safety
of its operator. Assembly lines in the factory were simulated in the environment. The path
tracking controller incorporated PPO and fuzzy control. Fuzzy control reduced the complexity of
the system and increased its robustness, enabling it to adapt favorably to various routing
conditions and have satisfactory fault tolerance. Moreover, the output of the fuzzy controller was
optimized through PPO. Hence, extensive experience in designing a fuzzy controller was not
required to improve its path tracking performance.

This study is divided into five sections. Section 2 introduces the experimental framework and the
hardware configuration for the AGV in this study. Section 3 discusses the algorithms employed
in this study and the structure of the fuzzy PPO (FPPO) controller. Section 4 presents the
experimental results for the AGV using developed the algorithms and the FPPO controller in the
simulated environment. Section 5 concludes the paper and discusses future research directions.

Section snippets
System architecture
In the experiment of this study, the simultaneous localization and mapping (SLAM) tool Hector
SLAM was applied to establish a map, and adoptive Monte Carlo localization and Anytime
Repairing A* were employed for positioning and routing. Subsequently, a model was built. The
position, orientation, and reference path of the AGV were the model input, and the forward and
turning speeds of the AGV was the model output (Fig. 1). A Mir200 was used as the AGV for a
physical demonstration (Fig. 2 and

Proposed method
First, we employed LiDAR for complete SLAM and subsequent path planning. Given the
limitations of LiDAR in certain scenarios—such as noise in the data, an overabundance of
ground points, and missing points caused by occlusions—a multistep data processing method
was adopted. Initially, outlier removal and a voxel grid filter were applied to reduce noise and
data density. Subsequently, a plane fitting method was used to eliminate ground points, thereby
enhancing perception accuracy. Finally,

Experimental results
The FPPO controller was trained in a virtual environment. In the experiments, the FPPO and
conventional fuzzy controllers were used to follow three paths, and the results were compared.
Finally, the FPPO controller and the conventional fuzzy controller were employed in a real AGV
for validation and comparison. The experimental results are available in [24].

Conclusion
This study combined fuzzy control theory and reinforcement learning to design and optimize a
path-tracking controller, culminating in the development of the FPPO controller within a virtual
environment. The virtual environment was constructed using the Gazebo simulator, which
includes a physics engine, providing a convenient platform for model training and controller
parameter testing while ensuring the safety of AGV operations. The research results
demonstrated that both the FPPO controll