IEEE ITAIC(ISSN:2693-2865)

A Learning-driven Method for Adaptive PID

Control
2022 IEEE 10th Joint International Information Technology and Artificial Intelligence Conference (ITAIC) | 978-1-6654-2207-9/22/$31.00 ©2022 IEEE | DOI: 10.1109/ITAIC54216.2022.9836627

Fei Tian1 , Ye Xu2 , Haizhou Du2 , Xiaohan Guo2 , Xin Li1 , Jianpeng Li1 ,
[Link] Corporation LTD. Gansu Branch, Gansu, China
[Link] University of Electric Power, Shanghai, China
Corresponding Author: Ye Xu Email: xuyevaga@[Link]

Abstract—Aiming at the slow convergence speed of existing samples compared to neural network PID, its structure is too
adaptive proportional-integral-derivative (PID) controllers, an complex to guarantee real-time performance. In view of this,
adaptive PID controller driven by deep reinforcement learning Adel [15] designed an adaptive PID controller based on the
is proposed. The controller uses the multi-threaded synchronous
learning feature of the Advantage-Actor-Critic (A2C) structure AC algorithm. The controller has a simple structure with
to train multiple Actor-Critic (AC) structure agents in parallel only one RBF network. Compared with the traditional PID
to reduce the correlation of learning samples. Each agent adopts control method, adaptive PID can adjust parameters online
a radial basis function (RBF) network to simultaneously approx- and improve the control performance of the system, which
imate the policy function and the value function to improve has become a research hotspot in recent years.
computational efficiency. Finally, we perform simulations in
the OpenAI gym environment to illustrate the efficiency and In general, the existing adaptive PID control strategies are
robustness of the proposed scheme. divided into two types including model-based adaptive PID
Index Terms—PID, deep reinforcement learning, advantage control [16] and neural network-based adaptive PID control
actor-critic, adaptive control [17]. However, it has been clarified that model-based adap-
tive PID control requires an assumption that the established
I. I NTRODUCTION model can accurately represent the real plant dynamics [18],
PID control is one of the most common control schemes and and modeling complex systems is time-consuming and lacks
dominates most industrial processes and mechanical systems precision, so it may not be difficult to obtain teaching signals
due to its versatility, high reliability, and ease of operation to tune PID parameters appropriately. On the other hand, neu-
[1]. When the mathematical model of the controlled device is ral network-based adaptive PID control [19] uses supervised
unknown, operators and control engineers can manually adjust learning to optimize network parameters, which is difficult
the PID controller parameters based on empirical knowledge. to obtain teaching signals and predict the value of unlabeled
Some classical tuning methods, such as the Ziegler-Nichols data. Therefore, the application of these methods has certain
method [2] and the Chien-Hrones-Reswich method [3], have limitations.
been applied to process control and have significantly supe- In recent years, the rapid development of machine learn-
rior performance compared to manual tuning methods. These ing algorithms, mainly deep learning [20] and reinforcement
methods are suitable for simple controlled devices, but for learning [21] algorithms, makes it possible to combine deep
complex systems with nonlinearities, the performance of the reinforcement learning with PID controllers.
controller cannot be guaranteed because an accurate mathe- The Asynchronous Advantage Actor-Critic (A3C) learn-
matical model cannot be constructed from the real system. ing algorithm [22] proposed by Google’s DeepMind team
In the scientific community, artificial intelligence technol- uses the multi-threaded parallelism feature of the CPU to
ogy emerged as a suitable solution to the adaptive control asynchronously train multiple agents. The agents in parallel
problem [4]. The capacity of intelligent systems, such as fuzzy will experience different learning states, thereby breaking the
logic [5]–[7], expert system, [8] or genetic algorithms [9], to correlation of learning samples. A2C [23] is a synchronous
adapt to changes seems suitable for this type of control prob- version of A3C. The notable difference is that at any time,
lem. Acosta [10] proposed to apply fuzzy rules to obtain PID different workers use the same set of policies, they are
parameters to control different types of equipment. Similarly, completely synchronized, and the timing of updates is also
Porter and Jones [11] also proposed applying a genetic algo- synchronized. Since each worker is the same as each other,
rithm to achieve the effect. Rossomando and Soria [12] used A2C is actually equivalent to only two networks, one of which
a combination of multilayer perceptron neural networks and is the Global network responsible for parameter updating, and
support vector machines to adaptively tune PID controllers; the other is responsible for interacting with the environment to
Jafari and Dhaouadi [13] used recurrent neural networks to collect experience, but it uses multiple parallel environments
make the mechanism of continuous online learning efficient. to collect decoupled networks. Multiple groups of independent
Aziz Khater [14] proposed a PID controller that combines the experiences. The synchronous update of A2C will make the
ASN reinforcement learning network with fuzzy mathematics. training more coordinated, thereby potentially speeding up the
Although this method does not require many accurate training convergence; it has been proved by practice that A2C has

978-1-6654-2207-9/22/$31.00 ©2022 IEEE 2185

Authorized licensed use limited to: Thammasat University. Downloaded on November 04,2025 at 13:56:24 UTC from IEEE Xplore. Restrictions apply.
higher utilization of GPU, better effect for large batch size, are: that no accumulation is required in the formula. The
and achieves better performance than A3C in the same task. determination of the control increment ∆u(t) is only related to
In this work, combined with the multi-thread synchronous the last three sampling values, and it is easy to obtain a better
training method of A2C structure and its own model-free control effect through weighting processing; the computer
online learning ability, the PID parameters are adjusted adap- only outputs the control increment each time, that is, the
tively, a design method of adaptive PID controller based on the change amount corresponding to the position of the actuator,
synchronous advantage actor-critic is researched and proposed, so the machine When a fault occurs, the scope of influence
and the simulation experiment is carried out. The superiority is small and the production process will not be seriously
and effectiveness of this method are verified. affected; Manual-automatic switching with little impact. When
the control is switched from manual to automatic, bumpless
II. PROBLEM STATEMENT switching is possible.
In this section, we introduce the main elements that support
our proposal. Thus, in the following subsections, the basic III. A DAPTIVE CONTROLLER DESIGN
concepts utilized in this work for our problem formulation A. Temporal Difference (TD) error
will be briefly explained.
First introduce a value function, which is defined as
A. Basic structure of PID controller ∞
X
A typical control system structure is shown in Fig. 1. V (t) = γ i−t r(x(i), u(i)) (3)
i=t

where 0 < γ <= 1 is the discount factor and the u(t) control
signal. The function r(x(i), u(i)) is called the reinforcement
signal and can be chosen according to the quadratic function.

∞
X
V (t) = r(x(t), u(t)) + γ γ i−(t+1) r(x(i), u(i)) (4)
i=t+1
Fig. 1. PID control structure
Instead of evaluating the infinite sum of the above equations,
the following equivalent difference equations can be solved
Z t
de(t) using the current control signal u(t):
u(t) = kp e(t) + ki e(t)dt + kd (1)
0 dt
where u(t) is the controller output,e(t)is the error at time t, and V (t) = r(x(t), u(t)) + γV (t + 1), V (0) = 0 (5)
Kp , Ki , Kd are the proportional, integral, and derivative con- This equation is also known as the Bellman equation.
stants (also called gains) of the PID [Link] According to the Bellman equation, the TD error can be
PID is an algorithm that performs PID control through the defined as the difference between the two sides:
increment of the control quantity. The calculation formula is
as follows:
δT D (t) = r(x(t), u(t)) + γV (t + 1) − V (t) (6)
u(t) =u(t − 1) + ∆u(t)
If the Bellman equation holds, the TD error is zero. There-
=u(t − 1) + Ki (t)e(t) + Kp (t)∆e(t) (2) fore, the current control signal can be regarded as the optimal
+ Kd (t)∆2 e(t) control strategy for each time t.
where:
B. A2C adaptive PID control
′
e(t) = y (t) − y(t)
∆e(t) = e(t) − e(t − 1)
∆2 e(t) = e(t) − 2 ∗ e(t − 1) + e(t − 2)
y ′ (t), y(t), e(t), ∆e(t), ∆2 e(t) respectively represents the
current actual signal value, the current system output value,
the system output error, the first-order difference of the error
and the second-order difference of the error. Incremental PID
control mainly replaces the cumulative effect of the original
integral link by finding the increment, so as to avoid the
integral link occupying a lot of computing performance and
storage space. The main advantages of incremental PID control Fig. 2. Parallel Architecture of A2C

2186
Authorized licensed use limited to: Thammasat University. Downloaded on November 04,2025 at 13:56:24 UTC from IEEE Xplore. Restrictions apply.
 As shown in Fig.2, A2C will build multiple processes,
including multiple parallel workers, to interact with indepen-
dent environments and collect independent experiences. These
workers are synchronized, that is, in each round of training,
the Global network will wait for each worker to complete the
current episode, and then aggregate and average the gradients
uploaded by these workers to obtain a unified gradient and use
it to update the parameters of the main network, and finally
use this parameter to update all workers at the same time. It
is equivalent to adding a synchronization link on the basis of
A3C.
The agents in parallel do not interfere with each other,
and the parameter update of the main structure is realized
by uploading and summarizing the update gradients of each Fig. 4. Network structure of actor-critic
parallel agent through the sub-structure. Therefore, compared
with the Actor-Critic algorithm, the data convergence in the
According to the form of the Gaussian kernel function, the
A2C algorithm is significantly improved.
output of the jth hidden node is as follows:
!
C. Structure of A2C-PID controller 2
∥x(t) − µj (t)∥
Φj (t) = exp − , j = 1, . . . , 5 (7)
The design framework of the A2C adaptive PID controller 2σj2
is to combine the incremental PID controller with the syn-
chronous learning structure of A2C, and then combine the where: µj = [µ1j µ2j µ3j ]T is the center vector of the jth
radial basis function to improve the computational efficiency. hidden node, σj is the width vector of the jth node, the hidden
Its structure is shown in Figure 3. layer adjusts the parameters of the activation function, adopts
a nonlinear optimization strategy, and the learning speed is
slow.
The third layer is the output layer. The first three outputs
are the three components of the output K′ (t) of the Actor
part, and the output of the fourth node is the value function
V(t) of the Critic part. The outputs are as follows:
5
X
′(t)
Km = wjm (t)Φj (t), m = 1, 2, 3 (8)
j=1

5
X
V (t) = wj4 (t)Φj (t) (9)
j=1

Among them, j=1,2,...,5 is the node number of the hid-

den layer; m=1,2,3 is the node number of the output layer;
Fig. 3. Adaptive PID control diagram based on A2C learning wj 1, 2, 3 is the jth node of the hidden layer and the output
layer Actor respectively The weights between the 1st, 2nd,
and 3rd nodes; wj4 is the weight between the jth node of the
D. A2C learning with RBF networks hidden layer and the critical (4th node) of the output layer.
The radial basis function network (RBF) simulates the Actor-network directly outputs the approximate values of
neural network structure in the human brain that is locally the three parameters Kp, Ki, and Kd, and adds Gaussian
adjusted and covers the receptive fields. It has the characteris- interference to the actual values.
tics of local approximation, strong generalization ability, and Regarding the network and parameter update, first of all,
fast convergence speed, and can approximate any continuous the learning performance index of the system is defined as:
function with arbitrary [Link] network is selected as 1 2
E(t) = δTD (t) (10)
the parameterization method to realize the learning of policy 2
function and value function at the same time. This paper chooses to use the gradient descent calculation
As shown in Fig. 4, the Actor-network has three layers: method to iteratively update the weights. In the learning
The first level is the input layer. The input vector Θi (t) = method of the algorithm, the calculation of the n-step TD error
[ei (t) ∆ei (t) ∆2 ei (t)] represents the state vector. δT D is realized by the difference between the state estimated
The second layer is the hidden layer which has 5 nodes, value V (Θi (t + n), Wv′ ) of the initial state and the estimated
and the activation function is a Gaussian kernel function. value after n steps, as follows:

2187
Authorized licensed use limited to: Thammasat University. Downloaded on November 04,2025 at 13:56:24 UTC from IEEE Xplore. Restrictions apply.
 δT D = qt − V (Θi (t), Wv′ ) (11)
qt =rt+1 + γrt+2 + · · · + γ n−1 rt+n + γ n
(12)
V (Θi (t + n), Wv′ )
The γ represents the discount factor used to determine the
ratio of delayed and immediate benefits. Wv′ is the weight of
the Critic network. The TD error δT D reflects the quality of
the selected actions in the actor network.
IV. E XPERIMENTS
A. Introduction of the simulation environment
In this paper, we test our algorithm in the self-built PID
simulation environment of the OpenAI gym module. In each
episode, the PID environment is initialized, the three parame-
ters are initialized to 0, and the reward for each control step is
defined as equation 7. Whenever the reward function value is
Fig. 5. The reward of the A2C-PID controller
0, that is, the error value is 0, it means that the current episode
ends.
B. Training The A2C algorithm creates multiple parallel environments,
and each parallel environment runs Actor-Critic at the same
The numerical simulation and comparative study are con- time, allowing multiple agents with secondary structures to
ducted in this section in order to evaluate the efficiency and simultaneously update the parameters in the main structure
feasibility of the proposed scheme. Demonstrate the control on these parallel environments. The agents in parallel do not
effect by taking the first-order inertial system as an example. interfere with each other, and the parameter update of the main
To start the training process, we randomly read Batch-size=128 structure is realized by uploading and summarizing the update
transformed samples from memory to train the neural network gradients of each parallel agent through the sub-structure. As a
in A2C, the learning rate of the actor is 0.001 at the beginning, dynamic variable parameter PID control algorithm, it has the
the learning rate of the critic is 0.01, and the learning rate feasibility, effectiveness, and universality of parameter self-
of both will decay exponentially. Train A2C-PID for 2000 tuning and optimization algorithm. By utilizing the multi-
episodes, respectively, and set the maximum number of steps thread parallel feature, the invention asynchronously trains
in set T to 500. multiple agents, undergoes different learning processes, and
C. Performance of the controller breaks the correlation between samples.
Figure 5 shows the discounted reward value curve of the ACKNOWLEDGMENT
A2C-PID controller. It can be seen that after 2000 iterations
of training, the reward value of A2C-PID gradually tends to a This work was supported by the Huadian group science and
stable state after about 250 iterations of training. technology project under Grant No.CHDKJ21-02-138.
The proposed scheme can work well given the reference R EFERENCES
signal, and eventually tends to reach a constant value, and
the DQN-PID overshoot is significantly larger than that of [1] Åström K J, Hägglund T. PID controllers: theory, design, and tuning[M].
ISA-The Instrumentation, Systems and Automation Society, 1995.
the proposed scheme, which indicates that the new update [2] Ziegler J G, Nichols N B. Optimum settings for automatic controllers[J].
rule works well within a certain range. The simulation results trans. ASME, 1942, 64(11).
show that the A2C-PID controller can effectively improve the [3] Chien K L. On the automatic control of generalized passive systems[J].
Trans. Asme, 1972, 74: 175-185.
large overshoot, nonlinearity, and hysteresis which has better [4] VanDoren V J. Adaptive controllers work smarter, not harder[J]. Control
adaptive ability than DQN-PID. Engineering, 2002, 49(10): 22-24.
[5] Petrov M, Ganchev I, Taneva A. Fuzzy PID control of nonlinear
V. CONCLUSIONS plants[C]//Proceedings First International IEEE Symposium Intelligent
Systems. IEEE, 2002, 1: 30-35.
Aiming at the problem that the PID controller cannot adjust [6] Tang, H. C., Li, Z. X., Wang, Z. T., et al. (2005). A fuzzy PID control
the parameters online, this paper proposes a control parameter system. Electric Machines and Control, 2, 136–138.
optimization algorithm based on reinforcement learning. By [7] Kang J, Meng W, Abraham A, et al. An adaptive PID neural network
for complex nonlinear system control[J]. Neurocomputing, 2014, 135:
combining the reward mechanism of the multi-threaded ad- 79-85.
vantageous action evaluation algorithm, the rapid learning of [8] Chee F, Fernando T L, Savkin A V, et al. Expert PID control system
RBF and the dynamic performance index evaluation module for blood glucose control in critically ill patients[J]. IEEE Transactions
on Information Technology in Biomedicine, 2003, 7(4): 419-425.
of the control system, online self-tuning, and optimization of [9] Zhang J, Zhuang J, Du H. Self-organizing genetic algorithm based tuning
controller parameters. of PID controllers[J]. Information Sciences, 2009, 179(7): 1007-1018.

2188
Authorized licensed use limited to: Thammasat University. Downloaded on November 04,2025 at 13:56:24 UTC from IEEE Xplore. Restrictions apply.
[10] Acosta G G, Mayosky M A, Catalfo J M. An expert PID controller
uses refined Ziegler and Nichols rules and fuzzy logic ideas[J]. Applied
Intelligence, 1994, 4(1): 53-66.
[11] Porter B, Jones A H. Genetic tuning of digital PID controllers[J].
Electronics Letters, 1992, 28(9): 843-844.
[12] Rossomando F G, Soria C M. Identification and control of nonlinear
dynamics of a mobile robot in discrete time using an adaptive technique
based on neural PID[J]. Neural Computing and Applications, 2015,
26(5): 1179-1191.
[13] Jafari R, Dhaouadi R. Adaptive PID control of a nonlinear servomecha-
nism using recurrent neural networks[J]. Adv. Reinforcement Learning,
2011: 275-296.
[14] Aziz Khater, A., El-Bardini, M., El-Rabaie, N. M. (2015). Embedded
adaptive fuzzy controller based on reinforcement learning for DC motor
with flexible shaft. Arabian Journal for Science and Engineering, 40(8),
2389–2406.
[15] Akbarimajd, A. (2015). Reinforcement learning adaptive PID controller
for an under-actuated robot arm. International Journal of Integrated
Engineering, 7(2), 20–27.
[16] Chang WD, Hwang R C, Hsieh J G. A multivariable on-line adaptive
PID controller using auto-tuning neurons[J]. Engineering Applications
of Artificial Intelligence, 2003, 16(1): 57-63.
[17] Liao Y, Koiwai K, Yamamoto T. Design and implementation of a
hierarchical-clustering CMAC PID controller[J]. Asian Journal of Con-
trol, 2019, 21(3): 1077-1087.
[18] Hou Z, Chi R, Gao H. An overview of dynamic-linearization-based
data-driven control and applications[J]. IEEE Transactions on Industrial
Electronics, 2016, 64(5): 4076-4090.
[19] Yongquan Y, Ying H, Bi Z. A PID neural network con-
troller[C]//Proceedings of the International Joint Conference on Neural
Networks, 2003. IEEE, 2003, 3: 1933-1938.
[20] LeCun Y, Bengio Y, Hinton G. Deep learning[J]. nature, 2015,
521(7553): 436-444.
[21] Arulkumaran K, Deisenroth M P, Brundage M, et al. A brief survey of
deep reinforcement learning[J]. arXiv preprint arXiv:1708.05866, 2017.
[22] Mnih V, Badia A P, Mirza M, et al. Asynchronous methods for
deep reinforcement learning[C]//International conference on machine
learning. PMLR, 2016: 1928-1937.
[23] Kyriakides G, Margaritis K G. Neural architecture search
with synchronous advantage actor-critic methods and partial
training[C]//Proceedings of the 10th Hellenic Conference on Artificial
Intelligence. 2018: 1-7.

2189
Authorized licensed use limited to: Thammasat University. Downloaded on November 04,2025 at 13:56:24 UTC from IEEE Xplore. Restrictions apply.