R2-17 SASIMI 2016 Proceedings

An Overlay Architecture for FPGA-based Industrial Control Systems

Designed with Functional Block Diagrams

Taisei Segawa1 , Yuichiro Shibata1 , Yudai Shirakura1 , Kenichi Morimoto1 , Hidenori Maruta
1
, Fujio Kurokawa1 , Masaharu Tanaka2 , and Masanori Nobe3
1
Nagasaki University 1-14, Bunkyo-machi, Nagasaki,852-8521 Japan
2
Mitsubishi Heavy Industries, LTD. 2-16-5 Konan, Minato-ku, Tokyo 108-8225 Japan
3
Mitsubishi Hitachi Power Systems, LTD. 3-3-1 Minatomirai, Nishi-ku, Yokohama, 220-8401 Japan

Abstract— This paper discusses FPGA implemen- can be inherited and enables long-term utilization of the
tation of industrial control logic described in a func- design, even when physical FPGA chips are exchanged.
tion block diagram (FBD) language. First, we evalu- Also, parallelism included in FBD descriptions will be ex-
ate an approach where FBD descriptions are directly ploited on FPGAs, by using a high-level synthesis (HLS)
translated to FPGA hardware using a high level syn- technique.
thesis technique. Second, aiming at improving re- Although research attempts that translate FBD de-
source utilization efficiency, we proposed an overlay scriptions to HDL descriptions using HLS have been car-
architecture which helps resource sharing of the same ried out [1][2][3], it has not been well addressed how these
arithmetic structure utilized in different control logic designs should be implemented on FPGAs in terms of per-
sheets. Evaluation results show that the proposed ar- formance and resource utilization. In this paper, we dis-
chitecture can significantly reduce resource require- cuss FPGA implementation of control logic described in
ments per control logic sheet, at a cost of acceptable an FBD language. We evaluate two approaches: (1) direct
performance degradation. conversion from FBD to FPGA hardware using HLS and
(2) introduction of an overlay architecture on the FPGA.
While many overlay architectures have been proposed to
I. INTRODUCTION speedup application mapping time on FPGAs [4][5][6][7],
our main aim of introducing an overlay architecture is to
Recently, demands for highly responsive real-time OS, improve efficiency of resource utilization.
highly intelligent communication, and high performance This paper is organized as follows. Section II explains
control operations are increasing in control systems for in- about FBD languages. Section III describes direct imple-
dustrial infrastructure such as thermal power generation mentation of control logic described in FBD and shows
plants. Although a standard solution to the demands is evaluation results. Section IV presents a proposed over-
multicore CPUs, their complex architecture is not neces- lay architecture. Section V explains how the proposed
sarily suited to the industrial control systems. architecture is implemented. Section VI discusses evalu-
Generally, long-term utilization of the same CPU ar- ation results on performance and resource utilization of
chitecture is difficult, since product cycles of CPUs are the proposed architecture. Finally, Section VII concludes
shorter than those of plant equipments. When the CPU the paper.
architecture installed in industrial control systems is ex-
changed with new architecture, often software applica-
tions and OS are also needed to be revised due to archi- II. FBD (Function Block Diagram)
tectural changes such as the number of cores. Note that
dedicated OS needs to be often used for industrial con- FBD is a graphical diagram, where instruction blocks
trol systems. Regarding development environments for (function blocks), which are reusable functional elements,
industrial control systems, function block diagram (FBD) and flows of data signals between function blocks are
languages are widely utilized. Usually, FBD descriptions described. FBD languages are also widely utilized for
are translated to standard programming languages such program description of programmable logic controllers
as C, and then executed sequentially on a CPU. How- (PLCs). In this work, we use a sort of FBD language
ever, this design and execution flow does not exploit any called IDOL, which was developed by Mitsubishi Hitachi
inherent parallelism in control logic. Power Systems for their DIASYS Netmation control sys-
As a solution to these problems, FPGAs are attract- tems.
ing attention. A logical design layer of FPGA circuits Typically, control logic for industrial plants consists of a

- 164 -
 (a) Control logic
(b) Plant model

Fig. 2. Control logic and plant model of PI control

Fig. 1. Loop logic and a sheet

(a) Control logic 1 (b) Control logic 2

number of feedback control loops. In IDOL, each feedback

loop is called loop logic, and is basically drawn in one
sheet as shown in Figure 1. (c) Plant model

Fig. 3. Control logic and plant model of PI control

III. Direct Translation from FBD to FPGA
by HLS
We implemented two simple control logic loops with example. Although IDOL scripts are similar to C descrip-
HLS to evaluate performance and resource utilization tions, IDOL has a dedicated grammar and has original
of direct translation from FBD to FPGA hardware: variable types. Therefore, to absorb a grammatical gap
proportional-integral (PI) control logic and on-off control between the IDOL scripts and HLS languages, we imple-
logic for controlling a liquid level in a water tank. mented a class library where IDOL original types and sys-
Figure 2 (a) shows diagram of the evaluated PI control tem functions are defined as C++ classes. By using this
loop, and Figure 2 (b) shows an assumed plant model as class library, IDOL scripts can be interpreted as standard
a control target. Based on input data such as the current C++ descriptions to be processed with HLS tools. In
liquid level, the PI loop logic controls the valve aperture this evaluation, Xilinx Vivado-HLS 2015.2 was utilized for
to keep the liquid level at the desired level. The AI and DI high-level synthesis. Single-precision floating point arith-
blocks shown in Figure 2 are used for data input, while the metic was utilized for real number calculations. The con-
AO block is used for data output. The other blocks such trol logic was mapped on a Xilinx Kintex-7 XC7K325T
as FX and PI are arithmetic blocks. In this evaluation, FPGA with a clock frequency constraint of 100 MHz. The
the target liquid level was set to 5,000. control target models were also described in C++ and
As Figure 3 illustrates, the on-off control logic consists mapped on the same FPGA to emulate the plant behav-
of two sheets of control loops, since its plant model has ior.
two control inputs, a water inlet valve and an outlet valve.
Comparing the current liquid level to predefined threshold B. Evaluation
levels, each control loop decides whether to open or close
the corresponding valve. In this evaluation, the inlet valve Figure 5 and Figure 6 show results of the emulation
was controlled to be opened and closed when the level experiments. For the PI control, the tank level converged
falls below 2,000 and exceeds 9,000, respectively. For the at the target level of 5,000. We confirmed the behavior of
outlet valve, the two threshold levels were set to 3,000 and the control logic was appropriately changed by parameters
8,000, respectively. The control cycle time of the systems of PI control. A ’k’ and ’t’ in Figure 5 show a proportional
was assumed to be 50 msec, which is a typical case in gain and a time constant, respectively.
current industrial plants. Also from the results of the on-off control, it is con-
firmed that the inlet and outlet valves were surely opened
and closed according to the designed threshold levels. The
A. Implementation
average arithmetic execution time per control loop was
In the DIASYS IDOL system, a script is prepared for 0.93 μsec for the PI control and 0.02 μsec for the on-off
each function block, which is eventually executed on a control, respectively. Considering a typical calculation
CPU. Figure 4 shows the script of the HMH block as an cycle time for many plant equipments is a few msec, the

- 165 -
 9000
8000
/************************ HMH*/ 7000

Tank Level
6000
5000
void SCR_HMH(Ain X,Ain H,Ain D,Dout Y) 4000
3000
{ 2000
1000
if(IS_MA_SET(Q(Y))) return; 0
0 100 200 300 400 500

1
if(D<0.0){

Inlet Valve Open

0.8

Y=1; 0.6
0.4
SET_CALC_ERROR(Q(Y));
0.2
return; 0
0 100 200 300 400 500
}
1

Outlet Valve Open

0.8
if (X>H) Y=1; 0.6
else if(X<=H-D) Y=0; 0.4

} 0.2
0
0 100 200 300 400 500

Fig. 6. Emulation results of the on-off logic

Fig. 4. Script of HMH block

6000
k=5, t=100
k=1, t=100
TABLE I
k=10, t=500
Resource utilization
5000

4000
PI Control On Off Control
Tank Level

Resource Utilization Utilization

3000 Utilization Utilization
Rate[%] Rate[%]
2000
SLICE 906 1.78 104 0.204
LUT 3093 1.52 332 0.162
1000
FF 1934 0.474 17 0.00417
DSP 12 1.42 0 0
0
BRAM 2 0.224 0 0
0 5000 10000 15000 20000 25000 30000 35000 40000 45000 50000
Simulation Step

Fig. 5. Emulation result of the PI logic

As shown Figure 7, processing elements (PEs) needed
for desired control logic are placed to form an array of
(n × m). The PEs in the same row and column are con-
control logic generated by HLS has a large performance nected with horizontal buses and vertical buses, respec-
advantage, more than three orders of magnitude. tively. For simplicity of the layout, PEs of the same type
Table I shows the amounts of FPGA resources con- of function block are placed in the same column. That
sumed by each control logic. When attention is paid is, n corresponds to how many different types of function
to the PI control, a total of 906 SLICEs, each of which blocks, and m corresponds to the maximum number of
consists of 4 look-up tables (LUTs) and flip-flops (FFs), PEs of the same type of function block are utilized.
were needed to implement this control logic. Since a A PE that finishes its calculation transmits the calcu-
Kintex-7 XC7K325T FPGA provides 50,950 SLICEs, this lation result to the PE that will use the result for the
FPGA can implement 56 sheets of PI control loops on the next operation. The data transmission is performed by
chip. However, recent control systems for large-scale en- the 2-D buses in the following manner. Here, we denote
ergy plants execute about 1,000 sheets of logic. Since the horizontal axis as x-axis and the vertical axis as y-axis
even a high-end Virtex-7 XC7V200T FPGA, which of- for explanation.
fers 30,5400 SLICEs, can implement only 337 sheets, it is
essential to improve area efficiently of the system. Con- 1. A PE which finished its calculation broadcasts the
sidering that the calculation speed of the control logic result to all the PEs on the same horizontal bus. The
is fast enough for the required performance, we propose coordinate of the destination PE is appended in the
an architecture where arithmetic hardware for multiple header of the packet.
loop logics is not simply expanded in space, but hardware
resources for the same function blocks used in different 2. Each PE that receives the broadcast data checks the
sheets are shared and reused. destination coordinate of the data. Only the PE that
has the same x-coordinate with the destination re-
tains the data, while the other PEs just discard the
IV. PROPOSED ARCHITECTURE data.

In this section, we explain a proposed architecture, 3. The PE which retains the data broadcasts the data
which focuses on sharing of hardware resources. to all the PEs on the same vertical bus.

- 166 -
 Fig. 9. Transmission to vertical direction

Fig. 7. Proposed architecture

Fig. 10. Execution order controller

Fig. 8. Transmission to horizontal direction

B. PE (Processing Element)
Figure 11 shows the structure of a PE in the proposed
architecture. The PE consists of a router, a register file,
4. Each PE that receives the data verifies the destina- and execution module. The router controls data trans-
tion, and only the PE that has the same y-coordinate mission and bus interfaces. In concrete, the router checks
with the destination retains the data. the destination of transmitted data and decides whether
it accepts or discards the data. When the execution mod-
For example, when the PE (0,1) transmits data to the ule finish its calculation, the router appends the routing
PE (j,k), first, the sender PE broadcasts the data on header to the result data and sends the packets. When
the same horizontal bus and the PE (j,1) retains data as a PE receives data, it is stored in the register file. The
shown in Figure 8. Next, the PE (j,1) sends the data on register file can hold data individually for each sheet and
the vertical bus as shown in Figure 9, and lastly PE (j,k) each input port, so that the functionality of the execu-
retains the data. In this way, communication between tion module is shared by different sheets. When a PE is
any combinations of PEs can be completed in two-hop activated by the execution controller, execution module
routing. reads required data from the register file and starts its
calculation.
A. Execution Order Controller
V. IMPLEMENTATION OF PROPOSED
All PEs are connected to an execution order controller, ARCHITECTURE
which controls the order of PE execution as shown Fig-
ure 10. The execution controller activates the PE to be Figure 12 and Figure 13 illustrate how the control logic
executed next by sending the coordinate of the PE. The shown in Figure 2 and Figure 3 were implemented on the
execution controller also sends the sheet number to be ex- proposed overlay architecture, respectively.
ecuted. The order of PE execution is stored in the table in To simplify the structure and routing logic in the archi-
the execution order controller, whose contents are written tecture, we placed different types of PEs along the hori-
when an application is mapped on the architecture. zontal direction and the same types of PEs along the ver-

- 167 -
 6000 9000
8000
7000
5000

Tank Level
6000
5000
4000 4000

Tank Level
3000
2000
3000 1000
0
2000 0 2000 4000 6000 8000 10000

1000 1

Inlet Valve Open

0.8
0
0 1000 2000 3000 4000 5000 6000 7000 0.6
0.4
10000
0.2
8000 0
0 2000 4000 6000 8000 10000

Valve Aperture
6000 1

Outlet Valve Open

0.8
4000
0.6
0.4
2000
0.2
0 0
0 1000 2000 3000 4000 5000 6000 7000 0 2000 4000 6000 8000 10000

(a) PI logic (b) On-off logic

Fig. 14. Emulation results of the proposed architecture

TABLE II
a Quantity of resource utilization of a proposed
architecture
Fig. 11. Inside of the PE

PI Control On Off Control
Resource Utilization Utilization
Utilization Utilization
Rate[%] Rate[%]
SLICE 4126 8.01 1154 2.26
LUT 12292 1.76 3595 1.76
FF 10883 0.566 2308 0.566
DSP 50 5.95 16 1.90
BRAM 76 17.1 19 4.27
Fig. 12. Implementation of the PI logic on the overlay architecture

VI. EVALUATION

tical direction, forming a rectangular array. The number Figure 14 show the emulation results of the control logic
of PEs in the vertical direction, which was four in both implemented on the proposed overlay architecture. For
examples in this evaluation, corresponds to the number of both examples, it was confirmed that the control logic
the PEs of the type with the most usage. Each number worked appropriately to regulate the tank level.
at the upper left corner of PEs shows the coordinate of Table II shows resource utilization for both control
the PE. The router module and the register file for each logic. Compared with the HLS implementation shown
PE and the execution order controller were designed in a in Table I, BRAM utilization increased by 30 times and
register transfer level with Verilog-HDL. RTL designs for other resources increased by 4 to 5 times. This increase
execution module were translated from IDOL scripts us- in the resources is mainly due to newly added mechanism
ing HLS. In this evaluation, the depth of the register file such as the routers and register files in PEs. In addition,
was set to 4096, so that 4096 sheets of control logic can since we formed a rectangular array of PEs to simplify
be executed. As in the case of the implementation using the architecture, some PEs were not utilized for the con-
only HLS, single-precision floating point arithmetic was trol logic. Also, unlike HLS implementations, the resource
utilized with the target clock frequency of 100 MHz. Fi- sharing between different PEs cannot be performed with
nally, the overlay architecture was mapped on a Kintex-7 this architecture.
XC7K325T FPGA with Xilinx Vivado 2015.2. However, the proposed architecture can process 4096
sheets of control logic, in contrast to the HLS implemen-
tation that can execute only one sheet. Table III shows
comparison results of resource utilization for PI control
per one sheet. It is shown that, utilization efficiency of
BRAM was improved by about 100 times for BRAM. Also
for other resources, improvement of approximately 800
times was achieved by introducing the overlay architec-
ture.
To analyze the hardware overhead imposed by the addi-
Fig. 13. Implementation of the on-off logic on the overlay tional mechanisms, we evaluated the breakdown of the re-
architecture source usage, which is summarized in Table IV. This time,
all BRAMs were utilized for the register files in PEs, since
any function blocks in the evaluated control logic did not
use BRAMs. For other resources, about 80 % were de-
voted to the arithmetic circuits. It is revealed that the

- 168 -
 formance and resource utilization between the proposed
TABLE III
Comparison resource utilization for PI control per one architecture and HLS direct implementation using two ex-
sheet amples of control logic.
As a result of evaluation, it was shown that the pro-
posed architecture still achieved practical control perfor-
Proposed HLS Utilization Efficiency
Resource
Architecture Implementation Improvement
mance while an overhead of data communication degraded
SLICE 1.01 906 897.0 the performance to some extent. On the other hand, the
LUT 3.00 3093 1031 resource sharing mechanism of the proposed architecture
FF 2.66 1934 727.1 improved the resource requirements per sheet by about
DSP 0.0122 12 983.6
100 times for BRAM and by about 800 times for the other
BRAM 0.0186 2 107.5
resources.
As future work, we will evaluate the architecture with
TABLE IV more practical control logic sheets that are utilized in ac-
Breakdown of resource utilization used overlay tual industrial plants. Although the proposed architec-
architecture
ture does not exploit inter-PE parallelism at this moment,
we will address improvement of the architecture so that
Resource SLICE[%] LUT[%] FF[%] DSP[%] BRAM[%] logic sheets that have no dependency each other are exe-
Calculation 79.8 90.9 93.0 100 0 cuted in parallel.
Routerɾ
19.2 8.3 6.9 0 100
Register File
Execution Order
1.0 0.8 0.1 0 0
Controller References
[1] D. A. Lee, E. s. Kim, J. Yoo, J. S. Lee, and J. G. Choi. Fb-
dtoverilog 2.0: An automatic translation of fbd into verilog
resource overhead of the proposed overlay architecture is to develop fpga. In Proc.2014 International Conference on
about 20 % in terms of SLICEs. Information Science Applications (ICISA), pages 1–4, May
Next, we evaluated the execution performance of the 2014.
control logic implemented on the proposed architecture. [2] J. Yoo, J. G. Choi, Y. J. Lee, and J. S. Lee. A technique for
The calculation times per control loop (minimum control demonstrating safety and correctness of program transla-
cycles) of the PI control and on-off control were 2.34 μsec tors: Strategy and case study. In [Link] Reliability
and 1.66 μsec, respectively. The performance degradation Engineering Workshops (ISSREW), 2014 IEEE Interna-
tional Symposium on, pages 210–215, Nov 2014.
compared to the HLS direct implementation was 3 times
for the PI control and 83 times for the on-off control. The [3] J. Yoo, E. S. Kim, D. A. Lee, J. G. Choi, Y. J. Lee, and
main reason is the data transfer on the two-dimension J. S. Lee. Nude 2.0: A model-based software development
buses, which is required for every interval between PE environment for the plc amp; fpga based digital systems
in nuclear power plants. In Proc.2014 International Sym-
executions. Especially in the on-off control logic, this
posium on Integrated Circuits (ISIC), pages 604–607, Dec
overhead was significant, since the time required for arith-
2014.
metic execution for this logic was originally quite small.
Another cause is the proposed architecture executes PEs [4] A. K. Jain, S. A. Fahmy, and D. L. Maskell. Efficient
overlay architecture based on dsp blocks. In [Link]-
sequentially and does not extract parallelism from differ-
Programmable Custom Computing Machines (FCCM),
ent PEs, while the HLS direct implementation spatially
2015 IEEE 23rd Annual International Symposium on,
expands arithmetic hardware so that inter PE parallelism pages 25–28, May 2015.
is fully exploited.
[5] A. K. Jain, D. L. Maskell, and S. A. Fahmy. Through-
However, the proposed architecture achieved the con-
put oriented fpga overlays using dsp blocks. In Proc.2016
trol cycle of several micro seconds, which is still several or-
Design, Automation Test in Europe Conference Exhibition
ders of magnitude faster than the required performance in (DATE), pages 1628–1633, March 2016.
general industrial control systems. Therefore, the perfor-
[6] G. Stitt and J. Coole. Intermediate fabrics: Virtual archi-
mance degradation observed by the proposed architecture
tectures for near-instant fpga compilation. IEEE Embedded
is considered to be acceptable as a trade-off for the 800
Systems Letters, 3(3):81–84, Sept 2011.
times improvement in the resource utilization efficiency.
[7] D. Capalija and T. S. Abdelrahman. A high-performance
overlay architecture for pipelined execution of data flow
VII. Conclusion graphs. In Proc.2013 23rd International Conference on
Field programmable Logic and Applications, pages 1–8,
In this paper, we proposed an overlay architecture fo- Sept 2013.
cusing on sharing of hardware resources, in which the re-
sources for the same function blocks that are utilized in
different sheets of control logic. We compared the per-

- 169 -