Distributed Industrial Robot Control using

Environment Perception and

Parallel Path Planning Cloud Services
Jonas Wassermann∗ , Vojtěch Vonásek† and Axel Vick‡
∗ Department of Industrial Automation Technology, Technische Universität Berlin, Germany, wassermann@[Link]
†Czech Technical University in Prague, Faculty of Electrical Engineering, Czech Republic, vonasek@[Link]
‡ Automation Technology Division, Fraunhofer IPK Berlin, Germany, [Link]@[Link]

Abstract—Recent developments in connected industries and II. R ELATED W ORK

internet of things identified demands for flexible reconfiguration
and reprogramming of robots and machine tools. The demand The trend for distributed control of machinery and robots is
for reconfiguration comes from changes in production process driven by the digitization and networking in the manufacturing
ordering or individual products; the demand for reprogramming
comes from changing workplace organization and material flow. field. The products become customized and individualized,
Yet these reconfiguration and reprogramming is often charac- the supply-chain is fully connected and the use of artificial
terized by constants for a specific use-case in terms of pre- intelligence in factories is investigated [1]. The performance of
computed trajectories. In this paper, we present an approach of the remote computing infrastructure as well as the connecting
monitoring the robot’s workspace and using an online replanning network is crucial for industrial control processes from the
of motion. We present a toolchain that is available and ready
to use for a big class of industrial robots that have a position cloud. Special studies on close-loop control behavior with
setpoint control interface. The feasibility is demonstrated in small uncertain latency conditions were presented by [2]. Authors
laboratory experiments with a modular industrial robot in a use an industrial robot dynamic model for a predictive control
common human-robot interface scenario. algorithm that is compensating for network packets delay or
Index Terms—Distributed Control, Industrial Robots, Path dropout during communication between the cloud controller
Planning
and the robot. Another network-adaptive approach by [3] is
providing resource or connectivity related reaction by shaping
I. I NTRODUCTION the control parameters at runtime. The outsourcing of robotic
path planning to the cloud execution was shown in [4].
The programming of industrial robots and machine tools is
From the environment perception point of view, there is
still a difficult task for experts. Recent developments in human-
much work available from the mobile robotics community. A
machine interaction like programming-by-demonstration and
multi-robot applications using simultaneous localization and
augmented reality allow the users to quickly and easy pro-
mapping (SLAM) is described in [5]. It is used to coordinate
gram or visualize the robotic tasks. For using these tools
and plan actions for systems of multiple mobile robots. En-
in a distributed robot control system, additional effort for
vironment perception in industrial production becomes more
modularization and virtualization is required. The environment
important with recent development in human-robot interaction.
perception methods as well as the path planning methods
In [6], a method for environment perception for human robot
need to be implemented completely independent from each
collaboration using multiple cameras is described. An ap-
other and from the sensor and robot hardware in terms of
proach for real-time collaborations and active collision avoid-
robot control services. Together with a suitable communication
ance for human-robot collaboration is proposed in [7]. The
framework, the services are able to exchange data and be
environment perception is achieved by background removal
combined into a comprehensive application.
and masking of the robot via a 3D model parameterized by
In this paper, we use a distributed robot control architecture joint positions from the real robot.
for single path planning and we expand it to an adaptive
Our approach relies on augment reality techniques to vi-
programming system. The system is able to detect changes
sualize the detected objects and trajectories of the robot.
in the robots environment and react to them by replanning the
Augmented Reality is used also in [8] for the visualization of
robot’s trajectory to avoid potential collisions.
the safe zones around an industrial robot or for highlighting
parts of the robot by color and basic interactions like adjusting
The work has been supported by Czech Ministry of Education under
Mobility projects No. 7AMB17DE026 and 7AMB17DE025. and German axis values. In [9], a concept for programming industrial robots
Academic Exchange Service (DAAD) under PPP project No. 57334517. using augmented reality with mobile devices is shown. The

978-1-5386-7108-5/18/$31.00 ©2018 IEEE 1263

 environment and re-plan the robot motion. All parts are
designed to run independently and can be deployed to different
execution environments like local PC, server, virtual machines
or cloud instances.

The service-oriented architecture of our system is shown

in Fig. 1. It consists of hardware services (Fig. 1, red)
and control services (Fig. 1, green). The hardware services
Fig. 1. Distributed System Architecture Overview with Services for Environ- are responsible only for the hardware dependent tasks and,
ment Perception, Path Planning, Visualization and Robot Control
therefore, they cannot be distributed to cloud environments.
They need to run near the hardware and have access to
robot’s movement is shown by displaying a virtual robot but the electric interface of the devices (e.g. USB, fieldbus). In
the trajectory is not visualized. contrast, the control services are responsible for analyzing the
A crucial part of this approach is motion planning. Mo- sensor data, calculating interpretations, deciding on behavior
tion planning of robotic manipulators leads to search of a and preparing actions. The visualization services (Fig. 1, blue)
high-dimensional configuration space, which is often realized are responsible for interaction with the human operator and
using sampling-based methods like Probabilistic roadmaps they live in both previous worlds. As they are hardware-
(PRM) [10] and Rapidly Exploring Random Tree (RRT) [11]. dependent to use displays and input devices, they belong to
RRT incrementally builds a tree of collision-free configura- the group of hardware services on one hand. On the other
tions. In each iteration, a random sample is generated in the hand, they can also run on almost any device and are freely
configuration space and its nearest node in the tree is found and deployable in different execution environments.
expanded towards the random sample. If the expansion leads to
The content of each service is now summarized briefly: The
a new collision-free configuration, this configuration is added
2D Camera is a standard USB Color Webcam with Full HD
to the tree. The search continues until the goal configuration
Resolution. The 3D Camera is a Microsoft Kinect V2 Time-of-
is reached.
Flight Depth sensor and Color Camera. The Marker Tracking
To speed up the expansion step, Task-space RRT biases
is done with the ALVAR framework. The Object Detection
the search using a low-dimensional task-space [12]. The task-
contains PointCloud segmentation, Clustering, Filtering and
space can be defined e.g. as the space of 3D positions of the
Oriented Bounding Box generation. The Coordinate Transfor-
end-effector. Many configurations are related to the same end-
mation combines the frames of 2D Camera, 3D Camera and
effector pose. Therefore, searching the nearest neighbors in
robot coordinate system to achieve a common representation
the task-space increases the chance that the tree reaches the
of the objects. That common representation is used to display
goal region, which is given by a subset of the configuration
the detected objects and the robot trajectory in the augmented
space and not only one configuration.
reality as well as for the calculation of bounding boxes in
Parallelization using multiple computing cores is often used
the robots task space. The Path Planning computes the motion
to speed up the planning process. In AND parallelization, mul-
trajectory for the robot, avoiding the detected objects. In fact,
tiple cores contribute to the same data structure which leads
the path planning service itself is split into sub-services and
to its faster construction [13]. In OR parallelization, each core
parallelized for the purpose n of reducing calculation time. The
solves the task separately and the whole process is terminated
Robot Control is handling the motion of the robot in terms of
after a core finds a feasible solution. Another approach is to
applying the desired joint angles to the motors.
divide the configuration space to several (possibly overlapping)
regions that are searched simultaneously on the computing
nodes [14], [15]. A master-slave architecture for distributed
planning was proposed in [13]. The configuration tree is
maintained only by the master who samples the configuration
space and selects nodes for the expansion. The expansion, that
is computationally demanding, is realized on the computing
cores who sent back to the master new nodes that expands
the tree. The master updates the main tree and informs other
computing cores (slaves) about the changes in the tree. A
similar technique is used in our system.
III. C LOUD - BASED I NDUSTRIAL ROBOT C ONTROL
A. System Architecture
Fig. 2. Left: 2D-Camera Image of robot environment with the augmented
Our Distributed Control System approach consists of three reality visualization of the detected objects (green boxes) and the robot base
major parts: Environment Perception, Parallel Path Planning coordinate system. Right: Combined Depth and Color Image of Kinect V2
and Robot Control. The task is to react to changes in the with the marker coordinate system and the detected objects (blue boxes)

1264
B. Intelligent Environment Perception α ε
To generate data for the environment perception, we in-
stalled a depth-image camera (Microsoft Kinect 2) above the
worktable of the industrial robot. The Camera is connected p p
1p q 1
to a computer that operates the camera through the open q 2 p init
init 3
source driver libfreenect2 and the ROS package iai kinect2. q
goal
The 2D marker is searched and located in the point cloud
using the ROS package ar track alvar. The marker position in a b
the camera image is provided to other ROS nodes in a topic. Fig. 3. Example of splitting the task using PRM. First, a PRM roadmap is
Another ROS node processes the point cloud with the Point built and a coarse path from the start qinit to the goal qgoal is found. The
Cloud Library. Image processing is performed in a series of path consists of five waypoints (qinit , p1 , p2 , p3 , qgoal ) and it is computed
using the resolution α (a). Each path segment is sent to a RRT service to
operations: obtain a fine plan with the resolution ε. Example of a RRT configuration tree
• First, the point cloud is filtered to remove excess infor- from qinit to p1 with a highlighted path (green) (b).
mation. A threshold filter limits the point cloud to the
area around the industrial robot. In this step, the point
collisions. After a path is found in the roadmap, the edges in
cloud is reduced from 217088 points to a mean of 31439
the path are checked for collision and if necessary, the path
points.
is recomputed. To further speed up the construction of the
• Next, the color data of the point cloud gets transformed
PRM roadmap, the collision-detection of the edges is realized
from RGB to HSV. After that, all points in the point cloud
using a user-defined resolution α. The runtime of the PRM
that are not green are filtered out.
computation decreases with the increasing α.
• In the next step, all remaining points are clustered.
To obtain the final motion plan, the waypoints have to
• The clusters are then transformed from the camera co-
be connected using a fine-resolution planner. Therefore, the
ordinate system into the base coordinate system of the
pairs of the consecutive waypoints (pi , pi+1 ) are sent to Slave
robot.
nodes. The fine-resolution planner is realized using the RRT-
• In order to transmit the separated clusters with little data
TS planner [12] running at the fine resolution ε, ε < α. The
traffic, oriented bounding boxes (OBB) are computed
process of splitting the planning task using the waypoints
around each cluster.
is depicted in Fig. 3. The short trajectories between each
The whole image processing process takes between 12 and consecutive pair of waypoints are then connected by the master
101 milliseconds with a mean of 46 milliseconds. to a single, final trajectory. The trajectory is then sent to the
With Rviz, a visualization node in ROS, the detected boxes robot controller.
are painted into the 2D-cameras image, creating an augmented The advantage of this master-slave architecture is that the
reality view. While moving the 2D-camera around an object, amount of information exchanged between the master and the
the OBB can be visually checked (see Fig. 2). In addition, other nodes is minimal. Only the definition of the task (i.e.,
it is possible to draw the computed trajectory of the robot only the waypoints pi , pi+1 ) is sent to the slave nodes, while
movement. the slave node responds only by the computed trajectory. The
information about configuration trees built by the slave nodes
C. Parallel Path Planning
is not exchanged. Moreover, no communication is required
Trajectories for the robot are computed using parallelized between the slaves. Splitting the planning task to several
sampling-based motion planning implemented in a master- waypoints brings also another effect. As the waypoints are
slave architecture. A configuration q ∈ C of the robot describes constructed using PRM, they consecutive distance is smaller
angles of the robot joints, and C is the set of all configura- than the distance from the start to the goal. As planning be-
tions. The task of the path planning is to find collision-free tween two close configurations is typically faster than between
path for the robotic manipulator from the start configuration two distant configurations, each RRT planner running on the
qinit ∈ Cf ree to a goal configuration qgoal ∈ Cf ree , where slave node has to construct only a small configuration tree,
Cf ree ⊆ C is the free region of the configuration space C. which is very fast.
The parallelization of the planning is achieved by first
finding a rough (approximate) plan consisting of few way- IV. E XPERIMENTS AND R ESULTS
points and connecting these waypoints in parallel using a During the experiments, we interacted with the robot system
fine-resolution motion planner. The sequence of m waypoints and 2D visualization. The shortest path to the goal position is
(p1 , . . . , pm ), where p1 = qinit , pn = qgoal and pi ∈ displayed as an overlay to the 2D camera image (Augmented
Cf ree , i = 1, . . . , m is constructed using Lazy-PRM [16], Reality). Moving an object through the workspace of the robot
which runs on the Master node. leads to detection of that object and it will be visualized by
In Lazy-PRM, only the nodes are checked for collisions highlighting it semi-transparent. In the case of interference of
and the free-ones are stored into the roadmap. The edges are the object and already planned trajectory, the path replanning
created between close nodes without checking them for the is triggered (Fig. 4).

1265
 in the map. Finally, we composed a robot control application
from these services to demonstrate the feasibility of interaction
with the robots environment while preserving reasonable robot
motions without collision.
The next steps include new experiments and user studies
of the proposed visual interaction scheme. The operator will
be allowed to work with the robot while sharing time and
space without risking harm. We also plan to use different
communication frameworks for our distributed architecture
like OPC UA or DDS.
R EFERENCES
[1] L. Ribeiro. Cyber-physical production systems’ design challenges. In
2017 IEEE 26th International Symposium on Industrial Electronics
(ISIE), pages 1189–1194, June 2017.
[2] A. Vick, J. Guhl, and J. Krüger. Model predictive control as a service –
concept and architecture for use in cloud-based robot control. In 2016
21st International Conference on Methods and Models in Automation
and Robotics (MMAR), pages 607–612, Aug 2016.
[3] S. S. P. Olaya and M. Wollschlaeger. Control as an industrie 4.0
component: Network-adaptive applications for control. In 2017 22nd
IEEE International Conference on Emerging Technologies and Factory
Automation (ETFA), pages 1–4, Sept 2017.
[4] A. Vick, V. Vonásek, R. Pěnička, and J. Krüger. Robot control as a
service - towards cloud-based motion planning and control for industrial
robots. In International Workshop on Robot Motion and Control
(RoMoCo), pages 33–39, 2015.
[5] Cheng Liying, Xue Dingyu, Cong Yang, Jiang Xuehui, and Zhao Shuy-
ing. The research of environment perception based on the cooperation
of multi-robot. In 2012 24th Chinese Control and Decision Conference
Fig. 4. Visualization of Online Planning Trajectory in 2D Camera Image. (CCDC), pages 1914–1919, May 2012.
When introducing an Object to the vicinity of the path, it will be replanned [6] S. Yang, W. Xu, Z. Liu, Z. Zhou, and D. T. Pham. Multi-source
to avoid the obstacle. vision perception for human-robot collaboration in manufacturing. In
2018 IEEE 15th International Conference on Networking, Sensing and
Control (ICNSC), pages 1–6, March 2018.
We conducted two types of experiments with the system. [7] Lihui Wang. Collaborative robot monitoring and control for enhanced
sustainability. The International Journal of Advanced Manufacturing
The first type considered the behavior of the path planner Technology, 81(9):1433–1445, Dec 2015.
during the interaction of the detected obstacle and the tra- [8] I. Malý, D. Sedláček, and P. Leitao. Augmented reality experiments
jectory. The time for re-planning was found to be acceptable, with industrial robot in industry 4.0 environment. In 2016 IEEE 14th
International Conference on Industrial Informatics (INDIN), pages 176–
although the resulting trajectory is not easily predictable due 181, July 2016.
to the randomness of the RRT approach. The second type [9] J. Guhl, S. Tung, and J. Kruger. Concept and architecture for pro-
considered the composition of a complete manipulation task, gramming industrial robots using augmented reality with mobile devices
like microsoft hololens. In 2017 22nd IEEE International Conference
e.g. pick&place. The task may consist of three phases: 1) on Emerging Technologies and Factory Automation (ETFA), pages 1–4,
approach motion towards the object to pick, 2) grasping the Sept 2017.
object, 3) via motion towards the area to place the object. [10] L. E. Kavraki, P. Svestka, J.-C. Latombe, and M. H. Overmars. Prob-
abilistic roadmaps for path planning in high-dimensional configuration
During the experiments, we introduced obstacles in the robot’s spaces. IEEE Transactions on Robotics and Automation, 12:566–580,
way in the phases 1 and 3, and we observed the behavior of the 1996.
visualization and the robot motion. In all cases where objects [11] S. M. LaValle. Rapidly-exploring random trees: A new tool for path
planning, 1998. Technical report 98-11.
have been detected, the trajectory was successfully replanned, [12] A. Shkolnik and R. Tedrake. Path planning in 1000+ dimensions using a
resulting in collision-free motions of the robot. task-space Voronoi bias. In IEEE international conference on Robotics
and Automation (ICRA), pages 2892–2898, 2009.
V. C ONCLUSION AND F UTURE W ORK [13] D. Devaurs, T. Siméon, and J. Cortés. Parallelizing RRT on distributed-
memory architectures. IEEE International Conference on Robotics and
In this paper, we presented a service-oriented architecture Automation (ICRA), pages 2261–2266, 2011.
for the control of industrial robots that can be deployed [14] S. A. Jacobs, N. Stradford, C. Rodriguez, S. L. Thomas, and N. M.
to remote computers or cloud infrastructures. The services Amato. A scalable distributed RRT for motion planning. In IEEE
International Conference on Robotics and Automation (ICRA), pages
themselves are designed to run independently and provide a 5088–5095, 2013.
specified function required in the intelligent robot control. We [15] C. Rodriguez, J. Denny, S. A. Jacobs, S. Thomas, and N. M. Amato.
implemented an environment perception service that is able Blind RRT: A probabilistically complete distributed RRT. In IEEE/RSJ
International Conference on Intelligent Robots and Systems (IROS),
to detect and visualize obstacles in the robot workspace. That pages 1758–1765, 2013.
perception service is connected to a parallel path planning ser- [16] R. Bohlin and E.E. Kavraki. Path planning using Lazy PRM. In IEEE
vice that provides fast calculation of collision-free trajectories International Conference on Robotics and Automation (ICRA), volume 1,
pages 521–528, 2000.
from given start configuration and a given set of obstacles

1266

Powered by TCPDF ([Link])