See discussions, stats, and author profiles for this publication at: [Link]

net/publication/335095635

Real-Time Object Coordinate Detection and Manipulator Control Using Rigidly

Trained Convolutional Neural Networks

Conference Paper · August 2019

DOI: 10.1109/COASE.2019.8842973

CITATIONS READS

6 306

3 authors, including:

Chih-Hung Li
National Taipei University of Technology
20 PUBLICATIONS 112 CITATIONS

SEE PROFILE

All content following this page was uploaded by Chih-Hung Li on 10 August 2019.

The user has requested enhancement of the downloaded file.

 Real-Time Object Coordinate Detection and Manipulator Control
Using Rigidly Trained Convolutional Neural Networks
Yu-Ming Chang, Chih-Hung G. Li*, Member, IEEE, and Yi-Feng Hong

Abstract— Objects embedded in the environment, such as

switches, control buttons, sockets, et al., are devices that need
frequent operations. To devise manipulators to operate such
devices automatically, we propose a visual-position control
scheme that directly converts the visual coordinate detections to
motor commands. We train ConvNets with rigid 3D coordinate
information, which is obtained from a single basis image of the
target object. Our proposed training data preparation
frameworks automatically generate and organize the required
structure of the training images for the network. The ConvNet's Basis photo Training set
superior image recognition capability results in high success rate
in object detection and high precision in coordinate estimation.
In our static experiments, in-range plane coordinate detection Training
offline
achieves an average success rate of 91% from various view-point
(x, y, z)
directions; the depth coordinate detection achieves an average
success rate of 86% based on an extended success range. In our ConvNet Estimation

dynamic experiments, a low-precision manipulator was used to Detection Manipulator control

online
press a down elevator call button and achieved an overall success
rate of 98%. A high-precision manipulator was used for an Figure 1. The needs to operate objects embedded in the environment such as
object localization task and achieved a precision of ±0.3 mm wall switches, control buttons, sockets, et al. are tremendous. Here we
using a low-resolution camera. demonstrate a novel approach using a rigidly trained ConvNet for 3D
coordinate estimation, and a monocular eye-in-hand system for direct control
I. INTRODUCTION of the manipulator. The ConvNet provides robustness in illumination and
Objects embedded in the environment such as wall view-point difference and sufficient precision on coordinate detection.
switches, control buttons, sockets, et al. are everywhere; the view-point differences, as a mobile manipulator may approach
needs to operate them are tremendous (see Fig. 1). Equipment the target differently at different times.
that can operate these objects automatically is of great use in
factories, hospitals, offices, houses, et al. Accurate operation Having recognized the differences between operating
of these objects requires precision localization of the embedded objects and handling free objects, in this paper, we
manipulator, which is now well guided by visual means. There contemplate the problem and propose a method suitable for
have been researches on pose estimation of free objects and embedded object coordinate detection and manipulator control.
manipulator control; however, the characteristics of operating Specifically, we demonstrate a novel approach using a rigidly
the aforementioned embedded devices are quite different from trained Convolutional Neural Network (ConvNet) for 3D
handling free objects. For example, to pick and place a bottle coordinate detection, and a monocular eye-in-hand system for
or a scissors, 6D pose estimation is needed to locate the center direct control of the manipulator (see Fig. 1). The ConvNet is
of the object and to find the best angle and position to grasp trained to “memorize” the 3D coordinates of the target center;
the object. However, to press a button on the wall only needs thus, the detector is capable of making predictions directly
to locate the 3D translational distances between the target and from the whole input image. Our method does not need any
the manipulator; the task can be completed by aligning the process of local-feature-matching or template-matching. As
manipulator with the button perpendicular to the visual plane will be clearer below, the rigidly-trained detector not only
and moving it toward the button until contact. The need on 6D remembers the target coordinates of the training images, but
pose estimation for pressing a button is low; however, a certain also provides accurate detections for cases that are subject to
level of precision in localization of the manipulator is required substantial illumination and view-point changes. The
for successful operation as switches and buttons are usually robustness was also confirmed by our dynamic tests, where
small. Furthermore, as the objects are embedded in the manipulators completed localization tasks with high success
environment, they are less susceptible to occlusions and rates while subject to various view-point conditions.
background changes, which are key issues in handling free Furthermore, our method does not require to pre-process
objects especially in cluttered scenes. The key challenge in data such as to manually annotate for the center of the objects
button pressing would be in dealing with illumination and for network training. Instead, for every object, only one basis

*This work was supported by the National Taipei University of The authors are with the Graduate Institute of Manufacturing Technology,
Technology – King Mongkut’s University of Technology Thonburi Joint National Taipei University of Technology, Taipei, 10608 Taiwan ROC (e-
Research Program: NTUT-KMUTT-107-02. mail: cL4e@[Link]).
photo on the target object is needed and the rest of the data level of abstraction, and makes it possible to identify
preparation process is automated. The average detection time homogeneous but highly variable signals [14], [15].
of our system is less than 0.032 s, making it possible to run in Investigation results also show that it outperforms SIFT on
real time. Low demand on computation and memory is also the descriptor matching [16] and other methods in visual
merit. The rest of the paper is organized as follows. Section II recognition, classification, and detection [17]. The robustness
describes related prior work. Details of the proposed method of ConvNet in image recognition is also one good reason to
and the static experiments are detailed in Section III. adopt it as the feature descriptor [18]. For example, effects of
Manipulator experiments are reported in Section IV. The illumination and view-point difference on place recognition
conclusions are in Section V. were studied; images with high variations due to season and
illumination differences were successfully recognized using a
II. RELATED WORK ConvNet [11]. In object pose estimation, techniques similar to
template matching are still widely adopted in ConvNet-related
Automatic visual positioning has been applied to mobile approaches. For example, R-CNN [8] uses region proposal
robots for fetch-and-delivery, equipment operation [1], SLAM methods to generate potential bounding boxes and then
[2], et al. It is also in great demand in automated production execute a classifier on these proposed boxes. However, sliding
lines such as workpiece placement, product assembly, AOI, windows and running a ConvNet on each window takes a
machinery operation, and more [3]. Traditional visual significant amount of processing time. Redmon et al. [12]
positioning techniques often involve feature detection [4] or reframed detection as a regression problem and achieved a
template matching [5] - [8]. Take pressing an elevator button very high detection speed in YOLO, which sees the entire
as an example, Wang et al. [5] used binary images of the image during training and test time, so it implicitly encodes
characters processed and captured from the whole image to contextual information about classes as well as their
perform pattern matching in order to detect the location of the appearance. As it learns object positions from manually
button. Troniak et al. [6] implemented a template matching annotated training sets, preprocessing the images can still be a
algorithm based on the FastMatchTemplate technique, which tedious task. To be relieved from heavily relying on manually
has been shown to be robust to small changes in lighting and annotated data, Oquab et al. [19] introduced multi-scale
viewing angle. Kang et al. [7] also used template matching for sliding-window training to train from images rescaled to
recognition of the elevator status; a threshold value was multiple different sizes and to treat fully connected adaptation
defined to reject non-objects. However, the feature templates layers as convolutions. They found that ConvNet learns to
often do not react to observations that are subject to localize objects despite having no object location annotation at
environmental or view-point changes [9], [10]. In addition, training. However, it is not clear whether their scheme can
template matching requires sliding the template across the provide the level of precision in coordinate estimation that
image and comparing the template to the local pixels; it often satisfies industrial applications.
consumes considerable computation time [10].
To devise a suitable method for embedded object detection
Recently, Deep ConvNet, which exhibits outstanding and precision coordinate estimation, we propose a method to
image recognition capabilities has been applied in many object adopt ConvNet for visual feature learning. However, the
or place recognition tasks [8], [11] - [13]. It provides higher scheme is not based on template matching nor bounding

Basis photo Level 0 Level 1 Level 2 Level nz

Rescale …

(0, 0, 0) (0, 0, 0)
Bound and Annotate
(0, 0, 0) (0, 0, 0)
(1, 0, 0) (1, 0, 0) (1, 0, 0)
(1, 0, 0)
(2, 0, 0) (2, 0, 0) (2, 0, 0)
.. .. (2, 0, 0) ..
. . ..
. . . .
.
(xn, 0, 0) .
(xn, 0, 0) (xn, 0, 0) (xn, 0, 0)
(0, 1, 0) (0, 1, 0) (0, 1, 0)
(0, 1, 0)
(1, 1, 0) (1, 1, 0) (1, 1, 0)
.. .. (1, 1, 0) ..
. . ..
. . . ..
(xn, yn, 0) .
(xn, yn, 0) (xn, yn, 0) (xn, yn, 0)

Camera

ConvNet
For Z
Detection
Figure 2. Illustration of the automatic training data generation and organization frameworks for the proposed 3D coordinate detection scheme. By
altering the size of the cropping window, images of the target object with different scales are generated and annotated automatically. By further sliding
a bounding window, the target object appears at different locations of strictly-defined plane coordinates. Here, each column contains images of the target
object of the same scale but different plane locations. By labeling each column of images with a different depth distance, a ConvNet for depth detection
is trained.
 regarded as belonging to the same class of depth distance. By
…… Y=0
combining various classes of depth distances, the data are
organized for training the depth coordinate detection (DCD).
For example, in Fig. 2, images in the left column are labeled

………
………… Y=1 ConvNet
as the deepest and the right column the nearest; a total of 𝑛𝑧 +
For Y
1 levels of depth are constructed.

………
Detection

By regrouping these images according to the object’s plane

..

coordinates, the same data can be reorganized for training

………… Y=Yn
plane coordinate detection (PCD) as shown in Fig. 3. The
images in each column in Fig. 3 have the same horizontal
X=0 X=1 X=2 ………… X=Xn coordinate, while each row has the same vertical coordinate.
As illustrated in Fig. 4, the bounding area of each training
ConvNet image is defined as,
For X
Detection Φ𝑘 −𝑠 Φ𝑘 −𝑠
0≤ − 𝑖∆𝑥 ≤ 𝑥̅𝑖,𝑗,𝑘 ≤ − 𝑖∆𝑥 + 𝑤𝑘 ≤ Φ𝑘
2 2
Figure 3. By reorganizing the images according to the target object’s plane Θ𝑘 −𝑟 Θ𝑘 −𝑟
coordinates relative to the image frame, the same data are constructed for 0≤ − 𝑗∆𝑦 ≤ 𝑦̅𝑖,𝑗,𝑘 ≤ − 𝑗∆𝑦 + ℎ𝑘 ≤ Θ𝑘 (1)
2 2
training the plane coordinate detection.
where Φ𝑘 and Θ𝑘 are the width and the height of the level k
boxing detection. Rather, we take a novel approach of holistic depth image, s and r are the width and the height of the target
image training by preparing training data with explicit window, which is usually defined as a fixed portion of Φ𝑘
coordinate information. Thus, the network is trained to and Θ𝑘 . wk and hk are the width and the height of a level k
memorize the coordinates of the object through the sliding window. Integers i, j, and k are the indices in the
deliberately organized training data. Unlike R-CNN and the horizontal, the vertical, and the depth directions:
other template-matching methods, our method can be very fast
and achieve real-time detection. Also, our method does not 𝑖 = 0, 1, 2, … , 𝑛𝑥
require any manual annotation on the training images; thus, it 𝑗 = 0, 1, 2, … , 𝑛𝑦 (2)
can be easily implemented by unskilled personnel for a large 𝑘 = 0, 1, 2, … , 𝑛𝑧
amount of applications. The coordinate detection scheme also
provides sufficient precision in coordinate estimation for where nx and ny must be even numbers, so the target object
manipulator control of industrial applications. always appear in the center at ic = nx/2 and jc = ny/2. The
horizontal and the vertical increments determine the precision
III. RIGIDLY TRAINED CONVNET DETECTOR of the plane coordinate detection and are defined as,
𝑤𝑘 −𝑠
We organize a large amount of images of the target object ∆𝑥 =
𝑛𝑥
appearing at different locations and train the ConvNet to
ℎ𝑘 −𝑟
memorize the location of the object and make predictions ∆𝑦 = (3)
𝑛𝑦
based on whole images. The automatic training data
preparation scheme is a key feature of the proposed method
and is detailed below.

A. Organization of Training Data

Preparation of the training image set begins with taking a
basis photo of the target object by placing it at the center of the
photo (see the left photo of Fig. 2). Basically, only one photo
is needed for each object to be trained, unless dramatic
illumination changes need to be dealt with. Placing the target
object at the center of the photo automatically annotates the
coordinate origin of the object. A series of systematic image
cropping and rescaling procedures then follow. New images in
which the target objects appear in different scales relative to hk
r Θ𝑘
each new image frame are generated in an automatic fashion. s
Naturally, a larger object infers a shorter distance between the
object and the observer; it can be estimated according to the wk
thin lens formula. As shown in Fig. 2, every rescaled image y
can be viewed as a photo taken at a different depth distance.
We then generate more images by sliding a bounding window
across the new images horizontally and vertically with fixed
Φ𝑘 x
moving increments. These new images are automatically
annotated with the plane coordinates of the target object Figure 4. Definitions of the parameters used in image preparation. The
relative to the image frame. Since all the images bounded from red lines define the sliding window that bounds the training image.
a single depth level have the same object scale, they are
 (a) PCD (b) DCD
Figure 5. The success rates of 3D coordinate detections of the elevator button from various view-point directions and depths.

depth range as the training set and without any rotation of

B. Static Experiment
viewing angle, the success rate of PCD is 100%; the success
For demonstration, 8 embedded objects shown in Fig. 1 rate of DCD is around 50%. As shown in Fig. 6, most of the
were tested, including a down elevator call button. A CMOS errors in depth detection are off by one depth level. If one
camera with a resolution of 640 × 480 pixels was used for all extends the definition of success range to include one upper
the static and dynamic experiments. For each object, training and one lower level, the average success rate becomes 86%.
images of 8 depth levels were generated, each containing 21
x and 21 y coordinates. The architecture of each ConvNet C. Effects of Viewing Angle and Out-of-range Detection
contains the input layer, two convolution layers followed by
PCD is quite robust in yaw rotation. At ±30 degrees, the
two maximum pooling layers, two flattened layers, and the
success rate is still higher than 94% (see sample images in Fig.
output layer. For the input layer, every image is transferred to 7 (a)). For the pitch rotation, the PCD success rate drops to
grey-scale and compressed to 28 × 28 pixels. The total 86% at ±15 degrees and 74% at ±30 degrees. It is suspected
number of trainable parameters is 243,481. The ConvNets are that the greater impact came from the ceiling lights that
trained on a 3.60 GHz Intel Core i7-7700 CPU with a RAM resulted in more illumination changes (see Fig. 7 (b)). Our
of 16.0 GB and an NVIDIA GeForce GTX 1050. The total other experimental results not presented here showed that
time for completing the entire training process, including such an effect can be easily remedied by including a few more
training set generation and the training of all three ConvNets basis photos under various illuminations. For the rolling effect,
is 3 minutes and 26 seconds, within which there were 300 the PCD success rate is 94% at ±15 degrees and 75% at ±30
epochs for x, 300 epochs for y, and 200 epochs for z. The degrees (see Fig. 7 (c)). There are two types of out-of-range
training accuracy is generally between 95% and 97%. detections. One is outside the depth range, e.g., near or far
(left two images in Fig. 7 (d)), and the other is partial inclusion
Real-time coordinate detection was then performed at a of the target object (right two images in Fig. 7 (d)). The
frame rate of 30 fps subject to unconstrained illumination. For training depth in the test ranges between 15 cm and 30 cm; in
each object in Fig. 1, the success rate of PCD is above 95%. the out-of-range test, the “near” range is between 5 cm and 15
We then tested the coordinate detection of the elevator
button from various oblique directions and outside the (a)
training range (see Fig. 5 (a)). Note that the basis photo was
taken perpendicular to the target object, so the ConvNets
never learned from oblique images. The success rate of
depth detection was also obtained by comparing the actual
and the predicted depths; the results are summarized in Fig. (b)
5 (b). For the test images that were taken within the same
70%

60%

50%
detection rate

(c)
40%

30%

20%

10%

0% (d)
0 1 2 3 4 5 6 7 8
detected depth
ground truth
0 1 2 3 4 5 6 7

Figure 6. The DCD success rates of the non-rotation cases. For each
depth ground truth, the detection rate of each depth level is presented.
For example, when the ground truth is 4, 55% detects 4, 27% detects 5, Figure 7. Testing images of various viewing angles: (a) yaw, (b) pitch, and
16% detects 3, 2% detects 6, and 0% detects the others. (c) roll. (d) Sample images of out-of-range detections.
 60

40
time (s)

20

0
0 degree yaw 15 yaw 30 pitch +15 pitch +30 pitch -15 pitch -30 roll 15 roll 30
degrees degrees degrees degrees degrees degrees degrees degrees
maximum completion time average completion time minimum completion time
Figure 8. Test results of the elevator button-pressing experiment. The average completion time is hardly affected by the attack angle within the orientation
range of ±30 degrees.

cm and the “far” range between 30 cm and 45 cm. Results of where ∆𝜎 denotes the commands on 3D movement. Γ𝜎
the out-of-depth-range PCDs are also shown in Fig. 5 (a). denotes the ConvNet detection, and 𝜋̅𝜎 denotes the policy on
About 80% of the partial inclusion cases still indicated the the manipulator action given the ConvNet detection result
ball-park position of the target object, although the accurate ω𝜎,𝑡 . The overall success rate of the task performance
coordinate estimates might not be provided. evaluated by (5) depends on 𝜋̅𝜎 and Γ𝜎 .

IV. MANIPULATOR CONTROL B. Manipulator Experiment and Result

We first conducted a dynamic experiment using a Uarm-
A. Direct Visual-position Control Metal manipulator to press a down elevator call button of
Automation of a visual-guided manipulator approaching 25 × 25 mm2. Uarm-Metal is a light manipulator with a
and operating an embedded object is devised as follows. The precision of approximately ±5 mm. In the test, the Uarm base
manipulator undergoes a series of activities such as visual was randomly placed at a depth distance about 10 cm to the
recognition, policy decision, and motor action. We let the target button. Nine cases of initial orientations were tested
policy function of the manipulator action be 𝜋𝜎 (𝑢𝑡 |𝑜𝑡 ) (𝑜𝑡 : (see Fig. 8). The eye-in-hand camera and the ConvNets
the observation; 𝑢𝑡 : the action) and 𝑥𝑡 denote the state of the detected the 3D coordinates of the elevator button; then the
manipulator. By discretizing the time, program sent motor commands to move the manipulator
toward the button in small steps (< 20 mm/step). Once the
t ∈ [1, ⋯ , 𝑇]. (4) manipulator reaches a depth of 5 cm to the button and is
In this Markov process, the probability of the current state aligned with it, the manipulator extends and presses the button.
is the outcome of the previous action and state In this test, the overall success rate is 98%; the manipulator
as 𝑝(𝑥𝑡+1 |𝑥𝑡 , 𝑢𝑡 ). Observation of the robot is also a probability successfully completed all the tasks, except the pitch +30 case
as 𝑝(𝑜𝑡 |𝑥𝑡 ) . Thus, the trajectory and the policy 𝜋𝜎 (𝜏) to that has a success rate of 80%. For every successful operation,
complete a task is [20], the manipulator completed the task in 6-12 steps; the average
elapsed time is approximately 30 s (see Fig. 8). A video of the
𝑇
manipulator experiment can be found at
𝜋𝜎 (𝜏) = 𝑝(𝑥1 ) ∏ 𝑝(𝑥𝑡+1 |𝑥𝑡 , 𝑢𝑡 ) ∫ 𝜋𝜎 (𝑢𝑡 |𝑜𝑡 )𝑝(𝑜𝑡 |𝑥𝑡 )𝑑𝑜𝑡 ; [Link] A sample imaging sequence
𝑡=1 of the Uarm approaching and pressing the elevator button is
𝜎 = 𝑥, 𝑦, 𝑧 (5) shown in Fig. 9. Comparing with the results of [6], our
average detection time of 0.032 s is shorter than template
where the trajectory  is defined as, matching by two orders of magnitude.
τ = {𝑥1 , 𝑢1 , 𝑥2 , 𝑢2 , ⋯ , 𝑥𝑇 , 𝑢 𝑇 } (6) The Uarm tests proved that the proposed visual-servo
Whereas the observation probability relies on a stochastic control scheme can successfully guide a low-precision
imaging process, the action policy here depends on the manipulator to complete a common button-pressing task. We
ConvNet detection and linear motion moving from the current then tested high-precision motion using a Yaskawa robot arm
coordinates to the next position within the time increment as, – MOTOMAN-GP7, which has a precision of ±0.01 mm (see
Fig. 10). In this test, a 3D-printed block with a square hole of
∆𝜎 = 𝜋𝜎 (𝑢𝜎,𝑡 |𝑜𝑡 ) = 𝜋̅𝜎 (𝑢𝜎,𝑡 |ω𝜎,𝑡 )Γ𝜎 (ω𝜎,𝑡 |𝑜𝑡 ) (7) 2 × 2 mm2 was used as the target object. PCD was set up to

Figure 9. A sample imaging sequence of the Uarm approaching and pressing the elevator button (from left to right) - a successful case of pitch +30
degrees.
 [7] J. -G. Kang, S. -Y. An, and S. -Y. Oh, “Navigation strategy for the
service robot in the elevator environment,” in Proc. Int. Conf. Control,
Enlarged Autom. Syst., Seoul, Korea, 2007.
target object
Robot arm [8] R. Girshick, “Fast R-CNN,” Int. Conf. Comput. Vision (ICCV), 2015.
[9] S. Oron, T. Dekel, T. Xue, W. T. Freeman, and S. Avidan, “Best-
Buddies Similarity—robust template matching using mutual nearest
neighbors,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 40, no. 8, pp.
Camera 1799–1813, 2018.
[10] W. Ouyang, F. Tombari, S. Mattoccia, L. Di Stefano, and W. -K. Cham,
“Performance evaluation of full search equivalent pattern matching
algorithms,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 34, no. 1, pp.
PC
127–143, 2012.
[11] N. Sunderhauf, S. Shirazi, F. Dayoub, B. Upcroft, and M. Milford, “On
the performance of ConvNet features for place recognition,” in Proc.
IEEE/RSJ Int. Conf. Intell. Robot. Syst. (IROS), 2015.
Target object [12] J. Redmon, S. Divvala, R. Girshick, A. Farhadi, “You Only Look Once:
unified, real-time object detection,” arXiv:1506.02640v5, 2016.
Figure 10. Setup of the Yaskawa robot arm experiment. The target object [13] Y. Xiang, T. Schmidt, V. Narayanan, and D. Fox, “PoseCNN: a
is movable, and the center of it was randomly placed within a square convolutional neural network for 6D object pose estimation in cluttered
range of 2 × 2 cm2. Only one camera and one target object were used in scenes,” arXiv:1711.00199, 2018.
this test.
[14] Y. Chen, Y. Shen, X. Liu, and B. Zhong, “3D object tracking via image
sets and depth-based occlusion detection,” Signal Processing, vol. ll2,
have a minimum detection increment of 0.5 mm. The center pp. 146–153, 2015.
of the target object was randomly placed within a square range [15] Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” Nature, vol. 521,
of 2 × 2 cm2. For the 100 trials, the robot arm successfully pp. 436–444, 2015.
located the target object within a maximum deviation of ±0.3 [16] P. Fischer, A. Dosovitskiy, and T. Brox, “Descriptor matching with
mm. convolutional neural networks: a comparison to SIFT,” arXiv:
1405.5769, 2014.
V. CONCLUSIONS [17] R. Girshick, J. Donahue, T. Darrell, and J. Malik, “Rich feature
hierarchies for accurate object detection and semantic segmentation,”
We proposed an object coordinate detection scheme using in Proc. IEEE Conf. Comput. Vision Pattern Recog. (CVPR), 2014.
rigidly trained ConvNets; we also demonstrated how to build [18] A. S. Razavian, H. Azizpour, J. Sullivan, and S. Carlsson, “CNN
the coordinate-articulate training data based on a single basis features off-the-shelf: an astounding baseline for recognition,” in Proc.
photo. Our experimental evidence shows that the PCD Comput. Vision Pattern Recog. Workshops (CVPRW), 2014.
success rate of perpendicular observation is 100%, and is [19] M. Oquab, L. Bottou, I. Laptev, J. Sivic, “Is object localization for free?
averagely higher than 80% for oblique observations. Using - weakly-supervised learning with Convolutional Neural Networks,”
the proposed real-time detection scheme, a low-precision IEEE Conf. Computer Vision Pattern Recog. (CVPR), 2015.
manipulator was controlled to localize and press a button, [20] S. Levine, C. Finn, T. Darrell, and P. Abbeel, “End-to-end training of
achieving an overall success rate of 98% and an average deep visuomotor policies,” J. Machine Learning Res., vol. 17, pp. 1–40,
completion time of 30 s. When tested on an industrial robot 2016.
arm, a localization precision of ±0.3 mm was achieved with
a 640 × 480 pixels CMOS camera.

REFERENCES
[1] N. Bellotto and H. Hu, “Multisensor-based human detection and
tracking for mobile service robots,” IEEE Trans. Systems, Man,
Cybernetics, Part B: Cybernetics, vol. 39, no. 1, pp. 167–181, February
2009.
[2] M. Veloso, J. Biswas, B. Coltin, and S. Rosenthal, “CoBots: robust
symbiotic autonomous mobile service robots,” in Proc. Twenty-Fourth
Int. Joint Conf. Artificial Intell. (IJCAI), 2015.
[3] Q. Wei, C. Yang, W. Fan, and Y. Zhao, “Design of demonstration-
driven assembling manipulator,” Appl. Sci., vol. 8, no. 5, pp. 797–807,
2018.
[4] J. L. Lowe, “Mobile robot localization and mapping with uncertainty
using scale-invariant visual landmarks,” Int. J. Robot. Res., vol. 21, pp.
735–758, 2002.
[5] W. -J. Wang, C. -H. Huang, I. -H. Lai, and H. -C. Chen, “A robot arm
for pushing elevator buttons,” in Proc. SICE Ann. Conf., August 18-21,
2010.
[6] D. Troniak, J. Sattar, A. Gupta, J. J. Little, W. Chan, E. Calisgan, E.
Croft, and M. Van der Loos, “Charlie rides the elevator–integrating
vision, navigation and manipulation towards multi-floor robot
locomotion,” in Int. Conf. Comput. Robot Vision, 2013.

View publication stats