Student Research Paper Conference Vol-2, No-15, July 2015

Design and Development of Human Following Robot

1
Muhammad Sarmad Hassan, 2Mafaz Wali Khan, 3Ali Fahim Khan
Department of Electrical Engineering
Institute of Space Technology, Pakistan
1
sarmadhassan@[Link]
2
mafazwali01@[Link]
3
alifahimkhan@[Link]

Abstract—For a robot that performs autonomously, the To perform this task accurately, robot needs a mechanism
communication between the person and the robot is the most that enables it to visualize the person and act accordingly [2]
important factor. A significant awareness has been observed [3]. The robot must be intelligent enough to follow a person
regarding the usage of such a technology. This research has a
in the crowded areas, vivid environment and in indoors and
trivial involvement in the development of such robots. A robot that
outdoors places [4].
functions fully autonomously should not only complete the jobs
that are desired of them but also somehow establish a connection
The image processing carried out to get the information about
between themselves and the person operating them. A lot of
the surroundings visually is a very important thing. The
research has been done of these kinds of robot and a lot of work
still needs to be done. In order for a robot to communicate and following points should be carefully noted while doing the
interact with the person, it should also be capable of following that processing.
particular person. Keeping this in mind, there should be a capacity
in the robot to get information from the surroundings while  The luminosity conditions should be very stable and
persuing the required object. The primary goal of our work was to should not fluctuate.
design and fabricate a robot that not only tracks the target but also  The ranges should be set properly for the desired
moves towards it while doing the tracking. In order to make things environment on which to perform the tracking.
simpler, a unique handmade tag was placed on the person that the  The target should not be very far from the visual
robot needs to follow. The main hindrance in this kind of work is
sensor as the distance matters a lot.
that the detection of the target is a sensitive thing to carry out. The
object has to be unique for the robot to recognize it and carry out  We should avoid the use of such colors around the
the objective. The simple tag removes this problem of uniqueness robot that matches with that of the target. Otherwise
and makes the task fairly easy. A small camera records the video the robot would get confused.
and the processor processes it to extract the desired information
from it. Protecting the robot from collision with the object is Typically human following robots are equipped with several
another problem that needs to be tackled so in order to do this, a different diverse combination of sensors i.e. light detection
sensor is used. All the processing is carried out by the and ranging sensor, radio frequency identification module
microprocessor while the control of the motors is carried out by (RFID), laser ranger finder (LFR), infrared (IR) sensing
the controller. modules, thermal imaging sensors, camera, wireless
transmitter/receiver etc. for recognition and locating the
Keywords — Human following, human tracking, visual imaging,
human robot interaction, laser range scanners, image tag, target. All the sensors and modules work in unison to detect
envelope detection, service robots, depth image mapping, thermal and follow the target.
conductivity.
The capability of a robot to track and follow a moving object
can be used for several purposes.
I. INTRODUCTION
 To help humans.
Robotic technology has increased appreciably in past couple
of years. Such innovations were only a dream for some  To create ease for people.
people a couple of years back. But in this rapid moving  Can be used for defense purpose.
world, now there is a need of robot such as “A Human
In this paper, we presented a method of a human following
Following Robot” that can interact and co-exist with them.
robot based on tag identification and detection by using a
[1]
camera. Intelligent tracking of specified target is carried out
by the use of different sensors and modules [5] i.e. ultrasonic
79
 Design and Development of Human Following Robot

sensor, magnetometer, infrared sensors and camera. An very simple technique was also used by a research. In this
intelligent decision is being made by the robot control unit technique, the person used distance sensors on the robot and
based on the information obtained from the above sensors the person. These sensors emitted radio waves and were
and modules, hence finding and tracking the particular object detected by the sensors on the person to be followed. This
by avoiding the obstacles and without collision with the way the robot followed the required target[22].
target.
III. SYSTEM COMPONENTS
II. RELATED WORK Our system consists of a three wheel robotic vehicle mounted
So far a lot of research has been done on the kinds of robot with a separate microprocessor and control unit along with
that fall into the category of the “Assisting Robots”. People different sensors and modules i.e. ultrasonic sensor,
have used different logics and alog‟s to implement their magnetometer, infrared sensors, and camera.
design. All of their primary focus has entirely been on the
design of robots that follows the target. The camera height is vertically self-adjusting and is initially
mounted on robot at a height of 4 ft. from ground to enhance
Laser sensor is used by Burgard in his tour guide robot for the visual capability and effectiveness.
human tracking [6]. LRF was incorporated by D. Schulz to
perform the „following‟. Using the above mentioned process, This robotic vehicle is controlled by the user by means of an
they performed the information linking for the detection[7]. identification tag.
Nicola, Husing used a technique for pointing out the
IV. METHODOLOGY
different styles of movement by using LRF. This information
was fused with the information obtained by the camera [8]. A systematic research methodology is adopted keeping in
Depth imaging was used by Songmin Jia to carry out the mind the ultimate goal of a fully functional and autonomous
detection. The model of a person was determined using the human following robot.
depth imaging [9]. The particular style of clothing was used
A decentralized top down approach is used for this project.
by Mehrez Kristou. He used a multidirectional camera. LRF
The project is divided in to five modules. Each module is
was also incorporated by him in the design [10]. A research
independent from one another. Different phases were carried
was conducted by Wilhelm with the focus on the color of the
out step by step, starting from basic sensor testing and
particular person‟s skin. Information from different sensors
proceeding towards obstacle avoidance, object detection,
was also used by him in the research[11].
object tracking and data transmission.
Some other research work was also conducted in this regard, Due to the decentralized approach, all modules and sensors
Depth imaging was used by Calisi and the target was persued act independently. Data obtained by different sensors and
by designing a special algorithm[12]. Ess and Leibe carried modules is collectively analyzed and an intelligent decision
out the same work. They did a lot of work on object tracking on the basis of information obtained is made that instruct the
and detection. The biggest advantage of their method was that robot to follow a particular direction. Two separate units are
their algorithm worked in complex environments as well[13] used i.e. microprocessor and a controller. The processing is
[14]. Stereo visiion was also carried out by Y. Salih in order carried out by microprocessor and the information obtained
to perform the detection[15]. This method enabled him to by the sensors is controlled by a controller i.e. Arduino board.
persue the required target with an effective manner. The A serial communication between microprocessor and
combination of different sensors were used by R. Munoz to controller is established to exchange the visual sensing
.get the information about the target to be tracked. In addition information.
to using different sensors, he also used stereo vision to get an
accurate information.[16]. The data of the sensors combined This approach was most suitable because if there is a fault in
with the information from the camera proved to be very any one of the modules then it would not affect the entire
helpful in carrying out the task[17]. system. Hence this provides the best possible results by
maintaining accuracy.
Different algorithms are being developed by the researchers
for the detection purposes. Laser was used in one research to Human tracking, obstacle avoidance, maintaining a specific
find the style of the moving legs [18] [19][20] and camera distance from the object and establishing a communication
was used to detect a particular object or a person[3] [21]. A

80
 Design and Development of Human Following Robot

link between microprocessor and controller are the main The software design of the mechanical structure is shown
aspects of this project. below.

V. IMPLEMENTATION
The implementation of human interactive robot seems to be
an easy task but there are some problems related to it. For
example, detection of a particular tag. Similarly the tag needs
to be unique so that it should not be merged into colors in a
vivid environment. So we have designed a novel algorithm to
overcome this problem.

There are several features of the project such as the

mechanical structure, design of circuits, tag detection and
intelligent tracking system.

The implementation of human interactive robot is as follows. Fig. 2. Software Design of Mechanical Structure

A. Design of a custom Tag: C. Design of Relay Based Motor Driver Module:

In order to make the target distinctive and unambiguous, a tag A relay based motor driver module is designed to control the
was designed in particular. This tag had four different colors differential drive of the robotic platform. The advantage of
in box arrangement. The whole research is based on the fact this relay based module is that it can easily drive the high
that the target has to be specific to the robot. The robot torques motors that require large fluxes. Hence it can easily
should only detect that target and not some other target. All handle heavy loads.
these colors enable the robot to detect the person by tracking
D. Image Processing Algorithm:
the tag. This tag is placed on the person that needs to be
pursued. Once the tag is put on the person the robot detects A novel algorithm to process the real time video is used to
the colors and recognizes the target through the detection. detect and follow the unique tag. We have used a computer
vision camera for recognizing the tag at the back of person
and an OpenCV python Platform to develop this algorithm.

In this algorithm, a tag having the four color are used. The
algorithm is designed in a way that the tag is only detected
when the four colors are all together and it that particular
order. Once the tag is detected, the centroid of the common
region of four colors is found to get the center coordinates of
the tag. These center coordinates are then serially transmitted
to control unit for further processing and to make an
intelligent decision by fusing it with information obtained by
the other sensors and modules.

Fig. 1. Custom Tag VI. SYSTEM DESIGN

The system design consists of separate processing and control
B. Design of Mechanical Structure:
unit. The processing unit only makes use of a camera and is
The mechanical structure of the robot is comprised of two linked with the control unit to serially transmit the visual
layers base and it consist of two wheel differential drive information after bulk processing. The control unit is serially
system and a free wheel. It is designed keeping in view that linked with the processor and it makes use of several sensors
the camera on robot has to be mounted over a certain height and modules i.e. ultrasonic sensor, magnetometer and
from the ground. The height of this camera is adjustable infrared sensors.
according to height of a person so that better visual
information can be obtained. So initially the height of the The above sensors and camera works in unison with each
camera is set up to 4 ft. other and helps the robot in its operation and to navigate its
81
 Design and Development of Human Following Robot

path by avoiding the obstacles and maintaining a specific

distance from the object. The decision is made on the basis of
information obtained from all above sensors.

The generalized system design incorporating different sensors

and modules is show below.

Fig. 4. Ultrasonic Sensor Principle

This ultrasonic sensor is placed at the top of robot along with

the camera module to maintain accuracy in measuring
distance between the robot and target object.

The flow chart to maintain specific distance from target is

shown below.

Fig. 3. System Design

Looking at the working of the above system, the first phase is

the detection of a tag by means of a camera and carrying out
the substantial processing in the processing unit. The
processor that we have used is Raspberry Pi single board
computer. After the detection of tag next phase is to establish
a serial communication between the processor and control
unit. We have used Arduino as a control unit. In this phase
Fig. 5: Flow Chart of Maintaining Specific Distance
center point coordinates of tag are serially transmitted to
Arduino for further processing. After ultrasonic sensor we interfaced the magnetometer to get
the orientation of robot in x-y-z coordinates. This module
Next phase is to interface modules and necessary sensors determines the orientation of robot and tells heading direction
with the control unit. For this purpose, we used ultrasonic of robot. This heading direction is used to determine the tilt
sensor, magnetometer and IR sensors for the proper of robot from its original position. [24]. On the basis of
functioning of robot. information obtained from this module, the control unit
determines that how much direction change is required to be
We used ultrasonic sensor for obstacle avoidance and to
back on track again by after avoiding the obstacle.
maintain a specific distance for the object. The ultrasonic
sensor works accurately works accurately with in a range of 4 After interfacing of above sensors, the next most
meters. Ultrasonic sensors operate by calculating the times important part of this system design is to interface the
differences. [23]. encoders to wheel calculate the distance travelled by
the robot to eliminate any further error in the robotic
movement due to displacement. For this purpose we
attached two slot sensors on top of the encoder‟s right beside
the wheels. The slot sensor has IR transmitter and a
photodiode mounted on it and facing each other. The light
emitted by the IR LED is blocked because of alternating slots

82
 Design and Development of Human Following Robot

of the encoder disc. This causes the logic level of the photo to make it able to respond to diverse situations.
diode to change and is detected by the controller [25].
VII. EXPERIMENTS AND RESULTS
Different experiments were conducted and the performance
of the human following robot was tested. Each experiment
that was performed took about 10 to 15 minutes. On the basis
of results obtained from these tests and experiments, we made
the necessary changes in the processing and control
algorithm.

First test was performed on the ultrasonic sensor. It was noted

that sensor was working accurately with in a range of 4
meters. Then we performed the test to check that weather the
robot maintains a specific distance with target object. Initially
Fig. 6. IR Slot Sensor
the we set the stopping of robot to 8 inches. It was observed
The distance is obtained by the number of counts recorded in that robot collided with the object as the distance between
memory of the controller and is used for distance calculations robot and target object approaches to 8 inches. This problem
by using the following formula. behind this was that the stopping distance was small enough
and robot was not stopping quickly because of its load on
board. So we increased the distance to 12 inches. Then we
again verified the routine.

The next test was performed on the magnetometer module.

This module gave us the heading direction of the robot with
Final phase is consisted of fusing all the information obtained respect to some reference. But we observed that there was
by the sensors and modules in the control unit. Hence, control some offset error in that heading direction. On observation
unit makes an intelligent decision to change the direction of we found that this was due to the wrong placement of the
robot and to get back on its track again and to follow the magnetometer module. This module was interfering with the
target having tag on basis of information obtained for all magnetic field of the electronic components due to which we
sensors and modules i.e. serially received coordinates from were getting the offset error. So we changed its placement
processor, distance information from ultrasonic sensor, and set it up at bottom layer and in the center of the robotic
heading direction from magnetometer, and distance structure. Now we had the heading degree without any offset
calculation from the IR sensor. error.

Then the next experiment was to test the detection of tag. We

observed that in certain lightning conditions the tag was not
detecting properly. So we adjusted the hue, saturation and
value of all the four colors as color thresholding in HSV
varies by the lighting conditions. So after changing the
threshold value we observed that this time the processor was
detecting the Tag properly.

Results are shown below.

Fig. 7. Human Following Robotic System.

The system is designed in such a way that it intelligently

makes use of the information obtained from different sensors
and modules. Different sensors are incorporated in the system
83
 Design and Development of Human Following Robot

Fig. 8. Tag Detection

Then we check the serial communication between

Arduino and Raspberry pi. We observed that when the
tag was not infront of camera the processor transmitted
a null value for both coordinates and some value when
tag was in front of camera. Hence, processor was
correctly transmitting the coordinates to control unit. Fig. 10. Human Follower Robot

VIII. APPLICATIONS
Results are shown below.
Looking deeply into environment or our surroundings, we
will be able interpret that “YES” there is a need of such robot
that can assist humans and can serve them. Such a robot can
be used for many purposes. With a few changings, the robot
can act as a human companion as well.

Some other applications of this robot are

 Can assist in carrying loads for people working in

hospitals, libraries, airports, etc.
 Can service people at shopping centers or public
areas.
 Can assist elderly people, special children and
babies.
Fig. 9. Serial Communication between Arduino and Raspberry Pi  Can follow a particular Vehicle.
Final test was performed on the differential drive system to IX. FUTURE WORK
check whether the robot intelligently follow the person or not.
We observed that the results produced were very satisfying. There are many interesting applications of this research in
different fields whether military or medical. A wireless
The robot was perfectly following the person wherever it
communication functionality can be added in the robot to
goes. We just adjusted its speed at the end. make it more versatile and control it from a large distance.
This capability of a robot could also be used for military
Hence the objective of implementing a good Human-Robot
purposes. By mounting a real time video recorder on top of
interaction was achieved by fast processing of the visual the camera, we can monitor the surroundings by just sitting in
input combined with the proper handling of the information our rooms. We can also add some modifications in the
from the sensors.

84
 Design and Development of Human Following Robot

algorithm and the structure as well to fit it for any other Proceedings. 2004 IEEE/RSJ International Conference on,
purpose. E-g a vehicle follower. 2004, vol. 2, pp. 1347–1352.

Similarly it can assist the public in shopping malls. So there it [8] N. Bellotto and H. Hu, “People tracking and identification
with a mobile robot,” in Mechatronics and Automation, 2007.
can act as a luggage carrier, hence no need to carry up the ICMA 2007. International Conference on, 2007, pp. 3565–
weights or to pull that. Using this algorithm the robot will 3570.
automatically follow that person.
[9] S. Jia, L. Zhao, and X. Li, “Robustness improvement of
X. CONCLUSION human detecting and tracking for mobile robot,” in
Mechatronics and Automation (ICMA), 2012 International
A successful implementation of a person follower robot is Conference on, 2012, pp. 1904–1909.
illustrated in this research. This robot does not only have the
detection capability but also the tracking and following [10] M. Kristou, A. Ohya, and S. Yuta, “Target person
identification and following based on omnidirectional camera
ability as well. The tracking is basically performed on the tag and LRF data fusion,” in RO-MAN, 2011 IEEE, 2011, pp.
and the human is followed on the basis of that detection. It 419–424.
was also kept in mind that the „following‟ capability of the
robot should be as efficient as possible. The tests were [11] T. Wilhelm, H. J. Böhme, and H. M. Gross, “Sensor fusion
for vision and sonar based people tracking on a mobile
performed on the different conditions to pin point the service robot,” in Proceedings of the International Workshop
mistakes in the algorithm and correct them. The different on Dynamic Perception, 2002, pp. 315–320.
sensors that were integrated with the robot added an
[12] D. Calisi, L. Iocchi, and R. Leone, “Person following through
additional advantage.
appearance models and stereo vision using a mobile robot.,”
in VISAPP (Workshop on on Robot Vision), 2007, pp. 46–56.
XI. REFERENCES
[1] K. Morioka, J.-H. Lee, and H. Hashimoto, “Human-following [13] A. Ess, B. Leibe, and L. Van Gool, “Depth and appearance
mobile robot in a distributed intelligent sensor network,” for mobile scene analysis,” in Computer Vision, 2007. ICCV
IEEE Trans. Ind. Electron., vol. 51, no. 1, pp. 229–237, Feb. 2007. IEEE 11th International Conference on, 2007, pp. 1–8.
2004.
[14] A. Ess, B. Leibe, K. Schindler, and L. Van Gool, “A mobile
[2] Y. Matsumoto and A. Zelinsky, “Real-time face tracking vision system for robust multi-person tracking,” in Computer
system for human-robot interaction,” in 1999 IEEE Vision and Pattern Recognition, 2008. CVPR 2008. IEEE
International Conference on Systems, Man, and Cybernetics, Conference on, 2008, pp. 1–8.
1999. IEEE SMC ’99 Conference Proceedings, 1999, vol. 2,
pp. 830–835 vol.2. [15] Y. Salih and A. S. Malik, “Comparison of stochastic filtering
methods for 3D tracking,” Pattern Recognit., vol. 44, no. 10,
[3] T. Yoshimi, M. Nishiyama, T. Sonoura, H. Nakamoto, S. pp. 2711–2737, 2011.
Tokura, H. Sato, F. Ozaki, N. Matsuhira, and H. Mizoguchi,
“Development of a Person Following Robot with Vision [16] R. Muñoz-Salinas, E. Aguirre, and M. García-Silvente,
Based Target Detection,” in 2006 IEEE/RSJ International “People detection and tracking using stereo vision and color,”
Conference on Intelligent Robots and Systems, 2006, pp. Image Vis. Comput., vol. 25, no. 6, pp. 995–1007, 2007.
5286–5291.
[17] J. Satake and J. Miura, “Robust stereo-based person detection
[4] H. Takemura, N. Zentaro, and H. Mizoguchi, “Development and tracking for a person following robot,” in ICRA
of vision based person following module for mobile robots Workshop on People Detection and Tracking, 2009.
in/out door environment,” in 2009 IEEE International
Conference on Robotics and Biomimetics (ROBIO), 2009, pp. [18] J. H. Lee, T. Tsubouchi, K. Yamamoto, and S. Egawa,
1675–1680. “People Tracking Using a Robot in Motion with Laser Range
Finder,” in 2006 IEEE/RSJ International Conference on
[5] N. Bellotto and H. Hu, “Multisensor integration for human- Intelligent Robots and Systems, 2006, pp. 2936–2942.
robot interaction,” IEEE J. Intell. Cybern. Syst., vol. 1, no. 1,
p. 1, 2005. [19] M. Lindstrom and J. O. Eklundh, “Detecting and tracking
moving objects from a mobile platform using a laser range
[6] W. Burgard, A. B. Cremers, D. Fox, D. Hähnel, G. scanner,” in 2001 IEEE/RSJ International Conference on
Lakemeyer, D. Schulz, W. Steiner, and S. Thrun, “The Intelligent Robots and Systems, 2001. Proceedings, 2001,
interactive museum tour-guide robot,” in AAAI/IAAI, 1998, vol. 3, pp. 1364–1369 vol.3.
pp. 11–18.
[20] M. Kristou, A. Ohya, and S. Yuta, “Panoramic vision and lrf
[7] M. Scheutz, J. McRaven, and G. Cserey, “Fast, reliable, sensor fusion based human identification and tracking for
adaptive, bimodal people tracking for indoor environments,” autonomous luggage cart,” in Robot and Human Interactive
in Intelligent Robots and Systems, 2004.(IROS 2004). Communication, 2009. RO-MAN 2009. The 18th IEEE

85
 Design and Development of Human Following Robot

International Symposium on, 2009, pp. 711–716.

[21] C. Schlegel, J. Illmann, H. Jaberg, M. Schuster, and R. Wörz,

“Vision Based Person Tracking with a Mobile Robot.,” in
BMVC, 1998, pp. 1–10.

[22] N.-Y. Ko, D.-J. Seo, and Y.-S. Moon, “A Method for Real
Time Target Following of a Mobile Robot Using Heading
and Distance Information,” J. Korean Inst. Intell. Syst., vol.
18, no. 5, pp. 624–631, Oct. 2008.

[23] K. S. Nair, A. B. Joseph, and J. I. Kuruvilla, “Design of a low

cost human following porter robot at airports.”

[24] V. Y. Skvortzov, H.-K. Lee, S. Bang, and Y. Lee,

“Application of Electronic Compass for Mobile Robot in an
Indoor Environment,” in 2007 IEEE International
Conference on Robotics and Automation, 2007, pp. 2963–
2970.

[25] “[Link] Slot

Sensor. .

86