Microprocessors and Microsystems 75 (2020) 103059

Contents lists available at ScienceDirect

Microprocessors and Microsystems

journal homepage: [Link]/locate/micpro

Interactive design of intelligent machine vision based on

human–computer interaction mode
Yufeng Shu a,b,∗, Changwei Xiong a,b, Sili Fan a,b
a
Department of Mechanical and Electrical Engineering, DongGuan Polytechnic, DongGuan, China
b
Guangdong textile industry intelligent detection Engineering Technology Research Center(DGPT), DongGuan, China

a r t i c l e i n f o a b s t r a c t

Article history: The intelligent machine vision technology based on man-machine interaction mode has the advantages of
Received 21 January 2020 weak intrusion, low adhesion and no device binding. With the development and progress of science and
Revised 21 February 2020
technology, the intelligent machine vision technology has become one of the most important directions in
Accepted 24 February 2020
the field of human–computer interaction. Compared with the traditional interactive mode, the intelligent
Available online 26 February 2020
machine vision interaction technology is quite convenient, and the existence of these problems can also
Keywords: influence the vision of the intelligent machine to a certain extent. And the wide application of the sense
Human–computer interaction. At present, the operation precision is the main basis of the machine vision interaction design.
Interaction Therefore, this paper studies the precision of the visual interaction of the intelligent machine. In this
Machine paper, three point-to-click experiments will be carried out on the basis of the Fitts’ Law. The accuracy
Vision of the machine vision algorithm is experimentally studied from the operating direction, the mapping
Operation precision
scale equation and the machine vision algorithm, and a reasonable suggestion is made for the intelligent
Interactive mapping
machine vision interactive design of the human interaction mode.
© 2020 Elsevier B.V. All rights reserved.

1. Introduction 2. Evaluation method and accuracy analysis of interactive

performance
Human–computer interaction is based on the research and im-
plementation of computer interaction. Human–computer interac- 2.1. Evaluation method
tion is based on the interdisciplinary fields of computer technol-
ogy and human–machine science [1]. It is also widely used in With the development of science and technology and the ad-
many fields, including modeling, design, evaluation and other theo- vancement of the times, non-keyboard input task types have
ries and methods. Good human–computer interaction methods will become more and more, which must include fixed-point and
fully understand human characteristics and pay more attention to indefinite-point interactive tasks. As for the conceptual model of
user demand, so as to better realize human–computer interaction pointing tasks, Fitts ’Law has extracted the most basic interactive
technology. In the process of human–computer interaction devel- tasks. This model concept is widely used in many areas of human–
opment, gesture interaction is increasingly used, thereby achiev- computer interaction, and it is also a commonly used evaluation
ing higher popularity [2]. The use of gesture interaction will in- method by related personnel [4,5].
evitably make people more convenient for human–computer inter- L Scott MacKenzie and others studied eye tracking systems. In
action. Therefore, in the process of using gestures to interact, it is this research, they studied eye tracking performance based on the
necessary to create a more natural and intuitive human–computer international standards of ISO 9241-9 [6]. They choose three ad-
communication mode [3]. vanced tracking technologies and compare them with traditional
methods. The performance index obtained through the conclusion
is used as the standard for user performance measurement, and
the conclusion is drawn based on these standards. The require-
ment for a long-term follow-up survey requires the respondent to
stare at the screen for a long time, and then stay for 750 ms to
∗
Corresponding author at: Department of Mechanical and Electrical Engineering, select a relevant action; the follow-up survey for a period of time
DongGuan Polytechnic, DongGuan, China.
requires the respondent to stay for 500 ms. The final conclusion
E-mail addresses: shuyf@[Link] (Y. Shu), 513268403@[Link] (C. Xiong),
592807575@[Link] (S. Fan). shows that the performance index of the long-term eye tracking

[Link]
0141-9331/© 2020 Elsevier B.V. All rights reserved.
2 Y. Shu, C. Xiong and S. Fan / Microprocessors and Microsystems 75 (2020) 103059

survey is smaller than the short-term tracking index. At the same

time, in the study, eye tracking was compared with a mouse click
to determine the time to determine the target, and then the three
technologies were compared to get the best performance index [7].
The eye tracking performance index was 4.36 bits / s. The perfor-
mance of the mouse The index is 4.56 bits / s, and there is little Fig. 1. Application of Fitz’s law in human–computer interaction.
difference between them.
The international standards of ISO9241-9 are defined as:
IDe 2.2. Usability analysis of gesture interaction accuracy
T hroughput =
T
 D
 Usability design goes to the design of interaction interfaces, ob-
IDe = log 2 1 + jects, and environments. The user’s fault tolerance rate is an impor-
We
tant content of gesture interaction accuracy. A user’s fault tolerance
W e = 4.133 × SD (1) rate can reflect a general aspect of interaction accuracy [12].
In the research of this paper, the main focus is on interaction
(1) Fitz’s Law [8]
accuracy. According to the operation, the operation accuracy is di-
Fitz’s law has a very important position in the field of human– vided into low, medium and high. Among these three levels, the
computer interaction, and it is widely used in evaluation. Fitz’s Law pixels selected are 8, 40, and 80. According to these three levels,
refers to a basic model of human–machine interaction and human the experimental levels are divided [13].
activity that is a real or virtual touch between human and ma-  D
  960

chine. It predicts the time required to reach the target area from Low accuracy : ID = log 2 1 + = log 2 1 + = 3.7bit
the beginning of the movement and What is the relationship be- W 80
tween the distance of the distance and the size of the target area. (3)
At the same time, Fitz’s Law also shows that a basic problem is
that when the target area is relatively far and small, it is more dif-
ficult to hit. Therefore, in order to improve the speed and accuracy  D
  800

of focusing on this problem, it is necessary to shorten the target Medium accuracy : ID = log 2 1 + = log 2 1 + = 4.39bit
W 40
distance and increase the size of the target area.
(4)
Fitz’s law has important applications in many fields and plays
an important role in the field of human–computer interaction. The
more common Fitz’s law is the Shannon formula proposed by a  D
  640

professor at York University: High accuracy : ID = log 2 1 + = log 2 1 + = 6.34bit
 D
 W 8
T = a + b log 2 1 + (5)
W
 D

ID = log 2 1 + (2) 2.3. Factors affecting the accuracy of gesture interaction
W

T refers to the average time to complete the job (1) Operating direction
a represents the start and end time of the device, and b rep-
resents the speed of the device. These data can be obtained The operation direction is one of the important factors affect-
through experiments. ing the accuracy of gesture interaction. Because most of the ges-
D refers to the distance from the starting position to the target ture interaction process is done in the air, the driving force of the
area human body in different operating directions, the habits of people,
W refers to the width of the target area during interpersonal and the difficulty in moving in the air are different. of. Therefore,
interaction when people use gesture interaction, it will affect accuracy [14,15].

ID indicates the difficulty of task completion. The larger the (2) Mapping scale equation
value, the higher the difficulty of the task and the more difficult
it is to complete. The mapping ratio equation refers to the law of the relative pro-
According to this equation of Fitz’s law, the following conclu- portion of the movement of the input device in the air equivalent
sions can be drawn. For small goals or distant goals, it often takes to the movement in the virtual space in reality [16].
more time and effort, and the difficulty factor is greater. Therefore, In order to increase the accuracy of gesture interaction in the
for near targets and large targets, the difficulty factor is relatively process of gesture interaction, it is necessary to appropriately en-
small, and it is relatively easy to complete. large or reduce the mapping distance equation in the operating
space, reduce or increase the scale of the mapping scale equation,
(2) Mathematical analysis methods and adjust the operation time And so on, thereby improving the
accuracy in the interaction process. Fig. 1.
The analysis of data needs to be based on the conclusions of
relevant experiments and data records, and use scientific and ef- (3) Equipment accuracy [17]
fective methods for analysis, so that scientific and accurate conclu-
sions can be obtained [9,10]. At the same time, these conclusions The accuracy of the device will affect the accuracy of gesture
can be obtained at an objective level verification. At the same time, interaction. For devices with higher accuracy, it is more desirable
it should be noted that in the process of mathematical analysis of for users to see. When the accuracy of the device is higher than
interpersonal interaction, it is necessary to make a survey of the the ideal accuracy of the user, the user will be more willing to
subjective impression of the user through a questionnaire to ob- use this accuracy, which will also increase the user’s satisfaction
tain a scientific result in many aspects [11]. [18,19].
 Y. Shu, C. Xiong and S. Fan / Microprocessors and Microsystems 75 (2020) 103059 3

Table 1
Test results.

D (pixel) W (pixel) ID (pixel) T (s)

960 80 3.70
960 72 3.84
945 64 3.98
920 56 4.12
870 48 4.26
800 40 4.39
910 32 4.88
970 24 5.37
895 16 5.83
640 8 6.34

The following data is calculated based on related formulas, as

the relevant data in horizontal, vertical and diagonal directions.
Through the data analysis, we can get that the correlation coef-
Fig. 2. Program interface. ficients of horizontal, vertical and diagonal are 0.95, 0.88 and 0.84,
respectively, which are more than 0.7, which are highly correlated.
Therefore, the effect of linear regression in these three directions is
3. Influence of operating direction on accuracy very significant, but relatively speaking, the data in the horizontal
direction is the most significant and stable.
3.1. Experimental process In terms of performance, the horizontal, vertical and diago-
nal directions are 1.57 bit/s, 1.49 bit/s and 1.46 bit/s respectively.
When studying the influence of the operation direction on ac- Therefore, from the perspective of performance direction, the per-
curacy, it is mainly based on the influence of the accuracy on the formance in the diagonal direction is the worst among the three,
gesture interaction when the user operates in different operation and the performance in the horizontal direction is the best.
directions. Fig. 2. Comfort and usability are the main factors for users to con-
In order to study the influence on accuracy in different oper- [Link] the perspective of usability, there is no obvious differ-
ating directions, this paper selects horizontal, vertical and exper- ence in the three directions, but it can be seen from the conclu-
imental research in the direction of the object. The experiment is sion that it is the most comfortable in terms of level and is also
first performed in the horizontal direction. The specific experimen- the user’s favorite. Therefore, in terms of comfort and usability, the
tal process is as follows: horizontal direction is operational best choice.
Before doing experiments, you need to modulate the resolution
of relevant equipment, prepare the environment, and adjust the 3.3. Design recommendations
mapping equation to:
⎧
⎪20Vhand Vhand ≤ 1.50 By analyzing and investigating the above problems and analyz-
⎪
⎪
⎨68 − 72 1.50 < Vhand < 4.00
ing the experimental data, it can be known that in the experiments
hand of clicking in these three directions, the target width is 24 pixels
V cursor − 800 (6)
⎪
⎪150Vhand − 400 4.00 < Vhand < 8.00 and the difficulty factor is about 5.38. The overall performance in
⎪
⎩ the diagonal direction is the worst Yes, horizontal is best. Although
200Vhand − 800 Vhand ≥ 8.00
the difficulty of operation is almost the same, the horizontal direc-
(1) Program interface tion is the most comfortable for the user in terms of comfort, so
it is designed to be the horizontal operation direction as much as
Set the direction to horizontal, vertical, and diagonal directions,
possible Table 1.
so its degree is set to 0, 90, 135;
It can be known from the experimental results that there is no
N is the number of times when the target is hit, the number of
obvious difference in the difficulty of operation in the three direc-
experimental hits in this article is 4 times, W is the width of the
tions at medium and low accuracy. When it comes to high pre-
operation target, and D is the horizontal moving distance.
cision, the user thinks that the horizontal direction is the most
In this article, in order to facilitate the recording of exper-
comfortable operation direction, and the horizontal direction is the
iments, the experiments should be numbered to facilitate later
best in terms of overall performance. Therefore, it is suggested that
recording.
when designing intelligent machine vision interaction based on
(2) Test the experiment, enter a width and a horizontal distance the human–computer interaction mode, the interactive interface
for each group, and then output the results. should try to make the width of the target not less than 24 pix-
(3) After the data is collected, a questionnaire survey is required els, and also control its difficulty factor to not exceed 5.37. When
for the relevant testers. performing interactive tasks, try to control and perform the inter-
(4) Change the research direction, and then conduct data collec- active tasks in the horizontal direction as much as possible.
tion and survey.
4. The impact of mapping equations on accuracy
3.2. Experimental results
4.1. Experimental process
In the course of the experiment, three directions of horizon-
tal, vertical and diagonal are selected as experiments. From these For the impact of the mapping equation on accuracy, there
three directions, the influence of the interaction accuracy in the are mainly long-distance and short-distance mapping equations, so
three directions is analyzed, and the results after the experiment you can study from these two aspects, one is to reduce the dis-
are statistically summarized: tance of movement, and the other is the amplitude of short-range
4 Y. Shu, C. Xiong and S. Fan / Microprocessors and Microsystems 75 (2020) 103059

Table 2 Table 4
Summarizes the data in horizontal, vertical, and diagonal directions. Take the data summary of 20, 30 and 40 pixels/s for Vcursor.

D W ID Horizontal direction Vertically Diagonal direction A (pixel) W (pixel) ID (bit) 20pixels/s T 20pixels/s T 20pixels/s T

960 80 3.7 1873.95 1792.44 1886.45 960 80 3.7 2401.03 1873.95 1688.67
960 72 3.84 1896.38 2175.50 1698.06 960 72 3.84 2046.67 1896.38 1751.08
945 64 3.98 2230.57 2187.39 2046.11 945 64 3.98 1885.67 2230.57 1953.11
920 56 4.12 2148.00 2252.33 2703.23 920 56 4.12 2678.45 2148.00 1842.83
870 48 4.26 2320.67 2296.00 2086.54 870 48 4.26 2431.33 2320.67 2050.00
800 40 4.39 2409.08 2324.72 2038.34 800 40 4.39 3424.50 2409.08 2339.38
910 32 4.88 2684.33 2448.62 2548.03 910 32 4.88 3362.78 2684.33 3395.22
970 24 5.37 3250.85 3277.89 2809.03 970 24 5.37 3330.28 3250.85 2923.38
895 16 5.83 4778.53 4461.45 4707.71 895 16 5.83 4018.89 4778.53 2794.50
640 8 6.34 6249.38 8175.39 9446.73 640 8 6.34 5492.67 6249.38 10452.67

Table 5
Table 3
Vcursor takes the parameter values of 20, 30 and 40 pixels/s respectively.
Test results.
20 pixels/s 30 pixels/s 40 pixels/s
Horizontal direction Vertically Diagonal direction
Mean 3107.41 2984.17 3318.98
Mean 2984.17 3139.18 3196.25
Throughput 1.50 1.57 1.41
Throughput 1.57 1.49 1.46
a −2047.01 −4064.14 −8667.99
a −4064.14 −5612.14 −7098.26
b 1103.49 1508.95 2566.25
b 1508.95 1873.54 2203.94
r 0.92 0.95 0.86
r 0.95 0.88 0.84

⎧
⎪26.7Vhand, Vhand ≤ 1.50
movement. Research to improve the accuracy of the interaction.
⎪
⎨ 64Vhand − 56, 1.50 < Vhand < 4.00
This experiment mainly adjusts from fast speed to slow speed. Vcursor (7)
⎪
⎪150Vhand − 400, 400 < Vhand < 8.00
At short distance, we select Vcursor ’of 20 pixel/s and 40 pixel/s ⎩
respectively, the rest are unchanged, and the most comfortable 20 0Vhand − 80 0, Vhand ≥ 8.00
horizontal direction is selected as the research object.
Table 4 summarizes the data when Vcursor takes 20, 30, and
At long distances, we select Vcursor ’s of 600 pixel/s and
40 pixels/s:
900 pixel/s respectively, the rest are unchanged, and the most
The above data is calculated by the formula as the values of a,
comfortable horizontal direction is selected as the research object.
b, and r above the horizontal, vertical, and diagonal directions.
The purpose of this experiment is to explore the impact of dif-
By analyzing the above data, it can be concluded that when
ferent mapping equations on interaction accuracy. The specific pro-
Vcursor takes 20, 30, and 40 pixels/s, the correlation coefficients
cess of the experiment is as follows: Table 2.
r are 0.91, 0.94, and 0.85, respectively. These three values are all
greater than 0.7, which are highly linearly correlated. Three map-
(1) Make preliminary preparations for the experimental equip- ping equations The linear regression effect is very significant, and
ment and experimental environment, adjust the screen res- Vcursor is most significant when taking 30 pixels/s. The stability is
olution and the corresponding mapping equations. the worst at 40 pixels/s. (Table 5).
(2) Familiarize the tester with the test equipment and operation
steps. After being familiar with it, you can test it.
4.3. Proposed amendments
(3) Measure the tester, complete each task in Table 3 at the
same time, and output the data after completion.
According to the experimental verification and the question-
(4) Complete the collection of data and conduct a questionnaire
naire, in the mapping equation of short-distance movement, there
survey on the test subjects.
is almost no impact on the accuracy of low or medium, but at
(5) Change the mapped equation, and then complete the test
high precision, the impact is relatively large. Therefore, it is sug-
again according to the previous steps to get the results.
gested that when designing the mapping equations of intelligent
machine vision interaction based on the human–computer interac-
tion mode, the mapping equations used for movement over short
4.2. Experimental results distances should not be too fast, and the mapping equations should
not be too fast when moving over long distances. It can’t be too
In the experiment, different mapping equations are used to ver- slow. So it is necessary to choose the proper mapping equation.
ify it experimentally, and based on the results of the experiment,
improvements are made, and proposals for improvement and de- 5. Conclusion
sign are proposed.
In the short-range modification, 20 pixels/s and 40 pixels/s With the development of science and technology and the devel-
were selected respectively, and the other conditions are un- opment of computer technology, intelligent machine vision inter-
changed. Click on the horizontal task. action based on human–computer interaction mode has gradually
The mapping equations are: become an important part of our daily lives. In terms of gesture in-
teraction technology, due to various high-tech developments, there
⎧
⎪13.3Vhand, Vhand ≤ 1.50 has been more understanding of its interpretation. Gestures, as
⎪
⎨ the most common connection between people, are bound to be
72Vhand − 88, 1.50 < Vhand < 4.00
Vcursor used more by people, and they are more intuitive to interact with.
⎪
⎪150Vhand − 400, 400 < Vhand < 8.00
⎩ Therefore, it is important to effectively use gesture interaction be-
20 0Vhand − 80 0, Vhand ≥ 8.00 tween human and machine.
 Y. Shu, C. Xiong and S. Fan / Microprocessors and Microsystems 75 (2020) 103059 5

This paper mainly studies the interaction accuracy of gesture [14] J. Liu, X. Sheng, D. Zhang, et al., Reduced daily recalibration of myoelectric
interaction in terms of usability. And according to the relevant data prosthesis classifiers based on domain adaptation, IEEE J. Biomed. Health In-
form. 20 (1) (2015) 166–176.
theory, its click experiment, and experiments on the accuracy of [15] M. Pilarski P, R. Dawson M, T. Degris, et al., Adaptive artificial limbs: a real–
gesture interaction, and design the above improvements for the ex- time approach to prediction and anticipation, IEEE Robot. Autom. Mag. 20 (1)
perimental results. (2013) 53–64.
[16] M. Liu, F. Zhang, H. Huang, An adaptive classification strategy for reliable loco-
motion mode recognition, Sensors 17 (9) (2017) 2020.
Declaration of Competing Interest [17] C. Vidaurre, M. Kawanabe, P. Von B, et al., Toward unsupervised adaptation
of LDA for brain-computer interfaces, IEEE Trans. Biomed. Eng. 58 (3) (2011)
587–597.
The authors declare that they have no known competing finan-
[18] F. Zhong, R. Kumar, C. Quan, RGB laser speckles based 3D profilometry, Appl.
cial interests or personal relationships that could have appeared to Phys. Lett. 114 (20) (2019) 201104.
influence the work reported in this paper. [19] X. Chen, D. Zhang, X. Zhu, Application of a self-enhancing classification method
to electromyography pattern recognition for multifunctional prosthesis control,
J. Neuroeng. Rehabil. 10 (1) (2013) 44.
References
YuFeng Shu. Institution: Department of Mechanical and
[1] A. Jones L, J. Lederman S, in: Human Hand Function, Oxford University Press, Electrical Engineering, DongGuan Polytechnic. Postal ad-
New York, 2006, pp. 251–263. dress: No. 3, University Road, high-tech Industrial Devel-
[2] E. Biddiss, T. Chau, Upper-limb prosthetics: critical factors in device abandon- opment Zone, Songshan Lake, Dongguan City,GuangDong
ment, Am. J. Phys. Med. Rehabil. 86 (12) (2007) 977–987. Province, China.
[3] D. Marasco P, K. Kim, E. Colgate J, et al., Robotic touch shifts perception of
embodiment to a prosthesis in targeted reinnervation amputees, Brain (134)
(2011) 747–758.
[4] Q. Zhong F, X. Shao X, C. Quan, 3D digital image correlation using a sin-
gle 3CCD colour camera and dichroic filter, Meas. Sci. Technol. 29 (4) (2018)
045401.
[5] D. Farina, N. Jiang, H. Rehbaum, et al., The extraction of neural information
from the surface EMG for the control of upper-limb prostheses: emerging
avenues and challenges, IEEE Trans. Neural Syst. Rehabil. Eng. 22 (4) (2014)
797–809. ChangWei Xiong. Institution: Department of Mechan-
[6] C. Antfolk, M. D’Alonzo, B. Rosén, et al., Sensory feedback in upper limb pros- ical and Electrical Engineering, DongGuan Polytechnic.
thetics, Expert Rev. Med. Dev. 10 (1) (2013) 45–54. Postal address: No. 3, University Road, high-tech In-
[7] L. Jiang, Q. Huang, D. Yang, et al., A novel hybrid closed-loop control approach dustrial Development Zone, Songshan Lake, Dongguan
for dexterous prosthetic hand based on myoelectric control and electrical stim- City,GuangDong Province, China.
ulation, Ind. Robot (2018), doi:10.1108/IR- 12- 2017- 0209.
[8] E. Scheme, K. Englehart, Electromyogram pattern recognition for control of
powered upper-limb prostheses: state of the art and challenges for clinical use,
J. Rehabil. Res. Dev. 48 (6) (2011) 643–659.
[9] Y. Gu, D. Yang, Q. Huang, et al., Robust EMG pattern recognition in the pres-
ence of confounding factors: features, classifiers and adaptive learning, Expert
Syst. Appl. 96 (2018) 208–217.
[10] W. Yang, D. Yang, Y. Liu, et al., A 3-DOF hemi-constrained wrist motion/force
detection device for deploying simultaneous myoelectric control, Med. Biol. SiLi Fan. Institution: Department of Mechanical and Elec-
Eng. Comput. 5 (2018) 1–13. trical Engineering, DongGuan Polytechnic. Postal address:
[11] D. Yang, W. Yang, Q. Huang, et al., Classification of multiple finger motions No. 3, University Road, high-tech Industrial Development
during dynamic upper limb movements, IEEE J. Biomed. Health Inform. 21 (1) Zone, Songshan Lake, Dongguan City,GuangDong Province,
(2017) 134. China.
[12] M. Yamagami, M. Peters K, I. Milovanovic, et al., Assessment of dry epider-
mal electrodes for long-term electromyography measurements, Sensors 18 (4)
(2018) 1269.
[13] J. Liu, X. Sheng, D. Zhang, et al., Towards zero retraining for myoelectric control
based on common model component analysis, IEEE Trans. Neural Syst. Rehabil.
Eng. 24 (4) (2016) 444–454.