TrackNet: A Deep Learning Network for Tracking

High-speed and Tiny Objects in Sports Applications

Yu-Chuan Huang I-No Liao Ching-Hsuan Chen Tsı̀-Uı́ İk∗ Wen-Chih Peng
Department of Computer Science, College of Computer Science
National Chiao Tung University
1001 University Road, Hsinchu City 30010, Taiwan
∗
Email: cwyi@[Link]
arXiv:1907.03698v1 [[Link]] 8 Jul 2019

topic in the areas of image processing and deep

Abstract—Ball trajectory data are one of the most
fundamental and useful information in the evaluation of
learning. In the applications of sports analyzing and
players’ performance and analysis of game strategies.
athletes training, videos are helpful in the post-game
Although vision-based object tracking techniques have
review and tactical analysis. In professional sports,
been developed to analyze sport competition videos, it is
high-end cameras have been used to record high
still challenging to recognize and position a high-speed
resolution and high frame rate videos and combined
and tiny ball accurately. In this paper, we develop a deep
with image processing for referee assistance or data
learning network, called TrackNet, to track the tennis
ball from broadcast videos in which the ball images are
collection. However, this solution requires enormous
small, blurry, and sometimes with afterimage tracks or
resources and is not affordable for individuals or
even invisible. The proposed heatmap-based deep learning
amateurs. Developing a low-cost solution for data
network is trained to not only recognize the ball image
acquisition from broadcast videos will be significant
from a single frame but also learn flying patterns from
for massive sports data collection.
consecutive frames. TrackNet takes images with the size
Ball trajectory data are one of the most funda-
of 640 × 360 to generate a detection heatmap from either
mental and useful information for game analysis.
a single frame or several consecutive frames to position
the ball and can achieve high precision even on public
However, for some sports such as tennis, badminton,
domain videos. The network is evaluated on the video of
baseball, etc., the ball is not only small but also may
the men’s singles final at the 2017 Summer Universiade,
fly as fast as several hundred kilometers per hour,
which is available on YouTube. The precision, recall, and
resulting in tiny and blurry images. That makes the
F1-measure of TrackNet reach 99.7%, 97.3%, and 98.5%,
ball tracking task becomes more challenging than
respectively. To prevent overfitting, 9 additional videos
other sports. In this paper, we design a heatmap-
are partially labeled together with a subset from the
previous dataset to implement 10-fold cross validation, and
based deep learning network, called TrackNet, to
the precision, recall, and F1-measure are 95.3%, 75.7%,
precisely position ball of tennis and badminton on
and 84.3%, respectively. A conventional image processing
broadcast videos or videos recorded by consumer’s
algorithm is also implemented to compare with TrackNet.
devices such as smartphones. TrackNet overcomes
Our experiments indicate that TrackNet outperforms con-
the issues of blurry and remnant images and can
ventional method by a big margin and achieves exceptional
even detect occluded ball by learning its trajectory
ball tracking performance. The dataset and demo video are
available at [Link]
patterns. The proposed network can be applied to
Index Terms—Deep Learning, neural networks, tiny
other ball-based sports and help both amateurs and
object tracking, heatmap, tennis, badminton
professional teams collect data with a moderate
budget.
I. I NTRODUCTION Conventional image recognition is usually based
Video considered as logs of visual sensors con- on the object’s appearance features such as shape,
tains a large amount of information. Information color, size, etc., or statistical features such as HOG,
extraction from videos has become a hot research SIFT, etc. Due to a relatively long shutter time of
consumer or prosumer cameras, images of high- detection. At last, the position of our target object
speed objects are prone to suffer from afterimage is calculated based on the heatmap generated by the
or blur issues, resulting in poor image recognition deep learning network. To meet the characteristics
accuracy. The performance of ball tracking can be of tennis and badminton games, our calculation and
improved by pairing candidates from frame to frame evaluation are based on the assumption that there is
according to trajectory models to find the most at most one ball on the court.
possible one [1]. In addition, a classical technique
in image processing to improve image quality is To evaluate the proposed network, we have la-
by fusing multiple low-quality images. Based on beled 20, 844 frames from the broadcast of men’s
the above observations, instead of using the rule- singles final at the 2017 Summer Universiade. To
based techniques, we propose to adopt deep learning assess the performance of the proposed consecu-
network to recognize the shape of the ball and tive input frames technique, both single-frame and
learn the trajectory patterns by applying multiple multiple-frame versions of TrackNet are imple-
consecutive frames to solve the mentioned issues. mented. Along with the conventional image recog-
Object classification and detection are two of the nition algorithm [1], a comprehensive comparison
earliest studies in deep learning. VGG-16 [2] is among different models is performed. Experiments
one of the most popular networks for feature map indicate that the proposed TrackNet outperforms the
encoding. To detect and classify multiple objects in conventional image recognition algorithm and effec-
an image, the R-CNN family [3] [4] [5] structurally tively locates fast-moving tennis ball from broadcast
examine the picture in two stages. It firstly selects sport competition videos. Moreover, to prevent the
many areas that may contain interesting objects, notorious overfitting issue that happens frequently
called Region of Interests (RoIs), and then applies in deep learning solutions, additional data from
object detection and classification techniques on 9 tennis games on different courts are added to
these regions. However, its performance cannot ful- the training dataset, including grass court, red clay
fill the needs of real-time applications. To speed up, court, hard court, etc. Additionally, to explore the
the YOLO family [6] develops a one-stage end-to- model extensibility, badminton tracking by Track-
end approach to detect objects in a limited search Net is evaluated. We have labeled 18, 242 frames
space, significantly reducing the computing time. from the video of 2018 Indonesia Open Final - TAI
The streamlined version of Tiny YOLO can even run Tzu Ying vs CHEN YuFei. Although badminton
on the Raspberry Pi. Compared to the block-based travels much faster than tennis, our experimental
algorithms, Fully Convolutional Networks (FCN) results exhibit a decent performance.
proceeds pixel-wise classification. To compensate
for the size reduction of the feature map during the The critical contribution of TrackNet comes from
encoding process, upsampling and DeconvNet [7] its capability of precisely tracking fast-moving and
are often used to decode the feature map, generating tiny objects by learning the dynamic behavior of the
an original size of the data array. trajectory. In the tennis tracking application, 10-fold
In this paper, a deep learning network, called cross validation results in an outstanding perfor-
TrackNet, is proposed to realize a precise trajec- mance of 95.3% precision, 75.7% recall, and 84.3%
tory tracking network. Firstly, VGG-16 is adopted F1-measure. Such capability shows great potential
to generate the feature map. Different from other in expanding the variety of computer vision applica-
deep learning networks, TrackNet can take multiple tions. The rest of the paper is organized as follows.
consecutive frames as input. In this way, TrackNet Section II provides an introduction to the relevant
learns not only the features of the ball but also the researches and the convolutional neural network.
characteristics of ball trajectories to enhance its ca- Section III introduces the datasets used in this paper.
pability of object recognition and positioning. Since Section IV elaborates the proposed deep learning
images are downsampled and encoded by pooling network and Gaussian heatmap techniques. Section
layers, the network follows the upsampling mech- V provides experimental results and performance
anism of FCN to generate the heatmap for object evaluation. At last, Section VI concludes this paper.

2
 II. R ELATED W ORKS sampling, and deconvolution/up-sampling. A soft-
max layer is usually used as the output layer.
In recent years, the analysis of player perfor- For example, the widely used VGG-16 [2] mainly
mance and game tactics based on the trajectory consists of convolutional, maximum pooling, and
data of balls and players has received more and ReLU layers. Conceptually, front-end layers learn
more attention [8] [9] [10] [11]. Many tracking to identify simple geometric features, and back-end
algorithms and systems have been developed to layers are trained to identify object features.
compute and collect the trajectory data. Current In CNNs, each layer is a W × H × D data array.
commercial solutions mainly rely on high resolution W , H, and D denote the width, height, and depth
and high frame rate video, resulting in high hard- of the data array, respectively. The convolution
ware investment. For example, the Hawk-Eye sys- operation is a filter with a kernel of size w × h × D
tem [12] has been extensively used in professional across the W × H range with the stride parameter
competitions to calculate ball trajectories and assist s being set as 1 in many applications. To avoid
the referee in clarifying controversial calls through information loss near the boundary or maintain the
3D visual depictions. Nonetheless, the system has to size of the output data array, columns and rows of
deploy high-end cameras with dedicated operators the data array can be padded with zero by setting the
at selected locations and angles. The expense is too padding parameter p. Figure 1 depicts the relevant
high for non-professional teams. parameters of the convolution operation. Let W 0 and
Attempting to position the ball from sports com- H 0 denote the width and height of the next layer.
petition videos has been studied for years. However, Then,
since the ball size is relatively small, it is prone to be W + 2p − w H + 2p − h
confused with objects having similar color or shape, W0 = + 1 and H 0 = + 1.
s s
causing false positives. Furthermore, due to the high
moving speed of the ball, the resulting image is
usually blurry, inducing false negatives. By explor-
ing the trajectory pattern from consecutive frames,
the ball positioning can be effectively improved.
In addition, the flight trajectory itself possesses
important information and is a subject in many
pieces of research [13]. For instance, combining
multiple cameras with 3D technology for tennis
detection [14], tracking tennis by particle filter in
low-quality films [15], and adopting two-layer data
Fig. 1. Convolution operation in deep learning networks.
association approach to calculate the most likely
ball trajectory from the results of failure detection Since the convolution operation is linear and
in the frame-by-frame image processing [16] are cannot effectively capture nonlinear behaviors, an
enlightening studies. activation function called rectifier is introduced to
The success of deep learning techniques in image capture nonlinear behaviors. The Rectified Linear
classification [2] [17] encourages more researchers Unit (ReLU) is the most commonly used activation
to adopt these methods to solve various problems function in deep learning models. If the input value
such as object detection and interception [5] [6] is negative, the function returns 0; otherwise, the
[18], computer games, network security, activity function returns the input value. ReLU can be
recognition [19] [20], text and image semantic anal- expressed as f (x) = max(0, x). Maximum pooling
ysis, and smart stores. The infrastructure of the deep provides the functionality of down-sampling and
learning network is a structured and huge convolu- feature fusion. Maximum pooling fuses features by
tional neural network trained with a large amount encoding data via down-sampling. The block of
of labeled data. The most common operations of data will be represented only by the largest one.
CNNs include convolution, rectifier, pooling/down- After pooling, the data size is reduced. On the other

3
hand, to achieve pixel-by-pixel classification, up- TABLE I
sampling is necessary to reconstruct an output with S EGMENTS OF LABEL FILES .
the same size as the original image [21] [22]. In up-
..
sampling, samples are duplicated to expand the data .
size. Batch normalization is a widely used technique [Link], 2, 727, 447, 0
to speed up the training process. Each W × H data [Link], 1, 735, 457, 0
[Link], 1, 722, 433, 1
array is independently standardized into a normal [Link], 1, 707, 403, 0
distribution. ..
.
Backward propagation is commonly used in train- [Link], 1, 555, 220, 0
ing neural networks to learn the filter coefficients. [Link], 1, 550, 218, 2
Firstly, forward propagation is performed to have [Link], 1, 547, 206, 0
..
a preliminary prediction. Then, compared the pre- .
diction with the ground truth, a loss function will
be evaluated. Finally, the weights of the model,
i.e., the filter coefficients, are updated according dilation and erosion to generate candidates. To filter
to the loss by the gradient descent method. Chain out wrong candidates, in our implementation, a
rule is adopted to calculate the gradient of the loss fully-connected neural network is trained to classify
function layer by layer. The process will be repeated candidates into positive and negative categories. The
again and again until a certain number of repetitions one that has the highest probability in the positive
is reached or the loss falls below an acceptable category is selected, indicating the position of the
threshold. The design of the loss function is an ball.
important factor that affects the training efficiency
and the performance of the network. Commonly III. DATASET
used loss functions include Root Mean Square Error Our first dataset is from the broadcast video of
(RMSE) and cross-entropy. the tennis men’s singles final at the 2017 Summer
In this paper, we propose a deep learning network Universiade. The resolution, frame rate, and video
named TrackNet to detect tennis and badminton length are 1280 × 720, 30 fps, and 75 minutes,
on broadcast sport competition videos. By training respectively. By screening out unrelated frames, 81
with consecutive input frames, TrackNet can not game-related clips are segmented and each of them
only recognize the ball but also learn its trajectory records a complete play, starting from ball serving to
pattern. A heatmap which is ideally a Gaussian score. There are 20, 844 frames in total. Each frame
distribution centered on the ball image is then possesses the following attributes: ”Frame Name”,
generated by TrackNet to indicate the position of ”Visibility Class”, ”X”, ”Y”, and ”Trajectory Pat-
the ball. The idea of exploiting heatmap for object tern”. Table I is pieces of label files.
detection has been adopted in many studies [23] ”Frame Name” is the name of the frame files.
[24]. ”Visibility Class”, VC for short, indicates the visi-
To compare and evaluate the performance of bility of the ball in each frame. The possible values
TrackNet, we implement Archanas algorithm [1] are 0, 1, 2, and 3. V C = 0 implies the ball is not
which uses conventional image processing tech- within the frame. V C = 1 implies the ball can be
niques to detect tennis ball. Archana’s algorithm easily identified. V C = 2 implies the ball is in the
firstly smooths the image of each frame by a median frame but can not be easily identified. For example,
filter to remove noise. After a background model as shown in Figure 2, the ball in [Link] is hardly
is calculated, background subtraction is performed visible since the color of the tennis ball is similar
to obtain the foreground. Then, the difference be- to the text ”Taipei” on the court. However, with the
tween frames by logical AND operation is examined help of neighboring frames, [Link] and [Link],
to identify fast-moving foreground objects. Those the unclear ball position of [Link] can be labeled.
objects are compared with shape, size, and aspect Figure 2 (d), (e), and (f) illustrate the labeling
ratio of the tennis ball and selected by applying results. V C = 3 implies the ball is occluded by

4
other objects. For example, as shown in Figure 3, the
ball in [Link] is occluded by the player. Similarly,
based on the information from neighboring frames,
[Link] and [Link], the ball position of [Link]
can be estimated. Figure 3 (d), (e), and (f) illustrate
the labeling results. In the dataset, the number of
frames of V C = 0, 1, 2, 3 are 659, 18035, 2143,
and 7, respectively.

Fig. 4. An example of the prolonged tennis trace.

hit, and bouncing. They are labeled by 0, 1, and 2,

respectively. Figure 5 is an example of striking a
ball. The ball is flying at [Link] and [Link]. At
[Link], the ball is labeled as hit. Figure 6 shows a
bouncing case. The ball has not reached the ground
Fig. 2. The ball image is hardly visible. at [Link] and [Link]. At [Link], the ball hits
the ground and is labeled as bouncing.

Fig. 5. A hit case: (a) and (b) are labeled as flying, and (c) is labeled
as hit.

Fig. 3. The ball is occluded by the player.

”X” and ”Y” indicate the coordinate of tennis in

the pixel coordinate. Due to the high moving speed,
tennis images in the broadcast video may be blurry
and even have afterimage trace. In such cases, ”X” Fig. 6. A bouncing case: (a) and (b) are labeled as flying, and (c)
and ”Y” are considered as the latest position of is labeled as bouncing.
the ball’s trace. For example, as shown in Figure
4, the ball is flying from Player1 to Player2 with a To enrich the variety of training dataset, addi-
prolonged trace and the red dot indicates the labeled tional 16, 118 frames are collected. These frames
coordinate. came from 9 videos recorded at different tennis
”Trajectory Pattern” indicates the ball movement courts, including grass court, red clay court, hard
types and are classified into three categories: flying, court etc. By learning diverse scenarios, the deep

5
learning model is expected to recognize tennis ball sometimes we could hardly identify the position of
at various courts. That increases the robustness the ball. An example of how we label the prolonged
of the model. Further details will be presented in images is shown in Figure 7.
Section V.
In addition to tennis, to explore the versatility
of the proposed TrackNet in the applications of
high-speed and tiny objects tracking, a trial run
on badminton match video is performed. Tracking
badminton is more challenging than tracking tennis
since the speed of badminton is much faster than
tennis. The fastest serve according to the official
records from the Association of Tennis Profession-
als is John Isner’s 253 kilometers per hour at the
2016 Davis Cup. On the other hand, the fastest
badminton hit in competition is Lee Chong Wei’s
Fig. 7. An example of the prolonged badminton trace.
417 kilometers per hour smash at the 2017 Japan
Open according to Guinness World Records, which
is over 1.6 times faster than tennis. Besides, in IV. T RACK N ET
professional competitions, the speed of badminton
is frequently over 300 kilometers per hour. The
faster the object moves, the more difficult it is to be
tracked. Hence, it is expected that the performance
will degrade for badminton compared with tennis.
Our badminton dataset comes from a video of
the badminton competition of 2018 Indonesia Open
Final - TAI Tzu Ying vs CHEN YuFei. The reso-
lution is 1280 × 720 and the frame rate is 30 fps.
Similarly, unrelated frames such as commercial or
highlight replays are screened out. The resulting Fig. 8. An example of the detection heatmap.
total number of frames is 18, 242. We label each
frame with the following attributes: ”Frame Name”, TrackNet is composed of a convolutional neural
”Visibility Class”, ”X”, and ”Y”. network (CNN) followed by a deconvolutional neu-
In badminton dataset, ”Visibility Class” is clas- ral network (DeconvNet) [7]. It takes consecutive
sified into two categories, V C = 0 and V C = 1. frames to generate a heatmap indicating the position
V C = 0 means the ball is not in the frame and of the object. The number of input frames is a
V C = 1 means the ball is in the frame. Unlike network parameter. One input frame is considered
our tennis dataset, we do not classify V C = 2 the conventional CNN network. TrackNet with more
and V C = 3 categories since the badminton moves than one input frame can improve the moving
so fast that blurry image happens very frequently. object detection by learning the trajectory pattern.
Therefore, in the badminton dataset, V C = 1 For the purpose of evaluation, two networks are
includes all status of badminton as long as the ball implemented. One is with single frame input, and
is within the frame no matter it is clearly visible or the other is with three consecutive frames input.
hardly visible. TrackNet utilizes the heatmap-based CNN which
”X” and ”Y” indicate the coordinate of bad- has been proved useful in several applications [23]
minton. Similar to tennis, ”X” and ”Y” are defined [24]. TrackNet is trained to generate a probability-
by the latest position of the ball’s trace considering like detection heatmap having the same resolution as
its moving direction if the image is prolonged. In the input frames. The ground truth of the heatmap is
badminton video, prolonged trace often happens and an amplified 2D Gaussian distribution located at the

6
 Fig. 9. The architecture of the proposed TrackNet.

center of the tennis ball. The coordinates of the ball TABLE II

are available in the labeled dataset and the variance N ETWORK PARAMETERS OF T RACK N ET.
of the Gaussian distribution refers to the diameter
of tennis ball images. Let (x0 , y0 ) be the ball center Layer Filter Size Depth Padding Stride Activation
Conv1 3×3 64 2 1 ReLU+BN
and the heatmap function is expressed as Conv2 3×3 64 2 1 ReLU+BN
   Pool1 2 × 2 max pooling and Stride = 2
1 − (x−x0 )2 +(y−y0)
2
2


G (x, y) = e 2σ 2 2πσ · 255 , Conv3 3×3 128 2 1 ReLU+BN
2πσ 2 Conv4 3×3 128 2 1 ReLU+BN
Pool2 2 × 2 max pooling and Stride = 2
where the first part is a Gaussian distribution cen- Conv5 3×3 256 2 1 ReLU+BN
tered at (x0 , y0 ) with variance of σ 2 , and the second Conv6 3×3 256 2 1 ReLU+BN
part scales the value to the range of [0, 255]. σ 2 = 10 Conv7 3×3 256 2 1 ReLU+BN
Pool3 2 × 2 max pooling and Stride = 2
is used in our implementation since the average ball Conv8 3×3 512 2 1 ReLU+BN
radius is about 5 pixels, roughly corresponding to Conv9 3×3 512 2 1 ReLU+BN
the region of G (x, y) ≥ 128. Figure 8 is a visualized Conv10 3×3 512 2 1 ReLU+BN
heatmap function of a tennis ball. UpS1 2 × 2 upsampling
Conv11 3×3 512 2 1 ReLU+BN
The implementation details of TrackNet is illus- Conv12 3×3 512 2 1 ReLU+BN
trated in Figure 9 and Table II. The input of the Conv13 3×3 512 2 1 ReLU+BN
proposed network can be some number of consec- UpS2 2 × 2 upsampling
Conv14 3×3 128 2 1 ReLU+BN
utive frames. The first 13 layers refer to the design Conv15 3×3 128 2 1 ReLU+BN
of the first 13 layers of VGG-16 [2] for object UpS3 2 × 2 upsampling
classification. The 14-24 layers refer to DeconvNet Conv16 3×3 64 2 1 ReLU+BN
[7] for semantic segmentation. To realize the pixel- Conv17 3×3 64 2 1 ReLU+BN
Conv18 3×3 256 2 1 ReLU+BN
wise prediction, upsampling is applied to recover Softmax
the information loss from maximum pooling layers.
Symmetric numbers of upsampling layers and max-
imum pooling layers are implemented.
The final black-white binary detection heatmap
is not directly available at the output of the deep
learning network. The network outputs a detection (639, 359) and depth within 0 ≤ k ≤ 255. The
heatmap that has continuous values within the range softmax layer calculates the probability distribution
of [0, 255] for each pixel. Let L (i, j, k) denote the of depth k from possible 256 grayscale values. Let
data array of coordinates within (0, 0) ≤ (i, j) ≤ P (i, j, k) denote the probability of depth k at (i, j).

7
The softmax function is given by TABLE III
K EY PARAMETERS USED IN MODEL TRAINING .
eL(i,j,k)
P (i, j, k) = P255 .
eL(i,j,l) Parameters Setting
l=0
Learning rate 1.0
Based on the probability given by the softmax Batch size 2
layer on each pixel, the depth k with the highest Steps per epoch 200
epochs 500
probability is selected as the heatmap value of the Initial weights random uniform
pixel. For each pixel, let Range of initial weights [−0.05, 0.05]

h (i, j) = arg max P (i, j, k)

k

denote the softmax layer output at (i, j), indicating frames are resized from 1280 × 720 to 640 × 360.
the selected grayscale value at (i, j). Once the To optimize weights of the network, the Adadelta
complete continuous detection heatmap is gener- optimizer [26] is applied. Table III summarizes
ated, the coordinate of the ball can be determined other key parameters. Among these parameters, the
by the following two steps. The first step is to number of epochs is one of the most critical factors
pixel-wisely convert the heatmap into a black-white in model training. Underfitting happens if it is too
binary heatmap by the threshold t. If a pixel has a small, while overfitting happens if it is too large.
value larger than or equal to t, the pixel is set to For TrackNet, the characteristic of loss versus the
255. On the contrary, if a pixel has a value smaller number of epochs is shown in Figure 10. Based on
than t, the pixel is set to 0. Based on the previous the simulation, we select 500 epochs as our optimal
discussion regarding the mean radius of a tennis value to prevent both underfitting and overfitting.
ball, threshold t is set as 128. The second step is to
exploit the Hough Gradient Method [25] to find the
circle on the black-white binary detection heatmap.
If exactly one circle is identified, the centroid of
the circle is returned. In other cases, the heatmap is
considered no ball detected.
During the training phase, the cross-entropy func-
tion is used to calculate the loss function based on
P (i, j, k). The corresponding ground truth function
denoted by Q (i, j, k) is given by

1, if G (i, j) = k;
Q (i, j, k) =
0, otherwise. Fig. 10. The loss curve of TrackNet model training.

Let HQ (P ) denote the loss function. Then, To compare the performance of TrackNet frame-
works with one input frame and three consecutive
X
HQ (P ) = − Q (i, j, k) log P (i, j, k) .
i,j,k
input frames, two versions of TrackNet are imple-
mented. For convenience, TrackNet that takes one
V. E XPERIMENTS input frame is named as Model I and TrackNet that
The experiment setup is as followed. The tennis takes three consecutive input frames is named as
dataset elaborated in Section III is used to evaluate Model II. For Model II, three consecutive frames
the performance of Archana’s algorithm, a conven- are used to detect the ball coordinate in the last
tional image processing technique, and the proposed frame. During the training phase, three consecutive
TrackNet. The dataset contains 20, 844 frames and frames are considered a training sequence if the last
is randomly divided to the training set and test set. frame belongs to the training set. Likewise, three
70% frames are the training set and 30% frames consecutive frames are considered a test sequence
are the test set. To speed up the training speed, all if the last frame belongs to the test set. Note that

8
 TABLE IV
P ERFORMANCE S UMMARY.

Archana’s TrackNet Model I TrackNet Model II TrackNet Model II’

VC0 VC1 VC2 VC3 VC0 VC1 VC2 VC3 VC0 VC1 VC2 VC3 VC0 VC1 VC2 VC3
TP - 4046 418 0 - 4933 497 0 - 5223 565 2 - 5234 598 1
FP 201 334 29 1 1 221 20 0 0 3 3 0 4 6 7 2
TN 9 - - - 195 - - - 210 - - - 206 - - -
FN - 947 214 6 - 241 139 7 - 101 93 5 - 87 56 4
Total 210 5327 661 7 196 5395 656 7 210 5327 661 7 210 5327 661 7

TABLE V
ACCURACY METRICS OF DIFFERENT MODELS .

Model Precision Recall F1-measure

Archana’s [1] 92.5% 74.5% 82.5%
TrackNet Model I 95.7% 89.6% 92.5%
TrackNet Model II 99.8% 96.6% 98.2%
TrackNet Model II’ 99.7% 97.3% 98.5%

TrackNet framework is scalable. Any number of

consecutive input frames are allowed.
To define a proper specification for prediction Fig. 11. The distribution of the positioning error.
error, the size of the tennis is investigated. The
diameter of tennis images in the video ranges from
2 to 12 pixels and the mean diameter is around TrackNet Model II are shown in Table IV, where TP,
5 pixels. Since the prediction error within a unit FP, TN, and FN stand for true positive, false posi-
size of the ball does not cause misleading in trajec- tive, true negative, and false negative, respectively.
tory identification, we define the positioning error The numbers are grouped by ”Visibility Class”, VC.
(PE) specification as 5 pixels to indicate whether False positive of VC1, VC2, and VC3 stands for
a ball is accurately detected. Detections with PE predictions with P E larger than 5 pixels. False
larger than 5 pixels belong to false predictions. negative of VC1, VC2, and VC3 means there is
P E is defined by the Euclidean distance between no ball detected or there is more than one ball
the model prediction and the ground truth. Figure detected when there is actually one ball in the frame.
11 shows the P E distribution of TrackNet models. Note that since TrackNet Model I and TrackNet
The x-axis represents P E in the unit of pixels and Model II utilize a different number of input frames,
the y-axis is the percentage of occurrence. x = 0 the training set and test set numbers are different.
stands for perfect detection. x = 1 means PE Archanas and TrackNet Model II’ follow the same
lies in 0 < P E ≤ 1, x = 2 means PE lies in training set and test set as TrackNet Model II.
1 < P E ≤ 2, and so on. Note that occurrence It is observed that compared to Archana’s algo-
percentages of P E > 5 of Model I and Model II rithm, both TrackNet Model I and TrackNet Model
are 4.3% and 0.1%, respectively. That is, 95.7% and II significantly reduce false positives and false neg-
99.9% detections of Model I and Model II fulfill the atives, resulting in an increase of both true positives
specification. and true negatives. The comparison presents an ex-
The Archana’s algorithm [1], an image processing ceptional object detection capability of deep learn-
technique developed by Archana and Geetha, is ing networks over conventional image processing
implemented for comparison. The prediction details algorithms. In addition, TrackNet Model II performs
of Archana’s algorithm, TrackNet Model I, and even better than TrackNet Model I, proving that

9
 TABLE VI the same training logic as TrackNet Model II with
ACCURACY ANALYSIS OF BADMINTON TRACKING . the only difference in the variety of training set.
The prediction details are shown in Table IV. As
Model Precision Recall F1-measure expected, the performance of TrackNet Model II’
TrackNet-Tennis 75.8% 22.9% 35.2%
TrackNet-Badminton 85.0% 57.7% 68.7% is similar to TrackNet Model II on the same test
set as shown in Table V. TrackNet Model II’
achieves 99.7% precision, 97.3% recall, and 98.5%
F1-measure. Furthermore, 10-fold cross validation
training TrackNet with consecutive input frames can is adopted on TrackNet Model II’ for the pur-
further improve its dynamic object tracking ability, pose of safety and comprehensive analysis. At last,
especially for small objects. Moreover, TrackNet TrackNet Model II’ with 10-fold cross validation
Model II even correctly positions occluded balls reaches 95.3% precision, 75.7% recall, and 84.3%
occasionally. 2 out of 7 occluded balls are precisely F1-measure.
detected. This discovery directly exhibits that con- In addition to tennis, we also apply the proposed
secutive frames provide critical information for the TrackNet to the badminton dataset as introduced in
network to learn trajectory patterns of the interested Section III. The badminton dataset contains 18, 242
object. By extracting information from neighboring frames with the resolution of 1280 × 720. Similarly,
frames, TrackNet Model II not only enhances its all frames are resized from 1280 × 720 to 640 × 360
tracking precision on normal objects but also on to speed up the training process. The dataset is
blurry or occluded objects. randomly divided to the training set and test set.
The overall performance in terms of precision, 70% frames are the training set and 30% frames are
recall, and F1-measure are summarized in Table V. the test set. For badminton dataset, model training
These three metrics are defined by parameters, including learning rate, batch size, num-
# of True Positive ber of epochs, etc., are set to the same values used
Precision = , in the training of tennis dataset as shown in Table
# of True Positive + False Positive
# of True Positive III.
Recall = , and Before evaluating TrackNet on badminton, the
# of VC1+VC2+VC3
2(Precision × Recall) specification of a correct detection is defined by
F1-measure = . analyzing the dimension of badminton images in
Precision + Recall
the video. Unlike tennis, badminton is not spherical,
The Archana’s algorithm reaches 92.5% precision, resulting in a larger size variation. We define the
74.5% recall, and 82.5% F1-measure. With the diameter of a badminton image by taking an average
help of powerful deep learning network, TrackNet on its largest length and width. The image exists
Model I outperforms the Archana’s algorithm and in two extreme cases. One happens when the bad-
reaches 95.7% precision, 89.6% recall, and 92.5% minton moves toward the camera at the backcourt
F1-measure. By learning how to extract trajec- and the other happens when the badminton moves
tory information from neighboring frames, TrackNet laterally at the frontcourt. In our dataset, such cases
Model II further improves the performance and result in a large variation in images’ diameter rang-
achieves 99.8% precision, 96.6% recall, and 98.2% ing from 3 to 24 pixels. Since the mean diameter is
F1-measure. around 7.5 pixels, we define the PE specification as
To prevent the overfitting issue that frequently 7.5 pixels to indicate whether a badminton image is
happens in deep learning solutions, another 16, 118 accurately detected. Detections with PE larger than
frames are added to the training set. These 16, 118 7.5 pixels belong to incorrect predictions. Compared
frames are collected from an additional 9 videos with tennis that has PE specification of 5 pixels, the
recorded at different tennis courts, including grass PE specification of badminton seems to be released.
court, red clay court, hard court etc. The model The main reason is that images of badminton are
trained by the enriched training set is named as larger than tennis in the video since the badminton
TrackNet Model II’. TrackNet Model II’ follows court is smaller than the tennis court. Therefore, the

10
camera uses a smaller focal length to capture the in traveling speed than tennis. For example, in
entire court, resulting in larger images of ball and badminton, a drop stroke and a smash stroke have a
players. significant difference in velocity. Such extreme sce-
To evaluate the badminton tracking ability of narios commonly happen during a badminton com-
TrackNet, we adopt the transfer learning idea that petition, making the model hard to fit both scenarios
directly applies the well-trained TrackNet model by perfectly. Nonetheless, although the performance in
tennis dataset for badminton trajectories recogni- tracking badminton is not as phenomenal as tennis,
tion. Here, we name the transfer learning mode as achieving a precision of 85.0% is accurate enough to
TrackNet-Tennis which is trained by tennis dataset correctly depict all trajectories in the game. Future
using three consecutive input frames. As shown in research on TrackNet improvement in the aspects of
Table VI, for badminton tracking, TrackNet-Tennis identifying trajectories of extreme fast objects and
only achieves precision, recall, and F1-measure of learning distinct patterns caused by significant speed
75.8%, 22.9%, and 35.2%, respectively. Although variation will be conducted.
the precision seems acceptable, the recall is too
poor to be used. Such a low recall is due to VI. C ONCLUSION
a large number of false negatives, implying that
In this paper, we proposed TrackNet, a heatmap-
the badminton cannot be recognized in many cir-
based deep learning network comprising both con-
cumstances. The main reason causing such poor
volutional and deconvolutional neural network.
performance lies in the fundamental characteristics
TrackNet is able to precisely position coordinates
difference between tennis and badminton, including
of high-speed and tiny objects such as tennis and
velocity, trajectories, shape, etc. To verify the feasi-
badminton. With TrackNet, accurate predictions can
bility of TrackNet framework on badminton track-
be achieved on broadcast sports videos without
ing, we train another model named as TrackNet-
high frame rate and high resolution, significantly
Badminton which is trained by badminton dataset
reducing the cost from recording and processing
using three consecutive input frames. As shown in
high specification videos. To enhance TrackNet’s
Table VI, TrackNet-Badminton reaches precision,
capability of identifying trajectory patterns of fast-
recall, and F1-measure of 85.0%, 57.7%, and 68.7%,
moving objects, we designed a scalable input that
respectively. As expected, TrackNet-Badminton is
allows feeding TrackNet with multiple consecutive
able to learn the features of badminton, leading to
input frames. By evaluating both conventional im-
significant performance improvement.
age processing algorithm and the proposed Track-
Furthermore, when we compare tennis and bad-
Net on the real tennis video dataset, we demon-
minton tracking performance using TrackNet frame-
strated that TrackNet can achieve an explainable
work, it can be observed that tennis tracking out-
and exceptional prediction performance by adopting
performs badminton tracking by a noticeable mar-
consecutive input frames concept on the deep neural
gin. This is because badminton travels much faster
network. Moreover, for the even faster objects such
than tennis, resulting in much more unclear object
as badminton, TrackNet achieves a decent tracking
images in badminton videos. As elaborated in Sec-
capability according to our experimental results,
tion III, the fastest recorded badminton moves in
exhibiting promising extensibility to related appli-
417 kilometers per hour, while the fastest recorded
cations.
tennis moves in 253 kilometers per hour. Such an
enormous increase in velocity causes performance
ACKNOWLEDGEMENT
degradation especially in the aspect of the recall
due to high false negatives. High traveling speed This work of T.-U. İk was supported in part by the
makes the badminton move across long distance Ministry of Science and Technology, Taiwan under
within only a few frames. The property of dynamic grant MOST 107-2627-H-009-001 and MOST 105-
trajectories in such high speed becomes hard to 2221-E-009-102-MY3. This work was financially
recognize by the model. In addition to the absolute supported by the Center for Open Intelligent Con-
speed, badminton possesses a much higher variation nectivity from The Featured Areas Research Center

11
Program within the framework of the Higher Edu- [13] X. Yu, C.-H. Sim, J. R. Wang, and L. F. Cheong, “A trajectory-
cation Sprout Project by the Ministry of Education based ball detection and tracking algorithm in broadcast tennis
video,” in 2004 International Conference on Image Processing
(MOE), Taiwan. (ICIP 2004), vol. 2. Singapore: IEEE, 24-27 October 2004,
pp. 1049–1052.
R EFERENCES [14] V. Renò, N. Mosca, M. Nitti, C. Guaragnella, T. D’Orazio, and
[1] M. Archana and M. K. Geetha, “Object detection and tracking E. Stella, “Real-time tracking of a tennis ball by combining 3d
based on trajectory in broadcast tennis video,” Procedia Com- data and domain knowledge,” in Technology and Innovation in
puter Science, vol. 58, pp. 225–232, 2015. Sports, Health and Wellbeing (TISHW), International Confer-
[2] K. Simonyan and A. Zisserman, “Very deep convolutional ence on. IEEE, 2016, pp. 1–7.
[15] F. Yan, W. Christmas, and J. Kittler, “A tennis ball tracking
networks for large-scale image recognition,” arXiv preprint algorithm for automatic annotation of tennis match,” in Pro-
arXiv:1409.1556, 2014. ceedings of the British Machine Vision Conference (BMVC
[3] R. Girshick, J. Donahue, T. Darrell, and J. Malik, “Rich feature 2005), vol. 2. Durham, England: BMVA, 5-8 September 2005,
hierarchies for accurate object detection and semantic segmen- pp. 619–628.
tation,” in Proceedings of the IEEE Conference on Computer [16] X. Zhou, L. Xie, Q. Huang, S. J. Cox, and Y. Zhang, “Tennis
Vision and Pattern Recognition (CVPR 2014), 23-28 June 2014, ball tracking using a two-layered data association approach,”
pp. 580–587. IEEE Transactions on Multimedia, vol. 17, no. 2, pp. 145–156,
[4] R. Girshick, “Fast R-CNN,” in International Conference on 2015.
Computer Vision (ICCV 2015), 11-18 December 2015, pp. [17] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classi-
1440–1448. fication with deep convolutional neural networks,” in Advances
[5] S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: Towards in neural information processing systems, 2012, pp. 1097–1105.
real-time object detection with region proposal networks,” in [18] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional
Advances in neural information processing systems, 2015, pp. networks for biomedical image segmentation,” in International
91–99. Conference on Medical image computing and computer-assisted
[6] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You only intervention. Springer, 2015, pp. 234–241.
look once: Unified, real-time object detection,” in Proceedings [19] W. Jiang and Z. Yin, “Human activity recognition using
of the IEEE conference on computer vision and pattern recog- wearable sensors by deep convolutional neural networks,” in
nition, 2016, pp. 779–788. Proceedings of the 23rd ACM international conference on
[7] H. Noh, S. Hong, and B. Han, “Learning deconvolution net- Multimedia. ACM, 2015, pp. 1307–1310.
work for semantic segmentation,” in Proceedings of the IEEE [20] Y. Chen and Y. Xue, “A deep learning approach to human ac-
International Conference on Computer Vision, 2015, pp. 1520– tivity recognition based on single accelerometer,” in 2015 IEEE
1528. International Conference on Systems, Man, and Cybernetics
[8] H.-T. Chen, W.-J. Tsai, S.-Y. Lee, and J.-Y. Yu, “Ball tracking (SMC). IEEE, 2015, pp. 1488–1492.
and 3D trajectory approximation with applications to tactics [21] V. Badrinarayanan, A. Handa, and R. Cipolla, “Segnet: A deep
analysis from single-camera volleyball sequences,” Multimedia convolutional encoder-decoder architecture for robust semantic
Tools and Applications, vol. 60, no. 3, pp. 641–667, October pixel-wise labelling,” arXiv preprint arXiv:1505.07293, 2015.
2012. [22] J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional
[9] X. Wang, V. Ablavsky, H. B. Shitrit, and P. Fua, “Take your networks for semantic segmentation,” in Proceedings of the
eyes off the ball: Improving ball-tracking by focusing on team IEEE conference on computer vision and pattern recognition,
play,” Computer Vision and Image Understanding, vol. 119, pp. 2015, pp. 3431–3440.
102–115, February 2014. [23] V. Belagiannis and A. Zisserman, “Recurrent human pose
[10] T.-S. Fu, H.-T. Chen, C.-L. Chou, W.-J. Tsai, and S.-Y. Lee, estimation,” in 2017 12th IEEE International Conference on
“Screen-strategy analysis in broadcast basketball video using Automatic Face and Gesture Recognition (FG 2017). IEEE,
player tracking,” in Processing of the 2011 IEEE Visual Com- 2017, pp. 468–475.
munications and Image (VCIP), 6-9 November 2011. [24] T. Pfister, J. Charles, and A. Zisserman, “Flowing convnets for
[11] H. Myint, P. Wong, L. Dooley, and A. Hopgood, “Tracking human pose estimation in videos,” in Proceedings of the IEEE
a table tennis ball for umpiring purposes,” in Proceedings of International Conference on Computer Vision, 2015, pp. 1913–
the 14th IAPR International Conference on Machine Vision 1921.
Applications (MVA 2015), 18-22 May 2015, pp. 170–173. [25] “Hough gradient method,” [Link]
[12] “Hawk-eye,” [Link] [26] M. D. Zeiler, “ADADELTA: an adaptive learning rate method,”
arXiv preprint, vol. abs/1212.5701, 2012. [Online]. Available:
[Link]

12