Running head: DEEPLABCUT IN SPORTS 1

Markerless Tracking in Sports using DeepLabCut: A Methodological Tutorial

Tércio Apolinário-Souza1 and Thiago José Leonardi2

1
Universidade Federal do Rio Grande do Sul, Porto Alegre, Brazil
2
Universidade Estadual de Campinas, Campinas, Brazil

Author Note

Correspondence concerning this article should be addressed to Tércio

Apolinário-Souza, address: R. Felizardo, 750 - Jardim Botânico, Porto Alegre - RS,
Brasil; CEP: 90690-200; E-mail: [Link]@[Link]; ORCID:
[Link]/0000-0002-2136-0238; ResearcherID: O-7470-2016
DEEPLABCUT IN SPORTS 2

Abstract

One of the major challenges in Motor Behavior research is translating findings from
controlled environments to complex, real-world contexts such as team sports. Recent
advances in machine learning, particularly deep learning, have enhanced the ability to
track and analyse movement in dynamic settings. DeepLabCut, an open-source toolbox
using deep neural networks, enables precise tracking of players and objects, offering
valuable insights for performance analysis. This tutorial aims to provide practical
guidance for implementing DeepLabCut and to evaluate its effectiveness in tracking
both athletes and the ball during gameplay, thereby supporting its application in sports
science research and professional practice.
Keywords: Behavior Analysis; Computer Vision; Machine Learning; Motion
Tracking; Pose Estimation.
 DEEPLABCUT IN SPORTS 3

Markerless Tracking in Sports using DeepLabCut: A Methodological Tutorial

1 Introduction

2 A key challenge in the field of Motor Behavior is bridging the gap between
3 insights obtained from controlled, simplified tasks and their application in complex,
4 real-world contexts typical of professional practice (Wulf and Shea, 2002). This
5 challenge is partly due to the lack of tools capable of accurately assessing Motor
6 Behavior constructs in such dynamic and complex environments.

7 Team sports often reveal this challenge more prominently. Team sports constitute
8 a complex and dynamic system shaped by the interactions of multiple elements (Davids,
9 Araújo, & Shuttleworth, 2005). Athletes’ behavior depends on continuous adaptations
10 to factors such as spatiotemporal demands (Sheehan et al., 2022). Performance analysis
11 research commonly focuses on individual and collective tactical behaviors—often relying
12 on positional data—to inform training and competition strategies (Low et al., 2020).
13 Over the decades, sports analysis techniques have seen remarkable advancements
14 (Camomilla, 2018), particularly with the advent of artificial intelligence (AI). Notably,
15 developments in AI have enabled accurate human motion tracking in dynamic and
16 natural environments (Akan & Varlı, 2023; Lauer et al., 2022; Sehara et al., 2021).

17 DeepLabCut, an open-source toolbox based on deep learning, exemplifies these

18 advancements in AI (Mathias & Mathis, 2019). DeepLabCut is a technology capable of
19 tracking behavior across different contexts, offering a practical, low-cost, and
20 simple-to-implement solution for the study of human movement (Mathis et al., 2018).
21 In sports science, DeepLabCut may be used to track athletes’ positions and ball
22 dynamics during gameplay. This application of DeepLabCut enables detailed
23 performance analysis and tactical evaluations, providing insights into player movements,
24 strategies, and interactions within a game context. Integrating such advanced tracking
25 technology into sports analysis represents a significant leap forward, offering researchers
26 and coaches a deeper understanding of athletic performance and strategy.

27 However, a significant barrier to leveraging these methods is the complexity

28 involved in setup, training, and evaluation, which often lacks user-friendly guidance.
 DEEPLABCUT IN SPORTS 4

29 Consequently, an accessible, tutorial-style article walks readers through practical

30 implementation steps and demonstrates how these tools can be deployed across diverse
31 applications, such as sports science. This tutorial aims to outline practical
32 implementation procedures and to assess the effectiveness of DeepLabCut in tracking
33 both players and the ball.

34 Online material

35 All scripts, videos, and other materials used in this paper can be found at the
36 following link: [Link]

37 See "Tutorial video 1" for this step in the online material. You can perform the
38 same procedure using our original video ("Original video") and Script 1, both available
39 in the online material.

40 For definitions of the concepts of AI, please see Cossich, Carlgren, Holash, and
41 Katz (2023).

42 Method

43 Step 1 - recording video

44 The first step is to record the video. We recommended using a drone for this
45 purpose, as it allows the camera to remain centred, thereby facilitating image
46 processing in the subsequent steps. However, if recording with a drone is not possible,
47 position the camera to ensure the image is as centred as possible.

48 Two pairs (four men aged 32± 3.0 years) played in a small-sided game to maintain
49 possession of the ball for as long as possible over 3 minutes. Ball handling should be
50 performed with the hands, and the participant was not allowed to move while in
51 possession of the ball (similar to basketball)If the ball crossed the established
52 boundaries, possession was awarded to the opposing team of the last player to touch the
53 ball. For more details about the game, please refer to the "Original video" in the online
54 materials. This experiment occurred on an open court within a delimited 6 x 6 meters.
55 The data were recorded at 30 Hz with a resolution of 640×480 pixels using a drone (DJI
 DEEPLABCUT IN SPORTS 5

56 Mini SE) positioned at a height of 10 meters. The moments in the video where the ball
57 left the designated area (’ball out of play’) were removed, leaving 107 seconds remaining
58 (online material: "Original video").

59 Step 2 - extract the frames

60 The next step is to extract the frames and label the frames of interest. These
61 frames represent the key points that the user intends to track. DeepLabCut provides a
62 function that allows all extracted frames to be labelled using an interactive graphical
63 user interface (GUI). In our example,using the GUI, we localized the center of nine key
64 points (the ball, the four players, and the four corners of the area) across 20 frames
65 sampled using the k-means clustering approach within DeepLabCut. The 20 frames
66 selected by the k-means clustering approach are the default setting in DeepLabCut;
67 however, the number of frames and k-means clustering can be adjusted as needed.

68 Step 3 - Create training dataset

69 We recommend using Google Colab with GPU support to create, train, and
70 evaluate the model. Personal computers often lack the necessary hardware
71 configurations (e.g., dedicated graphics cards) to perform such procedures efficiently. In
72 our example, we run the model using the Google Colab GPU. To create the model, we
73 employed the ResNet-50 architecture, a deep residual network known for its high
74 performance in image recognition tasks. Additionally, we utilized the imgaug library for
75 data augmentation, which enhances the variability of the training data by applying
76 random transformations such as rotations, flips, changes in brightness, and contrast
77 (online material: "Script 2"). However, the parameters used to create the dataset can be
78 modified.

79 Step 4 - train network

80 To train the model, use the function deeplabcut.train_network. This function

81 initiates the training process for the neural network dedicated to pose estimation. To
82 enhance the model’s robustness, employ the first shuffle of the dataset, providing a
 DEEPLABCUT IN SPORTS 6

83 varied training set. To monitor the progress, configure the training to display the status
84 every 100 iterations, offering frequent feedback on the model’s performance.
85 Additionally, set the function to save the model checkpoint every 500 iterations,
86 ensuring that training progress is preserved and allowing for resumption from the latest
87 checkpoint in case of interruptions. Finally, define the training process to run for a
88 maximum of 10,000 iterations, preventing overfitting and ensuring efficient use of
89 computational resources. Stopped the training when the loss function value reached
90 below 0.005 with a learning rate of 0.005.

91 You can perform the same procedure using Script 2 available in the online
92 material.

93 Step 5 - Evaluation of the Trained Network

94 The fifth step is to evaluate the performance of the trained model. We assess the
95 model’s performance by calculating the mean average Euclidean error between the
96 manual labels and the labels predicted by DeepLabCut. The acceptable error threshold
97 for each model depends on the level of precision required for the analysis. In our
98 example, the results showed that the mean training error was 2.62 pixels, while the
99 mean test error was 3.79 pixels. This error is relatively low for our application.

100 Step 6 - Perspective transformation

101 Centring the camera poses a challenge during recording; therefore, an important
102 step is the perspective transformation. This transformation helps minimise image
103 distortion effects. Drone footage results in a video with fluctuations in the camera’s
104 position due to the wind’s effect on the drone. In this example, we used known fixed
105 points (the four corners) to perform the perspective transformation, as described below.

106 We apply a perspective transformation to each frame, using the four known fixed
107 points in the image that delineate the playing area.

108 [Insert Figure 1 here]

 DEEPLABCUT IN SPORTS 7

109 Our implementation computed the homography matrix and applied the
110 perspective transformation (online material: "Script 3"). First, we extracted each
111 frame’s four corner points:

Bottom − lef t = (xA , yA )

Bottom − right = (xB , yB )

Top − right = (xC , yC )

Top − lef t = (xD , yD )

112 Then, we calculated the distances between these points to find the mean width
113 and height:

q
widthAD = (xA − xD )2 + (yA − yD )2
q
widthBC = (xB − xC )2 + (yB − yC )2
widthAD + widthBC
meanWidth =
2
114

q
heightAD = (xA − xB )2 + (yA − yB )2
q
heightBC = (xC − xD )2 + (yC − yD )2
heightAB + heightCD
meanHeight =
2

115 Based on the known real-world distance, we derive the pixels-per-meter ratio:

meanWidth
pixelsPerMeterHorizontal =
realDistance
meanHeight
pixelsPerMeterVertical =
realDistance
pixelsPerMeterHorizontal + pixelsPerMeterVertical
pixelsPerMeter =
2

116 Next, we define the desired coordinates in the new image using the pixels per
117 meter ratio. Assume d is the actual distance between the cones:
 DEEPLABCUT IN SPORTS 8

 
 0 0 
 
 
d · pixels_per_meter 0 
 
pts_dest = 



d · pixels_per_meter d · pixels_per_meter
 
 
 
0 d · pixels_per_meter
118 Here, pts_dest represents the coordinates in the destination plane, a transformed
119 version of the original coordinates. Each point is represented in the format [x,y]:

120 1. [0, 0]: bottom-left corner;

121 2. [d · pixels_per_meter, 0]: bottom-right corner;

122 3. [d · pixels_per_meter, d · pixels_per_meter]: top-right corner;

123 4. [0, d · pixels_per_meter]: top-left corner.

124 we calculated the homography matrix H using [Link] fuction of

125 OpenCV:
H = [Link](pts_orig, pts_dest)

126 For each point p = [x, y] of the elements, it is transformed using the homography
127 matrix H:

p′ = H · p

128 where p=[x,y,1] is the transformed coordinate. After, we performed by

129 multiplying the point vector by the homography matrix H:

    
′
x  h11 h12 h13  x
    
    
y ′  = h h22 h23 
 y 
 
   21
    
    
1 h31 h32 h33 1
130 Finally, we normalize the resulting vector by dividing it by the homogeneous
131 coordinates:
h11 x + h12 y + h13
x′ =
h31 x + h32 y + h33
132
h21 x + h22 y + h23
y′ =
h31 x + h32 y + h33
 DEEPLABCUT IN SPORTS 9

133 [Insert Figure 2 here]

134 However, the method for performing the perspective transformation should be
135 adjusted according to the type of recording. We always recommended using known
136 points from the recording to carry out the perspective transformation.

137 Step 7 - The filter with likelihood

138 The data exported from the analysis includes a column indicating the probability
139 that each signal is correct. It is advisable to disregard low-probability values. In the
140 example below, we used probability values greater than 0.8. The following outlines the
141 step-by-step procedure for our example.
142 After performing the perspective transformation of the data, we filtered the x and
143 y position values based on the probability. Based on their likelihood values, the
144 equation replaces values with "NaN" (Not a Number) in the x and y position rows.
145 Given a matrix E ∈ R3×n where E represents the positions (E1 = position x, E2 =
146 position y) and likelihoods (E3 ) of elements, as follows:


1 if E3,j < 0.80


Mj = for j = 1, 2, . . . , n


0

otherwise


NaN if Mj = 1


E1,j =  for j = 1, 2, . . . , n

E1,j

otherwise


NaN if Mj = 1


E2,j =  for j = 1, 2, . . . , n

E2,j

otherwise

147 Step 8 - The filter with threshold limits

148 Even after the final filtering, some data may not reflect reality. For instance, they
149 may indicate positional values that do not exist in the real world. In our example, the
150 playing area was limited to 6 by 6 meters, with an escape zone of approximately 2
151 meters. Therefore, any positional data above 8 meters or below -2 meters can be
152 excluded from the analysis. The following illustrates our example.
 DEEPLABCUT IN SPORTS 10

153 We established two threshold limits: an upper limit of 8 and a lower limit of -2.
154 The filtering process then replaced any values in the dataset that exceeded the upper
155 limit or fell below the lower limit with NaN. We subsequently added the filtered data to
156 the d matrix. This approach excludes data points outside the defined range, improving
157 the dataset’s accuracy and relevance for further analysis. The filtered data matrix
158 dfiltered is obtained by:
 

NaN












 if x > limup or y > limup or x < limdown or y < limdown
  
 NaN


dfiltered =  




 x
otherwise

 

  

  
 y



159 [Insert Figure 3 here]

160 The data from the player wearing the black shirt and the ball showed a loss of
161 around 50% of the data. This result was likely influenced by recording outdoors. The
162 black color of the shirt did not stand out against the shadows projected by the players,
163 making it difficult for the algorithm to recognize it. As for the ball, the sunlight
164 reflection created a color distinct from its red hue. However, upon analyzing the
165 complete video data (online material: "Video with DeepLabCut"), it is evident that the
166 losses of the ball and the player in the black shirt were distributed over time, having
167 little impact on the tracking. This indicates that, although the data loss was relatively
168 high, the phenomena that can be investigated in this video (e.g., ball passes) occur at a
169 relatively high frequency (around 3Hz).

170 [Insert Figure 4 here]

171 Final considerations

172 This tutorial aimed to outline practical implementation procedures and to assess
173 the effectiveness of DeepLabCut in tracking both players and the ball. We hope it
174 provides accessible guidance for researchers and practitioners unfamiliar with this tool.
175 Although our application focused on ball sports, DeepLabCut can be used to track any
 DEEPLABCUT IN SPORTS 11

176 element in an image, such as body segments or implements, particularly in the field of
177 Motor Behaviour. Finally, this represents the first attempt to present a tutorial
178 specifically tailored to DeepLabCut. We apologise if any aspect was insufficiently
179 covered, as some steps may require basic programming knowledge. We hope this
180 tutorial proves useful for those planning to adopt DeepLabCut in future investigations.

181 Funding Statements

182 This work was supported by the Brazilian National Council for Scientific and
183 Technological Development (CNPq, process 405316/2021-6).
 DEEPLABCUT IN SPORTS 12

184 References

185 Akan, S., & Varlı, S. (2023). Use of deep learning in soccer videos analysis: survey.
186 Multimedia Systems, 29 (3), 897–915.
187 Blostein, S. D., & Richardson, H. S. (1994). A sequential detection approach to target
188 tracking. IEEE Transactions on Aerospace and Electronic Systems, 30 (1),
189 197–212.
190 Camomilla, V. (2018). Advancements in sports analysis techniques. Journal of Sports
191 Science, 36 (4), 415-428.
192 Cioppa, A., Deliège, A., Magera, F., Giancola, S., Barnich, O., Ghanem, B., &
193 Van Droogenbroeck, M. (2021). Camera calibration and player localization in
194 soccernet-v2 and investigation of their representations for action spotting. In 2021
195 ieee/cvf conference on computer vision and pattern recognition workshops (cvprw)
196 (p. 4532-4541). doi: 10.1109/CVPRW53098.2021.00511
197 Cossich, V. R. A., Carlgren, D., Holash, R. J., & Katz, L. (2023). Technological
198 breakthroughs in sport: Current practice and future potential of artificial
199 intelligence, virtual reality, augmented reality, and modern data visualization in
200 performance analysis. Applied Sciences, 13 (23), 12965.
201 Cuevas, C., Quilón, D., & García, N. (2020). Techniques and applications for soccer
202 video analysis: A survey. Multimedia Tools and Applications, 79 (39),
203 29685–29721.
204 Davids, K., Araújo, D., & Shuttleworth, R. (2005). Applications of dynamical systems
205 theory to football. Science and football V , 537 , 550.
206 Fernandes, J. C., & Kenwright, B. (2022). Identifying and extracting football features
207 from real-world media sources using only synthetic training data. arXiv preprint
208 arXiv:2209.13254 . Retrieved from [Link]
209 Figueroa, P. J., Leite, N. J., & Barros, R. M. (2006). Tracking soccer players aiming
210 their kinematical motion analysis. Computer Vision and Image Understanding,
211 101 (2), 122–135.
212 Jiang, T., Billingham, J., Müksch, S., Zarate, J., Evans, N., Oswald, M. R., . . . Song, J.
 DEEPLABCUT IN SPORTS 13

213 (2025). Worldpose: A world cup dataset for global 3d human pose estimation. In
214 European conference on computer vision (pp. 343–362).
215 Kamble, P. R., Keskar, A. G., & Bhurchandi, K. M. (2019). A deep learning ball
216 tracking system in soccer videos. Opto-Electronics Review, 27 (1), 58–69.
217 Lauer, T., et al. (2022). Dynamic and natural environment motion tracking. Computer
218 Vision and Image Understanding, 214 , 103-115.
219 Low, B., Coutinho, D., Gonçalves, B., Rein, R., Memmert, D., & Sampaio, J. (2020). A
220 systematic review of collective tactical behaviours in football using positional
221 data. Sports Medicine, 50 , 343–385.
222 Mathias, A., & Mathis, M. (2019). Deeplabcut: A toolbox for analyzing animal
223 behavior. Nature Methods, 16 (6), 601-604.
224 Mathis, A., et al. (2018). Deeplabcut: Markerless pose estimation of user-defined body
225 parts. Nature Neuroscience, 21 (9), 1265-1272.
226 Monezi, L. A., Calderani Junior, A., Mercadante, L. A., Duarte, L. T., & Misuta, M. S.
227 (2020). A video-based framework for automatic 3d localization of multiple
228 basketball players: a combinatorial optimization approach. Frontiers in
229 bioengineering and biotechnology, 8 , 286.
230 Praça, G. M., Abreu, C. d. O., Rochael, M., & Moreira, P. D. (2022). How reliable are
231 the tactical measures obtained in soccer small-sided games? a test-retest analysis
232 of observational instruments and gps-based variables. Proceedings of the
233 Institution of Mechanical Engineers, Part P: Journal of Sports Engineering and
234 Technology, 17543371221113925.
235 Rico-González, M., Pino-Ortega, J., Nakamura, F. Y., Arruda Moura, F.,
236 Rojas-Valverde, D., & Los Arcos, A. (2020). Past, present, and future of the
237 technological tracking methods to assess tactical variables in team sports: A
238 systematic review. Proceedings of the Institution of Mechanical Engineers, Part P:
239 Journal of Sports Engineering and Technology, 234 (4), 281–290.
240 Santhosh, P., & Kaarthick, B. (2019). An automated player detection and tracking in
241 basketball game. Computers, Materials & Continua, 58 (3).
 DEEPLABCUT IN SPORTS 14

242 Sehara, S., et al. (2021). Deep learning for human motion tracking. IEEE Transactions
243 on Pattern Analysis and Machine Intelligence, 43 (11), 4562-4575.
244 Sheehan, W. B., Tribolet, R., Watsford, M. L., Novak, A. R., Rennie, M., & Fransen, J.
245 (2022). Tactical analysis of individual and team behaviour in professional
246 australian football: Original investigation. Science and Medicine in Football, 6 (2),
247 172–180.
248 Zheng, B. (2022). [retracted] soccer player video target tracking based on deep learning.
249 Mobile Information Systems, 2022 (1), 8090871.
DEEPLABCUT IN SPORTS 15

Figure 1 . The known points in the image for perspective transformation are A, B, C,
and D, with a distance of 6 meters between A and B, D and C. m = meters.
DEEPLABCUT IN SPORTS 16

Figure 2 . Graphical representation of the points in the image and the image at the 4th
second. A) Graphical representation of the points in the image without the
transformation. B) Graphical representation of the points in the image with the
transformation. C. Image with the indication of points of interest. D. Original image.
m = meters. px = pixels
DEEPLABCUT IN SPORTS 17

Figure 3 . Graphical representation of points of interest over time (A) and their x and y
positions (B). m = meters.

Figure 4 . Percentage of frames without information. A) Comparison among the 9

elements. B) The nine elements over time. sec = seconds