Statistics
Chapter 06 – Course Project
Comma Separated Values (CSV) File
HR1 HR2
A CSV file is a text file that has a specific format 73.38 62.11
71.97 63.74
which allows data to be saved in a table structured 67.81 60.63
format. 63.27 58.86
64.85 62.98
For example, HR1 denotes the true heart rate 68.8 63.27
(HR) and HR2 denotes the predicted HR. 66.48 67.99
73.58 69.09
In this file, HR values are tab-separated. 75.87 68.13
76.66 69.13
2
Reading CSV File in Python
HR1 HR2
You can read .CSV file using Pandas Library: 73.38 62.11
71.97 63.74
import pandas as pd 67.81 60.63
63.27 58.86
df = pd.read_csv(filename, sep="\t", skiprows=1,
64.85 62.98
header=None).values 68.8 63.27
66.48 67.99
HR1 = df[:, 0] → this how to access the 1st column
73.58 69.09
HR2 = df[:, 1] → this how to access the 2nd column 75.87 68.13
76.66 69.13
3
Error Measures: Mean Absolute Error
HR1 HR2
σ𝑛𝑖=1 𝑦𝑖 − 𝑥𝑖 73.38 62.11
MAE = 71.97 63.74
𝑛 67.81 60.63
○ MAE → mean absolute error 63.27 58.86
64.85 62.98
○ 𝑦𝑖 → prediction 68.8 63.27
○ 𝑥𝑖 → true value 66.48 67.99
73.58 69.09
○ 𝑛 → total number of data points 75.87 68.13
76.66 69.13
4
Error Measures: MAE
HR1 HR2 Diff Abs Diff
73.38 62.11 11.27 11.27
σ𝑛𝑖=1 𝑦𝑖 − 𝑥𝑖 71.97 63.74 8.23 8.23
MAE = 67.81 60.63 7.18 7.18
𝑛 63.27 58.86 4.41 4.41
64.85 62.98 1.87 1.87
Sum(Abs Diff) 59.76 68.8 63.27 5.53 5.53
MAE = =
𝑛 10 66.48 67.99 -1.51 1.51
73.58 69.09 4.49 4.49
= 5.98 beats/minute 75.87 68.13 7.74 7.74
76.66 69.13 7.53 7.53
5
Error Measures: Root Mean Square Error
HR1 HR2
73.38 62.11
σ𝑛𝑖=1 𝑥𝑖 − 𝑦𝑖 2
71.97 63.74
RMSE =
𝑛 67.81 60.63
63.27 58.86
64.85 62.98
○ RMSE → root mean square error 68.8 63.27
○ 𝑥𝑖 → true value 66.48 67.99
73.58 69.09
○ 𝑦𝑖 → prediction 75.87 68.13
76.66 69.13
○ 𝑛 → total number of data points
6
Error Measures: RMSE
HR1 HR2 Diff Square Diff
73.38 62.11 11.27 127.01
σ𝑛𝑖=1 𝑥𝑖 − 𝑦𝑖 2 71.97 63.74 8.23 67.73
RMSE = 67.81 60.63 7.18 51.55
𝑛 63.27 58.86 4.41 19.45
64.85 62.98 1.87 3.5
68.8 63.27 5.53 30.58
66.48 67.99 -1.51 2.28
438.87 73.58 69.09 4.49 20.16
RMSE = = 6.62 beats/minute
10 75.87 68.13 7.74 59.91
76.66 69.13 7.53 56.7
7
Similarity Measures: Correlation Coefficient
σ 𝑥𝑖 − 𝑥ҧ 𝑦𝑖 − 𝑦ത HR1 HR2
𝑟= 73.38 62.11
σ 𝑥𝑖 − 𝑥ҧ 2 σ 𝑦𝑖 − 𝑦ത 2
71.97 63.74
67.81 60.63
○ 𝑟 → correlation coefficient 63.27 58.86
○ 𝑥𝑖 → values of the x-variable in a sample (HR1) 64.85 62.98
68.8 63.27
○ 𝑥ҧ → mean of the values of the x-variable 66.48 67.99
73.58 69.09
○ 𝑦𝑖 → values of the y-variable in a sample (HR2)
75.87 68.13
○ 𝑦ത → mean of the values of the y-variable 76.66 69.13
8
Similarity Measures: Correlation Coefficient
Sum Square Sum Square Sum [Diff(HR1) *
Mean HR1 Mean HR2
Diff(HR1) Diff(HR2) Diff(HR2)]
70.27 64.59 196.66 124.66 102.2
σ 𝑥𝑖 − 𝑥ҧ 𝑦𝑖 − 𝑦ത HR1 HR2 Diff(HR1) Diff(HR2) Square Diff(HR1) Square Diff(HR2) Diff(HR1) * Diff(HR2)
𝑟=
σ 𝑥𝑖 − 𝑥ҧ 2 σ 𝑦𝑖 − 𝑦ത 2 73.38 62.11 3.11 -2.48 9.67 6.15 -7.7128
71.97 63.74 1.7 -0.85 2.89 0.72 -1.445
102.2 67.81 60.63 -2.46 -3.96 6.05 15.68 9.7416
𝑟=
196.66 ∗ 124.66 63.27 58.86 -7 -5.73 49 32.83 40.11
64.85 62.98 -5.42 -1.61 29.38 2.59 8.7262
102.2 68.8 63.27 -1.47 -1.32 2.16 1.74 1.9404
𝑟=
156.57 66.48 67.99 -3.79 3.4 14.36 11.56 -12.886
𝑟 = 0.65 73.58 69.09 3.31 4.5 10.96 20.25 14.895
75.87 68.13 5.6 3.54 31.36 12.53 19.824
76.66 69.13 6.39 4.54 40.83 20.61 29.0106
9
Bland-Altman Plot
○ It is a graphical method to
compare two measurements. The
differences between the two
techniques are plotted against the
averages of the two techniques.
○ Horizontal lines are drawn at the
mean difference, and at the limits of
agreement, which are defined as the
mean difference plus and minus 1.96
times the standard deviation of the
differences.
10
Bland-Altman Plot Meani = (HR1i + HR2i)/2
Diffi = HR1i − HR2i
HR1 HR2 Mean Diff
73.38 62.11 67.75 11.27
71.97 63.74 67.86 8.23
67.81 60.63 64.22 7.18
63.27 58.86 61.07 4.41
64.85 62.98 63.92 1.87
68.8 63.27 66.04 5.53
66.48 67.99 67.24 -1.51
73.58 69.09 71.34 4.49
75.87 68.13 72 7.74
76.66 69.13 72.9 7.53
11
Bland-Altman Plot Meani = (HR1i + HR2i)/2
Diffi = HR1i − HR2i
○ Upper Limit = Mean[Diff] + 1.96 * SD[Diff] HR1 HR2 Mean Diff
○ Lower Limit = Mean[Diff] - 1.96 * SD[Diff] 73.38 62.11 67.75 11.27
71.97 63.74 67.86 8.23
67.81 60.63 64.22 7.18
Mean Diff SD Diff
63.27 58.86 61.07 4.41
5.67 3.6 64.85 62.98 1.87
63.92
68.8 63.27 66.04 5.53
66.48 67.99 67.24 -1.51
Upper Limit Lower Limit
73.58 69.09 71.34 4.49
12.73 -1.39
75.87 68.13 72 7.74
76.66 69.13 72.9 7.53
12
Bland-Altman Plot Meani = (HR1i + HR2i)/2
Diffi = HR1i − HR2i
Mean Upper Limit Lower Limit HR1 HR2 Mean Diff
5.67 12.73 -1.39 73.38 62.11 67.75 11.27
5.67 12.73 -1.39 71.97 63.74 67.86 8.23
5.67 12.73 -1.39 67.81 60.63 64.22 7.18
5.67 12.73 -1.39 63.27 58.86 61.07 4.41
5.67 12.73 -1.39 64.85 62.98 63.92 1.87
5.67 12.73 -1.39 68.8 63.27 66.04 5.53
5.67 12.73 -1.39 66.48 67.99 67.24 -1.51
5.67 12.73 -1.39 73.58 69.09 71.34 4.49
5.67 12.73 -1.39 75.87 68.13 72 7.74
5.67 12.73 -1.39 76.66 69.13 72.9 7.53
13
Bland-Altman Plot
Mean Upper Limit Lower Limit
5.67 12.73 -1.39
5.67 12.73 -1.39
5.67 12.73 -1.39
5.67 12.73 -1.39
5.67 12.73 -1.39
5.67 12.73 -1.39
5.67 12.73 -1.39
5.67 12.73 -1.39
5.67 12.73 -1.39
5.67 12.73 -1.39
14