Predicting NFL plays
Predicting Plays in the National Football League
Group Three:
Drake Hath, Ritesh Patil, Harshavardhan Reddy Suddamalla, Benjamin Ilacqua
Central Michigan University
STA591: Data Mining Techniques
RUNNING HEAD TITLE 2
(SHORTENED)
Introduction
The National Football League is engrained into the culture of America, with millions
tuning in every Sunday to passionately watch their team. The NFL is a multi-billion-dollar
industry, and with so much at stake the head office of each team needs to be on top of their
game. Part of being on top of their game is being prepared to face your opponent each week.
Predicting what play your opponent will run is a large part of game preparation for the head
office of an NFL team. Therefore, within this study multiple variables will be used to predict the
probability of a play being a pass.
Data Source
We found a dataset from Kaggle that had NFL play-by-play data from 2009-2018. This
data set has 255 variables and 447,382 observations. This data set contains variables that
describe every single play spanning the nine seasons of this data set. Some examples of variables
that are contained within this data set that are planned on being used are:
Down – Nominal variable that shows what down it is for the play.
Yardline_100 – This is an interval variable that gives us the number of yards
away the team is from the endzone.
Pass_attempt – This is our target variable. It is binary, 0=no pass 1=pass.
Score_differential – interval variable that shows the score differential for the
possession team
Game_half – Nominal variable that describes what half it is.
Half_seconds_remaining – Interval variable that tells how much time is left in the
half.
Ydstogo – Interval variable that displays the number of yards left for a first down
RUNNING HEAD TITLE 3
(SHORTENED)
Data Cleaning
Our first task was trimming down the enormous dataset. The first decision we made was
to narrow the data down to just data from 2018. We choose 2018 data because it was the most
recent data that was available. This narrowed our observations down to 42,037. Next, we wanted
to narrow down the number of variables. We rejected all redundant variables, such as time left in
minutes and time left in second that give virtually the same information. We also rejected
variables that had to do with what happened during the play such as yards gained since this
would be information that is not available when making predictions before the play. Next, we
needed to take out the missing values from our interval variables. The original dataset labeled all
missing variables as N/A which turned all our interval variables into nominal variables. We used
Python to remove all observations where a variable had an N/A value. We also decided to
remove all the observations where playtype equals kickoff because it is impossible to have a pass
play on a kickoff. This narrowed our observations down to 33,071.
The following plot shows some basic statistics of our variables.
RUNNING HEAD TITLE 4
(SHORTENED)
RUNNING HEAD TITLE 5
(SHORTENED)
A table of the summery statistics of all our class variables
Correlation of the variables to the target
A table of the summery statistics of all our interval variables
RUNNING HEAD TITLE 6
(SHORTENED)
After cleaning the data and exploring our variables our next step was to partition our data
into training, validation and test. To do this we used the Data Partition node in SAS enterprise
miner. We tried many different combinations of training, validation and testing data and we
found that a 60%, 20%, 20% partition gave us the best models based on ROC values.
Variable Selection
To select the variables that are significant we used a variable selection node, this node is
used to find the variables that make a difference in the model. The variable selection node
improves the efficiency and accuracy of a model. The way that we selected our variables was
using Chi-Squared selection, we tried different options such as ProbF but we found that Chi-
Squared resulted in the best ROC scores. The following picture shows what variables were
decided as significant and the worth of each variable.
Variable worth table
RUNNING HEAD TITLE 7
(SHORTENED)
Variable worth plot
Modeling techniques
Decision Tree model:
Within our project we had three different modeling techniques. The first modeling
technique we tried was using a decision tree to predict whether a play would be a pass or not.
Some advantages of using decision trees are that decision trees are easier to understand and
RUNNING HEAD TITLE 8
(SHORTENED)
explain when presenting. Along with the decision tree being easy to understand, it is easy to
prepare the data for the decision tree, decisions trees do not require normalized data or scaling of
data. A few examples of weaknesses with decision tree models are that a small change in the data
can make a huge difference in the decision tree, not only this decision tree algorithms are
typically very complex compared to other modeling. Within our project we ran multiple different
variations of decision trees, the best one we found used Gini as the splitting rule. Our best model
had an ROC value of 0.76. The ROC value is a great way to determine how accurate a model
you have, in this case the ROC value means our model is accurate 76% of the time.
The Cumulative lift plot suggests that our model is effective for predicting pass attempt.
The plot shows that as our model is implemented, we become more effective at predicting
whether a play is going to be as pass.
RUNNING HEAD TITLE 9
(SHORTENED)
This is the diagram of our best decision tree model.
Logistic Regression model:
The next modeling technique that we tried was multiple linear regression for the data. A
few advantages of using multiple linear regression are that it is easy to understand and interpret,
along with this typically regression creates very flexible models. A couple of disadvantages of
multiple linear regression is that this model does not adapt to outlying data well, this kind of
model is also prone to overfitting. We tried many different versions of multiple regression
models, we found that our best model used stepwise selection for the variables, meaning that the
variables are added to the model one at a time to determine the effect that the variable has on the
model. Out of the different options the best linear regression model that we found was when we
used the Schwarz’ Bayesian criterion. These options gave our model the ROC value of .74, thus
meaning that our model was accurate 74% of the time.
RUNNING HEAD TITLE 10
(SHORTENED)
The estimates for each of our variables show the weights that our variables hold. The Wald Chi-Squared
test shows that all of the variables above are significant because they all have a Pr > ChiSq less than
0.05.
RUNNING HEAD TITLE 11
(SHORTENED)
One of the biggest draws for using a logistic regression model is the odds ratio estimates. This
allows for useful interpretations of our variables. The odds ratio estimates for the yardline_100
variable shows that for every yard a team gets closer to a touchdown they are 0.6% less likely to
pass. The odds ratio for the ydstogo shows that for every yard a team is closer to a first down
they are 4.1% more likely to pass the ball. The estimate for score_differential reveals that for
every point that a team is behind they are 2.7% more likely to pass the ball. The only variable
that we could not interpret using the odds ratio was safety_prob. This is because the variable
safety_prob is an interval variable for 0 to 1, thus we can’t interpret a one-point increase in safety
probability.
RUNNING HEAD TITLE 12
(SHORTENED)
The cumulative lift plot for the logistic regression model shows that our model is effective for
predicting pass attempt. The plot shows that as our model is implemented, the more effective, we
become at predicting pass attempt.
Neural Network Model:
The last modeling technique that we explored was a neural networking model. Out of all
the models that we explored neural networking model gave us the best model based on ROC
with value of 0.77. Although neural networking models are good at following intricate patterns
in data, the problem with the modeling technique is that it is a “black box”. This means that we
can’t derive any meaning from the system that the model creates. Our neural network model is
also only slightly better than our decision tree model that we created. Because of the lack of
interpretability and the fact that it is only slightly better than our next best model we conclude
that the decision tree is a better model for the purpose of helping a defense predict the probability
of a pass attempt.
RUNNING HEAD TITLE 13
(SHORTENED)
The cumulative lift chart shows that the neural network model is very effective at
prediction pass attempt. The plot shows a steady increase in cumulative lift as we implement our
model.
Results/Discussion
Out of all the techniques and options that we used to attempt to predict a play in the
National Football League, the best model based on ROC value was the neural network model
with an accuracy of 77%. However, the best model that we found for the purpose of our study
was the decision tree model. We concluded that the decision tree model is our best model
because it is easier to read, and we can better interpret our variables. In addition, it only had one
less ROC point with a value of 0.76. Using SAS Enterprise Miner along with Python for data
trimming, we were able to predict if a play was going to be a pass with at a high rate of success.
This statement is reinforced by our ROC plots.
RUNNING HEAD TITLE 14
(SHORTENED)
The ROC plots for our three models show that all three models are effective at predicting
the probability a team will pass on a given play. The more accurate the model is the farther away
it is from the orange line. The ROC lines for all three models show a healthy distance from the
orange which suggests that they all work well for predicting.
Model Neural Decision Regression
Networ Tree- (Stepwise,
k (Gini) SBC)
Test Data ROC 0.77 0.76 0.74
Misclassification 0.31 0.31 0.33
Rate
Average Squared 0.19 0.20 0.21
Error
Validation ROC 0.75 0.74 0.72
Data
Misclassification 0.32 0.33 0.35
Rate
Average Squared 0.20 0.20 0.21
Error
Training ROC 0.75 0.75 0.72
Data
Misclassification 0.32 0.32 0.35
Rate
Average Squared 0.20 0.20 0.21
RUNNING HEAD TITLE 15
(SHORTENED)
Error
The table compares each of the three models we fit using three different comparison
statistics: ROC, Misclassification Rate, and Average Squared Error. They also compare all these
comparison statistics for all three partitions of the data. As stated, the Neural Network model had
the highest ROC value followed closely by the decision tree model and the logistic regression
model for the training, validation, and test data. The neural networking model and the decision
tree model have the same misclassification rate for both the test and training data. The neural
network model has an average squared error of 0.19 followed closely by the decision tree model
with 0.2 and then the logistic regression model with a value of 0.21. This table shows how close
these three models are in accuracy and highlights how well they predict pass attempt.
The final diagram from Enterprise Miner looks like:
This diagram shows the recap of the different techniques that were used to wrangle and
model this data. The one question that needs to be addressed is; how is the data useful? The
National Football League is imperative to be a step ahead of the opponent, winning a single play
could sway the whole game one way or another. This data is useful because it can be used to get
RUNNING HEAD TITLE 16
(SHORTENED)
you a step ahead, knowing situations where a team is likely to pass or run means you can
strategically plan your plays to catch the opponent off-guard. This information is useful to
coaches across the league to potentially win more games and play based on a more statistical
game plan.
In conclusion, our study shows that the factors that influence a team's probability of
passing on a given play are down, safety probability, win probability, score probability, score
differential, expect points, yards to go for a first down, the team with the ball, the amount of time
left in the game, and what quarter it is. We also revealed that the decision tree model is the best
model for a defense to use if they want to try and predict if a team is going to pass on a given
play. To continue our research in the future we would try to find ways for a team to quickly
obtain the data so that our model could be seamlessly implemented for a defense.