Better Working World

Data Challenge Team Train-Test Split

 Executive Summary

THE CHALLENGE
The challenge was to predict the occurrence of a single species of frog
for a single location using a single data source at a coarse spatial resolution.

KEY PROBLEMS
Sampling Bias in training data, Feature and Model Selection, Preventing Overfitting

OUR APPROACH
We used climatic variables and random forests to predict the occurrence
of Litoria Fallax at a given coordinate

THE PERFORMANCE
.82
Test F1 Score
.78
Test Precision Score
.86
Test Recall Score
.77
Unseen data F1 Score

Overview Data Cleaning Sampling Bias Feature Engineering Model Selection Training Process An Alternative 1
 Approach At A Glance
3.
A brief overview of our solution
Understanding how the
variables will contribute to the
model and selection

1.
X=(
model_data

4.
.drop([‘get f ID',
"event Date",
'decimal Latitude',
'decimal Longitude',
oversample =
SMOTE()

2. v
Data cleaning to prepare it
to be utilized for
Choosing an appropriate
prediction
Exploratory data analysis on model for our purposes
provided data set

7. 6. 5.
Model performance evaluation and
Validating model to ensure it
hyperparameter tuning
hasn’t overfit to the training data
Submission generation

Overview Data Cleaning Sampling Bias Feature Engineering Model Selection Training Process An Alternative 2
 Steps To : Data Cleaning
”Garbage in, garbage out,” how do we mitigate that?

Predictor data - “Terra Climate Variables”, Target dataset - Frog occurrences data set
Species Selection
1. The frog occurrence data set was cleaned to include only Litoria Fallax and Crinia Signifera as they share a habitat. Also, the
other species in the original dataset were too spread out over the continent and thus, they would contribute minimally to the
model.

Geographical Restriction
2. Cleaned the data based on geography to restrict the area to the Eastern Coastline of Australia which is where Litoria Fallax is
most commonly found. In the process, several Crinia Signifera occurrence points were deleted and this also slightly reduced the
sampling bias present.

Chronological Specification
3. Only data points from 2005 onwards were kept in the dataset to take care of multiple rows with null values. Rows with null
values post 2005 were also dropped but this was done post joining the occurrence dataset with the predictive variables.

Correlation Measurements
4.
For feature engineering, correlations were measured to gain additional insight into the predictive variables. Pre training, a 𝝌𝟐
analysis on the features versus the labels and post training, permutation importance was used to identify high performing
features and to discount low performing ones to improve efficiency.

Overview Data Cleaning Sampling Bias Feature Engineering Model Selection Training Process An Alternative 3
 Solving Sampling Bias
Attempt at solving the sampling bias using Synthetic Minority Oversampling

The Problem Our Solution

70000
60000
50000
40000
30000
20000
10000
0
Crinea Signifera Litoria Fallax

There existed a clear sampling bias present between the 2 The imbalanced classes were solved using Synthetic
frog species. Minority Oversampling Technique (SMOTE).

Number of observations of Crinia Signifera were In this technique, a random example from the
more than Litoria Fallax. This would lead to the minority class is chosen, then a randomly selected
model being good at predicting the absence of neighbour is chosen and a synthetic example is
Litoria Fallax but not the presence as we assume created at a randomly selected point between the
the presence of Crinia Signifera is the absence of points in feature space.
Litoria Fallax.
Overview Data Cleaning Sampling Bias Feature Engineering Model Selection Training Process An Alternative 4
 Feature Engineering (1/2)
Rationale behind selection of the crucial features behind our model

Our Findings

Feature importance using permutation on full model

Highest Performer
Using permutation importance on the trained
0.025 model, the highest performing feature is
“vap_mean”.
0.020

0.015
Low Performers
The lowest performing features can be dropped
0.010
to increase predictive capacity at scale. We
0.005
haven’t done this in order to extract all possible
predictive powers from the dataset.
0.000
pet_mean

tmax_mean

vpd_mean
aet_mean

tmin_mean

ws_mean
soil_mean
q_mean

vap_mean
def_mean

ppt_mean

srad_mean

Vap_mean
Gas
The feature “vap_mean” is the mean vapour
pressure over the month

Overview Data Cleaning Sampling Bias Feature Engineering Model Selection Training Process An Alternative 5
 Feature Engineering (2/2)
Rationale behind selection of the crucial features behind our model

Insights
Our Findings
𝜒 ! Analysis of Features vs Label Feature Correlation
Correlation between the various features and
1000 also the occurrence status were checked to
reveal relations that could potentially contribute
800 highly to the model’s performance
600

Highest performer
400
A 𝝌𝟐 analysis on the feature vs the labels
200
revealed “soil_mean” to be a highly performing
feature.
0
pet_mean

tmax_mean

vpd_mean
aet_mean

tmin_mean

ws_mean
q_mean

vap_mean
def_mean

ppt_mean

soil_mean

srad_mean

Soil_mean
The feature “soil_mean” is the mean of the total
soil moisture present

Overview Data Cleaning Sampling Bias Feature Engineering Model Selection Training Process An Alternative 6
 Model Selection (1/2)
The decisions that led to the choosing of random forest as our model

Assumptions Selection Approach

The main assumption of our First Attempt – Logistic Regression
approach is that the presence The shortcoming of this model was that it assumes
of Crinia Signifera is the linearity between the dependent variable and the
absence of Litoria Fallax. independent variables thus it tends to perform poorly in
situations where there exists a complex relation between
the features and labels
Implications
Thus, we had to pick a model that Second Attempt – (K-NN) Model
would work well in binary Given the classification task at hand, the next method we
classification tasks (high prediction tried was k-nearest neighbours (k-NN). It requires no
accuracy), works well with large training period and thus is a fast approach but given the
amounts of data, is computationally large amount of noisy data at hand, more efficient and
efficient and robust against robust approaches would have better results.
overfitting.

Overview Data Cleaning Sampling Bias Feature Engineering Model Selection Training Process An Alternative 7
 Model Selection (2/2)
The decisions that led to the choosing of random forest as our model

Selection Approach Random Forests

Third Attempt – Decision Trees Random forests are an ensemble of
At first, with their extreme efficiency at finding non- multiple decision trees which utilise
linearly related relationships and non-parametric nature, features to split the dataset until a
decision trees seemed like an optimal choice. However, classification point is reached.
their unstable nature is something which had to be
mitigated. We did so by utilizing random forests. Multiple decision trees together
protect the individual decision trees
Fourth Attempt – Random Forests from errors they might make as
In attempts to utilise ensemble techniques further, models individual decision trees are sensitive
such as XGBoost and ADABoost were also applied but the to the training data as they form a
trade off between training time and marginal increases in fixed structure of nodes and leaves.
accuracy (only after a lot of hyperparameter tuning) made Multiple decision trees with randomly
random forests the most optimal choice. selected data minimises this, hence
preventing overfitting and increasing
accuracy.

Overview Data Cleaning Sampling Bias Feature Engineering Model Selection Training Process An Alternative 8
 Training Process
Elaborating on how we trained our model and how well it performed

Fitting The Model And Validation

The testing features and labels were used to Performance Metrics
train the random forest classifier and The similarity between these two sets of
predictions were made using the test metric implies our model hasn’t overfit
features and compared with the test labels
TESTING SCORE
to measure its performance.
Train – Test Splitting
The dataset was divided into features and
labels (X, y) and further split 70% and
.81 .78 .86
F1 Score Precision Score Recall Score
30% into training features, training labels MEAN CROSS-VALIDATION SCORE

.82 .78 .86

and testing features and testing labels so Cross Validation
that the model can be trained and then Next, to check against overfitting, a 10-
its performance validated. fold cross validation was carried out and F1 Score Precision Score Recall Score
the performance metrics were averaged
over each of the 10 folds to get mean
performance metrics.

Overview Data Cleaning Sampling Bias Feature Engineering Model Selection Training Process An Alternative 9
 An Alternate Feature
Though not incorporated in our model, we recommend incorporating this approach on field

Experimenting With Time

Month of prediction as a feature How To Go About It? Sine and Cosine Encoding of Months
We suggest adding the month in which It is crucial to preserve the cyclicity of
the prediction is to be made as a this feature which is the property that
component while training the model December is closer to January than 1.00

and making predictions September. Thus, we recommend 0.75

encoding the month aspect through a 0.50

The way Litoria Fallax exists
cyclic function such as sine or cosine.

Encoded value
geographically has a very strong 0.25

correlation with the time of the year and 0.00

Why we have not incorporated this
this feature will add an incredible - 0.25
predictive capacity to the model The challenge requires us to predict - 0.50
whether a frog occurs at a certain - 0.75
latitude or longitude. The absence of - 1.00 sine
cosine

.82 .91
date in any form makes this approach 0 2 4 6 8 10
unsuitable for the purposes of the Month
challenge
Jump In F1 score when tested

Overview Data Cleaning Sampling Bias Feature Engineering Model Selection Training Process An Alternative 10
Appendix
Correlation amongst the features

Checking the correlations between variables not only

helps enhance understanding of variables, but highly
correlated variables can be dropped to increase
computational efficiency.

11