Data Mining Process

Business Understanding

“The first step is Business Understanding. Here, the focus is on defining the objectives of the
problem. We try to understand the business context, goals, and the questions we want data mining
to answer. This step ensures that our analysis aligns with business needs.”For example, are we trying
to predict customer churn, improve product recommendations, or detect fraudulent transactions?

In this phase, we work closely with domain experts and stakeholders to translate the
business objectives into a data mining problem. Without a strong understanding of the goals, the
entire project could go in the wrong direction.”

Data Understanding

“Next comes Data Understanding. At this stage, we collect data, explore its structure, and
examine its quality. We identify missing values, inconsistencies, and potential challenges. The aim is
to get a clear picture of the dataset before preparing it for modeling.” The goal here is to become
familiar with the data, so we know what we have to work with, and also what additional data might
be needed.”

Data Preparation

“Data Preparation is often the most time-consuming part. It involves cleaning, handling
missing values, transforming data types, dealing with outliers, feature selection, and sometimes
sampling. Well-prepared data is critical for building accurate models.”

Modeling

“Once the data is ready, we move to Modeling. Here, we split the dataset into training and
testing sets. The dataset is typically split into training and testing sets. The training data is used to
build the model, while the testing data helps us evaluate how well it performs on unseen data.
Different algorithms can be tried and compared, and often the process involves several iterations of
fine-tuning until we achieve the desired level of accuracy.”

Evaluation

“Evaluation ensures the model actually meets business objectives. “Once the model is built,
we need to evaluate it. This step goes beyond just checking accuracy or error rates. We ask ourselves:
Does the model actually solve the business problem we defined in the first step?

We look at different performance metrics, test the model on new data, and assess its
robustness. If the results are not satisfactory, we may need to go back to data preparation or try
different modeling approaches.

The evaluation step ensures that we are not just building a good model, but also one that is
meaningful and useful in the business context.”

Deployment
 “The final step is Deployment. “After the model has been evaluated and refined, the next
step is Deployment. This is where the insights or predictions from the model are put into practical
use.

Deployment can take many forms: it might be a report for management, a dashboard for
monitoring trends, or even a fully automated system that makes real-time predictions.
The key here is to integrate the model into the business process, so that it provides continuous value
and helps with decision-making.”

Prior Knowledge

“In any data mining project, the very first step is gathering prior knowledge. This means gaining a
clear understanding of three things:

• First, the objective of the problem — what exactly are we trying to solve or predict? For
example, in the context of this dataset, the problem could be predicting interest rates based
on a borrower’s credit score.

• Second, the subject area of the problem — we need to understand the domain or field
we’re working in. Here, it’s finance, and specifically, lending decisions. A good grasp of the
subject area helps us interpret patterns correctly.

• Third, the data itself — we start by looking at what kind of data we have, how it’s structured,
and whether it matches the problem we want to solve. For instance, in this table we see
Borrower IDs, their Credit Scores, and the corresponding Interest Rates.

This prior knowledge sets the foundation for the rest of the data mining process. Without it, we risk
building a technically correct model that doesn’t solve the real problem.”

Data Preparation

“After gaining prior knowledge, the next step is Data Preparation. This is one of the most important
and time-consuming stages of the data mining process, because the quality of the model depends
heavily on the quality of the data.

We begin with data exploration — looking at the data visually and statistically, just as you can see in
the scatter plot on this slide. Here, it shows how credit scores and interest rates are related.

Next, we check data quality. Are there errors, duplicates, or inconsistencies? If the data is inaccurate,
any model we build will also be unreliable.
We also need to deal with missing values. For example, if a borrower’s credit score is missing, we
have to decide whether to fill it using averages, estimate it with other attributes, or simply remove
that record.

Another important step is data type conversion — making sure numerical values, categorical values,
and dates are in the right format for analysis.

Then comes transformation, which could mean scaling numbers to a consistent range, normalizing
values, or even creating new variables that make patterns clearer.

We also check for outliers — extreme values that don’t fit the general pattern. For example, a
borrower with an unusually low interest rate compared to their credit score might be an outlier that
could affect the model.

Another step is feature selection. Not all available data is useful, so we focus only on the variables
that have the most predictive power.

And finally, sometimes we use sampling. If the dataset is very large, we might take a representative
sample to make the analysis more efficient without losing accuracy.

Overall, data preparation ensures that the dataset is clean, consistent, and ready for modeling. As
the saying goes in data science: ‘Garbage in, garbage out.’ So, this step is critical to success.”

Modeling

“Once the data is prepared, the next step is Modeling. This is where we apply different algorithms to
learn patterns from the data.

The process begins with training data. This is the portion of the dataset we use to build the model.
For example, if we’re trying to predict interest rates from credit scores, the training data helps the
model learn the relationship between those two variables.

After the model is built, we use test data to evaluate it. Test data is kept separate from the training
data so that we can see how well the model performs on new, unseen cases. This helps us check
whether the model generalizes well, rather than just memorizing the training data.

Then comes the evaluation step. Here, we measure how good the model is using performance
metrics — depending on the problem, this could be accuracy, error rate, precision, recall, or other
measures.

If the results are not good enough, we may go back and adjust the model. This could mean tuning
the parameters, trying a different algorithm, or even revisiting data preparation. This is an iterative
process until we find the best possible outcome.

Finally, once the model passes evaluation, we confirm it as the final model. This model is then ready
for deployment, where it can be used to make predictions or provide insights for real-world decision-
making.”

“In modeling, one of the most critical steps is splitting the data into training and test sets.
Here you can see two tables. The first one is the Training Data Set. It contains most of the borrower
records along with their credit scores and interest rates. This is the data we use to train our model —
essentially teaching the algorithm how credit scores influence interest rates.

The second table is the Test Data Set. It contains a smaller set of borrower records that the model
has not seen during training. After building the model on the training data, we apply it to this test
data to check how well it predicts the interest rates.

The purpose of this split is to avoid overfitting — which happens when a model memorizes the
training data but performs poorly on new, unseen data. By keeping a separate test set, we ensure
that the model can generalize and make accurate predictions in real-world situations.

In practice, a common approach is to split the dataset into about 70–80% for training and 20–30%
for testing, although the exact ratio can vary depending on the size of the dataset.

Visualizing Training and Test Data Split

Training Data
Test Data
The black dots represent the training data, which is the majority of the dataset. These points are
used by the model to learn the relationship between credit score and interest rate.

The white dots represent the test data. These are not shown to the model during training. Instead,
they are held back and later used to evaluate the model’s performance.

What we can see here is that both the training and test data points follow the same general pattern
— as the credit score increases, the interest rate tends to decrease. This makes sense in the financial
domain, because borrowers with higher credit scores are usually offered lower interest rates.

By separating the dataset in this way, we can check whether the model generalizes well. If the model
performs well on both the training data and these test data points, then we know it is reliable. But if
it only performs well on the training data and fails on the test data, it means the model has overfitted
and will not be useful in the real world.

This split between training and testing is one of the most fundamental practices in data mining and
machine learning.”

Regression Line and Equation

In this example, we are using a linear regression model to capture the relationship between credit
scores and interest rates.

The scatter plot shows the data points — each representing a borrower’s credit score and the
corresponding interest rate they received. The dashed line is the regression line, which represents
the model’s prediction.

The equation of the line is shown on the right:

Here:

• y represents the predicted interest rate,

• the constants (0.1 and 6/100,000) are the model parameters, which have been calculated
based on the training data.

This means that for every increase in credit score, the interest rate decreases slightly, which matches
the expected financial behavior. Borrowers with higher credit scores are more trustworthy, so they
get lower interest rates.

The important point here is that the model generalizes the relationship — it doesn’t just memorize
the data, but creates a mathematical function that we can apply to new, unseen borrowers to predict
their interest rates.

This is a simple linear regression, but in practice, more complex algorithms may be used if the
relationship is not linear.”

Evaluation of Test Dataset

“After building the model, the next step is to evaluate it using the test dataset — the data the model
has never seen before.

This table shows how the model performed on three borrowers:

• For Borrower 04, with a credit score of 700, the actual interest rate was 6.40%. The model
predicted 6.11%, which gives an error of -0.29%.

• For Borrower 07, with a credit score of 750, the actual interest rate was 5.90%. The model
predicted 5.81%, with a very small error of -0.09%.

• For Borrower 10, with a credit score of 825, the actual rate was 5.70%, and the model
predicted 5.37%, leading to an error of -0.33%.

The Model Error column shows how far off the predictions were from the actual values. Here, all the
errors are quite small — less than half a percent. That means our regression model is doing a good
job at approximating the relationship between credit score and interest rate.

This evaluation is important because it tells us whether the model generalizes well. If the errors had
been very large, that would mean the model either overfitted the training data or was not capturing
the true pattern.

In this case, since the errors are low, we can be confident that our model is reliable and ready to
move on to the next stage.”

Application
“After modeling and evaluation, the next step is Application. This stage is about making the model
useful in the real world.

Product readiness. This means preparing the model so it can be used outside of the lab
environment. For example, making sure it can handle real business data and that the results are
understandable to decision-makers.

Technical integration. The model has to be connected with existing systems — whether that’s a
banking platform, a CRM system, or a web application. Without integration, the model’s predictions
can’t flow into the tools that the business actually uses.

Model response time. Especially in real-time applications, like fraud detection or credit scoring, the
model needs to give answers quickly. A model that takes hours to compute a result may not be
practical.

Remodeling. Models don’t last forever — data changes over time, business conditions change, and
customer behavior evolves. This means the model has to be monitored and retrained periodically to
maintain accuracy.

Assimilation. This is the process of embedding the model’s insights into everyday decision-making.
For example, in a bank, loan officers might start relying on the model’s risk scores to set interest
rates. Over time, the model becomes a natural part of business operations.

Knowledge (Posterior Knowledge)

“The last stage of the data mining process is about generating knowledge — also called posterior
knowledge.

Up to this point, we’ve gone through prior knowledge, data preparation, modeling, evaluation, and
application. But the real value of data mining lies in what we learn after the model has been applied.

Posterior knowledge means the new insights that the model uncovers — insights we didn’t have at
the beginning. For example, from our case study, we now know that there is a clear negative
relationship between credit score and interest rate. That’s not just a number coming out of a model
— it’s actionable knowledge that can guide financial decisions.

This stage is also about using that knowledge to improve strategies and decision-making. For
instance:

• A bank can use this model to design better loan products.

• Risk managers can adjust policies for borrowers with different credit scores.

• Business leaders can spot opportunities, like targeting high-score customers with more
attractive offers.

Finally, posterior knowledge isn’t static. As more data comes in, the knowledge can evolve. This
means organizations should treat data mining as a continuous learning process, where knowledge is
regularly updated and fed back into business decisions.

So, while prior knowledge helps us start the journey, posterior knowledge is the end goal — the real-
world understanding that data mining gives us, which can transform decision-making.”