CHAPTER: DATA MINING

(Definition, Representation of Input Data, Data Mining Process, Analysis Methodologies,

Data Preparation: Data Validation, Data Transformation, Data Reduction)

1. Introduction to Data Mining

In the modern era of information technology, organizations collect vast amounts of data
from multiple sources such as business transactions, sensors, social media interactions,
medical devices, and e-commerce activities. While storing such data is easy, extracting
meaningful insights from it is challenging. This motivated the development of Data Mining, a
discipline that lies at the intersection of database systems, statistics, machine learning, and
artificial intelligence.

Data mining plays a crucial role in converting raw data into valuable knowledge that
supports strategic decision-making, forecasting, and predictive analytics.

2. Definition of Data Mining

Data mining is commonly defined as:

“The process of automatically discovering useful patterns, hidden knowledge, trends, and
relationships from large datasets using algorithmic and statistical techniques.”

Data mining is a key component of Knowledge Discovery in Databases (KDD). It involves the
application of computational techniques that aim at identifying previously unknown and
meaningful information.

2.1 Key Characteristics of Data Mining

1. Automatic or semi-automatic process – uses algorithms to detect patterns.

2. Operates on large volumes of data – typically warehouses or big data systems.

3. Extracts previously unknown information – focuses on discovering new insights.

4. Generates actionable knowledge – the results support business strategies.

5. Involves predictive and descriptive tasks – classification, clustering, association, etc.

2.2 Examples of Data Mining

 Predicting customer churn in telecom industry.

 Fraud detection in credit card transactions.

 Identifying purchase patterns in retail stores.

  Diagnosing diseases based on medical history.

 Recommending movies or products on streaming platforms.

3. Representation of Input Data for Data Mining

The structure and representation of data play a critical role in determining the type of data
mining techniques that can be applied. Different formats of data require different
preprocessing and algorithms.

Data can exist in multiple forms such as relational tables, text documents, images, web
pages, multimedia, time-series, and graphs. Below are the major types of input data
representations:

3.1 Relational (Record-Based) Data

The most common representation is tabular form, consisting of rows (records) and columns
(attributes).
Each record is a tuple representing an entity in the domain.

Structure:

CustomerID Name Age Gender Salary City

Applications

 Classification

 Regression

 Clustering

Advantages

 Easy to store, query, and manipulate

 Supported by SQL and database systems

3.2 Transactional Data

Used when each record is a collection of items (cart items, medicines purchased, etc.)

Structure (Market Basket Example):

T1 → {Milk, Bread, Butter}

T2 → {Eggs, Milk}
Applications

 Association rule mining

 Market basket analysis

3.3 Time-Series and Sequential Data

Data collected over time with ordered timestamps.

Examples

 Stock prices

 Weather data

 Sensor readings

 Web-click stream

Applications

 Trend analysis

 Forecasting

 Sequence pattern mining

3.4 Spatial and Geographical Data

Represents physical locations like maps, GPS points, satellite imagery.

Applications

 GIS systems

 Traffic analysis

 Urban planning

3.5 Text and Unstructured Data

Includes documents, emails, logs, social media posts.

Applications

 Text mining

 Sentiment analysis
  Topic modeling

3.6 Graph-Based Data

Nodes represent entities; edges represent relationships.

Examples

 Social networks

 Web hyperlink structures

 Biological networks

Applications

 Link prediction

 Community detection

 Network analysis

3.7 Multimedia Data

Includes images, audio, video.

Applications

 Face recognition

 Image classification

 Speech processing

4. Data Mining Process (KDD Process)

The process of extracting knowledge is not a single step but a series of systematic stages
known as KDD (Knowledge Discovery in Databases).

The KDD process consists of the following major phases:

4.1 Data Selection

Relevant data is extracted from different sources such as operational databases, ERP
systems, sensors, logs, etc.

Main Tasks
  Identify required datasets

 Retrieve data from multiple sources

 Integrate and consolidate them into one dataset

4.2 Data Cleaning (Preprocessing)

Data in real-world applications is often:

 Noisy

 Incomplete

 Inconsistent

 Containing outliers

Data cleaning handles such issues.

Common Cleaning Techniques

 Filling missing values

 Removing duplicates

 Smoothing noisy data

 Correcting inconsistencies

4.3 Data Integration

Combining multiple datasets into a unified view.

Example

Joining customer data, sales data, and product data from different systems.

4.4 Data Transformation

Transforming data into forms suitable for mining.

Techniques:

 Normalization

 Aggregation

 Encoding
  Scaling

 Feature extraction

4.5 Data Mining (Core Phase)

Application of computational algorithms to extract patterns.

Techniques used:

 Classification

 Clustering

 Regression

 Association rule mining

 Anomaly detection

 Sequential pattern mining

4.6 Pattern Evaluation

Only interesting and useful patterns are selected based on measurable criteria:

 Interestingness measures

 Accuracy

 Support and confidence

 Lift

 Statistical significance

4.7 Knowledge Presentation

Results must be presented in a clear and understandable form through:

 Tables

 Graphs

 Dashboards

 Reports

 Visualization tools
5. Analysis Methodologies in Data Mining

Analysis methodologies define how the mined data is interpreted and what type of
knowledge is extracted. There are two major categories of data mining tasks:

5.1 Predictive Analysis

Uses known labels to predict unknown outcomes.

5.1.1 Classification

Classifies data into predefined categories.

Examples:

 Email: spam or not spam

 Cancer: benign or malignant

 Loan: approved or rejected

Algorithms:

 Decision Trees

 Naive Bayes

 SVM

 Neural Networks

5.2 Regression

Predicts continuous numerical values.

Examples:

 Predicting house prices

 Forecasting sales

 Estimating rainfall

Techniques:

 Linear regression

 Polynomial regression
5.3 Descriptive Analysis

Describes hidden structures in data.

5.3.1 Clustering

Groups similar objects without predefined labels.

Examples:

 Customer segmentation

 Grouping similar documents

 Image grouping

Algorithms:

 K-Means

 Hierarchical clustering

 DBSCAN

5.3.2 Association Rule Mining

Discovers relationships between items in large databases.

Example: Market Basket Analysis:

“If customer buys Bread → they often buy Butter.”

Measures:

 Support

 Confidence

 Lift

5.3.3 Anomaly Detection

Identifies unusual data objects.

Examples:

 Fraud detection

 Network intrusion detection

  Medical abnormalities

5.3.4 Sequential Pattern Mining

Discovers frequent patterns in sequential data such as:

 DNA sequences

 Click-streams

 User navigation patterns

6. Data Preparation

Before applying data mining algorithms, data must be prepared to ensure it is accurate,
consistent, and suitable for analysis. Data preparation involves three major activities: data
validation, data transformation, and data reduction.

6.1 Data Validation

Data validation ensures that the input data is correct, complete, consistent, and reliable.

Objectives

 To detect errors

 To maintain data quality

 To ensure reliable mining results

Types of Validation

1. Range Check:
Ensures values fall within permissible range (Age 18–60).

2. Format Check:
Ensures data follows a specific format (email, date).

3. Consistency Check:
Checks logical correctness (DOB matches age).

4. Uniqueness Check:
Ensures no duplicate primary keys.

5. Completeness Check:
Ensures no missing values in critical fields.
6.2 Data Transformation

Data transformation modifies data into formats more suitable for mining.

Major Techniques

(1) Normalization

Scaling numerical data to a standard range.

Types:

 Min-max normalization

 Z-score normalization

Used in: clustering, neural networks.

(2) Encoding

Converting categorical data into numerical form.

 Label encoding

 One-hot encoding

Used in: decision trees, ML models.

(3) Aggregation

Summarizing data.

Example:
Daily sales → Monthly sales.

(4) Feature Extraction

Deriving new attributes from existing ones.

Example:
From timestamps → derive year, month.

(5) Smoothing
Removing noise using:

 Binning

 Averaging

6.3 Data Reduction

Data reduction reduces the volume of data while retaining essential information.

Techniques

(1) Dimensionality Reduction

Reduces number of attributes.

Methods:

 Principal Component Analysis (PCA)

 Feature selection

(2) Numerosity Reduction

Represents data using smaller models.

Methods:

 Regression models

 Histograms

 Clustering algorithms

(3) Data Compression

Uses encoding methods (Huffman encoding, ZIP).

(4) Sampling

Selecting a representative subset of data.

Types:

 Random sampling
 Systematic sampling

 Stratified sampling