Chapter 3&4 Notes
Foundations of Data Mining and Analytics
Data are raw facts, numbers, text, or signals collected and stored to gain insights.
Data Mining is the specific process of discovering meaningful patterns and trends
within large, complex datasets to support decision-making and predictive modeling.
Core Data Mining Techniques
1. Classification: Assigning items to predefined categories (e.g., email spam
detection or credit risk scoring) using algorithms like Decision Trees or Naïve Bayes.
2. Clustering: Grouping similar data points without pre-labeled classes, often used
for customer segmentation.
3. Association Rule Mining: Discovering relationships and co-occurrences, such as
"market basket analysis" (identifying which products are frequently bought
together).
4. Regression Analysis: Predicting continuous numeric outcomes, such as
property prices or product demand.
5. Anomaly Detection: Identifying unusual or unexpected data points to detect
fraud or network intrusions.
Classification of Data Types
• By Structure:
◦ Structured: Stored in fixed fields, rows, and columns (relational databases).
◦ Semi-structured: Uses tags or markers for organization (e.g., JSON, XML).
◦ Unstructured: Lacks a predefined format, such as text, images, or videos.
• By Source:
◦ Primary: Collected directly for a study (surveys, interviews).
◦ Secondary: Previously recorded for other purposes (government data, reports).
◦ Internal vs. External: Internal data is owned by the organization (sales records);
external data is public information gained from third parties (social media).
Data Quality and Preprocessing
Real-world data is often of poor quality, meaning it may be incomplete, noisy, or
inconsistent. Quality decisions require quality data; therefore, preprocessing
(which consumes ~60% of data mining effort) is essential.
The 6 Measures of Data Quality
1. Accuracy: Free from errors.
2. Completeness: No missing values.
3. Consistency: No discrepancies in naming or codes.
4. Timeliness: Data is appropriate for current needs.
5. Believability: Acceptability of the source.
6. Interpretability: Easy for users to understand.
Data Cleaning Tasks
• Handling Missing Data: Methods include ignoring the tuple, manual filling, using a
global constant, or replacing values with the Mean (numeric) or Mode (categorical).
Advanced methods like the Expectation-Maximization (EM) Algorithm iteratively
predict missing values until they converge.
• Handling Noisy Data: "Noise" refers to random errors or outliers (e.g., a negative
salary). Techniques to "smooth" data include Binning, Regression, and Clustering
to detect and remove outliers.
Data Integration and Transformation
Data Integration combines data from multiple sources into a coherent store,
though it often faces challenges like schema integration (identifying that "cust-id"
and "cust-#" are the same entity) and format conflicts.
Data Transformation Techniques
1. Normalization & Standardization: Rescaling numerical data to a common range.
◦ Min-Max Normalization: Rescales data to a range like.
◦ Z-score Standardization: Centers data around a mean of 0 with a standard
deviation of 1.
2. Discretization: Converting continuous data into discrete "bins" or intervals (e.g.,
turning specific ages into ranges like 20–30). This can be done via Equal-width or
Equal-frequency binning.
3. Generalization: Replacing low-level specific data with higher-level concepts
using a Concept Hierarchy (e.g., Kebele → City → Region).
4. Encoding: Converting categorical text labels into numbers. Label Encoding is
used for ordered data, while One-hot Encoding creates binary columns for each
category.
Data Reduction and Feature Selection
Data Reduction aims for "less data, same information" to improve computational
efficiency and model performance.
Reduction Strategies
• Dimensionality Reduction: Reducing the number of attributes. Techniques
include Principal Component Analysis (PCA), which captures maximum variance,
and Linear Discriminant Analysis (LDA), which maximizes class separability.
• Numerosity Reduction: Reducing data volume. Parametric methods store only
model parameters (like regression slopes); Non-parametric methods use sampling,
histograms, or clustering.
• Data Compression: Exploiting redundancy through Lossless (ZIP, RLE) or Lossy
(JPEG) schemes.
Feature Selection Methods
This process chooses the optimal subset of original features to avoid redundancy.
• Filter Methods: Fast preprocessing steps using statistical tests like Information
Gain, Chi-square, or Correlation Coefficients.
• Wrapper Methods: Iterative "greedy" algorithms like Forward Selection (adding
features) or Backward Elimination (removing features).
• Embedded Methods: Feature selection is built directly into the learning algorithm
(e.g., Lasso Regularization or Decision Trees).
Privacy, Security, and Ethics
1. Data Privacy: Concerns the proper handling of sensitive data to meet regulatory
requirements (e.g., GDPR) and managing consent.
2. Data Security: Protecting digital information from unauthorized access or theft
throughout its lifecycle.
3. Data Ethics: Evaluating data practices that could adversely impact society.
Ethical behavior is vital for building consumer trust; 57% of consumers will stop
doing business with a company that uses data irresponsibly.
Analogy for Understanding
Think of the Data Analytics process as Cooking a Five-Course Meal. Data
Collection is like gathering ingredients from different farms (Primary sources) or
grocery stores (Secondary sources). Data Cleaning is washing the vegetables and
throwing away the rotten ones (Outliers/Noise). Data Transformation is chopping
ingredients into uniform sizes so they cook evenly (Normalization/Standardization).
Data Reduction is boiling down a large pot of stock into a concentrated sauce that
holds the same flavor but takes up less space (PCA/Compression). Finally, Data
Ethics is the "Chef’s Code," ensuring the food is prepared safely and honestly to
maintain the diners' trust.