DATA MINING & WAREHOUSING
ehensive Exam Study Notes Lectures 1 – 4 | Final Exam Pr
What's inside: Lecture 1 – Introduction to Data Mining | Lecture 2 – Data Mining Tasks & Techniques + KDD
| Lecture 3 – Data Mining Architecture & Warehousing vs Mining | Lecture 4 – Data Marts & Partitioning
Strategy
■ LECTURE 1: Introduction to Data Mining
What is Data Mining?
Definition: Data Mining is the process of extracting useful information and knowledge from large sets of data.
It involves more than just extraction — it includes Data Cleaning, Data Integration, Data Transformation,
Pattern Evaluation, and Data Presentation.
Brief History
In the 1960s, statisticians used the terms "Data Fishing" or "Data Dredging" to describe data analysis practices.
The modern term "Data Mining" emerged around 1990 within the database community.
Key Applications
Application How Data Mining Helps
Market Analysis Identify customer buying patterns and target marketing campaigns.
Fraud Detection Flag unusual behaviour — e.g., credit-card transactions from new locations.
Customer Retention Predict which customers are likely to leave and act proactively.
Production Control Detect faulty equipment and find optimal manufacturing parameters.
Science Exploration Glean insights from massive biological datasets; aid drug discovery.
Advantages by Industry
1. Marketing / Retail
Data mining builds predictive models from historical data to identify which customers will respond to new
campaigns. Through market basket analysis, stores can arrange products strategically and offer targeted
discounts to attract more customers.
2. Finance / Banking
Banks use data mining to classify loans as good or bad based on historical customer profiles, and to detect
fraudulent credit card transactions in real time.
3. Manufacturing
Manufacturers apply data mining to operational data to detect faulty equipment and determine optimal control
parameters — for example, finding the ideal conditions for defect-free semiconductor wafer production.
4. Government
Government agencies analyse financial transaction records to detect money laundering and other criminal
activities.
Challenges of Data Mining
Challenge Explanation
Privacy Issues Personal data collected for one purpose may be sold or leaked when businesses change
hands. Growing public concern, especially with social networks and e-commerce.
Challenge Explanation
Security Issues Hackers target organisations holding social security numbers, payroll data, etc. Identity
theft and credit card fraud are serious consequences.
Misuse / Inaccuracy Data mined for ethical purposes can be exploited. Inaccurate mining results used for
decision-making can cause serious harm.
Deriving Business Value
The real value of data mining lies in uncovering hidden patterns and relationships in data that can be used
to make accurate predictions — enabling better marketing, fraud prevention, product safety, and customer
behaviour forecasting.
■ LECTURE 2: Data Mining Tasks, Techniques & KDD
Key Components of Data Mining
Component Description
Preprocessing Build a target data set from a data mart or warehouse before applying algorithms.
Data Cleansing Remove noise, handle missing values, and filter outliers.
Association Rule Learning Discover relationships between variables (e.g., products bought together).
Clustering Group similar data points without predefined labels.
Classification Apply known structures to classify new data (e.g., spam vs. legitimate email).
Regression Predict numeric values such as sales, prices, or temperatures.
Summarization Produce compact representations through visualisation and reporting.
Data Mining Tasks: Two Categories
Predictive Tasks — Perform inference to predict future data: Classification, Prediction, Time-Series Analysis.
Descriptive Tasks — Characterise general properties of the data: Association, Clustering, Summarization.
Task (Type) Description & Example
Classification (Predictive) Derives a model to assign a class label to records based on their attributes.
Example: direct marketing — predict {purchase / don't purchase}.
Prediction (Predictive) Predicts missing or future values using a model. Example: predict employee
income based on education, experience, and demographics.
Time-Series Analysis Analyses sequences of events over time to extract trends and patterns. Key
(Predictive) application: stock market prediction.
Association (Descriptive) Discovers relationships between sets of items. Example: beer and nappies are
often bought together — put nappies on sale to boost beer sales.
Clustering (Descriptive) Groups similar data objects together. Example: an insurance company clusters
customers by age, income, and residence for personalised services.
Summarization (Descriptive) Generalises data into a smaller, aggregated set. Example: summarise a
customer's shopping into total products, total spending, offers used.
Data Mining Techniques
Technique Description
Classification Analysis Classifies data into predefined classes using algorithms. Example: email spam
filters that classify messages as spam or legitimate.
Technique Description
Association Rule Learning Identifies hidden relationships (dependency modelling) between variables. Used
in retail basket analysis, catalog design, and machine learning programs.
Anomaly / Outlier Detection Identifies data points that deviate significantly from expected patterns. Used in
fraud detection, intrusion detection, and health monitoring.
Clustering Analysis Groups objects so similarity within a cluster is highest and between clusters is
lowest. Used for customer profiling.
Regression Analysis Identifies relationships between dependent and independent variables to predict
future values. Used for sales forecasting and financial modelling.
Attributes of Mined Data
Mined data must be: 1. New (Novel) — previously unknown findings; 2. Correct — accurate and reliable; 3.
Potentially Useful — actionable for business decisions.
Knowledge Discovery in Databases (KDD)
KDD is the broad process of finding knowledge in data. Data mining is its core step. The terms are often
used interchangeably, but KDD is the overall pipeline.
KDD Step Description
1. Data Selection Retrieve relevant data from databases or data warehouses.
2. Pre-processing Transform/consolidate data via aggregation and summarisation.
3. Data Mining Apply algorithms (clustering, classification, regression) to find patterns.
4. Pattern Evaluation Evaluate discovered patterns for accuracy and interestingness.
5. Knowledge Presentation Display identified patterns visually (e.g., graphs, charts).
■ LECTURE 3: Data Mining Architecture & Warehousing
vs Mining
Data Mining Architecture — Key Components
Component Role
Data Sources Databases, data warehouses, WWW, text files, spreadsheets. Large volumes
of historical data are required.
Data Cleaning, Integration & Raw data from heterogeneous sources must be cleaned, integrated, and
Selection filtered before being passed to the server. Data may be incomplete or
unreliable.
Database / Data Warehouse Holds the processed data and retrieves relevant records based on mining
Server requests.
Data Mining Engine The core component. Contains modules for association, classification,
clustering, prediction, time-series analysis, and characterisation.
Pattern Evaluation Module Measures interestingness of patterns using a threshold. Interacts with the
engine to steer search toward interesting patterns.
Graphical User Interface (GUI) Allows users to specify queries and view results without needing to
understand the underlying complexity.
Knowledge Base Guides the search and evaluates result patterns. May contain user beliefs and
experience data. Updated regularly by the pattern evaluation module.
Data Warehousing vs. Data Mining
Dimension Data Warehousing Data Mining
Definition Process of extracting, compiling, and Process of discovering patterns and
storing data from multiple sources into a useful knowledge from the stored data.
central repository.
Purpose Store and organise large volumes of data. Extract meaningful insights and predictions.
Who does it? Carried out by engineers. Carried out by business users with engineer
support.
Sequence Must occur before data mining. Occurs after data warehousing.
Nature Architecture / repository. Process / analytical activity.
Analysis type Basic statistical analysis. Complex pattern recognition, profiling, fraud
detection.
Data stored Months or years of historical data. Works on warehouse data; focuses on
patterns.
Example A company stores all project/employee data Mine that data to generate profit reports,
in one database. predict employee turnover.
Why Use Data Mining? — Top Reasons
• Establish relevance and relationships among data to generate profitable insights.
• Enable businesses to make informed decisions quickly.
• Identify unusual shopping patterns in grocery stores.
• Optimise websites by providing customised offers to each visitor.
• Measure customer response rates in business marketing.
• Create and maintain new customer groups for marketing purposes.
• Predict customer defections — identify customers likely to switch suppliers.
• Differentiate between profitable and unprofitable customers.
• Identify all kinds of suspicious behaviour as part of fraud detection.
■ LECTURE 4: Data Marts & Partitioning Strategy
What is a Data Mart?
A Data Mart is a repository of operational and aggregated data that supports business decision-making. It
contains a predefined subset of data organised for rapid analysis and reporting, focused on meeting the
needs of a specific group of users.
Reasons to Create a Data Mart from a Data Warehouse
• Create a passive copy of fact tables that don't change often — deleted after analysis.
• Decrease query load on the data warehouse by pointing reports to a separate server refreshed daily.
• Simulate critical business scenarios by changing mart data without affecting the warehouse.
Data Warehouse vs. Data Mart
Dimension Data Warehouse Data Mart
Scope Multiple subject areas Single subject area only
Data detail Detailed data Summarised data
Data model Does not always use dimensional model Built on dimensional model (star schema)
Integration Integrates many different data sources Focuses on one subject area
Types of Data Marts
Independent (Stand-alone) Data Mart
Focuses on a particular subject area and is not designed within an enterprise context. BI/analytic tools query
it directly. Can be built quickly.
Dependent Data Mart
Data comes directly from a central data warehouse (Bill Inmon's definition). Data is aggregated, restructured,
and summarised on its way in. Building one or two dependent data marts is the best solution for improving
warehouse performance, as processing happens outside the warehouse. Each department owns and controls
its own mart — improving security.
Modeling a Data Mart
Step Details
Logical Design Design facts and dimensions systematically. Define granularity first (the lowest level of
data stored). Consider historical duration to inform storage design.
Schema Design Use the star schema — simplest design. Define relationships between facts and
dimensions based on established granularity. Star schema allows intuitive grouping and
easier retrieval.
Physical Design Convert logical design to tables and constraints. Key decisions: data partitioning and
data indexing for performance.
Step Details
Size Estimation Estimate rows in the fact table → estimate row size → multiply for fact table size. Data
mart size ≈ 3–5× the fact table size.
Advantages & Disadvantages of Data Marts
Advantages:
✔ Simpler and cheaper to build than a full corporate data warehouse.
✔ Breaks down complex data design into manageable pieces — easier to maintain.
✔ Reporting queries become simpler due to subject-specific filtering.
✔ Can run on smaller, distributed machines — no need for heavy hardware.
✔ Provides applicable data as demanded by end-user requirements.
✔ Easier for customers to understand since data is domain-specific.
Disadvantages:
✘ Often short-term, temporary solutions not part of a corporate architecture — can raise extraction costs.
✘ Continuous implementation creates tracking problems; daily refreshes may require hardware upgrades.
✘ Multiple extraction, administration, and operational activities increase hardware and software expenses.
Warehouse Partitioning Strategy
Partitioning splits a large fact table into multiple separate smaller partitions to enhance performance, simplify
management, and facilitate backup/recovery.
Why Partition?
Easy Management Fact tables can grow to hundreds of gigabytes — too large to manage as one
entity.
Backup / Recovery Load only the required data partitions rather than the entire table. Mark old
partitions read-only for fast backups.
Performance Query scans only relevant partitions — not the entire dataset. Significantly
speeds up query response time.
Horizontal Partitioning Strategies
Strategy Description Best Use Case
By Time — Equal Partition by a fixed time period (e.g., monthly). Each Monthly sales data; user
Segments period represents a meaningful retention window. queries for 'month-to-date' data.
Partitioned tables can be reused by removing old
data.
By Time — Small partitions for recent (frequently accessed) data; Mix of recent history queries
Different-sized larger partitions for older, inactive data. Keeps and full-history data mining.
Segments physical table count small and reduces operating
cost. Not suitable if partitioning profile changes
often.
Strategy Description Best Use Case
By Another Dimension Partition by product group, region, supplier, etc. Regional departments query
Speeds up queries restricted to one dimension. Risk: only their own region's data.
if the dimension changes, the entire table must be
repartitioned. Recommended only when you are
certain the dimension will not change.
By Table Size Used when no clear dimension basis exists. A new No natural partitioning key
partition is created when the table exceeds a exists.
predetermined size. Requires metadata to track what
data is in each partition. Complex to manage.
Other Partitioning Methods
Round Robin Partitions
When a new partition is needed, the old one is archived. Uses metadata to direct user access tools to the
correct partition. Easy to automate table management.
Vertical Partitioning
Splits data vertically (by columns) rather than rows. Two approaches:
• Normalization — standard relational method; collapses repeated data into a single row to save space
(separate store table from transaction table).
• Row Splitting — maintains a one-to-one map between partitions; used to speed up access to large tables
by reducing their size.
Important: When using vertical partitioning, ensure there is no requirement to perform major JOIN operations
between two partitions, as this negates the performance benefit.
Choosing the Right Partition Key
Choosing the wrong partition key forces a full reorganisation of the fact table — a costly operation. Always
analyse the majority of query patterns first.
Example: An Account_Txn_Table can be partitioned by region or transaction_date. If most queries are
region-specific, partition by region (30 partitions = manageable). Partitioning by date would force region-users
to scan multiple partitions — inefficient.
Rule of thumb: Partition only by the time dimension unless you are certain that the chosen dimension
grouping will not change during the life of the data warehouse.
■ QUICK-REFERENCE CHEAT SHEET (All Lectures)
Key Definitions to Remember
Term Definition
Data Mining Extracting useful information/knowledge from large datasets.
KDD Knowledge Discovery in Databases — the full pipeline; data mining is its core step.
Classification Assigns class labels to new records based on learned attributes. Predictive.
Clustering Groups similar records without predefined labels. Descriptive.
Association Finds relationships between items (e.g., market basket analysis). Descriptive.
Regression Predicts numeric values from independent variables.
Data Warehouse Central repository integrating multiple data sources for reporting & analysis.
Data Mart Subject-specific subset of a data warehouse for a specific user group.
Partitioning Splitting fact tables into smaller units for performance, management, and backup.
Star Schema Simplest schema for data mart design — fact table at centre, dimensions around it.
Granularity The lowest level of data stored in a fact table — first decision in fact design.
Normalization Relational DB method that collapses repeated rows to reduce space (vertical
partition).
Predictive vs. Descriptive Tasks
Category Task Purpose
Predictive Classification Assign class labels to new data
Predictive Prediction Forecast future/missing values
Predictive Time-Series Analysis Analyse trends over time
Descriptive Association Find item relationships
Descriptive Clustering Group similar items
Descriptive Summarization Compact data representation
Partitioning Strategy Summary
Strategy Key Points
Horizontal — Equal Time Same-size time intervals (monthly, quarterly). Reusable tables.
Horizontal — Varied Time Small recent + large old partitions. Lowers table count.
Horizontal — Dimension By region, product, supplier. Fast for dimension-specific queries.
Strategy Key Points
Horizontal — Size New partition when table exceeds set size. Most complex.
Round Robin Archives old partition when new one needed. Easy automation.
Vertical — Normalization Collapses repeated rows. Reduces space. Standard relational approach.
Vertical — Row Splitting 1:1 map between partitions. Speeds access by reducing table size.
Final Exam Tip: Know the difference between Data Warehousing (storing/organising data) and Data Mining
(extracting patterns). Understand all 6 data mining tasks, whether they are predictive or descriptive, and be
able to give a real-world example for each. For Lecture 4, focus on the reasons for partitioning, the
horizontal strategies, and how to choose the right partition key. Good luck!