Sahil Kumar Rai DWDM Assignment - 2

Data Warehousing and Data Mining

Assignment – 2

1. What are data mining primitives?

Ans: - Data mining primitives are the fundamental concepts or building blocks
that define how data mining tasks are performed. They include:
 Task-Relevant Data – Specifies the portion of data to be mined.
 Knowledge to be Mined – Defines the type of patterns to be discovered
(e.g., associations, classifications).
 Background Knowledge – Additional domain-specific knowledge that
improves mining accuracy.
 Interestingness Measures – Criteria to evaluate the usefulness of
discovered patterns.
 Presentation & Visualization – Defines how the results will be displayed
for interpretation.
These primitives help in designing effective data mining processes.

2. What is the difference between classification and clustering?

Ans: - The key difference between classification and clustering is:

 Classification is a supervised learning technique where data is assigned
to predefined categories based on labelled training data (e.g.,
classifying emails as spam or not spam).
 Clustering is an unsupervised learning technique where data is grouped
into clusters based on similarity without predefined labels (e.g.,
grouping customers based on purchasing behaviour).
In short, classification requires labelled data, while clustering finds hidden
patterns in unlabelled data.

1|Page
Sahil Kumar Rai DWDM Assignment - 2

3. How data mining is the primary step in knowledge discovery in data?

Ans: - Data mining is the core step in the Knowledge Discovery in Databases
(KDD) process because it applies intelligent techniques to extract patterns,
trends, and useful information from raw data. It transforms large volumes of
data into meaningful insights, which are then evaluated and interpreted to
support decision-making. Without data mining, the knowledge discovery
process cannot effectively identify valuable patterns hidden within complex
datasets.
4. Need for Data Pre-Processing.

Ans: - Need for Data Pre-Processing

Data pre-processing is essential in data mining because real-world data is
often incomplete, noisy, and inconsistent. It improves the quality of data and
enhances the accuracy of mining results. The key reasons for data pre-
processing include:
 Handling Missing Data – Filling or removing missing values to avoid
biased results.
 Noise Reduction – Removing errors or irrelevant information for better
pattern recognition.
 Data Integration – Combining data from multiple sources for a unified
analysis.
 Data Transformation – Normalizing or converting data into a suitable
format.
 Data Reduction – Reducing data size while maintaining its integrity for
efficient processing.
Effective data pre-processing ensures better data quality, improved model
accuracy, and meaningful insights in data mining.
5. Data Pre-processing Techniques

Ans: - Data Pre-Processing Techniques

2|Page
Sahil Kumar Rai DWDM Assignment - 2

Data pre-processing involves various techniques to clean, transform, and

prepare data for analysis. The key techniques include:
Data Cleaning – Handling missing, noisy, and inconsistent data by:
o Filling in missing values (e.g., mean, median, or predictive
methods).
o Removing duplicate or incorrect data.
Data Integration – Combining data from multiple sources while resolving
inconsistencies and redundancy.
Data Transformation – Converting data into a suitable format using:
o Normalization – Scaling values to a standard range (e.g., 0 to 1).
o Aggregation – Summarizing data (e.g., weekly sales from daily
sales).
Data Reduction – Reducing data size while retaining meaningful
information through:
o Dimensionality Reduction (e.g., Principal Component Analysis,
PCA).
o Data Compression and Sampling.
Data Discretization – Converting continuous data into discrete categories
for better analysis (e.g., binning age into groups like "young," "middle-
aged," "senior").
These techniques ensure higher data quality, efficiency, and accuracy in data
mining tasks.
6. “Every day in the data warehouse is stamped” Explain.
Ans: - The phrase "Every day in the data warehouse is stamped" refers to
the concept of time-variant data in a data warehouse. This means that all
data stored in a data warehouse is associated with a specific timestamp or
time period, allowing historical analysis and trend identification.
Key Points:

3|Page
Sahil Kumar Rai DWDM Assignment - 2

1. Time-Variant Nature – Unlike operational databases that focus on

current data, data warehouses store historical data with time
references.
2. Tracking Changes Over Time – Every record includes a time dimension
(e.g., date, week, month, year) to analyze trends and patterns.
3. Decision-Making – Businesses can compare past and present data to
make strategic decisions.
4. Example – A sales data warehouse might store daily sales transactions
with timestamps, enabling trend analysis over different periods.
Thus, the timestamping of data in a warehouse ensures accurate historical
tracking and better business intelligence.
7. List the major benefits of Data Mining
Ans: - Major Benefits of Data Mining
 Improved Decision-Making – Helps businesses make data-driven
decisions by uncovering hidden patterns and trends.
 Fraud Detection – Identifies suspicious activities in banking, finance,
and cybersecurity.
 Customer Insights – Analyses customer behaviour for targeted
marketing and personalized recommendations.
 Predictive Analysis – Forecasts future trends, sales, and risks in various
industries.
 Increased Efficiency – Automates data analysis, reducing manual effort
and time.
 Competitive Advantage – Helps businesses stay ahead by
understanding market trends and customer needs.
 Enhanced Data Quality – Improves data accuracy by detecting
inconsistencies and missing values.
 Optimized Operations – Enhances supply chain management, resource
allocation, and business processes.
Data mining transforms raw data into valuable insights, enabling smarter
business strategies and improved performance.
4|Page
Sahil Kumar Rai DWDM Assignment - 2

8. What do you mean by data mining functionalities?

Ans: -Data Mining Functionalities
Data mining functionalities refer to the types of patterns that can be
discovered in a dataset. They help in extracting meaningful information for
decision-making. The major functionalities include:
 Characterization – Summarizing general features of data (e.g., average
sales per region).
 Discrimination – Comparing two datasets to find differences (e.g., high
vs. low-performing customers).
 Association Rule Mining – Finding relationships between items (e.g.,
"Customers who buy bread also buy butter").
 Classification – Assigning data to predefined categories using a model
(e.g., spam or non-spam emails).
 Clustering – Grouping similar data without predefined categories (e.g.,
customer segmentation).
 Outlier Detection – Identifying anomalies or unusual data points (e.g.,
fraud detection).
 Evolution Analysis – Tracking changes in data over time (e.g., stock
market trends).
These functionalities help extract useful insights, trends, and patterns
from large datasets.
9. What is data mining? Explain the Knowledge Discovery Data ((KDD)
process in detail.
Ans: - Knowledge Discovery in Databases (KDD) Process
The KDD process consists of several steps to extract useful knowledge
from data. The main stages are:
 Data Selection – Identifying and selecting relevant data from multiple
sources for analysis.
 Data Preprocessing (Cleaning & Integration) – Handling missing values,
removing noise, and integrating data from different sources to improve
quality.
5|Page
Sahil Kumar Rai DWDM Assignment - 2

 Data Transformation – Converting data into an appropriate format,

such as normalization or aggregation, to enhance mining performance.
 Data Mining – Applying algorithms to extract patterns, trends, and
relationships from the processed data.
 Pattern Evaluation – Assessing the discovered patterns using
interestingness measures to ensure their relevance and usefulness.
 Knowledge Presentation – Visualizing the mined knowledge using
reports, charts, or graphs for better understanding and decision-
making.
The KDD process ensures that raw data is transformed into valuable
knowledge for insights and decision support.
10. What is data integration and transformation? How are these
performed?
Ans: - Data Integration:
Data integration is the process of combining data from multiple sources
into a single, unified dataset for analysis. It ensures consistency and
eliminates redundancy.
How Data Integration is Performed?
 Schema Integration: Merging different data structures (e.g., aligning
column names from different databases).
 Entity Resolution: Identifying and linking records referring to the same
entity (e.g., "John Doe" and "J. Doe" in customer records).
 Handling Redundancy: Removing duplicate or inconsistent data from
different sources.

Data Transformation:
Data transformation is the process of converting data into a suitable
format for mining and analysis.
How Data Transformation is Performed?
 Normalization: Scaling data to a common range (e.g., converting values
to a 0-1 scale).

6|Page
Sahil Kumar Rai DWDM Assignment - 2

 Aggregation: Summarizing data (e.g., monthly sales from daily sales).

 Smoothing: Reducing noise using techniques like binning, regression, or
clustering.
 Discretization: Converting continuous data into categories (e.g., age
groups: "Young," "Middle-aged," "Senior").
Both data integration and transformation are crucial pre-processing steps
that improve the quality and efficiency of data mining processes.
11. Explain the three data models used in Data Warehouse
Ans: - Three Data Models Used in a Data Warehouse
Data warehouses use different data models to structure and organize data
efficiently. The three primary models are:
Conceptual Data Model
o Focuses on high-level design and defines the business entities,
attributes, and relationships between them.
o It is technology-independent and used for understanding
business requirements.
o Example: A data warehouse for a retail company may have
entities like Customer, Product, Sales, and Supplier.
Logical Data Model
o Represents the detailed structure of the data warehouse using
tables, columns, and relationships but is independent of any
specific database system.
o Defines keys, constraints, normalization, and relationships
between tables.
o Example: The Sales table may have attributes like Sale_ID,
Customer_ID, Product_ID, Date, and Amount.
Physical Data Model
o Describes how the data will be stored physically in the database
system.

7|Page
Sahil Kumar Rai DWDM Assignment - 2

o Includes indexes, partitions, storage format, and access methods

to optimize performance.
o Example: Using indexing to speed up query performance or
partitioning data by year for fast retrieval.
12. Why is the entity-relationship model technique not suitable for the Data
warehouse? Explain with Example.
Ans: - The Entity-Relationship (ER) model is designed for transactional
(OLTP) systems, which focus on fast insert, update, and delete operations.
However, a data warehouse (OLAP) is optimized for analytical queries and
reporting, making the ER model inefficient for the following reasons:
Complex Joins and Performance Issues
 The ER model consists of normalized tables with many relationships,
requiring multiple joins to retrieve data.
 In a data warehouse, queries involve large datasets and complex
aggregations, making joins slow and inefficient.
 Example: A query to analyze total sales by region may require joins
across Customers, Orders, Products, and Sales tables, slowing down
performance.
Lack of Dimensional Organization
 ER models focus on entities and relationships, whereas a data
warehouse needs a dimensional model (Star Schema or Snowflake
Schema) for fast analytical queries.
 Example: A retail company needs to analyze sales by product, time, and
region. Using a dimensional model (fact & dimension tables) instead of
an ER model improves query efficiency.
Data Redundancy is Needed for Faster Access
 Data warehouses use denormalization to store pre-aggregated and
redundant data for quick retrieval.
 The ER model is fully normalized, which reduces redundancy but makes
queries complex and slow.

8|Page
Sahil Kumar Rai DWDM Assignment - 2

 Example: In a data warehouse, a Sales Fact Table might store total

revenue per product per day, whereas an ER model would require on-
the-fly aggregation, increasing query time.
ER Models Are Not Optimized for Historical Data
 ER models are designed for current transactional data with frequent
updates.
 Data warehouses store historical data for trend analysis and require
time-variant data storage.
 Example: A business may need 5 years of sales data to analyze trends,
which an ER model does not efficiently support.
13. What is the Knowledge Discovery Process (KDD)? Explain with suitable
examples how it works.
Ans: - The Knowledge Discovery in Databases (KDD) process is a
systematic approach to extracting meaningful patterns, trends, and
knowledge from large datasets. It involves multiple stages, from raw data
collection to actionable insights.

Steps in the KDD Process (with Example)

Data Selection
 Identifying and selecting relevant data from multiple sources.
 Example: A retail company collects data from online purchases, in-store
sales, and customer feedback.
Data Preprocessing (Cleaning & Integration)
 Handling missing values, removing noise, and resolving inconsistencies.
 Example: Fixing missing customer age data by replacing it with the
average age or removing duplicate records.
Data Transformation
 Converting data into a suitable format for analysis, such as
normalization or aggregation.

9|Page
Sahil Kumar Rai DWDM Assignment - 2

 Example: Converting raw sales data into a structured format with daily,
weekly, and monthly summaries.
Data Mining
 Applying machine learning, statistical techniques, or algorithms to find
patterns.
 Example: Using association rule mining to find that "Customers who
buy laptops often buy a mouse."
Pattern Evaluation
 Assessing the discovered patterns for their usefulness and significance.
 Example: Checking if the identified "laptop-mouse" association holds
for all customer segments.
Knowledge Presentation
 Visualizing and representing the results in a meaningful way for
decision-making.
 Example: Creating a dashboard showing customer buying patterns to
help managers optimize product placement.
14. Every Data Structure in a Data warehouse contains the Time element.
Why?
Ans: - In a data warehouse, the time element is essential because it
enables historical analysis, trend identification, and decision-making.
Unlike transactional databases (OLTP), which focus on current data, a data
warehouse (OLAP) stores and analyzes data over time.
Key Reasons for the Time Element in a Data Warehouse:
Historical Data Storage
o Data warehouses store past and present data for long-term
analysis.
o Example: A sales data warehouse may store daily, monthly, and
yearly sales trends to analyze business growth.
Time-Variant Nature

10 | P a g e
Sahil Kumar Rai DWDM Assignment - 2

o Data warehouses track changes over time rather than just the
latest state.
o Example: A customer's address changes; the warehouse retains
previous addresses for historical analysis.
Trend and Pattern Analysis
o Businesses use time-stamped data to identify seasonal trends,
customer behavior, and market shifts.
o Example: A retail company analyzes shopping trends during
different holidays over the years.
Comparison and Forecasting
o Time-stamped data helps compare past vs. present performance
and predict future trends.
o Example: Comparing last year’s Q1 sales with the current year to
measure growth.
Data Integrity and Consistency
o Maintaining time-stamped records ensures accurate tracking of
data changes.
o Example: Employee salary history records show changes over
time rather than overwriting old data.
15. What are the fields in which clustering techniques are used? Mention
any four fields. Discuss the basic requirements of cluster analysis.
Ans: - Clustering is widely used in various domains to group similar data
points together for analysis. Here are four key fields where clustering
techniques are applied:
Marketing & Customer Segmentation
o Used to identify different customer groups based on purchasing
behavior, demographics, or preferences.
o Example: A retail company segments customers into high-value,
budget-conscious, and occasional buyers.
Healthcare & Medical Diagnosis

11 | P a g e
Sahil Kumar Rai DWDM Assignment - 2

o Helps in disease classification, patient grouping, and genetic data

analysis.
o Example: Clustering patients based on symptoms to detect
patterns in disease outbreaks.
Image Processing & Pattern Recognition
o Used in facial recognition, object detection, and handwriting
recognition.
o Example: Grouping similar images in a photo album based on
facial features.
Anomaly Detection (Fraud Detection & Cybersecurity)
o Identifies unusual patterns that could indicate fraud or security
threats.
o Example: Detecting fraudulent credit card transactions by
clustering normal vs. abnormal behavior.

Basic Requirements of Cluster Analysis

For effective cluster analysis, certain requirements must be met:
Scalability
o The algorithm should handle large datasets efficiently.
o Example: Processing millions of customer records in an e-
commerce database.
High-Dimensional Data Handling
o Should work well with datasets having multiple attributes or
dimensions.
o Example: Analyzing medical records with multiple parameters
like age, symptoms, and test results.
Interpretability and Usability
o The clusters should be meaningful and provide actionable
insights.

12 | P a g e
Sahil Kumar Rai DWDM Assignment - 2

o Example: Customer clusters should clearly distinguish between

high-value and low-value customers.
Handling Different Data Types
o Clustering should work with categorical, numerical, and mixed
data types.
o Example: Grouping products based on numerical (price, rating)
and categorical (category, brand) features.
16. Why is Outlier Mining important? Briefly describe the different
approaches behind statistical-based outlier detection and distance-
based outlier detection.
Ans: - Importance of Outlier Mining
Outlier mining is crucial because it helps identify rare, unusual, or
unexpected patterns in data that could indicate:
Fraudulent Activities – Detects credit card fraud, cyber attacks, or financial
fraud.
Medical Diagnosis – Identifies abnormal health conditions from patient
records.
Fault Detection – Finds defects in manufacturing or sensor networks.
Data Quality Improvement – Helps in detecting and correcting errors in
datasets.

Approaches to Outlier Detection

Statistical-Based Outlier Detection
 Assumes that data follows a standard statistical distribution (e.g.,
Normal Distribution).
 Identifies outliers as data points that significantly deviate from the
expected statistical range.
Methods:
 Z-Score Method: Data points with a Z-score beyond a threshold (e.g.,
±3) are considered outliers.

13 | P a g e
Sahil Kumar Rai DWDM Assignment - 2

 Box Plot Method (IQR-Based): Uses the Interquartile Range (IQR) to

define thresholds (outliers are beyond Q1 - 1.5×IQR or Q3 + 1.5×IQR).
 Probability Distribution-Based: Outliers are detected if their probability
of occurring is extremely low under a given distribution.
Example: If the average salary in a company is $50,000 with a standard
deviation of $5,000, an employee earning $100,000 would be flagged as
an outlier using the Z-score method.

Distance-Based Outlier Detection

 Identifies outliers based on their distance from other data points in a
dataset.
 Useful when no clear statistical distribution exists in the data.
Methods:
 K-Nearest Neighbors (KNN): A point is an outlier if it is far from its
nearest neighbors.
 Density-Based (DBSCAN): Data points in low-density regions are
considered outliers.
 Clustering-Based: Outliers are points that do not belong to any cluster
or are in very small clusters.
Example: In a customer database, if most customers spend between
$100–$500 monthly, but one spends $10,000, distance-based methods
would classify them as an outlier.

17. How can you use the Web as a data source for your data warehouse?
What type of information do you get from the Web?
Ans: - You can use the Web as a data source for your data warehouse by
extracting, transforming, and loading (ETL) web data into your system.
Web scraping, APIs, and web logs are common methods for collecting
data.
Types of information you can get from the Web include:

14 | P a g e
Sahil Kumar Rai DWDM Assignment - 2

 Structured Data (e.g., APIs, databases, government datasets)

 Semi-Structured Data (e.g., JSON, XML, RSS feeds)
 Unstructured Data (e.g., social media posts, blogs, articles, reviews)
 Web Traffic Data (e.g., user behaviour, clickstream data, logs)
This data can be used for business intelligence, analytics, and decision-
making.
18. Name the major phase of Data Mining operations. Out of these pick any
two phases and describe the type of activities in these two phases.
Ans: - Major Phases of Data Mining Operations:
 Data Cleaning
 Data Integration
 Data Selection
 Data Transformation
 Data Mining
 Pattern Evaluation
 Knowledge Presentation
Data Cleaning
 Identifies and removes noise, errors, and inconsistencies in raw data.
 Handles missing values, duplicate records, and incorrect data entries.
 Ensures data quality for accurate analysis.
Data Integration
 Combines data from multiple sources into a unified dataset.
 Resolves data format conflicts, schema mismatches, and redundancy.
 Prepares a consolidated view for further processing.
Data Selection
 Extracts relevant data from the integrated dataset.
 Filters unnecessary or irrelevant information.

15 | P a g e
Sahil Kumar Rai DWDM Assignment - 2

 Ensures only useful data is passed to the next phase.

Data Transformation
 Converts data into a suitable format for mining.
 Normalization, aggregation, and attribute construction are performed.
 Enhances data consistency and reduces complexity.
Data Mining
 Applies algorithms to discover hidden patterns and relationships.
 Uses techniques such as classification, clustering, and association rule
mining.
 Extracts meaningful insights from the data.
Pattern Evaluation
 Assesses the discovered patterns to determine their relevance and
usefulness.
 Uses statistical measures, visualization techniques, and validation
methods.
 Selects significant patterns for decision-making.
Knowledge Presentation
 Represents mined patterns in an understandable format.
 Uses reports, graphs, charts, and interactive dashboards.
 Helps stakeholders interpret and utilize the findings effectively.
19. Explain Data granularity and how it applies to the data warehouse.
Ans: - Data Granularity in a Data Warehouse
Data granularity refers to the level of detail or summarization in a
dataset. It determines how detailed or aggregated the stored data is.
Application in a Data Warehouse:
Fine Granularity (Detailed Data)
o Contains transactional or raw data with high detail (e.g.,
individual sales transactions).

16 | P a g e
Sahil Kumar Rai DWDM Assignment - 2

o Enables in-depth analysis but requires more storage and

processing power.
Coarse Granularity (Aggregated Data)
o Summarized data (e.g., total sales per month instead of
individual sales).
o Reduces storage requirements and improves query performance.
Balancing granularity is crucial—detailed data supports flexible analysis,
while aggregated data enhances efficiency. Data warehouses often store
both levels to meet different analytical needs.
20. In real-world data, tuples with missing values for some attributes are
common occurrences. Describe any five methods for
handling this problem.
Ans: - Methods for Handling Missing Values in Data
Deletion (Removing Tuples)
o Case Deletion: Remove records (tuples) with missing values.
o Works well if missing data is minimal but can cause data loss if
excessive.
2. Mean/Median/Mode Imputation
o Replace missing values with the mean (numerical data), median
(skewed data), or mode (categorical data).
o Simple and effective but may introduce bias.
Prediction using Regression or Machine Learning
o Use regression models or ML algorithms to predict missing
values based on other attributes.
o More accurate but computationally expensive.
Interpolation & Forward/Backward Filling
o Estimate missing values using trends in sequential data (e.g.,
time series).
o Useful for structured datasets but assumes predictable patterns.

17 | P a g e
Sahil Kumar Rai DWDM Assignment - 2

Using "Unknown" or Default Values

o Assign a placeholder value (e.g., “Unknown” for categorical data
or 0 for numerical data).
o Maintains dataset size but may affect model performance.
The best method depends on the nature of the dataset and the extent of
missing values.

18 | P a g e