ENDSEM IMP BUSINESS INTELLIGENCE

UNIT – 4
Q.1] Explain data exploration in detail with example.
ANS : Data Exploration

Data exploration is the first step in data analysis, where we examine the
dataset to understand its structure, patterns, and important features. It
helps in identifying missing values, detecting outliers, and finding
relationships between variables.

Key Steps in Data Exploration:

1. Loading the Data:
o Import the dataset using tools like Python (Pandas), Excel, or
SQL.
o Example: df = pd.read_csv('[Link]') in Python.
2. Understanding the Data:
o Check the number of rows and columns.
o Example: [Link] shows the dataset size.
3. Checking Missing Values:
o Identify if any data is missing and decide how to handle it.
o Example: [Link]().sum() in Python shows missing values per
column.
4. Summary Statistics:
o Use descriptive statistics to get mean, median, standard
deviation, etc.
o Example: [Link]() in Python provides statistical summaries.
5. Visualizing the Data:
o Use graphs like histograms, scatter plots, and box plots to
understand distributions.
o Example: A histogram can show the age distribution of
customers.
6. Finding Relationships:
o Check correlations between variables to find meaningful
patterns.
o Example: A correlation matrix ([Link]()) shows how two
variables are related.
Example:
Imagine you have a dataset of students' exam scores.
 You check how many students are there ([Link]).
 You find missing scores ([Link]().sum()).
 You calculate average scores ([Link]()).
 You plot a histogram to see how scores are distributed.
 You check if study hours and scores are related ([Link]()).

This process helps in making data-driven decisions before applying

machine learning or deeper analysis.
Q.2] Explain data transformation in detail with example.
ANS : Data Transformation

Data transformation is the process of converting raw data into a suitable

format for analysis. It includes cleaning, structuring, and modifying data to
improve its quality and usability.

Key Steps in Data Transformation:

1. Handling Missing Values:
o Fill missing values with the mean, median, mode, or drop them.
o Example: [Link]([Link]()) fills missing values with the column
mean.
2. Normalization and Scaling:
o Converts data into a common range (e.g., 0 to 1) to improve
accuracy in models.
o Example: Min-Max Scaling scales values between 0 and 1.
 Formula:

3. Encoding Categorical Data:

o Converts text data into numerical values for analysis.
o Example: Replacing "Male" with 0 and "Female" with 1 (Label
Encoding).
4. Removing Duplicates and Outliers:
o Identifies and removes duplicate records and extreme values
that may distort analysis.
o Example: df.drop_duplicates() removes duplicate rows.
5. Feature Engineering:
o Creating new meaningful variables from existing data.
o Example: If a dataset has a birth year column, you can create an
"Age" column by subtracting the birth year from the current year.
6. Data Type Conversion:
o Converts data into appropriate types (e.g., string to date format).
o Example: pd.to_datetime(df['date_column']) converts a text date
into a DateTime format.
Example:

Imagine you have a dataset of employees with columns: Name, Age, Salary,
Gender, and Joining Date.
 You fill missing ages with the average age.
 You scale the Salary column to a 0-1 range.
 You encode Gender as 0 (Male) and 1 (Female).
 You remove duplicate employee records.
 You create a new column "Years of Service" from the Joining Date.
 You convert the date format correctly.

This process ensures the data is clean, structured, and ready for analysis.
Q.3] Explain data validation, incompleteness, noise, inconsistency of quality
of input data.
ANS : Data Quality Issues and Validation

Data quality is important for accurate analysis and decision-making. Poor-

quality data can lead to wrong conclusions. Below are key issues related to
data quality and validation:

1. Data Validation
 It is the process of checking if the input data is correct, complete, and
useful.
 Ensures data follows specific rules (e.g., correct format, valid values).
 Example: A phone number should have 10 digits; if not, it's invalid.
2. Data Incompleteness
 Missing values or incomplete records in a dataset.
 Can happen due to human errors, system failures, or data entry
mistakes.
 Example: A customer record without an email address is incomplete.
3. Data Noise
 Unwanted or irrelevant data that affects accuracy.
 Can occur due to sensor errors, transmission issues, or human
mistakes.
 Example: A temperature sensor recording extreme values (like -
100°C) due to malfunction.
4. Data Inconsistency
 When the same data appears in different formats or contradicts itself.
 Can happen when data is entered manually from multiple sources.
 Example: A person’s birthdate recorded as "1990-05-12" in one system
and "12-May-1990" in another.
Q.4] Explain data reduction in detial with example.
ANS : Data Reduction

Data reduction is the process of minimizing the amount of data while

preserving its important information. It helps in improving data processing
speed, reducing storage requirements, and making analysis easier.

Key Methods of Data Reduction:

1. Dimensionality Reduction:
o Reducing the number of features (columns) while keeping
essential information.
o Example: Principal Component Analysis (PCA) reduces multiple
related variables into fewer important ones.
2. Data Compression:
o Storing data in a compact form to reduce file size.
o Example: ZIP files or JPEG image compression reduces storage
but retains most details.
3. Numerosity Reduction:
o Replacing the original data with a smaller model or
representative data.
o Example: Using clustering (like K-Means) to group similar data
points instead of storing each one separately.
4. Sampling:
o Selecting a small portion of the data instead of using the full
dataset.
o Example: Instead of analyzing 1 million customer transactions,
selecting a random 10,000 transactions to find patterns.
5. Data Cube Aggregation:
o Summarizing data at different levels for easier analysis.
o Example: Instead of storing daily sales, storing only monthly or
yearly totals.
6. Feature Selection:
o Choosing only the most important variables and removing
irrelevant ones.
o Example: In a student performance dataset, keeping only study
hours and test scores while removing unnecessary columns like
student ID.
Example:

A company has a database with millions of customer purchase records.

 They use sampling to analyze only 50,000 transactions instead of the
entire dataset.
 They apply PCA to reduce 100 features to 10 key features.
 They store summarized sales data by month instead of daily records
(data cube aggregation).

By applying data reduction techniques, businesses can process data faster,

save storage, and improve decision-making efficiency.
Q.5] Difference between univariate, Bivariate, Multivariate analysis.
ANS : Difference Between Univariate, Bivariate, and Multivariate Analysis

1. Univariate Analysis
 Definition: Examining one variable at a time.
 Purpose: To understand the distribution, central tendency (mean,
median), and spread (variance, standard deviation) of a single
variable.
 Example: Checking the average height of students.
 Visualization: Histogram, box plot, bar chart.
2. Bivariate Analysis
 Definition: Analyzing the relationship between two variables.
 Purpose: To see how one variable affects another (correlation or
dependence).
 Example: Checking if study hours and exam scores are related.
 Visualization: Scatter plot, line chart, correlation matrix.
3. Multivariate Analysis
 Definition: Examining three or more variables together.
 Purpose: To understand complex relationships among multiple
factors.
 Example: Studying how age, income, and education affect spending
habits.
 Visualization: 3D plots, heatmaps, multiple regression models.

Key Differences

Analysis Number of Purpose

Type Variables
Univariate 1 Understand one variable
Bivariate 2 Find relationships between two variables
Multivariate 3 or more Analyze complex interactions between
multiple variables

These methods help in understanding data better and making informed

decisions.
Q.6] Write a short note on data discretization.
ANS : Data Discretization

Data discretization is the process of converting continuous data (numeric

values) into discrete categories or groups. It helps simplify data analysis
and improves the performance of machine learning models.

Key Points on Data Discretization:

1. Definition:
o It transforms continuous numerical data into discrete bins or
intervals.
2. Why is it Needed?
o Simplifies complex data for easier interpretation.
o Reduces noise in the data.
o Helps machine learning models that work better with categorical
data.
3. Methods of Data Discretization:
o Equal-width binning: Divides data into equal-sized intervals.
o Equal-frequency binning: Each bin has an equal number of data
points.
o Clustering-based discretization: Uses clustering techniques like
K-means.
4. Example:
o Suppose we have student ages: [16, 17, 18, 19, 20, 21, 22].
o Using equal-width binning, we create intervals:
 16-18 → "Young"
 19-21 → "Adult"
 22+ → "Senior"
5. Advantages:
o Improves efficiency in data processing.
o Helps in better visualization and pattern recognition.
6. Applications:
o Used in decision trees, Naïve Bayes classification, and data
mining.

Discretization makes continuous data easier to understand and analyze.

Q.7] Compute Mean, Median and Mode for following data
Class Interval 10-15 15-20 20-25 25-30 30-35 35-40 40-45 45-50
Frequency (f) 2 28 125 270 303 197 65 10
ANS : Mean, Median, and Mode Calculation

Given Data:
Class Interval 10-15 15-20 20-25 25-30 30-35 35-40 40-45 45-50
Frequency (f) 2 28 125 270 303 197 65 10

1. Mean Calculation
Steps:
1. Find the midpoint (x) of each class:
o Formula: Midpoint (x) = (Lower Bound + Upper Bound) / 2
o Example for 10-15: (10+15)/2 = 12.5
o Midpoints: 12.5, 17.5, 22.5, 27.5, 32.5, 37.5, 42.5, 47.5
2. Multiply frequency (f) by midpoint (x) to get fx for each class.
3. Find Σf (Total Frequency) = 1000
4. Find Σfx (Total fx) = 30775
5. Calculate Mean:
o Mean = Σfx / Σf
o Mean = 30775 / 1000 = 30.78

2. Median Calculation
Steps:
1. Find N/2 (Total frequency ÷ 2)
o N = 1000, so N/2 = 500
2. Locate the median class (where cumulative frequency reaches 500).
o Cumulative frequencies:
 2, 30, 155, 425, 728 (Median Class: 30-35)
3. Use Median Formula:
o Median = L + [(N/2 - CF) / f] × h
o L = 30, N/2 = 500, CF = 425, f = 303, h = 5
4. Median = 30 + [(500 - 425) / 303] × 5
o Median = 30 + (75 / 303) × 5
o Median ≈ 30 + 1.24 = 31.24
3. Mode Calculation
Steps:
1. Find the modal class (highest frequency).
o Highest frequency = 303 (Class: 30-35)
2. Use Mode Formula:
o Mode = L + [(f1 - f0) / (2f1 - f0 - f2)] × h
o L = 30, f1 = 303, f0 = 270, f2 = 197, h = 5
3. Mode = 30 + [(303 - 270) / (2 × 303 - 270 - 197)] × 5
o Mode = 30 + (33 / (606 - 270 - 197)) × 5
o Mode = 30 + (33 / 139) × 5
o Mode ≈ 30 + 1.19 = 31.19

Final Answers:
 Mean = 30.78
 Median = 31.24
 Mode = 31.19
Q.8] What is data Transformation? Explain Data Transformation Process in
Detail.
ANS : Data Transformation

Data transformation is the process of converting data into a suitable

format for analysis. It includes cleaning, structuring, and modifying data to
improve its quality and usefulness.

Data Transformation Process:

1. Data Cleaning:
o Remove duplicate or incorrect data.
o Handle missing values by filling them with the mean, median, or
removing them.
o Example: Replacing missing age values with the average age of
all records.
2. Data Integration:
o Combine data from multiple sources into a single dataset.
o Example: Merging customer data from different departments like
sales and support.
3. Data Normalization and Scaling:
o Convert data into a standard format to avoid large differences in
values.
o Example: Scaling income values between 0 and 1 for machine
learning models.
4. Data Encoding:
o Convert categorical data (text) into numerical form for analysis.
o Example: Changing "Male" and "Female" into 1 and 0.
5. Feature Selection and Extraction:
o Choose the most important variables (features) for analysis.
o Example: Selecting only "Age" and "Salary" to predict loan
approval.
6. Data Aggregation:
o Summarize data by grouping it.
o Example: Finding the average sales per month instead of daily
sales.
Example:

If you have customer data with missing ages, different currencies, and
gender as text:
 Fill missing ages with the average.
 Convert currency values to a standard unit.
 Encode gender as 1 (Male) and 0 (Female).
 Normalize income to a 0-1 scale.

This process ensures the data is clean, consistent, and ready for analysis
or machine learning.
Q.9] Explain univariate, bi variate and multivariate analysis with example
and applications.
ANS : Types of Data Analysis

1. Univariate Analysis (One Variable)

 Definition: Examining one variable at a time to understand its
distribution, patterns, and summary statistics.
 Example: Analyzing students' test scores to find the average, highest,
and lowest marks.
 Methods Used:
o Summary statistics (mean, median, mode)
o Histograms, box plots
 Application:
o Finding average customer age in a store.
o Identifying salary distribution in a company.

2. Bivariate Analysis (Two Variables)

 Definition: Studying the relationship between two variables to
understand how they influence each other.
 Example: Analyzing study hours and test scores to see if more study
time leads to higher marks.
 Methods Used:
o Scatter plots, correlation coefficient, regression analysis
 Application:
o Checking if marketing spend affects sales.
o Finding the relationship between temperature and ice cream
sales.

3. Multivariate Analysis (More than Two Variables)

 Definition: Analyzing three or more variables simultaneously to find
complex relationships.
 Example: Analyzing test scores, study hours, and sleep hours
together to see their combined effect on performance.
 Methods Used:
o Multiple regression, heatmaps, clustering
  Application:
o Predicting house prices using factors like location, size, and
number of bedrooms.
o Analyzing customer behavior based on age, income, and buying
habits.

This type of analysis helps in making better decisions by considering

multiple factors at the same time.
Q.10] What is a Contingency Table? What is Marginal Distribution? Justify
with suitable example.
ANS : Contingency Table & Marginal Distribution
1. Contingency Table:
A contingency table is a table used to organize and analyze the relationship
between two categorical variables. It shows the frequency (count) of
occurrences for different combinations of variables.
Example:
A survey of 100 students records their preference for tea or coffee based
on gender.
Tea Coffee Total
Male 30 20 50
Female 25 25 50
Total 55 45 100
 The table shows how many males and females prefer tea or coffee.

2. Marginal Distribution:
Marginal distribution is the total count for each category of a single
variable, found in the margins (last row/column) of a contingency table. It
helps in understanding the overall distribution of one variable without
considering the other.
Example from Table Above:
 Marginal distribution of drink preference:
o Tea: 55 students
o Coffee: 45 students
 Marginal distribution of gender:
o Males: 50 students
o Females: 50 students

Justification with Example:

 The contingency table helps compare two variables (gender & drink
preference).
 Marginal distribution helps analyze how each category behaves
independently.
 Example: From the table, more students prefer tea (55) over coffee
(45), which can guide decisions like stocking more tea in a cafeteria.
Q.11] Explain following Data reduction technique: Sampling, Feature
selection, Principal component analysis.
ANS : Data Reduction Techniques
Data reduction techniques help to reduce the complexity of a dataset,
making it easier to analyze and interpret, while maintaining the key
information. Here are three common techniques:

1. Sampling
Definition:
Sampling involves selecting a subset of the data from the full dataset to
represent the entire population.
Key Points:
 Purpose: To reduce the size of data while maintaining the essential
patterns.
 Types of Sampling:
o Random Sampling: Randomly selects data points.
o Stratified Sampling: Divides data into groups (strata) and
samples from each group.
 Example: If you have data from 10,000 customers, you might randomly
sample 1,000 to analyze trends in the data.
Advantages:
 Reduces computation time and resources.
 Helps with quicker analysis while still representing the dataset.

2. Feature Selection
Definition:
Feature selection involves choosing the most important features (or
variables) from the dataset to reduce its dimensionality.
Key Points:
 Purpose: To remove irrelevant or redundant features that do not
contribute significantly to the analysis.
 Methods of Feature Selection:
o Filter Methods: Uses statistical tests (e.g., correlation) to select
features.
o Wrapper Methods: Uses models to evaluate which features
contribute the most to prediction.
 o Embedded Methods: Feature selection is integrated into the
model building process (e.g., decision trees).
 Example: If you have data on height, weight, age, and income, and you
find that age and income do not influence the target variable, you can
remove them.
Advantages:
 Reduces overfitting by eliminating irrelevant data.
 Improves model accuracy and interpretability.

3. Principal Component Analysis (PCA)

Definition:
PCA is a statistical technique used to reduce the dimensionality of the data
by transforming the data into a smaller set of uncorrelated variables,
called principal components.
Key Points:
 Purpose: To reduce the number of features while retaining most of the
variance in the data.
 How It Works:
o It finds new axes (principal components) that capture the most
variance in the data.
o The original features are transformed into a smaller set of
components that summarize the data.
 Example: If you have a dataset with 10 features, PCA might reduce it to
2 or 3 components while keeping most of the data's information.
Advantages:
 Reduces dimensionality, making data easier to visualize and analyze.
 Removes correlations between features, making the dataset more
manageable.

Summary:
 Sampling: Selects a representative subset of the data.
 Feature Selection: Removes irrelevant or redundant features.
 PCA: Transforms data into a smaller set of uncorrelated components
while retaining most of the data's variance.
These techniques help make the data easier to handle, process, and
analyze without losing important information.
Q.12] Discuss the need for data pre-processing and any 2 techniques used.
ANS : Need for Data Pre-processing

Data pre-processing is essential because raw data often contains errors,

inconsistencies, and noise. To build accurate models, we need to clean and
transform the data. It helps to:

1. Handle Missing Data:

o Missing or incomplete data can lead to biased or incorrect
results. Pre-processing helps fill or remove such data.
2. Improve Data Quality:
o Raw data can have inaccuracies, duplicates, or irrelevant
information. Pre-processing ensures better quality data for
analysis.
3. Scale Data for Consistency:
o Features may have different units (e.g., weight in kg and height in
cm). Pre-processing scales them to a common range, ensuring
equal importance.
4. Feature Engineering:
o Raw data might not include all the necessary information. Pre-
processing can create new features that are more informative
for the model.
5. Prepare for Modeling:
o Many algorithms need the data in a specific format or range.
Pre-processing ensures that data meets these requirements.

Two Common Techniques Used in Data Pre-processing:

1. Handling Missing Data:
o Description: Missing values in the dataset can affect the model’s
accuracy.
o Technique:
 Imputation: Fill in missing values with the mean, median, or
mode of the column.
 Removal: Delete rows or columns that have missing values.
o Example: If a column has missing values for some students’
scores, you can replace them with the average score or remove
those rows entirely.
 2. Normalization/Standardization:
o Description: Features in a dataset might have different scales
(e.g., one feature in thousands and another in fractions).
o Technique:
 Normalization: Rescale data between 0 and 1.
 Standardization: Rescale data so that the mean is 0 and the
standard deviation is 1.
o Example: If one column is salary (in thousands) and another is
age, normalization or standardization helps ensure both features
are on the same scale for modeling.

Data pre-processing is a crucial step to ensure the data is clean,

consistent, and ready for building accurate models.
Q.13] What is data transformation? Why it is needed? Explain at least 3
techniques.
ANS : Data Transformation
Data transformation is the process of changing the format, structure, or
values of data to make it more suitable for analysis or modeling. It is a
crucial step in the data preprocessing phase, especially when data is raw,
inconsistent, or incomplete.
Why Data Transformation is Needed:
1. Consistency: Ensures that the data is in a consistent format, making it
easier to analyze.
2. Improved Accuracy: Transforming data helps to remove errors and
improve the quality of analysis.
3. Compatibility: Makes data compatible with algorithms or tools that
require specific formats or scales.
4. Feature Engineering: Helps in creating new features or modifying
existing ones for better model performance.

Three Common Data Transformation Techniques:

1. Normalization:
o What it is: Scaling the data so that all features have a similar
range, often between 0 and 1.
o Why it's needed: Prevents variables with larger values from
dominating the analysis or model.
o Example: If a dataset has one feature in range 1-1000 and
another in 0-1, normalization makes both ranges similar.
o How it's done:
 Formula: Normalized Value = (X - min(X)) / (max(X) -
min(X))
2. Standardization (Z-Score Normalization):
o What it is: Transforming the data so that it has a mean of 0 and a
standard deviation of 1.
o Why it's needed: Useful for algorithms that assume data is
normally distributed (like linear regression or SVM).
o Example: If a feature has values like 50, 100, 150, standardization
converts it to a new scale (e.g., -1.23, 0, 1.23).
o How it's done:
 Formula: Z = (X - mean(X)) / std(X)
 3. Categorical Encoding:
o What it is: Converting categorical data (like "Yes", "No" or "Red",
"Blue") into numerical values.
o Why it's needed: Many algorithms cannot handle categorical data
directly, so it needs to be transformed into numerical format.
o Example:
 Convert "Red" = 1, "Blue" = 2, "Green" = 3.
 Or using one-hot encoding, where each category is turned
into a separate binary column (e.g., "Red" = [1, 0, 0], "Blue" =
[0, 1, 0], "Green" = [0, 0, 1]).

These techniques make the data ready for analysis, ensuring better
accuracy and efficiency in models.
Q.14] What is data reduction? Explain Dimensionality Reduction and Data
Compression.
ANS : Data Reduction

Data reduction refers to techniques that reduce the size of a dataset while
maintaining its essential information. The goal is to improve computational
efficiency, reduce storage needs, and simplify data analysis without losing
important details.

Types of Data Reduction:

1. Dimensionality Reduction
2. Data Compression

1. Dimensionality Reduction:
Dimensionality reduction aims to reduce the number of features (variables)
in a dataset, making it easier to analyze without losing key information.
Key Points:
 Purpose: Simplifies complex data by reducing the number of features.
 Techniques:
o Principal Component Analysis (PCA): Finds the most important
features by transforming the data into new variables (principal
components) that capture the most variance.
o Linear Discriminant Analysis (LDA): Reduces dimensions while
preserving class separability, often used in classification tasks.
 Advantages:
o Reduces overfitting.
o Speeds up model training and prediction.
o Makes visualization easier (e.g., reducing data to 2D or 3D).
 Example:
If you have a dataset with 100 features, dimensionality reduction might
reduce it to 5 features while retaining the most important information.
2. Data Compression:
Data compression reduces the size of the dataset by encoding data more
efficiently, leading to reduced storage and faster processing.
Key Points:
 Purpose: Minimizes data storage requirements and speeds up
transmission.
 Techniques:
o Lossless Compression: Compresses data without losing any
information (e.g., ZIP files).
o Lossy Compression: Some information is discarded to achieve
higher compression rates (e.g., JPEG for images).
 Advantages:
o Saves storage space.
o Speeds up data transfer.
 Example:
Compressing an image file (e.g., PNG or JPEG) reduces its size,
making it easier to store and share while maintaining acceptable
quality.

Summary:
 Dimensionality Reduction simplifies complex data by reducing the
number of features, helping in better analysis and visualization.
 Data Compression reduces the size of data, helping in efficient
storage and faster processing while maintaining important
information or quality.
Q.15] Define dirty data. What are the reasons of dirty data.
ANS : Dirty Data

Dirty data refers to inaccurate, incomplete, or inconsistent data that can

lead to incorrect analysis or decisions. It can cause problems during data
processing, cleaning, and analysis.

Reasons for Dirty Data:

1. Missing Values:
o When data points are absent in a dataset, making it incomplete.
o Example: A customer’s age is missing in the dataset.
2. Duplicates:
o Repeated data entries, which can distort analysis.
o Example: A customer’s record appears twice with the same
information.
3. Inconsistent Data Formats:
o Data stored in different formats, making it hard to interpret.
o Example: Dates recorded as dd/mm/yyyy in some rows and
mm/dd/yyyy in others.
4. Incorrect Data:
o Data that doesn't match real-world values due to human or
system errors.
o Example: A person’s age listed as 150 years.
5. Outliers:
o Data points that are significantly different from others and may
be errors.
o Example: A salary of $1 million when the average salary is
$50,000.
6. Inconsistent Naming or Coding:
o Variations in how categories are named or coded.
o Example: One row lists "New York" while another lists "NY,"
leading to inconsistency.

Summary:
Dirty data can arise due to missing values, duplicates, inconsistent
formats, incorrect entries, outliers, and naming inconsistencies. It needs to
be cleaned to ensure accurate analysis and decision-making.
Q.16] Explain the working of binning with suitable example.
ANS : Binning

Binning is a data preprocessing technique used to group a set of

continuous values into discrete intervals or "bins." It helps in simplifying
the data, making it easier to analyze and model. The main idea is to convert
numerical data into categorical data.

Working of Binning:
1. Data Sorting:
o Sort the data in increasing or decreasing order.
o Example: Given test scores [45, 87, 34, 92, 56, 71], you first sort
them in increasing order: [34, 45, 56, 71, 87, 92].
2. Define Bins:
o Divide the data range into intervals (bins). The bin edges are
predefined.
o Example: If you have scores ranging from 30 to 100, you could
define the bins:
 30–50
 51–70
 71–100
3. Assign Data to Bins:
o Assign each data point to the appropriate bin based on its value.
o Example:
 34 → Bin 30–50
 45 → Bin 30–50
 56 → Bin 51–70
 71 → Bin 71–100
 87 → Bin 71–100
 92 → Bin 71–100
4. Labeling or Aggregation:
o Assign a label to each bin, or perform aggregation (e.g., calculate
the average) for data in the bin.
o Example: After binning, the labels could be:
 Bin 30–50 → Label "Low"
 Bin 51–70 → Label "Medium"
 Bin 71–100 → Label "High"
 5. Use of Binned Data:
o The binned data is now easier to analyze and model. It can be
used in various machine learning algorithms or for simpler
decision-making.
o Example: You can now analyze the "Low," "Medium," and "High"
categories to understand how performance varies across these
bins.

Example:
Imagine you are analyzing the ages of a group of people to categorize them
into age groups:
 Data: [23, 45, 18, 35, 50, 61, 29, 41]
 Define bins:
o 18–30 → "Young"
o 31–50 → "Middle-aged"
o 51+ → "Old"
 After binning, the data would look like:
o 23 → "Young"
o 45 → "Middle-aged"
o 18 → "Young"
o 35 → "Middle-aged"
o 50 → "Middle-aged"
o 61 → "Old"
o 29 → "Young"
o 41 → "Middle-aged"

Binning simplifies the dataset into categories, helping to spot patterns and
trends.
Q.17] What is Bivariate analysis? Why it is important. Discuss the different
types with example?
ANS : Bivariate Analysis

Bivariate analysis is the statistical analysis that examines the relationship

between two variables. It helps to understand how two variables are
related or if one affects the other. This analysis is crucial for exploring
correlations and identifying patterns that could inform further analysis or
decision-making.

Importance of Bivariate Analysis:

1. Identifies Relationships:
o Helps determine if two variables are related or not.
2. Predictive Insights:
o Helps predict the behavior of one variable based on the other.
3. Informs Decision-Making:
o Provides insights for strategic decisions in business, health,
education, etc.

Types of Bivariate Analysis:

1. Correlation Analysis:
o Measures the strength and direction of the relationship between
two continuous variables.
o Example:
 Examining the correlation between "hours studied" and
"exam score". If the correlation is strong and positive, it
indicates that as study hours increase, exam scores likely
increase.
o Method: Pearson correlation coefficient (r).
2. Regression Analysis:
o Determines how one variable (dependent) changes in response
to another (independent). It helps predict the dependent variable.
o Example:
 Predicting "house prices" based on "square footage" using
linear regression.
o Method: Linear regression model (Y = a + bX).
 3. Scatter Plots:
o Visual representation of the relationship between two continuous
variables.
o Example:
 A scatter plot showing how "height" and "weight" of
individuals are related.
o Method: Plotting data points on a graph to see if a pattern or
trend exists.
4. Chi-Square Test (for Categorical Variables):
o Tests the relationship between two categorical variables to see if
they are independent or related.
o Example:
 Testing if "gender" is related to "voting preference"
(male/female vs. party A/party B).
o Method: Chi-square statistic.
5. Cross-Tabulation (Contingency Tables):
o Displays the frequency distribution of variables to compare their
relationships.
o Example:
 A table showing the relationship between "education level"
(high school, college, graduate) and "employment status"
(employed, unemployed).
o Method: Using a pivot table or crosstab in Python (Pandas).

Example:
 Regression: Predicting income based on education level (years of
education).
 Chi-Square Test: Checking if there’s a relationship between smoking
habits (smoker, non-smoker) and gender (male, female).

In summary, bivariate analysis helps understand the interaction between

two variables, whether continuous or categorical, and provides insights for
decision-making or further statistical modeling.
Q.18] How Mean, Median and Mode are used during data cleaning.
ANS : Mean, Median, and Mode in Data Cleaning
Mean, median, and mode are important statistical measures used during
data cleaning to handle missing or incorrect values. They help in filling or
replacing missing data and ensuring the dataset is accurate for analysis.
1. Mean (Average):
 Used to Replace Missing Numerical Data:
o The mean is the sum of all values divided by the number of
values. It is commonly used to fill missing values in numerical
data when there is no specific pattern.
o Example: If the salary of one employee is missing in a dataset,
you can replace it with the average salary of all employees.
o When to Use: If the data is normally distributed and doesn't
contain outliers.
Formula:
Mean = (Sum of all values) / (Number of values)
2. Median:
 Used to Handle Skewed Data or Outliers:
o The median is the middle value when the data is sorted. It is less
affected by extreme values or outliers than the mean.
o When to Use: If the dataset has outliers or is skewed (not
symmetrical).
o Example: In income data, the median is often preferred because
a few very high incomes can distort the mean.
Steps to Find Median:
o Sort the values, then pick the middle one. If the count of values is
even, the median is the average of the two middle values.
3. Mode:
 Used to Handle Categorical Data:
o The mode is the most frequent value in a dataset. It is
particularly useful for categorical data to replace missing values
with the most common category.
o Example: In a dataset of customer preferences (e.g., "Red",
"Blue", "Green"), if the color preference for one customer is
missing, you can replace it with the most frequent color (mode).
o When to Use: When dealing with categorical data or when you
want to replace missing values with the most common option.
Practical Example:

Imagine a dataset of customer ages:

 If some ages are missing, you can:
o Use the mean if the ages are normally distributed.
o Use the median if the ages have a few extreme values (outliers).
o Use the mode if the dataset contains categorical data (e.g., city
names) or you want the most common value for replacing
missing entries.

By using these measures, data cleaning becomes more efficient and helps
ensure accurate analysis results.
Q.19] Explain the need of data cleaning and different methods.
ANS : Need for Data Cleaning

Data cleaning is the process of identifying and correcting errors or

inconsistencies in the dataset to ensure it is accurate, complete, and ready
for analysis. Clean data leads to more reliable results, better decision-
making, and improved performance of machine learning models.

Reasons for Data Cleaning:

1. Accuracy Improvement:
o Raw data can contain errors like typos, incorrect values, or
inconsistencies. Cleaning ensures data accuracy.
o Example: A date written as "13/31/2025" (invalid date) can be
fixed.
2. Handling Missing Data:
o Missing values can affect analysis or model performance.
Cleaning helps in dealing with these gaps.
o Example: A missing income field in a dataset might be replaced
with the median or mean value.
3. Consistency Across Data:
o Inconsistent formats (like different date formats) can create
confusion. Cleaning ensures uniformity.
o Example: Dates should be in the same format (e.g.,
"DD/MM/YYYY").
4. Removing Duplicates:
o Duplicate entries can distort analysis results. Cleaning removes
any redundant data.
o Example: Multiple rows for the same customer need to be
merged or deleted.
5. Outlier Detection:
o Outliers (extreme values) can skew analysis. Cleaning helps to
identify and handle outliers.
o Example: A salary of $1,000,000 in a dataset of $40,000 average
salaries may need attention.
6. Improving Data Quality for Models:
o Models perform better with high-quality data. Cleaning ensures
the data is reliable, improving model predictions.
Methods of Data Cleaning:
1. Handling Missing Values:
o Imputation: Replace missing values with the mean, median, or
mode.
 Example: Replace missing age data with the average age of
all individuals.
o Deletion: Remove rows or columns with missing data if they are
not significant.
 Example: Drop rows where critical data like income or
score is missing.
2. Removing Duplicates:
o Identify and remove duplicate rows to avoid overrepresentation
of data.
o Example: Use df.drop_duplicates() in Python to remove repeated
rows.
3. Correcting Data Types:
o Ensure that data types (numeric, string, datetime) are correct for
each column.
o Example: Convert a "Date" column to the proper datetime type.
4. Handling Outliers:
o Identification: Use statistical methods or visualizations (e.g., box
plots) to find outliers.
o Capping or Removal: Replace or remove extreme values that
may skew analysis.
o Example: Remove salary entries that are too high or low
compared to the average.
5. Standardizing Data Formats:
o Ensure consistency in formats (e.g., dates in "YYYY-MM-DD",
phone numbers with the same format).
o Example: Convert all dates to the same format: "01/15/2025" to
"2025-01-15".
6. Encoding Categorical Variables:
o Convert categorical variables (like "Male" and "Female") to
numeric codes (0 and 1).
o Example: Convert "Yes"/"No" to 1/0 for better compatibility in
models.
Q.20] Explain the working of PCA. Its application .
ANS : Principal Component Analysis (PCA)

Principal Component Analysis (PCA) is a technique used to reduce the

dimensionality of a dataset while retaining as much variability
(information) as possible. It transforms the data into a new set of variables
called "principal components," which are uncorrelated and ordered by
importance.

Working of PCA:
1. Standardize the Data:
o Before applying PCA, it’s important to standardize the data to
ensure that each feature contributes equally.
o Example: Subtract the mean and divide by the standard deviation
for each feature.
2. Calculate the Covariance Matrix:
o The covariance matrix shows how the features vary with respect
to each other. If the features are highly correlated, PCA will
capture this correlation.
o Example: A covariance matrix helps identify relationships
between features (like height and weight).
3. Compute Eigenvalues and Eigenvectors:
o Eigenvectors represent the directions of the new axes (principal
components), and eigenvalues show the amount of variance
along each eigenvector.
o Higher eigenvalue means more variance (information).
4. Sort Eigenvalues and Select Principal Components:
o Sort the eigenvalues in descending order and choose the top ‘k’
components based on the largest eigenvalues.
o Example: If you want to reduce dimensions from 5 to 2, you pick
the 2 components with the highest eigenvalues.
5. Transform the Data:
o Multiply the original data by the selected eigenvectors to obtain
the new dataset with reduced dimensions.
o Example: The transformed data will have fewer features but still
capture most of the information.
Applications of PCA:
1. Dimensionality Reduction:
o PCA reduces the number of features in large datasets, making it
easier to visualize and analyze.
o Example: Reducing a dataset with 100 features to 2 or 3 for
visualization.
2. Noise Reduction:
o By discarding components with small eigenvalues (which
represent less variance or noise), PCA helps in removing noise
from the data.
o Example: Reducing fluctuations in sensor data.
3. Data Visualization:
o PCA can be used to project high-dimensional data onto 2D or 3D
plots, making it easier to visualize.
o Example: Visualizing customer data based on purchasing
patterns in 2D.
4. Feature Extraction:
o PCA can identify new, meaningful features (principal
components) that explain the most variance in the data.
o Example: In image processing, PCA can identify key features of
an image for recognition tasks.
5. Compression:
o PCA is often used for compressing data without losing significant
information, especially in image and speech processing.
o Example: Image compression in facial recognition systems.

PCA is widely used in fields like machine learning, computer vision,

finance, and bioinformatics to make data more manageable and
interpretable.
Q.21] Explain the data discretization. What are two methods?
ANS : Data Discretization

Data discretization is the process of converting continuous data (numerical

values) into discrete categories or intervals. This helps simplify the data
and make it easier to analyze, especially in classification problems where
the model needs categorical data.

Key Steps in Data Discretization:

1. Identify Continuous Data:
o Find columns with continuous values (like age, income,
temperature) in your dataset.
2. Choose Discretization Method:
o Decide how to divide the continuous data into intervals or bins.
3. Convert Data:
o Use discretization techniques to group continuous data into
categories (bins).
4. Use Categorical Data for Analysis:
o Discrete data can now be used in algorithms that require
categorical inputs, like decision trees.
Two Methods of Data Discretization:
1. Equal Width Binning:
o Description: Divide the range of the continuous variable into
intervals of equal size.
o Example: If the data ranges from 0 to 100 and you want 5 bins,
each bin will cover an interval of 20 (0-20, 21-40, 41-60, 61-80,
81-100).
2. Equal Frequency Binning (Quantile Binning):
o Description: Divide the data so that each bin contains
approximately the same number of data points.
o Example: If you have 100 data points and want 4 bins, each bin
will contain 25 data points, even if the bin widths differ.
Summary:
 Equal Width Binning creates bins with equal ranges, while Equal
Frequency Binning creates bins with equal data point counts.
Both methods help in simplifying continuous data for further analysis
or machine learning.