CST322 Module -2 PYQ Answers

📝 Data Analytics Life Cycle Explanation (8 marks)

The Data Analytics Life Cycle is designed specifically for Big Data problems
and data science projects.
The lifecycle has six phases, and project work can occur in several phases at
once.

Phases of Data Analytics Life Cycle:

1. Discovery
 The team learns the business domain, including relevant history such
as whether the organization or business unit has attempted similar
projects in the past from which they can learn.
 The team assesses the resources available to support the project in
terms of people, technology, time, and data.
 Important activities include framing the business problem as an
analytics challenge and formulating initial hypotheses (IHs).
2. Data Preparation
 Requires the presence of an analytic sandbox.
 Execute extract, load, and transform (ELT) or extract, transform and
load (ETL).
 Data should be transformed so the team can work with it and analyse
it.
 The team familiarizes itself with the data and conditions the data.
3. Model Planning
 The team determines the methods, techniques, and workflow for
model building.
 Explores data to learn relationships between variables.
 Selects key variables and suitable models.
4. Model Building
 Develop datasets for testing, training, and production.
 Build and execute models.
 Consider whether existing tools are sufficient or more robust
environment is needed.
5. Communicate Results
 Determine whether results are success or failure based on criteria.
 Identify key findings and quantify business value.

2
  Develop a narrative to convey findings to stakeholders.
6. Operationalize
 Deliver final reports, briefings, code, and technical documents.
 Run pilot project to implement models in production environment.

Diagram also.

📝 Data Analytical Process Model (8 marks)

Step 1: Identify Business Problem

 In the first step thorough definition of the business problem to be
addressed is needed
 Requires close collaboration between data scientist and business
experts
 Both parties need to agree on key concepts like customer, transaction,
churn or fraud
Step 2: Identify Data Sources
 All source data that could be of potential interest need to be identified

3
  “The more data, the better”
 Model will later decide which data are relevant
Step 3: Select, Clean and Transform Data
 Select relevant data
 Clean data to remove errors and incompleteness
 Transform data into suitable format
Step 4: Model Building / Analysis
 Analytical model is estimated on pre-processed and transformed data
 Based on business objective, appropriate analytical technique is
selected and implemented
Step 5: Interpret, Evaluate and Deploy
 Results are interpreted and evaluated by business expert
 Results may be clusters, patterns or relations
 Find unknown yet interesting and actionable patterns
 Once validated, model is put into production
 Used as an analytics application

📝 Sampling (3/7/8 Marks)

Importance:
 A key requirement for a good sample is that it should be
representative of the future customers on which the analytical model
will be run.
 Sampling helps in building models using relevant and manageable
data.

4
  Sampling bias should be avoided as much as possible, since it affects
model accuracy.
 It ensures that the analytical model gives reliable and meaningful
results.
Sample vs Population:
 Population: Entire dataset or all customers
 Sample: Subset of past customer data used to build an analytical
model
Types of Sampling:
1. Random Sampling
o Sample is selected randomly from the population
o Example: Selecting customers randomly from a database
2. Stratified Sampling
o Sample is taken according to predefined strata
o Example: Dividing customers into groups (age/income) and
sampling from each group.

📝 MISSING VALUE HANDLING (6/7 MARKS)

Missing values occur due to:
 Information can be nonapplicable
 Information can be undisclosed
 Error during merging
 Some analytical techniques can directly deal with missing values, while
others need preprocessing.
Methods to deal with missing values:
1. Replace (Impute)
 Replacing the missing value with a known value

5
  Example: Replace missing income with average income
2. Delete
 Delete observations or variables with lots of missing values
 Assumes data is missing at random
 Example: Remove records with many missing fields
3. Keep
 Missing values can be meaningful
 Example: Income not disclosed may indicate unemployed customer

📝 ANALYTICAL MODEL REQUIREMENTS – 3/6/7

An analytical model should satisfy the following requirements:

1. Business Relevance
 The analytical model should actually solve the business problem.
 The problem must be appropriately defined, qualified, and agreed
upon.
2. Statistical Performance
 The model should have statistical significance and predictive power.
3. Interpretability
 Refers to understanding the patterns captured by the model.
 Depends on the business user’s knowledge.
4. Justifiability
 The model should correspond to prior business knowledge and
intuition.
5. Operational Efficiency

6
  Efforts needed to collect data, preprocess, evaluate, and deploy the
model.
 Includes monitoring, back testing, and re-estimation.
6. Economic Cost
 Costs of data collection, preprocessing, analysis, and implementation.
 Includes software, human, and computing resources.

7. Regulation and Legislation

 Model should comply with local and international regulations.
 Includes privacy and legal constraints.

📝 OUTLIER DETECTION

Outliers are extreme observations that are very dissimilar to the rest of the
population.
 Can be valid observations (e.g., very high salary)
 Can be invalid observations (e.g., age = 300)
Ways to detect outliers
 Minimum and Maximum Check
 Identify extreme values
 Visual Mechanisms
 Histograms
 Box plots
 Box Plot Method
 Based on quartiles (Q1, Median, Q3)
 Outliers lie beyond 1.5 × IQR
 Z-score Method

7
  Measures how many standard deviations an observation lies
from the mean
Treatment Methods:
1. Truncation / Capping / Winsorizing
 Set upper and lower limits
 Values beyond limits are adjusted
2. Using Z-score or IQR limits
 Upper/lower limit based on statistical measures
3. Sigmoid Transformation
 Transform values into a range between 0 and 1

📝 DIMENSIONALITY REDUCTION (3/6/7 MARKS)

Dimensionality reduction is a data reduction strategy that obtains a reduced

representation of the data set that is much smaller in volume, yet produces
the same (or almost the same) analytical results.
 It applies data encoding schemes to obtain a compressed
representation of the original data.
Why it is important in data analysis?
 Produces a reduced representation of the dataset
 Reduces data volume while maintaining analytical results
 Helps in efficient data analysis
Types of data reduction Techniques:
1. Data Compression Techniques
o Example: Wavelet transforms, Principal Components Analysis
(PCA)
o Transform or project data into a smaller space

8
 2. Attribute Subset Selection
o Removing irrelevant attributes
3. Attribute Construction
o Deriving a small set of useful attributes from original data
Algorithms to implement dimensionality reduction
1. Wavelet Transform
 A linear signal processing technique
 Transforms data vector into wavelet coefficients
 Uses hierarchical pyramid algorithm
2. Principal Components Analysis (PCA)
 Searches for k orthogonal vectors (k ≤ n)
 Projects data into a smaller space
 Combines attributes into a smaller set

📝 SOURCES OF DATA FOR DATA COLLECTION:

Data can be collected from various sources for analytics.

Types of Data Sources:
1. Transactional Data
 Structured, low-level, detailed information
 Captures characteristics of customer transactions
 Examples: purchase, claim, cash transfer, credit card payment
 Stored in OLTP relational databases
2. Unstructured Data
 Data embedded in text documents or multimedia content
 Examples: emails, web pages, claim forms

9
3. Qualitative / Expert-Based Data
 Based on expert knowledge and experience
4. Data Poolers
 Organizations that gather and sell data
 Examples: Dun & Bradstreet, Bureau Van Dijck, Thomson Reuters
5. Publicly Available Data
 Includes macroeconomic data
 Examples: GDP, inflation, unemployment
 Helps analyse impact of economic conditions

📝 APPLICATIONS OF DATA ANALYTICS (3 MARKS)

 Customer Lifetime Value (CLV) Analysis

 Analyse customer value over time
 Fraud Detection
 Identify fraudster or not
 Churn Prediction
 Predict whether a customer will leave
 Marketing Analytics / Campaign Management
 Improve marketing strategies
 Web Analytics
 Analyse web visits or sessions
 Risk Management
 Used in areas like credit risk modelling

📝 DIFFERENT TYPES OF DATA ELEMENTS (3 MARKS)

10
  Defined on an interval (limited or unlimited)
 Takes any value within a range
 Examples: income, sales, RFM (recency, frequency, monetary)
Categorical Data (3/6 marks)
Categorical data takes a limited set of values.
Types:
1. Nominal
o No meaningful ordering
o Example: marital status, profession
2. Ordinal
o Has meaningful ordering
o Example: credit rating, age groups
3. Binary
o Takes only two values
o Example: gender, employment status

📝 DIFFERENCE BETWEEN MIN–MAX NORMALIZATION AND DECIMAL

SCALING (3 MARKS)
Min–Max Normalization
 Transforms data into a specific range (usually 0 to 1)
 Formula:
𝒗 − 𝒎𝒊𝒏
𝒗 =
𝒎𝒂𝒙 − 𝒎𝒊𝒏

11
  Preserves relative relationships between data values
Normalization by Decimal Scaling
 Moves the decimal point of values
 Formula:
𝒗
𝒗 =
𝟏𝟎𝒋
,where j is chosen such that max(|v′|) < 1
 Depends on maximum absolute value

 Min–Max
 Feature  Decimal Scaling
Normalization
 Depends on
 Range  Fixed (0 to 1)
scaling
 Method  Uses min and max  Uses power of 10
 Sensitivity  Affected by outliers  Less sensitive
 Transformation  Linear scaling  Decimal shifting

📝 Box Plots (7 marks)

A box plot is a visual mechanism used for outlier detection and data
distribution.
 It represents three key quartiles:
o Q1 (First quartile) → 25% of observa ons have lower value
o Median (Q2) → 50% of observa ons have lower value
o Q3 (Third quartile) → 75% of observa ons have lower value
 These quartiles are represented as a box.
 The minimum and maximum values are added unless they are too far
away.

12
  Outliers are identified as values beyond 1.5 × IQR
 Where:
𝐼𝑄𝑅 = 𝑄3 − 𝑄1
Importance of Box Plot
 Helps in visual data exploration
 Identifies outliers easily
 Shows data distribution (spread, median, quartiles)
 Useful for comparing datasets
 Helps in detecting extreme observations
Problem: (7 marks):
Draw a boxplot for the following data set of scores: 76 79 76 74 75 71 85
82 82 79 81

📝 STEPS IN DATA PREPROCESSING (7 MARKS)

1. Identify Data Sources

 Identify all source data that could be of potential interest
 “The more data, the better the analytical model”
2. Select the Data
 Select relevant data from available sources
 The analytical model later decides which data are relevant

3. Clean the Data

 Ensure that there is no overlap or mistake
 Make sure data is not incomplete

13
  Helps to fix or eliminate errors before analysis
4. Transform the Data
 Convert data into suitable format
 Required before applying analytical techniques
5. Standardizing Data
 Scaling variables to similar ranges
 Example: bringing values to comparable scale
6. Categorization (Binning)
 Grouping or classification of data
 Methods:
o Equal interval binning
o Equal frequency binning
7. Outlier Detection
 Detect extreme observations
 Methods:
o Box plot
o Histogram
o Z-score

📝 Consider the following sorted data for price (in dollars) 4, 8, 15, 21,
21, 24, 25, 28, 34. Illustrate smoothing by bin mean. (3 marks)

📝 Given the dataset: 10, 20, 30, 40. Apply Min-Max Scaling to
transform the values to a range of 0 to 1. (3 marks)

14
15