UNIT III
Big Data Analytics
Life Cycle
Comprehensive Study Notes | Exam-Focused | PYQ Included
Data Science • TE / VI • SPPU Syllabus
■ Topics ■ Hours ■ Exam Weight ■ PYQs Covered
Big Data Sources, BI vs DS
High (8-9 marks 8 Questions
Data Analytic Lifecycle 07 Hours
per question) from 3 papers
Phases 1-6
1. Introduction to Big Data
1.1 What is Big Data?
Big Data: Extremely large datasets that cannot be processed or managed by traditional data processing
tools within an acceptable time frame. It requires advanced technologies to store, manage, and analyze.
The 5 V's of Big Data (Key Characteristics)
V Name Description Example
V1 Volume Massive amount of data generated Petabytes of social media data
V2 Velocity Speed at which data is generated Twitter: 6,000 tweets/second
V3 Variety Different types/formats of data Text, images, video, sensor data
V4 Veracity Accuracy and trustworthiness of data Noisy or uncertain data
V5 Value Usefulness of data after processing Business insights from raw logs
1.2 Sources of Big Data
• Social Media: Facebook posts, tweets, Instagram photos, YouTube videos
• Machine / Sensor Data: IoT devices, RFID, GPS trackers, industrial sensors
• Transaction Data: E-commerce orders, banking transactions, POS data
• Web Data: Clickstreams, server logs, search queries
• Scientific Data: Genomics, astronomy, climate simulations
• Healthcare Data: Electronic health records, medical imaging, wearables
• Enterprise Data: ERP systems, CRM databases, supply chain records
■ PYQ: Explain any three sources of Big Data. Differentiate BI versus Data Science. [8]
1.3 Business Intelligence (BI) vs Data Science
Aspect Business Intelligence (BI) Data Science
Focus Reporting & dashboards Prediction & discovery
Data Type Structured (SQL, Excel) Structured + Unstructured
Techniques OLAP, SQL queries, KPIs ML, statistics, NLP, AI
Output Reports, charts, summaries Predictive models, insights
Users Business analysts, managers Data scientists, engineers
Time Horizon Historical (what happened?) Future (what will happen?)
Tools Tableau, Power BI, QlikView Python, R, Spark, TensorFlow
Skill Level Moderate Advanced (stats + coding)
■ Tip: BI answers 'What happened?'; Data Science answers 'What will happen?' — remember this
distinction for exams!
2. Data Analytic Lifecycle
The Data Analytic Lifecycle provides a structured framework for managing and executing big data
analytics projects from inception to deployment. It consists of 6 iterative phases.
Phase 1 Phase 2 Phase 3 Phase 4 Phase 5 Phase 6
Discovery Data Prep Model Planning Model Building Communication Operationalize
Data Analytic Lifecycle – 6 Phases
■ Iterative process – phases can loop back
■■ Important: The lifecycle is ITERATIVE — teams can revisit earlier phases based on findings. This is
a key exam point!
2.1 Key Stakeholders in an Analytics Project
• Business Stakeholders: Define objectives, approve resources, evaluate final results
• Project Manager / Sponsor: Oversee timelines, budgets, and team coordination
• Data Scientist: Build and validate models, perform statistical analysis
• Data Engineer: Build data pipelines, ensure data quality and availability
• Business Analyst: Translate business needs into analytics requirements
• DBA / IT Team: Maintain infrastructure, databases, and security
Stakeholder Key Output Expected at Conclusion
Business Sponsor ROI improvement, actionable insights, cost savings
Data Scientist Working predictive model, model evaluation report
Business Analyst Business requirement docs, KPI dashboards
IT/DBA Deployed, scalable, secure data pipeline
End Users Easy-to-use tool or dashboard with reliable predictions
■ PYQ: List out different stakeholders of an analytics project. What do they usually expect at the
conclusion (key outputs) of a project? [8/9]
3. The Six Phases in Detail
Phase 1: Discovery
Goal: Understand the problem domain, define the objectives, and assess available resources.
Key Activities:
• Learn the business domain and existing analytics solutions
• Identify the problem to solve — translate business problem into analytics problem
• Frame the initial hypotheses to be tested
• Identify data sources needed for the project
• Assess resources: people, technology, time, and budget
• Conduct interviews with SMEs (Subject Matter Experts)
• Create a project charter with scope, goals, timeline
Key Outputs:
• Problem statement document
• List of hypotheses and analytics questions
• Resource requirements and availability report
• Initial data sources inventory
■ Tip: Think of Discovery as 'understanding WHAT you are solving and WHY before touching any data.'
Phase 2: Data Preparation
Goal: Collect, clean, transform, and prepare data for modelling. This is the most time-consuming phase
(~60-80% of project time).
Key Activities:
• Data collection from identified sources (databases, APIs, flat files, web scraping)
• Data cleaning: handling missing values, removing duplicates, correcting errors
• Data integration: merging data from multiple sources
• Data transformation: normalization, encoding categorical variables, feature extraction
• Exploratory Data Analysis (EDA): summary statistics, histograms, correlation matrices
• Feature engineering: creating new meaningful features from raw data
Analytics Sandbox
Analytics Sandbox: A dedicated environment (separate from production) where data scientists can
freely explore, experiment, and manipulate data without affecting live systems. Provides safe space for
trial-and-error analysis.
ETLT Process
ETLT (Extract, Transform, Load, Transform): A variant of the ETL process used in big data. Raw data
is first Extracted from sources, then partially Transformed, Loaded into the sandbox/warehouse, and
Transformed again for analytics. The second T allows in-database transformations for speed.
Step Description
Extract (E) Pull data from source systems (databases, APIs, logs, sensors)
Transform 1 (T) Basic cleaning, format standardization, deduplication
Load (L) Load into analytics sandbox / data warehouse / Hadoop/Spark
Transform 2 (T) In-depth transformations, feature engineering, aggregations
■ PYQ: What is the Data Preparation phase in Data Analytics Lifecycle? What is the Analytics Sandbox
and ETLT process in this phase? [8]
Phase 3: Model Planning
Goal: Identify which methods, tools, and techniques to use for building the analytical model.
Key Activities:
• Explore relationships among variables in the data
• Choose the modelling technique (regression, classification, clustering, etc.)
• Select appropriate tools for modelling
• Plan the model testing and validation approach
• Define model performance metrics (accuracy, RMSE, AUC-ROC, etc.)
Common Tools Used in Model Planning:
Category Tools
Statistical R, SAS, SPSS, MATLAB
Programming Python (scikit-learn, pandas, NumPy)
Big Data Apache Spark (MLlib), Hadoop
Visualization Tableau, matplotlib, seaborn, ggplot2
SQL Analytics SQL Server, PostgreSQL, Hive
■ PYQ: List out the activities to be carried out in model planning and model building phase. What are
different tools used for these phases? [8]
Phase 4: Model Building
Goal: Develop, test, and validate the analytical model using prepared data.
Key Activities:
• Build the model on the training dataset
• Apply machine learning algorithms (regression, decision trees, neural networks, etc.)
• Perform model training and tune hyperparameters
• Validate using cross-validation or hold-out test sets
• Evaluate using metrics: Accuracy, Precision, Recall, F1-Score, RMSE, AUC
• Iterate to improve model performance
• Document model assumptions, parameters, and results
Linear Regression (Most Frequently Asked)
Linear Regression: A statistical method to model the linear relationship between a dependent variable
(Y) and one or more independent variables (X). Used for prediction of continuous values.
Type Formula Use Case
Simple Linear RegressionY = b0 + b1*X + e 1 predictor → 1 output
Y = b0 + b1*X1 + b2*X2 + ... + eMultiple predictors → 1 output
Multiple Linear Regression
Performance Evaluation of Linear Regression:
• MAE (Mean Absolute Error): Average of absolute differences between predicted and actual
• MSE (Mean Squared Error): Average of squared differences — penalizes large errors
• RMSE (Root Mean Squared Error): Square root of MSE — same unit as output variable
• R² (Coefficient of Determination): Proportion of variance explained by the model (0–1, higher =
better)
• Adjusted R²: R² adjusted for number of predictors — use for multiple regression
■ PYQ: What is Model Building? Elaborate this phase of data analytics with the help of a suitable
example. [8/9]
■ PYQ: What is linear regression, and what are its primary objectives? What is the difference between
simple linear regression and multiple linear regression? How do you evaluate the performance of linear
regression? [8]
Phase 5: Communicate Results
Goal: Present findings, insights, and recommendations to business stakeholders in a clear and actionable
manner.
Key Activities:
• Compare model results against success criteria defined in Phase 1
• Identify key findings and business insights
• Quantify business value — ROI, cost savings, revenue uplift
• Create visualizations: dashboards, charts, heat maps, infographics
• Prepare a final report and presentation for stakeholders
• Highlight model limitations and assumptions
• Recommend next steps for operationalizing the model
Types of Analytics Outputs Communicated:
• Descriptive Analytics: What happened? (historical summaries, reports)
• Diagnostic Analytics: Why did it happen? (root cause analysis, drill-down)
• Predictive Analytics: What will happen? (forecasts, scores, predictions)
• Prescriptive Analytics: What should we do? (optimization, recommendations)
■ PYQ: Explain Descriptive, Diagnostic, Predictive analytics. [8/9]
Type Question Answered Technique Example
Descriptive What happened? Reports, dashboards, charts Monthly sales summary
Diagnostic Why did it happen? Drill-down, correlations Why did sales drop?
Predictive What will happen? ML models, regression Next month's demand
Prescriptive What to do? Optimization, simulation Best discount to offer
Phase 6: Operationalize
Goal: Deploy the model into production and integrate it into business processes for real-world use.
Key Activities:
• Pilot deployment: Deploy model in a limited production environment for testing
• Model integration: Embed model into existing applications or workflows
• Performance monitoring: Track model accuracy over time — detect concept drift
• Model maintenance: Retrain or update the model with new data periodically
• Documentation: Create technical and user documentation for the deployed model
• Training: Train business users on how to interpret and use model outputs
• Measure actual business impact vs original goals
■■ Important: Operationalization is NOT just about deployment — it includes monitoring, maintenance,
and ensuring the model continues to deliver value over time.
4. Quick Revision – All 6 Phases at a Glance
Phase Name Key Keyword Main Output
1 Discovery Problem Framing Project charter, hypotheses
2 Data Preparation ETLT + Sandbox Clean, analysis-ready dataset
3 Model Planning Tool & Technique Chosen algorithm + test plan
4 Model Building Train & Validate Trained, validated model
5 Communicate Results Visualize & Report Stakeholder presentation, insights
6 Operationalize Deploy & Monitor Production model + monitoring
■ PYQ: List and explain the various activities involved in identifying potential data resources as a part of
discovery phase in Data Analytics Life Cycle. [9]
5. All PYQs with Answer Pointers
Q1. What is Model Building? Elaborate this phase of data analytics with the help of a
suitable example.
Marks: [8/9] | Answer Pointers:
• Define Model Building as Phase 4 of Data Analytic Lifecycle
• Explain: training dataset, algorithm selection, hyperparameter tuning
• Example: Predicting house prices using linear regression
• Explain training → testing → validation workflow
• Mention evaluation metrics: RMSE, R², Accuracy
• Mention tools: Python (scikit-learn), R, Spark MLlib
Q2. Explain any three sources of Big Data. Differentiate BI versus Data Science.
Marks: [8] | Answer Pointers:
• Sources: Social media, IoT sensors, transaction data, web logs, healthcare records
• Explain 3 in detail with examples
• BI vs DS table: focus, data type, output, tools, time horizon
• BI = descriptive/historical; DS = predictive/future
Q3. What are the three characteristics of Big Data and what are the main considerations in
processing Big Data?
Marks: [8] | Answer Pointers:
• 3 Vs: Volume, Velocity, Variety (or all 5Vs for full marks)
• Processing considerations: Storage (Hadoop HDFS), Processing (Spark), Scalability
• Data quality, security, integration challenges
• Use of distributed computing frameworks
Q4. What is the Data Preparation phase? What is the Analytics Sandbox and ETLT
process?
Marks: [8] | Answer Pointers:
• Define Phase 2: collection, cleaning, transformation, integration
• Analytics Sandbox: isolated experimentation environment
• ETLT: Extract → Transform → Load → Transform (explain each step)
• Mention tools: Python (pandas), Spark, Hive, SQL
Q5. List out different stakeholders. What do they usually expect at the conclusion (key
outputs)?
Marks: [8/9] | Answer Pointers:
• Stakeholders: Business sponsor, PM, Data Scientist, Data Engineer, Business Analyst, IT/DBA, End
User
• Expected outputs for each stakeholder (use table format)
• Emphasize: ROI, working model, actionable insights, reliable system
Q6. List activities in model planning and model building. What tools are used?
Marks: [8] | Answer Pointers:
• Model Planning: explore variables, select technique, choose tools, define metrics
• Model Building: train model, validate, tune, evaluate, iterate
• Tools: R, Python, SAS, Spark, Tableau, scikit-learn
Q7. What is linear regression? Differentiate SLR and MLR. How to evaluate performance?
Marks: [8] | Answer Pointers:
• Definition of linear regression
• SLR formula: Y = b0 + b1*X; one predictor
• MLR formula: Y = b0 + b1X1 + b2X2 + ...; multiple predictors
• Evaluation: MAE, MSE, RMSE, R², Adjusted R²
• Example: Predicting salary based on experience (SLR)
Q8. Explain Descriptive, Diagnostic, and Predictive analytics.
Marks: [8/9] | Answer Pointers:
• Descriptive: What happened? — reports, dashboards, historical data
• Diagnostic: Why did it happen? — drill-down, root cause analysis
• Predictive: What will happen? — ML models, regression, forecasting
• (Bonus: Prescriptive: What to do? — optimization, recommendations)
• Give real-world example for each type
6. Exam Tips, Tricks & Memory Shortcuts
Memory Aids
Mnemonic for 6 Phases: Don't Delay Making Money Communicating Opportunities → Discovery, Data
Prep, Model Planning, Model Building, Communicate, Operationalize
Mnemonic for 5 Vs: Very Vast Varieties Verify Value → Volume, Velocity, Variety, Veracity, Value
Exam Strategy
• For 8-mark questions: Write definition (2 lines) + 4-5 bullet points + 1 example + small diagram if
possible
• For 9-mark questions: Include comparison table OR detailed sub-point expansion
• Always draw the lifecycle diagram when asked about any phase — earns extra marks
• Mention tools by name (Python, Spark, Tableau) — shows technical knowledge
• Use real-world examples: e-commerce, healthcare, banking scenarios
• For BI vs DS: always write in table format — 8 clear points
• For analytics types: write all 4 types (Descriptive, Diagnostic, Predictive, Prescriptive) even if asked for
3
Common Mistakes to Avoid
■■ Important: Do NOT confuse 'Data Preparation' (Phase 2) with 'Discovery' (Phase 1). Discovery =
understanding problem; Data Prep = handling actual data.
■■ Important: Do NOT say the lifecycle is sequential/linear — always emphasize it is ITERATIVE.
■■ Important: Do NOT mix up SLR and MLR formulas — SLR has ONE X, MLR has MULTIPLE Xs.
7. Topic Summary – Key Takeaways
Big Data
• Defined by 5 Vs: Volume, Velocity, Variety, Veracity, Value
• Sources: Social media, IoT, transactions, web, healthcare, enterprise
• BI focuses on historical reporting; Data Science focuses on future predictions
Data Analytic Lifecycle
• 6 iterative phases: Discovery → Data Preparation → Model Planning → Model Building →
Communicate → Operationalize
• Phase 2 (Data Prep) is the most time-consuming (~60-80% of effort)
• Analytics Sandbox = safe experimentation environment; ETLT = Extract-Transform-Load-Transform
• Phase 4 includes linear regression: SLR (one predictor) and MLR (multiple predictors)
• Evaluation metrics: MAE, MSE, RMSE, R², Adjusted R²
• 4 analytics types: Descriptive, Diagnostic, Predictive, Prescriptive
• Operationalize = deploy + monitor + maintain the model in production
All the best for your exam! ■ Focus on the lifecycle phases, BI vs DS comparison, linear regression,
and stakeholder outputs — these are the most frequently asked topics.