Data Science - Complete Study Guide Examination Preparation

DATA SCIENCE
COMPLETE EXAMINATION STUDY GUIDE
Semester Exam Preparation | Total: 75 Marks
4 Units x 15 Marks Each + 15 Marks Common Questions

UNIT 1 Introduction to Data Science (Big Data, Web Scraping, Analysis vs

Reporting, AI & DS, Myths)

UNIT 2 Programming Tools: Matplotlib, NumPy, Scikit-learn, NLTK, Charts,

Files, Web Scraping

UNIT 3 Data Science Methodology: 10 Stages (Business Understanding to

Feedback)

UNIT 4 Applications: Prediction & Elections, Recommendations, Business

Analytics, Clustering, Text Analytics

Page 1 of 34
Data Science - Complete Study Guide Examination Preparation

UNIT 1: INTRODUCTION TO DATA SCIENCE

1. Concept of Data Science
Data Science is an interdisciplinary field that focuses on extracting useful information, patterns,
and knowledge from data. It uses methods from statistics, mathematics, computer science, and
domain knowledge to analyze data and support decision-making.
In simple words: Data Science converts raw data into meaningful insights that help individuals,
organizations, and governments take better decisions.

Background and Need

Earlier, data was small and handled using traditional databases. With the growth of the internet,
smartphones, social media, digital payments, and sensors, data started growing rapidly in size
and complexity. Traditional tools became insufficient, which led to the emergence of Data
Science.
Modern society generates data continuously from: mobile phones, social media platforms, online
shopping websites, banking systems, healthcare systems, and government portals.

What Data Science Does

• Understand customer behavior
• Improve business and government policies
• Detect fraud in transactions
• Predict future trends
• Support automation and intelligent systems

Basic Flow of Data Science

1. Data Collection - Gathering data from websites, databases, sensors
2. Data Cleaning - Removing missing or incorrect data
3. Data Analysis - Understanding patterns using statistics
4. Data Visualization - Presenting insights via graphs and charts
5. Model Building - Creating predictive models using machine learning
6. Insights and Decision Making
Example: An online shopping website collects data about products searched, user location, and
time spent on pages. Data Science analyzes this to recommend products, predict customer
needs, and improve sales and customer satisfaction.
Exam Tip: Definition + Flow + Example is enough for a 7.5-mark answer on 'Concept of Data
Science'.

2. Traits (V's) of Big Data

Big Data refers to extremely large, fast, and diverse datasets that cannot be handled efficiently
using traditional data processing tools. Big Data is described using six major characteristics called
the '6 V's'.

Page 2 of 34
Data Science - Complete Study Guide Examination Preparation

The relationship: Big Data provides the raw material. Data Science provides the methods and
tools to analyze that data. Big Data without Data Science has no value.

V1 - Volume: Scale of Data

Volume refers to the huge amount of data generated and stored. The name 'Big Data' itself
indicates that data size is extremely large. Data is measured in terabytes, petabytes, and
exabytes.
Example: Facebook generates approximately 1 billion messages, 4.5 billion 'Like' button
records, and 350 million new posts daily. Global mobile data traffic exceeded 6.2 exabytes per
month in 2016.

V2 - Velocity: Speed of Data

Velocity refers to the speed at which data is generated, collected, and processed. Data flows
continuously from social media, machines and sensors, mobile devices, and online transactions.
Some data must be processed in real time to be useful.
Example: Google handles billions of searches every day. Online payment systems process
transactions instantly. Stock market data changes every second. High velocity requires fast data
processing systems.

V3 - Variety: Diversity of Data

Variety refers to the different types and formats of data. Big Data comes from many
heterogeneous sources and exists in different forms.
• Structured Data: Well organized in rows and columns (databases, spreadsheets). Stored
in RDBMS.
• Semi-Structured Data: Partially organized but no strict structure (XML, JSON, CSV, TSV,
email, log files).
• Unstructured Data: No predefined format (text, images, audio, videos, emails, social
media posts).
• Quasi-Structured Data: Textual data with inconsistent formats (e.g., web server logs).
Example: A single social media post may include text, image, video, hashtags, and comments -
all different data types.

V4 - Veracity: Trustworthiness of Data

Veracity refers to the quality, accuracy, and reliability of data. Big Data often contains missing
values, duplicate data, errors, noise, and inconsistent information. Because data comes from
many sources, ensuring accuracy becomes difficult.
Example: Customer feedback data may contain spelling mistakes, incomplete information, or
false reviews. Low-quality data leads to wrong analysis and incorrect business decisions.

V5 - Value: Usefulness of Data

Value refers to the importance and usefulness of data. Having a large amount of data is
meaningless unless it can be converted into valuable information. Value is often considered the
MOST IMPORTANT trait of Big Data.
Example: Sales data becomes valuable when it helps increase profit, understand customer
needs, and improve marketing strategies. Raw data without analysis has zero value.

Page 3 of 34
Data Science - Complete Study Guide Examination Preparation

V6 - Variability: Inconsistency of Data

Variability refers to changes in data meaning, structure, or flow over time. Data may behave
differently under different situations or contexts.
Example: During festivals or sales events, data volume suddenly increases. The word 'bad' may
mean negative in normal language but positive in slang. Such inconsistencies make data
analysis more challenging.
V What It Means Example
Volume Huge amount of data generated Facebook: 350 million posts per day
Speed of data generation and Stock market data changing every
Velocity
processing second
Different types: structured,
Variety Text, images, videos, JSON, XML
unstructured, semi-structured
Missing values, duplicates, false
Veracity Quality and reliability of data
reviews
Usefulness of data after analysis Sales data helping marketing
Value
(MOST IMPORTANT) strategy
Changes in meaning or structure 'Bad' = negative normally, positive in
Variability
over time slang

3. Web Scraping
Web Scraping (also called web data mining or web harvesting) is the process of automatically
extracting data from websites using tools, software, or programming languages. In Data Science,
web scraping is mainly used in the data collection stage, because most real-world data is
available on websites.
Web scraping requires two parts:
• Crawler: An AI algorithm that browses the web searching for required data by following
links across the internet.
• Scraper: A specific tool created to extract data from the website. Its design varies with the
complexity of the project.

Web Crawling vs Web Scraping

Feature Web Crawling Web Scraping
Downloading and storing contents of Extracting specific data elements
Definition
large numbers of websites from a website using its structure
Mostly done on large scale (search
Scale Can be implemented at any scale
engines)
Specific information (prices, names,
Output Generic information (index)
reviews)
Scraping Amazon for product prices
Example Googlebot indexing web pages
and ratings

Page 4 of 34
Data Science - Complete Study Guide Examination Preparation

Methods of Web Scraping

7. Chrome Extensions (Beginner Level): Tools: Web Scraper Extension, Data Miner.
Install extension, select data visually, export to Excel/CSV. Best for non-programmers.
8. Software Tools (No Coding): Tools: Octoparse, ParseHub. Drag-and-drop interface.
Used by companies to collect competitor pricing data daily.
9. Excel Web Query: Import table data directly from websites using Power Query. Only
works well with simple static websites.
10. Python Libraries (Most Powerful): Most commonly used in Data Science. Libraries:
Requests, Beautiful Soup, Selenium, Scrapy.

Python Libraries for Web Scraping

Library Purpose Best For
Sends HTTP request to website and
Requests Downloading webpage content
gets HTML content
Beautiful Reads HTML, finds specific tags, Static websites with fixed HTML
Soup extracts data structure
Controls browser automatically, Dynamic websites requiring login or
Selenium
clicks buttons, fills login forms JavaScript
Framework for large-scale scraping, Enterprise-level, thousands of pages
Scrapy
very fast daily

Step-by-Step: How Web Scraping Works

11. Program sends a request to the target website URL
12. Website sends back HTML code in response
13. Scraper parses the HTML to find required tags
14. Extract specific data elements (price, title, rating)
15. Store extracted data in Excel, CSV, or database
Example: On a product page: Product Name and Price are extracted. Beautiful Soup finds the
<h2>Product Name</h2> and <span>Rs.2000</span> tags and extracts those values into a
spreadsheet.

Applications of Web Scraping

• Market Research - Compare competitor product prices and analyze trends
• Stock Market Analysis - Collect daily stock prices to analyze patterns
• Sentiment Analysis - Scrape product reviews for positive/negative feedback analysis
• Job Market Analysis - Collect job titles, skills, and salary data from job portals
• E-Commerce Analysis - Track ratings and reviews to understand customer behavior
• News Monitoring - Get detailed reports on current news for business intelligence

Advantages
• Saves Time: A 5-6 hour manual process completes in 2-3 minutes automatically
• Large Data Collection: Enables collection of 50,000+ records at once
• Real-Time Data: Can run daily or hourly for up-to-date information
• Cost Effective: No need to buy expensive datasets from third parties
• Better Decisions: More data leads to better analysis and business strategy

Page 5 of 34
Data Science - Complete Study Guide Examination Preparation

Disadvantages
• Legal Issues: Some websites prohibit scraping of copyrighted content
• Website Blocking: Too many requests may result in IP ban
• Data Cleaning Required: Scraped data contains missing values, symbols, duplicates
• Dynamic Website Complexity: JavaScript-loaded content requires Selenium
• Structure Changes: If website changes HTML, scraper breaks and needs updating
Python Code Example:
import requests
from bs4 import BeautifulSoup
url = '[Link]
response = [Link](url)
soup = BeautifulSoup([Link], '[Link]')
title = [Link]('h1')
print([Link])

4. Analysis vs Reporting
Reporting involves organizing data into summaries. Analysis involves inspecting, cleaning,
transforming, and modeling these summaries to gain actionable insights for a specific purpose.
Aspect Reporting Analysis / Analytics
Monitor data; understand Interpret data deeper; provide
Purpose
performance of various functions recommendations on actions
Building, consolidating, organizing, Asking questions, examining,
Activities
formatting, summarizing comparing, interpreting, forecasting
Canned reports, dashboards, alerts Ad-hoc responses, insights,
Output
pushed to users recommended actions, forecasts
Repetitive tasks; used by functional Requires expertise; used by data
People
business heads scientists and business leaders
Clean, raw data; periodic Enriched data with big data; can
Data
(daily/weekly/monthly/quarterly) predict future trends
What happened? How much did we Why did it happen? What will
Question Asked
sell? happen? What should we do?
Provides visibility into what is Provides insight into why it
Value
happening happened and what to do next
Example: Reporting: Sales were Rs. 50 lakhs last month. Analysis: Sales dropped 20%
because a competitor launched a lower-priced product. Prescriptive: Reduce our price by 10% to
recapture market share.

5. Data Collection, Storing, and Processing

Page 6 of 34
Data Science - Complete Study Guide Examination Preparation

A. Data Collection
Data Collection is the process of gathering, measuring, and analyzing accurate data from various
sources to find answers to research problems and evaluate outcomes. The analyst must first
answer: What is the goal? What kinds of data are needed? What methods will be used?
Data Types:
• Qualitative Data: Descriptions: color, size, quality, appearance
• Quantitative Data: Numbers: statistics, percentages, poll numbers
Collection Methods:
• Primary Data: Original, first-hand data. More accurate but time-consuming and
expensive. Methods: Interviews, Focus Groups, Surveys, Delphi Technique.
• Secondary Data: Second-hand, already collected by others. Easier and cheaper but may
raise accuracy concerns.
Primary Collection Techniques:
• Interviews: Direct or via phone/mail; most common method
• Projective Technique: Indirect interview; respondents complete an incomplete question
• Delphi Technique: Expert panel provides and consolidates opinions
• Focus Groups: 6-12 people led by a moderator to discuss an issue

B. Data Storing
Data Storing refers to the systematic storage of collected data for use in the data science process.
Four key steps:
16. Identify Goals: Have a clear strategy for data saving aligned with business objectives
before jumping to technology.
17. Big Data vs Small Data: Big data: multiple servers, multiple sources, unstructured,
continuous generation. Small data: structured, stored in single databases, full control.
18. Avoid Data Fatigue: Don't store useless data. Focus on what aligns with specific goals.
Use encoding (e.g., geocodes) to reduce data size. Work with Database Administrators.
19. Data Management (SQL vs NoSQL): SQL for structured data with known schema
(MySQL, MariaDB). NoSQL for semi/unstructured data requiring faster complex queries
(MongoDB, Cassandra).
Feature SQL (e.g., MySQL) NoSQL (e.g., MongoDB)
Best For Structured, well-defined data Semi-structured or unstructured data
Schema Fixed schema required Flexible, schema-less
Complex Queries Joins are slower for complex queries Handles complex queries efficiently
Best for Big Data, high-performance
Scale Best for smaller, defined datasets
needs
Use Case Banking, financial systems Social media, e-commerce big data

C. Data Processing
Data Processing refers to the set of operations performed to transform raw data into useful
information. It involves multiple stages:
20. Discovery: Understand requirements, resources, budget. Define the business problem
and initial hypotheses (IH).

Page 7 of 34
Data Science - Complete Study Guide Examination Preparation

21. Information Preparation: Explore and pre-process data. Perform data cleaning,
transformation, and exploratory visualization.
22. Model Planning: Determine methods and techniques to draw connections between
variables. Apply Exploratory Data Analysis (EDA).
23. Model Building: Create training and testing datasets. Analyze classification, association,
clustering techniques. Build the best-fit model.
24. Operationalize: Deploy final code, technical reports, and run pilot project in real-time
environment.
25. Communicate Results: Evaluate outcomes against original objectives. Share key
findings with stakeholders.

6. Describing and Modelling

A. Describing (Data Visualization)
Data Visualization is the graphical representation of information and data. By using visual
elements like charts, graphs, and maps, it provides an accessible way to see and understand
trends, outliers, and patterns.
Common types of visualization:
• Charts: Bar chart, Pie chart, Line chart
• Tables: Structured rows and columns
• Graphs: Scatter plots, network graphs
• Maps: Geographic data visualizations
• Dashboards: Combined multiple visualizations for decision-making
• Infographics: Text combined with visual elements
• Advanced: Histogram, Heat Map, Box-and-whisker, Word Cloud, Treemap, Gantt Chart
Tools used: Tableau, Power BI, Python (Matplotlib, Seaborn), Excel

B. Data Modelling
Data Modelling describes information in a systematic way that allows it to be stored and retrieved
efficiently in a Relational Database Management System (RDBMS) such as SQL Server, MySQL,
or Oracle.
The model translates logic of accurately describing real-world things and their relationships into
rules enforceable by computer code. Data modelling also helps organizations ensure they are
collecting ALL necessary information.
Example: A sales transaction is broken down into related groups: customer, seller, item sold,
payment method. Without the date field, return policies cannot be enforced. Data modelling
helps capture all necessary data points.

7. Statistical Modelling and Algorithm Modelling

A. Statistical Modelling
Statistical Modelling is the process of applying statistical analysis to a dataset. A statistical model
is a mathematical representation (mathematical model) of observed data.

Page 8 of 34
Data Science - Complete Study Guide Examination Preparation

When data analysts apply statistical models, they can: identify relationships between variables,
make predictions about future datasets, and visualize data for stakeholders.
3 Reasons to Learn Statistical Modelling:
26. Better equipped to choose the right model: Understand which of the many statistical
models best answers the question and fits your data.
27. Better data preparation: Before any model can run, data must be cleaned. Statistical
knowledge helps identify what data is relevant and what is 'bad or incomplete'.
28. Become a better communicator: Present findings to both technical audiences (granular
details) and non-technical business teams (key takeaways).

B. Algorithm Modelling
An Algorithm is a set of carefully defined instructions that take a set of inputs, manipulate them,
and produce some output.
A recipe is an informal example: it takes ingredients (inputs), follows steps, and produces a dish
(output).
Three basic elements of algorithms:
• Sequence: Executing instructions one at a time, in order
• Selection: Choosing which instructions to execute based on some condition (if-else)
• Iteration: Repeating instructions until some condition is met (loops)
Feature Statistical Modelling Algorithm Modelling
Mathematical representation of Set of defined instructions to
Definition
observed data transform inputs to outputs
Finding patterns and relationships in Step-by-step computational
Focus
data procedure
Regression, probability, hypothesis Decision trees, neural networks,
Techniques
testing sorting algorithms
Output Statistical predictions and insights Precise computed outputs
Predicting house prices using Google Maps finding shortest route
Example
regression (A* algorithm)
Automating decisions and
Primary Use Understanding and interpreting data
computations

8. Artificial Intelligence (AI) and Data Science

What is Artificial Intelligence?
Artificial Intelligence is the branch of computer science that enables machines to mimic human
intelligence. This includes tasks like learning, reasoning, problem-solving, and decision-making.
AI uses algorithms to perform autonomous actions.
Types of AI:
• Narrow AI (Weak AI): Performs a specific task only. Example: Siri, Alexa, AlphaGo,
Netflix recommendations.

Page 9 of 34
Data Science - Complete Study Guide Examination Preparation

• General AI (Strong AI): Can perform any intellectual task a human can do. Still
theoretical.
• Super AI: Smarter than humans. Fictional concept, not yet real.

What is Data Science?

Data Science is a comprehensive process that involves pre-processing, analysis, visualization,
and prediction. A Data Scientist extracts data using SQL/NoSQL, cleans anomalies, analyzes
patterns, applies predictive models, and generates insights.

Key Differences: AI vs Data Science

Aspect Artificial Intelligence Data Science
Comprehensive process: pre-
Implementation of predictive models
Definition processing, analysis, visualization,
to forecast future events
prediction
Computer algorithms (deep learning, Statistical techniques (regression,
Techniques
neural networks) probability, hypothesis testing)
Specialized: TensorFlow, PyTorch, Diverse: Python, R, SQL, Tableau,
Tools
deep learning frameworks Excel, Power BI
Impart autonomy to data models; Find hidden patterns; build models
Goal
emulate human cognition using statistical insights
More analytical and practical; less
Science Level High degree of scientific processing
pure science
Role AI is a TOOL used by Data Data Science is the broader
Relationship Scientists DISCIPLINE that includes AI
Simple Analogy: Data Science provides the FUEL (data), AI provides the ENGINE (intelligence).
Example: Self-driving cars (AI) rely on sensor and traffic data (Data Science) to navigate safely.
ChatGPT (AI) was trained using a huge dataset of conversations (Data Science).

9. Myths of Data Science

There are many common misconceptions about Data Science. These myths prevent people from
entering the field or create wrong expectations. Understanding them is essential for aspiring data
scientists.
# Myth Reality
You need a PhD or advanced Curiosity, practice, and skills matter most. Projects >
1
degree Degrees.
Only engineers/CS grads can Anyone from arts, commerce, biology can succeed.
2
succeed Analytical thinking matters more than background.
Coding is ~30% of the job. Most work is cleaning,
3 Data Science is all about coding
analyzing, and communicating data.
AI automates tasks; humans interpret results, set
4 AI will replace Data Scientists
strategy, and handle ethics.

Page 10 of 34
Data Science - Complete Study Guide Examination Preparation

Small, clean datasets are enough to learn and create

5 You need huge datasets
meaningful insights.
It is only about building AI 60-70% of work is cleaning, exploring, and
6
models visualizing data before building any model.
Only big companies (Google, Every company with data needs insights - startups,
7
Amazon) hire banks, hospitals, even local shops.
You will get a huge salary Entry-level in India: ~Rs.7-8 LPA. Grows to Rs.15-20
8
instantly LPA with experience.
The field is saturated with no Demand is growing in healthcare, finance, agri-tech,
9
jobs startups, and all sectors.
1 Basic statistics, probability, and logical thinking are
You must be a math genius
0 enough.
1 Dirty data produces wrong results. Cleaning is often
Data cleaning is not important
1 70% of the project time.
1 Visualization tools replace Tools create charts; humans must interpret and
2 analysts explain what the charts mean.
1 Deep learning is always the best Simple models often outperform complex ones on
3 model small datasets and are easier to explain.
1 Coding, visualization, domain knowledge, and
Data Science is just statistics
4 communication are equally important.
1 100 clean, relevant records > 1 million messy
More data is always better
5 records.
1 Once you deploy a model, work Models degrade over time as data changes. MLOps
6 is done (monitoring and retraining) is essential.
Exam Tip: For a 7.5-mark question on 'Myths', explain any 5-6 myths with their reality and one
tip or example each. Do not just list them - explain briefly.

Page 11 of 34
Data Science - Complete Study Guide Examination Preparation

UNIT 2: PROGRAMMING TOOLS FOR DATA

SCIENCE
Programming tools are software and libraries that help analyze data using code. Python is the
most popular language for Data Science because it is easy to learn, easy to read, and has
powerful libraries for every data science task.

1. Matplotlib
Matplotlib is a low-level Python library used for data visualization. It is easy to use and emulates
MATLAB-like graphs. It is built on top of NumPy arrays and consists of plots like line chart, bar
chart, histogram, scatter plot, and pie chart.
• Creator: John D. Hunter
• Installation: pip install matplotlib
• Primary Interface: [Link] (provides MATLAB-like interface)

Key Components
• Figure Class: The overall window or page on which everything is drawn. Top-level
container holding one or more axes. Created using [Link]()
• Axes Class: The most basic and flexible unit for creating sub-plots. A figure can contain
many axes. Provides methods: set_title(), set_xlabel(), set_ylabel(), set_xlim(), set_ylim(),
legend()
• Pyplot Module: Provides convenient functions like plot(), title(), xlabel(), legend(), show()

Important Matplotlib Functions

• [Link](x, y) - Creates line chart
• [Link]('text') - Adds title to the plot
• [Link]() / [Link]() - Adds axis labels
• [Link]() / [Link]() - Sets axis value limits
• [Link]() / [Link]() - Sets tick marks and labels
• [Link]() - Adds a legend box
• [Link]() - Adds grid lines
• [Link]() - Displays the plot
• [Link](rows, cols, index) - Creates subplots (multiple charts in one figure)
• [Link](x, y) - Bar chart
• [Link](x, y) - Scatter plot
• [Link](x) - Histogram
• [Link](y, labels=...) - Pie chart

Multiple Subplots - 4 Methods

29. add_axes() Method: Figure.add_axes([left, bottom, width, height]) - adds axes at
specified position

Page 12 of 34
Data Science - Complete Study Guide Examination Preparation

30. subplot() Method: [Link](rows, cols, index) - places plot at given grid position
31. subplots() Method: fig, axes = [Link](rows, cols) - creates figure and all subplots at
once
32. subplot2grid() Method: plt.subplot2grid(shape, location, rowspan, colspan) - spans
across rows/columns
Simple Example:
import [Link] as plt
x = [10, 20, 30, 40]
y = [20, 25, 35, 55]
[Link](x, y)
[Link]('Linear Graph')
[Link]('X-Axis')
[Link]('Y-Axis')
[Link]()

Matplotlib vs Seaborn
Feature Matplotlib Seaborn
High-level, built on Matplotlib, less
Level Low-level, more control, more code
code
Syntax More verbose and complex Simpler and easier to learn
Attractive built-in themes and
Themes Basic styling, manual customization
statistical plots
More comfortable with Pandas
Data Handling Works with lists and NumPy arrays
DataFrames
Statistical data exploration and
Use Case Custom, precise visualizations
visualization
Can open multiple figures
Figures Better at avoiding overlapping plots
simultaneously

2. NumPy (Numerical Python)

NumPy stands for Numerical Python. It is a Python library used for working with arrays and
numerical data. It also provides functions for linear algebra, Fourier transform, and matrices.
• Created by: Travis Oliphant in 2005
• Installation: pip install numpy
• Import Convention: import numpy as np

Why Use NumPy?

• NumPy arrays are stored at one continuous memory location (locality of reference),
making them up to 50x faster than Python lists
• Provides the ndarray object - a powerful N-dimensional array
• Supports broadcasting - arithmetic between arrays of different shapes
• Provides mathematical functions for linear algebra, statistics, Fourier transforms
• Arrays are very frequently used in Data Science where speed and resources are critical

Page 13 of 34
Data Science - Complete Study Guide Examination Preparation

NumPy Array Key Terms

• ndarray: Main object; homogeneous multidimensional array (all elements same type)
• rank: Number of dimensions (axes) in the array
• shape: Tuple showing size in each dimension, e.g., (2, 3) for 2 rows, 3 columns
• dtype: Data type of elements in the array (int64, float64, etc.)
• Broadcasting: Method NumPy uses to allow arithmetic between arrays of different
shapes

Important NumPy Operations

• [Link]([1,2,3]) - Create array from list
• [Link]((3,4)) - Array of all zeros
• [Link]((3,4)) - Array of all ones
• [Link](0, 30, 5) - Sequence with step size
• [Link](0, 5, 10) - 10 evenly spaced values from 0 to 5
• [Link](2, 3) - Reshape dimensions
• [Link]() - Collapse to 1D array
• [Link](arr) / [Link]() / [Link]() / [Link]() - Statistical operations
• arr[arr > 4] - Boolean indexing (filter elements)
• arr[0:5] - Slicing
• arr.T - Transpose
Example:
import numpy as np
arr = [Link]([10, 20, 30, 40])
print('Average:', [Link](arr)) # Output: 25.0
arr2 = [Link](0, 90, 1).reshape(3, 30) # 3 rows, 30 cols
bool_arr = arr > 15
print(arr[bool_arr]) # Output: [20 30 40]

NumPy vs Pandas
Feature NumPy Pandas
Primary Use Numerical data, arrays, matrices Tabular data (rows and columns)
Main Object ndarray (N-dimensional array) DataFrame and Series
Consumes more memory than
Memory Memory efficient
NumPy
Better performance for rows >=
Performance Better performance for rows <= 50K
500K
Indexing Speed Very fast array indexing Relatively slower Series indexing
Heterogeneous (mixed types
Data Type Homogeneous (same type)
allowed)

Page 14 of 34
Data Science - Complete Study Guide Examination Preparation

3. Scikit-learn
Scikit-learn (sklearn) is an open-source Python library for machine learning. It is simple, efficient,
accessible to everyone, and built on NumPy, SciPy, and Matplotlib.
• License: BSD license (open source, commercially usable)
• Installation: pip install scikit-learn

Key Features / Capabilities

• Classification: Identifying which category an object belongs to. Example: spam or not
spam, disease or healthy.
• Regression: Predicting a continuous value. Example: house price prediction, stock price.
• Clustering: Automatic grouping of similar objects into sets. Example: customer
segmentation.
• Dimensionality Reduction: Reducing number of features (variables) to consider.
Example: PCA (Principal Component Analysis).
• Model Selection: Comparing, validating, and choosing parameters and models. Includes
cross-validation.
• Preprocessing: Feature extraction, normalization, scaling, imputation of missing values.

ML Workflow (Pipeline)
Scikit-learn provides the Pipeline class ([Link]) that executes a sequence of
steps in a pipe-like manner. Output of one step becomes input of the next.
33. Gather Data (real-time or from file/database)
34. Data Pre-processing (cleaning, scaling, handling missing values)
35. Split into Training Set (70-80%) and Test Set (20-30%)
36. Train Model on Training Data
37. Evaluate on Test Data
38. Compare Accuracy and Select Best Model

Train/Test Split
• Training Set: Used to train the model. Model learns from this data.
• Test Set: Used to evaluate the model on unseen data. Testing accuracy is a better
estimate of real performance.
• Validation Set: Sometimes used for hyperparameter tuning.
Example with Iris Dataset:
from [Link] import load_iris
from sklearn.model_selection import train_test_split
from [Link] import DecisionTreeClassifier
from [Link] import accuracy_score
iris = load_iris()
X_train, X_test, y_train, y_test = train_test_split(
[Link], [Link], test_size=0.25)
clf = DecisionTreeClassifier()
[Link](X_train, y_train)
print(accuracy_score(y_test, [Link](X_test))) # ~97%

KNNImputer (Handling Missing Values)

KNNImputer is a scikit-learn class to fill missing values in a dataset using the KNN algorithm
approach. It fills missing values based on the values of the K nearest neighbors.

Page 15 of 34
Data Science - Complete Study Guide Examination Preparation

• n_neighbors: Number of data points to include closer to the missing value

• metric: Distance metric for searching (default: nan_euclidean)
• weights: How neighboring values are weighted (uniform or distance)

4. NLTK (Natural Language Toolkit)

NLTK is a Python package for Natural Language Processing (NLP). It is used to preprocess,
analyze, and process unstructured human language text data before it can be used in data
analysis or machine learning.
• Installation: pip install nltk (or python -m pip install nltk==3.5)

Key NLTK Concepts

• Tokenizing by Word: Splits text into individual words. Words are the atoms of natural
language - the smallest unit of meaning. Used to identify frequently occurring words.
• Tokenizing by Sentence: Splits text into sentences to analyze how words relate to each
other and get context.
• Stop Words: Common words like 'in', 'is', 'an', 'the' that add no meaning and are filtered
out during processing.
• POS Tagging: Part-of-Speech tagging: labels each word as noun, verb, adjective, etc.
• Named Entity Recognition (NER): Identifies and classifies names, places, organizations,
dates in text.

Applications of NLTK
• Chatbots and virtual assistants (understanding user queries)
• Sentiment analysis (positive/negative product reviews)
• Language translation
• Spam detection (email filtering)
• Resume filtering for job recruitment
• Document classification and topic modeling
Tokenization Example:
from [Link] import word_tokenize
text = 'Python is useful for data science'
words = word_tokenize(text)
print(words)
# Output: ['Python', 'is', 'useful', 'for', 'data', 'science']

5. Visualizing Data: Bar Charts, Line Charts, Scatterplots

A. Bar Charts
A Bar Chart displays categorical data with rectangular bars. The length or height of each bar
corresponds to the data value. Bars can be vertical (bar()) or horizontal (barh()).
Properties:
• All bars have a common base
• Each bar's length corresponds to its data value

Page 16 of 34
Data Science - Complete Study Guide Examination Preparation

• All bars have the same width

• Bar charts MUST start at zero to avoid misleading comparisons
• Best for: Comparing categories of data or multiple data series
Example: Comparing student enrollment across 4 courses: C=20, C++=15, Java=30,
Python=35.
Code Example:
import [Link] as plt
data = {'C':20, 'C++':15, 'Java':30, 'Python':35}
[Link](list([Link]()), list([Link]()),
color='maroon', width=0.4)
[Link]('Courses')
[Link]('Number of Students')
[Link]('Student Enrollment by Course')
[Link]()

B. Line Charts
Line Charts connect data points with a line and are best for showing trends over time. They
illustrate how values change continuously.
Best for: Time-series data, showing increases/decreases, trends, continuous change.
Common Line Styles:
• '-' solid line, '--' dashed line, '-.' dash-dot, ':' dotted
• Markers: 'o' circle, 's' square, '^' triangle, '*' star
Example: Showing the bias-variance tradeoff: as model complexity increases, variance
increases and bias decreases. Using [Link]() with different colors and linestyles for each line.
Code Example:
import [Link] as plt
x = [10, 20, 30, 40]
y = [20, 25, 35, 55]
[Link](x, y, color='green', linewidth=3,
marker='o', markersize=10, linestyle='--')
[Link]('Line Chart')
[Link]('X-Axis')
[Link]('Y-Axis')
[Link](['Data Series 1'])
[Link]()

C. Scatterplots
A Scatterplot visualizes the relationship between two paired sets of data. Each observation is
represented as a single dot in 2D space. The pattern of dots reveals correlations.
Best for: Identifying correlations or relationships between two continuous variables.
Example: Relationship between number of friends a user has (x-axis) and minutes spent on the
social media site per day (y-axis). More friends generally means more time on site.
Code Example:
import [Link] as plt
import numpy as np
x = [Link]([5,7,8,7,2,17,2,9,4,11])
y = [Link]([99,86,87,88,111,86,103,87,94,78])
[Link](x, y)
[Link]('Car Age vs Speed')
[Link]('Age of Car (years)')
[Link]('Speed (km/h)')

Page 17 of 34
Data Science - Complete Study Guide Examination Preparation

[Link]()
Chart Type Best For Key Function X-Axis Data
Categorical (names,
Bar Chart Comparing categories [Link](x, y)
groups)
Trends over Continuous or time-
Line Chart [Link](x, y)
time/sequence based
Relationship between 2
Scatter Plot [Link](x, y) Continuous (numerical)
variables
Histogram Frequency distribution [Link](x) Continuous (bins)
Pie Chart Part-to-whole proportions [Link](y) Categories (labels)

6. Working with Data: Reading Files and Scraping the Web

A. Reading Files in Python
Python uses the open() function to work with files. It takes two parameters: filename and mode.
File Opening Modes:
Mode Meaning File Exists? File Not Exist?
Read
'r' Opens for reading Raises error
(default)
'w' Write Opens, OVERWRITES content Creates new file
'a' Append Opens, adds to end Creates new file
'x' Create Raises error Creates new file
Text mode
't' - -
(default)
Binary mode
'b' - -
(images)
Reading Methods:
• read(): Reads the entire file content as a single string
• readline(): Reads one line at a time
• readlines(): Returns a list where each item is a line from the file
Examples:
# Reading a file (best practice with 'with' statement)
with open('[Link]', 'r') as f:
data = [Link]()
print(data)

# Writing to a file (overwrites existing content)

with open('[Link]', 'w') as f:
[Link]('some data to write')

# Appending to a file
with open('[Link]', 'a') as f:

Page 18 of 34
Data Science - Complete Study Guide Examination Preparation

[Link]('additional data')

# Reading a specific line (e.g., line 4)

with open('[Link]', 'r') as fo:
for currentline, line in enumerate(fo, 1):
if currentline == 4:
print(line)
break

B. Scraping the Web with Python

Python is the most popular language for web scraping because of its simplicity and wide range of
dedicated libraries (Requests, BeautifulSoup, Selenium, Scrapy).
Uses of Web Scraping:
• Price Monitoring: Scrape product data to fix optimal pricing and increase revenue
• Market Research: Collect high-quality data to analyze consumer trends
• News Monitoring: Detailed reports on current news for companies dependent on daily
news
• Sentiment Analysis: Collect social media data to understand general sentiment for
products
• Email Marketing: Collect Email IDs for bulk promotional campaigns
Python Web Scraping Example:
import requests
from bs4 import BeautifulSoup
# Step 1: Send request to website
url = '[Link]
response = [Link](url)
# Step 2: Parse the HTML
soup = BeautifulSoup([Link], '[Link]')
# Step 3: Find and extract data
title = [Link]('h1').text
price = [Link]('span', class_='price').text
print(f'Product: {title}, Price: {price}')
Exam Tip: In the exam, for Q on 'Working with Data', explain file reading modes + show code +
explain web scraping methods + one Python library with code. This covers a 10-mark answer
well.

Page 19 of 34
Data Science - Complete Study Guide Examination Preparation

UNIT 3: DATA SCIENCE METHODOLOGY

Data Science Methodology indicates the routine for finding solutions to a specific problem. It is a
cyclic process that undergoes critical behaviour, guiding business analysts and data scientists to
act accordingly. The methodology has 10 stages.

Stage 1: Business Understanding

Business Understanding forms the concrete foundation before solving any problem. Before
solving a problem in the business domain, it needs to be understood properly. Business sponsors
who need the analytic solution play the MOST CRITICAL role at this stage.

Goals
• Specify the key variables that are to serve as model targets
• Identify the relevant data sources the business has access to or needs to obtain
• Specify metrics used to determine the success of the project

Tasks
• Define Objectives: Work with customers and stakeholders to understand and identify
business problems. Formulate 'sharp' questions that are relevant, specific, and
unambiguous.
• Identify Data Sources: Find relevant data that helps answer the questions defining
project objectives.

5 Types of Questions Data Science Can Answer

Question Type ML Approach Example
How much / How many? Regression What will sales be next quarter?
Which category? Classification Is this email spam or not spam?
Which customers are similar to each
Which group? Clustering
other?
Is this weird? Anomaly Detection Is this banking transaction fraudulent?
Which option? Recommendation What product should we show this user?

SMART Success Metrics

The success metrics for a project must be SMART:
• S - Specific: Clearly defined and focused on a specific area
• M - Measurable: Quantifiable; progress can be tracked numerically
• A - Achievable: Realistic and attainable given available resources
• R - Relevant: Aligned to business goals and objectives
• T - Time-bound: Has a specific deadline or timeframe

Page 20 of 34
Data Science - Complete Study Guide Examination Preparation

Example: We want to achieve a customer churn prediction with an accuracy rate of 85% within
this 3-month project. With this data, we can offer promotions to reduce churn.

Artifacts (Deliverables)
• Charter Document: Living document updated throughout the project as new discoveries
are made
• Data Sources: Lists raw data sources and their destination locations
• Data Dictionaries: Descriptions of data schema, data types, validation rules, entity-relation
diagrams

Stage 2: Analytic Approach

Based on business understanding, the analyst decides which analytical approach to follow. The
approach is chosen based on the type of question being answered.
Approach Question Answered Method Used
What is the current
Descriptive Summary statistics, visualization of current state
status?
What is happening and
Diagnostic Statistical analysis, correlation, root cause analysis
why?
What will happen in the
Predictive Machine learning, time-series forecasting
future?
How should the problem Optimization, decision models, action
Prescriptive
be solved? recommendations

Stage 3: Data Requirements

The chosen analytical approach indicates the necessary data content, formats, and sources to be
gathered. The analyst must find answers for:
• WHAT data is needed to answer the question?
• WHERE will the data come from?
• WHEN was the data generated?
• WHY is this data relevant to the problem?
• HOW will it be collected and processed?
• WHO owns or collected this data?

Stage 4: Data Collection

Data can be collected in any random format. According to the analytical approach chosen and the
output to be obtained, the collected data should be validated. More data may be gathered or
irrelevant data discarded.

Page 21 of 34
Data Science - Complete Study Guide Examination Preparation

Types of Data Collected

• Structured Data: Databases, spreadsheets (e.g., transaction records)
• Unstructured Data: Text, images, audio, video, social media posts
• Semi-Structured Data: JSON, XML, CSV, log files

Data Acquisition Goals

• Produce a clean, high-quality data set whose relationship to target variables is understood
• Locate the data set in the appropriate analytics environment ready for modeling
• Develop a solution architecture for the data pipeline that refreshes and scores data
regularly

Data Pipeline Options

• Batch-based: Data is collected and processed in groups at scheduled times
• Streaming / Real-time: Data is processed continuously as it arrives
• Hybrid: Combination of batch and real-time processing

Artifacts
• Data Quality Report: Data summaries, attribute-target relationships, variable ranking
• Solution Architecture: Diagram of the data pipeline
• Checkpoint Decision: Decide whether to proceed, collect more data, or abandon the
project
Example: A hospital collecting patient data needs structured diagnosis records, semi-structured
doctor notes, and images from different departments. Gaps in data collection require revising
requirements.

Stage 5: Data Understanding

Data Understanding answers the question: 'Is the data collected representative of the problem to
be solved?' It involves exploring the data to verify quality and identify patterns.

Activities
• Explore data using descriptive statistics: mean, median, mode, standard deviation,
percentiles
• Audit data quality: check for missing values, outliers, duplicates, inconsistencies
• Use data summarization and visualization to understand patterns
• Determine if the data quality is adequate to answer the question
• This step may lead to reverting back to the data collection stage for corrections
The goal is to understand patterns inherent in the data to choose an appropriate predictive model.
If data is insufficient or low quality, new data sources may need to be found.

Page 22 of 34
Data Science - Complete Study Guide Examination Preparation

Stage 6: Data Preparation

Data Preparation encompasses all activities to construct the final dataset used in the modeling
stage. It is usually the MOST TIME-CONSUMING step in a data science project (often 60-80% of
project time).

Activities in Data Preparation

• Data Cleaning: Dealing with missing or invalid values, eliminating duplicates, proper
formatting
• Data Integration: Combining data from multiple sources: files, tables, platforms
• Data Transformation: Converting data into more useful variables, scaling, normalization
• Feature Engineering: Creating additional explanatory variables (features/predictors)
through domain knowledge and existing structured variables. Text analytics is useful here
for unstructured data.
Today's high-performance, massively parallel systems allow data scientists to prepare data much
more rapidly using very large datasets. Automating certain data preparation steps can minimize
ad-hoc preparation time.

Stage 7: Modeling
Modeling decides whether the prepared data is appropriate for processing and focuses on
building predictive or descriptive models. This is the stage where machine learning algorithms are
applied.

Goals
• Determine optimal data features for the machine-learning model
• Create an informative ML model that predicts the target most accurately
• Create an ML model that is suitable for production

Three Main Tasks

• Feature Engineering: Create data features from raw data to facilitate model training.
Balancing act: include informative variables, exclude unrelated ones (which add noise).
• Model Training: Find the model that answers the question most accurately by comparing
success metrics.
• Production Suitability: Determine if the model is ready for deployment in a production
environment.

Model Training Steps

39. Split input data randomly into training set and test set
40. Build models using the training data set
41. Evaluate using competing ML algorithms with various tuning parameters (parameter
sweep)
42. Determine the 'best' solution by comparing success metrics between methods

CRITICAL WARNING: Avoid Data Leakage

Data Leakage occurs when data from OUTSIDE the training set is included, allowing the model to
make unrealistically good predictions. This is a common reason why data scientists are

Page 23 of 34
Data Science - Complete Study Guide Examination Preparation

suspicious when results seem 'too good to be true'. These dependencies are hard to detect. To
avoid leakage: iterate between building analysis data set, creating model, and evaluating
accuracy.

Stage 8: Evaluation
Before deployment, the data scientist evaluates the model to understand its quality and ensure it
properly and fully addresses the business problem. Evaluation happens during model
development, not just before deployment.

Evaluation Process
• Testing Set: Independent of training set but follows the same probability distribution and
has known outcomes. Used to evaluate model performance.
• Diagnostic Measures: Computing various metrics, tables, and graphs to interpret model
quality
• Validation Set: Sometimes a final validation set is used for final assessment
• Statistical Significance Tests: Provide additional proof of model quality, especially for
high-stakes decisions

Key Checkpoint Questions

• Does the model answer the question with sufficient confidence given the test data?
• Should alternative approaches be tried?
• Should additional data be collected or more feature engineering be done?
• Should different algorithms be tested?

Stage 9: Deployment
Once a satisfactory model is developed and approved by business sponsors, it is deployed into
the production or production-like environment. Initial deployment is usually limited until
performance is fully evaluated.

Goal
Deploy models with a data pipeline to a production environment for final user acceptance.

Types of Deployment
• Batch Deployment: Predictions made on groups of data at scheduled intervals
• Real-time Deployment: Predictions made instantly as new data arrives (live API)
• Hybrid Deployment: Combination of batch and real-time

How Models Are Made Available

• Open API interface: models exposed to be consumed by online websites, spreadsheets,
dashboards, line-of-business applications, back-end applications
• Build telemetry and monitoring into the deployed model for status reporting and
troubleshooting

Page 24 of 34
Data Science - Complete Study Guide Examination Preparation

Artifacts
• Status dashboard displaying system health and key metrics
• Final modeling report with deployment details
• Final solution architecture document
Example: A sales response propensity model deployed through a campaign management
process: built by development team, administered by marketing group, used in sales process.

Stage 10: Feedback

By collecting results from the implemented model, the organization gets feedback on the model's
performance and its impact on the environment. Feedback enables refinement and improvement.

Purpose of Feedback
• Understand how the model is actually performing in production
• Identify gaps between expected and actual performance
• Enable data scientists to refine the model to improve accuracy and usefulness
• Some or all feedback-gathering and model refinement steps can be automated
Example: Feedback could be the response rate to a promotional campaign targeting customers
identified by the model as high-potential. A low response rate signals the model needs
refinement.

Customer Acceptance (Final Stage)

• Confirm deployed model and pipeline meet customer's business needs
• Validate that the system answers questions with acceptable accuracy
• All documentation finalized and reviewed
• Project handed off to operations team
• EXIT REPORT produced: technical report containing all project details

Complete Methodology Overview

Stage Core Question Key Output
1. Business Charter document, data sources, SMART
What is the problem?
Understanding metrics
Selected approach:
2. Analytic Approach Which method to use?
descriptive/diagnostic/predictive/prescriptive
Data requirements list (what, where, when,
3. Data Requirements What data do we need?
why, how, who)
Where does the data come Raw dataset, data quality report, pipeline
4. Data Collection
from? architecture
5. Data
Is data representative? Exploratory analysis, quality assessment
Understanding
6. Data Preparation How to clean and structure it? Prepared dataset with engineered features
7. Modeling Which model works best? Trained ML model with evaluated accuracy

Page 25 of 34
Data Science - Complete Study Guide Examination Preparation

8. Evaluation Is the model good enough? Model evaluation metrics, checkpoint decision
Deployed model + API + monitoring
9. Deployment How to put it in production?
dashboard
10. Feedback Is it performing well? Refined model, exit report, improved system

Page 26 of 34
Data Science - Complete Study Guide Examination Preparation

UNIT 4: DATA SCIENCE APPLICATIONS

1. Prediction and Elections

What is Prediction?
Prediction is the process of using historical and current data to estimate future outcomes.
Machine learning models are trained on historical data to learn relationships between variables
and make accurate predictions about unknown or future values.
• Predictor: The model constructed from a training set used to predict unknown values. Its
accuracy refers to how well it estimates new data.
Note: Prediction does not always mean the future. It can also mean determining something about
the present that is unknown - e.g., whether a transaction that already occurred was fraudulent.

Classification vs Prediction
Aspect Classification Prediction
Identifying or estimating
Finding a model to describe and
Definition missing/unavailable continuous data
distinguish data classes/categories
values
Output Type Categorical value (class label) Continuous numerical value
Categorize new data into one of the
Goal Predict a missing or future element
known classes
Classifying email as 'spam' or 'not Predicting the correct treatment
Example
spam' dosage for a patient
Decision Tree, KNN, SVM, Naive Linear Regression, Random Forest,
ML Algorithm
Bayes Neural Networks

Why are Predictions Important?

• Allow businesses to make highly accurate guesses about likely outcomes based on
historical data
• Provide insights that result in tangible business value
• Enable proactive action: e.g., if model predicts customer churn, company can intervene
with promotions
• Reduce uncertainty in business decisions

Role of Data Science in Elections

In elections, Data Science is used for analyzing large amounts of voter data to predict possible
outcomes and help political parties design better strategies.
• Predict winning candidates or political parties
• Analyze voter behavior and demographic preferences
• Study influence of age, income, education, and region on voting

Page 27 of 34
Data Science - Complete Study Guide Examination Preparation

• Monitor public opinion through surveys and social media analysis

• Identify key issues that are most important to different voter segments

How Election Prediction Works (Step by Step)

43. Collect large amounts of data: previous election results, voter demographics (age, income,
education), opinion polls, social media posts
44. Clean and analyze the data to identify patterns and correlations
45. Apply ML algorithms: regression, classification, time-series models
46. Model learns relationships: which age group supports which party, which economic
conditions affect voting behavior
47. Generate predictions about election outcomes and seat distributions
Example: National election scenario: Data scientists analyze voter data and find that urban
youth strongly prefer Party A while rural areas support Party B. Middle-income groups are
undecided. Using this, Party A focuses campaigns on undecided urban middle-class voters and
allocates campaign budget accordingly.

Why Prediction in Elections is Important

• Helps political parties design better, data-driven campaign strategies
• Allows early insights into possible results for media and analysts
• Enables efficient resource allocation (which constituencies to focus on)
• Reduces guesswork and improves strategic planning

Limitations of Prediction
• Predictions are NOT always 100% accurate
• Depends heavily on the quality and representativeness of data
• Sudden political events, scandals, or shifts in public sentiment can invalidate predictions
• Bias in data can lead to systematically incorrect predictions

Applications of Prediction (Beyond Elections)

Domain Application Example
Sales forecasting, demand Amazon predicting holiday season sales
Business
prediction volume
Disease risk prediction, patient
Healthcare Predicting diabetes risk from patient records
outcomes
Stock market prediction, fraud
Finance Credit card fraud detection in real time
detection
Rainfall, temperature
Weather Predicting cyclone paths 72 hours ahead
forecasting
Predicting football match results using team
Sports Match outcome prediction
statistics

Page 28 of 34
Data Science - Complete Study Guide Examination Preparation

2. Recommendations and Business Analytics

A. Recommendation Systems
A Recommendation System is a data-driven technique that suggests products, services, or
content to users based on their preferences, behavior, and past activities. It is a subclass of
machine learning that predicts ratings or rankings a user might give to a specific item.
Core Question a Recommendation System Answers: 'What should we show to the user next?'
Used by: Google, Instagram, Spotify, Amazon, Reddit, Netflix, YouTube - to increase user
engagement with the platform.

Types of Recommendation Algorithms

Algorithm How It Works Best For Example
Recommends based on
Collaborative what similar users liked. Users with shared Netflix: Users who liked Movie
Filtering 'People like you also preferences X also liked Movie Y
bought...'
Recommends items
Content-Based similar to what user Personalized Spotify: Recommends songs
Filtering previously liked. preferences with similar genre/tempo/artist
Analyzes item features.
Combines collaborative +
Most production Amazon: Combines user
Hybrid Methods content-based filtering for
systems history + item features
better accuracy

How Recommendation Systems Work (Process Flow)

48. User Activity: Clicks, searches, purchases, ratings
49. Data Collection: Record all user interactions
50. Data Processing: Clean and structure interaction data
51. Apply Algorithm: Collaborative, content-based, or hybrid
52. Generate Recommendations: Ranked list of items
53. Show to User: Display recommendations in the interface
54. User Feedback: User clicks/ignores; system learns and improves
Example: If a user buys a mobile phone, the system recommends: phone cover, earphones,
and screen protector. Amazon shows 'Customers who bought this also bought...' increasing
average order value by 35%.

Evaluating Recommendation Systems

• RMSD (Root Mean Square Deviation): Measures prediction error. Lower is better.
• MAE (Mean Absolute Error): Average absolute difference between predicted and actual
ratings. Lower is better.
• K-Fold Cross Validation: Splits data into K folds; trains on K-1, tests on 1. Average
accuracy across all folds shows how well the system generalizes.

Why Recommendation Systems are Important

• Helps users find relevant content quickly (reduces choice overload)
• Increases customer satisfaction and engagement

Page 29 of 34
Data Science - Complete Study Guide Examination Preparation

• Boosts sales and revenue (Amazon: 35% of revenue from recommendations)

• Creates personalized user experience

B. Business Analytics
Business Analytics is the process of analyzing data to make better, data-driven business
decisions. It uses statistical methods, data analysis, and machine learning to understand trends,
understand past performance, and predict future outcomes.
Business Analytics answers: What happened? Why did it happen? What will happen? What
should we do?
Business Analytics combines: management (domain knowledge), business (practical limitations),
and computer science (data analysis).

Four Types of Business Analytics

Type Question Method Example
Descriptive What Reporting, aggregation, Monthly sales report: Rs. 50 lakh
Analytics happened? dashboards revenue last month
Diagnostic Why did it Root cause analysis, Sales dropped 20% because competitor
Analytics happen? statistical testing launched at lower price
Predictive What will Machine learning, Customer likely to churn within 30 days
Analytics happen? regression, forecasting based on patterns
Prescriptive What should Optimization, Reduce price by 10% in Region X to
Analytics we do? recommendation engines increase market share
Note: Most companies progress through these types in order. Building core competencies in
descriptive analytics before advancing to predictive is essential.

Benefits of Business Analytics

• Enable data-driven decision making with potential to increase profits and improve
efficiency
• Allow businesses to plan for the future in ways that were previously impossible
• Help make informed decisions by modeling outcomes and understanding the past
• Minimize guesswork in business strategy
• Present meaningful, clear data to convince stakeholders

Real-World Example
Fast-food drive-thru: By monitoring how busy the drive-thru is, digital order boards change based
on queue length. When the line is long, boards highlight items that can be prepared quickly. When
lines are short, higher-margin items are featured. This responds to real-time needs to improve
efficiency - an example of Prescriptive Analytics.

Page 30 of 34
Data Science - Complete Study Guide Examination Preparation

3. Clustering and Text Analytics

A. Clustering
Clustering (Cluster Analysis) is an unsupervised machine learning technique used to group similar
data points together. Objects in the same cluster are MORE SIMILAR to each other than to
objects in other clusters.
Classification is done using criteria such as: smallest distances, density of data points, graph
relationships, or statistical distributions.

Why Clustering?
• Discover hidden patterns and structures in data
• Understand customer behavior and segmentation without predefined labels
• Simplify large datasets by reducing them to a few meaningful groups
• Anomaly detection: find data points that do NOT belong to any cluster (outliers)
• Data exploration: understand what groups naturally exist in the data

Clustering Algorithms (4 Main Types)

Algorithm How It Works Best For Limitation
Assigns each point to
nearest of K centroids; Large datasets, well-
Must specify K in advance;
K-Means iterates until stable. separated round-shaped
sensitive to outliers
Minimizes within-cluster clusters
distance.
Density-Based Spatial
Clustering of Applications
Irregular-shaped
with Noise. Groups Struggles with varying density
DBSCAN clusters, noisy data,
densely packed points; clusters
outlier detection
marks low-density points
as noise/outliers.
Builds multilevel tree
(dendrogram) from
When K is unknown;
Hierarchical bottom-up Computationally expensive for
creating visual
Clustering (agglomerative) or top- large datasets
dendrograms
down (divisive). No need
to specify K.
Similarity graph-based
approach. Models
Complex, non-convex
Spectral nearest-neighbor Computationally expensive;
shapes; graph structured
Clustering relationships as an requires similarity matrix
data
undirected graph; uses
graph cuts.

Clustering Use Cases

• Customer Segmentation: Group customers by age, income, spending habits for targeted
marketing campaigns
• Market Research: Group markets into segments for more effective advertising and
product positioning

Page 31 of 34
Data Science - Complete Study Guide Examination Preparation

• Document Analysis: Cluster documents by theme for automatic organization of large

document collections
• Network Traffic Classification: Group traffic types to identify spam, bot traffic, and block
malicious sources
• Fraud Detection: Identify unusual transaction patterns that deviate from normal cluster
behavior
• Image Segmentation: Group pixels of similar characteristics for image processing and
computer vision
• Medical Research: Group patients with similar symptoms or genetic markers for clinical
studies
Example: Shopping mall example: Customers are grouped based on age, income, and
spending habits. High-income customers form Cluster A (Premium products). Low-income
customers form Cluster B (Budget products). Middle-income customers form Cluster C (Value
products). This enables targeted marketing for each segment.

Clustering Flow
55. Raw Data collection
56. Feature Selection (choosing which variables to cluster on)
57. Apply Clustering Algorithm (K-Means, DBSCAN, etc.)
58. Form Clusters (groups of similar data)
59. Analyze Each Cluster (what characteristics define each group)
60. Business Insights (how to use this segmentation)

B. Text Analytics
Text Analytics (or Text Mining) involves the use of unstructured text data and processing it into
usable structured data. It is an application of Natural Language Processing (NLP). Text Analytics
has gained importance because millions of people generate enormous amounts of text data
online daily.
Sources of text data: blogs, social media posts, tweets, product reviews, surveys, customer
service logs, forum discussions, research papers.

Why Text Analytics is Important

• Huge amount of data is in text (unstructured) form - the majority of all data
• Helps organizations understand customer opinions, sentiment, and needs
• Supports automation of document processing and customer service
• Improves decision-making by extracting insights from customer feedback
• Serves as foundation for advanced NLP tasks: classification, categorization, sentiment
analysis

Text Analytics Process

61. Text Data Collection: Gather relevant text from the appropriate sources
62. Text Cleaning: Remove stop words, punctuation, special characters, HTML tags
63. Tokenization: Break text into individual words or sentences
64. Feature Extraction: Convert text to numerical form (TF-IDF, word vectors, embeddings)
65. Apply NLP/ML Model: Classification, clustering, sentiment analysis model
66. Insights/Output: Sentiment scores, topics, keywords, classifications

Page 32 of 34
Data Science - Complete Study Guide Examination Preparation

Important Text Analytics Techniques

• Tokenization: Splitting text into words (word tokenization) or sentences (sentence
tokenization)
• Stop Word Removal: Filtering out common words ('the', 'is', 'in') that add no analytical
value
• Word Frequency Calculation: Counting how often each word appears to find the most
important terms
• Sentiment Analysis: Determining whether text expresses positive, negative, or neutral
opinion
• Named Entity Recognition (NER): Identifying and classifying names, places,
organizations, dates in text
• Topic Modeling (LDA): Discovering abstract topics that occur in a collection of
documents
• Text Classification: Categorizing text into predefined categories (e.g., spam detection,
news categorization)

Applications of Text Analytics

• Sentiment Analysis: Analyzing product reviews on Amazon/Flipkart to understand
customer satisfaction
• Chatbots: Processing and understanding customer queries for automated customer
service
• Spam Detection: Filtering spam emails by analyzing text content and patterns
• Social Media Monitoring: Tracking brand sentiment on Twitter, Facebook, Instagram
• Resume Filtering: Recruiters use text analytics to filter hundreds of applications by
matching skills
• Scientific Discovery: Analyzing thousands of research papers to find new insights or
connections
• Competitive Intelligence: Monitoring news and competitor websites for business
intelligence
Example: Customer review: 'The product is very good and delivery was fast.' Text analytics
identifies: positive sentiment, high satisfaction with product AND delivery. Companies use this to
understand which aspects customers value most and where improvement is needed.
Resume Text Analytics Example:
import pandas as pd
from [Link] import word_tokenize
df = pd.read_csv('Resume_Data.csv', encoding='utf-8')
# Tokenize each resume
df['tokens'] = df['Resume_str'].apply(word_tokenize)
# Count categories
print(df['Category'].value_counts())

Clustering vs Text Analytics Comparison

Feature Clustering Text Analytics
Data Type Primarily structured/numerical data Unstructured text data
ML Category Unsupervised Machine Learning NLP + Supervised/Unsupervised ML
Primary Technique Distance-based grouping algorithms Tokenization, sentiment analysis,

Page 33 of 34
Data Science - Complete Study Guide Examination Preparation

NER
Group similar data points without
Goal Extract meaningful insights from text
labels
Sentiment scores, keywords,
Output Cluster assignments / Segment IDs
classifications
Customer segmentation by Analyzing Twitter posts for brand
Example
purchase patterns sentiment

Summary: Unit 4 at a Glance

Primary
Topic Key Definition Real-World Example
Technique
Using historical
Regression,
data to estimate Predicting election results using voter
Prediction Classification,
future/unknown demographics
Time-series
outcomes
Using Data Science
Survey analysis,
Elections to analyze voter Party targeting undecided middle-income
Social media
Analytics data and predict urban voters
mining, Regression
political outcomes
Suggesting
Collaborative /
Recommenda relevant items to Netflix recommending shows; Amazon
Content-based /
tion Systems users based on 'also bought'
Hybrid Filtering
preferences
Descriptive,
Using data to drive
Business Diagnostic, Fast-food chain adjusting menu boards
business decisions
Analytics Predictive, based on queue length
across 4 types
Prescriptive
Grouping similar K-Means,
data points without DBSCAN, Customer segmentation for targeted
Clustering
labels Hierarchical, marketing
(unsupervised) Spectral
Extracting insights Tokenization,
Analyzing product reviews for customer
Text Analytics from unstructured Sentiment Analysis,
satisfaction
text data NER, LDA

Page 34 of 34