DATA SCIENCE

Complete Exam Notes — All 4 Units

Session 2024-25 | IIT-Level Preparation

Unit Topics

Unit 1 Introduction to Data Science — Big Data, Web Scraping, Myths, Modelling

Unit 2 Programming Tools — Python, NumPy, Matplotlib, Scikit-Learn, NLP, Scraping

Unit 3 Data Science Methodology — CRISP-DM, Understanding, Preparation, Modelling

Unit 4 Applications — Elections, Recommendations, Clustering, Text Analytics

 UNIT 1 — Introduction to Data Science

1.1 Concept of Data Science

Data Science is an interdisciplinary field that uses scientific methods, processes, algorithms, and systems to
extract knowledge and insights from structured and unstructured data. It combines statistics, computer science,
and domain expertise.

Computer Domain
Statistics + + = DATA SCIENCE
Science Knowledge

Key Roles in Data Science

Role Focus

Data Analyst Analyze & visualize existing data

Data Engineer Build data pipelines & infrastructure

Data Scientist Build predictive models & find insights

ML Engineer Deploy & scale ML models in production

1.2 Traits of Big Data (The 5 Vs)

V Trait Meaning

Volume Scale Terabytes to Petabytes of data

Velocity Speed Real-time / near-real-time generation

Variety Form Structured, semi-structured, unstructured

Veracity Quality Accuracy, trust, and consistency of data

Value Worth Actionable insights from raw data

1.3 Web Scraping

Web scraping is the automated extraction of data from websites. Python's BeautifulSoup and Scrapy are the
primary tools.

import requests
from bs4 import BeautifulSoup

url = "[Link]
response = [Link](url)
soup = BeautifulSoup([Link], '[Link]')

# Extract all headings

headings = soup.find_all('h1')
for h in headings:
print([Link])

# Extract table data

table = [Link]('table')
 rows = table.find_all('tr')
for row in rows:
cols = row.find_all('td')
print([[Link]() for c in cols])

1.4 Analysis vs Reporting vs Collection vs Storing

Activity Description Example

Collection Gathering raw data from sources APIs, sensors, surveys, scraping

Storing Persisting data for future use SQL DB, HDFS, Data Warehouse

Processing Cleaning & transforming data Pandas, Spark ETL pipelines

Analysis Deriving insights from data EDA, hypothesis testing

Reporting Communicating findings Dashboards, charts, PowerBI

Modelling Building predictive systems Regression, SVM, Neural Nets

1.5 Statistical & Algorithmic Modelling

Statistical Modelling: Uses probability distributions and statistical tests to describe relationships in data (e.g.,
Linear Regression, ANOVA).
Algorithmic/ML Modelling: Uses optimization algorithms to learn patterns (e.g., Decision Trees, Neural
Networks).

# Statistical Modelling Example (Linear Regression)

import numpy as np
from sklearn.linear_model import LinearRegression

X = [Link]([[1],[2],[3],[4],[5]])
y = [Link]([2, 4, 5, 4, 5])

model = LinearRegression()
[Link](X, y)

print(f"Slope (m): {model.coef_[0]:.2f}")

print(f"Intercept (c): {model.intercept_:.2f}")
print(f"Prediction for x=6: {[Link]([[6]])[0]:.2f}")

1.6 AI and Data Science

--contains-
AI Machine Learning --subset--> Deep Learning
->

Data Science uses AI/ML for prediction, classification, and clustering. AI provides the toolbox; Data Science
provides the methodology to apply it.

1.7 Myths of Data Science

Myth 1: Data Science = Machine Learning
DS is much broader — it includes data collection, cleaning, analysis, and storytelling.
Myth 2: More data always means better results
Quality > Quantity. Dirty data produces bad models ("Garbage In, Garbage Out").

Myth 3: Data Scientists only code

They also design experiments, communicate insights, and think statistically.

Myth 4: DS is only for tech companies

Used in healthcare, agriculture, finance, government, and more.
 UNIT 2 — Programming Tools for Data Science

2.1 Python Toolkits Overview

Library Purpose Key Functions

NumPy Numerical computing array, linspace, reshape, dot, linalg

Pandas Data manipulation DataFrame, read_csv, groupby, merge

Matplotlib 2D Plotting plot, scatter, bar, hist, subplot

Scikit-Learn Machine Learning fit, predict, train_test_split, metrics

NLTK Natural Language Processing

tokenize, stem, pos_tag, FreqDist

Requests+BS4 Web Scraping get, BeautifulSoup, find_all, select

2.2 NumPy — Numerical Python

import numpy as np

# Array creation
a = [Link]([1, 2, 3, 4, 5])
b = [Link]((3, 3)) # 3x3 zeros matrix
c = [Link](0, 10, 5) # [0, 2.5, 5, 7.5, 10]
d = [Link](0, 20, 2) # [0,2,4,...,18]

# Array operations (vectorized — NO loops needed)

print(a * 2) # [2,4,6,8,10]
print([Link](a)) # element-wise sqrt
print([Link](a), [Link](a)) # statistics

# Matrix operations
A = [Link]([[1,2],[3,4]])
B = [Link]([[5,6],[7,8]])
print([Link](A, B)) # matrix multiplication
print([Link](A)) # matrix inverse
print([Link](A)) # determinant

# Reshaping
x = [Link](12).reshape(3, 4) # 3 rows, 4 cols

2.3 Visualizing Data

Matplotlib is the foundational visualization library. Three main chart types are tested:

Chart Type When to Use Code

Bar Chart Compare categories [Link](x, y)

Line Chart Trend over time [Link](x, y)

Scatter Plot Relationship between 2 vars [Link](x, y)

Histogram Distribution of 1 var [Link](data, bins=10)

Pie Chart Proportions [Link](sizes, labels=lbl)

 import [Link] as plt
import numpy as np

fig, axes = [Link](1, 3, figsize=(15, 4))

# Bar Chart
categories = ['A', 'B', 'C', 'D']
values = [23, 45, 12, 67]
axes[0].bar(categories, values, color='steelblue')
axes[0].set_title('Bar Chart')

# Line Chart
x = [Link](0, 10, 100)
axes[1].plot(x, [Link](x), color='red', label='sin(x)')
axes[1].plot(x, [Link](x), color='blue', label='cos(x)')
axes[1].legend()
axes[1].set_title('Line Chart')

# Scatter Plot
x = [Link](100)
y = 2*x + [Link](100)
axes[2].scatter(x, y, alpha=0.5, color='green')
axes[2].set_title('Scatter Plot')

plt.tight_layout()
[Link]('[Link]', dpi=150)
[Link]()

2.4 Working with Data — Reading Files

import pandas as pd

# Reading data
df = pd.read_csv('[Link]')
df_excel = pd.read_excel('[Link]')
df_json = pd.read_json('[Link]')

# Exploration
print([Link]()) # first 5 rows
print([Link]) # (rows, cols)
print([Link]()) # dtypes + nulls
print([Link]()) # stats summary

# Data Cleaning
[Link](inplace=True) # remove NaN rows
df['age'].fillna(df['age'].mean(), inplace=True) # fill NaNs
df.drop_duplicates(inplace=True) # remove duplicates
df['salary'] = df['salary'].astype(float) # type conversion

# Filtering
young = df[df['age'] < 30]
high_earners = df[df['salary'] > 50000]

# Grouping & Aggregation

dept_avg = [Link]('department')['salary'].mean()
print(dept_avg)
2.5 Web Scraping the Web
import requests
from bs4 import BeautifulSoup
import pandas as pd

def scrape_table(url):
headers = {'User-Agent': 'Mozilla/5.0'}
page = [Link](url, headers=headers)
soup = BeautifulSoup([Link], '[Link]')

table = [Link]('table', {'class': 'data-table'})

rows = []
for tr in table.find_all('tr')[1:]: # skip header
cols = [[Link]() for td in tr.find_all('td')]
[Link](cols)

df = [Link](rows, columns=['Name','Value','Date'])
return df

# Scrapy Spider (basic structure)

import scrapy
class QuotesSpider([Link]):
name = 'quotes'
start_urls = ['[Link]

def parse(self, response):

for quote in [Link]('[Link]'):
yield {
'text': [Link]('[Link]::text').get(),
'author': [Link]('[Link]::text').get(),
}

2.6 NLTK — Natural Language Processing

NLTK (Natural Language Toolkit) provides tools for processing human language data — text tokenization,
stemming, tagging, and more.

import nltk
from [Link] import word_tokenize, sent_tokenize
from [Link] import stopwords
from [Link] import PorterStemmer, WordNetLemmatizer
from nltk import FreqDist, pos_tag

text = "Data Science is amazing. It transforms raw data into insights!"

# Tokenization
words = word_tokenize(text) # ['Data','Science','is',...]
sents = sent_tokenize(text) # split into sentences

# Stopword removal
stop = set([Link]('english'))
filtered = [w for w in words if [Link]() not in stop]

# Stemming (aggressive: running -> run)

ps = PorterStemmer()
stemmed = [[Link](w) for w in filtered]

# Lemmatization (context-aware: running -> run, better -> good)

lem = WordNetLemmatizer()
lemmatized = [[Link](w) for w in filtered]

# POS Tagging
tags = pos_tag(words) # [('Data','NNP'), ('Science','NNP'),...]

# Frequency Distribution
fd = FreqDist(filtered)
print(fd.most_common(5)) # top 5 words

Remove Stem /
Raw Text -> Tokenize -> -> -> Feature Vector
Stopwords Lemmatize
 UNIT 3 — Data Science Methodology

3.1 CRISP-DM Framework Overview

CRISP-DM (Cross-Industry Standard Process for Data Mining) is the most widely used lifecycle for data science
projects. It has 6 iterative phases. The process is cyclical — insights from later phases feed back into earlier ones.

1. Business 2. Data 3. Data

--> -->
Understanding Understanding Preparation

6. Deployment <-- 5. Evaluation <-- 4. Modeling

1. Business Understanding
• Define the BUSINESS OBJECTIVE (what problem to solve)
• Assess the situation — inventory resources, constraints, risks
• Define Data Mining Goal — translate business goal to DS goal
• Produce Project Plan — timeline, tools, success criteria
• Key Q: What decisions will this model support?

2. Data Understanding
• Collect initial data from available sources
• Describe data — format, quantity, field meanings
• Explore data — EDA: distributions, correlations, outliers
• Verify data quality — missing values, inconsistencies
• Tools: pandas profiling, matplotlib, seaborn

3. Data Preparation
• Select relevant features/records
• Clean data — handle missing values, outliers, duplicates
• Construct new features (Feature Engineering)
• Integrate data from multiple sources
• Format data — scaling, encoding, train/test split

4. Modeling
• Select modeling techniques (e.g., Decision Tree, SVM, NN)
• Generate test design — cross-validation strategy
• Build model — train on training data
• Assess model — evaluate on validation set
• Key: Iterate — tune hyperparameters, try different algorithms

5. Evaluation
• Evaluate results against Business Objectives
• Review the entire process for mistakes or improvements
• Determine next steps — deploy, iterate, or restart
• Metrics: Accuracy, Precision, Recall, F1, RMSE, AUC-ROC
6. Deployment
• Plan deployment — how will model be used in production?
• Monitor and maintain the model over time
• Produce final report and document findings
• Review project — lessons learned
• Formats: REST API, batch job, embedded in app

3.2 Analytic Approach

Choosing the right analytic approach depends on the type of question being asked:

Question Type Approach Example Algorithm

Descriptive What happened? Summary stats, Frequency tables

Diagnostic Why did it happen? Correlation, Regression analysis

Predictive What will happen? Regression, Decision Tree, SVM

Prescriptive What should we do? Optimization, Reinforcement Learning

Classification Which category? Logistic Reg., Random Forest, SVM

Clustering Natural groupings? K-Means, DBSCAN, Hierarchical

3.3 Data Preparation in Detail

import pandas as pd
import numpy as np
from [Link] import StandardScaler, LabelEncoder
from sklearn.model_selection import train_test_split

df = pd.read_csv('raw_data.csv')

# 1. Handle Missing Values

df['age'].fillna(df['age'].median(), inplace=True) # numerical
df['city'].fillna(df['city'].mode()[0], inplace=True) # categorical
[Link](subset=['target'], inplace=True) # drop if target missing

# 2. Remove Outliers (IQR method)

Q1 = df['salary'].quantile(0.25)
Q3 = df['salary'].quantile(0.75)
IQR = Q3 - Q1
df = df[(df['salary'] >= Q1 - 1.5*IQR) & (df['salary'] <= Q3 + 1.5*IQR)]

# 3. Encode Categorical Variables

le = LabelEncoder()
df['gender_enc'] = le.fit_transform(df['gender']) # Male->1, Female->0

# One-Hot Encoding (for multi-class)

df = pd.get_dummies(df, columns=['city'], drop_first=True)

# 4. Feature Scaling
scaler = StandardScaler()
df[['age','salary']] = scaler.fit_transform(df[['age','salary']])
 # 5. Train-Test Split
X = [Link]('target', axis=1)
y = df['target']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2,
random_state=42)

3.4 Modelling — Key Algorithms

from [Link] import DecisionTreeClassifier
from [Link] import RandomForestClassifier
from [Link] import SVC
from [Link] import accuracy_score, classification_report

# Decision Tree
dt = DecisionTreeClassifier(max_depth=5, random_state=42)
[Link](X_train, y_train)
y_pred = [Link](X_test)
print(f"Decision Tree Accuracy: {accuracy_score(y_test, y_pred):.2f}")

# Random Forest
rf = RandomForestClassifier(n_estimators=100, random_state=42)
[Link](X_train, y_train)
print(f"Random Forest Accuracy: {accuracy_score(y_test, [Link](X_test)):.2f}")

# Support Vector Machine

svm = SVC(kernel='rbf', C=1.0, gamma='scale')
[Link](X_train, y_train)

# Full Report
print(classification_report(y_test, [Link](X_test)))
# Shows: Precision, Recall, F1-Score per class

3.5 Evaluation Metrics

Metric Formula Use When

Accuracy Correct / Total Balanced classes

Precision TP / (TP + FP) Cost of false positives is high

Recall TP / (TP + FN) Cost of false negatives is high

F1-Score 2 * (P*R) / (P+R) Balance Precision & Recall

RMSE sqrt(mean((y-y_hat)^2)) Regression problems

AUC-ROC Area under ROC curve Classifier performance comparison

 UNIT 4 — Data Science Applications

4.1 Prediction and Elections

Electoral prediction uses historical voting data, demographic data, polling data, and social media sentiment to
forecast election outcomes. Key techniques include ensemble models, Bayesian forecasting, and sentiment
analysis.

import pandas as pd
from [Link] import RandomForestClassifier
from sklearn.model_selection import train_test_split
from [Link] import accuracy_score

# Sample election prediction pipeline

data = pd.read_csv('election_data.csv')
# Features: demographics, historical turnout, polling avg, economic indicators
features = ['age_median','income_median','past_turnout','poll_avg','unemployment']
X = data[features]
y = data['winner'] # 0=Party A, 1=Party B

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

model = RandomForestClassifier(n_estimators=200, random_state=42)
[Link](X_train, y_train)

print(f"Prediction Accuracy: {accuracy_score(y_test, [Link](X_test)):.2%}")

# Feature importance
import [Link] as plt
importance = [Link](model.feature_importances_, index=features)
importance.sort_values().plot(kind='barh', title='Feature Importance')
plt.tight_layout()
[Link]()

4.2 Recommendation Systems

Recommendation systems suggest relevant items to users. Two main approaches:

Approach How it Works Example Drawback

Collaborative
Recommend based on similar users'
Netflix:
preferences
users who watchedCold
X also
Start
watched
problem
Y
Filtering

Content-Based
Recommend based on item features
Spotify: songs with similar Limited
genre/tempo
novelty
Filtering

Hybrid Combine both methods Amazon product recommendations

Complex to implement

# Collaborative Filtering with Cosine Similarity

import numpy as np
from [Link] import cosine_similarity

# User-Item rating matrix (rows=users, cols=items)

ratings = [Link]([
[5, 3, 0, 1], # User A
[4, 0, 4, 1], # User B
[1, 1, 0, 5], # User C
 [0, 0, 5, 4], # User D
])

# Compute user similarity

user_sim = cosine_similarity(ratings)
print("User similarity matrix:")
print([Link](user_sim, 2))

# Recommend for User A: find most similar user, recommend their top items
user_a_idx = 0
similar_user = [Link](user_sim[user_a_idx])[-2] # 2nd most similar
unrated = [Link](ratings[user_a_idx] == 0)[0] # items A hasn't rated
for item in unrated:
print(f"Recommend item {item} with score {ratings[similar_user][item]}")

4.3 Business Analytics

Business analytics applies data science to business problems — sales forecasting, customer segmentation, churn
prediction, A/B testing, and KPI dashboards.

import pandas as pd
from sklearn.linear_model import LogisticRegression
from [Link] import StandardScaler

# Churn Prediction Example

df = pd.read_csv('[Link]')

# Features: usage, tenure, support_calls, contract_type

X = df[['monthly_charges','tenure','support_calls','num_products']]
y = df['churned'] # 1 = churned

scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)

model = LogisticRegression()
[Link](X_scaled, y)

# Predict churn probability for new customers

new_customer = [[45, 12, 3, 1]] # monthly_charge, tenure, calls, products
prob = model.predict_proba([Link](new_customer))[0][1]
print(f"Churn probability: {prob:.1%}")

4.4 Clustering
Clustering is unsupervised learning — grouping similar data points without predefined labels. Key algorithm:
K-Means.

Step 2: Assign each Step 3:

Step 1: Place k Step 4: Repeat until
--> point to nearest --> Recompute -->
centroids converged
centroid centroids

from [Link] import KMeans

from [Link] import StandardScaler
import [Link] as plt
import numpy as np
 # Generate sample data
[Link](42)
X = [Link]([[Link](100, 2) + center
for center in [[0,0],[5,5],[-3,5],[5,-3]]])

# Elbow Method — find optimal k

inertias = []
k_range = range(1, 10)
for k in k_range:
km = KMeans(n_clusters=k, random_state=42, n_init=10)
[Link](X)
[Link](km.inertia_)

[Link](k_range, inertias, 'bo-')

[Link]('k')
[Link]('Inertia (Within-cluster sum of squares)')
[Link]('Elbow Method for Optimal k')
[Link]()

# Fit final model

kmeans = KMeans(n_clusters=4, random_state=42, n_init=10)
labels = kmeans.fit_predict(X)

# Plot clusters
[Link](X[:, 0], X[:, 1], c=labels, cmap='viridis', alpha=0.6)
[Link](kmeans.cluster_centers_[:, 0], kmeans.cluster_centers_[:, 1],
marker='X', s=200, c='red', label='Centroids')
[Link]()
[Link]('K-Means Clustering (k=4)')
[Link]()

4.5 Text Analytics

Text analytics converts unstructured text into structured, analyzable form. Core pipeline: collect → preprocess →
represent → analyse/model.

Tokenize TF-IDF / Sentiment /

Raw Text -> -> -> Model (NB/SVM) ->
+Clean Bag-of-Words Topic

from sklearn.feature_extraction.text import TfidfVectorizer, CountVectorizer

from sklearn.naive_bayes import MultinomialNB
from [Link] import Pipeline
from [Link] import classification_report

# Sample corpus for Sentiment Analysis

docs = [
"This product is excellent and amazing", # positive
"Terrible quality, very disappointed", # negative
"Okay product, nothing special", # neutral
"Love it! Best purchase ever", # positive
"Waste of money, do not buy", # negative
]
labels = ['pos', 'neg', 'neu', 'pos', 'neg']

# TF-IDF Vectorization
tfidf = TfidfVectorizer(stop_words='english', ngram_range=(1,2))
 X_tfidf = tfidf.fit_transform(docs)
print("Vocabulary size:", len(tfidf.vocabulary_))
print("Top features:", tfidf.get_feature_names_out()[:10])

# Full Pipeline: Text -> TF-IDF -> Naive Bayes Classifier

pipeline = Pipeline([
('tfidf', TfidfVectorizer(stop_words='english')),
('clf', MultinomialNB()),
])
[Link](docs, labels)

# Predict sentiment
new_reviews = ["Fantastic experience!", "Completely useless product"]
predictions = [Link](new_reviews)
print(dict(zip(new_reviews, predictions)))

4.6 Sentiment Analysis — Advanced

# Using VADER (Valence Aware Dictionary for Sentiment Reasoning)
from [Link] import SentimentIntensityAnalyzer
import nltk
[Link]('vader_lexicon', quiet=True)

sia = SentimentIntensityAnalyzer()

reviews = [
"Absolutely love this! 10/10 would recommend.",
"It's okay, not great but not terrible either.",
"Worst. Product. Ever. Complete waste of money!!",
]

for review in reviews:

scores = sia.polarity_scores(review)
# scores = {'neg':0.2, 'neu':0.3, 'pos':0.5, 'compound':0.8}
label = 'Positive' if scores['compound'] >= 0.05 else 'Negative' if scores['compound'] <=
-0.05 else 'Neutral'
print(f"{label:9s} | compound={scores['compound']:+.3f} | {review[:40]}")

EXAM QUICK REVISION — Key Definitions

CRISP-DM phases (in order): Business Understanding → Data Understanding → Data Preparation → Modeling
→ Evaluation → Deployment
5 Vs of Big Data: Volume, Velocity, Variety, Veracity, Value
Web Scraping tools: BeautifulSoup (static pages), Scrapy (large-scale crawling)
Feature Engineering: Creating new features from existing ones to improve model performance
Collaborative Filtering weakness: Cold Start — cannot recommend to new users with no history
K-Means needs k specified: Use Elbow Method — plot inertia vs k, pick the "elbow" point
TF-IDF: Term Frequency × Inverse Document Frequency — weighs rare words higher
F1-Score = 0: When either Precision or Recall is 0. F1 is the harmonic mean, not arithmetic mean.
 PRACTICE EXAM QUESTIONS

2-Mark Questions
Q1. Define Data Science. How does it differ from Business Intelligence?

Q2. List the 5 Vs of Big Data with one example each.

Q3. What is the difference between Supervised and Unsupervised learning?

Q4. Define TF-IDF. Why is it preferred over raw frequency counts?

Q5. What is the Cold Start problem in recommendation systems?

Q6. Differentiate between Stemming and Lemmatization.

5-Mark Questions
Q1. Explain the CRISP-DM methodology with a neat diagram. Why is it iterative?

Q2. Compare Collaborative Filtering and Content-Based Filtering for recommendation systems with examples.

Q3. Explain the K-Means clustering algorithm with the Elbow Method for selecting k.

Q4. Describe the data preparation phase in CRISP-DM. What techniques are used to handle missing values?

Q5. Explain Sentiment Analysis. Describe a pipeline from raw text to sentiment label with code.

10-Mark Questions
Q1. A company wants to predict customer churn. Describe the complete CRISP-DM workflow, including data
preparation steps, model selection rationale, evaluation metrics, and deployment strategy.

Q2. Explain the complete Text Analytics pipeline for classifying news articles into categories. Include
tokenization, stopword removal, TF-IDF vectorization, and Naive Bayes classification with code snippets.

Q3. Compare Statistical Modelling vs Algorithmic Modelling in Data Science. Discuss with examples how each
approach is applied in election prediction and business analytics.