DATA SCIENCE

Complete Notes
Lifecycle • Statistics • Python/SQL • EDA & Visualization • Feature
Engineering • ML • Time Series • NLP • Big Data • MLOps • A/B Testing •
Code Examples

A single reference document covering the full Data Science landscape

— from statistics and data wrangling to modeling, deployment, and
business communication — with runnable Python/SQL code throughout.
Table of Contents
1. What is Data Science?
2. The Data Science Lifecycle
3. Statistics & Probability Foundations
3.1 Descriptive Statistics
3.2 Probability Basics
3.3 Probability Distributions
3.4 Inferential Statistics & Hypothesis Testing
3.5 Correlation & Covariance
4. Data Collection & Storage
5. SQL for Data Science
6. Python Data Stack (NumPy & Pandas)
7. Data Cleaning & Wrangling
8. Exploratory Data Analysis (EDA)
9. Data Visualization
10. Feature Engineering & Selection
11. Machine Learning Overview
12. Time Series Analysis
13. Natural Language Processing (NLP)
14. A/B Testing & Experimentation
15. Big Data Tools
16. Model Deployment & MLOps
17. Data Ethics, Bias & Privacy
18. Data Storytelling & Communication
19. Popular Data Science Tools & Libraries
20. Master Cheat Sheet
1. What is Data Science?
Data Science is the interdisciplinary field that uses statistics, programming, domain knowledge, and
machine learning to extract insights and actionable knowledge from structured and unstructured data. It
combines three core pillars:

• Mathematics & Statistics: the theoretical foundation for analyzing data.

• Computer Science / Programming: for collecting, processing and modeling data at scale.
• Domain Expertise: understanding the business or scientific context to ask the right questions.

Data Science vs Related Fields

• Data Analytics: focuses on analyzing historical data to answer specific business questions.
• Machine Learning: a subset of data science focused on building predictive models.
• Data Engineering: builds the infrastructure/pipelines that move and store data.
• Business Intelligence (BI): dashboards and reporting on past/current performance.
• Data Science: encompasses all of the above to solve open-ended problems and predict the future.

2. The Data Science Lifecycle

• 1. Business Understanding - define the problem and success metrics.
• 2. Data Collection - gather data from databases, APIs, files, web scraping, sensors.
• 3. Data Cleaning - handle missing values, duplicates, outliers, inconsistent formats.
• 4. Exploratory Data Analysis (EDA) - understand distributions, relationships, and patterns.
• 5. Feature Engineering - create and select meaningful input variables.
• 6. Modeling - apply statistical or machine learning models.
• 7. Evaluation - validate model performance against business goals.
• 8. Deployment - integrate the model into production systems.
• 9. Monitoring & Maintenance - track model drift and retrain as needed.
• 10. Communication - present findings/insights to stakeholders via reports and dashboards.

3. Statistics & Probability Foundations

3.1 Descriptive Statistics
Summarizes and describes the main features of a dataset.

• Mean: the average value of a dataset.

• Median: the middle value when data is sorted.
• Mode: the most frequently occurring value.
• Variance: average squared deviation from the mean - measures spread.
• Standard Deviation: square root of variance, in the same unit as the data.
• Range: difference between maximum and minimum values.
• Percentile / Quartile: values below which a given percentage of data falls (Q1, Q2=median, Q3).
• Interquartile Range (IQR): Q3 - Q1, used to detect outliers.
• Skewness: measures asymmetry of the distribution.
• Kurtosis: measures 'tailedness' of the distribution.
Python - Descriptive statistics with pandas
import pandas as pd

df = pd.read_csv("[Link]")
print([Link]()) # count, mean, std, min, quartiles, max
print("Skewness:", df["col"].skew())
print("Kurtosis:", df["col"].kurt())

# IQR-based outlier detection

Q1, Q3 = df["col"].quantile([0.25, 0.75])
IQR = Q3 - Q1
outliers = df[(df["col"] < Q1 - 1.5*IQR) | (df["col"] > Q3 + 1.5*IQR)]

3.2 Probability Basics

• Probability: likelihood of an event, between 0 and 1.
• Conditional Probability: P(A|B) - probability of A given B has occurred.
• Bayes' Theorem: P(A|B) = [P(B|A) * P(A)] / P(B) - updates beliefs given new evidence.
• Independence: two events are independent if P(A and B) = P(A) * P(B).
• Random Variable: a variable whose value is a numerical outcome of a random phenomenon.

3.3 Probability Distributions

Distribution Use Case

Normal (Gaussian) Continuous data symmetric around the mean (heights, errors)

Binomial Number of successes in n independent yes/no trials

Poisson Number of events in a fixed interval (arrivals, defects)

Bernoulli A single binary trial (success/failure)

Uniform All outcomes in a range are equally likely

Exponential Time between events in a Poisson process

Python - working with distributions (scipy)

from scipy import stats
import numpy as np

# Generate samples
normal_samples = [Link](loc=0, scale=1, size=1000)
binomial_samples = [Link](n=10, p=0.5, size=1000)

# PDF / CDF
x = 1.5
print("PDF at x:", [Link](x, loc=0, scale=1))
print("CDF at x:", [Link](x, loc=0, scale=1))

3.4 Inferential Statistics & Hypothesis Testing

Used to draw conclusions about a population from a sample.

• Null Hypothesis (H0): the default assumption of no effect/difference.

• Alternative Hypothesis (H1): what you're trying to prove.
• P-value: probability of observing the data if H0 is true; p < 0.05 typically rejects H0.
• Confidence Interval: a range likely to contain the true population parameter.
• Type I Error: false positive (rejecting a true H0).
• Type II Error: false negative (failing to reject a false H0).
• t-test: compares means of two groups.
• Chi-Square Test: tests independence between categorical variables.
• ANOVA: compares means across three or more groups.
Python - Hypothesis testing ([Link])
from scipy import stats

# Independent two-sample t-test

t_stat, p_value = stats.ttest_ind(group_a, group_b)
print("t-statistic:", t_stat, "p-value:", p_value)

# Chi-square test of independence

chi2, p, dof, expected = stats.chi2_contingency(contingency_table)

# One-way ANOVA
f_stat, p_val = stats.f_oneway(group1, group2, group3)

3.5 Correlation & Covariance

Covariance measures how two variables vary together (unbounded, unit-dependent). Correlation (e.g.
Pearson's r) normalizes this to a value between -1 and 1, indicating the strength and direction of a linear
relationship. Note: correlation does not imply causation.
Python - Correlation
corr_matrix = [Link](method="pearson") # or "spearman", "kendall"
import seaborn as sns
[Link](corr_matrix, annot=True, cmap="coolwarm")

4. Data Collection & Storage

• Databases (SQL): structured, relational storage (MySQL, PostgreSQL, SQL Server).
• NoSQL Databases: flexible schema for unstructured/semi-structured data (MongoDB, Cassandra).
• APIs: programmatic access to external data sources (REST, GraphQL).
• Web Scraping: extracting data directly from websites (BeautifulSoup, Scrapy, Selenium).
• Flat Files: CSV, JSON, Parquet, Excel files.
• Data Warehouses: centralized storage optimized for analytics (Snowflake, BigQuery, Redshift).
• Data Lakes: store raw structured/unstructured data at scale (S3, Azure Data Lake).
Python - reading data from various sources
import pandas as pd
import requests

df_csv = pd.read_csv("[Link]")
df_json = pd.read_json("[Link]")
df_excel = pd.read_excel("[Link]", sheet_name="Sheet1")
df_parquet = pd.read_parquet("[Link]")

# API request
response = [Link]("[Link]
data = [Link]()

# SQL database
from sqlalchemy import create_engine
engine = create_engine("postgresql://user:pass@host:5432/dbname")
df_sql = pd.read_sql("SELECT * FROM table_name", engine)

5. SQL for Data Science

SQL (Structured Query Language) is essential for querying and manipulating relational databases.
SQL - core query patterns
-- Basic SELECT
SELECT customer_id, SUM(amount) AS total_spent
FROM orders
WHERE order_date >= '2024-01-01'
GROUP BY customer_id
HAVING SUM(amount) > 1000
ORDER BY total_spent DESC
LIMIT 10;

-- JOINs
SELECT o.order_id, [Link], [Link]
FROM orders o
INNER JOIN customers c ON o.customer_id = c.customer_id;

-- Window functions
SELECT
customer_id,
order_date,
amount,
RANK() OVER (PARTITION BY customer_id ORDER BY amount DESC) AS rnk,
SUM(amount) OVER (PARTITION BY customer_id) AS customer_total
FROM orders;

-- Common Table Expression (CTE)

WITH monthly_sales AS (
SELECT DATE_TRUNC('month', order_date) AS month, SUM(amount) AS total
FROM orders
GROUP BY 1
)
SELECT * FROM monthly_sales ORDER BY month;

-- Subquery
SELECT name FROM customers
WHERE customer_id IN (SELECT customer_id FROM orders WHERE amount > 5000);

6. Python Data Stack (NumPy & Pandas)

6.1 NumPy Essentials
Python - NumPy
import numpy as np

arr = [Link]([1, 2, 3, 4, 5])

matrix = [Link]([[1, 2], [3, 4]])

print([Link](), [Link](), [Link]())

print(matrix.T) # transpose
print([Link](matrix, matrix)) # matrix multiplication
print([Link](0, 10, 5)) # 5 evenly spaced numbers
print(arr[arr > 2]) # boolean indexing

6.2 Pandas Essentials

Python - Pandas
import pandas as pd
df = pd.read_csv("[Link]")
[Link]() # first 5 rows
[Link]() # column types & non-null counts
[Link] # (rows, columns)

[Link]("category")["sales"].sum()
df.sort_values("sales", ascending=False)
df.pivot_table(values="sales", index="region", columns="category", aggfunc="sum")

df["new_col"] = df["col1"] + df["col2"]

[Link](lambda row: row["a"] * row["b"], axis=1)

[Link](other_df, on="id", how="left")

[Link]([df1, df2], axis=0)

7. Data Cleaning & Wrangling

• Handling missing data: drop, impute (mean/median/mode), or flag as a category.
• Removing duplicates: identify and drop duplicate records.
• Fixing data types: convert strings to dates, numbers, categories.
• Outlier treatment: cap (winsorize), remove, or transform (log) extreme values.
• Standardizing formats: consistent casing, date formats, units.
• String cleaning: trimming whitespace, regex-based extraction/replacement.
Python - common cleaning operations
df = df.drop_duplicates()
df["date"] = pd.to_datetime(df["date"], errors="coerce")
df["category"] = df["category"].[Link]().[Link]()
df["price"] = df["price"].fillna(df["price"].median())

# Regex extraction
df["area_code"] = df["phone"].[Link](r"\((\d{3})\)")

# Capping outliers (winsorizing)

lower, upper = df["value"].quantile([0.01, 0.99])
df["value"] = df["value"].clip(lower, upper)

8. Exploratory Data Analysis (EDA)

EDA is the process of visually and statistically summarizing data to uncover patterns, spot anomalies, test
assumptions, and check relationships between variables before modeling.

• Univariate analysis: examine one variable at a time (histograms, box plots, value counts).
• Bivariate analysis: relationships between two variables (scatter plots, cross-tabulation).
• Multivariate analysis: relationships among 3+ variables (pair plots, correlation heatmaps).
• Check class balance for classification targets.
• Detect multicollinearity between features (Variance Inflation Factor - VIF).
Python - quick EDA workflow
import pandas as pd
import [Link] as plt
import seaborn as sns

[Link](include="all")
[Link]().sum()
df["target"].value_counts(normalize=True)

[Link](df["age"], kde=True)
[Link](x="category", y="sales", data=df)
[Link](df, hue="target")
[Link]([Link](), annot=True)
[Link]()

9. Data Visualization
Chart Type Best For

Histogram Distribution of a single numeric variable

Box Plot Spread, median, and outliers across categories

Scatter Plot Relationship between two numeric variables

Line Chart Trends over time

Bar Chart Comparing values across categories

Heatmap Correlation matrices, tabular intensity data

Pie Chart Proportions of a whole (use sparingly)

Pair Plot Pairwise relationships across multiple variables

Python - visualization with matplotlib/seaborn/plotly

import [Link] as plt
import seaborn as sns
import [Link] as px

# Matplotlib
[Link](df["date"], df["sales"])
[Link]("Sales Over Time")
[Link]("Date"); [Link]("Sales")
[Link]()

# Seaborn
[Link](x="category", y="sales", data=df)

# Interactive Plotly
fig = [Link](df, x="age", y="income", color="segment", size="spend")
[Link]()

10. Feature Engineering & Selection

• Binning: converting continuous variables into categorical bins.
• Polynomial features: creating interaction/power terms.
• Date/time features: extracting day, month, weekday, is_weekend, etc.
• Text features: TF-IDF, word counts, embeddings.
• Aggregation features: rolling means, group-level statistics.
• Encoding: one-hot, label, target/mean encoding for categorical variables.
• Feature Selection: filter methods (correlation, chi-square), wrapper methods (RFE), embedded
methods (Lasso, tree feature importance).
Python - feature engineering examples
df["day_of_week"] = df["date"].[Link]
df["is_weekend"] = df["day_of_week"].isin([5, 6]).astype(int)
df["income_bucket"] = [Link](df["income"], bins=[0, 30000, 70000, float("inf")],
labels=["low", "mid", "high"])

# TF-IDF for text

from sklearn.feature_extraction.text import TfidfVectorizer
tfidf = TfidfVectorizer(max_features=500)
X_text = tfidf.fit_transform(df["review_text"])

# Recursive Feature Elimination

from sklearn.feature_selection import RFE
from sklearn.linear_model import LogisticRegression
selector = RFE(LogisticRegression(), n_features_to_select=10)
[Link](X, y)

11. Machine Learning Overview

Machine Learning is the modeling core of data science. It is grouped into Supervised Learning
(Regression: Linear/Ridge/Lasso, Random Forest, XGBoost; Classification: Logistic Regression, KNN,
Decision Trees, SVM, Naive Bayes), Unsupervised Learning (K-Means, Hierarchical Clustering, DBSCAN,
PCA), Reinforcement Learning (Q-Learning, DQN, PPO), and Deep Learning (ANN, CNN, RNN/LSTM,
Transformers).
Python - a typical modeling pipeline
from sklearn.model_selection import train_test_split
from [Link] import RandomForestClassifier
from [Link] import classification_report

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

model = RandomForestClassifier(n_estimators=200, random_state=42)

[Link](X_train, y_train)

y_pred = [Link](X_test)
print(classification_report(y_test, y_pred))
For a full breakdown of every ML algorithm with detailed theory and code, see the companion document:
Machine_Learning_Complete_Notes.pdf.

12. Time Series Analysis

• Trend: long-term increase or decrease in the data.
• Seasonality: repeating patterns at fixed intervals (daily, weekly, yearly).
• Stationarity: statistical properties (mean, variance) don't change over time - required by many models.
• Autocorrelation (ACF) / Partial Autocorrelation (PACF): correlation of a series with its own lags.
• ARIMA (AutoRegressive Integrated Moving Average): classical forecasting model.
• Prophet: Facebook's forecasting tool, robust to missing data and outliers.
• LSTM: deep learning approach for complex sequential patterns.
Python - Time series forecasting (ARIMA & Prophet)
import pandas as pd
from [Link] import ARIMA
from prophet import Prophet

# ARIMA
model = ARIMA(ts_data, order=(1, 1, 1))
fitted = [Link]()
forecast = [Link](steps=12)
# Prophet
df_prophet = [Link](columns={"date": "ds", "sales": "y"})
m = Prophet()
[Link](df_prophet)
future = m.make_future_dataframe(periods=30)
forecast = [Link](future)

13. Natural Language Processing (NLP)

• Tokenization: splitting text into words/subwords.
• Stopword Removal: filtering out common uninformative words (the, is, and).
• Stemming / Lemmatization: reducing words to their root form.
• Bag of Words / TF-IDF: converting text into numeric feature vectors.
• Word Embeddings: dense vector representations capturing semantic meaning (Word2Vec, GloVe).
• Transformers: attention-based models (BERT, GPT) that power modern NLP and LLMs.
• Sentiment Analysis, Named Entity Recognition (NER), Text Summarization, Machine Translation are
common NLP tasks.
Python - basic NLP pipeline
import nltk
from [Link] import stopwords
from [Link] import WordNetLemmatizer
from sklearn.feature_extraction.text import TfidfVectorizer

lemmatizer = WordNetLemmatizer()
stop_words = set([Link]("english"))

tokens = nltk.word_tokenize([Link]())
clean_tokens = [[Link](t) for t in tokens if [Link]() and t not in stop_words]

tfidf = TfidfVectorizer(max_features=1000)
X = tfidf.fit_transform(corpus)

# Using a transformer model (Hugging Face)

from transformers import pipeline
sentiment_pipe = pipeline("sentiment-analysis")
print(sentiment_pipe("This product is amazing!"))

14. A/B Testing & Experimentation

A/B testing compares two (or more) versions of a product/feature to determine which performs better on a
target metric, using randomized controlled experiments.

• Control Group vs Treatment Group: baseline vs the new variant.

• Sample Size Calculation: ensures the test has enough statistical power.
• Statistical Significance: p-value < 0.05 typically indicates a real effect.
• Statistical Power: probability of correctly detecting a true effect (commonly 80%).
• Guardrail Metrics: secondary metrics monitored to ensure no unintended harm.
Python - A/B test significance check
from scipy import stats
import numpy as np

# Conversion rate test (two-proportion z-test)

from [Link] import proportions_ztest

conversions = [Link]([180, 220]) # [control, treatment]

totals = [Link]([2000, 2000])

z_stat, p_value = proportions_ztest(conversions, totals)

print("z-stat:", z_stat, "p-value:", p_value)

15. Big Data Tools

Tool Purpose

Hadoop (HDFS + MapReduce) Distributed storage and batch processing of massive datasets

Apache Spark Fast, in-memory distributed data processing (batch & streaming)

Apache Kafka Real-time distributed event streaming/messaging

Apache Airflow Workflow orchestration for data pipelines (DAGs)

Hive SQL-like querying on top of Hadoop

Snowflake / BigQuery / Redshift Cloud-based data warehousing for large-scale analytics

Python - PySpark example

from [Link] import SparkSession

spark = [Link]("DataScienceApp").getOrCreate()
df = [Link]("large_data.csv", header=True, inferSchema=True)

[Link]("category").agg({"sales": "sum"}).show()

16. Model Deployment & MLOps

• Model Serialization: save trained models (pickle, joblib, ONNX).
• REST APIs: serve models via Flask/FastAPI for real-time predictions.
• Containerization: package models with Docker for consistent deployment.
• CI/CD for ML: automated testing and deployment pipelines.
• Model Monitoring: track prediction drift, data drift, and performance decay over time.
• Model Registry: version and manage models (MLflow, SageMaker Model Registry).
• A/B testing in production: gradually roll out new models and compare performance.
Python - saving a model & serving with FastAPI
import joblib
[Link](model, "[Link]")

# FastAPI serving
from fastapi import FastAPI
import joblib
import numpy as np

app = FastAPI()
model = [Link]("[Link]")

@[Link]("/predict")
def predict(features: list):
prediction = [Link]([Link](features).reshape(1, -1))
return {"prediction": [Link]()}
17. Data Ethics, Bias & Privacy
• Bias: models can reflect/amplify biases present in training data (historical, sampling, label bias).
• Fairness: ensuring model outcomes don't systematically disadvantage protected groups.
• Privacy: protecting personally identifiable information (PII); techniques include anonymization and
differential privacy.
• Transparency & Explainability: using tools like SHAP/LIME to explain model predictions.
• Regulations: GDPR, CCPA and similar laws govern data collection and usage.
• Data Governance: policies for data quality, access control, and lifecycle management.
Python - model explainability with SHAP
import shap

explainer = [Link](model)
shap_values = explainer.shap_values(X_test)
shap.summary_plot(shap_values, X_test)

18. Data Storytelling & Communication

• Know your audience: tailor technical depth to stakeholders vs technical peers.
• Lead with the insight, not the method - state the 'so what' up front.
• Use clear, minimal visualizations - avoid chart clutter and misleading axes.
• Build dashboards for ongoing monitoring (Tableau, Power BI, Looker, Streamlit).
• Support conclusions with confidence intervals / uncertainty, not just point estimates.
• Structure: Context -> Insight -> Evidence -> Recommendation -> Next Steps.
Python - quick interactive dashboard with Streamlit
import streamlit as st
import pandas as pd

[Link]("Sales Dashboard")
df = pd.read_csv("[Link]")

region = [Link]("Select Region", df["region"].unique())

filtered = df[df["region"] == region]

st.line_chart(filtered.set_index("date")["sales"])
[Link]([Link]())

19. Popular Data Science Tools & Libraries

Category Tools / Libraries

Programming Languages Python, R, SQL, Scala

Data Manipulation Pandas, NumPy, dplyr (R), Polars

Visualization Matplotlib, Seaborn, Plotly, Tableau, Power BI

Machine Learning scikit-learn, XGBoost, LightGBM, CatBoost

Deep Learning TensorFlow, Keras, PyTorch

Big Data Hadoop, Spark, Kafka, Hive

Databases PostgreSQL, MySQL, MongoDB, Snowflake, BigQuery

Notebooks/IDE Jupyter, Google Colab, VS Code, RStudio

MLOps MLflow, Docker, Kubernetes, Airflow, DVC

Cloud Platforms AWS (SageMaker), GCP (Vertex AI), Azure ML

20. Master Cheat Sheet
Task Go-To Tools / Methods

Query structured data SQL (SELECT, JOIN, GROUP BY, window functions)

Clean & transform data Pandas (fillna, drop_duplicates, apply, merge)

Explore data [Link](), histograms, box plots, correlation heatmaps

Test a hypothesis t-test, chi-square test, ANOVA, p-value < 0.05

Predict a number Linear Regression, Random Forest, XGBoost

Predict a category Logistic Regression, Random Forest, SVM, XGBoost

Segment customers K-Means, Hierarchical Clustering

Reduce dimensions PCA, t-SNE, UMAP

Forecast over time ARIMA, Prophet, LSTM

Analyze text TF-IDF, Naive Bayes, Transformers (BERT/GPT)

Run an experiment A/B test with two-proportion z-test or t-test

Process massive data Spark, Hadoop, distributed cloud warehouses

Deploy a model Flask/FastAPI + Docker, or a managed cloud ML platform

Explain a model SHAP, LIME, feature importance

Communicate results Dashboards (Tableau/Power BI/Streamlit), clear narrative

End of Notes - Data Science Complete Reference