Data Science, Data Analytics

& Artificial Intelligence /

Machine Learning
A Complete Guide: From Basics to Advanced

Covers Python · Statistics · SQL · ML · Deep Learning · NLP · MLOps · GenAI

2024 Edition | 550+ Concepts | Beginner to Expert

 Table of Contents
PART I: FOUNDATIONS
Chapter 1 — Introduction to Data Science & Analytics
Chapter 2 — Python for Data Science
Chapter 3 — Mathematics & Statistics Essentials
Chapter 4 — SQL & Database Fundamentals

PART II: DATA ANALYSIS

Chapter 5 — Data Collection & Web Scraping
Chapter 6 — Data Cleaning & Preprocessing
Chapter 7 — Exploratory Data Analysis (EDA)
Chapter 8 — Data Visualization

PART III: MACHINE LEARNING

Chapter 9 — Machine Learning Fundamentals
Chapter 10 — Supervised Learning
Chapter 11 — Unsupervised Learning
Chapter 12 — Model Evaluation & Tuning

PART IV: DEEP LEARNING & AI

Chapter 13 — Neural Networks & Deep Learning
Chapter 14 — Computer Vision with CNNs
Chapter 15 — Natural Language Processing
Chapter 16 — Generative AI & Large Language Models

PART V: ADVANCED TOPICS

Chapter 17 — Time Series Analysis & Forecasting
Chapter 18 — Reinforcement Learning
Chapter 19 — MLOps, Deployment & Production
Chapter 20 — Big Data & Cloud Platforms
Appendix — Resources, Libraries & Career Guide
 PART I

Foundations
Python · Statistics · SQL · Core Concepts
Chapter 1: Introduction to Data Science & Analytics

1.1 What Is Data Science?

Data Science is an interdisciplinary field that uses scientific methods, algorithms, processes, and
systems to extract knowledge and insights from structured and unstructured data. It blends elements of
statistics, computer science, domain expertise, and communication to turn raw data into actionable
intelligence.

The Data Science Venn Diagram

• Mathematics & Statistics — probability, inference, modelling

• Computer Science & Programming — algorithms, software engineering

• Domain Expertise — business context, subject-matter knowledge

The overlap of all three is where Data Science lives.

1.2 Data Science vs. Data Analytics vs. AI vs. ML

Term Focus Output Example

Data Analytics Past & present data Reports, dashboards Monthly sales KPIs

Data Science Past, present & future Predictions, models Customer churn model

Machine Learning Pattern learning from data Trained models Spam classifier

Artificial Intelligence Human-like intelligence Decisions, reasoning Self-driving car

Deep Learning Neural networks Feature learning Image recognition

1.3 The Data Science Lifecycle

• 1. Problem Definition — clearly state the business question
• 2. Data Collection — gather relevant structured/unstructured data
• 3. Data Cleaning — handle missing values, outliers, inconsistencies
• 4. Exploratory Data Analysis — understand distributions, patterns, correlations
• 5. Feature Engineering — create informative input variables for models
• 6. Modelling — select, train, and validate algorithms
• 7. Evaluation — measure performance against business metrics
• 8. Deployment — serve the model in production
• 9. Monitoring — track drift, retrain as needed
1.4 Key Roles in a Data Team
Role Primary Skills Typical Output

Data Analyst SQL, Excel, Tableau/Power BI Dashboards, reports

Data Scientist Python/R, ML, Statistics Predictive models

ML Engineer Python, Spark, Cloud, MLOps Production ML systems

Data Engineer SQL, Spark, Airflow, ETL Data pipelines

AI Researcher Math, PyTorch, Publications Novel algorithms

Chapter 2: Python for Data Science

2.1 Setting Up Your Environment

The standard data science environment uses Anaconda (a Python distribution bundling 250+
packages) or a virtual environment with pip. Jupyter Notebook / JupyterLab provides an interactive
coding experience.
# Install Anaconda or create a virtual environment pip install jupyterlab numpy pandas
matplotlib seaborn scikit-learn

2.2 Python Basics Refresher

2.2.1 Data Types & Variables
x = 42 # int y = 3.14 # float name = "Alice" # str flag = True # bool nums = [1,2,3] #
list pairs = {"a":1} # dict uniq = {1,2,3} # set t = (1,2) # tuple

2.2.2 Control Flow

for i in range(5): if i % 2 == 0: print(f'{i} is even') else: print(f'{i} is odd')

2.2.3 Functions & Lambda

def square(n): return n ** 2 double = lambda x: x * 2 result = list(map(double,
[1,2,3,4])) # [2,4,6,8]

2.3 NumPy – Numerical Computing

import numpy as np arr = [Link]([1,2,3,4,5]) print([Link](), [Link]()) # 3.0 1.414
matrix = [Link]((3,3)) # 3x3 zero matrix dot = [Link](arr, arr) # 55

2.4 Pandas – Data Manipulation

import pandas as pd # Load data df = pd.read_csv('[Link]') print([Link]()) # first 5
rows print([Link]()) # dtypes, nulls print([Link]()) # statistics # Selection &
filtering df[df['age'] > 30] # filter rows df[['name','age']] # select columns #
Aggregation [Link]('dept')['salary'].mean() # Merging merged = [Link](df1, df2,
on='id', how='left')

2.5 Essential Libraries Overview

Library Purpose Key Functions

NumPy Array computing array, dot, linalg, random

Pandas Data manipulation DataFrame, read_csv, groupby, merge

Library Purpose Key Functions

Matplotlib Plotting plot, scatter, hist, subplots

Seaborn Statistical plots heatmap, boxplot, pairplot

Scikit-learn Machine Learning fit, predict, Pipeline, GridSearchCV

SciPy Scientific computing stats, optimize, signal

Statsmodels Statistical modelling OLS, ARIMA, logit

Chapter 3: Mathematics & Statistics Essentials

3.1 Descriptive Statistics

• Mean (average) — sum of values divided by count
• Median — middle value; robust to outliers
• Mode — most frequent value
• Variance — average squared deviation from the mean
• Standard Deviation — square root of variance; same unit as data
• Skewness — asymmetry of distribution (positive = right tail)
• Kurtosis — 'peakedness'; high kurtosis = heavy tails

3.2 Probability Fundamentals

Probability measures how likely an event is, ranging from 0 (impossible) to 1 (certain).
• P(A or B) = P(A) + P(B) - P(A and B) [Addition Rule]
• P(A and B) = P(A) * P(B|A) [Multiplication Rule]
• P(A|B) = P(B|A)*P(A) / P(B) [Bayes' Theorem]

3.3 Key Probability Distributions

Distribution Use Case Parameters

Normal (Gaussian) Heights, errors, test scores mean (mu), std (sigma)

Binomial # successes in n trials n trials, p probability

Poisson Count of rare events/time lambda (rate)

Exponential Time between events lambda (rate)

Uniform Equal probability in range a (min), b (max)

Beta Probabilities, proportions alpha, beta

3.4 Hypothesis Testing

Hypothesis testing is a formal procedure to decide whether sample data provide enough evidence to
reject a null hypothesis (H0).
• Step 1: State H0 (null) and H1 (alternative) hypotheses
• Step 2: Choose significance level alpha (commonly 0.05)
• Step 3: Select and compute the test statistic
• Step 4: Find p-value — probability of observing the result under H0
 • Step 5: If p-value < alpha, reject H0

Common Tests

• t-test — compare means of one or two groups

• chi-squared — test independence of categorical variables

• ANOVA — compare means across 3+ groups

• Mann-Whitney U — non-parametric alternative to t-test

3.5 Linear Algebra for ML

• Scalars, Vectors, Matrices, Tensors — the building blocks of ML
• Matrix multiplication — core operation in neural networks
• Eigenvalues & Eigenvectors — used in PCA dimensionality reduction
• Dot product — similarity measure, basis of cosine similarity
• Gradient — direction of steepest ascent; flipped for gradient descent
Chapter 4: SQL & Database Fundamentals

4.1 Relational Database Concepts

A relational database stores data in tables (relations) with rows and columns. Tables are linked through
foreign keys. SQL (Structured Query Language) is the standard language to query and manipulate this
data.

4.2 Core SQL Syntax

-- Select specific columns SELECT name, salary, department FROM employees WHERE salary >
60000 AND department = 'Engineering' ORDER BY salary DESC LIMIT 10;

-- Aggregations SELECT department, COUNT(*) AS headcount, AVG(salary) AS avg_salary,

MAX(salary) AS top_salary FROM employees GROUP BY department HAVING COUNT(*) > 5;

4.3 JOINs
JOIN Type Returns

INNER JOIN Only matching rows in both tables

LEFT JOIN All rows from left + matches from right (NULL if no match)

RIGHT JOIN All rows from right + matches from left

FULL OUTER JOIN All rows from both tables

CROSS JOIN Cartesian product of both tables

SELECT [Link], d.dept_name FROM employees e INNER JOIN departments d ON e.dept_id = [Link];

4.4 Window Functions

SELECT name, salary, department, RANK() OVER (PARTITION BY department ORDER BY salary
DESC) AS dept_rank, AVG(salary) OVER (PARTITION BY department) AS dept_avg FROM employees;

4.5 Common Table Expressions (CTEs)

WITH high_earners AS ( SELECT * FROM employees WHERE salary > 100000 ), ranked AS ( SELECT
*, RANK() OVER (ORDER BY salary DESC) AS rnk FROM high_earners ) SELECT * FROM ranked
WHERE rnk <= 10;
 PART II

Data Analysis
Collection · Cleaning · EDA · Visualization
Chapter 5: Data Collection & Web Scraping

5.1 Data Sources

Source Type Examples Tools

CSV / Excel files Company exports pandas.read_csv / read_excel

Databases PostgreSQL, MySQL, SQLite SQLAlchemy, psycopg2

APIs Twitter, Weather, Finance requests, httpx

Web scraping News, e-commerce BeautifulSoup, Scrapy, Selenium

Streaming Kafka, IoT sensors Kafka-python, Faust

Public datasets Kaggle, UCI, [Link] Kaggle API, wget

5.2 REST APIs

import requests url = '[Link] params = {'q':
'London', 'appid': 'YOUR_API_KEY', 'units': 'metric'} resp = [Link](url,
params=params) data = [Link]() print(data['main']['temp']) # temperature in Celsius

5.3 Web Scraping with BeautifulSoup

from bs4 import BeautifulSoup import requests html =
[Link]('[Link] soup = BeautifulSoup(html, '[Link]') # Find
all article titles titles = [[Link] for h2 in soup.find_all('h2',
class_='article-title')] print(titles)

■ Always check a site's [Link] and Terms of Service before scraping. Use delays between requests.
Chapter 6: Data Cleaning & Preprocessing

6.1 Handling Missing Values

import pandas as pd [Link]().sum() # count missing per column [Link]() # drop rows
with any NaN [Link]([Link]()) # fill with column mean df['col'].fillna('Unknown') #
fill categorical with placeholder # Interpolation for time series
df['price'].interpolate(method='linear')

6.2 Outlier Detection & Treatment

• Z-score method: flag values more than 3 standard deviations from mean
• IQR method: flag values below Q1 - 1.5*IQR or above Q3 + 1.5*IQR
• Winsorizing: cap extreme values at the 1st and 99th percentiles
Q1 = df['col'].quantile(0.25) Q3 = df['col'].quantile(0.75) IQR = Q3 - Q1 clean =
df[(df['col'] >= Q1-1.5*IQR) & (df['col'] <= Q3+1.5*IQR)]

6.3 Encoding Categorical Variables

Technique When to Use Example

Label Encoding Ordinal categories Low=0, Med=1, High=2

One-Hot Encoding Nominal categories, low cardinality Gender -> [male, female]

Target Encoding High-cardinality categoricals City -> mean(target) per city

Binary Encoding High-cardinality, saves space Hash then binary digits

pd.get_dummies(df, columns=['city', 'category'], drop_first=True)

6.4 Feature Scaling

from [Link] import StandardScaler, MinMaxScaler # Standardization: mean=0,
std=1 (best for most ML) scaler = StandardScaler() X_scaled =
scaler.fit_transform(X_train) # Min-Max: scale to [0,1] (good for neural networks) mm =
MinMaxScaler() X_mm = mm.fit_transform(X_train)
Chapter 7: Exploratory Data Analysis (EDA)

7.1 The EDA Workflow

• 1. Understand shape & schema: [Link], [Link], [Link]()
• 2. Summary statistics: [Link](include='all')
• 3. Distribution of each feature: histograms, box plots
• 4. Relationships between features: scatter plots, correlation matrix
• 5. Target variable analysis: class balance, value distribution
• 6. Identify patterns, anomalies, hypotheses

7.2 Correlation Analysis

import seaborn as sns, [Link] as plt corr = [Link]() [Link](corr,
annot=True, cmap='coolwarm', fmt='.2f') [Link]('Feature Correlation Matrix') [Link]()

■ Correlation does NOT imply causation. A high correlation between two variables may be due to a
confounding third variable.

7.3 Univariate & Bivariate Analysis

# Univariate df['age'].hist(bins=30) # distribution df['age'].plot(kind='box') # outlier
view # Bivariate (numerical vs numerical) [Link](x='age', y='income',
hue='gender', data=df) # Bivariate (categorical vs numerical) [Link](x='department',
y='salary', data=df)
Chapter 8: Data Visualization

8.1 Choosing the Right Chart

Goal Best Chart Library

Distribution of one variable Histogram, KDE, Box plot Matplotlib/Seaborn

Compare categories Bar chart, Grouped bar Matplotlib/Plotly

Show relationship Scatter plot, Bubble chart Seaborn/Plotly

Show trend over time Line chart Matplotlib/Plotly

Show composition Pie chart, Stacked bar Matplotlib

Show correlation matrix Heatmap Seaborn

Geographic data Choropleth map Folium/Plotly

Multi-variable overview Pair plot, Parallel coords Seaborn/Plotly

8.2 Matplotlib Essentials

import [Link] as plt import numpy as np fig, axes = [Link](1, 2,
figsize=(12, 5)) # Line plot x = [Link](0, 10, 100) axes[0].plot(x, [Link](x),
color='blue', label='sin(x)') axes[0].plot(x, [Link](x), color='red', label='cos(x)')
axes[0].set_title('Trigonometric Functions') axes[0].legend() # Scatter plot
axes[1].scatter([Link](100), [Link](100), alpha=0.6)
axes[1].set_title('Random Scatter') plt.tight_layout() [Link]('[Link]', dpi=150)

8.3 Dashboard Tools

Tool Best For Language

Tableau Business BI dashboards No-code / drag & drop

Power BI Microsoft ecosystem DAX, M language

Plotly Dash Custom Python web apps Python

Streamlit Fast prototyping ML apps Python

Looker/Data Studio Google ecosystem LookML / SQL

Metabase Self-serve analytics SQL

 PART III

Machine Learning
Supervised · Unsupervised · Evaluation · Tuning
Chapter 9: Machine Learning Fundamentals

9.1 What Is Machine Learning?

Machine Learning (ML) is a subset of AI where systems learn from data to improve performance on a
task without being explicitly programmed. Instead of writing rules by hand, we feed examples to an
algorithm that discovers patterns automatically.

9.2 Types of Machine Learning

Type Description Examples

Supervised Learn from labelled (X,y) pairs Classification, Regression

Unsupervised Find structure in unlabelled data Clustering, Dimensionality Reduction

Semi-supervised Mix of labelled & unlabelled Label propagation

Reinforcement Agent learns via rewards Game playing, Robotics

Self-supervised Labels generated from data itself GPT pre-training, SimCLR

9.3 The ML Pipeline with Scikit-learn

from [Link] import Pipeline from [Link] import StandardScaler
from [Link] import RandomForestClassifier from sklearn.model_selection import
train_test_split from [Link] import classification_report # Split data X_train,
X_test, y_train, y_test = train_test_split( X, y, test_size=0.2, random_state=42,
stratify=y ) # Build pipeline pipe = Pipeline([ ('scaler', StandardScaler()), ('model',
RandomForestClassifier(n_estimators=100)) ]) # Train & evaluate [Link](X_train, y_train)
preds = [Link](X_test) print(classification_report(y_test, preds))

9.4 Bias-Variance Tradeoff

Key Concept

• High Bias (underfitting) — model too simple; fails on training data

• High Variance (overfitting) — model too complex; fails on new data

• Goal: find the sweet spot that minimises total error

Regularisation (L1/L2), cross-validation, and ensemble methods help manage this tradeoff.
Chapter 10: Supervised Learning

10.1 Linear Regression

Linear Regression models the relationship between features X and a continuous target y as a straight
line (or hyperplane in higher dimensions): y = w0 + w1*x1 + ... + wn*xn
from sklearn.linear_model import LinearRegression from [Link] import r2_score,
mean_squared_error import numpy as np model = LinearRegression() [Link](X_train,
y_train) preds = [Link](X_test) print('R2:', r2_score(y_test, preds))
print('RMSE:', [Link](mean_squared_error(y_test, preds)))

10.2 Logistic Regression

Despite its name, Logistic Regression is a classification algorithm. It models the probability of class
membership using the sigmoid function, outputting values between 0 and 1.
from sklearn.linear_model import LogisticRegression model = LogisticRegression(C=1.0,
max_iter=1000) [Link](X_train, y_train) probs = model.predict_proba(X_test)[:, 1] #
probability of class 1

10.3 Decision Trees

Decision Trees partition feature space into rectangular regions using a series of binary splits chosen to
maximise information gain or minimise Gini impurity.
• Interpretable and handles mixed data types well
• Prone to overfitting — addressed by limiting depth or pruning

10.4 Ensemble Methods

Method Idea Key Models

Bagging Train on bootstrap samples; average Random Forest

Boosting Sequentially correct errors XGBoost, LightGBM, AdaBoost

Stacking Use model outputs as features for meta-model StackingClassifier

from xgboost import XGBClassifier model = XGBClassifier( n_estimators=500,

learning_rate=0.05, max_depth=6, subsample=0.8, colsample_bytree=0.8,
use_label_encoder=False, eval_metric='logloss' ) [Link](X_train, y_train,
eval_set=[(X_test, y_test)], early_stopping_rounds=30, verbose=50)

10.5 Support Vector Machines

SVMs find the maximum-margin hyperplane that separates classes. With the kernel trick (RBF,
polynomial), they handle non-linear decision boundaries efficiently.
from [Link] import SVC model = SVC(kernel='rbf', C=10, gamma='scale',
probability=True) [Link](X_train, y_train)
Chapter 11: Unsupervised Learning

11.1 K-Means Clustering

K-Means partitions n data points into k clusters by minimising within-cluster variance. The algorithm
alternates between assigning points to the nearest centroid and recomputing centroids.
from [Link] import KMeans from [Link] import silhouette_score # Find
optimal k using elbow method inertias = [] for k in range(2, 11): km =
KMeans(n_clusters=k, random_state=42, n_init=10) [Link](X) [Link](km.inertia_) #
Fit chosen k km = KMeans(n_clusters=4, random_state=42) labels = km.fit_predict(X)
print('Silhouette Score:', silhouette_score(X, labels))

11.2 Hierarchical & DBSCAN Clustering

Algorithm Pros Cons

K-Means Fast, scalable Needs k upfront; spherical clusters

DBSCAN Finds arbitrary shapes; noise detection Struggles with varying density

Hierarchical No k needed; dendrogram O(n^3) time; not scalable

Gaussian Mixture Soft assignments; probabilistic More parameters to tune

11.3 Principal Component Analysis (PCA)

PCA reduces dimensionality by projecting data onto orthogonal axes (principal components) that
capture maximum variance. It is used for visualisation, noise reduction, and speeding up downstream
models.
from [Link] import PCA import [Link] as plt pca =
PCA(n_components=2) X_2d = pca.fit_transform(X_scaled) print('Variance explained:',
pca.explained_variance_ratio_.sum()) [Link](X_2d[:,0], X_2d[:,1], c=y,
cmap='viridis') [Link]('PC1'); [Link]('PC2') [Link]('PCA Projection')
Chapter 12: Model Evaluation & Hyperparameter Tuning

12.1 Classification Metrics

Metric Formula Use When

Accuracy (TP+TN)/(Total) Balanced classes

Precision TP/(TP+FP) Cost of false positives is high

Recall TP/(TP+FN) Cost of false negatives is high (e.g. cancer detection)

F1 Score 2*P*R/(P+R) Imbalanced classes; balance P & R

ROC-AUC Area under ROC curve Overall ranking quality of model

PR-AUC Area under Precision-Recall Imbalanced datasets

12.2 Cross-Validation
from sklearn.model_selection import StratifiedKFold, cross_val_score cv =
StratifiedKFold(n_splits=5, shuffle=True, random_state=42) scores =
cross_val_score(model, X, y, cv=cv, scoring='roc_auc') print(f'CV AUC: {[Link]():.4f}
+/- {[Link]():.4f}')

12.3 Hyperparameter Tuning

from sklearn.model_selection import GridSearchCV, RandomizedSearchCV # Grid Search
param_grid = {'max_depth': [3,5,7], 'n_estimators': [100,200,300]} grid =
GridSearchCV(RandomForestClassifier(), param_grid, cv=5, scoring='f1') [Link](X_train,
y_train) print('Best params:', grid.best_params_) # Bayesian Optimisation (optuna) import
optuna def objective(trial): n = trial.suggest_int('n_estimators', 50, 500) d =
trial.suggest_int('max_depth', 2, 10) m = RandomForestClassifier(n_estimators=n,
max_depth=d) return cross_val_score(m, X_train, y_train, cv=3, scoring='f1').mean() study
= optuna.create_study(direction='maximize') [Link](objective, n_trials=50)
 PART IV

Deep Learning & AI

Neural Networks · CNNs · NLP · Generative AI
Chapter 13: Neural Networks & Deep Learning

13.1 The Neuron & Perceptron

An artificial neuron computes a weighted sum of its inputs, adds a bias, and passes the result through
a non-linear activation function: output = activation(w1x1 + w2x2 + ... + wnxn + b)

13.2 Activation Functions

Activation Formula Common Use

Sigmoid 1/(1+e^-x) Binary output layer

Softmax exp(xi)/sum(exp(xj)) Multi-class output layer

ReLU max(0, x) Hidden layers (default)

Leaky ReLU max(0.01x, x) Avoids dying ReLU

Tanh (e^x - e^-x)/(e^x + e^-x) RNNs, zero-centred

GELU x*Phi(x) Transformers (BERT, GPT)

13.3 Building a Neural Network with Keras

import tensorflow as tf from tensorflow import keras model = [Link]([
[Link](256, activation='relu', input_shape=(n_features,)),
[Link](0.3), [Link](128, activation='relu'),
[Link](0.3), [Link](1, activation='sigmoid') # binary
classification ]) [Link]( optimizer=[Link](learning_rate=1e-3),
loss='binary_crossentropy', metrics=['accuracy', [Link]()] ) history =
[Link]( X_train, y_train, validation_data=(X_val, y_val), epochs=50, batch_size=64,
callbacks=[[Link](patience=5, restore_best_weights=True)] )

13.4 Backpropagation & Gradient Descent

• Forward pass: compute predictions and loss
• Backward pass: compute gradients via chain rule
• Parameter update: w = w - learning_rate * gradient
• Optimisers — SGD, Adam (adaptive), RMSProp
• Batch sizes — full batch (slow), mini-batch (typical), SGD (noisy)
Chapter 14: Computer Vision with CNNs

14.1 Convolutional Layers

A Convolutional Neural Network (CNN) applies learnable filters (kernels) across the spatial dimensions
of an image to detect features like edges, textures, and objects. Each filter produces a feature map.
• Convolution — slides a kernel over the image; dot product at each position
• Pooling — downsamples feature maps (MaxPool, AvgPool)
• Stride — step size of the kernel; stride>1 reduces spatial size
• Padding — adds border pixels to preserve spatial dimensions

14.2 Famous CNN Architectures

Architecture Year Key Innovation

LeNet-5 1998 First successful CNN for digits

AlexNet 2012 Deep CNN; ReLU; GPU training

VGGNet 2014 Very deep (16-19 layers); 3x3 convolutions

ResNet 2015 Residual connections; 152 layers

EfficientNet 2019 Compound scaling; SOTA efficiency

Vision Transformer (ViT) 2020 Patches as tokens; pure attention

14.3 Transfer Learning

Transfer Learning reuses a model pre-trained on a large dataset (e.g. ImageNet) as the starting point
for a related task. This dramatically reduces training time and data requirements.
from [Link] import EfficientNetB0 from [Link] import
layers, Model base = EfficientNetB0(include_top=False, weights='imagenet',
input_shape=(224, 224, 3)) [Link] = False # Freeze pretrained weights x =
[Link] x = layers.GlobalAveragePooling2D()(x) x = [Link](256,
activation='relu')(x) output = [Link](num_classes, activation='softmax')(x) model =
Model(inputs=[Link], outputs=output)
Chapter 15: Natural Language Processing (NLP)

15.1 NLP Tasks Overview

Task Description Example

Text Classification Assign category to text Spam detection, sentiment

Named Entity Recognition Find entities in text Extract names, dates, places

Machine Translation Translate between languages English to French

Summarisation Condense long text News article summary

Question Answering Answer from context Reading comprehension

Text Generation Produce coherent text ChatGPT, story writing

15.2 Text Preprocessing

import re from [Link] import stopwords from [Link] import PorterStemmer def
clean_text(text): text = [Link]() text = [Link](r'[^a-z\s]', '', text) # remove
punctuation tokens = [Link]() tokens = [t for t in tokens if t not in
[Link]('english')] stemmer = PorterStemmer() tokens = [[Link](t) for t in
tokens] return ' '.join(tokens)

15.3 Word Embeddings

• Bag of Words — simple word count vector; sparse; ignores order
• TF-IDF — weights words by importance; reduces common-word noise
• Word2Vec — dense 300-d vectors; captures word semantics
• GloVe — global co-occurrence statistics; fast training
• FastText — sub-word representations; handles OOV words
• BERT embeddings — contextualised; same word, different meaning

15.4 Transformers & BERT

The Transformer architecture (Vaswani et al., 2017) uses self-attention to process all tokens in parallel,
capturing long-range dependencies. BERT (Bidirectional Encoder) is pre-trained on masked language
modelling and next-sentence prediction.
from transformers import pipeline # Sentiment analysis classifier =
pipeline('sentiment-analysis') result = classifier('This movie was absolutely fantastic!')
print(result) # [{'label': 'POSITIVE', 'score': 0.9998}] # Named Entity Recognition ner =
pipeline('ner', grouped_entities=True) ner('Elon Musk founded SpaceX in Hawthorne,
California.')
Chapter 16: Generative AI & Large Language Models

16.1 What Are LLMs?

Large Language Models (LLMs) are transformer-based models trained on trillions of tokens of text.
They learn to predict the next token, and in doing so develop rich world knowledge, reasoning, and
language generation capabilities.

16.2 Key LLM Families (2024)

Model Family Creator Strengths

GPT-4o OpenAI Multimodal; strong reasoning

Claude 3 Opus Anthropic Long context; safety-focused

Gemini Ultra Google DeepMind Multimodal; code & science

Llama 3 Meta Open weights; strong open-source baseline

Mistral Mistral AI Efficient; strong instruction following

Falcon TII Open; Arabic + English

16.3 Prompt Engineering

• Zero-shot: just describe the task with no examples
• Few-shot: provide 2-5 input/output examples in the prompt
• Chain-of-thought: ask the model to reason step-by-step
• ReAct: Reasoning + Acting; model calls tools and reflects
• System prompt: sets the persona, constraints, and output format

16.4 Fine-tuning & RAG

Technique When to Use Cost

Prompt Engineering Quick task specification Free

RAG Ground answers in private documents Low-Medium

LoRA / QLoRA Adapt model to new style/domain Medium

Full Fine-tuning Maximum control; task specialisation High

RLHF Align to human preferences Very High

 PART V

Advanced Topics
Time Series · RL · MLOps · Big Data · Cloud
Chapter 17: Time Series Analysis & Forecasting

17.1 Time Series Components

• Trend — long-term increase or decrease
• Seasonality — repeating patterns at fixed periods (daily, weekly, yearly)
• Cyclicality — irregular multi-year fluctuations
• Residual/Noise — random variation after removing above components

17.2 Classical Models

Model Description Use Case

Naive Forecast Last value = next value Baseline

Moving Average Average of last n periods Smooth noisy series

ARIMA AutoRegressive Integrated Moving Average Stationary univariate

SARIMA Seasonal ARIMA Series with seasonality

Exponential Smoothing Weighted average; more weight to recent Trending data

Prophet Additive model by Meta; handles holidays Business KPIs

from [Link] import ARIMA model = ARIMA(train, order=(2,1,2)) results

= [Link]() forecast = [Link](steps=30) print(forecast)

17.3 ML/DL Forecasting

• LSTM — learns sequential dependencies; good for long sequences
• Temporal Fusion Transformer — SOTA; handles multiple time series
• N-BEATS — pure DL for univariate; interpretable decomposition
• LightGBM with lag features — often fastest to prototype and competitive
Chapter 18: Reinforcement Learning

18.1 Core Concepts

Term Definition

Agent The learner/decision-maker

Environment The world the agent interacts with

State Current situation of the agent

Action Choice the agent makes

Reward Scalar feedback signal

Policy Mapping from states to actions

Value Function Expected cumulative reward from a state

Q-Value Expected reward for action a in state s

18.2 Key RL Algorithms

Algorithm Type Famous Application

Q-Learning Model-free, value-based Atari games (DQN)

SARSA On-policy Q-learning Gridworld navigation

PPO Policy gradient ChatGPT RLHF, robotics

SAC Off-policy, continuous actions Robot locomotion

AlphaZero MCTS + RL Chess, Go, Shogi

Chapter 19: MLOps, Deployment & Production

19.1 The MLOps Lifecycle

• 1. Version Control — Git for code; DVC for data and models
• 2. Experiment Tracking — MLflow, Weights & Biases
• 3. Feature Store — Feast, Tecton; reuse features across models
• 4. Model Registry — store, version, and stage models
• 5. CI/CD — automated testing and deployment pipelines
• 6. Serving — REST API (FastAPI), batch jobs, streaming
• 7. Monitoring — data drift, concept drift, model performance

19.2 Serving a Model with FastAPI

from fastapi import FastAPI import joblib import numpy as np from pydantic import
BaseModel app = FastAPI() model = [Link]('[Link]') class InputData(BaseModel):
features: list[float] @[Link]('/predict') def predict(data: InputData): X =
[Link]([Link]).reshape(1, -1) pred = [Link](X)[0] prob =
model.predict_proba(X)[0].max() return {'prediction': int(pred), 'confidence':
round(float(prob), 4)}

19.3 Containerisation with Docker

# Dockerfile FROM python:3.11-slim WORKDIR /app COPY [Link] . RUN pip install -r
[Link] COPY . . EXPOSE 8000 CMD ["uvicorn", "main:app", "--host", "[Link]",
"--port", "8000"] # Build & run # docker build -t my-ml-api . # docker run -p 8000:8000
my-ml-api
Chapter 20: Big Data & Cloud Platforms

20.1 Big Data Technologies

Technology Role Key Concept

HDFS Distributed file storage Commodity hardware clusters

Apache Spark Distributed computing RDDs, DataFrames, lazy evaluation

Apache Kafka Event streaming Topics, partitions, consumers

Apache Airflow Workflow orchestration DAGs, tasks, operators

Delta Lake Reliable data lake ACID transactions on Parquet

Apache Iceberg Table format Time-travel, schema evolution

20.2 PySpark Quickstart

from [Link] import SparkSession from [Link] import col, avg, count
spark = [Link]('DataScience').getOrCreate() # Load data df =
[Link]('s3://bucket/[Link]', header=True, inferSchema=True) # Transform result =
([Link](col('age') > 25) .groupBy('department')
.agg(avg('salary').alias('avg_salary'), count('*').alias('headcount'))
.orderBy(col('avg_salary').desc())) [Link]()

20.3 Cloud ML Platforms

Cloud ML Service Storage Compute

AWS SageMaker S3, Redshift EC2, Lambda

Google Cloud Vertex AI GCS, BigQuery GCE, Cloud Run

Azure Azure ML Blob, Synapse AKS, Azure Functions

Databricks MLflow + Spark Delta Lake Autoscaling clusters

Appendix: Resources, Libraries & Career Guide

A.1 Essential Python Libraries

Library Version (2024) Purpose

NumPy 1.26 Numerical arrays

Pandas 2.2 Data manipulation

Scikit-learn 1.4 Classical ML

TensorFlow/Keras 2.16 Deep Learning

PyTorch 2.3 Research DL

XGBoost 2.0 Gradient boosting

LightGBM 4.3 Fast gradient boosting

HuggingFace Transformers 4.40 NLP & LLMs

Plotly 5.21 Interactive visualisation

Streamlit 1.35 ML web apps

MLflow 2.13 Experiment tracking

FastAPI 0.111 Model serving

A.2 Free Learning Resources

• [Link] — practical deep learning for coders (free)
• Kaggle Learn — short micro-courses on ML, DL, SQL, Python
• Google ML Crash Course — ML fundamentals with TensorFlow
• Stanford CS229 (YouTube) — Andrew Ng's ML course
• Deep Learning Specialisation (Coursera) — Andrew Ng, [Link]
• Hugging Face NLP Course — modern NLP with Transformers
• Full Stack Deep Learning — production ML systems

A.3 Recommended Books

• Hands-On ML with Scikit-Learn, Keras & TensorFlow — Aurélien Géron
• The Elements of Statistical Learning — Hastie, Tibshirani, Friedman (free PDF)
• Deep Learning — Goodfellow, Bengio, Courville (free online)
• Python for Data Analysis — Wes McKinney (pandas creator)
• Designing ML Systems — Chip Huyen (MLOps best practices)
A.4 Career Roadmap
Stage Skills to Build Target Roles

0-3 months Python, SQL, Statistics basics —

3-6 months Pandas, Matplotlib, EDA Junior Data Analyst

6-12 months Scikit-learn, ML fundamentals Data Analyst / Junior DS

1-2 years DL, NLP, MLOps, Cloud Data Scientist

2-4 years Distributed systems, LLMs, Research Senior DS / ML Engineer

4+ years Architecture, strategy, leadership Principal / Director

Final Note
Data Science is a rapidly evolving field. The best practitioners combine solid mathematical intuition with
strong software engineering skills and clear communication. Stay curious, build projects, participate in
Kaggle competitions, contribute to open source, and never stop learning.

Good luck on your data science journey!