Python for ML — Complete Notes

PYTHON FOR ML
The Complete Pre-ML Library Playbook
NumPy · Pandas · Matplotlib · Seaborn · SciPy · Scikit-learn

NumPy Pandas Matplotlib/Seaborn Scikit-learn

Arrays & Math Data Wrangling Visualization ML API

Prepared for: IIT Kharagpur placement prep — Data Analyst / Business Analyst / Product Analyst / Product
Management tracks.

How to use this document: it is written assuming zero prior exposure to these libraries, but goes deep enough
(internals, gotchas, complexity, real interview questions) to hold up in a rigorous IIT-level interview. Read
topic-wise, run every code snippet yourself, and use the Interview Questions block at the end of each chapter as
active recall.

Page 1
 Python for ML — Complete Notes

0. Why These Libraries, and In What Order

Before any machine learning model is trained, ~80% of real work is data handling. The 'pre-ML stack' has a
natural dependency order:

Library Role Depends on

Fast numerical arrays & math — the foundation

NumPy Pure Python
everything else is built on

Labeled, tabular data (rows/columns) — built on top of

Pandas NumPy
NumPy arrays

Low-level plotting engine — draws every chart you'll

Matplotlib NumPy
ever make

Statistical plotting — a friendlier layer on top of

Seaborn Matplotlib, Pandas
Matplotlib, understands DataFrames

SciPy (stats) Probability distributions, hypothesis tests, optimization NumPy

Preprocessing, model training, evaluation — the

Scikit-learn NumPy, SciPy, (Pandas)
standard ML toolkit

Suggested learning order

●​ NumPy first — everything else silently assumes you understand arrays, shapes, and vectorization.
●​ Pandas second — 90% of a Data/Business/Product Analyst's daily work lives here.
●​ Matplotlib + Seaborn third — you cannot do EDA (exploratory data analysis) without plotting.
●​ [Link] fourth — needed to formally test hypotheses (A/B tests, significance) common in Product
Analyst interviews.
●​ Scikit-learn last — by the time you reach it, you already know arrays, tables, and plots; you just learn the
modeling API.

Why Product/Business Analyst candidates still need this stack

Even 'non-core-tech' analytics and PM interviews at top firms (Amazon, Flipkart, Uber, Meesho, consulting
analytics wings) increasingly include a live Python/SQL round, a case study requiring you to manipulate a CSV, or
a take-home in a Jupyter notebook. Interviewers also ask conceptual questions ('what is vectorization and why
does it matter for a 10M-row dataset?') to test whether you understand tools or just memorized syntax.

Page 2
 Python for ML — Complete Notes

1. NumPy — Numerical Python

NumPy (Numerical Python) provides the ndarray — a fast, fixed-type, multi-dimensional array — plus vectorized
math operations implemented in C. It is the substrate every other data/ML library (Pandas, Matplotlib,
scikit-learn, TensorFlow, PyTorch) is built on.

1.1 Why not just use Python lists?

●​ Python lists are arrays of pointers to arbitrary objects — every element access involves pointer
dereferencing and type checking. NumPy arrays store raw, contiguous, fixed-type data (like a C array), so
operations run in compiled C loops, not the Python interpreter loop.
●​ NumPy supports vectorization — applying an operation to an entire array at once, with no explicit
Python-level for-loop.
●​ NumPy arrays use far less memory (no per-element Python object overhead).
●​ NumPy supports broadcasting (see §1.6) to combine arrays of different shapes without manual loops.
PYTHON
import numpy as np, time
n = 5_000_000
py_list = list(range(n))
np_arr = [Link](n)

t0=[Link](); py_result=[x*2 for x in py_list]; print('list:', [Link]()-t0)

t0=[Link](); np_result=np_arr*2; print('numpy:', [Link]()-t0)
# numpy is typically 20-100x faster for numeric workloads

1.2 Creating Arrays

PYTHON
import numpy as np

a = [Link]([1, 2, 3]) # 1-D array from a list

b = [Link]([[1,2,3],[4,5,6]]) # 2-D array (matrix)
c = [Link]((2,3)) # array of zeros, shape (2,3)
d = [Link]((3,3)) # array of ones
e = [Link]((2,2), 7) # filled with a constant
f = [Link](3) # 3x3 identity matrix
g = [Link](0, 10, 2) # [0 2 4 6 8] (like range())
h = [Link](0, 1, 5) # 5 evenly spaced points in [0,1]
i = [Link](2,3) # uniform [0,1) random array
j = [Link](2,3) # standard normal random array
k = [Link](0, 10, size=(2,3)) # random ints in [0,10)
l = [Link]((2,2)) # uninitialized (garbage) values — fast allocation

Key array attributes

Example (arr =
Attribute Meaning
[Link]([[1,2,3],[4,5,6]]))

[Link] Tuple of dimension sizes (2, 3)

Page 3
 Python for ML — Complete Notes

Example (arr =
Attribute Meaning
[Link]([[1,2,3],[4,5,6]]))

[Link] Number of dimensions 2

[Link] Total number of elements 6

[Link] Data type of elements dtype('int64')

[Link] Bytes per element 8

[Link] Total bytes (size × itemsize) 48

arr.T Transposed view shape becomes (3, 2)

1.3 Data Types (dtype)

Every NumPy array has a single, fixed dtype — this is the key difference from Python lists, which can hold mixed
types. Common dtypes: int8/16/32/64, uint8..64, float16/32/64, bool, complex64/128, object (fallback, loses
speed benefits), <U (unicode string).
PYTHON
a = [Link]([1, 2, 3], dtype=np.float32)
[Link](np.int64) # cast to another dtype (returns a new array)
[Link]([1, 'two', 3.0]) # dtype becomes '<U32' — everything upcast to string!
[Link]([1, 2, 3.5]) # dtype becomes float64 — int upcast to float

Interview gotcha
Mixing types in [Link]() triggers upcasting to the 'most general' common type (int -> float -> complex -> string
-> object). This silently changes behavior — e.g. integer division becomes float division. Always check .dtype after
creating arrays from mixed sources.

1.4 Indexing & Slicing

PYTHON
a = [Link]([10,20,30,40,50])
a[0] # 10
a[-1] # 50 (last element)
a[1:4] # [20 30 40] (stop index excluded, like Python lists)
a[::2] # [10 30 50] (step of 2)
a[::-1] # [50 40 30 20 10] (reversed)

m = [Link]([[1,2,3],[4,5,6],[7,8,9]])
m[1,2] # 6 -> row 1, col 2
m[1] # [4 5 6] -> entire row 1
m[:,1] # [2 5 8] -> entire column 1
m[0:2, 1:3] # [[2 3],[5 6]] -> sub-matrix (rows 0-1, cols 1-2)

Boolean masking & fancy indexing

PYTHON
a = [Link]([1,2,3,4,5,6])
mask = a > 3 # array([False, False, False, True, True, True])

Page 4
 Python for ML — Complete Notes
a[mask] # array([4, 5, 6])
a[a % 2 == 0] # array([2, 4, 6]) -- even numbers
a[a > 3] = 0 # conditional assignment -> array([1,2,3,0,0,0])

idx = [0, 2, 4]
a[idx] # fancy indexing -> picks elements at positions 0,2,4

[Link](a > 3, 'high', 'low') # element-wise conditional, like Excel IF()

View vs Copy — the #1 NumPy bug source

Basic slicing (a[1:4]) returns a VIEW — it shares memory with the original array, so modifying the slice modifies
the original! Fancy indexing (a[[0,2,4]]) and boolean masking (a[a>3]) always return a COPY. Use .copy() explicitly
whenever you need an independent array: b = a[1:4].copy().

1.5 Reshaping & Combining Arrays

PYTHON
a = [Link](12)
[Link](3,4) # reshape to 3 rows, 4 cols (must have same total size)
[Link](3,-1) # -1 means 'infer this dimension automatically'
[Link](-1,1) # turn a 1-D array into a column vector, shape (12,1)
[Link]() # collapse to 1-D — always returns a COPY
[Link]() # collapse to 1-D — returns a VIEW when possible (faster)

x = [Link]([1,2,3]); y = [Link]([4,5,6])
[Link]([x,y]) # [1 2 3 4 5 6]
[Link]([x,y]) # stack as rows -> [[1 2 3],[4 5 6]]
[Link]([x,y]) # horizontal stack -> [1 2 3 4 5 6]
[Link]([x,y], axis=0) # like vstack but creates a new axis
[Link]([Link](9), 3) # split into 3 equal parts

1.6 Broadcasting — the most important NumPy concept

Broadcasting lets NumPy perform element-wise operations on arrays of different (but compatible) shapes
without writing explicit loops or copying data. Two dimensions are compatible when they are equal, or one of
them is 1.

PYTHON
a = [Link]([[1,2,3],[4,5,6],[7,8,9]]) # shape (3,3)

Page 5
 Python for ML — Complete Notes
b = [Link]([10,20,30]) # shape (3,) -- treated as (1,3)
a + b
# [[11 22 33]
# [14 25 36]
# [17 28 39]] -- b's row is 'stretched' across all 3 rows of a

# Broadcasting rule (compare shapes from the RIGHT):

# (3,3) and (3,) -> align as (3,3) vs (1,3) -> compatible, result (3,3)
# (3,1) and (1,4) -> compatible, result (3,4)
# (3,4) and (3,) -> incompatible! ValueError: operands could not be broadcast together

Real use-case
Normalizing a dataset: (X - [Link](axis=0)) / [Link](axis=0) subtracts a (n_features,) mean vector from every row
of an (n_samples, n_features) matrix in one broadcasted line — this exact line appears constantly in ML
preprocessing and interview whiteboard questions.

1.7 Vectorized Operations & Universal Functions (ufuncs)

PYTHON
a = [Link]([1,2,3,4])
a + 5, a - 5, a * 2, a / 2, a ** 2, a % 2
[Link](a); [Link](a); [Link](a); [Link](a)
[Link](a); [Link](a)

x = [Link]([1,2,3]); y = [Link]([4,5,6])
x + y # element-wise addition -> [5 7 9]
x * y # element-wise (Hadamard) product -> [4 10 18]
x @ y # DOT PRODUCT (matrix mult operator) -> 32
[Link](x,y) # same as x @ y

Interview trap
'*' is element-wise multiplication, NOT matrix multiplication. Use @ or [Link]()/[Link]() for true matrix
multiplication. This is one of the most commonly asked distinctions in ML interviews.

1.8 Aggregations & the axis parameter

PYTHON
m = [Link]([[1,2,3],[4,5,6]]) # shape (2,3)
[Link]() # 21 -- sums ALL elements
[Link](axis=0) # [5 7 9] -- sums DOWN each column (collapses rows)
[Link](axis=1) # [6 15] -- sums ACROSS each row (collapses columns)
[Link](), [Link](), [Link](), [Link](), [Link]()
[Link](), [Link]() # INDEX of min/max (flattened by default)
[Link]() # cumulative sum -> [1 3 6 10 15 21]
[Link](m, 50) # median via percentile
[Link](m)

Remember axis this way

Page 6
 Python for ML — Complete Notes

axis=0 moves DOWN the rows (result has one value per COLUMN). axis=1 moves ACROSS the columns (result has
one value per ROW). This trips up almost everyone at first — verify with a small example every time you're
unsure.

1.9 Linear Algebra ([Link])

PYTHON
A = [Link]([[1,2],[3,4]])
A.T # transpose
[Link](A) # matrix inverse
[Link](A) # determinant
[Link](A) # eigenvalues & eigenvectors
[Link](A) # Frobenius norm (or vector norm for 1-D)
[Link](A, b) # solves Ax = b directly (faster & more stable than inv(A) @ b)
[Link](A) # sum of diagonal elements

1.10 Random module & reproducibility

PYTHON
rng = [Link].default_rng(seed=42) # modern, recommended generator
[Link](5) # 5 uniform floats in [0,1)
[Link](0, 10, 5) # 5 random ints in [0,10)
[Link](0, 1, 5) # 5 samples from N(0,1)
[Link]([1,2,3,4], size=3, replace=False) # random sample without replacement
[Link](arr) # in-place shuffle

# Legacy API (still very common in interviews / older code):

[Link](42) # sets global seed for reproducibility
[Link](3)

Why set a seed?

Setting a seed makes 'random' results reproducible — critical for debugging and for interviewers/graders to
reproduce your exact output. Always seed before train_test_split, bootstrapping, or any simulation you show in
an interview.

1.11 NaN, Inf, and Handling Missing Numeric Data

PYTHON
a = [Link]([1, [Link], 3, [Link], -[Link]])
[Link](a) # [False True False False False]
[Link](a) # [False False False True True]
[Link](a) # mean ignoring NaNs
[Link](a) # sum ignoring NaNs
a[[Link](a)] = 0 # replace NaNs with 0

Gotcha
[Link] != [Link] (NaN is never equal to itself, even to itself — it's part of the IEEE-754 float spec). Never write `if
x == [Link]`; always use [Link](x).

Page 7
 Python for ML — Complete Notes

1.12 Vectorization vs [Link] vs loops — performance ranking

Approach Speed When to use

Always prefer this — pure C loops

Native vectorized ops (a + b, [Link]()) Fastest
under the hood

Only for readability on small data;

[Link](func) Slow (just a for-loop in disguise)
NOT a performance tool

Avoid on large arrays — defeats the

Python for-loop over array elements Slowest
purpose of NumPy

PYTHON
# Bad (slow):
result = [Link]([x**2 if x>0 else 0 for x in arr])

# Good (vectorized):
result = [Link](arr>0, arr**2, 0)

1.13 NumPy — Interview Questions

Q1. Why is NumPy faster than plain Python lists for numeric computation?
NumPy arrays store homogeneous, fixed-type data contiguously in memory and push loops down into compiled C
code (vectorization), avoiding per-element Python object overhead, type dispatch, and interpreter loop overhead
that plain lists incur.

Q2. What's the difference between a view and a copy in NumPy? Give an example.
A view shares the underlying data buffer with the original array — modifying it modifies the original (e.g., basic
slicing arr[1:4]). A copy is an independent array with its own memory (e.g., fancy indexing arr[[1,3]], boolean
masking arr[arr>0], or explicit .copy()). Use np.shares_memory(a,b) to check.

Q3. Explain broadcasting rules with an example where it fails.

NumPy compares shapes from the trailing (rightmost) dimension; dimensions are compatible if equal or one of
them is 1. Example failure: shape (3,4) and shape (3,) are NOT broadcastable directly, because comparing from the
right gives 4 vs 3 (neither equal nor 1) -> ValueError. You'd need to reshape the (3,) array to (3,1) first.

Page 8
 Python for ML — Complete Notes

Q4. What is the difference between axis=0 and axis=1 in an aggregation?

axis=0 collapses/aggregates down the rows, producing one result per column. axis=1 collapses across the columns,
producing one result per row.

Q5. How do you multiply two matrices element-wise vs as true matrix multiplication?
Element-wise: A * B (Hadamard product), requires identical shapes (or broadcastable). True matrix multiplication:
A @ B or [Link](A,B) or [Link](A,B), requires inner dimensions to match (m,n) @ (n,p) -> (m,p).

Q6. Why does [Link]([1, 2, 'three']) turn all elements into strings?
NumPy arrays require a single dtype for all elements. When mixed types are given, NumPy upcasts to the most
general common type that can represent all values — here, string, since integers can always be represented as
strings but not vice versa.

Q7. How would you efficiently normalize a (n_samples, n_features) matrix without loops?
(X - [Link](axis=0)) / [Link](axis=0) — this relies on broadcasting to subtract a (n_features,) row vector from every
row of X and divide similarly, entirely vectorized.

Q8. What's the time complexity of [Link]()? What algorithm does it use by default?
O(n log n) average case. Default is 'quicksort' (technically introsort), but 'mergesort' (stable) and 'heapsort' are also
available via the kind parameter — pick mergesort when a stable sort is required.

Q9. Why should you avoid [Link] for performance?

[Link] is essentially a thin wrapper around a Python-level for-loop — it improves code
readability/broadcasting-compatibility but gives none of NumPy's C-loop speed benefit. True speed requires native
vectorized ufuncs.

Q10. How do you handle NaNs when computing a mean in NumPy?

Use [Link]() (or nansum, nanstd, etc.), which ignores NaN values. Plain [Link]() propagates NaN — a
single NaN anywhere makes the entire result NaN.

Page 9
 Python for ML — Complete Notes

2. Pandas — Data Wrangling & Analysis

Pandas provides two core structures — Series (1-D labeled array) and DataFrame (2-D labeled table) — built on
top of NumPy. This is the single most important library for a Data/Business/Product Analyst: filtering, grouping,
joining, and reshaping data is the day-to-day job.

2.1 Series & DataFrame basics

PYTHON
import pandas as pd

s = [Link]([10,20,30], index=['a','b','c']) # 1-D labeled array

s['b'] # 20 -- label-based access

df = [Link]({
'name': ['Aditi','Rohan','Meera'],
'age': [24, 29, 22],
'city': ['Pune','Delhi','Kolkata'],
'salary': [55000, 72000, 48000],
})
[Link](2) # first 2 rows
[Link](2) # last 2 rows
[Link] # (3, 4)
[Link]() # dtypes, non-null counts, memory usage
[Link]() # summary stats (count, mean, std, min, quartiles, max) for numeric cols
[Link]; [Link]; [Link]

2.2 Reading & Writing Data

PYTHON
pd.read_csv('[Link]')
pd.read_csv('[Link]', sep=';', header=0, index_col=0, usecols=['a','b'], nrows=1000)
pd.read_excel('[Link]', sheet_name='Sheet1')
pd.read_json('[Link]')
Page 10
 Python for ML — Complete Notes
pd.read_sql('SELECT * FROM orders', con=connection)
pd.read_clipboard() # paste directly from clipboard -- handy in live interviews

df.to_csv('[Link]', index=False)
df.to_excel('[Link]', index=False)
df.to_json('[Link]')
df.to_sql('table_name', con=connection, if_exists='replace')

2.3 Selecting & Filtering Data — loc vs iloc

PYTHON
df['age'] # single column -> Series
df[['name','age']] # multiple columns -> DataFrame
[Link][0] # row by LABEL (index value)
[Link][0] # row by POSITION (0-based)
[Link][0:2] # label-based slice -- END IS INCLUSIVE
[Link][0:2] # position-based slice -- END IS EXCLUSIVE
[Link][0, 'age'] # single cell by (row label, col label)
[Link][0, 1] # single cell by (row pos, col pos)
[Link][df['age'] > 23] # boolean filtering
[Link][(df['age']>23) & (df['city']=='Pune')] # multiple conditions -- use & | ~, NOT and/or
[Link]('age > 23 and city == "Pune"') # readable alternative
df[df['city'].isin(['Pune','Delhi'])] # membership filter
[Link][0, 'age'] # fast SCALAR access by label (faster than .loc for single
values)
[Link][0, 1] # fast scalar access by position

Interview favorite: loc vs iloc

loc is LABEL-based and its slice end is INCLUSIVE. iloc is POSITION-based (integer only) and its slice end is
EXCLUSIVE, matching Python's normal slicing rules. This asymmetry is one of the most frequently tested pandas
facts.

2.4 Adding, Modifying, Dropping Columns/Rows

PYTHON
df['bonus'] = df['salary'] * 0.1 # new column, vectorized
df['seniority'] = df['age'].apply(lambda x: 'Sr' if x>25 else 'Jr')
[Link](columns={'name':'employee_name'}, inplace=True)
[Link](columns=['bonus'], inplace=True) # drop column(s)

Page 11
 Python for ML — Complete Notes
[Link](index=[0], inplace=True) # drop row(s)
df.drop_duplicates(subset=['city'], keep='first')
[Link](1, 'id', range(len(df))) # insert column at a specific position
df['age'] = df['age'].astype(float) # change dtype

2.5 Handling Missing Data

PYTHON
[Link]() # boolean mask of missing values (isnull() is an alias)
[Link]().sum() # count of missing values per column
[Link]() # drop rows with ANY NaN
[Link](subset=['age']) # drop rows where 'age' specifically is NaN
[Link](thresh=3) # keep rows with at least 3 non-NaN values
[Link](0) # fill all NaNs with 0
df['age'].fillna(df['age'].mean(), inplace=True) # fill with column mean -- very common
[Link](method='ffill') # forward-fill (propagate last valid value)
[Link](method='bfill') # backward-fill
df['age'].interpolate(method='linear') # interpolate numeric gaps

Analyst tip
Never blindly fillna(0) on a numeric column like salary or age — it silently distorts the mean/distribution. Prefer
mean/median imputation, or flag+impute (add a boolean 'was_missing' column) so downstream models/analysis
can account for it.

2.6 GroupBy — split, apply, combine

PYTHON
[Link]('city')['salary'].mean() # avg salary per city
[Link]('city').agg({'salary':'mean','age':'max'}) # multiple aggregations
[Link]('city').agg(
avg_salary=('salary','mean'),
max_age=('age','max')
) # NAMED aggregation (cleaner column
names)
[Link]('city').size() # row count per group
[Link]('city')['salary'].transform('mean') # broadcast group mean back to EVERY row (same
shape as df)
[Link]('city').filter(lambda g: len(g) > 1) # keep only groups matching a condition
[Link](['city','seniority']).mean() # multi-level (hierarchical) grouping

agg vs transform vs apply

agg() reduces each group to a single value (shape shrinks). transform() returns a result with the SAME shape as
the input, broadcasting the group result back to every row — perfect for creating features like
'salary_vs_city_avg'. apply() is the most flexible/slowest — the function can return anything (scalar, Series, or
DataFrame).

2.7 Merging, Joining, Concatenating

PYTHON
[Link](df1, df2, on='id', how='inner') # SQL-style join

Page 12
 Python for ML — Complete Notes
# how: 'inner' (intersection), 'left', 'right', 'outer' (union)
[Link](df1, df2, left_on='emp_id', right_on='id', how='left')
[Link](df2, on='id') # join on index (or a key column)
[Link]([df1, df2], axis=0) # stack vertically (append rows)
[Link]([df1, df2], axis=1) # stack horizontally (append columns)
[Link]([df1, df2], ignore_index=True) # reset the index after stacking

Join type Result

inner Only rows with matching keys in BOTH DataFrames

left All rows from left + matches from right (NaN where no match)

right All rows from right + matches from left

outer All rows from both, NaN where no match (union)

2.8 Reshaping: pivot, pivot_table, melt, stack/unstack

PYTHON
[Link](index='date', columns='city', values='sales') # reshape long -> wide (no
aggregation, needs unique combos)
df.pivot_table(index='city', columns='seniority', values='salary', aggfunc='mean') # like pivot
+ groupby combined
[Link](df, id_vars=['name'], value_vars=['q1_sales','q2_sales']) # wide -> long
df.set_index(['city','seniority']).unstack() # move inner index level to columns
[Link](df['city'], df['seniority']) # frequency cross-tabulation

2.9 Sorting & Ranking

PYTHON
df.sort_values('salary', ascending=False)
df.sort_values(['city','salary'], ascending=[True, False]) # multi-column sort
df.sort_index()
df['rank'] = df['salary'].rank(ascending=False, method='dense')
[Link](3, 'salary') # top-3 rows by salary -- faster than sort+head
[Link](3, 'salary')

2.10 apply, map, applymap / [Link]

PYTHON
df['age'].map({22:'young', 24:'young', 29:'senior'}) # [Link] -- element-wise value
substitution
df['age'].apply(lambda x: x*2) # [Link] -- function per element
[Link](lambda row: row['salary']/row['age'], axis=1) # [Link] -- function per ROW
(axis=1) or COLUMN (axis=0)
[Link](lambda x: str(x).upper()) # element-wise over the WHOLE DataFrame (deprecated in
favor of [Link] in pandas 2.1+)

Performance ranking (fastest to slowest)

Page 13
 Python for ML — Complete Notes

1) Vectorized ops (df['a']+df['b']) 2) NumPy vectorized ([Link]) 3) [Link] with a dict 4) .apply() 5)
Python for-loop / iterrows(). Avoid iterrows() on large data — it's dramatically slower because it reconstructs a
Series object per row.

2.11 String, DateTime, and Categorical operations

PYTHON
df['name'].[Link](); df['name'].[Link]('a')
df['name'].[Link](' ').str[0] # first token of each string
df['name'].[Link]('a','@', regex=False)

df['date'] = pd.to_datetime(df['date'])
df['date'].[Link]; df['date'].[Link]; df['date'].dt.day_name()
df['date'].[Link] # 0=Monday
df.set_index('date').resample('M').sum() # monthly resampling/aggregation of a time series

df['seniority'] = df['seniority'].astype('category') # categorical dtype -- saves memory,

speeds up groupby

2.12 Duplicate handling, MultiIndex, window functions

PYTHON
[Link]().sum() # count exact duplicate rows
df.drop_duplicates()

df.set_index(['city','name']) # MultiIndex (hierarchical index)

[Link]('Pune', level='city') # cross-section lookup on a MultiIndex

df['salary'].rolling(window=3).mean() # rolling (moving) average -- window functions

df['salary'].expanding().mean() # expanding (cumulative) average
df['salary'].shift(1) # shift values down by 1 row (lag) -- e.g. for
month-over-month diff
df['salary'].diff() # difference from previous row
df['salary'].pct_change() # percentage change row-to-row

2.13 Performance & Memory Optimization

●​ Use vectorized operations instead of loops/apply wherever possible.
●​ Downcast numeric dtypes: pd.to_numeric(df['col'], downcast='integer') to shrink memory footprint.
●​ Convert low-cardinality string columns to 'category' dtype — huge memory & groupby speed win.
●​ Read only needed columns: pd.read_csv(..., usecols=[...]).
●​ For very large files, read in chunks: pd.read_csv(..., chunksize=100000) and process iteratively.
●​ df.memory_usage(deep=True) to inspect per-column memory use.

2.14 Pandas — Interview Questions

Q1. What's the fundamental difference between loc and iloc?

Page 14
 Python for ML — Complete Notes

loc selects by LABEL and its slice end is inclusive; iloc selects by integer POSITION and its slice end is exclusive
(standard Python slicing behavior).

Q2. How would you find and handle missing values in a DataFrame?
Use [Link]().sum() to profile missingness per column, then decide: dropna() if missing rows are rare and
non-informative, or fillna() with mean/median/mode/forward-fill for numeric or categorical columns, or add an
explicit 'missing' indicator column when missingness itself is informative (e.g., MNAR data).

Q3. Difference between merge, join, and concat?

merge() performs SQL-style key-based joins (inner/left/right/outer) on columns. join() is a convenience method
primarily for joining on the index. concat() simply stacks DataFrames along an axis (rows or columns) without
matching on keys — it's structural, not key-based, combination.

Q4. Explain agg vs transform vs apply in groupby.

agg reduces each group to one summary value (output shrinks). transform returns an output the same length as
the input, broadcasting the group-level result back to every original row (great for creating comparison features).
apply is the most general — can return scalars, Series, or DataFrames, and pandas infers how to combine results,
at some performance cost.

Q5. How do you find duplicate rows and remove them?

[Link]() flags duplicate rows (default: all columns, keeping the first occurrence as False);
df.drop_duplicates(subset=[...], keep='first'/'last'/False) removes them, optionally based on a subset of columns.

Q6. Why is iterrows() considered bad practice on large DataFrames?

iterrows() reconstructs a pandas Series object for every row (with its own dtype inference and indexing overhead),
making it orders of magnitude slower than vectorized operations or even itertuples(). Prefer vectorized
column-wise operations, [Link], or at worst itertuples().

Q7. What is the difference between a Series and a DataFrame?

A Series is a 1-dimensional labeled array (single column with an index). A DataFrame is a 2-dimensional labeled
table — essentially a dict of Series sharing a common index, one per column.

Q8. How would you pivot a 'long' sales table (date, city, sales) into a 'wide' table with cities as columns?
[Link](index='date', columns='city', values='sales'), or pivot_table if there could be duplicate (date, city)
combinations requiring aggregation.

Q9. What does inplace=True actually do, and why is it often discouraged in production code?
It modifies the DataFrame in place and returns None instead of a new object. It's discouraged because it can
silently mutate shared references, doesn't chain well with method chaining, and (in some pandas versions) can
raise SettingWithCopyWarning when used on a view/slice rather than the original object.

Q10. How do you efficiently compute a 7-day moving average of sales?

df.set_index('date')['sales'].rolling(window=7).mean() — rolling() creates a sliding window over which .mean() (or
sum/std/etc.) is applied.

Q11. What's the SettingWithCopyWarning and how do you avoid it?

Page 15
 Python for ML — Complete Notes

It's raised when pandas can't determine if you're modifying a view or a copy of a DataFrame (e.g., df[df.a>0]['b']=1
chained indexing). Avoid it by using .loc explicitly for both selection and assignment in a single step: [Link][df.a>0,
'b'] = 1, or call .copy() explicitly when you intend to create an independent DataFrame.

Q12. How would you detect and treat outliers in a salary column using pandas?
Common approaches: IQR method (flag values outside Q1 - 1.5*IQR to Q3 + 1.5*IQR), or z-score method (flag |z|
> 3). E.g. q1,q3=[Link]([.25,.75]); iqr=q3-q1; outliers = df[([Link] < q1-1.5*iqr) | ([Link] >
q3+1.5*iqr)].

Page 16
 Python for ML — Complete Notes

3. Matplotlib & Seaborn — Data Visualization

Matplotlib is the low-level plotting engine every Python chart is ultimately drawn with. Seaborn sits on top of it,
offering statistical plot types and DataFrame-aware syntax with sensible defaults. Learn Matplotlib's Figure/Axes
model first — Seaborn will make far more sense afterward.

3.1 The Figure / Axes architecture

PYTHON
import [Link] as plt

# Object-oriented API (RECOMMENDED, scales to multiple subplots)

fig, ax = [Link](figsize=(6,4))
[Link]([1,2,3],[4,5,6])
ax.set_title('My Plot'); ax.set_xlabel('X'); ax.set_ylabel('Y')
[Link]()

# pyplot (state-based) API -- quick and dirty, fine for a single simple plot
[Link]([1,2,3],[4,5,6])
[Link]('My Plot'); [Link]()

Figure vs Axes
A Figure is the entire window/canvas. An Axes is a single plot area living inside a Figure — one Figure can contain
many Axes (subplots). Almost all customization methods (title, labels, legend, limits) exist on the Axes object in
the OO API.

Page 17
 Python for ML — Complete Notes

3.2 Common plot types

PYTHON
fig, axes = [Link](2,3, figsize=(12,7))
axes[0,0].bar(x, y) # bar chart -- compare categories
axes[0,1].plot(x, y) # line chart -- trend over continuous/time axis
axes[0,2].scatter(x, y) # scatter -- relationship between 2 numeric vars
axes[1,0].hist(data, bins=20) # histogram -- distribution of 1 numeric var
axes[1,1].boxplot([g1, g2]) # box plot -- distribution + outliers, compare groups
axes[1,2].pie(sizes, labels=labels) # pie chart -- part-to-whole (use sparingly!)
plt.tight_layout(); [Link]()

Plot Best for Avoid when

Too many categories (>10-15) — gets

Bar Comparing a metric across categories
cluttered

Line Trend over time/ordered axis Unordered/categorical x-axis

Relationship/correlation between 2 Too many overlapping points (use alpha

Scatter
numeric variables or hexbin)

Shape of a single numeric variable's Comparing many groups at once (use

Histogram
distribution KDE/violin instead)

Need to see the full shape of distribution

Box plot Comparing spread & outliers across groups
(bimodal etc.)

Simple part-to-whole with few (<=5) Comparing precise values — humans are
Pie chart
categories bad at judging angles

3.3 Customization essentials

Page 18
 Python for ML — Complete Notes
PYTHON
ax.set_xlim(0,10); ax.set_ylim(0,100)
ax.set_xticks([0,2,4,6,8]); ax.set_xticklabels(['a','b','c','d','e'], rotation=45)
[Link](loc='upper right')
[Link](alpha=0.3)
[Link](x, y, color='#C24E7A', linewidth=2, linestyle='--', marker='o', label='revenue')
[Link](y=50, color='red', linestyle=':') # reference line
[Link]('Peak', xy=(5,90), xytext=(6,95), arrowprops=dict(arrowstyle='->'))
[Link]('[Link]', dpi=300, bbox_inches='tight') # save high-res, no clipped labels

3.4 Subplots layouts

PYTHON
fig, axes = [Link](nrows=2, ncols=2, figsize=(10,8), sharex=True)
fig, axes = [Link](1, 3, figsize=(15,4))
axes[0].plot(...); axes[1].bar(...); axes[2].hist(...)
[Link]('Overall Dashboard Title')
plt.tight_layout() # prevents overlapping labels/titles

3.5 Seaborn — statistical plotting

PYTHON
import seaborn as sns
sns.set_style('whitegrid')

[Link](data=df, x='salary', hue='city', kde=True) # distribution + smooth density curve

[Link](data=df, x='city', y='salary') # grouped box plot
[Link](data=df, x='city', y='salary') # box plot + density shape combined
[Link](data=df, x='age', y='salary', hue='city', size='bonus')
[Link](data=df, x='city', y='salary', estimator='mean', errorbar='sd') # bar + confidence
interval
[Link]([Link](), annot=True, cmap='coolwarm') # correlation matrix heatmap
[Link](df, hue='city') # scatterplot matrix -- great first
EDA step
[Link](data=df, x='age', y='salary') # scatter + fitted regression line
[Link](data=df, x='city') # frequency bar chart of a
categorical column

Why Seaborn over raw Matplotlib for EDA

Seaborn understands 'tidy' DataFrames directly (x='col', y='col', hue='col'), auto-computes aggregates/confidence
intervals, and ships with better default color palettes and statistical plot types (violin, KDE, pairplot, heatmap)
that would take many lines of raw Matplotlib to replicate.

3.6 Choosing the right chart — a quick decision guide

●​ 1 numeric variable -> histogram / KDE / box plot
●​ 1 categorical variable -> bar chart / count plot
●​ 2 numeric variables -> scatter plot (+ regression line if checking correlation)
●​ 1 numeric + 1 categorical -> box plot / violin plot / bar plot (grouped)
●​ 2 categorical variables -> heatmap of a crosstab, or grouped/stacked bar chart

Page 19
 Python for ML — Complete Notes

●​ Time series -> line chart (always keep time on the x-axis)
●​ Many numeric variables at once -> correlation heatmap, or pairplot for pairwise relationships

3.7 Matplotlib / Seaborn — Interview Questions

Q1. What's the difference between the pyplot interface and the object-oriented interface in Matplotlib?
The pyplot interface ([Link](), [Link]()) implicitly tracks a 'current' figure/axes globally — convenient for one-off
quick plots but fragile for multi-subplot figures. The OO interface (fig, ax = [Link](); [Link]()) explicitly
references the Figure and Axes objects, making it the recommended, scalable approach for anything beyond a
single simple plot.

Q2. When would you use a box plot vs a histogram to show a distribution?
A histogram shows the full shape of a single distribution (including multi-modality) but is hard to compare across
many groups. A box plot compactly summarizes median/quartiles/outliers and is ideal for comparing the spread of
a numeric variable across several categories side by side, at the cost of hiding the detailed shape.

Q3. Why are pie charts generally discouraged in professional analytics?

Humans are poor at accurately judging and comparing angles/areas versus lengths, so pie charts make precise
comparisons hard, especially with more than ~5 slices or similarly-sized slices. A bar chart usually communicates
the same information more accurately.

Q4. How do you visualize the correlation between multiple numeric variables at once?
Compute a correlation matrix with [Link]() and visualize it as a heatmap via [Link]([Link](), annot=True,
cmap='coolwarm'), or use [Link]() to see pairwise scatter plots for every combination of variables.

Q5. What does the hue parameter do in Seaborn?

It adds a third (typically categorical) dimension to a plot by color-coding points/bars/lines according to the values
of that column, letting you compare subgroups within the same plot without manually looping and re-plotting.

Q6. How would you avoid overlapping points in a scatter plot with 100,000 rows?
Reduce opacity with alpha (e.g., alpha=0.1), downsample the data, use a 2-D density plot/hexbin ([Link] or
[Link] with fill), or aggregate before plotting.

Page 20
 Python for ML — Complete Notes

4. SciPy — Statistics, Distributions & Optimization

SciPy extends NumPy with scientific computing tools. For analyst/ML interviews, the [Link] module is by far
the most tested: probability distributions, and hypothesis tests (used constantly in A/B testing questions for
Product/Business Analyst roles).

4.1 Probability distributions

PYTHON
from scipy import stats

[Link](0, loc=0, scale=1) # density of N(0,1) at x=0

[Link](1.96) # P(Z <= 1.96) ~ 0.975
[Link](0.975) # inverse CDF (quantile) -> 1.96 (critical value)
[Link](k=3, n=10, p=0.5) # P(X=3) for Binomial(10, 0.5)
[Link](k=2, mu=3) # P(X=2) for Poisson(lambda=3)
[Link](loc=0, scale=1, size=100) # random samples

4.2 Hypothesis testing — the interview favorite

PYTHON
# One-sample / two-sample t-test
stats.ttest_1samp(sample, popmean=50)
stats.ttest_ind(group_a, group_b, equal_var=False) # Welch's t-test (unequal variances)
stats.ttest_rel(before, after) # paired t-test

# Chi-square test of independence (categorical vs categorical)

chi2, p, dof, expected = stats.chi2_contingency([Link]([Link], [Link]))

# ANOVA (compare means across 3+ groups)

stats.f_oneway(group_a, group_b, group_c)

# Correlation
[Link](x, y) # linear correlation + p-value
[Link](x, y) # rank/monotonic correlation, robust to outliers

# Proportions z-test (A/B test on conversion rate)

from [Link] import proportions_ztest
proportions_ztest(count=[120,150], nobs=[1000,1000])

A/B testing interview pattern

1) State H0 (no difference between variants) and H1. 2) Pick a test based on data type: proportions -> z-test for
proportions or chi-square; means -> t-test; 3+ groups -> ANOVA. 3) Compute the p-value. 4) Compare to
significance level alpha (usually 0.05): p < alpha -> reject H0 (statistically significant). Always also discuss effect
size and practical significance, not just p-values.

4.3 SciPy — Interview Questions

Q1. Which test would you use to compare conversion rates between two app versions (A/B test)?

Page 21
 Python for ML — Complete Notes

A two-proportion z-test (or equivalently a chi-square test of independence on a 2x2 table), since we're comparing
a binary outcome's rate across two independent groups.

Q2. What's the difference between a t-test and a z-test?

A z-test assumes known population variance/large sample size (uses normal distribution); a t-test is used when
population variance is unknown and estimated from the sample, especially with small samples (uses the
t-distribution, which has heavier tails and approaches normal as sample size grows).

Q3. When would you use Spearman correlation instead of Pearson?

Spearman correlation measures monotonic (rank-based) relationships and is robust to outliers and
non-linear-but-monotonic relationships, whereas Pearson specifically measures linear correlation and is sensitive
to outliers.

Page 22
 Python for ML — Complete Notes

5. Scikit-learn — The ML Toolkit API

Scikit-learn provides a remarkably consistent API across every algorithm: every model is an 'estimator' with .fit(),
and (depending on type) .predict(), .transform(), or .predict_proba(). Understanding this consistent contract is
more valuable in interviews than memorizing every algorithm's math.

5.1 The core API contract

Method Used by What it does

Learns parameters from training

.fit(X, y) All estimators
data (y omitted for unsupervised)

Returns predicted labels/values for

.predict(X) Classifiers, Regressors
new data

Returns class probabilities instead

.predict_proba(X) Classifiers
of hard labels

Transformers (scalers, encoders, Applies a learned transformation to

.transform(X)
PCA) data

Shortcut: fit() then transform() in

.fit_transform(X) Transformers
one call

Returns a default metric (accuracy

.score(X, y) Most estimators
for classifiers, R² for regressors)

5.2 Train/test split & the golden rule of preprocessing

PYTHON
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, random_state=42, stratify=y
)

Golden rule: fit on train, transform on both

Page 23
 Python for ML — Complete Notes

ALWAYS call .fit() (or .fit_transform()) only on the TRAINING data, then use the already-fitted transformer's
.transform() on the test set. Fitting a scaler/encoder on the full dataset before splitting leaks information from
the test set into training (data leakage) — one of the most common interview trick questions.

5.3 Preprocessing
PYTHON
from [Link] import StandardScaler, MinMaxScaler, OneHotEncoder, LabelEncoder

scaler = StandardScaler() # (x - mean) / std -- 'standardization', mean 0, std 1

X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = [Link](X_test) # NOTE: transform only, using train's mean/std

mm = MinMaxScaler() # (x - min) / (max - min) -- scales to [0,1]

ohe = OneHotEncoder(sparse_output=False, handle_unknown='ignore')

ohe.fit_transform(df[['city']]) # one column per category, 0/1 values

le = LabelEncoder() # encodes a single target/label column as integers

0..k-1
le.fit_transform(df['seniority'])

Technique Use for Watch out for

Features with roughly normal distribution;

distance-based/gradient-based models Sensitive to outliers (uses
StandardScaler
(KNN, SVM, linear/logistic regression, mean/std)
neural nets)

When you need bounded [0,1] range (e.g., Very sensitive to outliers
MinMaxScaler
neural net inputs, image pixels) (uses min/max)

Nominal categorical features with no order High cardinality -> curse of

OneHotEncoder
(city, color) dimensionality

Never use LabelEncoder on

LabelEncoder / Ordinal categories with a natural order nominal FEATURES fed to
OrdinalEncoder (low/medium/high), or the target column linear models — implies a
false order

5.4 Pipelines & ColumnTransformer

PYTHON
from [Link] import Pipeline
from [Link] import ColumnTransformer
from [Link] import SimpleImputer

numeric_features = ['age','salary']
categorical_features = ['city']

numeric_pipe = Pipeline([
('imputer', SimpleImputer(strategy='median')),
('scaler', StandardScaler()),
Page 24
 Python for ML — Complete Notes
])
categorical_pipe = Pipeline([
('imputer', SimpleImputer(strategy='most_frequent')),
('onehot', OneHotEncoder(handle_unknown='ignore')),
])

preprocessor = ColumnTransformer([
('num', numeric_pipe, numeric_features),
('cat', categorical_pipe, categorical_features),
])

full_pipeline = Pipeline([
('preprocessing', preprocessor),
('model', LogisticRegression()),
])
full_pipeline.fit(X_train, y_train)
full_pipeline.predict(X_test)

Why use Pipeline at all?

A Pipeline bundles preprocessing + model into a single object, guaranteeing the exact same transformations are
applied identically to train, test, and future production data — and it prevents data leakage automatically
because cross-validation refits the pipeline (including preprocessing) on each fold's training data only.

5.5 Baseline models you should be able to explain in 30 seconds each

PYTHON
from sklearn.linear_model import LinearRegression, LogisticRegression
from [Link] import DecisionTreeClassifier
from [Link] import RandomForestClassifier
from [Link] import KMeans
from [Link] import PCA

lr = LinearRegression().fit(X_train, y_train) # predicts a continuous number

logreg = LogisticRegression().fit(X_train, y_train) # predicts a class probability via
sigmoid
dt = DecisionTreeClassifier(max_depth=5).fit(X_train, y_train) # rule-based splits
rf = RandomForestClassifier(n_estimators=200).fit(X_train, y_train) # ensemble of trees,
reduces overfitting
km = KMeans(n_clusters=3, random_state=42).fit(X) # unsupervised clustering, no y needed
pca = PCA(n_components=2).fit_transform(X) # dimensionality reduction

5.6 Model evaluation & metrics

Page 25
 Python for ML — Complete Notes

PYTHON
from [Link] import (accuracy_score, precision_score, recall_score, f1_score,
confusion_matrix, roc_auc_score, mean_squared_error, r2_score, mean_absolute_error)

accuracy_score(y_test, y_pred)
precision_score(y_test, y_pred) # of predicted positives, how many were correct? TP/(TP+FP)
recall_score(y_test, y_pred) # of actual positives, how many did we catch? TP/(TP+FN)
f1_score(y_test, y_pred) # harmonic mean of precision & recall
roc_auc_score(y_test, y_proba) # area under ROC curve -- threshold-independent
confusion_matrix(y_test, y_pred)

mean_squared_error(y_test, y_pred) # regression: average squared error

mean_absolute_error(y_test, y_pred) # regression: average absolute error (robust to
outliers)
r2_score(y_test, y_pred) # regression: proportion of variance explained (1.0 =
perfect)

Metric Formula Use when

False positives are costly (e.g., flagging a good

Precision TP / (TP + FP)
transaction as fraud)

False negatives are costly (e.g., missing an actual

Recall TP / (TP + FN)
fraud/disease case)

Need a single balance between precision & recall,

F1-score 2 · P·R / (P+R)
esp. with imbalanced classes

Classes are roughly balanced — misleading

Accuracy (TP+TN) / total
otherwise

Need a threshold-independent ranking quality

ROC-AUC Area under TPR vs FPR curve
measure

5.7 Cross-validation & hyperparameter tuning

Page 26
 Python for ML — Complete Notes
PYTHON
from sklearn.model_selection import cross_val_score, GridSearchCV, KFold, StratifiedKFold

scores = cross_val_score(model, X, y, cv=5, scoring='f1') # 5-fold CV

param_grid = {'n_estimators':[100,200], 'max_depth':[3,5,None]}

grid = GridSearchCV(RandomForestClassifier(), param_grid, cv=5, scoring='roc_auc')
[Link](X_train, y_train)
grid.best_params_; grid.best_score_

Why cross-validation over a single train/test split?

A single split gives a noisy, potentially lucky/unlucky estimate of model performance. K-fold CV trains/evaluates
the model K times on different splits and averages results, giving a far more reliable estimate of how the model
generalizes — especially important with smaller datasets.

5.8 Handling class imbalance

●​ Use class_weight='balanced' in most sklearn estimators to upweight the minority class automatically.
●​ Resampling: oversample the minority class (e.g., SMOTE from imbalanced-learn) or undersample the
majority class.
●​ Prefer F1/recall/ROC-AUC/PR-AUC over raw accuracy — accuracy is misleading when one class
dominates (e.g., 99% negative -> a model predicting 'always negative' gets 99% accuracy but is useless).
●​ Adjust the classification threshold (default 0.5) based on the business cost of false positives vs false
negatives.

5.9 Scikit-learn — Interview Questions

Q1. Why must you fit a scaler only on training data, not the full dataset?
Fitting on the full dataset lets statistics from the test set (mean, std, min/max) leak into training, giving an overly
optimistic performance estimate. Fit only on X_train, then use that fitted scaler's .transform() on X_test, simulating
truly unseen data.

Q2. Explain the difference between fit, transform, and fit_transform.

fit() learns parameters from data (e.g., mean/std for a scaler) without changing anything. transform() applies
previously learned parameters to (possibly different) data. fit_transform() is a convenience method that does both
in one call — typically used only once, on the training set.

Q3. What's the difference between precision and recall, and when would you optimize for one over the
other?
Precision = of everything predicted positive, how much was actually positive (minimizes false positives). Recall = of
everything actually positive, how much was caught (minimizes false negatives). Optimize for precision when false
positives are expensive (spam filter blocking real emails); optimize for recall when false negatives are expensive
(cancer screening missing a real case).

Q4. Why is accuracy a poor metric for imbalanced classification problems?

Page 27
 Python for ML — Complete Notes

A trivial model that always predicts the majority class can achieve very high accuracy while being completely
useless for detecting the minority class of actual interest — accuracy doesn't reflect performance on the class that
usually matters most.

Q5. What is the purpose of a scikit-learn Pipeline?

It chains preprocessing steps and a final estimator into a single object so the exact same transformations are
consistently applied to train/test/production data, and so that cross-validation correctly refits preprocessing steps
within each fold, preventing data leakage.

Q6. What is k-fold cross-validation and why use it over a single train-test split?
It splits data into k folds, trains on k-1 folds and validates on the remaining fold, repeating k times so every
observation is used for validation exactly once, then averages the results — giving a more robust, lower-variance
estimate of generalization performance than one arbitrary split.

Q7. What's the difference between bagging (Random Forest) and boosting (e.g., XGBoost) at a high level?
Bagging trains many models in parallel on bootstrapped samples and averages/votes their predictions to reduce
variance (Random Forest). Boosting trains models sequentially, where each new model focuses on correcting the
errors of the previous ones, reducing bias (AdaBoost, Gradient Boosting, XGBoost).

Q8. How would you decide between OneHotEncoder and LabelEncoder for a categorical feature?
Use OneHotEncoder for nominal categories with no inherent order (feeding to distance-based or linear models,
since LabelEncoder would falsely imply an ordinal relationship). Use LabelEncoder/OrdinalEncoder only for
genuinely ordinal categories, or for encoding the target/label column, or for tree-based models which can handle
integer-encoded categories reasonably well.

Q9. What is data leakage and give an example specific to preprocessing?

Data leakage is when information from outside the training set (often from the future or the test set) improperly
influences model training, producing unrealistically good validation performance that won't hold in production.
Example: computing a StandardScaler's mean/std on the full dataset before splitting, or imputing missing values
using statistics computed across train+test combined.

Q10. What does PCA do and when would you use it?
Principal Component Analysis projects data onto a smaller number of orthogonal directions (components) that
capture maximum variance, used for dimensionality reduction, noise reduction, visualization of high-dimensional
data, and mitigating multicollinearity — at the cost of losing direct interpretability of the transformed features.

Page 28
 Python for ML — Complete Notes

6. Other Libraries Worth Knowing Before ML

6.1 Statsmodels — statistics-first modeling
Where scikit-learn is optimized for prediction, statsmodels is optimized for statistical inference — it gives you
p-values, confidence intervals, and coefficient significance out of the box, which is what Business/Product
Analyst interviews often actually want.
PYTHON
import [Link] as sm
X = sm.add_constant(X) # adds an intercept column of 1s
model = [Link](y, X).fit() # Ordinary Least Squares regression
print([Link]()) # coefficients, p-values, R-squared, confidence intervals

6.2 Jupyter Notebook essentials

PYTHON
%matplotlib inline # show plots directly below cells
%timeit [Link]('city').mean() # benchmark a single line
%%time # benchmark an entire cell
[Link]() # auto-displays as a formatted HTML table (last line of a cell)
!pip install seaborn # run shell commands directly

6.3 Plotly (interactive visualization) — brief mention

Plotly Express ([Link]) creates interactive, hoverable charts with a syntax very close to Seaborn — useful
for dashboards/stakeholder-facing deliverables, but rarely required in a live coding interview. Worth knowing it
exists: [Link](df, x='city', y='salary'), [Link](df, x='age', y='salary', hover_data=['name']).

6.4 Quick reference: which library solves which problem?

Task Library

Fast numeric array math, linear algebra NumPy

Load/clean/reshape tabular data Pandas

Static charts for reports/EDA Matplotlib, Seaborn

Interactive/dashboard charts Plotly

Hypothesis tests, distributions SciPy (stats)

Regression with p-values/inference Statsmodels

Train/evaluate ML models, preprocessing Scikit-learn

Notebook environment / experimentation Jupyter

Page 29
 Python for ML — Complete Notes

7. Consolidated Interview Prep — Product / Business

/ Data Analyst Angle
Beyond syntax, non-core-tech interviewers (Product Analyst, Business Analyst, Product Manager with analytics
rounds) tend to ask conceptual + applied questions that connect these libraries to business decisions. Below are
the most common patterns.

7.1 Case-study style questions

Q1. Walk me through how you'd analyze why weekly active users dropped 15% last month.
Structure: (1) Confirm the metric definition and data source, rule out a tracking/logging bug. (2) Segment the drop
— by platform, geography, cohort (new vs returning), and feature usage using pandas groupby to isolate where the
drop is concentrated. (3) Check for confounds — seasonality, a recent release, marketing spend changes — using a
time series line plot. (4) Form a hypothesis and test it statistically (e.g., a t-test/chi-square comparing before/after
cohorts). (5) Recommend an action and a metric to monitor going forward.

Q2. You have a CSV of user transactions. How would you find the top 10% of users by revenue contribution,
using pandas?
Group by user_id and sum revenue: rev = [Link]('user_id')['revenue'].sum(); threshold = [Link](0.9);
top_users = rev[rev >= threshold]. This directly uses groupby + quantile, both core pandas operations.

Q3. How would you design and analyze an A/B test for a new checkout flow?
Define the primary metric (e.g., conversion rate) and a minimum detectable effect, compute required sample size,
randomly assign users to control/treatment ensuring no leakage, run the test for a pre-committed duration (avoid
peeking), then run a two-proportion z-test or chi-square test on the results, checking both statistical significance
(p-value) and practical/business significance (effect size, revenue impact).

7.2 Rapid-fire conceptual questions

Q1. What is vectorization and why does it matter for a 10-million-row dataset?
Vectorization applies operations to entire arrays via compiled, low-level loops instead of Python-level for-loops,
giving order-of-magnitude speedups — critical when row-by-row Python iteration over 10M rows would take
minutes instead of seconds.

Q2. What's the difference between correlation and causation, and how does this relate to a regression
coefficient?
Correlation (or a regression coefficient) shows association between variables, not that one causes the other —
confounding variables, reverse causality, or coincidence can produce a strong correlation with no causal link.
Establishing causation typically requires a randomized experiment (A/B test) or careful causal inference methods
(e.g., instrumental variables, diff-in-diff).

Q3. Explain p-value in one sentence a non-technical stakeholder would understand.

It's the probability of seeing a result at least this extreme purely by chance if there were actually no real effect — a
small p-value (typically < 0.05) suggests the observed effect is unlikely to be random noise.

Page 30
 Python for ML — Complete Notes

Q4. Why might you choose median over mean when reporting typical user spend?
Mean is sensitive to outliers (a few very high spenders can inflate it); median is robust and better represents the
'typical' user when the distribution is skewed, which spend/income data usually is.

Page 31
 Python for ML — Complete Notes

8. Final Quick-Reference Cheat Table

I want to... Code

Load a CSV pd.read_csv('[Link]')

See data types & nulls [Link]()

Filter rows df[df['col'] > 5]

Select rows by position [Link][0:5]

Select rows by label [Link]['a':'c']

Group & aggregate [Link]('col').agg({'x':'mean'})

Join two tables [Link](df1, df2, on='key', how='left')

Handle missing values [Link]([Link]())

Plot a distribution [Link](df['col'], kde=True)

Compare group means [Link](x='group', y='value', data=df)

Run a hypothesis test stats.ttest_ind(a, b)

Split train/test train_test_split(X, y, test_size=0.2, random_state=42)

Scale features StandardScaler().fit_transform(X_train)

Train a baseline model LogisticRegression().fit(X_train, y_train)

Evaluate a classifier f1_score(y_test, y_pred)

Before your interview

Re-derive every code snippet in
this document from memory in
a blank notebook, not by
re-reading — active recall is
what actually sticks. Then re-do
the Interview Questions sections
out loud, as if answering a real
interviewer, in under 60 seconds
each.

Page 32