PYTHON

The Complete One-Stop Study Guide

Concepts • Internals • NumPy/Pandas • Testing • Coding Practice Set • Big-O • Interview Prep
Data Structures • Functions/Decorators • Generators • OOP • Typing • DE Stdlib • Memory/GIL • Concurrency •
NumPy • Pandas • pytest • ETL Patterns • 20 Solved Coding Problems • FAANG/JPMC Interviews
Table of Contents
TOC \h \o "1-1"
0. How to Use This Guide + Structured Learning Path
A complete, interview-grade Python reference for a Data Engineer targeting JPMC / FAANG-level roles. It
blends language concepts, internals, the DE toolkit (NumPy/pandas), engineering practices
(typing/testing/packaging), and a solved coding practice set — because FAANG Python rounds are live
coding, not trivia.

4-Week Roadmap
● Week 1 — Language core: Parts 1–6. How Python runs, data structures + Big-O,
functions/closures/decorators, comprehensions/generators, OOP, exceptions/context managers.
● Week 2 — Engineering: Parts 7–12. Modules/packaging, typing, the DE standard library, file I/O &
formats, memory/GIL/performance, concurrency.
● Week 3 — Data toolkit + quality: Parts 13–17. NumPy, pandas, testing (pytest), DE/ETL patterns,
pitfalls. Start the coding practice set.
● Week 4 — Interview: Parts 18–21. Big-O reference, drill the 20 coding problems from a blank page
(<15 min each), then scenarios + FAANG/JPMC interview questions.

How Python interviews are graded

● Correctness + edge cases → complexity (know the Big-O and state it) → clean, idiomatic code
(comprehensions, stdlib, no reinventing) → communication (clarify, test mentally, discuss trade-
offs).
1. Fundamentals & How Python Runs
Python is high-level, dynamically typed, interpreted. CPython compiles source (.py) to bytecode (.pyc)
executed by the Python Virtual Machine (PVM).
● Dynamically typed — names are references bound to objects; types checked at runtime.
● Everything is an object — numbers, functions, classes all have a type + identity (`id()`).
● Indentation defines blocks (no braces). CPython is the reference implementation (also PyPy = JIT,
Jython, etc.).

Mutable vs immutable (interview staple)

● Immutable: int, float, bool, str, tuple, frozenset, bytes — can't change in place; 'modifying' creates a
new object; hashable (valid dict keys).
● Mutable: list, dict, set, bytearray, most objects — changed in place; same `id()`; NOT hashable.
● Why it matters: mutable default args and shared references cause classic bugs (Part 17).
2. Built-in Data Structures + Internals + Big-O
● list `[]` — dynamic array; index O(1), append amortized O(1), insert/delete-at-i O(n), search O(n).
Ordered, mutable, duplicates allowed.
● tuple `()` — immutable, hashable, slightly smaller/faster than list; use for fixed records / dict keys.
● set `{}` — hash table; add/remove/`in` O(1) average. Unordered, unique. Best for dedup +
membership.
● dict `{k:v}` — hash table; get/set/del O(1) average; insertion-ordered since 3.7. Keys must be
hashable.

How hashing works (why dict/set are O(1))

● A dict/set stores items in buckets indexed by `hash(key)`. Lookups hash the key → jump to the
bucket → compare. Collisions are resolved by open addressing; a bad hash or many collisions
degrades to O(n). This is why keys must be hashable (immutable).
nums = [1, 2, 2, 3]
uniq = set(nums) # {1,2,3} O(1) membership
d = {"a": 1, "b": 2}
[Link]("z", 0) # 0 (no KeyError)
# comprehensions (fast, Pythonic)
squares = [x*x for x in range(5)]
evens = {x for x in range(10) if x % 2 == 0}
freq = {k: len(k) for k in ["hi","world"]}

Choose: ordered+mutable → list; fixed record/key → tuple; uniqueness/fast membership → set; key→value
lookup → dict.
3. Functions, Args, Closures, Decorators, Functional
def greet(name, greeting="Hi", *args, **kwargs):
# name=positional, greeting=default,
# *args=extra positionals (tuple), **kwargs=extra keywords (dict)
return f"{greeting}, {name}"

add = lambda a, b: a + b # anonymous

from functools import reduce, partial

list(map(lambda x: x*2, [1,2,3])) # [2,4,6]
list(filter(lambda x: x>1, [1,2,3]))
reduce(lambda a,b: a+b, [1,2,3,4])# 10
inc = partial(add, 1) # inc(5) -> 6

● LEGB scope — name lookup: Local → Enclosing → Global → Built-in. Use `global`/`nonlocal` to rebind
outer names.
● Closure — inner function capturing enclosing variables. Decorator — a callable that wraps a function
(logging, timing, retry, caching via `@functools.lru_cache`). Use `@[Link]` to keep the
original's metadata.
import functools, time
def retry(n=3, delay=1): # parametrized decorator
def deco(fn):
@[Link](fn)
def wrap(*a, **kw):
for i in range(n):
try: return fn(*a, **kw)
except Exception:
if i == n-1: raise
[Link](delay)
return wrap
return deco

@retry(n=5, delay=2)
def call_api(): ...
4. Comprehensions, Iterators, Generators & Itertools
● Iterable — loopable (`__iter__`). Iterator — yields items one at a time (`__iter__` + `__next__`).
Generator — a function with `yield`; produces values lazily, O(1) memory — ideal for big/streaming
data.
def read_big(path): # memory-efficient streaming
with open(path) as f:
for line in f: # yields one line at a time
yield [Link]()

gen = (x*x for x in range(10**6)) # generator expression: tiny memory

# vs [x*x for x in range(10**6)] # list: builds all in RAM

● itertools — `chain, islice, groupby, product, combinations, count, cycle, tee`. functools — `reduce,
lru_cache, partial, wraps`. collections — `Counter, defaultdict, deque, namedtuple, OrderedDict`.
from collections import Counter, defaultdict, deque
Counter("aabbbc") # {'b':3,'a':2,'c':1}
d = defaultdict(list); d["k"].append(1)
q = deque([1,2,3]); [Link](0) # O(1) both ends
5. Object-Oriented Programming
from dataclasses import dataclass
from abc import ABC, abstractmethod

@dataclass # auto __init__/__repr__/__eq__

class Order:
id: int
amount: float
status: str = "NEW"

class Source(ABC): # abstract base class

@abstractmethod
def read(self): ...

class Account:
bank = "Global" # class attribute (shared)
def __init__(self, owner, bal=0):
[Link] = owner # instance attribute
self.__bal = bal # name-mangled 'private'
@property # computed / read-only attribute
def balance(self): return self.__bal
def __repr__(self): return f"Account({[Link]},{self.__bal})"

class Savings(Account): # inheritance + super()

def __init__(self, owner, bal=0, rate=0.03):
super().__init__(owner, bal); [Link] = rate

● 4 pillars: Encapsulation (`__private`), Inheritance (`super()`), Polymorphism (overriding / duck

typing), Abstraction (ABCs).
● Dunder methods: `__init__, __repr__/__str__, __len__, __eq__, __hash__, __iter__,
__enter__/__exit__, __call__`.
● @classmethod (gets `cls`; alt constructors), @staticmethod (no self/cls), @property
(getter/computed), @dataclass (boilerplate-free records).
● MRO (Method Resolution Order) — C3 linearization determines attribute lookup with multiple
inheritance (`Class.__mro__`); mixins add behavior.
6. Exceptions & Context Managers
try:
v = int(user_input)
except ValueError as e:
[Link]("bad number: %s", e)
except (FileNotFoundError, PermissionError) as e:
raise RuntimeError("io failure") from e # chain the cause
else:
print("ok") # runs if no exception
finally:
cleanup() # always runs

# context manager: guaranteed cleanup even on error

with open("[Link]") as f:
for line in f: process(line)

from contextlib import contextmanager

@contextmanager
def timer():
import time; t=[Link]()
try: yield
finally: print([Link]()-t)

● Catch specific exceptions (never bare `except:`). Use `with` (context managers via
`__enter__/__exit__`) for files/locks/DB txns. Raise custom exceptions (`class
ValidationError(Exception): pass`) and chain with `raise ... from e`.
7. Modules, Packaging & Project Structure
● Module = a .py file; package = a directory with `__init__.py`. Import: `from [Link] import fn`. `if
__name__ == '__main__':` guards script entry points.
● Virtual environments — `python -m venv .venv` → `source .venv/bin/activate`; install with `pip
install -r [Link]`; pin versions. (Tools: pip, poetry, uv, conda.)
● Project layout — `src/pkg/`, `tests/`, `[Link]` (build config + deps), `[Link]`,
`README`. Editable install: `pip install -e .`.
myproject/
[Link]
src/etl/__init__.py
src/etl/[Link]
src/etl/[Link]
tests/test_transform.py
8. Type Hints (typing)
from typing import Optional, Union, Any, Callable, Iterator
from [Link] import Sequence

def parse(rows: list[dict[str, Any]],

key: str,
default: Optional[int] = None) -> dict[str, int]:
...

Reader = Callable[[str], Iterator[str]] # type alias

x: int | None = None # 3.10+ union syntax

● Type hints are optional + not enforced at runtime but enable mypy/pyright static checking, better
IDE help, and self-documenting APIs — expected in production DE code. Use `dataclasses`/`pydantic`
for typed records + validation.
9. Standard Library for Data Engineering
● collections — Counter (freq), defaultdict (grouping), deque (queues), namedtuple/dataclass
(records).
● itertools/functools — chain/islice/groupby, reduce/lru_cache/partial.
● datetime — parsing/formatting, timezones (`zoneinfo`), arithmetic (`timedelta`).
● pathlib — modern paths (`Path('/x')/'y'`, `.glob`, `.exists`); os/shutil for FS ops.
● json / csv — read/write; gzip/zipfile for compressed data.
● logging — structured logs (levels, handlers, formatters) — never `print` in pipelines.
● argparse — CLI args for scripts. configparser / [Link] — config/secrets. subprocess — run
external tools.
import logging
[Link](level=[Link],
format="%(asctime)s %(levelname)s %(name)s %(message)s")
log = [Link]("etl")
[Link]("processed %d rows", n)
10. File I/O & Data Formats
import csv, json
# stream a large CSV (don't load all rows)
with open("[Link]", newline="") as f:
for row in [Link](f):
handle(row)

# newline-delimited JSON (JSONL) - common in DE

with open("[Link]") as f:
records = ([Link](line) for line in f) # lazy

# chunked read of a huge file

def chunks(path, size=10000):
batch = []
with open(path) as f:
for line in f:
[Link](line)
if len(batch) == size:
yield batch; batch = []
if batch: yield batch

● Formats: CSV/TSV (simple), JSON/JSONL (semi-structured), Parquet/ORC (columnar — via

pyarrow/pandas, preferred for analytics), Avro. Stream large files with generators; process in
chunks/batches.
11. Memory, GIL & Performance
● Reference counting frees objects at refcount 0; a cyclic GC handles reference cycles. Small ints (-
5..256) and some strings are interned/cached.
● GIL (Global Interpreter Lock) — in CPython only ONE thread runs Python bytecode at a time →
threads don't parallelize CPU-bound pure-Python work.
● `is` vs `==` — `is` = identity (same object); `==` = value. Interning makes `is` unreliable for value
comparison; use `is` only for `None`.
● Shallow vs deep copy — `[Link]` shares nested refs; `[Link]` recurses.
● Performance levers: use built-ins/comprehensions (C-level), generators for memory, `__slots__` to
cut per-object memory, local-variable caching in hot loops, vectorize with NumPy/pandas instead of
Python loops.
class Point:
__slots__ = ("x", "y") # no __dict__ -> less memory
def __init__(self, x, y): self.x, self.y = x, y

# profiling
# python -m cProfile -s cumtime [Link]
# import timeit; [Link]("f()", setup="from __main__ import f")
12. Concurrency & Parallelism
● threading — good for I/O-bound work (GIL released during I/O: network/disk). Use
`[Link]`.
● multiprocessing — true parallelism for CPU-bound work (separate processes, separate GILs). Use
`ProcessPoolExecutor`. Watch pickling overhead of args/results.
● asyncio — single-threaded cooperative concurrency for high-I/O fan-out (thousands of sockets):
`async def` + `await` + event loop.
from [Link] import ThreadPoolExecutor
def fetch(url): ...
with ThreadPoolExecutor(max_workers=8) as ex:
results = list([Link](fetch, urls)) # I/O-bound: threads help

from [Link] import ProcessPoolExecutor

def crunch(x): return heavy(x)
with ProcessPoolExecutor() as ex: # CPU-bound: processes
results = list([Link](crunch, data))

Decision: I/O-bound → threads/asyncio; CPU-bound → multiprocessing (or push compute to NumPy/Spark).

13. NumPy for Data Engineering
● ndarray — a fixed-type, contiguous array → vectorized operations run in C (10–100× faster than
Python loops), with far less memory than lists.
import numpy as np
a = [Link]([1,2,3,4], dtype=np.int64)
a * 2 # [2,4,6,8] vectorized (no loop)
a[a > 2] # boolean mask -> [3,4]
[Link](); [Link](); [Link]()
m = [Link](12).reshape(3,4) # 3x4 matrix
[Link](axis=0) # column sums
[Link](a > 2, a, 0) # conditional

● Key ideas: vectorization (avoid Python loops), broadcasting (operate on mismatched shapes),
boolean masking, `dtype` control, `axis` semantics. NumPy underpins pandas.
14. Pandas for Data Engineering
import pandas as pd
df = pd.read_csv("[Link]") # or read_parquet/read_json
df = pd.read_parquet("[Link]") # columnar, preferred

# select / filter / new column

df[["id","amount"]]
df[[Link] > 1000]
df = [Link](amount_usd = [Link] * 0.012)

# groupby aggregation (SQL GROUP BY)

([Link]("country")
.agg(total=("amount","sum"), n=("id","count"), avg=("amount","mean"))
.reset_index())

# join / merge
[Link](a, b, on="id", how="left")

# nulls, dedup, sort

[Link](subset=["amount"]); [Link]({"amount":0})
df.drop_duplicates(subset=["id"], keep="last")
df.sort_values("amount", ascending=False)

# window-ish: rank / cumulative / rolling

df["rnk"] = [Link]("country")["amount"].rank(ascending=False)
df["running"] = [Link]("country")["amount"].cumsum()
df["mov3"] = df["amount"].rolling(3).mean()

# apply a function (last resort - slow; prefer vectorized)

df["tier"] = [Link]([Link] > 1000, "HIGH", "LOW")

● Best practices: prefer vectorized ops + `groupby/agg/merge` over `.apply`/loops; read Parquet not
CSV for big data; use `dtype`/`category` to cut memory; process in chunks (`read_csv(...,
chunksize=)`) for large files; know when to switch to Spark/Dask (data > RAM).
● pandas vs Spark: pandas = single-machine, in-memory (great < a few GB); Spark = distributed
(bigger-than-RAM / cluster). Same mental model (DataFrames, groupBy, join).
15. Testing (pytest)
# [Link]
def clean_amount(x):
return 0.0 if x is None else round(float(x), 2)

# tests/test_transform.py
import pytest
from [Link] import clean_amount

def test_none_becomes_zero():
assert clean_amount(None) == 0.0

@[Link]("inp,exp", [("1.239",1.24),(2,2.0)])
def test_rounding(inp, exp):
assert clean_amount(inp) == exp

@[Link]
def sample_df():
import pandas as pd
return [Link]({"amount":[1,None,3]})

def test_pipeline(sample_df):
assert sample_df["amount"].fillna(0).sum() == 4

● pytest — plain `assert`, fixtures (reusable setup), parametrize (data-driven cases), `[Link]` for
exceptions, mock/patch (`[Link]`) for external calls (DB/API/S3). Aim for unit tests on
transforms + integration tests on pipelines; run in CI.
16. Data-Engineering Patterns
● Idempotent ETL — re-running yields the same result (dedup by key, upsert/MERGE, overwrite
partitions).
● Chunked/streaming processing — generators + batches so memory stays flat on huge inputs.
● Retry with backoff — decorate flaky I/O (APIs/DB) with retries + exponential delay (Part 3).
● Config + secrets — env vars / config files, never hard-code credentials; use `[Link]` / a secrets
manager.
● Structured logging + metrics — log record counts, durations, failures; make pipelines observable.
● Schema validation — validate incoming data (pydantic/dataclasses/great_expectations) before load;
fail fast on bad data.
# a clean ETL script skeleton
def extract(path): ... # yield records (generator)
def transform(rec): ... # pure, testable
def load(batch): ... # idempotent write

def main():
batch = []
for rec in extract(SRC):
[Link](transform(rec))
if len(batch) >= 10000:
load(batch); [Link]()
if batch: load(batch)

if __name__ == "__main__":
main()
17. Common Pitfalls & Gotchas
● Mutable default argument — `def f(x=[])` shares ONE list across calls. Fix: `def f(x=None): x = x or []`.
● Late-binding closures — `funcs = [lambda: i for i in range(3)]` all return 2. Fix: `lambda i=i: i` (bind
now).
● `is` vs `==` — comparing values with `is` breaks unpredictably (interning). Use `==`; `is` only for None.
● Modifying a list while iterating — skips/repeats. Iterate a copy or build a new list.
● Float precision — `0.1 + 0.2 != 0.3`; use `[Link]` or `[Link]` for money/comparisons.
● Shared mutable class attribute — a list defined at class level is shared by all instances; init mutables
in `__init__`.
● `UnboundLocalError` — assigning a name inside a function shadows the global; use
`global`/`nonlocal` if intended.
● Catching too broadly — bare `except:` hides bugs and swallows KeyboardInterrupt; catch specific
types.
18. Big-O Complexity Reference
● list: index O(1), append amortized O(1), insert/pop(0) O(n), `in`/search O(n), sort O(n log n).
● dict / set: get/set/del/`in` O(1) average, O(n) worst (collisions).
● deque: append/appendleft/pop/popleft O(1).
● heapq (min-heap): push/pop O(log n), peek O(1) — top-K, priority queues.
● string: concat in a loop O(n^2) (use `''.join`), slicing O(k).
● Common algos: binary search O(log n), two-pointer/sliding-window O(n), sort O(n log n), hashing
pass O(n), nested loops O(n^2).
Rule: know the complexity of each built-in operation; interviewers expect you to state time AND space and
improve a brute-force O(n^2) to O(n) with a hash/two-pointer/sliding-window.
19. Solved Coding Practice Set (the core drill)
Rewrite each from a blank page until fluent; state the Big-O each time. Patterns: hashing, two-pointer, sliding
window, heap, stack, recursion/DP, and DE-flavored parsing/grouping.

Hashing / arrays
● P1 — Two Sum (indices summing to target). O(n).
def two_sum(nums, target):
seen = {}
for i, x in enumerate(nums):
if target - x in seen:
return [seen[target - x], i]
seen[x] = i

● P2 — Group anagrams. O(n·k log k).

from collections import defaultdict
def group_anagrams(words):
g = defaultdict(list)
for w in words:
g["".join(sorted(w))].append(w)
return list([Link]())

● P3 — First non-repeating character. O(n).

from collections import Counter
def first_unique(s):
c = Counter(s)
for i, ch in enumerate(s):
if c[ch] == 1: return i
return -1

● P4 — Find duplicates / dedup preserving order. O(n).

def dedup(seq):
seen = set(); out = []
for x in seq:
if x not in seen:
[Link](x); [Link](x)
return out

Two-pointer / sliding window

● P5 — Pair sum in a sorted array. O(n).
def pair_sum(a, target):
i, j = 0, len(a)-1
while i < j:
s = a[i] + a[j]
if s == target: return (i, j)
if s < target: i += 1
else: j -= 1

● P6 — Longest substring without repeating chars. O(n).

def longest_unique(s):
last = {}; start = 0; best = 0
for i, ch in enumerate(s):
if ch in last and last[ch] >= start:
start = last[ch] + 1
last[ch] = i
best = max(best, i - start + 1)
return best
 ● P7 — Max sum subarray of size k. O(n).
def max_window(a, k):
s = sum(a[:k]); best = s
for i in range(k, len(a)):
s += a[i] - a[i-k]
best = max(best, s)
return best

● P8 — Kadane: max subarray sum. O(n).

def max_subarray(a):
cur = best = a[0]
for x in a[1:]:
cur = max(x, cur + x)
best = max(best, cur)
return best

Heap / stack
● P9 — Top-K frequent elements. O(n log k).
import heapq
from collections import Counter
def top_k(nums, k):
c = Counter(nums)
return [x for x,_ in [Link](k, [Link](), key=lambda kv: kv[1])]

● P10 — Valid parentheses. O(n).

def valid(s):
pairs = {")":"(","]":"[","}":"{"}; st = []
for ch in s:
if ch in "([{": [Link](ch)
elif not st or [Link]() != pairs[ch]: return False
return not st

● P11 — Merge intervals. O(n log n).

def merge(intervals):
[Link]()
out = []
for s, e in intervals:
if out and s <= out[-1][1]:
out[-1][1] = max(out[-1][1], e)
else:
[Link]([s, e])
return out

Recursion / DP / search
● P12 — Fibonacci with memoization. O(n).
from functools import lru_cache
@lru_cache(maxsize=None)
def fib(n):
return n if n < 2 else fib(n-1) + fib(n-2)

● P13 — Binary search. O(log n).

def bsearch(a, t):
lo, hi = 0, len(a)-1
while lo <= hi:
mid = (lo + hi)//2
if a[mid] == t: return mid
if a[mid] < t: lo = mid+1
else: hi = mid-1
 return -1

● P14 — Merge two sorted lists. O(n+m).

def merge_sorted(a, b):
i = j = 0; out = []
while i < len(a) and j < len(b):
if a[i] <= b[j]: [Link](a[i]); i += 1
else: [Link](b[j]); j += 1
[Link](a[i:]); [Link](b[j:])
return out

● P15 — Reverse a linked list. O(n).

class Node:
def __init__(self, v, nxt=None): self.v, [Link] = v, nxt
def reverse(head):
prev = None
while head:
[Link], prev, head = prev, head, [Link]
return prev

DE-flavored
● P16 — Word frequency (word count). O(total chars).
from collections import Counter
def word_count(text):
return Counter([Link]().split())

● P17 — Group-by aggregation (SQL GROUP BY in Python).

from collections import defaultdict
def group_sum(rows, key, val):
agg = defaultdict(float)
for r in rows:
agg[r[key]] += r[val]
return dict(agg)

● P18 — Flatten a nested list (arbitrary depth).

def flatten(x):
for e in x:
if isinstance(e, list):
yield from flatten(e)
else:
yield e

● P19 — Running/moving average over a stream.

from collections import deque
def moving_avg(stream, k):
win = deque(maxlen=k); s = 0
for x in stream:
if len(win) == k: s -= win[0]
[Link](x); s += x
yield s / len(win)

● P20 — Chunk an iterable into batches (for bulk loads).

from itertools import islice
def batched(it, size):
it = iter(it)
while (chunk := list(islice(it, size))):
yield chunk
20. Scenario-Based Questions
Q1. A function `def add(item, items=[])` accumulates across calls. Why + fix?
→ The default list is created once and shared (mutable default). Fix: items=None then items = items or []
inside.
Q2. Process a 50 GB log with low memory. Approach?
→ Generator that yields one line at a time (for line in f: yield ...); process lazily / in chunks — never
readlines().
Q3. `a=[1,2]; b=a; [Link](3)` — what is a and why?
→ a is [1,2,3]; b copies the reference, not the list. Use [Link]()/list(a) for an independent copy.
Q4. A CPU-bound job doesn't speed up with threads. Why + fix?
→ The GIL serializes Python bytecode → threads don't parallelize CPU work. Use multiprocessing (or push
to NumPy/Spark).
Q5. Your pandas `.apply` over 10M rows is slow. Speed up.
→ Replace row-wise apply with vectorized ops / [Link] / groupby-agg; use categorical dtypes; read
Parquet; chunk or move to Spark if > RAM.
Q6. Guarantee a file/DB connection is closed even on error. How?
→ Use a context manager (with-statement / @contextmanager) — __exit__ runs even on exceptions.
Q7. Cache expensive pure-function results by args. Simplest way?
→ @functools.lru_cache(maxsize=...) — memoizes by (hashable) arguments.
Q8. Dedup a huge list fast. Structure?
→ Convert to a set for O(1) membership (loses order), or [Link](seq) to dedup preserving order.
Q9. Make an ETL step idempotent so re-runs don't duplicate data.
→ Dedup by natural key (ROW_NUMBER-style / drop_duplicates), upsert/MERGE, or overwrite the target
partition.
Q10. Improve a brute-force O(n^2) pair-search. How?
→ Use a hash set/dict for O(n) (Two Sum pattern) or two pointers on a sorted array for O(n log n) total.
21. FAANG / JPMC-Level Interview Questions
1. Mutable vs immutable — examples and one practical consequence.
A: Immutable: int/str/tuple/frozenset (hashable → dict keys, safe defaults). Mutable: list/dict/set (aliasing
bugs, not hashable). Consequence: mutable defaults + shared refs cause surprising state; immutables are
safe keys/defaults.
2. Explain the GIL and how you get real parallelism.
A: CPython's GIL lets one thread run bytecode at a time, so CPU-bound threads don't scale. Use
multiprocessing (separate processes/GILs), C-extensions/NumPy (release the GIL), or a cluster engine
(Spark). Threads/asyncio still help I/O-bound work.
3. Why are dict/set O(1), and what breaks it?
A: They're hash tables: hash(key) → bucket → compare. Requires hashable (immutable) keys. Many
collisions or a poor hash degrade to O(n); resizing keeps load factor low.
4. Generators vs lists — when and why?
A: Generators yield lazily with O(1) memory (ideal for large/streaming data, pipelines, infinite sequences);
lists materialize everything. Use generators to keep memory flat and enable streaming ETL.
5. Decorators — what are they and give a production use.
A: A callable that wraps a function, applied with @. Uses: logging/timing, retry with backoff, caching
(lru_cache), auth. Use [Link] to preserve metadata; parametrized decorators add a layer.
6. threading vs multiprocessing vs asyncio — pick one for: web-scraping 10k URLs; hashing 10k files; a high-
concurrency socket server.
A: Scraping (I/O-bound) → threads or asyncio; hashing (CPU-bound) → multiprocessing; socket server
(massive I/O fan-out) → asyncio.
7. How do you make a data pipeline memory-safe on huge inputs?
A: Stream with generators, process in chunks/batches, avoid loading whole files, use columnar Parquet +
dtype/category in pandas, and switch to Spark/Dask when data exceeds RAM.
8. is vs == and a gotcha.
A: is = identity (same object); == = value. Small-int/str interning makes is unreliable for values (256 is 256
True, 257 is 257 maybe False). Use == for values, is only for None.
9. Explain the four OOP pillars in Python with a DE example.
A: Encapsulation (private state in a Source class), Inheritance (KafkaSource(Source) via super()),
Polymorphism (each [Link]() overridden / duck-typed), Abstraction (abstract base Source with
@abstractmethod read()).
10. Improve this brute force: count pairs in a list summing to K.
A: Replace the O(n^2) double loop with a single pass + hash set/dict: for each x check if K-x was seen —
O(n) time, O(n) space (Two Sum pattern).
11. How do you test a data transform, and why does it matter?
A: pytest unit tests on pure transform functions (parametrize edge cases: nulls, bad types, empty), fixtures
for sample DataFrames, mock external I/O (DB/API/S3), and run in CI — catching data bugs before they hit
production.
12. When do you leave pandas for Spark, and what stays the same?
A: Leave pandas when data exceeds single-machine memory or you need distributed compute; the
DataFrame mental model (select/filter/groupBy/join) transfers, but Spark is lazy, distributed, and
partitioned.
13. Design a retry decorator with exponential backoff.
A: Wrap the function; loop up to n attempts; on exception sleep delay*2**attempt (with optional jitter);
re-raise on the last attempt; use [Link] and make n/base/max configurable — apply to flaky
API/DB calls.
Appendix: Quick Reference & Mastery Checklist
Idioms
a, b = b, a # swap
first, *rest = [1,2,3,4] # unpack
for i, v in enumerate(xs): ...
for a, b in zip(la, lb): ...
",".join(str(n) for n in nums)
{k: v for k, v in pairs} # dict comp
val = "P" if x > 0 else "N" # ternary
all(v > 0 for v in nums)

Golden rules
● Never use a mutable default arg; bind loop vars in closures.
● == for value, is only for None; catch specific exceptions.
● Generators for big data; comprehensions for clarity; with for cleanup.
● GIL: threads for I/O, multiprocessing for CPU, NumPy/Spark for scale.
● Vectorize with NumPy/pandas; type-hint + test production code.

Mastery checklist
● Solve the 20 coding problems from a blank page, stating Big-O.
● Turn an O(n^2) brute force into O(n) with a hash/two-pointer/window.
● Write clean pandas: groupby/agg/merge, no needless apply.
● Explain GIL, generators, decorators, and OOP pillars clearly.
● Structure + test an idempotent, chunked ETL script.
● Choose threading/multiprocessing/asyncio correctly per workload.