Python + Pandas Study Notes

A 12-page original guide for data projects, pandas workflows, and research coding habits.

Created as a practical reference for building cleaner Python data workflows and avoiding common pandas
mistakes.

What this PDF is for

This is an original learning handout designed for quick review, tutoring, interview preparation, or personal study.
It is not a copied textbook chapter. Work through the examples actively: cover the solution, attempt the
problem, then compare your steps.

How to use it
●
Read one page at a time and write out the key method in your own words.
●
Redo each worked example without looking after you understand it once.
●
Use the practice prompts as a quick check rather than a full course replacement.
●
Mark any weak spots and turn them into a short daily review list.

Python + Pandas Study Notes Page 1

 1. The practical Python mindset
For data work, Python is less about memorizing syntax and more about building a repeatable workflow. You
want code that is correct, readable, and easy to rerun. A clean notebook is useful for exploration, but a script is
better for a pipeline step you will run many times.

Think in layers: load data, clean data, create features, model, evaluate, and save outputs. Each layer should be
understandable on its own.
●
Use notebooks for discovery and explanation.
●
Use scripts for repeatable steps.
●
Keep raw data separate from processed data.
●
Name variables for meaning, not for convenience.
●
Prefer small functions over one giant cell.

Worked example
Instead of one cell that downloads, cleans, models, and plots everything, split it into functions: load_prices(),
make_features(), create_labels(), fit_model(), evaluate_model().

Practice
1 Write three signs that a notebook cell should become a function.

2 Explain why raw data should be kept untouched.

3 Name the main stages of a data project pipeline.

Python + Pandas Study Notes Page 2

 2. Core Python containers
Lists, tuples, dictionaries, and sets are the everyday building blocks of Python. Knowing when to use each one
makes your code clearer and faster.

A list is ordered and mutable. A tuple is ordered and often used for fixed pairs or records. A dictionary maps
keys to values. A set stores unique items and supports fast membership checks.
●
List: tickers = ["SPY", "QQQ", "IWM"]
●
Tuple: date_ticker = ("2024-01-05", "SPY")
●
Dict: weights = {"SPY": 0.5, "QQQ": 0.5}
●
Set: unique_tickers = {"SPY", "QQQ", "IWM"}
●
Use dictionaries when you need lookup by name.

Code sketch
tickers = ["SPY", "QQQ", "IWM"]
weights = {ticker: 1 / len(tickers) for ticker in tickers}
print(weights["SPY"])

Practice
1 When would a set be better than a list?

2 Create a dictionary mapping three tickers to sectors.

3 Explain what a tuple can represent in a MultiIndex.

Python + Pandas Study Notes Page 3

 3. Functions and clean scripts
Functions let you give a name to a piece of logic. A good function should have a clear input, clear output, and a
clear reason to exist. If a function changes global variables or silently depends on hidden state, it becomes
harder to test.

For research code, functions are especially useful because they let you rerun the same experiment with
different parameters.
●
Use verbs in function names: load_data, calculate_returns, evaluate_model.
●
Keep functions short enough that you can explain them in one sentence.
●
Return values instead of printing everything.
●
Add basic checks for expected columns or missing values.
●
Use if __name__ == "__main__" for script entry points.

Code sketch
from pathlib import Path
import pandas as pd

def load_prices(path: str | Path) -> [Link]:

df = pd.read_csv(path, parse_dates=["Date"])
if "Date" not in [Link]:
raise ValueError("Expected a Date column")
return df

Practice
1 Rewrite a repeated code block as a function.

2 What should a function return if it creates features?

3 Why is printing not a substitute for returning data?

Python + Pandas Study Notes Page 4

 4. NumPy arrays and vectorization
NumPy is built around arrays. An array stores values in a rectangular structure and lets you perform operations
on many values at once. This is usually faster and clearer than writing manual loops for numerical work.

Broadcasting means NumPy can apply an operation across compatible shapes. It is powerful, but you should
check shapes when results look strange.
●
[Link] creates arrays from lists.
●
Array operations are elementwise by default.
●
[Link] handles vector and matrix products.
●
axis=0 usually means down rows; axis=1 usually means across columns.
●
Check [Link] often.

Code sketch
import numpy as np
returns = [Link]([0.01, -0.02, 0.015])
weights = [Link]([0.4, 0.3, 0.3])
portfolio_return = [Link](weights, returns)
print(portfolio_return)

Practice
1 What is the shape of a 100 by 5 return matrix?

2 Explain axis=0 in plain English.

3 Why might vectorized code be less error-prone than loops?

Python + Pandas Study Notes Page 5

 5. pandas objects: Series and DataFrame
A pandas Series is a labeled one-dimensional array. A DataFrame is a table with labeled rows and columns.
The labels are not just decoration: they control alignment during arithmetic and joining.

This alignment is helpful, but it can surprise you. If two Series have different indexes, pandas aligns by label
rather than by position. That is usually what you want for time series, but only if the indexes are correct.
●
Series: one column with an index.
●
DataFrame: many columns with an index.
●
Index: labels used for alignment and selection.
●
Use .head(), .info(), and .describe() for quick inspection.
●
Use .isna().sum() to check missing values.

Code sketch
import pandas as pd
s = [Link]([1, 2, 3], index=["a", "b", "c"])
print([Link]["b"])

Practice
1 What is the difference between .loc and .iloc?

2 Why does index alignment matter for labels and features?

3 What command checks column types?

Python + Pandas Study Notes Page 6

 6. Reading and writing project data
A good data project has clear file boundaries. Raw files should be saved as they came in. Processed files
should be created by scripts. This gives you a trail from source data to modeling data.

CSV is convenient, but it loses some type information. Always parse dates explicitly when reading and inspect
dtypes afterward.
●
Use [Link] for file paths.
●
Use parse_dates for date columns.
●
Save intermediate outputs only when they are useful checkpoints.
●
Avoid editing processed CSVs by hand.
●
Make folders like data/raw, data/processed, and reports.

Code sketch
from pathlib import Path
raw = Path("data/raw/[Link]")
processed = Path("data/processed/[Link]")
[Link](parents=True, exist_ok=True)

Practice
1 Why should raw data be read-only in your workflow?

2 What does parents=True do in mkdir?

3 When should you save an intermediate CSV?

Python + Pandas Study Notes Page 7

 7. Filtering, indexing, and MultiIndex
Indexing is where many pandas bugs begin. A simple Date index is common for one asset. For many assets
over many dates, a MultiIndex such as (Date, Ticker) is often cleaner.

A MultiIndex lets you represent panel data. The important rule is that both features and labels must use the
same index names, same date type, and same ordering before joining.
●
Use set_index(["Date", "Ticker"]) for panel data.
●
Use sort_index() before joins and slicing.
●
Use .loc for label-based selection.
●
Use reset_index() when you need columns again.
●
Check index names and dtypes when joins create unexpected NaNs.

Code sketch
df = df.set_index(["Date", "Ticker"]).sort_index()
spy = [Link]("SPY", level="Ticker")
print([Link]())

Practice
1 What does xs do?

2 Why might two identical-looking dates fail to join?

3 How do you turn index levels back into columns?

Python + Pandas Study Notes Page 8

 8. Groupby, joins, and alignment
groupby lets you apply the same calculation to each group, such as each ticker or each date. Joins combine
datasets by index or columns. In finance projects, groupby and joins are everywhere.

The safest mental model is: groupby changes how calculations are applied; join changes how tables are
combined. Before joining, inspect the keys. After joining, inspect the row count and missing values.
●
groupby("Ticker") calculates separately for each ticker.
●
groupby("Date") calculates cross-sectional values for each date.
●
inner join keeps only matching keys.
●
left join keeps all keys from the left table.
●
Unexpected NaNs usually mean key mismatch or missing observations.

Code sketch
features = [Link](labels, how="inner")
missing = [Link]().sum().sort_values(ascending=False)
print([Link]())

Practice
1 When is an inner join appropriate?

2 What does groupby(level="Ticker") mean?

3 Why check row counts before and after a join?

Python + Pandas Study Notes Page 9

 9. Date handling and time series rules
Time series data has order. You cannot shuffle it casually if your model is supposed to predict the future. Any
operation that uses future information in a feature creates lookahead bias.

Rolling windows should use past data. Forward returns can be used as labels, but not as features. Train/test
splits should respect time order unless the problem is genuinely not time-based.
●
Sort by date before rolling calculations.
●
Use shift(1) if a feature must be known before the prediction date.
●
Use shift(-h) for h-step-forward labels.
●
Never let test data influence feature scaling or parameter selection.
●
Keep a final holdout that stays untouched until evaluation.

Code sketch
df["ret_5"] = [Link]("Ticker")["Close"].pct_change(5)
df["fwd_ret_5"] = [Link]("Ticker")["Close"].pct_change(5).shift(-5)

Practice
1 Why is shift(-5) valid for a label but dangerous for a feature?

2 What does a rolling 20-day volatility use?

3 Why should scalers be fit on training data only?

Python + Pandas Study Notes Page 10

 10. Feature engineering for research
Feature engineering turns raw data into model inputs. In financial data, common features include returns,
volatility, volume changes, relative performance, and ranks. The key is not to create as many features as
possible; it is to create features with a plausible reason to matter.

Every feature should have a timestamp interpretation: what would you know at the moment you make the
prediction?
●
Return features measure recent price movement.
●
Volatility features measure recent variation.
●
Relative return features compare an asset to a benchmark or universe.
●
Ranks can reduce sensitivity to outliers.
●
Winsorization or clipping can control extreme values, but should be fit carefully.

Code sketch
df["rel_ret_5"] = df["ret_5"] - [Link]("Date")["ret_5"].transform("mean")
df["vol_20"] = [Link]("Ticker")["ret_1"].rolling(20).std().reset_index(level=0, drop=True)

Practice
1 Create one feature based on relative performance.

2 What makes a feature leaky?

3 Why might ranks be useful in cross-sectional models?

Python + Pandas Study Notes Page 11

 11. Debugging and testing data code
Debugging data code is mostly about checking assumptions. Do the indexes match? Are the dates sorted? Are
there missing values? Are labels shifted correctly? Is the row count what you expected?

Small checks save hours. Add assertions when a condition must be true for the rest of the pipeline to make
sense.
●
Use [Link] after every major transformation.
●
Use [Link].is_monotonic_increasing for sorted indexes.
●
Use assert not [Link].has_duplicates when duplicates are impossible.
●
Print a few rows around boundary dates.
●
Test functions on a tiny hand-made dataset.

Code sketch
assert [Link] == ["Date", "Ticker"]
assert [Link] == ["Date", "Ticker"]
combined = [Link](labels, how="inner")
assert "label" in [Link]

Practice
1 Write an assertion for no missing labels.

2 Why are tiny toy datasets useful?

3 What should you inspect after a failed join?

Python + Pandas Study Notes Page 12

 12. Mini project checklist
Use this page as a project template. A strong small project is better than a messy large one. The goal is to
show that you can ask a question, build a dataset, run an experiment, and explain the result honestly.
●
Question: state one specific research question.
●
Data: identify source, frequency, tickers/assets, and date range.
●
Features: list every feature and when it is known.
●
Labels: define exactly what y means.
●
Split: explain train, validation/CV, and holdout.
●
Model: start simple before adding complexity.
●
Evaluation: use metrics that match the problem.
●
Robustness: test at least one alternative assumption.
●
Write-up: explain what worked, what failed, and what you would do next.

Worked example
Example question: Does purged cross-validation choose hyperparameters that generalize better than naive
cross-validation for a cross-sectional ETF classification task?

Practice
1 Write a one-sentence research question.

2 List three checks you would include in a README.

3 Explain the difference between a result and a conclusion.

Python + Pandas Study Notes Page 13