Perfect — here’s your enhanced step-by-step workflow, rewritten to fully align with your
Yubo Wang lecture sequence and your demo notebook style.
It keeps your clear structure but now mirrors the logic, terminology, and habits shown in
your lecture slides and coded examples.
(Still no code — only workflow and reasoning.)
🧠 Assignment Workflow — Lecture-Aligned Version
(PS5841 Data Science in Finance & Insurance — HW-4: Market Model, Bootstrap, and Cross-
Validation)
0) Set up and sanity-check the data
Purpose (lecture framing): establish the sample so the model assumptions can hold —
independent, identically distributed with finite variance (slide 2).
Steps
1. Load SPY_history.csv and XLK_history.csv.
2. Keep only Date, Close, and Dividends (Open is not needed).
3. Ensure dates are monthly and properly aligned by end-of-month across both ETFs.
4. Sort by ascending date and check for missing or duplicated months.
5. Verify that both series cover the same period.
Model assumption reminder
Treat the data as independent monthly observations following
[
R_{\text{XLK}} = f(R_{\text{SPY}}) + \varepsilon,\qquad
E[\varepsilon]=0,; \text{Var}(\varepsilon)<\infty
]
and approximate (f(X)) by a linear form ( \alpha + \beta X).
1) Compute monthly total returns
Formula (from assignment)
[
R_t=\frac{\text{Close}_t+\text{Dividends}t}{\text{Close}{t-1}}-1
]
Actions
Compute this for both SPY and XLK.
Drop the first row (no lagged Close).
Confirm: no missing values and both series remain perfectly date-aligned.
Concept link: this creates the observed pairs ((x_i,y_i)) used in the sample
({(R_{\text{SPY},i},R_{\text{XLK},i})}_{i=1}^n) (slide 2).
2) Keep the most recent 60 months
Slice the last 60 aligned observations for both SPY and XLK.
These will be your (n=60) samples for the regression and resampling steps.
3) Part (a): Estimate XLK’s beta vs SPY — Simple Linear Regression
Model (market model):
[
R_{\text{XLK}} = \alpha + \beta,R_{\text{SPY}} + \varepsilon
]
Steps
1. Fit an OLS regression of XLK returns on SPY returns.
2. Save the fitted coefficients ( \hat\alpha, \hat\beta ) and fitted values ( \hat y_i ).
3. Plot: scatter of (R_{\text{SPY}}) (x-axis) vs (R_{\text{XLK}}) (y-axis), overlay the
regression line.
o Annotate the plot with ( \hat\beta ) (the estimated slope).
4. Residual check: plot residuals vs fitted values to verify no curvature or fan-shape
(homoscedasticity).
5. Report:
o ( \hat\beta ) (numerical value)
o RSE ( \hat\sigma = \sqrt{\frac{1}{n-2}\sum( y_i - \hat y_i )^2} )
o R² ( = 1 - \frac{\text{RSS}}{\text{TSS}} )
Deliverable (a)
Scatter + line figure with ( \hat\beta ) printed on it.
Reported numeric ( \hat\beta ), RSE, and R².
(Links to lecture: “Simple Linear Regression”, “Goodness of Fit” — slides 2 & 7.)
4) Part (b): Bootstrap test of “high beta?”
Hypotheses:
[
H_0: \beta = 1.10 \quad\text{vs}\quad H_a: \beta > 1.10
]
b1) Bootstrap distribution of β̂
Lecture method: “Quantifying Uncertainty via bootstrap” (slide 3).
Use residual bootstrap under H₀:
Keep SPY returns (x_i), fitted residuals (\hat\varepsilon_i), and your observed (\hat\
beta_{\text{obs}}).
For B = 5000 replications:
1. Resample residuals with replacement from (\hat\varepsilon).
2. Build synthetic XLK series under the null slope
[
R_{\text{XLK}}^{(b)} = \hat\alpha + 1.10,R_{\text{SPY}} + \varepsilon^{(b)}
]
3. Refit the regression on each synthetic dataset and record (\hat\beta^{(b)}).
Collect all (\hat\beta^{*(b)}) values → empirical null distribution.
Plot (b1):
Histogram or kernel density of (\hat\beta^{*}).
Add vertical lines for
o (E(\hat\beta^{*})) (mean),
o (se(\hat\beta^{*})) (SD),
o and optionally the observed (\hat\beta_{\text{obs}}).
Optionally compute a bootstrap 95 % CI for β̂ to show whether 1.10 lies inside it.
b2) One-sided p-value
[
p = \Pr(\hat\beta^{*} \ge \hat\beta_{\text{obs}} \mid H_0)
]
Estimate as the fraction of bootstrap β̂ ≥ β̂ _obs.
Report p to 4 decimals (reference ≈ 0.0142).
Deliverables (b)
Bootstrap distribution plot with mean & SE annotated.
Optional 95 % CI of β̂ .
One-sided p-value.
(Links to lecture: “Quantifying Uncertainty”, “Bootstrap” — slides 3–4.)
5) Part (c): Estimate prediction error — Cross-Validation
Goal: approximate expected prediction error
[
E_{\text{samples}}!\left[(\hat y - y)^2\right]
]
as defined in lecture (slide 5).
Steps
1. Use LOOCV (leave-one-out CV):
o For each month i (1 to 60), fit the model on the other 59 months.
o Predict (R_{\text{XLK},i}) and compute squared error.
o Average the 60 errors → estimated test MSE.
2. Take square root → test RMSE (monthly).
3. Compute mean absolute return of XLK over the 60 months:
(\text{mean}(|R_{\text{XLK}}|)).
4. Report the ratio ( \text{RMSE} / \text{mean}(|R_{\text{XLK}}|) ) × 100 %.
o Reference numbers: MSE ≈ 0.00064, RMSE ≈ 0.0253 (≈ 49.6 % of mean |
return| ≈ 0.05097).
Deliverables (c)
Numeric test MSE, RMSE, mean |return|, and percentage ratio.
(Links to lecture: “k-Fold Cross Validation” & “Leave-One-Out CV” — slides 5–6.)
6) Optional diagnostic — power transform awareness
As per slides 10–11, if residuals look highly skewed or variance changes with x,
note that a Box–Cox (for positive y) or Yeo–Johnson transform could stabilize variance.
(You don’t have to apply it — simply acknowledge it as a check.)
7) Final interpretation and packaging
Interpretation (as in slide 25 “That Was Easy!”):
Write 2–3 sentences summarizing what you found:
Does ( \hat\beta > 1.10 )?
Is the p-value small enough to reject H₀?
What does the RMSE relative to average movement tell you about model accuracy?
Packaging
Notebook includes:
o Data checks & return construction,
o (a) Figure + β̂ + RSE + R²,
o (b1) Bootstrap plot + (b2) p-value (+ CI),
o (c) MSE / RMSE / ratio table,
o Final interpretation cell.
Export both .ipynb and .html versions.
This version now follows your lectures line-for-line — from “truth vs model” → “OLS fit &
RSE/R²” → “bootstrap inference” → “cross-validation prediction error” → “optional
transform” → “interpretation.”
Would you like me to next turn this into a Markdown skeleton (section headers + short
prompts) ready for your Jupyter notebook?