Sub-problems explained in depth
A — Reading CDF files is your foundation. CDF is a binary NASA format, not a spreadsheet.
You open it with cdflib, call .cdf_info() to see what variables are inside, then extract the
exact timestamp (Epoch) and flux variable by name. The catch is that GOES-13/15 and GOES-
16/17/18/19 use different variable names across instrument generations, so you must inspect
each file before coding. The whole ingestion layer is a loop that processes one file at a time and
concatenates 11 years into one pandas DataFrame.
B — Cleaning is where most teams underestimate the work. Three separate problems: fill values
(replace -9999 with NaN immediately), instrument spikes (use a rolling median/MAD filter but
only remove spikes when Kp < 3 — if Kp ≥ 4 it's a real storm), and short data gaps (interpolate
up to 3 hours, leave longer gaps as NaN). Crucially, log-transform the flux before any ML work
— it spans four orders of magnitude and models train extremely poorly on that raw range.
C — Feature selection is where you earn scientific credibility. You run a lagged cross-
correlation: shift each solar wind variable backward in time by 0, 1, 2 … 72 hours and compute
correlation with current flux. The plot shows that Vsw peaks at ~24–48h lag (slow belt
acceleration) and Bz peaks at ~6–12h lag (fast magnetic injection). This plot, shown in your
presentation, proves you understand the physics — not just the maths.
D — Training the model must start with a baseline (persistence model: "future = current"), then
a Gradient Boosting model, then LSTM. Never skip the baseline. Your LSTM takes a sliding
window of the past 48 hours of 5 features and outputs 3 numbers simultaneously (the three
forecast horizons). Train/val/test split must be strictly chronological — random shuffling of
time-series data is a visible red flag to domain scientists.
E — Validation has two layers. Standard RMSE/R² on the held-out GOES test set, and Heidke
Skill Score (HSS) for threshold crossings at the NOAA alert level of 1000 pfu — the HSS is far
more operationally meaningful than average error. The second layer is GRASP cross-validation:
running your GOES-trained predictions against ISRO's own Indian-longitude satellite data. Even
moderate correlation (R ~ 0.5) at Indian longitude is a genuine, honest, impressive finding.
F — Dashboard is built in Streamlit: one Python file, [Link]() for the solar wind gauges,
Plotly for the forecast chart with a shaded uncertainty band and the 1000 pfu alert line, and a
colour-coded risk badge (Green → Extreme). For the demo, replay a known storm period at 5×
speed — it looks and behaves identically to a live system.