INTEGRATED DATA SCIENCE AND PYTHON
FOR ENHANCED INSIGHTS
PYTHON LIBRARIES FOR DATA WRANGLING
NumPy (Basics of NumPy arrays)
1) Arrays (creation & types)
[Link] is the main container: homogeneous typed n-dimensional arrays.
Shape, dtype, ndim, size
2) Aggregations
[Link], [Link], [Link], [Link], [Link], [Link], [Link], [Link].
Axis parameter: axis=0 (columns), axis=1 (rows)
3) Computations on arrays (vectorized ops)
Elementwise arithmetic: add, subtract, multiply, divide.
Matrix multiplication: @ or [Link].
4) Comparisons, masks, boolean logic
Comparisons return boolean arrays: a > 3.
Masks filter: a[a>3].
Boolean logic: np.logical_and, np.logical_or, ~ (not).
5) Fancy indexing
Integer array indexing: a[[0,2,4]].
Boolean indexing used above.
Slicing returns views (usually) whereas fancy indexing returns copies.
6) Structured arrays
Allow fields like columns in a NumPy array with different dtypes (like lightweight tables).
Example use-cases: small datasets where you want NumPy speed but per-column dtypes.
Data manipulation with Pandas
Pandas is the main library for tabular data: Series (1D) and DataFrame (2D).
1) Data index & selection
Indexing rows: .loc (label-based) and .iloc (position-based).
Column selection: df ['col'] or [Link].
2) Operating on data
Vectorized ops: df['A'] * 2, df[['A','B']].sum(axis=1).
Use apply for custom row/column operations (but prefer vectorized ops for speed)
3) Missing data
Represented as NaN for floats, [Link] optionally.
Functions: [Link](), [Link](), [Link](value), [Link]().
4) Hierarchical indexing (MultiIndex)
Multi-level index for rows and/or columns (useful for panel-like data).
Create with [Link].from_tuples or df.set_index([col1,col2]).
5) Combining data (merge/concat/join)
[Link]([df1, df2]) for stacking (axis=0 or axis=1).
[Link](df1, df2, on='key', how='inner|left|right|outer') for relational joins (SQL-like).
[Link] () for index-based joins.
6) Aggregation and grouping (groupby)
[Link]('col').agg({'col2':['mean','sum']}) or [Link]('col')['col2'].mean().
Useful for split-apply-combine patterns.
7) Pivot tables
[Link](index='row', columns='col', values='val')pd.
pivot_table(df, values='val', index=['row1'], columns=['col1'], aggfunc='mean')Pivot tables generalize groupby +
reshape.
DESCRIBING RELATIONSHIPS
Scatter plots
What: A scatter plot displays paired observations (𝑥𝑖,𝑦𝑖)(xi,yi) as points on a 2D plane.
Why: Visualize relationship: linear, nonlinear, clusters, outliers, heteroscedasticity.
Key points to look for on a scatter plotDirection:
positive (upward), negative (downward), none.
Form: linear vs non-linear (curved).
Strength: tight cloud (strong) vs spread out (weak).
Outliers/influential points.
correlation coefficient for quantitative data
What: Pearson correlation coefficient 𝑟r measures linear association between two quantitative variables X
and Y.
Formula (sample):
Ranges from −1 to +1.
Interpretation
𝑟≈1: strong positive linear relationship.
𝑟≈−1: strong negative linear relationship.
𝑟≈0: no linear relationship (could be non-linear).
Magnitude indicates strength, sign indicates direction.
Computational note: Pearson r is sensitive to outliers; always inspect scatter plot.
Regression — regression line
What: Regression fits a model predicting Y from X. For simple linear regression:
𝑦^=𝑏0+𝑏1𝑥
where b1= slope and b0= intercept (estimated by least squares).
Least squares estimates
Derivation idea: Minimizes sum of squared
residuals ∑(yi−y^i)2.
LEAST SQUARES REGRESSION LINE
Residuals: 𝑒𝑖=𝑦𝑖−𝑦^𝑖. Least squares minimizes ∑𝑒𝑖^2
Assumptions of classical linear regression (for inference):
Linearity: relationship is linear in parameters.
Independence of errors.
Homoscedasticity: constant variance of errors.
Normality of errors (for small-sample inference).
No perfect multicollinearity (relevant in multiple regression).
Diagnostics to run
Residual vs fitted plot (check non-linearity).
Q-Q plot of residuals (normality).
Leverage and Cook’s distance (influential points).
Error of estimate (standard error of estimate)
What: A measure of the spread of observed y around the regression line — essentially the standard
deviation of residuals (root-mean-square error, RMSE).
Formula (for simple regression):
The denominator is 𝑛−2 because two parameters were estimated (intercept & slope).
Interpretation of r2 (coefficient of determination)
What: 𝑅2 = proportion of variance in Y explained by the model. For simple regression:
R2=r2
Ranges 0 to 1. Example: R2=0.7 means 70% of variance in Y explained by X.
Caveats: High 𝑅2 does not imply causation; adding predictors always increases R2 (use adjusted R2 to penalize).
Multiple regression equations (multiple linear regression
Model: Predict 𝑌Y using multiple predictors 𝑋1,𝑋2,…,𝑋𝑝
Matrix form: β^=(X⊤X)−1X⊤y (ordinary least squares).
Interpretation: Each βj is effect of Xj holding other variables constant.
Concerns:
• Multicollinearity (predictors highly correlated) inflates variance of estimates.
• Overfitting if too many predictors for sample size.
• Need diagnostics analogous to simple regression.
Regression towards the mean
What: If a variable is extreme on its first measurement, it tends to be closer to the mean on a second
measurement (random variability). Important when interpreting changes — e.g., athletes' performance or
test scores.
Implication: Be cautious attributing causality to interventions if pre/post contrast could be due to
regression to the mean.
Thank You
Feel free to ask if you have any further
questions or encounter any issues. I'm here to
help!
Contact: + 91 8489317579