Python Basics and Syntax

Begin by understanding what Python is and how to run code. Python is a high-level, interpreted
programming language known for its readability and clear syntax 1 . It uses English-like keywords and
simple structure, making it beginner-friendly. Python code can be written in many environments: for
example, IDLE, which comes with Python by default, provides a simple interface for writing and running
code 2 ; and Google Colab is a free, cloud-hosted Jupyter Notebook service that requires no setup and
even offers free GPUs and TPUs for computation 3 . You might start with a simple “Hello, World!” program
to verify your setup:

print("Hello, world!") # This line prints the message to the screen.

• print is a built-in Python function that outputs data to the screen or console.
• The parentheses (...) enclose the argument passed to the function.
• "Hello, world!" is a string literal (text enclosed in quotes). In this case, it’s the message that
will be printed.
• The # symbol starts a comment which explains the code; comments are ignored when the
program runs.

Variables store data and can be created by simple assignment. For example:

message = "Hello, world!"

print(message) # Prints the value stored in variable 'message'.

• message is a variable name; variable names should be descriptive and follow naming rules
(letters, numbers, and underscores, not starting with a number).
• = assigns the string "Hello, world!" to the variable message .
• The second line calls print(message) , which outputs the contents of the message variable.

Python supports basic data types like integers ( 42 ), floats ( 3.14 ), strings ( "text" ), and booleans
( True / False ). You can perform arithmetic, use lists ( [ ] ), dictionaries ( { } ), and more. For example,
a list of numbers can be defined and summed:

numbers = [1, 2, 3, 4]
total = sum(numbers) # sum() adds all items in the list.
print(total) # Outputs: 10

• numbers = [1, 2, 3, 4] creates a list containing four integers.

• sum(numbers) calls a function that returns the sum of list elements. The result is assigned to
total .

1
 • print(total) then displays the result (here, 10 ).

Moving from IDLE to Colab

To switch from a local environment (like IDLE) to an online one like Colab, simply create a new notebook on
Google Colab and copy your code. Colab notebooks allow you to run Python code in cells and visualize
outputs inline. This is helpful for data science because you can combine code, outputs, and explanatory text
in one document.

Key Python Libraries for Data Science

• pandas – powerful for data manipulation and analysis (tables, CSV files) 4 .
• NumPy – for numerical operations on arrays and matrices.
• scikit-learn – machine learning library with algorithms and tools like OneHotEncoder ,
train_test_split , etc.
• Matplotlib and Seaborn – for creating visualizations (plots, charts) 4 5 .

Each library adds functionality: for example, pandas provides DataFrame objects for tabular data, and
[Link] has plotting functions like plot() and show() .

Common Errors and Tips

• Indentation is crucial in Python. Indents (spaces or tabs) define code blocks (loops, functions). A
common error is inconsistent indentation.
• Remember case-sensitivity: Variable and variable are different.
• Use print(type(var)) to check a variable’s type if you’re unsure.
• If a function or variable is not found, ensure you have the correct import statement, e.g., import
pandas as pd .

Quiz: Python Basics

1. What does the print() function do?

2. A. Defines a variable named print
3. B. Outputs text or values to the screen (Correct)
4. C. Reads user input from the keyboard

5. D. Imports a library

6. Which of the following is a valid variable name in Python?

7. A. 2ndValue
8. B. my-var
9. C. my_var (Correct)

10. D. print

11. What will this code print?

2
 x = 5
y = 2
print(x * y)

12. A. 52
13. B. 7
14. C. 10 (Correct)
15. D. Error

References

Basic Python: its readable syntax and design philosophy 1 ; running code in IDLE 2 and Colab 3 .

Data Cleaning and Preprocessing

Real-world data is often messy. Data cleaning involves fixing or removing incorrect, corrupted, or missing
data to prepare datasets for analysis. Common tasks include handling missing values (e.g., NaN in pandas)
and removing duplicates. The goal is to build a tidy, reliable dataset for modeling.

Handling missing values: In pandas, the dropna() and fillna() methods are key. Use
[Link]() to remove rows with any missing values. As per pandas documentation: “The dropna()
method removes the rows that contain NULL values.” 6 . For example:

import pandas as pd

df = [Link]({
'A': [1, None, 3],
'B': [4, 5, None]
})
clean_df = [Link]()
print(clean_df)

• [Link]() returns a new DataFrame with only the rows that have no missing data in any
column. In this example, only row 0 would remain because rows 1 and 2 have None (missing)
values.

If you want to fill missing values instead of dropping, [Link](value) replaces NaN s with a given
value. Pandas notes that “fillna() replaces NA values with non-NA data.” 7 . For instance:

3
 filled_df = [Link](0) # Replaces all missing values with 0.
print(filled_df)

• Now any None becomes 0 in the output. This keeps the shape of the DataFrame intact.

Use dropna() when missingness is minimal or likely random; use fillna() (with mean, median, or
constant) to retain rows. You can also forward-fill ( [Link]() ) or backward-fill ( [Link]() ) to
propagate values.

Removing duplicates: Duplicate records can bias analysis. Pandas provides drop_duplicates() to
remove duplicate rows. By default, “Return DataFrame with duplicate rows removed.” 8 . For example:

df2 = [Link]({
'Name': ['Alice', 'Bob', 'Alice', 'Charlie'],
'Score': [90, 80, 90, 85]
})
unique_df = df2.drop_duplicates()
print(unique_df)

• Only one of the identical 'Alice', 90 rows would be kept. You can specify subset columns or
keep first/last occurrence with parameters.

Additional cleaning steps: After fixing missing data and duplicates, common tasks include:
- Fixing data types: Ensure numeric columns are numeric ( df['age'] = df['age'].astype(int) ).
- Dealing with outliers: Identify values far from the norm and decide whether to keep or adjust them.
- Standardizing text: Lowercase strings, strip whitespace, correct typos.

Quality data cleaning prevents garbage-in-garbage-out and builds trust in your analysis.

Quiz: Data Cleaning

1. What does [Link]() do by default?

2. A. It fills missing values with the mean.
3. B. It removes rows that contain any missing values. (Correct)
4. C. It replaces missing values with zeros.

5. D. It normalizes numeric columns.

6. How can you fill missing values in pandas?

7. A. Using drop_duplicates()
8. B. Using fillna() (Correct)
9. C. Using fill_values()

10. D. Using handle_missing()

4
 11. What is the effect of df2.drop_duplicates(subset=['Name']) ?

12. A. Removes all rows with duplicate names, keeping only unique names. (Correct)
13. B. Removes duplicate scores only.
14. C. Fills duplicate names with NaN.
15. D. Combines duplicates into one row.

References

Pandas methods for missing values and duplicates 6 7 8 .

Data Visualization (Matplotlib, Seaborn)

Visualizing data helps uncover patterns and communicate findings. The go-to Python libraries are
Matplotlib and Seaborn. Matplotlib is a foundational plotting library: “Matplotlib is a comprehensive library
for creating static, animated, and interactive visualizations in Python.” 5 . Seaborn builds on Matplotlib with a
simpler interface and attractive default styles: it is “a Python visualization library built on top of Matplotlib,
designed for creating attractive and informative statistical graphics.” 4 .

Matplotlib Basics

Import Matplotlib’s pyplot interface (usually aliased as plt ):

import [Link] as plt

[Link](figsize=(6,4)) # Creates a new figure of given size (width,
height in inches).
x = [1, 2, 3, 4]
y = [2, 4, 1, 3]
[Link](x, y, label='Line')
# Plots a line connecting (1,2), (2,4), (3,1), (4,3).
[Link]('X-axis') # Label for the x-axis.
[Link]('Y-axis') # Label for the y-axis.
[Link]('Simple Line Plot') # Title of the plot.
[Link]() # Displays the legend.
[Link]() # Renders and displays the figure.

• [Link](figsize=(6,4)) creates a blank canvas (Figure) to draw on, with specified size.
• [Link](x, y) draws a line plot with points (x[i], y[i]) . It accepts many parameters
(color, linestyle, etc.).
• [Link]/ylabel/title() add labels and a title to make the plot self-explanatory.
• [Link]() shows labels (set via label='Line' ) in the corner.
• [Link]() tells Python to display the plot. In Jupyter or Colab, it is often optional because cells
auto-render, but in scripts it’s needed.

You can also create scatter plots, histograms, bar charts, etc. For example:

5
 [Link](['A', 'B', 'C'], [10, 5, 15])
[Link]('Bar Chart')
[Link]()

Matplotlib’s syntax can be procedural (as above) or object-oriented using Figure and Axes objects (for
more control). The example in the docs:

fig, ax = [Link]() # Creates a Figure and a single Axes.

[Link]([1,2,3,4], [1,4,2,3]) # Plot on that Axes object.
[Link]()

Matplotlib can output publication-quality figures and supports interactive features (zooming, panning) in
GUI backends.

Seaborn for Statistical Graphics

Seaborn simplifies common statistical plots with pleasing styles. It integrates with pandas DataFrames
easily. First, import Seaborn (often alias sns ):

import seaborn as sns

sns.set_style('whitegrid') # Sets a background style.

# Example: Create a scatter plot with regression line

[Link](x='age', y='income', data=df, hue='gender', markers=['o','x'])
[Link]('Income vs Age by Gender')
[Link]()

• sns.set_style('whitegrid') changes the default background to a clean grid.

• [Link] plots data and fits a linear regression line. Here hue='gender' colors by gender
category, and different markers are used for each group.
• Seaborn has many plot functions like [Link]() , [Link]() , [Link]() ,
and more, often requiring fewer lines than pure Matplotlib. For example, plotting a heatmap of
correlations:

corr = [Link]()
[Link](corr, annot=True, square=True, cmap='coolwarm')
[Link]('Feature Correlation Heatmap')
[Link]()

• [Link]() computes the correlation matrix.

• [Link] visualizes it, annot=True adds the correlation numbers on the squares.

6
Matplotlib and Seaborn together “make easy things easy and hard things possible” 5 when it comes to
plotting.

Quiz: Data Visualization

1. What library is Seaborn built on?

2. A. NumPy
3. B. Matplotlib (Correct)
4. C. Plotly

5. D. Bokeh

6. What does sns.set_style('whitegrid') do?

7. A. Enables 3D plots.
8. B. Sets a background style with white grid lines. (Correct)
9. C. Clears all existing plots.

10. D. Specifies color palette.

11. Which command shows a Matplotlib plot in a script?

12. A. [Link]()
13. B. [Link]() (Correct)
14. C. [Link]()
15. D. print(plt)

References

Matplotlib description and capabilities 5 ; Seaborn purpose and relation to Matplotlib 4 .

Feature Engineering and Encoding

Feature engineering creates or transforms variables to improve model performance. It includes
techniques like encoding categorical variables, scaling numeric features, or deriving new features from
existing ones.

Encoding Categorical Variables

Machine learning models generally require numeric input. One-hot encoding converts a categorical
feature with k categories into k binary columns (each category has its own column) 9 . For example, if a
feature Color has values ['Red','Blue','Green'] , one-hot encoding creates three new features:
Color_Red , Color_Blue , Color_Green , with 1/0 indicating presence. In scikit-learn:

7
 from [Link] import OneHotEncoder
encoder = OneHotEncoder(sparse=False)
X = encoder.fit_transform([['Red'], ['Blue'], ['Red']])
print(X)

• OneHotEncoder() creates an encoder object.

• fit_transform learns categories and returns a new numeric array. The output might look like:

[[0., 0., 1.], # Red

[0., 1., 0.], # Blue
[0., 0., 1.]] # Red

• The order of columns corresponds to the sorted categories (e.g., ['Blue', 'Green', 'Red'] ).

One-hot encoding is essential for algorithms that cannot work directly with text labels 9 . Remember it can
create many new columns (if categories are many), which might be problematic (the “curse of
dimensionality”). Sometimes you can reduce columns by dropping one (using drop='first' ) to avoid
multicollinearity in linear models.

Label encoding (Ordinal encoding) is another approach: convert categories to integer labels. This is done
for target labels with LabelEncoder in scikit-learn. It assigns each class a number between 0 and n-1
10 . For example:

from [Link] import LabelEncoder

le = LabelEncoder()
y = le.fit_transform(['spam', 'ham', 'spam'])
print(y) # Might output: [1, 0, 1]

• Here, le.fit_transform learned that ‘ham’ and ‘spam’ map to some integer classes.
• LabelEncoder is for encoding the target label (often the y in supervised learning) 10 . It’s not
recommended for input features if the categories are nominal, because the numeric values might
imply an order. If categories have an inherent order, consider using ordinal encoding or mapping
manually.

Scaling and Normalization

For numeric features, scaling can help. Common techniques:

- Min-max scaling: transforms data to [0,1] range. Use [Link] .
- Standardization: centers data to mean 0 and variance 1 using StandardScaler .

Example of standardization:

8
 from [Link] import StandardScaler
scaler = StandardScaler()
scaled_feats = scaler.fit_transform(df[['height', 'weight']])

• This is important for distance-based models (KNN, SVM) or methods that assume normal
distribution. It prevents features with large numeric ranges from dominating the model.

Creating New Features

Sometimes combining or extracting information yields better predictors. For example, from the Titanic
dataset: create a new feature FamilySize as the sum of SibSp + Parch + 1 (siblings + parents +
self). Or extract titles from names (e.g., 'Mr', 'Mrs') using string methods or regex, as shown:

import re

def get_title(name):
match = [Link](' ([A-Za-z]+)\.', name)
return [Link](1) if match else ''

df['Title'] = df['Name'].apply(get_title)

• This code defines a function to extract a title (sequence of letters followed by a dot) from the
Name .
• [Link] finds the first match. group(1) returns the matched title.
• df['Name'].apply(get_title) applies this function to each row’s Name, creating a new
column Title .

Engineering features often requires domain knowledge. Always cross-validate that new features help your
model by testing performance.

Quiz: Feature Engineering

1. What does one-hot encoding do to a categorical feature?

2. A. Converts categories to a single integer value.
3. B. Creates binary columns for each category. (Correct)
4. C. Randomly assigns numbers to categories.

5. D. Normalizes numeric features.

6. LabelEncoder should primarily be used to encode:

7. A. Continuous numerical features.

8. B. Target class labels (e.g., class names). (Correct)
9. C. Input categorical features without order.

10. D. Only text data in pandas.

9
 11. How do you use StandardScaler in scikit-learn?

12. A. StandardScaler.fit_transform(data) (Correct)

13. B. [Link](data)
14. C. [Link](data)
15. D. StandardScaler (no arguments)

References

One-hot encoding and necessity in ML 9 ; LabelEncoder for targets 10 .

Exploratory Data Analysis (EDA)

Exploratory Data Analysis (EDA) is the process of summarizing and visualizing data to understand its main
characteristics 11 . EDA helps uncover patterns, spot anomalies, test hypotheses, and check assumptions
12 . It is an essential step before formal modeling.

Key EDA steps include:

- Summary statistics: Use [Link]() in pandas to get count, mean, std, min, max, quartiles for
numerical columns 13 . For example:

[Link]()

This outputs a table with count, mean, std, min, 25%, 50%, 75%, max for each numeric column
13 . For categorical columns, it shows count, unique, top, freq .

• Value counts: For categorical data, df['column'].value_counts() shows how many entries
per category. This quickly reveals class imbalances or dominant categories.

• Grouping and aggregation: Use groupby . For instance, in Titanic:

df[['Pclass','Survived']].groupby('Pclass').mean()

Groups data by passenger class and computes average survival rate. This might show, for example,
that 1st class had a higher average survival than 3rd class. The code groups rows by Pclass and
then takes the mean of Survived in each group.

• Pivot tables: df.pivot_table(values='target', index='feature', aggfunc='mean')

can summarize a target (e.g., Survived ) by another feature.

• Correlations: Calculate [Link]() to see linear correlations between numeric features. Visualize
with [Link]([Link](), annot=True) . Highly correlated features might be redundant.

10
 • Visual EDA: Plot histograms ( [Link]() ), box plots ( [Link]() ), scatter plots
( [Link]() ), etc. For example, a histogram of ages shows age distribution; a boxplot by
category shows medians and outliers.

EDA is iterative: generate plots and stats, then investigate any interesting or unexpected results. Always
document insights: e.g., “The majority of students came from São Paulo (74%), and over 90% of students
scored at least 5.0 on the CSAT survey,” if such pattern appears. Good EDA reveals data quality issues and
guides feature engineering.

Quiz: EDA

1. What is the primary goal of EDA?

2. A. To train a machine learning model.
3. B. To clean data by removing missing values.
4. C. To summarize data’s main characteristics and discover patterns. (Correct)

5. D. To deploy a predictive model in production.

6. What does [Link]() show for numeric columns?

7. A. Data types of each column.

8. B. Count, mean, std, min, quartiles, max. (Correct)
9. C. The top and frequency of each category.

10. D. It drops null values.

11. Which library can quickly display a correlation matrix heatmap?

12. A. scikit-learn
13. B. pandas (no built-in heatmap)
14. C. seaborn (Correct)
15. D. NumPy

References

EDA concept and purpose 11 13 .

Case Studies
Working through real datasets cements understanding. We include three case studies: Titanic Survival,
Student Dropout, and EasyShop CSAT. Each demonstrates a complete workflow: data loading, cleaning,
EDA, feature engineering, modeling, and evaluation.

11
Titanic Survival (Kaggle)

Dataset: Contains passenger info (age, sex, class, etc.) and whether each person survived the Titanic
disaster. The goal is to predict survival.

Example steps:
- Load data with pd.read_csv('[Link]') .
- Data cleaning: fill missing ages (e.g., df['Age'].fillna(df['Age'].median(), inplace=True) ),
encode Sex ( df['Sex'].map({'male':0,'female':1}) ), and Embarked (e.g., map 'S','C','Q'
to 0,1,2).
- Feature engineering: Extract titles from names (e.g., use regex as in [44] ), create
FamilySize = SibSp + Parch + 1 .
- EDA: Use groupby to check survival rates by class, sex, etc. For instance, one may find that women and
children in higher classes had much higher survival rates.
- Modeling: Train a classifier (e.g., logistic regression) on features like Pclass, Sex, Age, Fare, Title, FamilySize.
- Evaluation: Assess accuracy, confusion matrix, ROC on a held-out test set.

Insight: Typically, female passengers and first-class passengers had higher survival rates. For example,
grouping by sex and averaging survival shows ~74% survival for women vs ~19% for men 14 . This aligns
with the “women and children first” policy. Titanic data is iconic for teaching EDA and feature engineering
(adding titles, handling cabin numbers, etc.).

Student Dropout

Dataset: Contains demographics, grades, and dropout indicators for students. The task is to predict if a
student will drop out.

Workflow:
- Clean data: handle missing test scores, attendance, etc.
- Convert categorical data (e.g., education level, gender) using encoding.
- Visualize distributions of grades, attendance and how they differ between dropouts vs persisters.
- Feature engineering might include: the number of failed courses, or change in grades over semesters.
- Split into train/test, train models like Random Forest.
- Evaluate using accuracy, precision/recall (since dropout is often imbalanced).

Insight: Key predictors might include socio-economic status, academic performance, and engagement
metrics. Typically, lower grades and low parental support correlate with higher dropout risk.

EasyShop CSAT (Customer Satisfaction)

Dataset: Survey scores and customer info for an e-commerce platform. Task is to predict if a customer is
“satisfied” (CSAT) or not.

Workflow:
- Data integration: merge customer survey responses with purchase history or demographics using
[Link]() .
- EDA: Look at satisfaction scores distribution, and correlate CSAT with number of purchases or support

12
calls.
- Feature engineering: Create features like average spend, frequency of purchases.
- Handle imbalanced classes if few are unsatisfied (see next section).
- Train models (e.g., XGBoost) and measure metrics like ROC AUC.

Insight: It might be found that customers with more service tickets are less satisfied, or that repeat
customers tend to give higher satisfaction.

Each case study illustrates how techniques (cleaning, encoding, EDA, modeling) come together. In a full
study guide, these cases would include step-by-step code with explanations and visualizations from start to
finish.

Quiz: Case Studies

1. In Titanic data, which group had the highest survival rate?

2. A. Male third-class passengers
3. B. Female first-class passengers (Correct)
4. C. Male first-class passengers

5. D. Children in third-class

6. Why merge multiple real-world datasets?

7. A. To increase the number of rows.

8. B. To combine related information into one analysis set. (Correct)
9. C. To reduce the number of features.

10. D. To apply PCA more effectively.

11. Which metric is crucial when dealing with imbalanced classes (e.g., churn dataset)?

12. A. ROC AUC (often important)

13. B. Accuracy (can be misleading)
14. C. Precision/Recall (for minority class) (Correct, often recall or F1 for minority)
15. D. R^2 (for regression)

Real-World Data Integration and Merging

Often you must combine data from different sources. Pandas provides tools to merge/join datasets like a
database. Use [Link]() to combine DataFrames on common columns. The pandas documentation
describes it as a “database-style join” 15 . For example, suppose df_customers has customer info and
df_orders has orders with a customer ID:

13
 merged = [Link](df_customers, df_orders, how='inner', on='customer_id')

• how='inner' yields rows where customer_id exists in both tables. Other options: 'left' ,
'right' , 'outer' .
• on='customer_id' specifies the key column to join.
• This creates a single DataFrame merged with columns from both sources aligned by customer.

If joining on index, you can set left_index=True or use [Link]() . Concatenation ( [Link] ) is
another method for stacking data.

Merging is used to enrich datasets. For instance, merge a demographic table with purchase data, or
combine survey responses with transaction logs. Always check for unintended duplication or missing
entries after merging (e.g., ensure many-to-many joins behave as expected).

After merging, proceed to clean or engineer features on the unified dataset. This step often reveals
inconsistencies (e.g., mismatched IDs) requiring further cleaning.

Quiz: Data Integration

1. What does [Link](df1, df2, how='outer') do?

2. A. Keeps only rows with keys in both.
3. B. Keeps all rows from df1 and df2, filling NaN where no match. (Correct)
4. C. Keeps only rows from df1.

5. D. Removes duplicates in df1.

6. If two DataFrames have no columns in common, which merge key could you use?

7. A. on=None with how='cross' for a Cartesian product. (Correct)

8. B. on=None is not allowed.
9. C. Must drop one DataFrame.
10. D. Merge on index only.

References

Pandas DataFrame merge function 15 .

Handling Imbalanced Data (e.g., SMOTE)

In classification tasks, one class may far outnumber others (e.g., 95% negative, 5% positive). This imbalance
can cause models to be biased toward the majority class. For example, predicting that all customers did not
churn yields 95% accuracy if 95% are non-churners, which is misleading. We need techniques to handle
imbalance so the model pays attention to minority classes (e.g., fraud cases, churners).

14
One popular method is SMOTE (Synthetic Minority Over-sampling Technique). SMOTE generates
synthetic samples of the minority class to balance the dataset. According to the imbalanced-learn library
documentation, SMOTE “performs over-sampling using the Synthetic Minority Over-sampling
Technique” 16 . It works by taking each minority class sample and introducing synthetic examples along the
line segments joining that sample and its nearest minority neighbors. In other words, it interpolates new
points between existing minority points 17 . In practice:

from imblearn.over_sampling import SMOTE

sm = SMOTE(random_state=42)
X_resampled, y_resampled = sm.fit_resample(X_train, y_train)

• After fit_resample , X_resampled contains additional synthetic minority rows, making classes
more balanced.
• Note: SMOTE should be applied only to training data after splitting, to avoid data leakage.

SMOTE (and variants like SMOTENC for categorical features) are widely used. Be aware that over-sampling
can risk overfitting minority examples, so sometimes undersampling or combined methods (SMOTE +
Tomek links) are used. The key is balancing without losing information or introducing bias.

After resampling, proceed to train classifiers on the balanced data. You should evaluate performance using
metrics sensitive to imbalance (recall, precision, F1, ROC AUC) rather than accuracy alone.

Quiz: Imbalanced Data

1. What problem does SMOTE address?

2. A. Missing data.
3. B. Class imbalance by oversampling minority. (Correct)
4. C. Scaling numeric features.

5. D. Dimensionality reduction.

6. SMOTE creates new samples by:

7. A. Copying minority records.

8. B. Interpolating between minority class examples. (Correct)
9. C. Deleting majority class records.

10. D. Clustering majority class data.

11. Why must you avoid applying SMOTE before train-test split?

12. A. SMOTE is slow on large datasets.

13. B. It would create synthetic data in the test set, causing data leakage. (Correct)
14. C. It can only work on balanced data.
15. D. SMOTE does not work on text data.

15
References

SMOTE definition and purpose 16 17 .

Train-Test Splitting and Data Leakage Prevention

After preprocessing and feature engineering, you build models. Always split your data into training and
testing sets to evaluate performance on unseen data. The train_test_split function in scikit-learn
quickly partitions datasets. According to the docs: “Split arrays or matrices into random train and test
subsets.” 18 . For example:

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, random_state=0)

• test_size=0.2 reserves 20% of data for testing.

• random_state ensures reproducibility.
• By default, data is shuffled before splitting. You can stratify splits for classification: stratify=y
keeps the same class proportions in train and test.

Preventing data leakage: Data leakage occurs when information from the test set influences the training
model, leading to overly optimistic evaluations. A common mistake is performing preprocessing (like scaling
or imputation) on the entire dataset before splitting. Instead, always split first, then fit transformers on
X_train only. For example:

from [Link] import MinMaxScaler

scaler = MinMaxScaler()
X_train_scaled = scaler.fit_transform(X_train) # Learn scaling on training
data
X_test_scaled = [Link](X_test) # Apply same scaling to test

If you scaled before splitting, information about the test data (e.g., the true range of values) would leak into
the train scaler. MachineLearningMastery emphasizes correct workflows: perform data preparation within
cross-validation or after splitting to avoid leakage 19 . Their example shows using pipelines and cross-
validation to ensure no leakage, which slightly improved model performance when done properly 20 .

Always keep the test set completely separate until final evaluation. Use cross-validation on the training set
for model selection, but never peek at test labels during training or preprocessing.

Quiz: Train/Test and Leakage

1. What does train_test_split return?

2. A. A trained model and its accuracy.

16
 3. B. Separate train and test subsets of features and target. (Correct)
4. C. The confusion matrix.

5. D. A merged DataFrame.

6. Why is data leakage dangerous?

7. A. It slows down training.

8. B. It causes models to underfit.
9. C. It makes the test evaluation too optimistic. (Correct)

10. D. It only affects unsupervised learning.

11. Which practice can cause data leakage?

12. A. Splitting the data after scaling with the entire dataset. (Correct)
13. B. Stratifying splits.
14. C. Using cross-validation on training data only.
15. D. Imputing missing values on train set only.

References

train_test_split usage 18 ; data leakage avoidance 19 .

Evaluation Metrics and Model Assessment

To judge model performance, use appropriate metrics. Classification metrics include:

• Accuracy: (TP+TN)/Total. Useful when classes are balanced.

• Precision: TP/(TP+FP) 21 . Intuitively, the proportion of predicted positives that are true positives 21 .
High precision means few false alarms.
• Recall (Sensitivity): TP/(TP+FN) 22 . Proportion of actual positives that are correctly identified 22 . High
recall means few missed positives.
• F1 Score: Harmonic mean of precision and recall. Balances both. Useful when classes are
imbalanced.
• Confusion Matrix: Table of predicted vs actual counts (TP, FP, FN, TN). Helps compute the above
metrics.

For multi-class classification, metrics can be averaged (‘macro’, ‘micro’, ‘weighted’). Scikit-learn’s
classification_report gives precision, recall, F1 for each class.

ROC Curve and AUC: For binary classifiers, plot the Receiver Operating Characteristic (ROC) curve. The ROC
curve is a plot of the True Positive Rate (recall) vs False Positive Rate at various thresholds 23 . The Area
Under the Curve (AUC) summarizes it: “the probability that the model will rank a randomly chosen positive
instance higher than a negative one” 24 . AUC ranges from 0.5 (random) to 1.0 (perfect). A higher AUC
indicates better overall discrimination between classes.

17
Example: After predicting probabilities y_proba with a model:

from [Link] import roc_auc_score, roc_curve

probs = model.predict_proba(X_test)[:,1]
fpr, tpr, thresholds = roc_curve(y_test, probs)
auc = roc_auc_score(y_test, probs)
print("AUC:", auc)

• roc_curve gives FPR and TPR arrays to plot the ROC.

• roc_auc_score returns the AUC value.

Regression metrics: For predicting numeric values, common metrics include: - Mean Squared Error (MSE):
Average of squared differences (lower is better). sklearn’s mean_squared_error describes it as “Mean
squared error regression loss.” 25 .
- Root MSE (RMSE): Square root of MSE, in the same units as the target.
- Mean Absolute Error (MAE): Average absolute difference (less sensitive to outliers than MSE).
- R² (Coefficient of Determination): Proportion of variance explained by the model (ranges 0–1; higher is
better).

Always choose metrics that align with your problem. For imbalanced classification, consider precision-
recall or ROC AUC over accuracy. For regression, compare MSE/RMSE or MAE, and visualize residuals to
detect bias.

Quiz: Evaluation Metrics

1. Precision measures:
2. A. TP/(TP+FN)
3. B. TP/(TP+FP) (Correct)
4. C. (TP+TN)/Total

5. D. TN/(TN+FP)

6. A ROC AUC of 0.5 indicates:

7. A. Perfect classification.
8. B. Random guessing. (Correct)
9. C. Cannot tell.

10. D. Overfitting.

11. Which metric is not suitable for regression?

12. A. Mean Squared Error.

13. B. Precision. (Correct)
14. C. R² Score.
15. D. Mean Absolute Error.

18
References

Precision and recall definitions 21 22 ; ROC AUC interpretation 24 ; regression loss function descriptions
25 .

This comprehensive study guide covers the data science workflow from Python basics through data cleaning,
visualization, feature engineering, analysis, and model evaluation. It weaves together concepts, examples, and
case studies to build a solid foundation for beginner to intermediate learners in Python-based data science.

19
 1 Python Overview
[Link]

2 Getting Started With Python IDLE Quiz – Real Python

[Link]

3 [Link]
[Link]

4 Top 8 Python Libraries for Data Visualization - GeeksforGeeks

[Link]

5 Matplotlib — Visualization with Python

[Link]

6 Pandas DataFrame dropna() Method

[Link]

7 Working with missing data — pandas 2.3.1 documentation

[Link]

8 [Link].drop_duplicates — pandas 2.3.1 documentation

[Link]

9 OneHotEncoder — scikit-learn 1.7.1 documentation

[Link]

10 LabelEncoder — scikit-learn 1.7.1 documentation

[Link]

11 12 What is Exploratory Data Analysis? | IBM

[Link]

13 [Link] — pandas 2.3.1 documentation

[Link]

14 Analysis of Titanic Survival Data

[Link]

15 [Link] — pandas 2.3.1 documentation

[Link]

16 SMOTE — Version 0.13.0

[Link]

17 ML | Handling Imbalanced Data with SMOTE and Near Miss Algorithm in Python - GeeksforGeeks
[Link]

18 train_test_split — scikit-learn 1.7.1 documentation

[Link]

19 20 How to Avoid Data Leakage When Performing Data Preparation - [Link]

[Link]

21 precision_score — scikit-learn 1.7.1 documentation

[Link]

20
22 recall_score — scikit-learn 1.7.1 documentation
[Link]

23 24 Classification: ROC and AUC | Machine Learning | Google for Developers

[Link]

25 mean_squared_error — scikit-learn 1.7.1 documentation

[Link]

21