Academic Year:2025-26 SAP ID:60003240302
DEPARTMENT OF INFORMATION TECHNOLOGY
COURSE CODE: DJS23IPC253L DATE:
COURSE NAME: Programing Laboratory CLASS: S.Y. [Link]
EXPERIMENT NO. 7
CO/LO: CO2.
AIM / OBJECTIVE: Write a Python program to implement data analysis using pandas.
DESCRIPTION OF EXPERIMENT:
Pandas is a software library written for the Python programming language for data manipulation
and analysis. It provides fast, flexible, and expressive data structures designed to make working
with “relational” or “labeled” data easy and intuitive. It aims to be the fundamental high-level
building block for doing practical, real-world data analysis in Python.
Pandas in Python is built on top of NumPy and is intended to integrate well within a scientific
computing environment with many other 3rd party libraries.
Pandas Data Structure
The two primary data structures of pandas:
• Series (1-dimensional)
• DataFrame (2-dimensional)
handle the vast majority of typical use cases in finance, statistics, social science, and many areas
of engineering.
Pandas is suitable for:
• Easy handling of missing data (represented as NaN) in floating point as well as non-floating
point data
• Size mutability
• Intelligent label-based slicing, fancy indexing, and subsetting of large data sets
• Flexible reshaping and pivoting of data sets
• Robust IO tools for loading data from flat files (CSV and delimited), Excel files, databases,
and saving/loading data from the ultrafast HDF5 format
• Time series data
Academic Year:2025-26 SAP ID:60003240302
DEPARTMENT OF INFORMATION TECHNOLOGY
• Tabular data with heterogeneously-typed columns, as in an SQL table or Excel spreadsheet
• Arbitrary matrix data (homogeneously typed or heterogeneous) with row and column labels
• Any other form of observational/statistical data sets. The data need not be labeled at all to be
placed into a pandas data structure
1.) Install the package
Before starting with the demo, you will have to install the package. Run the below command to
install the Pandas package.
!pip install pandas
2.) Importing the Pandas package
Step 2: In your Jupyter notebook, run the following command to install the required package.
import pandas as pd
Short Trick: The common shortcut of Pandas is pd so instead of writing “pandas.” you can write
“pd.”, but note that there is a dot after “pd” which is used to call a method from Pandas library.
3.) Import the dataset with read_csv
Now that the package is successfully installed, we will import the dataset.
Step 3: To read a dataset, we are going to use read_csv.
here df stands for dataframe (pandas dataframe)
df = pd.read_csv("//YOUR FILE Path")
Step 4: To view the contents of the dataset we are going to use the [Link]().
print([Link])
print([Link])
print([Link]())
Academic Year:2025-26 SAP ID:60003240302
DEPARTMENT OF INFORMATION TECHNOLOGY
Note: if the () is left blank then the first 5 elements of the table will be displayed. You want a
specific number of entries, then you can give the command as [Link](20) to print the first 20
rows.
Step 5: To view the last rows of the table, run the following command [Link]()
Step 6: Now if you want to read some specific columns only from the dataset, you can use
the usecols argument to specify the column names that we want to work with. Let’s work with
just PassengerId, Survived, Pclass columns.
df = pd.read_csv("\\Your File Path\\[Link]", usecols= ["PassengerId", "Survived", "Pclass"])
[Link]()
Step 7: Run the describe() command to get a summary of numeric values in your dataset.
Academic Year:2025-26 SAP ID:60003240302
DEPARTMENT OF INFORMATION TECHNOLOGY
In order to see statistics on non-numerical features, one has to explicitly indicate data types of
interest in the include parameter.
[Link](include=["object", "bool"])
4.) Sort Columns based on specific criteria
In this section, we will sort columns based on numeric data, string, etc.
Step 8: Run the below command to view the first 8 lowest ticket prices (fare column).
df.sort_values("Fare").head(8)
Step 9: To view the highest-paying passengers, run the below command.
df.sort_values("Fare", ascending = False).head(8)
5.) Count the occurrences of variables
Step 10: Using .value_counts() to count the occurences of each variable in a column.
6.) Data Filtering
Academic Year:2025-26 SAP ID:60003240302
DEPARTMENT OF INFORMATION TECHNOLOGY
Step 11: Run the below command to display those passengers who are female and their fare is
less than 100.
df_fare_mask = df["Fare"] < 100
df_sex_mask = df["Sex"] == "female"
df[df_fare_mask & df_sex_mask]
7.) Null values (NaN)
One of the most common problems in data science is missing values. To detect them, there is a
beautiful method which is called .isnull(). With this method, we can get a boolean series (True or
False).
Step 12: Run the below command to show the passengers whose cabin is unknown (NaN).
null_mask = df["Cabin"].isnull()
df[null_mask]
Academic Year:2025-26 SAP ID:60003240302
DEPARTMENT OF INFORMATION TECHNOLOGY
Exploratory Data Analysis
Sorting
A DataFrame can be sorted by the value of one of the variables (i.e columns). For example, we
can sort by Total day charge (use ascending=False to sort in descending order):
We can also sort by multiple columns:
Indexing and retrieving data
A DataFrame can be indexed in a few different ways.
To get a single column, you can use a DataFrame['Name'] construction. Let's use this to answer a
question about that column alone:
what is the proportion of churned users in our dataframe?
Academic Year:2025-26 SAP ID:60003240302
DEPARTMENT OF INFORMATION TECHNOLOGY
Boolean indexing with one column is also very convenient. The syntax is df[P(df['Name'])],
where P is some logical condition that is checked for each element of the Name column. The
result of such indexing is the DataFrame consisting only of rows that satisfy the P condition on
the Name column.
Let's use it to answer the question:
What are average values of numerical features for churned users?
How much time (on average) do churned users spend on the phone during daytime?
What is the maximum length of international calls among loyal users (Churn == 0) who do
not have an international plan?
Academic Year:2025-26 SAP ID:60003240302
DEPARTMENT OF INFORMATION TECHNOLOGY
DataFrames can be indexed by column name (label) or row name (index) or by the serial
number of a row. The loc method is used for indexing by name, while iloc() is used
for indexing by number.
In the first case below, we say "give us the values of the rows with index from 0 to 5
(inclusive) and columns labeled from State to Area code (inclusive)". In the second case, we
say "give us the values of the first five rows in the first three columns" (as in a typical
Python slice: the maximal value is not included).
If we need the first or the last line of the data frame, we can use the df[:1] or df[-1:] construct:
Applying Functions to Cells, Columns and Rows
To apply functions to each column, use apply():
For example, if we need to select all states starting with W, we can do it like this:
Academic Year:2025-26 SAP ID:60003240302
DEPARTMENT OF INFORMATION TECHNOLOGY
The map method can be used to replace values in a column by passing a dictionary of the
form {old_value: new_value} as its argument:
The same thing can be done with the replace method:
Academic Year:2025-26 SAP ID:60003240302
DEPARTMENT OF INFORMATION TECHNOLOGY
Grouping
In general, grouping data in Pandas works as follows:
[Link](by=grouping_columns)[columns_to_show].function()
1. First, the groupby method divides the grouping_columns by their values. They become a
new index in the resulting dataframe.
2. Then, columns of interest are selected (columns_to_show). If columns_to_show is not
included, all non groupby clauses will be included.
3. Finally, one or several functions are applied to the obtained groups per selected columns.
Here is an example where we group the data according to the values of the Churn variable and
display statistics of three columns in each group:
Summary tables
Suppose we want to see how the observations in our sample are distributed in the context of two
variables - Churn and International plan. To do so, we can build a contingency table using
the crosstab method:
Academic Year:2025-26 SAP ID:60003240302
DEPARTMENT OF INFORMATION TECHNOLOGY
pivot tables are implemented in Pandas: the pivot_table method takes the following parameters:
• values – a list of variables to calculate statistics for,
• index – a list of variables to group data by,
• aggfunc – what statistics we need to calculate for groups, ex. sum, mean, maximum,
minimum or something else.
Let's take a look at the average number of day, evening, and night calls by area code:
Academic Year:2025-26 SAP ID:60003240302
DEPARTMENT OF INFORMATION TECHNOLOGY
DataFrame transformations
Like many other things in Pandas, adding columns to a DataFrame is doable in many ways.
For example, if we want to calculate the total number of calls for all users, let's create
the total_calls Series and paste it into the DataFrame:
Academic Year:2025-26 SAP ID:60003240302
DEPARTMENT OF INFORMATION TECHNOLOGY
To delete columns or rows, use the drop method, passing the required indexes and
the axis parameter (1 if you delete columns, and nothing or 0 if you delete rows).
The inplace argument tells whether to change the original DataFrame. With inplace=False,
the drop method doesn't change the existing DataFrame and returns a new one with dropped
rows or columns. With inplace=True, it alters the DataFrame.
Academic Year:2025-26 SAP ID:60003240302
DEPARTMENT OF INFORMATION TECHNOLOGY
First attempt at predicting telecom churn
Let's see how churn rate is related to the International plan feature. We'll do this using
a crosstab contingency table and also through visual analysis with Seaborn.
Academic Year:2025-26 SAP ID:60003240302
DEPARTMENT OF INFORMATION TECHNOLOGY
Feature Engineering in Machine Learning
Feature engineering is the process of transforming raw data into meaningful features that
improve the performance of machine learning models. It involves creating, selecting, and
modifying features to enhance predictive power.
1. Handling Missing Values
Missing data can reduce model accuracy and introduce bias. Common strategies include:
• Mean/Median Imputation (for numerical data)
• Mode Imputation (for categorical data)
• Forward/Backward Fill
• Dropping Missing Values (only if minimal data is lost)
2. Encoding Categorical Variables
Machine learning models work best with numerical data. Encoding techniques include:
• Label Encoding (for ordinal data)
• One-Hot Encoding (for nominal data)
Academic Year:2025-26 SAP ID:60003240302
DEPARTMENT OF INFORMATION TECHNOLOGY
3. Feature Scaling
Feature scaling ensures all variables have similar ranges, preventing models from being biased
toward larger values. Common methods:
• Standardization (Z-score scaling)
• Min-Max Scaling (Rescales values between 0 and 1)
4. Feature Selection
Selecting the most important features improves model efficiency and reduces overfitting.
Techniques include:
• Statistical Tests (e.g., ANOVA, chi-square)
• Recursive Feature Elimination (RFE)
• Tree-Based Feature Selection
5. Feature Transformation
Transforming features can make data more suitable for modeling:
• Log Transformation (for skewed distributions)
• Polynomial Features (for capturing non-linearity)
• Binning (converting continuous features into categorical bins)
6. Feature Creation
Creating new features from existing ones can improve model performance:
• Interaction Features (multiplication of two variables)
• Aggregations (sum, mean, max)
• Date-Time Features (extracting day, month, year)
Data balancing:
Data balancing is the process of handling class imbalances in datasets, which occurs when one
class has significantly more samples than another. An imbalanced dataset can lead to biased
models that favor the majority class, resulting in poor generalization.
1. Identifying Class Imbalance
Before applying balancing techniques, check the class distribution.
2. Techniques for Data Balancing
A. Undersampling (Reduce Majority Class)
Removes samples from the majority class to create a balanced dataset.
Academic Year:2025-26 SAP ID:60003240302
DEPARTMENT OF INFORMATION TECHNOLOGY
Pros: Reduces training time.
Cons: May lose important data.
B. Oversampling (Increase Minority Class)
Duplicates or synthetically generates new samples for the minority class.
Pros: Preserves valuable information.
Cons: Risk of overfitting.
1. Random Oversampling
Duplicates random samples from the minority class.
QUESTIONS:
1. Examine the dataset for missing or null values. If any are found, apply appropriate
techniques to handle them.
2. Retrieve the details of students who have scored more than 85 marks in Mathematics and
have attendance greater than 90%.
Academic Year:2025-26 SAP ID:60003240302
DEPARTMENT OF INFORMATION TECHNOLOGY
3. Create two new columns in the dataset:
(a) Total Marks (sum of Maths, Science, and English)
(b) Average Marks (mean of the three subjects)
4. Rank all students based on their Average Marks in descending order. Assign ranks
accordingly.
Academic Year:2025-26 SAP ID:60003240302
DEPARTMENT OF INFORMATION TECHNOLOGY
5. Calculate the average marks in Maths, Science, and English for each gender group.
6. Identify students who study for more than 4 hours per day and have internet usage of less
than 2 hours.
7. Display the top 10 students based on their Total Marks.
Academic Year:2025-26 SAP ID:60003240302
DEPARTMENT OF INFORMATION TECHNOLOGY
8. Analyze the relationship between:
(a) Study Hours and academic performance
(b) Internet Usage and academic performance
QUESTIONS FOR OBSERVATION SHEET:
1. Why is Pandas preferred for data analysis?
2. Difference between Series and DataFrame?
3. How do you handle missing values in Pandas?
4. Difference between dropna() and fillna()?
5. How would you merge two datasets with common columns? Give example
•
OBSERVATIONS / DISCUSSION OF RESULT:
This section should interpret the outcome of the experiment. The observations can be visually
represented using images, tables, graphs, etc. This section should answer the question "What do
Academic Year:2025-26 SAP ID:60003240302
DEPARTMENT OF INFORMATION TECHNOLOGY
the result tell us?" Compare and interpret your results with expected behavior. Explain
unexpected behavior, if any.
CONCLUSION:
Base all conclusions on your actual results; describe the meaning of the experiment and the
implications of your results.
REFERENCES:
Website References
1. [Link]
2. [Link]
3. [Link]
: