00:00:00 - 00:00:23

Data Loading and Cleaning Overview

All right, so let's get started. So given the task you're going to perform
these specific following steps for successfully complete this assignment.
So you are going to load the data. You will proceed to for the data
cleaning, you will perform a data, you know, exploratory analysis, OK,
wherein you will be applying bivariate and multivariate analysis.

00:00:24 - 00:00:38

Exploratory Data Analysis Steps

OK. Then there is a, you need to create a visualization to support the

analysis, derive multiple insights and try to come up with the conclusions.
So we have already discussed about the data understanding here.

00:00:39 - 00:01:11

Data Set Description and Structure

Now let's move forward. So first thing, we have already gone through the
entire data overview. We have seen the data dictionary. The data set
contain obviously 12 files, right? And each represent one month taxi trip.
OK, all the files include details such as trip distance, fare, passenger count
and pick up and drop off times and locations. OK, so let me show you here
because we have written all the detailed information pertaining to the
data.

00:01:14 - 00:01:21

Importing Libraries and Recommended Versions

Now first thing what you need to do right? The first thing obviously we
need to import the required libraries right?

00:01:21 - 00:02:01

Handling Large Data Files and Sampling Techniques

Now let me give you there are some recommended versions which has
been mentioned it here OK. So these are the recommended versions
which we are going to use downloading all this recommended version.
Now you have to take a look at one of the files OK. You will find that these
have around 3 million record for a single month. So try to upload a single
file here, OK And it will have a multiple records which are available, OK,
figure it out how many number of rows are there? How many do you think
how much large number of rows is computationally feasible for us to to
calculate?

00:02:01 - 00:02:26
Sampling Strategy and Data Combination

Like if there is no such memory error or other information should not

appear all this information try to you try to reflect here on your Jupiter
notebook. Then obviously you can perform a sampling the data. Right now
the challenge here is given. But given so many record, is it possible to
directly combine all the files, right? It will be too large to handle.

00:02:27 - 00:03:12

Nested Loop Sampling Methodology

So to handle it, we will take a different approach. We will take a sample of

the portion of the data and work on it, right? So let's say you will take a
some kind of fraction say 5% from each and every file and then combine
all these files into a single data file. That is a thought here. OK now
another question arise here is how do we sample it or which values do we
select. So this is the sampling technique which you can apply. As you can
see here we are taking a sample right and you are taking our data and
there is some amount of 5% fraction which you are taking it here and then
in the end you are concatenating all the specific files. That is the process
which you need to apply.

00:03:14 - 00:03:39

Data Sampling Function and Code Implementation

OK, so now, so basically it's OK to select 5% of the record randomly. Why?

Because the task at the hand to find patterns and trends in the data,
right? So even if you select the random sample here, right. So our
assumption here is it will preserve these specific trends, OK. But yes,
again you need to figure it out whether it is available there or not.
00:03:39 - 00:04:43

Data Cleaning and Column Standardization

It's it's your understanding, the idea is to sample 5% of the daily trips,
right? This will help us preserve the daily trends. But we can go a bit more
deeper than this if you want to. OK, OK, So what if I say if we take a
sample by the trip, pick over pick up hours? OK, so this will also help us to
uncover even the hourly trends of the trip, uncover it may uncover more
insights, right? However, given this specific part, OK, if we just sample by
the hours, then again the daily patterns will get overlooked, OK? Because
sampling 5% of the pickup in each hour may lead to some days being
missed out and other days having more data than others. So, this does
not give out any accurate representation of the data. OK to solve this we
will make a some kind of nested loop.

00:04:44 - 00:05:26

Handling Missing Values and Outlier Management

OK, so this is what we are trying to do. We are trying to create a some
kind of nested loop here. OK. So first we will iterate through the days to
capture the daily records. OK, that is our thought. And then while iterating
through each day's record, we will loop into the trips in each hour of that
day. Now from those trips we can select 5% at random. That is a thought
here. OK, now then what we will do, we will merge the sample data of all
the 24 hours to make up the sample data of a one day and merge the
sample data of all 365 days to make up the yearly sample data.

00:05:27 - 00:06:10

Exploratory Data Analysis Techniques

Right now we are already showing the sampling function technique which
is available here. OK, so this is the sampling technique which is already I
just showed you and that's how you will sample the data. Probably you
can load your data by using your Google Drive, OK, which are available in
your data and then in your Google Drive and then you can try to load it
and do the OK. So this is the sample function which we which has been
provided here, OK? So obviously here we are going to take a sampling
section in random state, OK? So you can keep the random state any
number you like.

00:06:11 - 00:07:03

Summary Statistics and Data Distribution

Now this is as I said earlier, this is the code sampling. OK how here some
code is pre filled. So we are going to fill some of the codes for you. As you
can see OK this will create a list of almost whatever the number of files is
available. So in our case there are 12 files are available. Now you can
easily loop through these 12 files right? But make a note the the code loop
for all the days and all a nested loop inside it. All the hours in that day is
to be filled by you only. OK then in the end you going you will concatenate
the and the data frames of hours and days to make up final data and then
save the and then save this as a per quit file or CSA file to your device. OK
so this is here.

00:07:03 - 00:08:27

Temporal and Financial Analysis

That's how you can do it. OK, now let's talk about data cleaning. OK, so
data cleaning, as we already know, data cleaning is one of the most
important steps of EDA. It is absolutely crucial for extracting any useful
insights. OK, now what you need to do, You need to figure it out all the
irregularities in your column. How you can do it. OK, a standardized
column name so that we can we can ensure all the consistencies, convert
data type if necessary, OK fix the index and drop all unnecessary columns
whichever is not required, for example, OK OK. For example, take a look
at the column airport fee. There are two with the same name, OK, It may
be possible that name for the column registering the airport fee might
have been changed sometime along along with the year. So you will have
to combine these columns into one. So these are kind of, you know, fixing
steps and assumption that you might need to take to improve the data
quality. Again, while fixing the negative monitoring values, you can look at
rate code ID.

00:08:28 - 00:09:13

Geographical Analysis and Visualization

OK, there is a one column called rate code ID, right which tells the rate
card information for the trip. It will help filter out the invalid or cancelled
trip and so on. OK, that's how you can you can perform all your fixing
column strategies here. So far so good. Now proceeding further, let's
figure it out about the handling missing values. OK, so try to detect and
address missing entries in a critical columns. For example, let's say trip
distances or fares right Connect invalid entries such as negative trip
distance or 0 passenger counts. Most probably you will find some of the
missing values in in few of the columns.

00:09:13 - 00:10:34

Operational Efficiency and Route Optimization

Likewise rate code ID or congestion surcharges right now, but because

these are since we have taken a random sampling here, there may be a
different set of missing values might be available. So, so you need to
handle those information accordingly. OK, now the next thing is we are
going to perform a outlier handling. OK, now outlier management you
need to identify in the numerical columns like distance, fare or passenger
counts what all outliers are available right. Then decide and treatment
strategy, OK, any kind of capping or transformation or any exclusion is
your is your choice to go with figure it out here you will see there are fair
amounts which is going beyond $300.00 for a zero trip distance. So in, in,
in some places, the trip distance are more than 250 miles. So without
proper rate codes or fair amount, it make it makes sense to drop these
specific records. OK, so you can also create a trip duration the column it's
it all depends upon you. Now the next thing is we are going to perform an
exploratory data analysis.

00:10:34 - 00:11:22

Demand Trends and Resource Allocation

OK, so let's do some kind of basic EDA, right? And we already discussed it
about the to start with the very initial exploit data analysis, then move
further. Now having cleaned our data, let's start with complete EDA
process. What you need needs to do here. OK. Now the first thing is
obviously you need to perform a summary statistics, OK, calculate
measuring like mean, median, minimum value, maximum value values or
any kind of key columns which are available figure it or your figure it out
your data distribution. We try to create a histogram plot box plot, scatter
plot and try to understand the data spreads, right?

00:11:22 - 00:12:23

Pricing Strategy and Customer Insights

What is the variance, What is the standard deviation which are available?
All this information, our basic information, right? Then you will perform
your temporal analysis here. OK. Now you what you need to do, you need
to examine the patterns in a trip across the time, OK, Based on that, so it
could be like peak hours or busiest month, all this information you need to
figure it out, OK? Try to analyze the passenger trains during different
times of day or season, OK. Then obviously try to perform some kind of
financial analysis here, OK? Now by using the financial analysis, try to
investigate relationship between fair amount, trip length and tips, OK
Identify trends in total revenue generation over the time, OK? That is what
you need to perform it here in this specific section, OK.

00:12:23 - 00:13:37

Conclusion and Actionable Insights

Then obviously later on once you do this, you need to go ahead and
proceed for the geographical analysis. OK, now use XE zone shape files as
mentioned here. OK, map pickup and drop off zones to identify all the
hottest spots. Visualize trip density across neighborhood and draws. OK.
Then obviously once you do this, you need to perform your detailed EDA,
OK? Now, so there are multiple information which we have mentioned it
here once you do this, right? So now there is another important factor
here is you need to perform operational efficiency, OK? So obviously the
first step is route optimization, OK analyze route with frequent delays or
treatment delays or slower travel speeds, OK Identify all the zones with
high congestion at specific times, OK, Try to figure it out your all
demands.