DataFrame Filtering and Sorting

[00:00:00.00] [MUSIC PLAYING]

[00:00:08.36] RYAN AHMED: Hello, everyone, and welcome to this lesson on dataframe
filtering and sorting. Data sorting is the process of organizing data into some meaningful order to
make it easier to understand and examine. You can sort data in ascending or descending orders.
For example, data analysts might need to sort client data based on the size of their investment
portfolio in an ascending order. Data filtering is the process of filtering Pandas dataframes based
on a specific condition.

[00:00:40.90] For example, a data analyst who works at an investment firm might need to filter
out loyal clients who have been with the firm for, let's say, 15 years or more. Many of you might
be wondering, well, I can do this in Excel. Why do I need Pandas for this? Well, Pandas offers
key advantages over other data manipulation tools, since it can handle the large amount of data,
allows for process automation, and offers highly efficient data manipulation capabilities.

[00:01:10.53] Here are the key learning objectives of this lesson. Sort Pandas dataframes by the
values of one or more columns. Use the ascending parameter to choose the sort order, which
could be ascending or descending. Use in place parameters enforce Pandas dataframe changes to
take place in memory. Filter out Pandas dataframes using a specific condition. So let's head over
to our Jupyter notebook and get started.

[00:01:40.52] [MUSIC PLAYING]

[00:01:48.11] All right. So let's go ahead and cover dataframe filtering and sorting. So right now
I'm in the dataframe filtering and sorting Jupyter notebook. So first let's cover dataframe
filtering. Dataframe filtering is the process of filtering Pandas dataframes based on a specific
condition. So here I have my Pandas dataframe that has multiple clients, their first name, their
last name. I also have their age, portfolio size, years with investment firm. I also have the risk
tolerance and their goal as well.

[00:02:24.92] What I wanted to do is I might want to filter out specific rows from my Pandas
dataframe based on a specific condition. For example, let's assume that I wanted to select only
investors who have years with investment firm of at least 15 years. I'm only interested in filtering
out loyal customers with the investment firm. To do that, this is simply the syntax that I'm going
to follow. And I'm going to show you all these details in code.

[00:02:56.60] I'm going to grab my investor_df. This is simply my input Pandas dataframe. You
open a square bracket. And then you write the specific condition in here. For example, I wanted
to say, OK, go to my years with investment firm column. If you find that years with investment
firm column greater than or equals 15, this is simply the condition that I have here, once that
condition is true, if that condition is satisfied, filter out my Pandas dataframe and place the new
rows in my loyal_df. And that is going to be my output dataframe.
[00:03:36.08] So what I got here is based on that specific condition, I was able to filter out the
entire Pandas dataframe and I ended up with this new one here. So only these rows have been
able to satisfy my condition. And you would notice that the years with investment firm. Here I
have 30 years. Basil, for example, have 15 years. And Nancy had 17 years. So that's how we can
do dataframe filtering using Pandas.

[00:04:07.52] Next I wanted to show you dataframe sorting. So dataframe sorting is the process
of organizing data into some meaningful order to make it easier to understand and examine. So
for example, let's assume that here I have my Pandas dataframe. It's the same one that I had
before. And I might want to sort my Pandas dataframe based on the portfolio size in an
ascending order.

[00:04:35.96] To do that, all you need to do is to grab the dataframe name investor_df. You just
say .sort_values. This is the sorting method. You specify which column you want to sort the
dataframe by. So here I can say by equals 2. And you write the name of the column. That was
here and my portfolio size. And I wanted to enforce that change to take place in memory, and
that's why I set in place to be equal to true.

[00:05:04.86] So after you apply these sort values method to my investor_df dataframe, you will
get here another dataframe. And you will notice that the portfolio size have been sorted in an
ascending order. So we started with the smallest one, 1,500, and then 41,000. We kept on going
up until we reached the maximum value with Sherif here at the end.

[00:05:29.39] All right, so let me go ahead and walk you through the code. First I wanted to
import Pandas SPD. We have done that already so many times in the past. If you press Shift and
Enter, that is going to run or execute this code set. Next I'm going to use pandas.read_csv. And
I'm going to specify the name of the CSV as file. And that is going to be investors_data. I'm
going to read the data, and then I'm going to put it in investor_df dataframe. Again, we have
done that as well in previous lessons.

[00:06:04.33] Now to the fun part. Now I wanted to grab my Pandas dataframe, and I wanted to
filter out only rows that have been loyal simply with the bank. Only people who have been with
the bank for at least 15 years. So I'm going to say investor_df. If I here apply that condition, that
is going to be investor_df. Years with investment firm greater than or equals to 15. And you run
the send. That is going to return back for me another Pandas dataframe. And that is going to be
loyal_df. And here only containing customers who have been with the firm for at least 15 years.
So here, for example, 30 years, 15, and 17.

[00:06:52.87] What you might want it to do is you might need to reset the indexed for the new
Pandas dataframe. So if you notice here on the left hand side, the index is still the same as the
original Pandas dataframe. So I have 1, 5, and 7. So to reset index, you just grab the name of the
dataframe. So loyal_df. And then you say .resetindex and you enforce the change to take place in
memory. You say in place equals true. Press Shift Enter.

[00:07:21.91] Here we go. You will notice that right now I was able to reset my index. So here I
have 0, 1, 2. And I also have the original index captured in here as well. You can go ahead and
drop the original index if you want. But I just want to show you how we can reset the index here
if you wanted to.

[00:07:41.49] Next I wanted to sort the Pandas dataframe by portfolio size. To do that, I'm going
to grab my investor_df and then I'm going to say .sortvalues. And then you specify the by. The
by is going to be portfolio size. So if you press Shift and Enter. And please note that what I've
done here is I did not set in place to be equals to true. And I just wanted to show you if we set in
place to be equal to true, what's going to happen? And if we just ignore it, if we set it to false by
default, what's going to happen as well?

[00:08:17.89] So what I've done here is I grab my investor_df. I said .sortvalues. I specify the
name of the column that I wanted to sort the values by, and that is going to be portfolio size. And
you will notice that here we started with the smallest portfolio size at 1,500 and we went up, up
until we reached the maximum value here. And of course, I have NAN or Not A Number here.

[00:08:42.19] So what you would notice is we actually sorted my Pandas dataframe just for
display purposes. Meaning if you wanted to go and see what's actually happening right now in
investor_df, if you run this send right now, you will notice something interesting. You will notice
that nothing has changed in memory. So the original investor_df did not actually change in
memory. It was unsorted. Here I have the original raw investor_df. And the reason is because we
did not set in place to be equals to true when we apply the sort values operation.

[00:09:22.22] And now I wanted to show you if we apply in place to be equals to true, what is
going to happen? So I'm going to say investor_df .sortvalues. And the by is going to be by
portfolio size. And right now I'm going to enforce the change to take place in memory by setting
in place to be equal to true. If you press Shift Enter, if you check out what's investor_df, here we
go. Right now we actually sorted our Pandas dataframe in an ascending order according to the
portfolio size from the smallest value to the maximum value.

[00:09:56.21] All right, so that's it. That's all I have for this lesson. I hope you enjoyed it. In the
next lesson, we'll have our practice opportunity. Please stay tuned. I hope you enjoyed this
lesson, and see you in the next one.

[00:10:06.89] [MUSIC PLAYING]