What is Pandas?

Pandas is a Python library used for working with data sets.

It has functions for analyzing, cleaning, exploring, and manipulating data.

The name "Pandas" has a reference to both "Panel Data", and "Python Data
Analysis" and was created by Wes McKinney in 2008.

Why Use Pandas?

Pandas allows us to analyze big data and make conclusions based on
statistical theories.

Pandas can clean messy data sets, and make them readable and relevant.

Relevant data is very important in data science.

Data Science: is a branch of computer science where we study how to

store, use and analyze data for deriving information from it.

What Can Pandas Do?

Pandas gives you answers about the data. Like:

 Is there a correlation between two or more columns?

 What is average value?
 Max value?
 Min value?

Pandas are also able to delete rows that are not relevant, or contains wrong
values, like empty or NULL values. This is called cleaning the data.

What is a Series?

A Pandas Series is like a column in a table.

It is a one-dimensional array holding data of any type.

Example
Create a simple Pandas Series from a list:

import pandas as pd

a = [1, 7, 2]

myvar = [Link](a)

print(myvar)

Output:
0 1
1 7
2 2
dtype: int64

Labels
If nothing else is specified, the values are labeled with their index number.
First value has index 0, second value has index 1 etc.

This label can be used to access a specified value.

Example
Return the first value of the Series:

import pandas as pd

a = [1, 7, 2]

myvar = [Link](a)

print(myvar[0])

Output:
1

Create Labels

With the index argument, you can name your own labels.
Example
Create your own labels:

import pandas as pd

a = [1, 7, 2]

myvar = [Link](a, index = ["x", "y", "z"])

print(myvar)

Output:
x 1
y 7
z 2
dtype: int64

When you have created labels, you can access an item by referring to the
label.

Example
Return the value of "y":

import pandas as pd

a = [1, 7, 2]

myvar = [Link](a, index = ["x", "y", "z"])

print(myvar["y"])

Output:
7

Key/Value Objects as Series

You can also use a key/value object, like a dictionary, when creating a
Series.

Example
Create a simple Pandas Series from a dictionary:
import pandas as pd

calories = {"day1": 420, "day2": 380, "day3": 390}

myvar = [Link](calories)

print(myvar)

Output:
day1 420
day2 380
day3 390
dtype: int64

Note: The keys of the dictionary become the labels.

To select only some of the items in the dictionary, use the index argument
and specify only the items you want to include in the Series.

Example
Create a Series using only data from "day1" and "day2":

import pandas as pd

calories = {"day1": 420, "day2": 380, "day3": 390}

myvar = [Link](calories, index = ["day1", "day2"])

print(myvar)

Output:
day1 420
day2 380
dtype: int64

Pandas DataFrames

What is a DataFrame?

A Pandas DataFrame is a 2 dimensional data structure, like a 2 dimensional

array, or a table with rows and columns.
Example
Create a simple Pandas DataFrame:

import pandas as pd

data = {
"calories": [420, 380, 390],
"duration": [50, 40, 45]
}

#load data into a DataFrame object:

df = [Link](data)

print(df)

Result

calories duration
0 420 50
1 380 40
2 390 45

Locate Row

As you can see from the result above, the DataFrame is like a table with
rows and columns.

Pandas use the loc attribute to return one or more specified row(s)

Example
Return row 0:

import pandas as pd

data = {
"calories": [420, 380, 390],
"duration": [50, 40, 45]
}

#load data into a DataFrame object:

df = [Link](data)

print([Link][0])
Output:
calories 420
duration 50
Name: 0, dtype: int64
Note: This example returns a Pandas Series.

Example
Return row 0 and 1:

import pandas as pd

data = {
"calories": [420, 380, 390],
"duration": [50, 40, 45]
}

#load data into a DataFrame object:

df = [Link](data)

print([Link][[0, 1]])
Output:
calories duration
0 420 50
1 380 40
Note: When using [], the result is a Pandas DataFrame.

Named Indexes

With the index argument, you can name your own indexes.

Example
Add a list of names to give each row a name:

import pandas as pd

data = {
"calories": [420, 380, 390],
"duration": [50, 40, 45]
}

df = [Link](data, index = ["day1", "day2", "day3"])

print(df)
Result

calories duration
day1 420 50
day2 380 40
day3 390 45

Locate Named Indexes

Use the named index in the loc attribute to return the specified row(s).

Example
Return "day2":

import pandas as pd

data = {
"calories": [420, 380, 390],
"duration": [50, 40, 45]
}

df = [Link](data, index = ["day1", "day2", "day3"])

print([Link]["day2"])

Output:
calories 380
duration 40
Name: day2, dtype: int64

Load Files Into a DataFrame

If your data sets are stored in a file, Pandas can load them into a DataFrame.

Example
Load a comma separated file (CSV file) into a DataFrame:

import pandas as pd

df = pd.read_csv('[Link]')

print(df)
Output:
Duration Pulse Maxpulse Calories
0 60 110 130 409.1
1 60 117 145 479.0
2 60 103 135 340.0
3 45 109 175 282.4
4 45 117 148 406.0
.. ... ... ... ...
164 60 105 140 290.8
165 60 110 145 300.4
166 60 115 145 310.2
167 75 120 150 320.4
168 75 125 150 330.4

[169 rows x 4 columns]

Head and tail

1) Select first N Rows from a Dataframe using head() method of Pandas

DataFrame :
Pandas head() method is used to return top n (5 by default) rows of a data
frame or series
Syntax: [Link](n).
Parameters: (optional) n is integer value, number of rows to be returned.
Return: Dataframe with top n rows .
Let’s Create a dataframe
# import pandas library as pd
import pandas as pd

# List of Tuples
students = [('Ankit', 22, 'Up', 'Geu'),
('Ankita', 31, 'Delhi', 'Gehu'),
('Rahul', 16, 'Tokyo', 'Abes'),
('Simran', 41, 'Delhi', 'Gehu'),
('Shaurya', 33, 'Delhi', 'Geu'),
('Harshita', 35, 'Mumbai', 'Bhu' ),
('Swapnil', 35, 'Mp', 'Geu'),
('Priya', 35, 'Uk', 'Geu'),
('Jeet', 35, 'Guj', 'Gehu'),
('Ananya', 35, 'Up', 'Bhu')
]

# Create a DataFrame object from

# list of tuples along with columns
# and indices.
details = [Link](students, columns =['Name', 'Age',
'Place', 'College'],
 index =['a', 'b', 'c', 'd', 'e',
'f', 'g', 'i', 'j', 'k'])

Details

Output:

Example 1:
# Show first 5 rows of the details dataframe
# from top
[Link]()

Output:

Example 2:
# display top 3 rows of the dataframe
[Link](3)

Output:
Example 3:
# display top 2 rows of the specific columns
details[['Name', 'Age']].head(2)

Output:

2) Select last N Rows from a Dataframe using tail() method of Pandas

DataFrame :
Pandas tail() method is used to return bottom n (5 by default) rows of a
data frame or series.
Syntax: [Link](n)
Parameters: (optional) n is integer value, number of rows to be returned.
Return: Dataframe with bottom n rows .
Example 1:
# Show bottom 5 rows of the dataframe
[Link]()

Output:

Example 2:
# Show bottom 3 rows of the dataframe
[Link](3)

Output:
Example 3:
# Show bottom 2 rows of the specific
# columns from dataframe
details[['Name', 'Age']].tail(2)

Output:

DataFrame - Binary operator functions

In mathematics a binary operator or a dyadic operator is a function that combines two values to
produce a new value. The binary operator function could perform addition, subtraction and so on
to return the new value. [Link] has several binary operator functions defined for
combining two DataFrames. The binary operator functions return a new DataFrame as a result of
combining the two DataFrames.

Adding Of Two Data Frames

 The Python library pandas, offers several methods to handle two-

dimensional data through the class DataFrame.
 Two DataFrames can be added by using the add() method of
pandas DataFrame class.
 Calling add() method is similar to calling the operator +. However,
the add() method can be passed a fill value, which will be used
for NaN values in the DataFrame.

 Example:
import pandas as pd

dataSet1 = [(10, 20, 30),

(40, 50, 60),
 (70, 80, 90)];

dataFrame1 = [Link](data=dataSet1)

dataSet2 = [(5, 15, 25),

(35, 45, 55),
(65, 75, 85)];
dataFrame2 = [Link](data=dataSet2)

print("DataFrame1:");
print(dataFrame1);

print("DataFrame2:");
print(dataFrame2);

result = [Link](dataFrame2);
print("Result of adding two pandas dataframes:");
print(result)


 Output:


DataFrame1:
0 1 2
0 10 20 30
1 40 50 60
2 70 80 90
DataFrame2:
0 1 2
0 5 15 25
1 35 45 55
2 65 75 85
Result of adding two pandas dataframes:
0 1 2
0 15 35 55
1 75 95 115
2 135 155 175

Subtracting A Pandas DataFrame From Another DataFrame

 Python pandas library provides multitude of functions to work on

two dimensioanl Data through the DataFrame class.
 The sub() method of pandas DataFrame subtracts the elements of
one DataFrame from the elements of another DataFrame.
 Invoking sub() method on a DataFrame object is equivalent to
calling the binary subtraction operator(-).
 The sub() method supports passing a parameter for missing
values([Link], None).

 Example:
import pandas as pd

# Create Data
data1 = [(2, 4, 6, 8),
(1, 3, 5, 7),
(5, 0, 0, 9)];

data2 = [(1, 1, 0 , 1),

(1, 0, 1 , 1),
(0, 1, 1 , 0)];

# Construct DataFrame1
dataFrame1 = [Link](data=data1);
print("DataFrame1:");
print(dataFrame1);
# Construct DataFrame2
dataFrame2 = [Link](data=data2);
print("DataFrame2:");
print(dataFrame2);

# Subtracting DataFrame2 from DataFrame1

subtractionResults = dataFrame1 - dataFrame2;
print("Result of subtracting dataFrame1 from dataFrame2:");
print(subtractionResults);


 Output:
DataFrame1:
0 1 2 3
0 2 4 6 8
1 1 3 5 7
2 5 0 0 9
DataFrame2:
0 1 2 3
0 1 1 0 1
1 1 0 1 1
2 0 1 1 0
Result of subtracting dataFrame1 from dataFrame2:
0 1 2 3
0 1 3 6 7
1 0 3 4 6
2 5 -1 -1 9
Multiplying A DataFrame With Another DataFrame, Series Or A
Python Sequence
The mul() method of DataFrame object multiplies the elements of
a DataFrame object with another DataFrame object, series or any other
Python sequence.
mul() does an elementwise multiplication of a DataFrame with
another DataFrame, a pandas Series or a Python Sequence.
Calling the mul() method is similar to using the binary multiplication
operator(*).
The mul() method provides a parameter fill_value using which values
can be passed to replace the [Link], None values present in the data.

Example:
import pandas as pd

# Create data
dataSet1 = [(2, 0, 1, 0),
(1, 0, 3, 0),
(4, 3, 2, 0)];

dataSet2 = [(1, 1, 2, 1),

(1, 2, 1, 1),
(3, 1, 1, 3)];

# Construct pandas DataFrame instances

dataFrame1 = [Link](data=dataSet1);
dataFrame2 = [Link](data=dataSet2);

print("Elements present in DataFrame1:");

print(dataFrame1);

print("Elements present in DataFrame2:");

print(dataFrame2);

# Multiply two DataFrames

multiplicationResults = [Link](dataFrame2);
print("Result of element-wise multiplication of two Data Frames:"
print(multiplicationResults);

Output:
Elements present in DataFrame1:
0 1 2 3
0 2 0 1 0
1 1 0 3 0
2 4 3 2 0
Elements present in DataFrame2:
0 1 2 3
0 1 1 2 1
1 1 2 1 1
2 3 1 1 3
Result of element-wise multiplication of two Data Frames:
0 1 2 3
0 2 0 2 0
1 1 0 3 0
2 12 3 2 0

Dataframe Division Operations

 div() method divides element-wise division of one pandas

DataFrame by another.
 DataFrame elements can be divided by a pandas series or by a
Python sequence as well.
 Calling div() on a DataFrame instance is equivalent to invoking the
division operator (/).
  The div() method provides the fill_value parameter which is
used for replacing the [Link] and None values present in
the DataFrame or in the resultant value with any other value.

Example:
import pandas as pd

# Data creation part

data1 = [(20, 40, 60, 80),
(10, 30, 50, 70),
(15, 25, 35, 45)];

data2 = [(2, 3, 4, 6),

(5, 1, 0, 3),
(3, 6, 9, 1)];

# Creation of pandas DataFrames

dataFrame1 = [Link](data=data1);
dataFrame2 = [Link](data=data2);

# Divide the DataFrame1 elements by the elements of DataFrame2

divisionResults = [Link](dataFrame2);

print("Elements of DataFrame1:")
print(dataFrame1);

print("Elements of DataFrame2:")
print(dataFrame2);

print("DataFrame1 elements divided by DataFrame2 elements:")

print(divisionResults);
Output:

Integer Division Or Floor Division Of A Dataframe Using Python

Pandas
Unlike div(), the floordiv() method of the DataFrame class from
Python pandas library, does an element-wise integer division. By integer
division, it is meant that only the quotient, an integer is returned when one
element of a DataFrame is divided by another element of a DataFrame.

Example:
import pandas as pd

# Data
m1 = [(3, 5, 7),
(13, 15, 17),
(21, 23, 25)];

m2 = [(1, 2, 3),
 (4, 5, 6),
(7, 8, 9)];

# Data loaded into pandas DataFrames

dataFrame1 = [Link](data=m1);
dataFrame2 = [Link](data=m2);

# Do an integer division
resultantDataFrame = [Link](dataFrame2);

print("DataFrame1 floor divided by DataFrame2:")

print(resultantDataFrame);

Output:
DataFrame1 floor divided by DataFrame2:
0 1 2
0 3 2 2
1 3 3 2
2 3 2 2

Modulo Division Of A Dataframe Using Python Pandas

 The mod() method is one of the several binary operations provided

by the DataFrame class of Python pandas library.
 The mod() method does a modulo division of an element from a
DataFrame using another DataFrame.
 While the floordiv() method of DataFrame returns the quotient
from the integer division, the mod() method returns the remainder
of the integer division.

 Example:
import pandas as pd
# Data for DataFrame instances
m1 = [(3, 6, 7),
(13, 14, 17),
(21, 24, 25)];

m2 = [(3, 2, 3),
(4, 0, 16),
(7, 0, 9)];

# DataFrames
dataFrame1 = [Link](data=m1);
dataFrame2 = [Link](data=m2);

# Do a modulo division
resultantDataFrame = [Link](dataFrame2);

print("DataFrame1 modulo divided by DataFrame2:")

print(resultantDataFrame);

Output:
DataFrame1 modulo divided by DataFrame2:
0 1 2
0 0.0 0.0 1.0
1 1.0 NaN 1.0
2 0.0 NaN 7.0

Applying Pow() Function On To A Pandas DataFrame Instance

 The built-in function pow() raises a Python numeric variable to a

power exponent.
  In the similar way, the pow() method of a pandas DataFame
instance, raises its elements to a numeric power as given by the
elements of another pandas DataFrame instance.

 Example:
# Example Python program to raise a dataframe elements
# to the power exponent as defined by the elements of
# another dataframe

import pandas as pd

# Data 1
m1 = [(0, 1, 2),
(3, 4, 5),
(6, 7, 8)];

# Data 2
m2 = [(1, 1, 1),
(2, 2, 2),
(3, 3, 3)];

# Create pandas dataframe instances from lists of tuples

dataFrameInstance1 = [Link](data=m1);
dataFrameInstance2 = [Link](data=m2);

raised = [Link](dataFrameInstance2);

# Print input data

print("Pandas DataFrame Instance1:");
print(dataFrameInstance1);

print("Pandas DataFrame Instance2:");

print(dataFrameInstance2);

# Print resultant data

print("Dataframe1 elements raised to the power of Dataframe2 elem
print(raised);


 Output:
Pandas DataFrame Instance1:
0 1 2
0 0 1 2
1 3 4 5
2 6 7 8
Pandas DataFrame Instance2:
0 1 2
0 1 1 1
1 2 2 2
2 3 3 3
Dataframe1 elements raised to the power of Dataframe2 elements:
0 1 2
0 0 1 2
1 9 16 25
2 216 343 512

Functional Statistics:
Performing various complex statistical operations in python can be easily reduced to single
line commands using pandas. We will discuss some of the most useful and common statistical
operations in this post. We will be using the Titanic survival dataset to demonstrate such
operations.

# Import Pandas Library

import pandas as pd
 # Load Titanic Dataset as Dataframe

dataset = pd.read_csv('[Link]')

# Show dataset

# head() bydefault show

# 5 rows of the dataframe

[Link]()

Output:

1. Mean:
Calculates the mean or average value by using DataFrame/[Link]() method.
Syntax: DataFrame/[Link](self, axis=None, skipna=None, level=None,
numeric_only=None, **kwargs)
Parameters:
 axis: {index (0), columns (1)}
Specify the axis for the function to be applied on.
 skipna: This parameter takes bool value, default value is True
It excludes null values when computing the result.
 level: This parameter takes int value or level name, default value is None.
If the axis is a MultiIndex, count along a particular level, collapsing into a Series.
 numeric_only: This parameter takes bool value, default value is None
Include only float, int, boolean columns. If None, will attempt to use everything, then
use only numeric data values. Not implemented for Series.
 **kwargs: Additional arguments to be passed to the function.
Returns: Mean of Series or DataFrame (if level specified)
Code:

# Calculate the Mean

# of 'Age' column
mean = dataset['Age'].mean()
 # Print mean
print(mean)

Output:
29.69911764705882
2. Median:
Calculates the median value by using DataFrame/[Link]() method.
Syntax: DataFrame/[Link](self, axis=None, skipna=None, level=None,
numeric_only=None, **kwargs)
Parameters:
 axis: {index (0), columns (1)}
Specify the axis for the function to be applied on.
 skipna: This parameter takes bool value, default value is True
It excludes null values when computing the result.
 level: This parameter takes int or level name, default None
If the axis is a MultiIndex, count along a particular level, collapsing into a Series.
 numeric_only: This parameter takes bool value, default value is None
Include only float, int, boolean columns. If value is None, will attempt to use
everything, then use only numeric data.
 **kwargs: Additional arguments to be passed to the function.
Returns: Median of Series or DataFrame (if level specified)
Code:

# Calculate Median of 'Fare' column

median = dataset['Fare'].median()

# Print median
print(median)

Output:
14.4542

3. Mode:
Calculates the mode or most frequent value by using [Link]() method.

Syntax: DataFrame/[Link](self, axis=0, numeric_only=False, dropna=True)

Parameters:
 axis: {index (0), columns (1)}
The axis to iterate over while searching for the mode value:
0 value or ‘index’ : get mode of each column
 1 value or ‘columns’ : get mode of each row.
 numeric_only: This parameter takes bool value, default value is False.
If True, only apply to numeric value columns.
 dropna: This parameter takes bool value, default value is True.
Don’t consider counts of NaN/None value.
Returns: Highest frequency value.

Code:

# Calculate Mode of 'Sex' column

mode = dataset['Sex'].mode()

# Print mode
print(mode)

Output:
0 male
dtype: object

4. Count:
Calculates the count or frequency of non-null values by
using DataFrame/[Link]() Method.
Syntax: DataFrame/[Link](self, axis=0, level=None, numeric_only=False)
Parameters:
 axis: {0 or ‘index’, 1 or ‘columns’}, default value is 0
If value is 0 or ‘index’ counts are generated for each column. If value is 1 or
‘columns’ counts are generated for each row.
 level: (optional)This parameter takes int or str value.
If the axis is a MultiIndex type, count along a particular level, collapsing into a
DataFrame. A str is used specifies the level name.
 numeric_only: This parameter takes bool value, default False
Include only float, int or boolean [Link]: Return the highest frequency value
Returns: For each column/row the number of non-null entries. If level is specified returns a
DataFrame structure.
Code:

# Calculate Count of 'Ticket' column

count = dataset['Ticket'].count()
 # Print count
print(count)

Output:
891

5. Standard Deviation:
Calculates the standard deviation of values by using DataFrame/[Link]() method.
Syntax: DataFrame/[Link](self, axis=None, skipna=None, level=None, ddof=1,
numeric_only=None, **kwargs)
Parameters:
 axis: {index (0), columns (1)}
 skipna: This parameters takes bool value, default value is True.
Exclude NA/null values. If an entire row/column has NA values, the result will be NA
value.
 level: This parameters takes int or level name, default value is None.
If the axis is a MultiIndex, count along a particular level, collapsing into a Series.
 ddof: This parameter take int value, default value is 1.
Delta Degrees of Freedom. The divisor used in calculations is N – ddof, where N
value represents the number of elements.
 numeric_only: This parameter takes bool value , default None
Include only float, int, boolean columns. If None, will attempt to use everything, then
use only numeric data. Not implemented for Series.
Returns: Standard Deviation
Code:

# Calculate Standard Deviation

# of 'Fare' column
std = dataset['Fare'].std()

# Print standard deviation

print(std)

Output:
49.693428597180905

6. Max:
Calculates the maximum value using DataFrame/[Link]() method.
Syntax: DataFrame/[Link](self, axis=None, skipna=None, level=None,
numeric_only=None, **kwargs)
Parameters:
 axis: {index (0), columns (1)}
Specify the axis for the function to be applied on.
 skipna: bool, default True
It excludes null values when computing the result.
 level: int or level name, default None
If the axis is a MultiIndex type, count along a particular level, collapsing into a
Series.
 numeric_only: bool, default None
Include only float, int, boolean columns. If None value, will attempt to use
everything, then use only numeric data.
 **kwargs: Additional keyword to be passed to the function.
Returns: Maximum value in Series or DataFrame (if level specified)
Code:

# Calculate Maximum value in 'Age' column

maxValue = dataset['Age'].max()

# Print maxValue
print(maxValue)

Output:
80.0

7. Min:
Calculates the minimum value using DataFrame/[Link]() method.
Syntax: DataFrame/[Link](self, axis=None, skipna=None, level=None,
numeric_only=None, **kwargs)
Parameters:
 axis: {index (0), columns (1)}
Specify the axis for the function to be applied on.
 skipna: bool, default True
It excludes null values when computing the result.
 level: int or level name, default None
If the axis is a MultiIndex type, count along a particular level, collapsing into a
Series.
 numeric_only: bool, default None
Include only float, int, boolean columns. If None value, will attempt to use
everything, then use only numeric data.
 **kwargs: Additional keyword to be passed to the function.
Returns: Minimum value in Series or DataFrame (if level specified)
Code:
 # Calculate Minimum value in 'Fare' column
minValue = dataset['Fare'].min()

# Print minValue
print(minValue)

Output:
0.0000

8. Describe:

Summarizes general descriptive statistics using DataFrame/[Link]() method.

Syntax: DataFrame/[Link](self: ~ FrameOrSeries, percentiles=None,

include=None, exclude=None)
Parameters:
 percentiles: list-like of numbers, optional
 include: ‘all’, list-like of dtypes or None values (default), optional
 exclude: list-like of dtypes or None values (default), optional,
Returns: Summary statistics of the Series or Dataframe provided.

# Statistical summary
[Link]()

Output:

How to Sort Pandas DataFrame?

Creating a dataframe for demonstration
 # importing pandas library
import pandas as pd

# creating and initializing a nested list

age_list = [['Afghanistan', 1952, 8425333, 'Asia'],
['Australia', 1957, 9712569, 'Oceania'],
['Brazil', 1962, 76039390, 'Americas'],
['China', 1957, 637408000, 'Asia'],
['France', 1957, 44310863, 'Europe'],
['India', 1952, 3.72e+08, 'Asia'],
['United States', 1957, 171984000, 'Americas']]

# creating a pandas dataframe

df = [Link](age_list, columns=['Country', 'Year',
'Population', 'Continent'])

Df

Output :

Sort Pandas DataFrame

Sorting Pandas Data Frame

In order to sort the data frame in pandas, function sort_values() is
used. Pandas sort_values() can sort the data frame in Ascending or
Descending order.
Example 1: Sorting the Data frame in Ascending order
# Sorting by column 'Country'
df.sort_values(by=['Country'])

Output :
 Sort Pandas DataFrame

Example 2: Sorting the Data frame in Descending order

# Sorting by column "Population"
df.sort_values(by=['Population'], ascending=False)

Output :

Sort Pandas DataFrame

Example 3: Sorting Pandas Data frame by putting missing values first

# Sorting by column "Population"
# by putting missing values first
df.sort_values(by=['Population'], na_position='first')

Output :
 Sort Pandas DataFrame

Example 4: Sorting Data frames by multiple columns

# Sorting by columns "Country" and then "Continent"
df.sort_values(by=['Country', 'Continent'])

Output :

Sort Pandas DataFrame

Example 5: Sorting Data frames by multiple columns but different order

# Sorting by columns "Country" in descending
# order and then "Continent" in ascending order

df.sort_values(by=['Country', 'Continent'],
ascending=[False, True])

Output:
 Sort Pandas DataFrame

Indexing and Selecting Data with Pandas

Indexing in Pandas :
Indexing in pandas means simply selecting particular rows and columns of
data from a DataFrame. Indexing could mean selecting all the rows and
some of the columns, some of the rows and all of the columns, or some of
each of the rows and columns. Indexing can also be known as Subset
Selection.
Let’s see some example of indexing in Pandas. In this article, we are using
“[Link]” file to download the CSV, click here.
Selecting some rows and some columns
Let’s take a DataFrame with some fake data, now we perform indexing on
this DataFrame. In this, we are selecting some rows and some columns from
a DataFrame. Dataframe with dataset.

Suppose we want to select columns Age, College and Salary for only rows
with a labels Amir Johnson and Terry Rozier
Our final DataFrame would look like this:

Selecting some rows and all columns

Let’s say we want to select row Amir Jhonson, Terry Rozier and John
Holland with all columns in a dataframe.

Our final DataFrame would look like this:

 Selecting some columns and all rows
Let’s say we want to select columns Age, Height and Salary with all rows in a
dataframe.

Our final DataFrame would look like this:

Pandas Indexing using [ ], .loc[], .iloc[ ], .ix[ ]

There are a lot of ways to pull the elements, rows, and columns from a
DataFrame. There are some indexing method in Pandas which help in
getting an element from a DataFrame. These indexing methods appear very
similar but behave very differently. Pandas support four types of Multi-axes
indexing they are:
 Dataframe.[ ] ; This function also known as indexing operator
 [Link][ ] : This function is used for labels.
  [Link][ ] : This function is used for positions or integer
based
 [Link][] : This function is used for both label and integer
based
Collectively, they are called the indexers. These are by far the most
common ways to index data. These are four function which help in getting
the elements, rows, and columns from a DataFrame.

Indexing a Dataframe using indexing operator [] :

Indexing operator is used to refer to the square brackets following an object.
The .loc and .iloc indexers also use the indexing operator to make
selections. In this indexing operator to refer to df[].
Selecting a single columns
In order to select a single column, we simply put the name of the column in-
between the brackets

# importing pandas package

import pandas as pd

# making data frame from csv file

data = pd.read_csv("[Link]", index_col ="Name")

# retrieving columns by indexing operator

first = data["Age"]

print(first)
Output:

Selecting multiple columns

In order to select multiple columns, we have to pass a list of columns in an
indexing operator.

# importing pandas package

import pandas as pd

# making data frame from csv file

data = pd.read_csv("[Link]", index_col ="Name")

# retrieving multiple columns by indexing operator

first = data[["Age", "College", "Salary"]]

first
Output:

Indexing a DataFrame using .loc[ ] :

This function selects data by the label of the rows and columns.
The [Link] indexer selects data in a different way than just the indexing
operator. It can select subsets of rows or columns. It can also simultaneously
select subsets of rows and columns.
Selecting a single row
In order to select a single row using .loc[], we put a single row label in
a .loc function.
# importing pandas package

import pandas as pd
 # making data frame from csv file

data = pd.read_csv("[Link]", index_col ="Name")

# retrieving row by loc method

first = [Link]["Avery Bradley"]

second = [Link]["R.J. Hunter"]

print(first, "\n\n\n", second)

Output:
As shown in the output image, two series were returned since there was only
one parameter both of the times.

Selecting multiple rows

In order to select multiple rows, we put all the row labels in a list and pass
that to .loc function.
import pandas as pd
 # making data frame from csv file

data = pd.read_csv("[Link]", index_col ="Name")

# retrieving multiple rows by loc method

first = [Link][["Avery Bradley", "R.J. Hunter"]]

print(first)

Output:

Selecting two rows and three columns

In order to select two rows and three columns, we select a two rows which
we want to select and three columns and put it in a separate list like this:
[Link][["row1", "row2"], ["column1", "column2", "column3"]]

import pandas as pd

# making data frame from csv file

data = pd.read_csv("[Link]", index_col ="Name")

# retrieving two rows and three columns by loc method

first = [Link][["Avery Bradley", "R.J. Hunter"],

["Team", "Number", "Position"]]

print(first)
Output:

Selecting all of the rows and some columns

In order to select all of the rows and some columns, we use single
colon [:] to select all of rows and list of some columns which we want to
select like this:
[Link][:, ["column1", "column2", "column3"]]

import pandas as pd

# making data frame from csv file

data = pd.read_csv("[Link]", index_col ="Name")

# retrieving all rows and some columns by loc method

first = [Link][:, ["Team", "Number", "Position"]]

print(first)

Output:

Indexing a DataFrame using .iloc[ ] :

This function allows us to retrieve rows and columns by position. In order to
do that, we’ll need to specify the positions of the rows that we want, and the
positions of the columns that we want as well. The [Link] indexer is very
similar to [Link] but only uses integer locations to make its selections.
Selecting a single row
In order to select a single row using .iloc[], we can pass a single integer
to .iloc[] function.
import pandas as pd

# making data frame from csv file

data = pd.read_csv("[Link]", index_col ="Name")

# retrieving rows by iloc method

row2 = [Link][3]

print(row2)

Output:

Selecting multiple rows

In order to select multiple rows, we can pass a list of integer
to .iloc[] function.
import pandas as pd

# making data frame from csv file

data = pd.read_csv("[Link]", index_col ="Name")

# retrieving multiple rows by iloc method

row2 = [Link] [[3, 5, 7]]

 row2

Output:

Selecting two rows and two columns

In order to select two rows and two columns, we create a list of 2 integer for
rows and list of 2 integer for columns then pass to a .iloc[] function.
import pandas as pd

# making data frame from csv file

data = pd.read_csv("[Link]", index_col ="Name")

# retrieving two rows and two columns by iloc method

row2 = [Link] [[3, 4], [1, 2]]

print(row2)

Output:

Selecting all the rows and a some columns

In order to select all rows and some columns, we use single colon [:] to
select all of rows and for columns we make a list of integer then pass to
a .iloc[] function.
import pandas as pd

# making data frame from csv file

data = pd.read_csv("[Link]", index_col ="Name")

 # retrieving all rows and some columns by iloc method

row2 = [Link] [:, [1, 2]]

print(row2)

Output:

Indexing a using [Link][ ] :

Early in the development of pandas, there existed another indexer, ix. This
indexer was capable of selecting both by label and by integer location. While
it was versatile, it caused lots of confusion because it’s not explicit.
Sometimes integers can also be labels for rows or columns. Thus there were
instances where it was ambiguous. Generally, ix is label based and acts just
as the .loc indexer. However, .ix also supports integer type selections (as in
.iloc) where passed an integer. This only works where the index of the
DataFrame is not integer based .ix will accept any of the inputs
of .loc and .iloc.
Note: The .ix indexer has been deprecated in recent versions of Pandas.
Selecting a single row using .ix[] as .loc[]
In order to select a single row, we put a single row label in a .ix function.
This function act similar as .loc[] if we pass a row label as a argument of a
function.
# importing pandas package

import pandas as pd

# making data frame from csv file

data = pd.read_csv("[Link]", index_col ="Name")

# retrieving row by ix method

first = [Link]["Avery Bradley"]

print(first)

Output:

Selecting a single row using .ix[] as .iloc[]

In order to select a single row, we can pass a single integer
to .ix[] function. This function similar as a iloc[] function if we pass an
integer in a .ix[] function.
 # importing pandas package

import pandas as pd

# making data frame from csv file

data = pd.read_csv("[Link]", index_col ="Name")

# retrieving row by ix method

first = [Link][1]

print(first)

Output:

Advanced uses of Pandas for Data Analysis:

The pandas library offers core functionality when preparing your data using Python. But,
many don't go beyond the basics, so learn about these lesser-known advanced methods that
will make handling your data easier and cleaner.
Pandas is the gold standard library for all things data. With the functionality to load, filter,
manipulate, and explore data, it’s no wonder that it’s a favorite among Data Scientists.
Most of us naturally stick to the very basics of Pandas. Load up data from a CSV file, filter a
few columns, and then jump right into the data visualizations. Yet Pandas actually comes with
many lesser-known but useful functions that can make handling data a whole lot easier and
cleaner.
The following are the 5 more advanced functions:
1) Configuring Options and Settings

Pandas comes with a set of user-configurable options and settings. They’re huge productivity

boosters since they let you tailor your Pandas environment exactly to your liking.

We can, for example, change some of Pandas’s display settings to change how many rows

and columns are shown and to what precision floating point numbers are displayed.
import pandas as pd

display_settings = {
'max_columns': 10,
'expand_frame_repr': True, # Wrap to multiple pages
'max_rows': 10,
'precision': 2,
'show_dimensions': True
}

for op, value in display_settings.items():

pd.set_option("display.{}".format(op), value)

The code above ensures that Pandas always displays 10 rows and 10 columns at a maximum,
with floating-point values showing 2 decimal places at most. That way, our terminal or
Jupyter Notebook won’t look like a mess when we try to print out a big DataFrame!

(2) Combining DataFrames

A relatively unknown part of Pandas DataFrames is that there are actually two different ways

to combine them. Each method produces a different result, so selecting the proper one based

on what you want to achieve is very important. In addition, they contain many parameters

that further customize the merging. Let’s check them out.

Concatenating

Concatenating is the most well-known method of combining DataFrames and can be thought

of intuitively as “stacking.” That stacking can be done either horizontally or vertically.

Imagine that you have a huge dataset in CSV format. It makes sense to split it up into

multiple files for easier handling (this is common practice for large datasets, referred to

as sharding).

When you load it into pandas you can vertically stack the DataFrame of each CSV to create

one big DataFrame for all of the data. For example, if we have 3 shards, each with 5 Million

rows, then after we vertical stack them all, our final DataFrame will have 15 Million rows.

The code below shows how to concatenate DataFrames in Pandas vertically.

# Vertical concat
[Link]([october_df, november_df, december_df], axis=0)
Merging
(3) Reshaping DataFrames

Transpose

The easiest of them all. Transposing swaps a DataFrame’s rows with its columns. If you have

5000 rows and 10 columns, and then transpose your DataFrame, you’ll end up with 10 rows

and 5000 columns.

Groupby

Stacking

Stacking transforms the DataFrame into having a multi-level index, i.e., each row has

multiple sub-parts. These sub-parts are created using the DataFrame’s columns, compressing

them into the multi-index. Overall, stacking can be thought of as compressing columns into

multi-index rows.

df=[Link]()

print(df)

"""
0 Player Superman
Year 2000
Points 23
1 Player Batman
Year 2000
Points 43
2 Player Thanos
Year 2000
Points 45
3 Player Batman
Year 2001
Points 65
4 Player Thanos
Year 2001
 Points 76
5 Player Superman
Year 2002
Points 34
6 Player Batman
Year 2002
Points 23
7 Player Thanos
Year 2002
Points 78
8 Player Black Widow
Year 2003
Points 89
9 Player Batman
Year 2004
Points 76
10 Player Thanos
Year 2004
Points 92
11 Player Superman
Year 2005
Points 87

(4) Working with time data

The Datetime library is a staple in Python. Whenever you’re dealing with anything related to

real-world date and time information, it’s your go-to library. And lucky for us, Pandas also

comes with functionality for using Datetime objects.

Let’s illustrate with an example. In the code below, we first create a DataFrame with 4

columns: Day, Month, Year, and data, and then sort it by year and month. As you can see, it’s

quite messy; we’re using up 3 columns just to store the date, when in actuality, we know that

a calendar date is just one value.

from itertools import product

import pandas as pd
import numpy as np
 col_names = ["Day", "Month", "Year"]

df = [Link](list(product([10, 11, 12], [8, 9], [2018, 2019])),

columns=col_names)

df['data'] = [Link](len(df))

df = df.sort_values(['Year', 'Month'], ascending=[True, True])

print(df)

Output:
Day Month Year data
0 10 8 2018 1.685356
4 11 8 2018 0.441383
8 12 8 2018 1.276089
2 10 9 2018 -0.260338
6 11 9 2018 0.404769
10 12 9 2018 -0.359598
1 10 8 2019 0.145498
5 11 8 2019 -0.731463
9 12 8 2019 -1.451633
3 10 9 2019 -0.988294
7 11 9 2019 -0.687049
11 12 9 2019 -0.067432

We can clean things up with datetime.

Pandas conveniently comes with a function called to_datetime() that can compress and

convert multiple DataFrame columns into a single Datetime object. Once it’s in that format,

you have all the flexibility of the Datetime library at your disposal.
To use the to_datetime() function, you’ll need to pass it all of the “date” data from the

relevant columns. That’s the “Day”, “Month”, and “Year” columns. Once we have things in

Datetime format, we no longer need the other columns and can simply drop them. Check out

the code below to see how that all works!

import pandas as pd
import numpy as np
col_names = ["Day", "Month", "Year"]

df = [Link](list(product([10, 11, 12], [8, 9], [2018, 2019])),

columns=col_names)

df['data'] = [Link](len(df))

df = df.sort_values(['Year', 'Month'], ascending=[True, True])

[Link](loc=0, column="date", value=pd.to_datetime(df[col_names]))

df = [Link](col_names, axis=1).squeeze()

print(df)

Output:
date data
0 2018-08-10 -0.328973
4 2018-08-11 -0.670790
8 2018-08-12 -1.360565
2 2018-09-10 -0.401973
6 2018-09-11 -1.238754
10 2018-09-12 0.957695
1 2019-08-10 0.571126
5 2019-08-11 -1.320735
9 2019-08-12 0.196036
3 2019-09-10 -1.717800
 7 2019-09-11 0.074606
11 2019-09-12 -0.643198

(5) Mapping Items into Groups

Mapping is a neat trick that helps with organizing categorical data. Imagine, for

example, that we have a huge DataFrame with thousands of rows where one of the

columns has items we wish to categorize. Doing so can greatly simplify both the
training of Machine Learning models and visualizing the data effectively.
import pandas as pd

foods = [Link](["Bread", "Rice", "Steak", "Ham", "Chicken",

"Apples", "Potatoes", "Mangoes", "Fish",
"Bread", "Rice", "Steak", "Ham", "Chicken",
"Apples", "Potatoes", "Mangoes", "Fish",
"Apples", "Potatoes", "Mangoes", "Fish",
"Apples", "Potatoes", "Mangoes", "Fish",
"Bread", "Rice", "Steak", "Ham", "Chicken",
"Bread", "Rice", "Steak", "Ham", "Chicken",
"Bread", "Rice", "Steak", "Ham", "Chicken",
"Apples", "Potatoes", "Mangoes", "Fish",
"Apples", "Potatoes", "Mangoes", "Fish",
"Apples", "Potatoes", "Mangoes", "Fish",
"Bread", "Rice", "Steak", "Ham", "Chicken",
"Bread", "Rice", "Steak", "Ham", "Chicken",])

groups_dict = {
"Protein": ["Steak", "Ham", "Chicken", "Fish"],
"Carbs": ["Bread", "Rice", "Apples", "Potatoes", "Mangoes"]
}

In the code above, we put our list into a pandas series. We’ve also created a dictionary

showing the mapping we want, categorizing each food item as a “Protein” or a “Carb.” This

is a toy example, but if this series was at a large scale, say a length of 1,000,000 items, then

looping through it wouldn’t be practical at all.

Instead of the basic for-loop, we can write a function using Pandas’s built-in .map() function

to perform the mapping in an optimized way. Check out the code below to see the function

and how it’s applied.

def membership_map(pandas_series, groups_dict):
groups = {x: k for k, v in groups_dict.items() for x in v}
 mapped_series = pandas_series.map(groups)
return mapped_series

mapped_data = membership_map(foods, groups_dict)

print(list(mapped_data))

In the function, we first loop through our dictionary to create a new dictionary where the keys

represent every possible item in the pandas series and the value represents the new mapped

item, “Protein” or “Carbs”. Then, we simply apply Pandas’s built-in map function to map all

of the values in the series

Check out the output below to see the results!

['Carbs', 'Carbs', 'Protein', 'Protein', 'Protein', 'Carbs', 'Carbs',
'Carbs', 'Protein', 'Carbs', 'Carbs', 'Protein', 'Protein', 'Protein',
'Carbs', 'Carbs', 'Carbs', 'Protein', 'Carbs', 'Carbs', 'Carbs', 'Protein',
'Carbs', 'Carbs', 'Carbs', 'Protein', 'Carbs', 'Carbs', 'Protein',
'Protein', 'Protein', 'Carbs', 'Carbs', 'Protein', 'Protein', 'Protein',
'Carbs', 'Carbs', 'Protein', 'Protein', 'Protein', 'Carbs', 'Carbs',
'Carbs', 'Protein', 'Carbs', 'Carbs', 'Carbs', 'Protein', 'Carbs', 'Carbs',
'Carbs', 'Protein', 'Carbs', 'Carbs', 'Protein', 'Protein', 'Protein',
'Carbs', 'Carbs', 'Protein', 'Protein', 'Protein']

The index is like an address, that’s how any data point across the data frame
or series can be accessed. Rows and columns both have indexes, rows
indices are called index and for columns, it’s general column names.
Hierarchical Indexes
Hierarchical Indexes are also known as multi-indexing is setting more than
one column name as the index. In this article, we are going to
use [Link] file.
 Python3
# importing pandas library as alias pd

import pandas as pd

# calling the pandas read_csv() function.

# and storing the result in DataFrame df

df = pd.read_csv('[Link]')
 print([Link]())

Output:

In the following data frame, there is no indexing.

Columns in the Dataframe:
 Python3
# using the pandas columns attribute.

col = [Link]

print(col)

Output:
Index([‘Unnamed: 0’, ‘region’, ‘state’, ‘individuals’, ‘family_members’,
‘state_pop’],
dtype=’object’)
To make the column an index, we use the Set_index() function of pandas. If
we want to make one column an index, we can simply pass the name of the
column as a string in set_index(). If we want to do multi-indexing or
Hierarchical Indexing, we pass the list of column names in the set_index().
Below Code demonstrates Hierarchical Indexing in pandas:
 Python3
# using the pandas set_index() function.

df_ind3 = df.set_index(['region', 'state', 'individuals'])

# we can sort the data by using sort_index()

df_ind3.sort_index()

print(df_ind3.head(10))

Output:
Now the dataframe is using Hierarchical Indexing or multi-indexing.
Note that here we have made 3 columns as an index (‘region’, ‘state’,
‘individuals’ ). The first index ‘region’ is called level(0) index, which is on top
of the Hierarchy of indexes, next index ‘state’ is level(1) index which is below
the main or level(0) index, and so on. So, the Hierarchy of indexes is formed
that’s why this is called Hierarchical indexing.
We may sometimes need to make a column as an index, or we want to
convert an index column into the normal column, so there is a
pandas reset_index(inplace = True) function, which makes the index column
the normal column.
Selecting Data in a Hierarchical Index or using the Hierarchical
Indexing:
For selecting the data from the dataframe using the .loc() method we have to
pass the name of the indexes in a list.
 Python3
# selecting the 'Pacific' and 'Mountain'

# region from the dataframe.

# selecting data using level(0) index or main index.

df_ind3_region = df_ind3.loc[['Pacific', 'Mountain']]

print(df_ind3_region.head(10))

Output:
We cannot use only level(1) index for getting data from the dataframe, if we
do so it will give an error. We can only use level (1) index or the inner
indexes with the level(0) or main index with the help list of tuples.

 Python3
# using the inner index 'state' for getting data.

df_ind3_state = df_ind3.loc[['Alaska', 'California', 'Idaho']]

print(df_ind3_state.head(10))

Output:

Using inner levels indexes with the help of a list of tuples:

Syntax:
[Link][[ ( level( 0 ) , level( 1 ) , level( 2 ) ) ]]

 Python3
# selecting data by passing all levels index.

df_ind3_region_state = df_ind3.loc[[("Pacific", "Alaska", 1434),

("Pacific", "Hawaii", 4131),

("Mountain", "Arizona", 7259),

("Mountain", "Idaho", 1297)]]

df_ind3_region_state

Output:
 The Panel Data
A panel is a 3D container of data. The term Panel data is derived from econometrics and is
partially responsible for the name pandas − pan(el)-da(ta)-s.
The names for the 3 axes are intended to give some semantic meaning to describing operations
involving panel data. They are −
 items − axis 0, each item corresponds to a DataFrame contained inside.
 major_axis − axis 1, it is the index (rows) of each of the DataFrames.
 minor_axis − axis 2, it is the columns of each of the DataFrames.

[Link]()
A Panel can be created using the following constructor −
[Link](data, items, major_axis, minor_axis, dtype, copy)
The parameters of the constructor are as follows −

Parameter Description

data Data takes various forms like ndarray, series, map, lists, dict, constants and also another DataFrame

items axis=0

major_axis axis=1

minor_axis axis=2

dtype Data type of each column

copy Copy data. Default, false

Create Panel
A Panel can be created using multiple ways like −
  From ndarrays
 From dict of DataFrames
From 3D ndarray
# creating an empty panel
import pandas as pd
import numpy as np

data = [Link](2,4,5)
p = [Link](data)
print p
Its output is as follows −
<class '[Link]'>
Dimensions: 2 (items) x 4 (major_axis) x 5 (minor_axis)
Items axis: 0 to 1
Major_axis axis: 0 to 3
Minor_axis axis: 0 to 4
Note − Observe the dimensions of the empty panel and the above panel, all the objects are
different.

From dict of DataFrame Objects

#creating an empty panel
import pandas as pd
import numpy as np

data = {'Item1' : [Link]([Link](4, 3)),

'Item2' : [Link]([Link](4, 2))}
p = [Link](data)
print p
Its output is as follows −
Dimensions: 2 (items) x 4 (major_axis) x 3 (minor_axis)
Items axis: Item1 to Item2
Major_axis axis: 0 to 3
Minor_axis axis: 0 to 2
Create an Empty Panel
An empty panel can be created using the Panel constructor as follows −
#creating an empty panel
import pandas as pd
p = [Link]()
print p
Its output is as follows −
<class '[Link]'>
Dimensions: 0 (items) x 0 (major_axis) x 0 (minor_axis)
Items axis: None
Major_axis axis: None
Minor_axis axis: None

Selecting the Data from Panel

Select the data from the panel using −

 Items
 Major_axis
 Minor_axis
Using Items
# creating an empty panel
import pandas as pd
import numpy as np
data = {'Item1' : [Link]([Link](4, 3)),
'Item2' : [Link]([Link](4, 2))}
p = [Link](data)
print p['Item1']
Its output is as follows −
0 1 2
0 0.488224 -0.128637 0.930817
1 0.417497 0.896681 0.576657
2 -2.775266 0.571668 0.290082
3 -0.400538 -0.144234 1.110535
We have two items, and we retrieved item1. The result is a DataFrame with 4 rows and 3
columns, which are the Major_axis and Minor_axis dimensions.

Using major_axis
Data can be accessed using the method panel.major_axis(index).
# creating an empty panel
import pandas as pd
import numpy as np
data = {'Item1' : [Link]([Link](4, 3)),
'Item2' : [Link]([Link](4, 2))}
p = [Link](data)
print p.major_xs(1)
Its output is as follows −
Item1 Item2
0 0.417497 0.748412
1 0.896681 -0.557322
2 0.576657 NaN
Using minor_axis
Data can be accessed using the method panel.minor_axis(index).
# creating an empty panel
import pandas as pd
import numpy as np
data = {'Item1' : [Link]([Link](4, 3)),
'Item2' : [Link]([Link](4, 2))}
p = [Link](data)
print p.minor_xs(1)
Its output is as follows −
Item1 Item2
0 -0.128637 -1.047032
1 0.896681 -0.557322
2 0.571668 0.431953
3 -0.144234 1.302466
Note − Observe the changes in the dimensions.