Python Pandas
Uploaded by
punamrattanPython Pandas
Uploaded by
punamrattanPROGRAMMING IN PYTHON
Created By:
Kumar Vishal
Handling data with pandas
introduction to pandas
How to install pandas before going to use?
Create a Series from ndarray,list,tuple,dictionary
Series
Create a Series from Scalar
If data is a scalar value, an index must be provided. The value will be
repeated to match the length of index
Labels
If nothing else is specified, the values are labeled with their index number. First
value has index 0, second value has index 1 etc.
Create Labels
With the index argument, you can name your own labels.
Data frame
A Data frame is a two-dimensional data structure, i.e., data is aligned in a tabular
fashion in rows and columns.
DataFrame can be created using :Lists,dict,Series,Numpy ndarrays another DataFrame
DataFrame can be created using the following constructor −
[Link]( data, index, columns, dtype, copy)
data can be Lists,dict,Series,Numpy ndarrays another DataFrame
index: For the row labels
columns: For column labels
dtype: data type of each column
copy: copying of data
Name Age
Alex 10
Bob 12
Clarke 13
John 14
Create a DataFrame from Dictionary of Lists
Create a DataFrame from List of Dicts
create an indexed DataFrame
Addition of Rows
Add new rows to a DataFrame using the append function. This function
will append the rows at the end.
Deletion of Rows
Use index label to delete or drop rows from a DataFrame. If label is
duplicated, then multiple rows will be dropped.
Sorting Pandas Data Frame
sort_values() can sort the data frame in Ascending or Descending order.
Sorting the Data frame in Ascending order
Sorting the Data frame in Descending order
Working with csv files
CSV stands for “Comma Separated Values.” It is the simplest form of
storing data in tabular form as plain text.
Structure of CSV:
We have a file named “Salary_Data.csv.”
The first line of a CSV file is the header
and contains the names of the
fields/features. After the header, each line
of the file is an observation/a record. The
values of a record are separated by
“comma.”
Read CSV Files
Opening a local CSV file
Use read_csv()
Opening a CSV file from a URL
If you have a large DataFrame with many rows,
Pandas will only return the first 5 rows, and the
last 5 rows
DataFrame with the to_string() method
It will display all rows and cols
[Link]
[Link] attribute to return the column labels of the given
dataframe
nrows
Using nrows keyword you can display no. of rows according to your
choice.
usecols
Using usecols keyword you can display no. of columns according to your
choice.
head()
[Link](): use to display top 5 rows
tail()
[Link](): use to display bottom 5 rows
skiprows
If you want to skip rows from your dataset then use argument skiprows
index_col
If you want to change index with any column name then use argument
index_col
finding data using column name
finding data row-wise
Apply conditions in dataframe
selecting rows based on condition
Create CSV file using to_csv()
Data cleaning
When working with multiple data sources, there are many chances for data
to be incorrect, duplicated, or mislabeled In such situation your decision can
be wrong during data analysis.
Data cleaning is the process of changing or eliminating garbage, incorrect,
duplicate, corrupted, or incomplete data in a dataset.
So, A data set can contain:
• Empty cells
• Data in wrong format
• Wrong data and
• Duplicates
Steps to Cleaning Data:
• Import Dataset : To import the dataset we use the read_csv() function of
pandas and store it in the DataFrame named as data.
• Merge Dataset: Merging the dataset is the process of combining two
datasets in one, and line up rows based on some particular or common
property for data analysis. We can do this by using the merge() function of
the dataframe.
• Rebuild Missing Data: To find and fill the missing data in the dataset we
will use another function. There are 4 ways to find the null values if present
in the dataset.
• Using isnull() function
• Using isna() function
• Using isna().any()
• Using isna(). sum()
• Using isna().any().sum()
• Standardization and Normalization
• De-Duplicate: De-Duplicate means remove all duplicate values. There is no
need for duplicate values in data analysis. These values only affect the
accuracy and efficiency of the analysis result. To find duplicate values in the
dataset we will use a simple dataframe function i.e. duplicated().
• Verify and Enrich: After removing null, duplicate, and incorrect values, we
should verify the dataset and validate its accuracy. In this step, we have to
check that the data cleaned so far is making any sense. If the data is
incomplete we have to enrich the data again by data gathering activities like
approaching the clients again, re-interviewing people, etc.
• Export Dataset: This is the last step of the data cleaning process. After
performing all the above operations, the data is transformed into clean the
dataset and it is ready to export for the next process in Data Science or Data
Analysis.
Empty Cells
One way to deal with empty cells is to remove rows that contain empty
cells.
• Return a new Data Frame with no empty cells using dropna()
• If you want to change the original DataFrame, use the inplace = True
argument
• Another way of dealing with empty cells is to insert a new value instead using
fillna() method
Data of Wrong Format
Convert all cells in the 'Date' column into dates. Pandas has a
to_datetime() method for this.
Replacing Values
Set "Duration" = 45 in row 7:
[Link][7, 'Duration'] = 45
• Discovering Duplicates