Unit-8

Python for Data Analysis

8.1. Introduction to NumPy
NumPy, which stands for Numerical Python, is a powerful Python library for scientific
computing. It provides support for large, multi-dimensional arrays and matrices, along with a
collection of mathematical functions to operate on these arrays efficiently. NumPy is widely used
in fields such as data analysis, machine learning, image processing, and scientific research.

Here are some key features and concepts of NumPy:

1. Multi-dimensional Arrays: NumPy's main feature is its `ndarray` (n-dimensional array) object,
which represents a multi-dimensional, homogeneous array of fixed-size items. It allows efficient
storage and manipulation of large datasets. Arrays can be 1-dimensional, 2-dimensional
(matrices), or higher-dimensional.

2. Mathematical Operations: NumPy provides a wide range of mathematical functions that can
be applied element-wise on arrays, such as addition, subtraction, multiplication, division,
exponentiation, trigonometric functions, logarithmic functions, etc. These operations are
optimized for performance and can be performed on entire arrays without the need for explicit
loops.

3. Indexing and Slicing: NumPy offers powerful indexing and slicing capabilities to access
specific elements, rows, columns, or subarrays within an array. It allows for both integer and
boolean indexing, making it easy to extract and manipulate data.

4. Broadcasting: NumPy supports broadcasting, which is a powerful mechanism for performing

operations between arrays of different shapes. Broadcasting allows NumPy to handle operations
between arrays that have different dimensions, automatically aligning them to perform element-
wise operations.

5. Linear Algebra Operations: NumPy provides a comprehensive suite of linear algebra

functions, including matrix multiplication, matrix inversion, determinant calculation, eigenvalues
and eigenvectors, solving linear equations, and more. These operations are efficient and built on
top of highly optimized numerical libraries.

6. Integration with Python Ecosystem: NumPy seamlessly integrates with other popular Python
libraries such as SciPy (Scientific Python), pandas (data manipulation), Matplotlib (data
visualization), and scikit-learn (machine learning). Together, these libraries form a powerful
ecosystem for scientific computing and data analysis in Python.

To use NumPy in your Python code, you need to import the `numpy` module:
import numpy as np

After importing NumPy, you can create arrays, perform mathematical operations, access
elements, and leverage the various features and functionalities offered by NumPy.

Here's a simple example that demonstrates the basic usage of NumPy:

import numpy as np

# Create a 1-dimensional array

arr1d = [Link]([1, 2, 3, 4, 5])

# Create a 2-dimensional array

arr2d = [Link]([[1, 2, 3], [4, 5, 6]])

# Perform element-wise operations

result = arr1d + arr2d

# Access elements using indexing

print(result[2]) # Output: [4 6 8]

# Perform mathematical functions

mean = [Link](arr1d)
std = [Link](arr2d)

print(mean) # Output: 3.0

print(std) # Output: 1.707825127659933

# Use broadcasting for operations

arr3d = [Link]([[1, 2, 3]])
result = arr3d + 5
print(result) # Output: [[6 7 8]]

Mean ([Link]()):
Reflects the average value of the data.
Susceptible to outliers because it considers all data points equally.
Median ([Link]()):
Represents the middle value of a dataset when arranged in ascending order.
Less affected by outliers compared to the mean.
Useful for datasets with extreme values.
Standard Deviation ([Link]()):
Measures the dispersion or spread of data from its mean.
Provides a sense of how much the values deviate from the mean value.
Indicates the data's variability or consistency.

Q. How to generate random numbers using NumPy?

import numpy as np
random_array = [Link](3, 3) # Generate a 3x3 array of random numbers between 0
and 1
print(random_array)
Q. How to save and load NumPy arrays to/from a file?
my_array = [Link]([1, 2, 3, 4, 5])
[Link]('my_array.npy', my_array) # Save array to file

loaded_array = [Link]('my_array.npy') # Load array from file

print(loaded_array)

Q. Write a program to create array and perform operations on arrays like element-wise
arithmetic, slicing, reshaping, etc.
import numpy as np

# Element-wise arithmetic
a = [Link]([1, 2, 3])
b = [Link]([4, 5, 6])
print("Element-wise addition:")
print(a + b)

# Slicing
arr = [Link]([[1, 2, 3], [4, 5, 6], [7, 8, 9]])
print("\nSliced array:")
print(arr[:2, 1:])

# Reshaping
arr = [Link](1, 10) # Creates array [1, 2, 3, ..., 9]
reshaped_arr = [Link](3, 3)
print("\nReshaped array:")
print(reshaped_arr)

Indexing an NumPy Array

Indexing is used to extract individual elements from a one-dimensional array.
It can also be used to extract rows, columns, or planes in a multi-dimensional NumPy array.
Example: Index in NumPy array

Element 23 21 55 65 23

Index 0 1 2 3 4

In the above example, we have highlighted the element “55” which is at index “2”.
Indexing Using Index arrays
Indexing can be done in NumPy by using an array as an index.
Numpy arrays can be indexed with other arrays or any other sequence with the exception
of tuples. The last element is indexed by -1 second last by -2 and so on.
import numpy as np
 # Create a sequence of integers from 10 to 1 with a step of -2
a = [Link](10, 1, -2)
print("\n A sequential array with a negative step: \n",a)
# Indexes are specified inside the [Link] method.
newarr = a[[Link]([3, 1, 2 ])]
print("\n Elements at these indices are:\n",newarr)

Basic Slicing and indexing

Basic slicing and indexing is used to access a specific element or range of elements from a
NumPy array.
Basic slicing and indexing only return the view of the array.
Consider the syntax x[obj] where “x” is the array and “obj” is the index. The slice object is the
index in the case of basic slicing.
Basic slicing occurs when obj is :
1. A slice object that is of the form start: stop: step
2. An integer
3. Or a tuple of slice objects and integers
All arrays generated by basic slicing are always „view‟ of the original array.
Example: Basic Slicing in NumPy array
import numpy as np
# Arrange elements from 0 to 19
a = [Link](20)
print("\n Array is:\n ",a)
print("\n a[15]=",a[15])
# a[start:stop:step]
print("\n a[-8:17:1] = ",a[-8:17:1])
print("\n a[10:] = ",a[10:])

Q. Using boolean indexing on NumPy array to find numbers greater than 50

# You may wish to select numbers greater than 50
import numpy as np

a = [Link]([10, 40, 80, 50, 100])

print(a[a>50])

Q. Using boolean indexing on NumPy array to find numbers whose sum row is 10
# You may wish to select those elements whose
# sum of row is a multiple of 10.
import numpy as np

b = [Link]([[5, 5],[4, 5],[16, 4]])

sumrow = [Link](-1)
print(b[sumrow%10==0])
[Link]()
With the help of Numpy [Link](), We can perform the simple function of
transpose within one line by using [Link]() method of Numpy. It can transpose the
2-D arrays on the other hand it has no effect on 1-D arrays. This method transpose the 2-D
numpy array.

# importing python module named numpy

import numpy as np
# making a 3x3 array
gfg = [Link]([[1, 2, 3],
[4, 5, 6],
[7, 8, 9]])

# before transpose
print(gfg, end ='\n\n')

# after transpose
print([Link]())

O/P:
[[1 2 3]
[4 5 6]
[7 8 9]]

[[1 4 7]
[2 5 8]
[3 6 9]]

Parameters:
axes : [None, tuple of ints, or n ints] If anyone wants to pass the parameter then you can but
it’s not all required.
But if you want than remember only pass (0, 1) or (1, 0). Like we have array of shape (2, 3)
to change it (3, 2) you should pass (1, 0) where 1 as 3 and 0 as 2.
Returns: ndarray

# importing python module named numpy

import numpy as np
# making a 3x3 array
gfg = [Link]([[1, 2],
[4, 5],
[7, 8]])
# before transpose
print(gfg, end ='\n\n')

# after transpose
print([Link](1, 0))
O/P:
[[1 2]
[4 5]
[7 8]]

[[1 4 7]
[2 5 8]]

This is just a brief introduction to NumPy. NumPy offers many more advanced features, such as
advanced indexing, reshaping, sorting, and statistical functions. You can refer to the official
NumPy documentation ([Link] for detailed information and examples on using
NumPy for various scientific computing tasks.

8.2 Arrays and Scalars

To create arrays and use them along with scalars, you can follow the guidelines specific to the
programming language you are working with. Here, I'll provide examples using Python, which
has built-in support for arrays through the NumPy library.

First, you need to install NumPy if you haven't already. You can do this by running the following
command in your Python environment:
pip install numpy

Once NumPy is installed, you can import it in your Python script to use its array functionality:
import numpy as np

Now, let's see how to create arrays, perform operations with scalars, and manipulate array
elements:

1. Creating arrays:
- Creating an array from a list:
arr1 = [Link]([1, 2, 3, 4, 5])
print(arr1) # Output: [1 2 3 4 5]

- Creating a 2D array (matrix) from nested lists:

arr2 = [Link]([[1, 2, 3], [4, 5, 6]])
print(arr2)
# Output:
# [[1 2 3]
# [4 5 6]]
- Creating an array of zeros:
zeros_arr = [Link]((3, 4)) # Creates a 3x4 array filled with zeros
print(zeros_arr)
# Output:
 # [[0. 0. 0. 0.]
# [0. 0. 0. 0.]
# [0. 0. 0. 0.]]

2. Using arrays with scalars:

- Scalar addition:
arr3 = [Link]([1, 2, 3])
scalar = 5
result = arr3 + scalar
print(result) # Output: [6 7 8]

- Scalar multiplication:
arr4 = [Link]([1, 2, 3])
scalar = 2
result = arr4 * scalar
print(result) # Output: [2 4 6]

3. Manipulating array elements:

- Accessing elements by index:
arr5 = [Link]([1, 2, 3, 4, 5])
print(arr5[0]) # Output: 1
print(arr5[2:4]) # Output: [3 4]

- Modifying elements by index:

arr6 = [Link]([1, 2, 3, 4, 5])
arr6[0] = 10
print(arr6) # Output: [10 2 3 4 5]

- Performing operations on array elements:

arr7 = [Link]([1, 2, 3, 4, 5])
result = arr7 ** 2 # Squaring each element
print(result) # Output: [ 1 4 9 16 25]

Creating Scalars:
Scalars in NumPy are represented using basic Python types like int or float.
import numpy as np
scalar_int = np.int32(5)
scalar_float = np.float64(3.14)

print("Integer Scalar:", scalar_int)

print("Float Scalar:", scalar_float)

These examples demonstrate basic operations with arrays and scalars using the NumPy library in
Python. Remember to adapt the syntax according to the programming language you are using, as
array functionality and syntax may differ.
8.3 Universal Array Function
Universal functions (ufuncs) in NumPy are functions that operate element-wise on arrays,
applying the same operation to each element of the array. They are designed to efficiently handle
large arrays without the need for explicit loops. Here are some commonly used universal array
functions in NumPy:

1. `[Link](x)` - Compute the absolute value of each element in the array `x`.

2. `[Link](x)` - Compute the square root of each element in the array `x`.

3. `[Link](x)` - Compute the exponential (e^x) of each element in the array `x`.

4. `[Link](x)` - Compute the natural logarithm of each element in the array `x`.

5. `[Link](x)`, `[Link](x)`, `[Link](x)` - Compute the sine, cosine, and tangent of each element in
the array `x`, respectively. Similarly, there are trigonometric functions like `[Link]()`,
`[Link]()`, and `[Link]()` to compute inverse trigonometric values.

6. `[Link](x)`, `[Link](x)`, `[Link](x)`, `[Link](x)`, `[Link](x)` - Compute the sum, mean,

standard deviation, minimum, and maximum values of the elements in the array `x`, respectively.
These functions can also operate along a specific axis by specifying the `axis` parameter.

7. `[Link](x, y)`, `[Link](x, y)`, `[Link](x, y)`, `[Link](x, y)` - Perform element-
wise addition, subtraction, multiplication, and division between arrays `x` and `y`.

8. `[Link](x, y)` - Compute the exponentiation of each element in array `x` with the
corresponding element in array `y`.

9. `[Link](x)`, `[Link](x)`, `[Link](x)` - Compute the floor, ceiling, and rounding of each
element in the array `x`.

These are just a few examples of universal array functions in NumPy. There are many more
functions available to perform various mathematical operations on arrays efficiently. You can
explore the official NumPy documentation for a comprehensive list of available ufuncs and their
usage.

8.4 Array Input and Output

When working with arrays, it is often necessary to save them to a file or load them from a file for
later use. NumPy provides functions for input and output (I/O) operations to facilitate this
process. Here, I'll explain how to save and load arrays using NumPy in Python.

1. Saving Arrays:
To save an array to a file, you can use the `[Link]()` or `[Link]()` function. Here's how you
can use them:

- `[Link](file, arr)` - Saves a single array to a binary file with the `.npy` extension.
import numpy as np
arr = [Link]([1, 2, 3, 4, 5])
[Link]('[Link]', arr)

- `[Link](file, arr1, arr2, ...)` - Saves multiple arrays into a single compressed `.npz` file.
import numpy as np
arr1 = [Link]([1, 2, 3])
arr2 = [Link]([4, 5, 6])
[Link]('[Link]', arr1=arr1, arr2=arr2)

2. Loading Arrays:
To load arrays from files, you can use the `[Link]()` or `[Link]()` function. Here's how to use
them:

- `[Link](file)` - Loads a single array from a `.npy` file.

import numpy as np
arr = [Link]('[Link]')
print(arr) # Output: [1 2 3 4 5]

- `[Link](file)` - Loads multiple arrays from a `.npz` file. The returned object works as a
dictionary-like structure.
import numpy as np
data = [Link]('[Link]')
arr1 = data['arr1']
arr2 = data['arr2']
print(arr1) # Output: [1 2 3]
print(arr2) # Output: [4 5 6]

The saved arrays can be loaded into the program at a later time using these functions. It's
important to note that the file extensions `.npy` and `.npz` are used to differentiate between
single and multiple array files.

Additionally, NumPy provides other I/O functions such as `[Link]()` and `[Link]()` for
saving and loading arrays in plain text format, `[Link]()` for loading arrays from
delimited text files, and more. These functions offer various options for customizing the output
format and loading data with specific configurations.
Remember to import the NumPy library (`import numpy as np`) before using these functions.

8.5 Pandas- What are pandas? Where it is used?

Pandas is an open-source Python library that provides powerful data manipulation and analysis
tools. It is built on top of NumPy and is widely used in data science, data analysis, and machine
learning applications. Pandas introduces two key data structures: Series and DataFrame.
1. Series:
A Series is a one-dimensional labeled array that can hold any data type. It is similar to a
column in a spreadsheet or a single column of a database table. The Series object consists of two
main components: the data (values) and the index (labels). The index provides a label for each
element in the Series, allowing for easy and efficient data access and alignment.

2. DataFrame:
A DataFrame is a two-dimensional labeled data structure, resembling a table or a spreadsheet.
It consists of rows and columns, where each column can hold different types of data. DataFrames
are highly versatile and provide powerful functionalities for data manipulation, cleaning,
reshaping, grouping, merging, and more. They allow for efficient handling of structured data and
are commonly used for data analysis and preprocessing tasks.

Pandas provides a wide range of functions and methods to perform various operations on Series
and DataFrame objects. Some of the key features and use cases of Pandas include:

- Data cleaning and preprocessing: Pandas offers functions to handle missing values, duplicate
data, and outliers, as well as tools for data transformation, normalization, and feature
engineering.

- Data exploration and analysis: Pandas provides efficient methods for descriptive statistics,
aggregations, filtering, sorting, and visualizations. It allows you to gain insights into the data,
identify patterns, and extract meaningful information.

- Data integration and merging: Pandas enables combining multiple datasets based on common
columns or indices, performing joins, concatenation, and merging operations.

- Time series analysis: Pandas has extensive support for working with time series data, including
date/time indexing, resampling, shifting, and rolling window calculations.

- Input and output: Pandas can read and write data in various file formats, including CSV, Excel,
SQL databases, and more. It simplifies the process of loading and saving data from different
sources.

Pandas is widely used in industries such as finance, healthcare, marketing, social sciences, and
more. It provides a high-level, intuitive interface for data manipulation and analysis, making it a
popular choice among data scientists, analysts, and researchers. Its integration with other
libraries like NumPy, Matplotlib, and scikit-learn further enhances its capabilities in data
analysis workflows.

8.6 Series in pandas, pandas DataFrames, Index objects, ReIndex

In pandas, a Series is a one-dimensional labeled array that can hold any data type, similar to a
column in a spreadsheet or a single column of a database table. It consists of two primary
components: the data (values) and the index. The index provides labels for each element in the
Series, allowing for easy and efficient data access and alignment.

Here's an example of creating a Series in pandas:

import pandas as pd

# Creating a Series from a list

data = [10, 20, 30, 40, 50]
s = [Link](data)
print(s)
Output:
0 10
1 20
2 30
3 40
4 50
dtype: int64

In this example, a Series object `s` is created from a Python list `data`. The default index is
assigned, starting from 0, and the values from the list are stored in the Series.

Pandas DataFrames, on the other hand, are two-dimensional labeled data structures resembling
tables or spreadsheets. A DataFrame consists of rows and columns, where each column can hold
different types of data. It provides a tabular structure with additional functionalities for data
manipulation, analysis, and integration.

Here's an example of creating a DataFrame in pandas:

import pandas as pd

# Creating a DataFrame from a dictionary

data = {'Name': ['John', 'Emma', 'Michael'],
'Age': [25, 30, 35],
'City': ['New York', 'London', 'Paris']}
df = [Link](data)
print(df)
Output:
Name Age City
0 John 25 New York
1 Emma 30 London
2 Michael 35 Paris

In this example, a DataFrame `df` is created from a dictionary `data`. The keys of the dictionary
represent column names, and the values are the corresponding data. The DataFrame is structured
with rows and columns, and each column holds a different type of data.

Index objects in pandas are immutable arrays that hold the axis labels (row labels or column
labels) for Series and DataFrames. Index objects provide several functionalities for data
alignment, selection, and reindexing.

Reindexing is the process of creating a new object with a different index. It allows you to change
the row or column labels of a Series or DataFrame, aligning the data with the new index labels.
The `reindex()` method in pandas facilitates this operation.

Here's an example of reindexing a DataFrame:

import pandas as pd

data = {'Name': ['John', 'Emma', 'Michael'],

'Age': [25, 30, 35],
'City': ['New York', 'London', 'Paris']}
df = [Link](data)

# Reindexing the DataFrame

new_index = ['A', 'B', 'C']
df_reindexed = [Link](new_index)
print(df_reindexed)
Output:
Name Age City
A John 25.0 New York
B Emma 30.0 London
C Michael 35.0 Paris

In this example, the original DataFrame `df` is reindexed with the labels `['A', 'B', 'C']`. The
`reindex()` method creates a new DataFrame `df_reindexed` with the specified index labels, and
the data is aligned accordingly.

Reindexing allows you to handle missing values, align data with different datasets, or change the
order of the rows/columns in a DataFrame.

8.7 Drop Entry, Selecting Entries

In pandas, you can drop entries (rows or columns) from a DataFrame using the `drop()` method.
To select specific entries from a DataFrame, you can use indexing and slicing techniques.

1. Dropping Entries:
To drop entries from a DataFrame, you can use the `drop()` method and specify the labels of
the rows or columns you want to remove. By default, `drop()` removes rows, but you can also
drop columns by specifying the `axis` parameter.

Here's an example of dropping entries from a DataFrame:

import pandas as pd

# Creating a DataFrame
data = {'Name': ['John', 'Emma', 'Michael'],
 'Age': [25, 30, 35],
'City': ['New York', 'London', 'Paris']}
df = [Link](data)

# Dropping entries (rows)

df_dropped = [Link]([1]) # Drop the row with index 1
print(df_dropped)

Output:
Name Age City
0 John 25 New York
2 Michael 35 Paris

In this example, the row with index 1 is dropped using the `drop()` method. The resulting
DataFrame `df_dropped` no longer contains the dropped row.

If you want to drop columns, you need to specify the `axis` parameter as 1:
df_dropped = [Link](['Age', 'City'], axis=1) # Drop columns 'Age' and 'City'

2. Selecting Entries:
To select specific entries (rows and columns) from a DataFrame, you can use indexing and
slicing techniques.
- Selecting rows by index label or index position:
# Selecting rows by index label
row = [Link][1] # Get the row with index label 1

# Selecting rows by index position

row = [Link][1] # Get the row at index position 1

- Selecting columns by column name or column position:

# Selecting columns by column name
column = df['Age'] # Get the 'Age' column

# Selecting columns by column position

column = [Link][:, 1] # Get the column at position 1

- Selecting specific entries using both row and column selection:

entry = [Link][1, 'Age'] # Get the value at row with index 1 and column 'Age'

These are just a few examples of selecting entries from a DataFrame. Pandas provides various
indexing and slicing techniques to select specific rows, columns, or individual entries based on
labels or positions.

Keep in mind that indexing in pandas starts from 0, and you can also use boolean indexing and
other advanced selection methods for more complex selection operations.
8.8 Data Alignment, Rank and Sort
Data alignment, ranking, and sorting are important operations in pandas that allow you to align
data, assign ranks to values, and sort the data based on specific criteria. Let's go through each of
these operations:

1. Data Alignment:
Data alignment in pandas refers to the process of aligning data based on index labels. When
performing operations on multiple Series or DataFrames, pandas automatically aligns the data
based on their index labels, ensuring that corresponding elements are matched together.

Here's an example of data alignment:

import pandas as pd

# Creating two Series objects

s1 = [Link]([1, 2, 3], index=['A', 'B', 'C'])
s2 = [Link]([4, 5, 6], index=['B', 'C', 'D'])

# Adding the two Series

result = s1 + s2
print(result)
Output:
A NaN
B 6.0
C 8.0
D NaN
dtype: float64
In this example, when adding the two Series `s1` and `s2`, pandas aligns the data based on
index labels. The resulting Series `result` contains the addition of values where the index labels
match, and `NaN` (Not a Number) is assigned where the index labels do not match.

2. Rank:
The rank method in pandas assigns ranks to values within a Series or DataFrame based on their
numerical order. The `rank()` function is used to perform this operation.
import pandas as pd

# Creating a Series
s = [Link]([3, 1, 2, 4, 5])

# Ranking the values

ranks = [Link]()
print(ranks)
Output:
0 3.0
1 1.0
2 2.0
 3 4.0
4 5.0
dtype: float64

In this example, the `rank()` function assigns ranks to the values in the Series `s`. The smallest
value (1) receives the lowest rank, while the largest value (5) receives the highest rank.

3. Sorting:
Sorting in pandas allows you to sort the data in a Series or DataFrame based on specific
criteria, such as index labels or column values. The `sort_values()` and `sort_index()` functions
are commonly used for sorting.
import pandas as pd

# Creating a Series
s = [Link]([3, 1, 2, 4, 5])

# Sorting by values
sorted_values = s.sort_values()
print(sorted_values)

# Sorting by index
sorted_index = s.sort_index()
print(sorted_index)
Output:
1 1
2 2
0 3
3 4
4 5
dtype: int64

0 3
1 1
2 2
3 4
4 5

In this example, `sort_values()` sorts the Series `s` based on its values in ascending order,
while `sort_index()` sorts the Series based on its index labels in ascending order.

For DataFrames, you can specify the axis parameter (`axis=0` for rows, `axis=1` for columns) to
control the sorting direction.

These operations are fundamental in data manipulation and analysis using pandas. They help in
aligning data, assigning ranks, and sorting the data to extract meaningful insights and perform
further computations.
8.9 Summary Statics, Missing Data, Index Hierarchy
Summary Statistics, Missing Data, and Index Hierarchy are important concepts in pandas for
data analysis and manipulation. Let's explore each of these concepts:

1. Summary Statistics:
Pandas provides several functions to compute summary statistics for numerical data in a
DataFrame or Series. These functions allow you to gain insights into the data distribution, central
tendency, dispersion, and other statistical measures.

Here are some common summary statistics functions in pandas:

- `mean()`: Compute the mean (average) of the data.
- `median()`: Compute the median of the data.
- `sum()`: Compute the sum of the data.
- `min()`: Find the minimum value in the data.
- `max()`: Find the maximum value in the data.
- `std()`: Compute the standard deviation of the data.
- `var()`: Compute the variance of the data.
- `describe()`: Generate various descriptive statistics of the data, including count, mean,
standard deviation, minimum, quartiles, and maximum.

Here's an example of computing summary statistics for a DataFrame:

import pandas as pd

# Creating a DataFrame
data = {'Name': ['John', 'Emma', 'Michael'],
'Age': [25, 30, 35],
'Salary': [50000, 60000, 70000]}
df = [Link](data)

# Computing summary statistics

print([Link]()) # Compute the mean of each column
print([Link]()) # Generate descriptive statistics
Output:
Age 30.000000
Salary 60000.000000
dtype: float64

Age Salary
count 3.000000 3.000000
mean 30.000000 60000.000000
std 5.000000 10000.000000
min 25.000000 50000.000000
25% 27.500000 55000.000000
50% 30.000000 60000.000000
 75% 32.500000 65000.000000
max 35.000000 70000.000000

In this example, summary statistics such as mean and descriptive statistics are computed for the
numerical columns 'Age' and 'Salary' in the DataFrame `df`.

2. Missing Data:
Missing data refers to the presence of null or NaN (Not a Number) values in a DataFrame or
Series. Pandas provides functions to handle missing data, including identifying missing values,
removing or filling missing values, and handling missing data in computations.

Here are some functions for handling missing data in pandas:

- `isnull()`: Check if values are missing (NaN) in a DataFrame or Series.
- `notnull()`: Check if values are not missing in a DataFrame or Series.
- `dropna()`: Remove rows or columns with missing values.
- `fillna()`: Fill missing values with a specified value or using interpolation methods.
- `interpolate()`: Interpolate missing values based on different methods, such as linear
interpolation.

Here's an example of handling missing data in a DataFrame:

import pandas as pd
import numpy as np

# Creating a DataFrame with missing values

data = {'A': [1, [Link], 3, 4],
'B': [5, 6, [Link], 8],
'C': [9, 10, 11, [Link]]}
df = [Link](data)

# Handling missing data

print([Link]()) # Check for missing values
print([Link]()) # Remove rows with missing values
print([Link](0)) # Fill missing values with 0
Output:
A B C
0 False False False
1 True False False
2 False True False
3 False False True

A B C
0 1.0 5.0 9.0

A B C
0 1.0 5.0 9.0
1 0.0 6.0 10.0
 2 3.0 0.0 11.0
3 4.0 8.0 0.0
In this example, missing values are identified using `isnull()`. Rows with missing values are
removed using `dropna()`. Missing values are filled with 0 using `fillna()`.

3. Index Hierarchy:
Index hierarchy, also known as MultiIndex, allows you to have multiple levels of index in a
DataFrame. It provides a way to represent higher-dimensional data in a tabular structure. With
index hierarchy, you can perform advanced data selection, grouping, and analysis.

Here's an example of creating a DataFrame with index hierarchy:

import pandas as pd

# Creating a DataFrame with index hierarchy

data = {'Value': [10, 20, 30, 40, 50],
'Category': ['A', 'B', 'A', 'B', 'A'],
'Subcategory': ['X', 'Y', 'X', 'Y', 'Z']}
df = [Link](data)
multi_index = [Link].from_arrays([df['Category'], df['Subcategory']],
names=['Category', 'Subcategory'])
df.set_index(multi_index, inplace=True)
[Link](['Category', 'Subcategory'], axis=1, inplace=True)
print(df)
Output:
Value
Category Subcategory
A X 10
B Y 20
A X 30
B Y 40
A Z 50

In this example, a DataFrame `df` is created with two levels of index hierarchy: 'Category' and
'Subcategory'. The `MultiIndex` object is created from the columns 'Category' and 'Subcategory',
and the DataFrame is set with this multi-index using `set_index()`. The columns 'Category' and
'Subcategory' are dropped using `drop()` to achieve the desired index hierarchy.

Index hierarchy allows for more advanced data manipulation, slicing, and analysis by
leveraging multiple levels of indexing.

These concepts in pandas provide powerful tools to perform data analysis, handle missing data,
compute summary statistics, and work with multi-level index structures. They are essential for
data cleaning, exploratory data analysis, and extracting insights from structured datasets.

8.10 Python for Data Visualization - Matplotlib , Visualization Tools

Matplotlib is a popular plotting library in Python that provides a wide range of tools for creating
visualizations. It is widely used for generating charts, graphs, histograms, scatter plots, and many
other types of plots to explore and present data.

Here's a brief overview of how to use Matplotlib:

1. Importing Matplotlib:
Before using Matplotlib, you need to import it. Typically, Matplotlib is imported under
the alias `plt`.
import [Link] as plt

2. Basic Line Plot:

A basic line plot can be created using the `plot()` function in Matplotlib. You provide the x-
axis values and corresponding y-axis values as arguments to the function.

import [Link] as plt

x = [1, 2, 3, 4, 5]
y = [2, 4, 6, 8, 10]

[Link](x, y)
[Link]()

This will display a simple line plot with the x-axis values `[1, 2, 3, 4, 5]` and the y-axis values
`[2, 4, 6, 8, 10]`.

3. Customizing the Plot:

Matplotlib provides a wide range of options to customize the appearance of plots. You can add
labels, titles, legends, change line styles, colors, markers, and more.

import [Link] as plt

x = [1, 2, 3, 4, 5]
y = [2, 4, 6, 8, 10]

[Link](x, y, linestyle='--', marker='o', color='r')

[Link]('X-axis')
[Link]('Y-axis')
[Link]('Line Plot')
[Link](True)
[Link]()

In this example, the line style is set to a dashed line (`linestyle='--'`), the marker style is set to a
circle (`marker='o'`), and the color is set to red (`color='r'`). The x-axis label, y-axis label, and
plot title are added using `xlabel()`, `ylabel()`, and `title()` functions, respectively. The `grid()`
function adds a grid to the plot.

4. Other Types of Plots:

 Matplotlib supports various types of plots, including scatter plots, bar plots, histogram plots,
pie charts, and more. You can explore the Matplotlib documentation and examples to learn more
about each plot type and its specific customization options.

import [Link] as plt

# Scatter plot
x = [1, 2, 3, 4, 5]
y = [2, 4, 6, 8, 10]
[Link](x, y)
[Link]()

# Bar plot
x = ['A', 'B', 'C', 'D']
y = [3, 7, 2, 5]
[Link](x, y)
[Link]()

# Histogram
data = [1, 2, 3, 3, 4, 5, 5, 5, 6, 7]
[Link](data, bins=5)
[Link]()

# Pie chart
sizes = [15, 30, 45, 10]
labels = ['A', 'B', 'C', 'D']
[Link](sizes, labels=labels, autopct='%1.1f%%')
[Link]()
These examples demonstrate how to create scatter plots, bar plots, histograms, and pie charts
using Matplotlib. Each plot type has its own set of customization options.

Matplotlib offers extensive flexibility and control over plot customization, allowing you to create
visually appealing and informative plots. You can refer to the Matplotlib documentation and
examples for more in-depth guidance on specific plot types, customization options, and advanced
plotting techniques.

Solved Programs
Q.1 Create 5×5 2D numpy assay and retrieve top left corner 2×2 array fromit.[2marks]
Ans:
import numpy as np
x = [Link]((5,5),7)
print("Original Array:")
print(x)
print(x[0:2,1:3])
Q.2 Create pandas dataframe using two dimensional list. Perform following operations.
Count number of [Link] missing values in first column. Display number of columns in data
frame.
Ans:
import pandas as pd# List1
lst = [['tom', 'reacher', 25], ['krish', 'pete', 30],
['nick', 'wilson', 26], ['juli', 'williams', 22]]

df = [Link](lst, columns =['FName', 'LName', 'Age'])

print(df)
#Count number of rows:
print('Row count is:', len(df))

#Count missing values in first column:

[Link]().sum()

#Display number of columns in data frame.

# Getting the list of columns
col = [Link]
print('Number of columns :', len(col))

Q.3 Create a series from numpy array and find max and mean of unique itemsof series.
import pandas as pd

# Creating the Series

sr = [Link]([10, 25, 3, 25, 24, 6])

# Create the Index

index_ = ['Coca Cola', 'Sprite', 'Coke', 'Fanta', 'Dew', 'ThumbsUp']

# set the index

[Link] =
index_result =
[Link]()

# Print the result

print(result)

Excercizes
Q.1 Write a Python program to create a time series plot using Pandas and Matplotlib.
Q.2 Write a Python program to create multiple subplots using Pandas and Matplotlib.
Q.3 Write a Python program to customize various aspects of a plot such as colors, labels, titles,
legends, etc., using Pandas and Matplotlib
Q.4 Write a Python program to save a plot as an image file (e.g., PNG, JPEG) using Pandas and
Matplotlib.
Q.5 Write a Python program to create an interactive plot using Pandas, Matplotlib, and an
interactive backend like Plotly or Bokeh.