Scribd
Upload
13 views11 pages

Python Data Science: NumPy & Pandas Guide

The document provides a comprehensive guide on using Python for data science, focusing on the installation of libraries like NumPy and Pandas, as well as practical examples of working with arrays and data frames. It covers various operations such as reading CSV files, performing calculations like mean and variance, and visualizing data with scatter plots. The document also includes specific examples using datasets like the Iris and Pima Indian Diabetes datasets.

Uploaded by

Suthiksha
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd
13 views11 pages

Python Data Science: NumPy & Pandas Guide

The document provides a comprehensive guide on using Python for data science, focusing on the installation of libraries like NumPy and Pandas, as well as practical examples of working with arrays and data frames. It covers various operations such as reading CSV files, performing calculations like mean and variance, and visualizing data with scatter plots. The document also includes specific examples using datasets like the Iris and Pima Indian Diabetes datasets.

Uploaded by

Suthiksha
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd

Data science with python

Install inside script folder

In command prompt go to the installation path of python

C:\Program Files\Python311\Scripts

Type

pip install numpy


to install numpy

pip install SciPy


to install SciPy

pip install pandas


to install pandas
2. working with numpy arrays

import numpy as np

arr = [Link]([1, 2, 3, 4, 5])

print(arr)

print(type(arr))

import numpy as np

arr = [Link]((1, 2, 3, 4, 5))

print(arr)

import numpy as np

arr = [Link](42)

print(arr)

1D array

import numpy as np

arr = [Link]([1, 2, 3, 4, 5])

print(arr)

2D array

import numpy as np
arr = [Link]([[1, 2, 3], [4, 5, 6]])
print(arr)
3D array

import numpy as np

arr = [Link]([[[1, 2, 3], [4, 5, 6]], [[1, 2, 3], [4, 5, 6]]])

print(arr)

check no of dimensions

import numpy as np

a = [Link](42)
b = [Link]([1, 2, 3, 4, 5])
c = [Link]([[1, 2, 3], [4, 5, 6]])
d = [Link]([[[1, 2, 3], [4, 5, 6]], [[1, 2, 3], [4, 5, 6]]])

print([Link])
print([Link])
print([Link])
print([Link])

PANDAS
import pandas as pd
df = pd.read_csv('[Link]')
print(df.to_string())
What Can Pandas Do?

Pandas gives you answers about the data. Like:

 Is there a correlation between two or more columns?


 What is average value?
 Max value?
 Min value?

Pandas are also able to delete rows that are not relevant, or contains wrong values, like empty or NULL values. This is called cleaning the data.

import pandas as pd
mydataset = {
'cars': ["BMW", "Volvo", "Ford"],
'passings': [3, 7, 2]
}
myvar = [Link](mydataset)

print(myvar)

pandas version

import pandas as pd

print(pd.__version__)

What is a Series?

A Pandas Series is like a column in a table.

It is a one-dimensional array holding data of any type.

import pandas as pd

a = [1, 7, 2]

myvar = [Link](a)

print(myvar)

Create Labels

With the index argument, you can name your own labels.

import pandas as pd

a = [1, 7, 2]

myvar = [Link](a, index = ["x", "y", "z"])

print(myvar)

Key/Value Objects as Series

You can also use a key/value object, like a dictionary, when creating a Series.
import pandas as pd

calories = {"day1": 420, "day2": 380, "day3": 390}

myvar = [Link](calories)

print(myvar)

To select only some of the items in the dictionary, use the index argument and specify only the items you want to include in the Series.

import pandas as pd

calories = {"day1": 420, "day2": 380, "day3": 390}

myvar = [Link](calories, index = ["day1", "day2"])

print(myvar)

DataFrames

Data sets in Pandas are usually multi-dimensional tables, called DataFrames.

Series is like a column, a DataFrame is the whole table.

Create a DataFrame from two Series:

import pandas as pd

data = {
"calories": [420, 380, 390],
"duration": [50, 40, 45]
}

myvar = [Link](data)

print(myvar)

Iris dataset
import pandas as pd
import numpy as np
df = pd.read_csv("iris_csv.csv")
print([Link]())

10 records
import pandas as pd
import numpy as np

df = pd.read_csv("iris_csv.csv")
print([Link](10))

Column title

import pandas as pd
import numpy as np

df = pd.read_csv("iris_csv.csv")
print([Link])

total rows columns

import pandas as pd
import numpy as np

df = pd.read_csv("iris_csv.csv")
print([Link])

Display the whole dataset

import pandas as pd
import numpy as np

df = pd.read_csv("iris_csv.csv")
print(df)

Slicing the rows.

import pandas as pd
import numpy as np

df = pd.read_csv("iris_csv.csv")
print(df[0:10])

or

import pandas as pd
import numpy as np

df = pd.read_csv("iris_csv.csv")
print(df[0:10])

sliced_data=df[10:21]
print(sliced_data)

Displaying only specific columns.


import pandas as pd
import numpy as np

df = pd.read_csv("iris_csv.csv")

specific_data=df[["sepalwidth","class"]]
print(specific_data.head(10))

Mean value calculation

___________________

method 1

import pandas as pd

import numpy as np

df = pd.read_csv("iris_csv.csv")

df2=df["sepalwidth"]

print("The sum of all sepalwidthn values : ", [Link]())

sum=[Link]()

print("Total number of rows in sepalwidth : ",[Link])

column=[Link]

print("The average of all sepalwidth values : ",sum/150)

method 2

import pandas as pd

import numpy as np
df = pd.read_csv("iris_csv.csv")

df1=df[["sepalwidth"]]

print("the sum of:",[Link]())

sum = [Link]()

total = [Link]()

print(total)

avg = sum/total

print("the average:",avg)

method 3

import pandas as pd
import numpy as np
df = pd.read_csv("iris_csv.csv")
sum=df["sepalwidth"].sum()
print("sum",sum)
mean=df["sepalwidth"].mean()
print("mean",mean)
median=df["sepalwidth"].median()
print("median",median)

pima dataset

import pandas as pd
import numpy as np
df = pd.read_csv("pima-indians-diabetes(2).csv")
data1=df["pregnancies"].sum()
print("sum of pregnancies:",data1)
data2=df["pregnancies"].mean()
print("Mean of pregnancies:",data2)
data3=df["pregnancies"].median()
print("Mean of pregnancies:",data3)
data4=df["pregnancies"].value_counts()
print("Frequencies of pregnancies:\n",data4)

data5=df["pregnancies"]. mode()

print(data5)

pima dataset

import pandas as pd
import numpy as np
df=pd.read_csv("[Link]")
d1=df["blood pressure"].skew()
print("Skewness=",d1)
d2=df["blood pressure"].kurtosis()
print("Kurtosis=",d2)
d3=[Link](df["blood pressure"])
print("Variance=",d3)

import pandas as pd
import numpy as np
import math
df = pd.read_csv("[Link]")
data1 = df["pregnancies"].sum()
print("Sum of pregnancies:", data1)
data2 = df["pregnancies"].mean()
print("Mean of pregnancies:", data2)
data3 = df["pregnancies"].median()
print("Median of pregnancies:", data3)
data4 = df["pregnancies"].mode()
print("Mode of pregnancies:", data4)
data5 = [Link](df["pregnancies"])
print("Variance of pregnancies:", data5)
data6 = df["pregnancies"]
var = sum((x - data2) ** 2 for x in data6) / len(data6)
print("Variance of pregnancies (manual calculation):", var)
data7=[Link](df["pregnancies"])
print("standard dev usin fun ",data7)
data8=(var)**0.5
print("satndard dev manual cal",data8)
data9=df["pregnancies"].skew()
print("skewness using fun",data9)
data10=(3*(data2-data3))/data8
print("skewness manual cal",data10)
#The expected output should now show that data9 and data10 have matching
values for skewness.
#These values will be close but may not match exactly due to rounding dif-
ferences.
#They should be very close, indicating that both methods are working cor-
rectly.

import pandas as pd
import [Link] as plt
df=pd.read_csv("[Link]")
df1=df["insulin"]
df2=df["bp"]
x=df1
y=df2
[Link](x, y)
[Link]()

VARIANCE USING FORMULA

import pandas as pd
import statistics
import numpy as np
df = pd.read_csv("[Link]")
df1=df["bp"]
length = int(len(df1))
mean = sum(df1) / length
ans = sum((i - mean) ** 2 for i in df1) / length
print("The variance of BP is : " + str(ans))

STANDARD DEVIATION

import pandas as pd
import statistics
import numpy as np
df = pd.read_csv("[Link]")
df1=df["bp"]
i = [Link](df1)
print("Standard Deviation of the BP:",i)

SKEWNESS

from [Link] import skew


import pandas as pd
import statistics
import numpy as np
df = pd.read_csv("[Link]")
df1=df["bp"]
print ("df1 : \n", df1)
print("Skewness for BP : ", skew(df1))

KURTOSIS

import pandas as pd
import statistics
import numpy as np
df = pd.read_csv("[Link]")
df1=df["bp"]
print(df1)
result = [Link]()
print("kurtosis",result)

Common questions

Powered by AI

Variance is calculated in Python using both direct formulae and library functions. For instance, directly using the formula involves subtracting the mean from each data point, squaring the result, summing these squared differences, and dividing by the number of data points . Alternatively, a function like `np.var()` can be used. Understanding variance is critical as it quantifies the degree to which data points differ from the mean, highlighting data consistency and variability which are essential in assessing the reliability of data and predicting future trends .

DataFrames in Pandas are multi-dimensional data structures akin to data tables with rows and columns, enabling complex data manipulation and analysis . Unlike Series, which are one-dimensional and similar to single columns, DataFrames can be seen as a collection of Series, allowing for relationships between data variables to be handled within a single structure. DataFrames provide enhanced capability for structured data operations and are thus more suitable for comprehensive datasets that require extensive adjustments or analyses.

Plotting relationships between data variables in Python can be accomplished using libraries like Matplotlib, as demonstrated with scatter plots in `plt.scatter(x, y)`, where `x` and `y` are different variables . Such plots can reveal correlations, trends, or outliers within the data, providing a visual understanding of how variables interact and aiding in constructing hypotheses or further statistical testing to support data findings.

The number of dimensions in a NumPy array is checked using the `.ndim` attribute. For example, if `a = np.array(42)` then `a.ndim` will return 0 indicating a scalar. `b = np.array([1, 2, 3, 4, 5])` results in `b.ndim` returning 1 indicating a one-dimensional array. Similarly, `c = np.array([[1, 2, 3], [4, 5, 6]])` returns 2 and `d = np.array([[[1, 2, 3], [4, 5, 6]], [[1, 2, 3], [4, 5, 6]]])` returns 3 indicating a three-dimensional array .

Slicing techniques in Pandas allow the extraction of specific parts of a dataset by specifying the range of rows or the subset of columns desired. For instance, `df[0:10]` extracts the first 10 rows while `df[['sepalwidth','class']]` selects those specific columns . These techniques facilitate focused analysis by tailoring the view of data to relevant aspects, enhancing efficiency during exploratory data analysis or when preparing data features for machine learning models.

Skewness measures the asymmetry of a data distribution. A skewness of zero indicates a symmetrical distribution, while negative or positive values indicate left or right skew respectively . Skewness affects data interpretation by revealing potential bias or non-normality which can affect statistical analyses assumptions like regression or hypothesis tests which often assume normal distribution. Recognizing skewness helps in deciding on data normalization or transformation strategies to improve analysis accuracy.

Initial steps when examining datasets like the Iris dataset include loading the dataset properly using commands like `pd.read_csv()` and then performing basic inspections such as `df.head()` to understand data shape and contents, checking column titles via `df.columns`, and determining data dimensions using `df.shape` . These steps are crucial for identifying structure, understanding variable types, and planning any necessary preprocessing like handling missing values or normalization, ensuring that subsequent analysis can be carried out effectively and accurately, minimizing errors or biases in the data.

Pandas provides tools to handle missing data by enabling users to delete rows that contain NULL or inappropriate values, which is referred to as 'cleaning the data' . This is important for data analysis as missing or incorrect data can lead to misleading results, so ensuring data accuracy and completeness is crucial for generating reliable insights.

The mean is calculated by summing all values and dividing by the count, offering insight into the dataset's central tendency . The median is the middle value when data points are ordered, providing a better measure of central tendency in skewed datasets . The mode is the most frequently occurring value, useful for understanding the commonality within a dataset . Each of these measures provides different insights, such as sensitivity to outliers (mean) or the shape of the data distribution (mode and median).

Using key/value objects like dictionaries to create Pandas Series allows for intuitive data handling and labeling, as each key becomes an index, which can make data access and analysis more precise and intelligible . For example, in `calories = {'day1': 420, 'day2': 380, 'day3': 390}`, creating a Series with `pd.Series(calories)` results in a structured data form that is easy to access and manipulate .

You might also like