DATA SCIENCE WITH PYTHON

UNIT-1:

DESCRIPTIVE STATISTICS

[Link] of Central Tendency

In data science, measures of central tendency help summarize a dataset by identifying the
central point around which the data is clustered. These measures are essential in understanding
the distribution and the characteristics of the dataset. The common measures of central
tendency are mean, median, and mode.

1. Mean (Arithmetic Mean):

The mean is the average of a set of values. It's calculated by adding all the values in the dataset
and dividing by the number of values.

Formula:
x̄ = ( Σ xi ) / n

 Xi is each individual value in the dataset.
 n is the total number of values.

Calculate mean in Python:

Import numpy as np
data = [1, 2, 3, 4, 5] # Calculate mean
mean = [Link](data)
print("Mean:", mean)

Output:
Mean: 3.0

2. Median:

The median is the middle value in a dataset when the values are sorted in ascending or
descending order. If the dataset has an even number of elements, the median is the average of
the two middle values.

Formula:

 If the number of observations in the data is odd the median is ((n+1)/2)th observation

 If the number of observations in the data is Even the median is (n/2)th observation

Prepared by: B. Sai Hemanth,

Assistant Professor, Dept. Of MCA
Steps to calculate median:

1. Sort the dataset in ascending order.

2. If the number of data points is odd, the median is the middle value.
3. If the number of data points is even, the median is the average of the two
middle values.

Example:

For data: 1, 3, 5, 7, 9 → Median is 5.

For data: 1, 3, 5, 7 → Median is (3+5)/2=4

Calculate median in Python:

import numpy as np
data = [1, 2, 3, 4, 5]
median = np. median (data)
print("Median:", median)

Output:
Median: 3.0

3. Mode:

The mode is the value that appears most frequently in a data set. There can be more than one
mode if multiple values occur with the same highest frequency.

Example:

For data: 1, 2, 2, 3, 3, 3, 4 → Mode is 3 (as it appears most frequently).

Calculate mode in Python:

from scipy import stats

data = [1, 2, 2, 3, 3, 3, 4]
mode = stats. mode (data)
print("Mode:", mode)

Output:
Mode: ModeResult(mode=3, count=3)

4. Harmonic Mean:

The harmonic mean is the reciprocal of the arithmetic mean of the reciprocals of the data
values. It’s useful for rates and ratios.

Formula:

Prepared by: B. Sai Hemanth,

Assistant Professor, Dept. Of MCA
 𝑛
𝒉𝒂𝒓𝒎𝒐𝒏𝒊𝒄 𝒎𝒆𝒂𝒏 = 1
∑( )
𝑋𝑖

Where:

 n is the number of values,

 ∑(1/Xi) are the individual values.

Calculate Hormonic mean in Python:

from scipy import stats

data = [1, 2, 3, 4, 5]
harmonic_mean = [Link](data)
print("Harmonic Mean:", harmonic_mean)

Output:
Harmonic Mean: 2.18978102189781

5. Geometric Mean:

The geometric mean is the nth root of the product of all values in a dataset. It's useful for data
involving growth rates or multiplicative processes.

Formula:

Calculate Geometric mean in Python:

from scipy import stats

data = [1, 2, 3, 4, 5]
geometric_mean = [Link](data)
print("Geometric Mean:", geometric_mean)

Output:
Geometric Mean: 2.6051710846973517

Full example code for measures of central tendency:

Here is the full code to calculate all the measures of central tendency:

import numpy as np
from scipy import stats

# Sample data

Prepared by: B. Sai Hemanth,

Assistant Professor, Dept. Of MCA
data = [1, 2, 3, 4, 5]

# Mean
mean = [Link](data)
print("Mean:", mean)
# Median
median = [Link](data)
print("Median:", median)
# Mode
mode = [Link](data)
print("Mode:", mode)
# Harmonic Mean
harmonic_mean = [Link](data)
print("Harmonic Mean:", harmonic_mean)
# Geometric Mean
geometric_mean = [Link](data)
print("Geometric Mean:", geometric_mean)

Output:

Mean: 3.0
Median: 3.0
Mode: 1
Harmonic Mean: 2.1897810218978104
Geometric Mean: 2.6051710846973517

2. Measures of Dispersion

In data science, measures of dispersion like mean deviation from the mean, standard deviation,
and variance are crucial for understanding the spread of the data and its variability. These
measures help in making informed decisions based on the distribution of the data.

Let's implement these measures of dispersion using Python with the help of libraries like
NumPy and Pandas.

1. Mean Deviation from Mean

The mean deviation is the average of the absolute differences between each data point and the
mean of the data set. It gives a sense of how much the individual values deviate from the mean.

Formula:

Prepared by: B. Sai Hemanth,

Assistant Professor, Dept. Of MCA
Where:

 xi = individual data points

 xˉ = mean of the data set
 n = number of data points

Calculate Mean Deviation from mean in Python:

import numpy as np
import pandas as pd
data = [4, 8, 6, 5, 3, 7, 10, 12, 9, 6] # Example data set
data_series = [Link](data) # Converting the data into a Pandas Series

# 1. Mean Deviation from Mean

mean = [Link](data)
mean_deviation = [Link]([Link](data_series - mean))
print(f"Mean Deviation from Mean: {mean_deviation}")

Output:
Mean Deviation from Mean: 2.2

2. Standard Deviation:

Standard deviation is the square root of the variance and is used to quantify the amount of
variation or dispersion of a data set.

Formula:

Where:

 σ = standard deviation
 Xi = individual data points
 xˉ = mean of the data set
 n = number of data points

Calculate Standard Deviation in Python:

import numpy as np
import pandas as pd
data = [4, 8, 6, 5, 3, 7, 10, 12, 9, 6]
data_series = [Link](data) # Converting the data into a Pandas Series

Prepared by: B. Sai Hemanth,

Assistant Professor, Dept. Of MCA
std_deviation = [Link](data_series) # Standard Deviation
print(f"Standard Deviation: {std_deviation}")

Output:
Standard Deviation: 2.6457513110645907

3. Variance

Variance is the average squared deviation from the mean. While it’s not as interpretable as the
standard deviation (because it’s in squared units), it’s still useful for understanding data spread.
The square root of variance gives the standard deviation.

Formula:

Where:

 σ2 = variance
 xi = individual data points
 xˉ = mean of the data set
 n = number of data points

Calculate Variance in Python:

import numpy as np
import pandas as pd
data = [4, 8, 6, 5, 3, 7, 10, 12, 9, 6]
data_series = [Link](data) # Converting the data into a Pandas Series
variance = [Link](data_series) # Variance
print(f"Variance: {variance}")

Output:
Variance: 7.0

3. Central moments in data science with python

In data science, central moments are important statistical quantities that help describe the
shape and distribution of data. They are used to understand features like the spread, skewness,
and peakedness of a dataset. Central moments are calculated by measuring how data points
deviate from the mean, raised to a particular power. Here's an overview of the most common
central moments in data science, along with Python examples.

Prepared by: B. Sai Hemanth,

Assistant Professor, Dept. Of MCA
1. First Central Moment: (Mean)

The first central moment is always zero, because it's based on the deviation from the mean
(which is how the data is centered).

Formula:

Where: xi is the data point.

μ is the mean of the data.

Example in python:

import numpy as np
data = [2, 4, 6, 8, 10]
mean = [Link](data)
first_central_moment = [Link](([Link](data) - mean) ** 1) # This will always be 0
print("First Central Moment (Mean-centered):", first_central_moment)

Output:
First Central Moment (Mean-centered): 0.0

2. Second Central Moment: (Variance)

The second central moment is called variance and is a measure of the spread or dispersion of
the data. It gives an idea of how far the data points are from the mean.

Formula:

Example in Python:

import numpy as np
data = [2, 4, 6, 8, 10]
mean = [Link](data)
variance = [Link](data) # Variance in NumPy
print("Variance (Second Central Moment):", variance)

Output:
Variance (Second Central Moment): 8.0

Prepared by: B. Sai Hemanth,

Assistant Professor, Dept. Of MCA
3. Third Central Moment: (Skewness)

The third central moment is related to skewness, which tells you how asymmetrical the data
distribution is.

 If skewness is positive, the data distribution has a right tail (longer on the right).
 If skewness is negative, the data distribution has a left tail (longer on the left).
 If skewness is zero, the data is symmetrical.

Formula:

The calculation emphasizes the skewness in the data. If the data points are above the mean,
their cubes will be positive; if they are below the mean, their cubes will be negative.

Example in python:

import numpy as np
data = [2, 4, 6, 8, 10]
mean = [Link](data)
third_central_moment = [Link](([Link](data) - mean) ** 3)
print("Third Central Moment (Skewness-related):", third_central_moment)

(Or)

To directly calculate skewness (using SciPy):

from [Link] import skew

data = [2, 4, 6, 8, 10]
skewness = skew(data)
print("Skewness (Third Central Moment):", skewness)

Output:
Third Central Moment (Skewness-related): 0.0
Skewness (Third Central Moment): 0.0

4. Fourth Central Moment: (Kurtosis)

The fourth central moment is related to kurtosis, which measures the "tailedness" of the
distribution—how heavy or light the tails of the distribution are.

 High kurtosis indicates heavy tails (outliers or extreme values).

 Low kurtosis indicates light tails (fewer extreme values).
 Normal distribution has a kurtosis of 3, but excess kurtosis is calculated as kurtosis
minus 3.

Prepared by: B. Sai Hemanth,

Assistant Professor, Dept. Of MCA
Formula:

Example in python:
import numpy as np
data = [2, 4, 6, 8, 10]
mean = [Link](data)
fourth_central_moment = [Link](([Link](data) - mean) ** 4)
print("Fourth Central Moment (Kurtosis-related):", fourth_central_moment)

(Or)

To directly calculate kurtosis (using SciPy):

from [Link] import kurtosis

data = [2, 4, 6, 8, 10]
kurt = kurtosis(data) # Excess kurtosis by default
print("Kurtosis (Fourth Central Moment):", kurt)

Output:
Fourth Central Moment (Kurtosis-related): 108.8
Kurtosis (Fourth Central Moment): -1.3

[Link] and Rank Correlation in Data Science

Linear Correlation:

Linear correlation refers to the degree to which two variables have a straight-line relationship.
This is commonly measured using the Pearson correlation coefficient.

Example in python:

import numpy as np
x = [Link]([1, 2, 3, 4, 5]) # Example data
y = [Link]([2, 4, 5, 4, 5])
# Calculate Pearson correlation
correlation, _ = pearsonr(x, y)
print(f'Pearson correlation: {correlation}')

Output:
Pearson correlation: 0.7745966692414834

Prepared by: B. Sai Hemanth,

Assistant Professor, Dept. Of MCA
Rank Correlation:

Rank correlation (such as Spearman’s rank correlation) measures how well the relationship
between two variables can be described using a monotonic function.

Example in python:
from [Link] import spearmanr
# Calculate Spearman's rank correlation
rank_corr, _ = spearmanr(x, y)
print(f'Spearman rank correlation: {rank_corr}')

Output:
Spearman rank correlation: 0.7378647873726218

[Link] and Correlation

Covariance:

Covariance measures the direction of the linear relationship between variables.

o Positive covariance indicates the variables tend to increase together.

o Negative covariance indicates that one variable tends to increase as the other decreases.

Formula:

Example in python:

import numpy as np
# Example data
x = [Link]([1, 2, 3, 4, 5])
y = [Link]([2, 4, 5, 4, 5])
cov_matrix = [Link](x, y)
print(f'Covariance matrix: \n{cov_matrix}')

Output:
Covariance matrix:
[[2.5 1.5]
[1.5 1.5]]

Prepared by: B. Sai Hemanth,

Assistant Professor, Dept. Of MCA
Correlation:

Correlation is a normalized version of covariance, allowing for comparison between different

datasets. The most common correlation is Pearson's r. The formula for the Pearson correlation
coefficient is:

Formula:

Example in python:I
import numpy as np
# Example data
x = [Link]([1, 2, 3, 4, 5])
y = [Link]([2, 4, 5, 4, 5])
# Calculate Pearson correlation
correlation, _ = pearsonr(x, y)
print(f'Pearson correlation: {correlation}')

Output:
Pearson correlation: 0.7745966692414834

[Link] Testing (Means, Proportions, Variances, and Correlations)

Hypothesis testing is used to make inferences about population parameters based on sample
data. We test a null hypothesis (H₀) against an alternative hypothesis (H₁) to draw conclusions.

a. Hypothesis Testing for Means:

For testing means, the t-test is commonly used to compare the means of one or more samples
with the population mean or between two sample groups.

1. Null hypothesis (H₀): The sample mean is equal to the population mean.
2. Alternative hypothesis (H₁): The sample mean is not equal to the population mean.

Example in python:

from scipy import stats

import numpy as np
# Sample data
sample_data = [Link]([12, 14, 16, 18, 20, 22, 24])

Prepared by: B. Sai Hemanth,

Assistant Professor, Dept. Of MCA
 # Population mean
population_mean = 20
# One-sample t-test
t_stat, p_value = stats.ttest_1samp(sample_data, population_mean)
print(f"T-statistic: {t_stat}, P-value: {p_value}")
Output:
T-statistic: -1.224744871391589, P-value: 0.266569703380069

b. Hypothesis Testing for Proportions:

For testing proportions, we use the z-test.

 Null hypothesis (H₀): The population proportion is equal to a specified value.

 Alternative hypothesis (H₁): The population proportion is different from the specified
value.

Example in python: Testing whether the proportion of heads in 500 coin flips is 0.5.

import numpy as np
import [Link]
# Data: 500 coin flips, 260 heads
successes = 260
n = 500
p_null = 0.5
# Perform the z-test for proportions
z_stat, p_value = [Link].proportions_ztest(successes, n, p_null)
print(f"Z-statistic: {z_stat}, P-value: {p_value}")

Output:

Z-statistic: 0.8951435925492919, P-value: 0.3707103335043911

c. Hypothesis Testing for Variances:

For testing variances, the Chi-Square test is used. This test compares the sample variance to
the population variance.

 Null hypothesis (H₀): The population variance is equal to the sample variance.
 Alternative hypothesis (H₁): The population variance is not equal to the sample
variance.

Example in python:

import numpy as np
from [Link] import chi2

Prepared by: B. Sai Hemanth,

Assistant Professor, Dept. Of MCA
 # Sample data (assumed population variance = 10^2)
sample_data = [Link]([11, 12, 13, 14, 15])
sample_variance = [Link](sample_data, ddof=1)
population_variance = 10**2
n = len(sample_data)
# Chi-squared statistic
chi_squared_stat = (n - 1) * sample_variance / population_variance
# P-value for Chi-squared test
p_value = 1 - [Link](chi_squared_stat, df=n-1)
print(f"Chi-squared statistic: {chi_squared_stat}, P-value: {p_value}")
Output:

Chi-squared statistic: 0.1, P-value: 0.9987908957257497

d. Hypothesis Testing for Correlation:

For testing the correlation between two variables, we use the Pearson correlation coefficient.

 Null hypothesis (H₀): There is no linear correlation between the two variables.
 Alternative hypothesis (H₁): There is a significant linear correlation between the two
variables.

Example in python:

import numpy as np
from [Link] import pearsonr
# Sample data (two variables)
x = [Link]([1, 2, 3, 4, 5])
y = [Link]([2, 3, 5, 7, 11])
# Calculate Pearson correlation coefficient
correlation, p_value = pearsonr(x, y)
print(f"Pearson correlation: {correlation}, P-value: {p_value}")
Output:

Pearson correlation: 0.9722718241315028, P-value: 0.005519518537275836

[Link] of Random Variable and Probability

Random Variable: A random variable is a variable whose value is subject to random

fluctuations. It can be classified into:

o Discrete Random Variables: Can take on a finite or countable number of values.

Example: The number of heads in 10 coin tosses.
o Continuous Random Variables: Can take any value within a given range. Example:
Height or weight of individuals.

Probability: Probability is a measure of how likely an event is to occur. For a random

variable XXX, the probability distribution describes the likelihood of different outcomes.

Prepared by: B. Sai Hemanth,

Assistant Professor, Dept. Of MCA
 [Link] Probability Distributions in Python

Discrete probability distributions model random variables that take on a finite or countably
infinite number of values. These distributions are commonly used in data science when the
outcomes of random events can be clearly enumerated. Some common discrete probability
distributions include:

1. Binomial Distribution
2. Poisson Distribution

1. Binomial Distribution:

The Binomial distribution models the number of successes in a fixed number of independent
trials of a binary experiment (e.g., flipping a coin). The trials are identical, and each has two
possible outcomes: success (usually coded as 1) and failure (usually coded as 0).

Properties:

 n: Number of trials
 p: Probability of success on a single trial
 X: Number of successes in n trials

Example in python:I
mport numpy as np
import [Link] as plt
from [Link] import binom
# Parameters
n = 10 # Number of trials
p = 0.5 # Probability of success (e.g., heads in coin toss)
k = [Link](0, n+1) # Possible number of successes
# Probability Mass Function (PMF)
pmf = [Link](k, n, p)
# Plot the PMF
[Link](k, pmf, color='blue', alpha=0.6)
[Link](f"Binomial Distribution (n={n}, p={p})")
[Link]("Number of successes")
[Link]("Probability")
[Link]()

Prepared by: B. Sai Hemanth,

Assistant Professor, Dept. Of MCA
Output:

2. Poisson Distribution

The Poisson distribution is used to model the number of events that occur in a fixed interval of
time or space. It is commonly used for modeling rare events, such as the number of customer
arrivals in an hour or the number of emails received in a day.

Properties:

 λ (lambda): The average rate of occurrence of events.

Example in python:

mport numpy as np
import [Link] as plt
from [Link] import binom
# Parameters
lambda_ = 3 # Average rate of events
# Generate random samples from the Poisson distribution

samples = [Link](lambda_, 1000)

# Plot the histogram of the samples
[Link](samples, bins=30, density=True, alpha=0.6, color='orange')
[Link](f"Poisson Distribution (λ={lambda_})")
[Link]("Number of events")
[Link]("Frequency")
[Link]()

Prepared by: B. Sai Hemanth,

Assistant Professor, Dept. Of MCA
Output:

[Link] Probability Distributions

In data science, continuous probability distributions are used to model the probability of
different outcomes in a continuous range. These distributions are essential in statistical
modeling and are used to understand the underlying structure of data, conduct hypothesis
testing, and make predictions.

Here are three commonly used continuous probability distributions:

1. Gaussian (Normal) Distribution

2. Exponential Distribution
3. Chi-Square Distribution

[Link] (Normal) Distribution:

The Gaussian distribution (or Normal distribution) is one of the most widely used distributions.
It is symmetric and bell-shaped, characterized by its mean (μ) and standard deviation (σ). It is
commonly used in statistics to represent real-valued random variables whose distributions are
not known.

Properties:

 Mean (μ): the center of the distribution.

 Standard Deviation (σ): controls the spread of the distribution.
 The probability density function (PDF) for a normal distribution is given by:

Prepared by: B. Sai Hemanth,

Assistant Professor, Dept. Of MCA
Example in python:

import numpy as np
import [Link] as plt
from [Link] import norm
# Parameters
mu = 0 # Mean
sigma = 1 # Standard Deviation
# Generate random samples from a normal distribution
samples = [Link](mu, sigma, 1000)
# Plot histogram and PDF
[Link](samples, bins=30, density=True, alpha=0.6, color='b')
# Plot the theoretical PDF of the normal distribution
xmin, xmax = [Link]()
x = [Link](xmin, xmax, 100)
p = [Link](x, mu, sigma)
[Link](x, p, 'k', linewidth=2)
[Link]("Normal Distribution (Mean = 0, SD = 1)")
[Link]()

Output:

Prepared by: B. Sai Hemanth,

Assistant Professor, Dept. Of MCA
[Link] Distribution:

The Exponential distribution is often used to model time between events in a Poisson process,
where events occur continuously and independently at a constant average rate. This
distribution is characterized by the rate parameter λ, which is the reciprocal of the mean.

Properties:

 The probability density function (PDF) is given by:

Where: λ is the rate parameter (mean = 1/λ).

Example in python:

import numpy as np
import [Link] as plt
from [Link] import norm
lambda_ = 1 # Rate paramete
# Generate random samples from an exponential distribution
samples = [Link](1/lambda_, 1000)
# Plot histogram and PDF
[Link](samples, bins=30, density=True, alpha=0.6, color='g')
# Plot the theoretical PDF of the exponential distribution
x = [Link](0, [Link](samples), 100)
p = lambda_ * [Link](-lambda_ * x)
[Link](x, p, 'k', linewidth=2)
[Link]("Exponential Distribution (lambda = 1)")
[Link]()

Output:

Prepared by: B. Sai Hemanth,

Assistant Professor, Dept. Of MCA
[Link]-square Distribution:

The Chi-Square distribution is a special case of the Gamma distribution and is used primarily
in statistical inference, such as in hypothesis testing (e.g., the Chi-square test). It is defined by
its degrees of freedom (df). The distribution is right-skewed and becomes more symmetric as
the degrees of freedom increase.

Example in python:

import numpy as np
import [Link] as plt
from [Link] import norm
import math
df = 2 # Degrees of freedom”
# Generate random samples from a chi-square distribution
samples = [Link](df, 1000)
# Plot histogram and PDF
[Link](samples, bins=30, density=True, alpha=0.6, color='r')
# Plot the theoretical PDF of the chi-square distribution
x = [Link](0, [Link](samples), 100)
p = (x**(df/2 - 1) * [Link](-x/2)) / (2**(df/2) * [Link](df/2))
[Link](x, p, 'k', linewidth=2)
[Link](f"Chi-Square Distribution (df = {df})")
[Link]()

Output:

Prepared by: B. Sai Hemanth,

Assistant Professor, Dept. Of MCA
 [Link] Probability in Data Science

Bayesian Probability is a method in probability theory that allows you to update the
probability for a hypothesis as more evidence or information becomes available. It is widely
used in data science and machine learning for reasoning about uncertain events, updating
beliefs based on data, and making predictions.

Bayesian Theorem:

Bayes' Theorem gives us a way to calculate the conditional probability of a hypothesis H

given some observed data D:

Where:

 P(H∣D) is the posterior probability: the probability of the hypothesis H being true
after considering the data D.
 P(D∣H) is the likelihood: the probability of observing the data D given that H is true.
 P(H) is the prior probability: the initial belief about the hypothesis H before
considering the data D.
 P(D) is the evidence: the total probability of observing the data, often calculated as a
sum over all possible hypotheses.

In Python, you can implement Bayesian updates using libraries such as PyMC3, but here's a
simple example using scipy:

Step-by-Step:

1. Prior: Your initial belief is that the coin is fair, so the prior probability for heads is 0.5.
2. Likelihood: You observe a coin flip and get heads.
3. Posterior: You update your belief based on the observation of getting heads.

prior = 0.5 # Initial belief (50% chance the coin is biased)

likelihood = 0.8 # Likelihood of observing heads if the coin is biased
evidence = 0.6 # Probability of observing heads from any coin
# Bayes' Theorem
posterior = (likelihood * prior) / evidence
print(f'Posterior probability: {posterior}')

Output:

Posterior probability: 0.6666666666666667

Prepared by: B. Sai Hemanth,

Assistant Professor, Dept. Of MCA