Statistics

Table of Contents
Descriptive............................................................................................................................................2
Inferancial stats.....................................................................................................................................2
Population & Samples.....................................................................................................................2
Sampling Techniques..................................................................................................................2
Varialbe:................................................................................................................................................3
Quantative...................................................................................................................................3
Qualitative/Categorical...............................................................................................................3
variale measurement scale...............................................................................................................3
Nominal.......................................................................................................................................3
ordinal.........................................................................................................................................3
Interval........................................................................................................................................3
Ratio............................................................................................................................................3
Descriptive Statistics............................................................................................................................3
Measure of central tendency............................................................................................................3
Mean............................................................................................................................................3
Median........................................................................................................................................3
Mode...........................................................................................................................................4
Measure f Spread (Variaility)...........................................................................................................4
Range..........................................................................................................................................4
Variance.......................................................................................................................................4
Standard Deviation......................................................................................................................4
Measures of Shape...........................................................................................................................4
Symmetry....................................................................................................................................4
Modality......................................................................................................................................4
Inferantial Statistics..............................................................................................................................5
Confidence Interval..........................................................................................................................5
Point Estimate.............................................................................................................................5
Interval Estimate.........................................................................................................................5
Hypothesis testing............................................................................................................................5
Null Hypothesis:.........................................................................................................................5
Alternate Hypothesis...................................................................................................................5
Regression Analysis.........................................................................................................................5
Statistics is a branch of mathamatis that involvs the colelction, aalsis, interpretation,
presentaion and organization fo data.

Types of Statistics

Descriptive
Organizing and summarizing data in a meaningful way. Ex: what si the everage marks of students in
a calss? What is the percentae of students passin out of the exam

Inferancial stats
Data is measured to form inferences/conclusions based on samples of data collected. Ex: are the
ages of the students of thsi calls ssimilar tot e age of the Maths class in the college?

Population & Samples

Population is denoted by N and samples are denoted by n
Q: are the ages of the students of thsi calls ssimilar tot e age of the Maths class in the college?
Here entire college is called the population and the maths class students are called the samples

Ex: Election and exit Poll in a area

entire area peopl is called population and the people you choose to get exit poll survey is called
samples

Sampling Techniques
1. Simple Random sampling: Every member of the populatio has equal chance of bei selected for
your sample n. ex: go and pick rando people for survay for exit poll
2. Stratified Samplig: Where the population is split into non-overlapping groups(strata). Ex:
gender/age group/continent/profession/earning groups/etc based survey. : RBI’s housin sruvey.
Drug sampling for testing
3. Systematic Sampling: ex: pick every Nth population(or regular interval) as sample.
4. Convenience/cluster Sampling: divindin the populaton into smaller sections or clusters hen
randomly slecting one or more of the clusters and using lal of its members as members of the
sample. Tsi techniques is used when the population is large spread across geographic area. : small
business owners sampling based on random zip code

Varialbe:
A variable is a property that can take any value. Their are two typoes of avriable
Quantative
Measured numerically. Can add, subtract, multiply ad device. Two types: Descrete and continuous

Qualitative/Categorical
Based on characteristics we can derive some categorical variale. Doesnot support arithemetic
operation. Ex: gender(male/Femele), IQ(0-50, 51-100), lood group(0+, 0-)

variale measurement scale

Nominal
Categorical: gender, color,

ordinal
Order(rank) of the data matters but value doesnot

Interval
Order matters,value also atters, natural zero not present s

Ratio

Frequency Destribution
Comulative frequency
Descriptive Statistics
Measure of central tendency
Refers to the measure used to deterine the centre of the distriution of data. Mean, median and mode.

Mean
Also called the averae, calculated as
sum of total elements / number of elements

works well with numerical data

Median
Formula: (N+1)/2
 If number of values is ODD, the median is the middle value when arranged in ascending
order
If the number of values is EVEN, the median is the average of the two middle value when
arranged ascending order.
Median works well with outliers.

Mode
Most frequently occuring value in a given dataset
used to fill the missing elements of the dataset
works well with categorical elements

Measure of Spread (Variaility)/Dispersion

Range
Forula: Max value(last item) – min value(first item)

Variance

Standard Deviation

Formula = Variance

Percentile
A percentile is a value elow which a certain percentage fo observations lie.

Percentile Rank of x = (% of values belwo x) * 100 / n

5 Number summary
Minimum
First Quartile
median
third Quartile
maximum

Quartile

Removing the outliers

Lowewr Fence <==> Upper Fence

Lower fence = Q1 - 1.5(IQR)
Upper Fence = Q3 + 1.5(IQR)
IQR = Q3 – Q1
Q1 = 25
Q3 = 75
using these formulas we can draw oxplot and ientify outliers

Distribution
Normal Distribution

Standard normal Distriution

Apply z score to the distribution to get the standard normal distribution

Z score
Formula = (Xi - mean)/standard deviation
1.50minutes fo video
Log Normal Disytriution
erailis Distribution
Binomial Distriution

Measures of Shape
Symmetry
Postive skew: when the tail in the graph is longer and flatter on the right side
Negativer skew: when the tail in the graph is longer ad flatter on the left side

Modality
Determined by the number of peaks it has.
Unimodal: 1 peak
Bimodal: 2 peak
Multimodal: multiple peaks

Inferantial Statistics
Confidence Interval
Estimate: is specific observed numerical value used to estimate an uknown population paramater.
Two types:
Point Estimate
Single numerical value used to estimate an unknown population parameter. Sample ea is a point
estimate fo a population mean.

Interval Estimate
Range of values used to estimate an known population parameter.
Hypothesis testing
Hypothesis testing is a statistical method used to make inferences or draw conclusions about a
population based on a sample of data. In the context of AI, hypothesis testing plays an important
role in model evaluation, model selection, and decision-making processes. Here’s how it is applied:

1. Model Evaluation
When developing AI models, particularly in machine learning, hypothesis testing can be used to
evaluate the effectiveness of the model on a given dataset. For example, one might want to test if a
new model performs significantly better than a baseline model.
Example:
 Null hypothesis (H₀): The new model's performance is the same as the baseline model.
 Alternative hypothesis (H₁): The new model's performance is different (or better) than the
baseline model.
Using statistical tests like the t-test, we can decide whether to accept or reject the null hypothesis
based on model performance metrics (such as accuracy, precision, or F1 score).

2. A/B Testing for AI Systems

In AI systems, A/B testing is a common approach where two or more variations of a system are
compared to see which one performs better. Hypothesis testing is central to A/B testing, where we
test whether one variation is statistically better than the other.
Example:
 Null hypothesis (H₀): Version A and Version B of the AI system perform the same.
 Alternative hypothesis (H₁): Version A and Version B perform differently.
In AI, this could be applied to things like user experience, system response time, or conversion rates
(in recommendation systems, for example).

3. Feature Selection
In machine learning, hypothesis testing can help in selecting relevant features. By testing whether
the correlation between features and the target variable is statistically significant, one can decide
which features should be retained.
Example:
 Null hypothesis (H₀): There is no significant relationship between a feature and the target
variable.
 Alternative hypothesis (H₁): The feature has a significant relationship with the target
variable.
Hypothesis tests such as the chi-square test or correlation tests can be used to evaluate these
relationships.

4. Evaluating Algorithmic Fairness

Hypothesis testing can also be used to ensure fairness in AI models. For example, one might want to
test if a model’s performance is significantly biased across different demographic groups (e.g.,
gender or ethnicity).
Example:
 Null hypothesis (H₀): There is no significant difference in model performance across
different groups.
 Alternative hypothesis (H₁): There is a significant difference in model performance across
groups.
Statistical tests can be used to verify whether bias exists, and corrective actions can be taken if
necessary.

5. Model Robustness Testing

AI models should perform well not just on training data, but on new, unseen data. Hypothesis
testing can be used to evaluate whether the model is overfitting to a particular set of training data or
if its performance on new, unseen data is consistent.
Example:
 Null hypothesis (H₀): The model's performance on the test data is the same as its
performance on the training data.
 Alternative hypothesis (H₁): The model's performance on the test data is different from its
performance on the training data.

Common Statistical Tests in AI Hypothesis Testing:

 t-test: Compares the means of two groups (e.g., model A vs. model B).
 Chi-square test: Used for categorical data to test the relationship between variables.
 ANOVA (Analysis of Variance): Compares the means of more than two groups.
 Mann-Whitney U test: A non-parametric test for comparing two independent samples.
 Bootstrap methods: Resampling technique for estimating the distribution of a statistic.

The null hypothesis serves as a starting point for statistical testing, establishing a baseline
assumption of no effect or no relationship. Its role is critical in determining whether observed
patterns or differences in data are statistically significant or if they occurred by random chance.

Null Hypothesis:
The null hypothesis (denoted as H₀) is a fundamental concept in statistical hypothesis testing. It
represents a statement or assumption that there is no effect, no difference, or no relationship
between variables in a given context. In other words, it suggests that any observed effect or
relationship in the data is due to random chance, rather than being a result of a specific factor or
intervention.
States there is no signficant difference between the populatio mean and the sample mean.
Key Characteristics of the Null Hypothesis:
1. Assumes no effect or no relationship: It suggests that whatever you are testing does not
cause a change or relationship. For example, if you're testing whether a new drug works, the
null hypothesis would state that the drug has no effect.
2. Tested against alternative hypothesis (H₁): The null hypothesis is typically tested against
an alternative hypothesis, which posits that there is an effect or relationship.
3. Basis for statistical tests: Statistical tests, such as the t-test, chi-square test, or ANOVA, are
used to determine whether the null hypothesis can be rejected in favor of the alternative
hypothesis. This is done by analyzing sample data and calculating the likelihood of
obtaining the observed results if the null hypothesis were true.
4. Reject or fail to reject: Based on the test results, we either:
 Reject the null hypothesis if the data provides sufficient evidence that the null
hypothesis is unlikely to be true.
 Fail to reject the null hypothesis if there is not enough evidence to support the
alternative hypothesis. Importantly, failing to reject the null hypothesis does not
prove it is true, just that there is insufficient evidence to conclude otherwise.

Alternate Hypothesis
The alternative hypothesis (denoted as H₁ or Ha) is a statement in statistical hypothesis testing
that suggests there is an effect, a difference, or a relationship between variables. It is the opposite of
the null hypothesis (H₀), and it reflects the researcher's expectation or the theory being tested.
In simple terms, the alternative hypothesis proposes that something meaningful is happening in the
data—whether it's a relationship, a change, or an effect that is not due to random chance.
Key Characteristics of the Alternative Hypothesis:
1. Contrasts with the null hypothesis (H₀): The alternative hypothesis is essentially the claim
that challenges the null hypothesis. While the null hypothesis asserts that there is no effect or
relationship, the alternative hypothesis asserts that there is an effect or relationship.
2. Non-zero effect or relationship: It suggests that any observed difference or effect in the
data is real, not just a result of random variation.
3. Supports evidence against H₀: If the data shows strong evidence against the null
hypothesis (i.e., if the test statistic is sufficiently extreme), we reject the null hypothesis in
favor of the alternative hypothesis.
4. Can be one-tailed or two-tailed:
 One-tailed alternative hypothesis: The alternative hypothesis specifies the
direction of the effect. For example, the mean of a new treatment is greater than the
mean of an old treatment.
  Two-tailed alternative hypothesis: The alternative hypothesis does not specify the
direction of the effect, only that there is a difference. For example, the mean of a new
treatment is different from the mean of an old treatment (it could be either higher or
lower).

T-test
The t-test is a statistical test used to determine if there is a significant difference between the means
of two groups or samples. It is commonly used when the sample size is small (typically less than
30) and the population standard deviation is unknown. The t-test helps to assess whether the
difference observed in the sample data is likely to be genuine or if it could have occurred by random
chance.

Types of t-Tests:
1. One-Sample t-test: Used to compare the mean of a single sample to a known value or
population mean.
 Example: Testing whether the average height of a sample of students is different
from the national average height.
2. Independent Two-Sample t-test: Used to compare the means of two independent groups
(i.e., two different samples or populations).
 Example: Testing whether the average test scores differ between two classes of
students.
3. Paired Sample t-test (Dependent t-test): Used to compare the means of two related or
matched groups (i.e., when the data consists of paired observations).
 Example: Testing whether a new teaching method improves students' scores by
comparing their scores before and after the method is applied.

Key Assumptions of the t-test:

 Independence: The observations in the sample must be independent of each other (except in
the case of the paired sample t-test).
 Normality: The data should be approximately normally distributed. This assumption is
especially important for small sample sizes.
 Homogeneity of variance: For the independent two-sample t-test, the variances of the two
groups should be equal (this can be tested using Levene's test or other methods).

t-test Formula:
The formula for the t-statistic depends on the type of t-test you are conducting. Here's the general
formula for the independent two-sample t-test:
Where:
 X1ˉ,X2ˉ are the sample means of the two groups.
 s12,s22 are the sample variances of the two groups.
 n1,n2 are the sample sizes of the two groups.
For a paired t-test, the formula would involve the differences between the paired observations:

Where:
 dˉ is the mean of the differences between paired observations.
 sd is the standard deviation of those differences.
 n is the number of pairs.

Steps to Perform a t-test:

1. State the hypotheses:
 Null hypothesis (H₀): There is no significant difference between the means (e.g., μ1
=μ2).
 Alternative hypothesis (H₁): There is a significant difference between the means
(e.g., μ1=μ2).
2. Calculate the t-statistic: Use the appropriate formula for the t-test based on your data and
hypotheses.
3. Determine the degrees of freedom (df):
 For an independent two-sample t-test, df = n1+n2−2.
 For a paired t-test, df = n−1, where n is the number of pairs.
4. Find the critical t-value: This value depends on your chosen significance level (usually
α=0.05) and the degrees of freedom. You can look this up in a t-distribution table or use
statistical software.
5. Compare the t-statistic to the critical value:
 If the absolute value of the calculated t-statistic is greater than the critical value from
the t-distribution table, reject the null hypothesis.
 If the absolute value of the t-statistic is less than the critical value, fail to reject the
null hypothesis.
6. Interpret the results: If you reject the null hypothesis, it suggests that there is a statistically
significant difference between the two means. If you fail to reject the null hypothesis, it
means there is insufficient evidence to conclude a significant difference.
The t-test is a powerful statistical tool for comparing the means of two groups, helping to determine
if any observed differences are statistically significant or if they could have arisen due to random
variation. It's widely used in research, medical studies, business analysis, and many other fields to
assess differences between groups based on sample data.

Chi-square test

he Chi-square test (often denoted as χ2) is a statistical test used to determine whether there is a
significant association or difference between categorical variables. It is commonly used in cases
where you want to compare observed data with expected data, or to check the independence of two
categorical variables. There are two primary types of Chi-square tests:
1. Chi-square Test of Independence
2. Chi-square Goodness-of-Fit Test

1. Chi-square Test of Independence:

This test is used to assess whether two categorical variables are independent of each other or if
there is an association between them.
Example:
Suppose you want to test if there is an association between gender (male, female) and voting
preference (Democrat, Republican, Independent). You collect data and set up a contingency table
showing the frequencies of these categories.

Democrat Republican Independent Total

Male 30 20 10 60
Female 25 15 20 60
Total 55 35 30 120
 Null hypothesis (H₀): There is no association between gender and voting preference (the
variables are independent).
 Alternative hypothesis (H₁): There is an association between gender and voting preference
(the variables are dependent).
The Chi-square test will determine if the distribution of voting preference across gender is
consistent with what would be expected if the variables were independent.
Formula for the Chi-square Test of Independence:

Where:
 Oij is the observed frequency for category i,j.
  Eij is the expected frequency for category i,j under the assumption that the variables are
independent.
The expected frequency Eij is calculated as:

2. Chi-square Goodness-of-Fit Test:

This test is used to determine if a sample data fits a specific theoretical distribution. It is typically
used when you have observed frequencies of a categorical variable and you want to compare them
to expected frequencies based on a known distribution (such as a uniform, binomial, or Poisson
distribution).
Example:
You are testing whether a six-sided die is fair. You roll the die 60 times and record the number of
occurrences of each face:

Face Observed Frequency

1 10
2 12
3 8
4 14
5 7
6 9
 Null hypothesis (H₀): The die is fair, meaning each face has an equal probability of
appearing (i.e., the expected frequency for each face is 10).
 Alternative hypothesis (H₁): The die is not fair, meaning the observed frequencies do not
match the expected frequencies.
Formula for the Chi-square Goodness-of-Fit Test:

Where:
 Oi is the observed frequency for category i.
 Ei is the expected frequency for category i, based on the hypothesis (e.g., if the die is fair,
each face should have an expected frequency of 60/6=10).

ANOVA (Analysis of Variance)

ANOVA (Analysis of Variance) is a statistical method used to compare means across multiple
groups to determine if there are any statistically significant differences between them. It tests the
null hypothesis that all group means are equal, and the alternative hypothesis that at least one group
mean is different from the others.

ANOVA is commonly used when you have more than two groups or conditions to compare, and it
helps you avoid performing multiple t-tests (which would increase the risk of Type I errors).
Instead, ANOVA evaluates the overall differences among group means in a single test.

Types of ANOVA
1. One-Way ANOVA: Used when you have one independent variable (factor) with more than
two levels (groups), and you're interested in testing if the means of these groups are
significantly different from each other.
 Example: Testing the average test scores of students taught with three different
teaching methods (Method A, Method B, Method C).
2. Two-Way ANOVA: Used when you have two independent variables (factors), and you want
to assess:
 The main effect of each factor (independent variable).
 The interaction effect between the two factors.
 Example: Investigating how both teaching method (Method A, Method B, Method
C) and gender (Male, Female) affect test scores.
3. Repeated Measures ANOVA: Used when the same subjects are measured multiple times,
such as when each participant undergoes several treatments or assessments.
 Example: Measuring the effect of three different diets on weight loss over three
months in the same group of individuals.
4. Multivariate Analysis of Variance (MANOVA): A generalization of ANOVA that allows
for more than one dependent variable.
 Example: Testing how different teaching methods affect both test scores and
student satisfaction.

Z-test
Z-Test: Definition and Overview
A Z-test is a statistical test used to determine whether there is a significant difference between the
sample mean and the population mean, or between two sample means, under the assumption that
the data follows a normal distribution. It is called a Z-test because it uses the Z-score (a standard
score) to calculate the probability of observing the data.
Z-tests are most commonly used when the sample size is large (typically n>30) or when the
population standard deviation (σ) is known.

Types of Z-Tests
There are different variations of the Z-test, depending on the specific situation:
1. One-Sample Z-Test: Used to compare the sample mean to the known population mean.
2. Two-Sample Z-Test: Used to compare the means of two independent samples.
 3. Z-Test for Proportions: Used to compare the observed proportion in a sample to a known
population proportion.

Formula for Z-Score

The Z-test uses the Z-score to standardize the data. The formula for the Z-score is:

Where:
 X = sample mean (for a one-sample test) or difference in means (for a two-sample test)
 μ = population mean (for a one-sample test) or the hypothesized population mean (for a two-
sample test)
 σ = population standard deviation (for a one-sample test) or pooled standard deviation (for a
two-sample test)
 n = sample size

One-Sample Z-Test
This test is used when you want to compare the mean of a single sample to a known population
mean.
Hypotheses:
 Null hypothesis (H₀): The sample mean is equal to the population mean (μ).
 Alternative hypothesis (H₁): The sample mean is not equal to the population mean.
Steps:
1. State the Hypotheses:
 H₀: μ=μ0 (the population mean is equal to a known value).
 H₁: μ=μ0 (the population mean is different from the known value).
2. Set the Significance Level (α): Typically, α=0.05 (5%), though other values can be used
depending on the situation.
3. Calculate the Z-Score:

Where X is the sample mean, μ0 is the population mean, σ is the population standard deviation, and
n is the sample size.
 Find the P-value: Using the Z-score, find the corresponding p-value from the standard
normal distribution. If the test is two-tailed, you’ll look for the probability of getting a Z-
score as extreme or more extreme than the observed one.
 Decision Rule:
 If the p-value is less than or equal to the significance level α, reject the null hypothesis.
 If the p-value is greater than α, fail to reject the null hypothesis.
Example:
Suppose you want to test if the average height of a group of people differs from the known
population mean of 170 cm. You collect a sample of 100 people with a sample mean height of 172
cm and a population standard deviation of 10 cm.
Using the Z-test formula:

Where:
 Xˉ1 and Xˉ2 are the sample means,
 σ1 and σ2 are the population standard deviations of the two samples,
 n1 and n2 are the sample sizes of the two groups.
Steps for Two-Sample Z-Test:
1. State the Hypotheses:
 H₀: μ1=μ2 (the means of the two populations are equal).
 H₁: μ1=μ2 (the means of the two populations are different).
2. Set the Significance Level: Typically, α=0.05.
3. Calculate the Z-Score using the formula above.
4. Find the P-value from the Z-distribution.
5. Decision Rule: Reject the null hypothesis if the p-value is less than or equal to the
significance level α.

Z-Test for Proportions

A Z-test for proportions is used to test whether the proportion of success in a sample is different
from a known population proportion. The formula for the Z-score in this case is:

Where:
 p is the sample proportion,
 P is the population proportion,
 n is the sample size.
Steps for Z-Test for Proportions:
1. State the Hypotheses:
 H₀: p=P (the sample proportion is equal to the population proportion).
 H₁: p=P (the sample proportion is not equal to the population proportion).
2. Set the Significance Level: Typically, α=0.05.
 3. Calculate the Z-Score using the formula.
4. Find the P-value from the Z-distribution.
5. Decision Rule: Reject the null hypothesis if the p-value is less than or equal to α.
Example (Proportion):
Suppose a company claims that 80% of their products pass quality control. You sample 200
products and find that 170 pass the test. The sample proportion is p=200170=0.85.
Now, calculate the Z-score to see if this proportion significantly differs from the claimed 80%
(0.80):

For a two-tailed test with α=0.05, the critical Z-value is approximately ±1.96. Since 1.77 <
1.96, you fail to reject the null hypothesis, meaning there is no significant difference between
the sample proportion and the claimed population proportion.

When to Use a Z-Test

 Large sample size: The Z-test is typically used when the sample size is large (n>30),
because the Central Limit Theorem ensures that the sampling distribution of the sample
mean will be approximately normal.
 Known population standard deviation: The Z-test is ideal when the population standard
deviation (σ) is known. If σ is unknown, you might use a t-test instead.

Conclusion
The Z-test is a powerful statistical tool used to test hypotheses about population parameters (mean
or proportions) based on sample data. It is commonly used when the sample size is large and the
population standard deviation is known. Z-tests are used for:
 One-sample Z-test (comparing a sample mean to a population mean),
 Two-sample Z-test (comparing two independent sample means),
 Z-test for proportions (comparing observed proportions to known population proportions).
By calculating the Z-score and comparing it to the critical value (or finding the p-value), you can
make inferences about the population based on the sample data.

Mann-Whitney U test
Bootstrap methods test
P-Vaqlues
A p-value (short for probability value) is a fundamental concept in statistical hypothesis testing. It
helps you assess the strength of the evidence against the null hypothesis (H₀). In simple terms, the
p-value tells you how likely it is that the observed data (or something more extreme) would occur if
the null hypothesis were true.

Definition:
 P-value is the probability of obtaining results at least as extreme as the observed results,
assuming that the null hypothesis is true.
 A small p-value suggests that the observed data is unlikely under the null hypothesis, and
you might reject the null hypothesis.
 A large p-value suggests that the observed data is consistent with the null hypothesis, and
you may fail to reject it.

Key Points:
1. Interpretation:
 The p-value is a measure of how compatible your data is with the null hypothesis.
 A small p-value (typically ≤ 0.05) indicates strong evidence against the null
hypothesis, leading you to reject the null hypothesis.
 A large p-value (> 0.05) suggests that the data is consistent with the null hypothesis,
leading you to fail to reject the null hypothesis.
2. Threshold/Significance Level (α):
 The p-value is compared to a pre-determined threshold known as the significance
level, often denoted by α. The most commonly used value for α is 0.05 (5%
significance level).
 If p ≤ α (e.g., 0.05), you reject the null hypothesis (indicating a statistically
significant result).
 If p > α, you fail to reject the null hypothesis (indicating no statistically
significant result).
3. Not "Proof" of the Hypothesis:
 A p-value does not prove that the null hypothesis is true or false. It simply provides
evidence to support or undermine the null hypothesis.
 A p-value is not the probability that the null hypothesis is true or false. Instead, it’s
the probability of observing your data (or something more extreme) assuming the
null hypothesis is true.

Example:
Suppose you're conducting a hypothesis test to determine if a new drug affects blood pressure more
than a placebo.
 Null hypothesis (H₀): The new drug has no effect on blood pressure (i.e., the mean
difference in blood pressure between the drug and placebo groups is zero).
 Alternative hypothesis (H₁): The new drug has an effect on blood pressure (i.e., the mean
difference in blood pressure between the drug and placebo groups is not zero).
After conducting the statistical test, you get a p-value of 0.03.
 If you use a significance level of α=0.05, since p = 0.03 ≤ 0.05, you would reject the null
hypothesis, concluding that there is significant evidence to suggest that the drug does affect
blood pressure.
If the p-value had been 0.08, you would fail to reject the null hypothesis, because 0.08 is greater
than 0.05, suggesting that there is not enough evidence to support the claim that the drug affects
blood pressure.

P-value Range and Interpretation:

 p ≤ 0.01: Strong evidence against the null hypothesis (very statistically significant).
 0.01 < p ≤ 0.05: Moderate evidence against the null hypothesis (statistically significant).
 0.05 < p ≤ 0.10: Weak evidence against the null hypothesis (marginally significant).
 p > 0.10: Weak evidence in favor of the null hypothesis (not statistically significant).

Limitations of p-values:
1. Arbitrary Thresholds: The threshold of 0.05 for statistical significance is somewhat
arbitrary and can lead to over-interpretation. A p-value just below 0.05 is not necessarily
strong evidence, and a p-value just above 0.05 is not necessarily weak evidence.
2. Doesn't Measure Effect Size: A small p-value doesn’t tell you anything about the
magnitude of the effect (e.g., how large or important the difference between groups is). For
that, you need to look at the effect size.
3. Multiple Comparisons: When conducting multiple hypothesis tests, the probability of
obtaining at least one significant result due to random chance increases. This is called the
multiple comparisons problem, and adjustments (like the Bonferroni correction) are
often needed to account for this.
4. Not the Probability of the Null Hypothesis: A common misconception is that the p-value
tells you the probability that the null hypothesis is true. In reality, the p-value is a measure of
the likelihood of the observed data given the null hypothesis, not the probability that the null
hypothesis is true.

Example in Context (Two-Tailed Test):

Consider a two-tailed t-test where you are testing whether the mean of a sample is equal to a
population mean (e.g., testing if the average height of a population is 170 cm).
 Null hypothesis (H₀): The mean height of the sample is 170 cm.
 Alternative hypothesis (H₁): The mean height is not 170 cm.
Suppose you conduct the test and get a p-value of 0.03.
 If your significance level is 0.05, then because 0.03 < 0.05, you would reject the null
hypothesis, indicating that there is significant evidence that the mean height is not 170 cm.
 If you had a p-value of 0.08, you would fail to reject the null hypothesis, because 0.08 >
0.05, suggesting no significant evidence that the mean height is different from 170 cm.
Conclusion:
The p-value is a tool for helping you make decisions about the validity of the null hypothesis in
hypothesis testing. It quantifies the strength of the evidence against the null hypothesis and helps
you determine whether the observed data is consistent with what you would expect under the null
hypothesis. However, p-values should be interpreted with care and in context, considering factors
like effect size, study design, and multiple comparisons.

Type I and Type II error

Type I Error and Type II Error
In statistical hypothesis testing, errors can occur when making decisions about the null hypothesis
(H₀). These errors are categorized into two types: Type I error and Type II error. These errors
represent incorrect conclusions about the null hypothesis, which can arise based on sample data and
statistical tests.
Type I Error (False Positive)
 Definition: A Type I error occurs when the null hypothesis (H₀) is rejected when it is
actually true. In other words, you mistakenly conclude that there is an effect or a difference
when there isn't one.
 Example: In a clinical trial, a Type I error would occur if you concluded that a new
drug is effective (rejecting the null hypothesis that the drug has no effect) when in
reality the drug has no effect.
 Consequences: A Type I error is often considered more serious because it leads to the
incorrect belief that something has an effect or relationship, even though it doesn't. This
might lead to false conclusions, potentially causing unnecessary changes or actions based on
incorrect findings (e.g., approval of ineffective treatments, unnecessary policies, etc.).
 Significance Level (α\alpha): The probability of committing a Type I error is denoted by α\
alpha, which is also called the significance level of the test. Common values for α\alpha are
0.05 (5%), 0.01 (1%), etc. A 5% significance level means that there's a 5% chance of making
a Type I error if the null hypothesis is true.
 Example: If α=0.05\alpha = 0.05, you are willing to accept a 5% chance of wrongly
rejecting the null hypothesis.
Type II Error (False Negative)
 Definition: A Type II error occurs when the null hypothesis (H₀) is not rejected when it is
actually false. In other words, you fail to detect a true effect or difference.
 Example: In the context of a clinical trial, a Type II error would occur if you
concluded that a new drug has no effect (failing to reject the null hypothesis that the
drug has no effect) when, in reality, the drug is effective.
 Consequences: A Type II error is problematic because it leads to missing out on a real effect
or relationship, which could have been beneficial (e.g., failing to approve an effective drug,
missing a potential scientific discovery, etc.).
  Beta (β\beta): The probability of committing a Type II error is denoted by β\beta. The
power of a statistical test is defined as 1−β1 - \beta, which represents the probability of
correctly rejecting the null hypothesis when it is false. Higher power indicates a lower
chance of making a Type II error.
 Example: A study with 80% power (β=0.20\beta = 0.20) means there is an 80%
chance of correctly rejecting the null hypothesis when it is false, and a 20% chance
of making a Type II error.

The Relationship Between Type I and Type II Errors

There is a trade-off between Type I and Type II errors: reducing one often increases the other.
Here's how:
1. Decreasing α\alpha (lowering the significance level) reduces the probability of a Type I
error, but this also increases the probability of a Type II error because you become more
conservative in rejecting the null hypothesis.
 Example: If you lower α\alpha from 0.05 to 0.01 (stricter criteria for rejecting the
null hypothesis), you reduce the risk of a Type I error, but it becomes harder to detect
true effects, increasing the risk of a Type II error.
2. Increasing sample size: Larger sample sizes generally reduce both Type I and Type II
errors. This is because larger samples provide more precise estimates of the population
parameters, increasing the likelihood of detecting true effects and making the results more
reliable.
3. Effect Size: The size of the effect you're trying to detect also influences Type I and Type II
errors. Larger effects are easier to detect, reducing the chance of a Type II error, but smaller
effects are harder to detect, increasing the chance of a Type II error.

Visualizing Type I and Type II Errors (Decision Making)

Imagine you're testing whether a new drug improves patient recovery times. Here's how Type I and
Type II errors relate to the decision-making process:
 Type I Error: You conclude the drug works (reject H₀) when it actually does not (H₀ is
true).
 Type II Error: You conclude the drug does not work (fail to reject H₀) when it actually does
work (H₀ is false).

Decision Null Hypothesis (H₀) True State of Nature

Drug does not work (H₀ is
Reject H₀ (Drug works) Type I Error (False Positive)
true)
Fail to Reject H₀ (Drug does not Type II Error (False
Drug works (H₀ is false)
work) Negative)

Example Scenarios:
1. Medical Testing:
 Type I error: A medical test incorrectly identifies a healthy person as sick (false
positive), leading to unnecessary treatment.
  Type II error: The test fails to identify a sick person, leading to no treatment or
diagnosis (false negative).
2. Drug Testing:
 Type I error: Approving a drug as effective when it actually has no effect, leading to
potentially harmful or ineffective treatment being sold.
 Type II error: Failing to approve a drug that is actually effective, causing a delay in
patient care or lost opportunity for effective treatment.
3. Quality Control in Manufacturing:
 Type I error: Rejecting a batch of products that is actually of good quality (false
positive).
 Type II error: Accepting a batch of products that is defective (false negative).

Minimizing Type I and Type II Errors:

1. Increase Sample Size: Larger sample sizes increase the precision of estimates and decrease
both types of errors.
2. Choose an Appropriate Significance Level: Depending on the context, you may want to
adjust α\alpha. For example, in medical tests where false positives (Type I errors) can be
very harmful, you may choose a smaller α\alpha, like 0.01, to minimize the risk of
approving ineffective treatments.
3. Improve Study Design: Ensuring good experimental design and using proper statistical
methods can reduce both errors. For example, using a well-designed random controlled trial
can help reduce bias and increase the power of the test.

Conclusion:
 Type I Error: Rejecting a true null hypothesis (false positive).
 Type II Error: Failing to reject a false null hypothesis (false negative).
Understanding and balancing Type I and Type II errors is critical in designing experiments and
making informed decisions based on statistical tests. Researchers aim to control both errors through
appropriate study design, sample size, significance level, and power analysis to ensure reliable and
meaningful results.

Level of Significance
Level of Significance (α\alpha)
The level of significance (denoted as α\alpha) is a key concept in hypothesis testing. It represents
the threshold for deciding whether the results of a statistical test are statistically significant or not.
In other words, it is the probability of rejecting the null hypothesis (H₀) when it is actually true
(i.e., the probability of making a Type I error).

Key Points about α\alpha:

1. Threshold for Rejecting the Null Hypothesis:
  The level of significance is the probability of making a Type I error (false
positive), which means rejecting a null hypothesis that is actually true.
 If the p-value from your statistical test is less than or equal to α\alpha, you reject
the null hypothesis and conclude that the result is statistically significant.
 If the p-value is greater than α\alpha, you fail to reject the null hypothesis,
meaning there is not enough evidence to support the alternative hypothesis.
2. Common Values of α\alpha:
 0.05 (5% level of significance): This is the most commonly used significance level. It
means you are willing to accept a 5% chance of rejecting the null hypothesis when it
is actually true.
 0.01 (1% level of significance): Used when you want to be more stringent and
minimize the chance of a Type I error.
 0.10 (10% level of significance): Occasionally used when researchers are willing to
accept a higher risk of Type I errors, especially in exploratory research.
3. Interpretation of α\alpha:
 α=0.05\alpha = 0.05 means that you are allowing a 5% chance of making a Type I
error (incorrectly rejecting the null hypothesis when it is true).
 In other words, if you run a statistical test 100 times under the same conditions, you
would expect to reject the null hypothesis incorrectly (Type I error) about 5 times
out of 100.

Relationship Between P-value and Level of Significance

The p-value is compared to the level of significance α\alpha to make a decision in hypothesis
testing. Here's how the comparison works:
 If p-value ≤ α\alpha:
 Reject the null hypothesis (H0H_0).
 Conclude that there is sufficient evidence to support the alternative hypothesis
(H1H_1).
 The result is considered statistically significant.
 If p-value > α\alpha:
 Fail to reject the null hypothesis (H0H_0).
 Conclude that there is not enough evidence to support the alternative hypothesis
(H1H_1).
 The result is considered not statistically significant.

Example:
Imagine you are testing whether a new drug has a statistically significant effect on blood pressure,
compared to a placebo.
 Null hypothesis (H₀): The new drug has no effect on blood pressure (the mean difference
between the drug and placebo groups is zero).
 Alternative hypothesis (H₁): The new drug does have an effect on blood pressure (the mean
difference is not zero).
Suppose you choose a significance level of α=0.05\alpha = 0.05.
  After conducting the statistical test (e.g., a t-test), you get a p-value of 0.03.
 Since p-value (0.03) < α\alpha (0.05), you reject the null hypothesis and conclude
that the drug has a statistically significant effect on blood pressure.
 Alternatively, if you get a p-value of 0.08:
 Since p-value (0.08) > α\alpha (0.05), you fail to reject the null hypothesis, meaning
there is not enough evidence to support the claim that the drug has an effect on blood
pressure.

Choosing the Right Level of Significance:

1. Context of the Study: The choice of α\alpha depends on the context of the research and the
consequences of Type I errors. For example:
 In medical research, where false positives could lead to approving an ineffective or
harmful drug, a lower α\alpha (e.g., 0.01) is often chosen to minimize the risk of
Type I errors.
 In exploratory or preliminary research, a higher α\alpha (e.g., 0.10) might be
chosen to reduce the risk of Type II errors (false negatives), especially when the
focus is on detecting any potential effects.
2. Balancing Type I and Type II Errors:
 The choice of α\alpha is related to the balance between Type I errors (false
positives) and Type II errors (false negatives). Lowering α\alpha (making it more
stringent) reduces the probability of a Type I error but increases the probability of a
Type II error.
 Similarly, increasing α\alpha reduces the probability of a Type II error but increases
the probability of a Type I error.
3. Power of the Test:
 The power of a statistical test is the probability of correctly rejecting the null
hypothesis when it is false. It is related to the sample size, effect size, and the level of
significance.
 A larger sample size and a higher significance level (α\alpha) generally increase the
power of a test, making it easier to detect a true effect.

Visualizing the Level of Significance in a Hypothesis Test

Consider a normal distribution curve:
 Critical region (rejection region): The part of the distribution where we would reject the
null hypothesis. For example, if α=0.05\alpha = 0.05, the critical region corresponds to the
5% of the extreme values on the tail(s) of the distribution.
 If the p-value falls within the critical region (i.e., the extreme 5% in a two-tailed test), you
reject the null hypothesis.
 If the p-value falls outside the critical region (i.e., within the 95% middle portion), you fail
to reject the null hypothesis.
Here’s how it looks visually for a two-tailed test:
 Rejection Region (Critical Region) Non-Rejection Region (Fail to Reject)
α=0.05\alpha = 0.05 (total area in the tails) 95% of the area (centered around the mean)
p-value ≤ 0.05: Reject H₀ p-value > 0.05: Fail to reject H₀

Summary:
 Level of Significance (α\alpha): The threshold probability for rejecting the null hypothesis.
It is the probability of making a Type I error (false positive).
 Common Values: 0.05, 0.01, 0.10, depending on the field of study and consequences of
errors.
 Decision Rule: Compare the p-value with α\alpha:
 If p ≤ α\alpha, reject the null hypothesis.
 If p > α\alpha, fail to reject the null hypothesis.
 Choosing α\alpha: Depends on the desired balance between Type I and Type II errors and
the consequences of making those errors in the context of the research.
The level of significance plays a crucial role in the hypothesis testing framework and helps
determine whether the observed data provides sufficient evidence to reject the null hypothesis.
Regression Analysis

Model Evaluation in Machine Learning

Model evaluation refers to the process of assessing the performance of a machine learning model.
The goal is to understand how well the model is generalizing to unseen data, which can be
measured through various metrics, visualizations, and techniques depending on the type of problem
(e.g., classification, regression, etc.).

Why Model Evaluation is Important

1. Assessing Accuracy: It helps in understanding whether the model is accurate enough for the
task.
2. Avoiding Overfitting or Underfitting: Evaluation allows you to detect overfitting (model
is too complex) or underfitting (model is too simple).
3. Comparing Models: It provides a standard way to compare different models or algorithms
to select the best performing one.
4. Improving Model: Evaluation metrics give you insights into which aspects of the model
need improvement (e.g., precision, recall, etc.).

Steps in Model Evaluation

1. Data Split:
 Typically, you split your data into at least two sets: training and testing datasets
(sometimes a validation set is also used).
 Training Set: Used to train the model.
 Testing Set: Used to evaluate the model’s performance on unseen data
(generalization).
 Validation Set (optional): Sometimes used to tune hyperparameters during training.
2. Model Training: Train the model using the training data.
3. Prediction: Use the trained model to make predictions on the testing (or validation) set.
4. Evaluation: Compare the model’s predictions to the actual outcomes and use appropriate
evaluation metrics to quantify the performance.

Evaluation Metrics for Different Types of Problems

1. Classification Metrics
In classification tasks (e.g., spam detection, image classification, etc.), the goal is to predict discrete
class labels. Some common metrics include:
1. Accuracy:
 Definition: The proportion of correct predictions out of the total predictions.
  Formula:
Positives + True Negatives SamplesAccuracy=Total SamplesTrue Positives + True N
egatives
 Limitations: Accuracy can be misleading in imbalanced datasets where one class is
much more frequent than the other.
2. Confusion Matrix:
 A confusion matrix is a table that describes the performance of a classification
model, showing the counts of True Positives (TP), True Negatives (TN), False
Positives (FP), and False Negatives (FN).
 The confusion matrix helps compute several other important metrics like precision,
recall, and F1 score.
Predicted Positive Predicted Negative
Actual Positive TP (True Positive) FN (False Negative)
Actual Negative FP (False Positive) TN (True Negative)
3. Precision:
 Definition: The proportion of positive predictions that are actually correct.
 Formula:
Positives Positives + False PositivesPrecision=True Positives + False PositivesTrue P
ositives
 Use Case: Precision is important when the cost of a false positive is high (e.g., in
fraud detection, where wrongly identifying a legitimate transaction as fraudulent is
costly).
4. Recall (Sensitivity, True Positive Rate):
 Definition: The proportion of actual positives that are correctly identified.
 Formula:
Positives Positives + False NegativesRecall=True Positives + False NegativesTrue P
ositives
 Use Case: Recall is critical when the cost of a false negative is high (e.g., in medical
diagnoses, where missing a disease diagnosis could be dangerous).
5. F1-Score:
 Definition: The harmonic mean of precision and recall. It balances the trade-off
between precision and recall.
 Formula:

 Use Case: The F1 score is useful when you need a balance between precision and recall and
there is an uneven class distribution.
 ROC Curve and AUC (Area Under the Curve):
 ROC Curve: A graphical representation of the trade-off between true positive rate
(sensitivity) and false positive rate (1 - specificity) at various thresholds.
  AUC: The area under the ROC curve. AUC ranges from 0 to 1, with 1 indicating a perfect
model and 0.5 indicating a random model.
 Use Case: AUC is a good metric when you want to compare classifiers and their ability to
distinguish between classes.
 Specificity (True Negative Rate):
 Definition: The proportion of actual negatives that are correctly identified.
 Formula:

1. Use Case: Specificity is important in cases where the focus is on avoiding false positives.
2. Regression Metrics
In regression tasks (e.g., predicting house prices, stock prices, etc.), the goal is to predict continuous
numeric values. Some common metrics include:
1. Mean Absolute Error (MAE):
 Definition: The average of the absolute differences between predicted and actual
values.
 Formula:

 Use Case: MAE is interpretable and gives a clear measure of the average error.
 Mean Squared Error (MSE):
 Definition: The average of the squared differences between predicted and actual values.
 Formula:

 Use Case: MSE penalizes larger errors more significantly than MAE due to the squaring of
the error terms.
 Root Mean Squared Error (RMSE):
 Definition: The square root of the mean squared error.
 Formula:
  Use Case: RMSE is sensitive to large errors and provides an interpretable error measure in
the same units as the target variable.
 R-squared (R2):
 Definition: The proportion of variance in the dependent variable that is predictable from the
independent variables. R-squared is a measure of how well the model fits the data.
 Formula:

 Use Case: A higher R-squared value indicates that the model explains a larger proportion of
the variance in the target variable.
 Adjusted R-squared:
 Definition: A modification of R2 that adjusts for the number of predictors in the model,
preventing overfitting.
 Formula:

1. Where:
 n is the number of observations,
 p is the number of independent variables.
 Use Case: Adjusted R2 is particularly useful when comparing models with different
numbers of predictors.
3. Cross-Validation
Cross-validation is a technique used to assess the generalization ability of a model. It involves
splitting the data into multiple subsets (folds), training the model on some subsets, and testing it on
others. The most common method is k-fold cross-validation, where the data is split into k subsets,
and the model is trained and tested k times, each time using a different fold for testing.
 Benefits: Cross-validation helps mitigate the bias that might arise from a single train-test
split, giving a more reliable estimate of model performance.
Other Considerations
 Bias-Variance Trade-off: Model evaluation helps identify whether the model is suffering
from bias (underfitting) or variance (overfitting).
 Learning Curves: Plotting the model’s training and validation errors as the number of
training samples increases can provide insights into how the model is learning and whether
it needs more data or hyperparameter tuning.
 Hyperparameter Tuning: Evaluation metrics can guide you in tuning model
hyperparameters (e.g., regularization strength, number of trees in a random forest, etc.) to
improve performance.