AIDS TE Statistics for Artificial Intelligence Data
Science
Dr. Pooja Singh
Assistant Professor
Dept. of Humanities & Applied Sciences,
SIES Graduate School of Technology
[Link] Singh
• Methods Based on the Cumulative Distribution Function (CDF)
are foundational in descriptive statistics and probability theory
for understanding and visualizing the distribution of data.
• These methods help in analyzing both empirical data and
theoretical distributions
[Link] Singh
Here are some key methods based on CDFs:
• Cumulative Distribution Function (CDF)
• Empirical Cumulative Distribution Function (ECDF)
• Survival Function (Complement of CDF)
• Quantile Function (Inverse CDF)
• Quantile-Quantile (Q-Q) Plots
[Link] Singh
[Link] Singh
● The Cumulative Distribution Function (CDF) is a fundamental concept in
probability and statistics. It describes the probability that a random variable
takes a value less than or equal to a given number.
[Link] Singh
What is a Cumulative Distribution Function?
• The Cumulative Distribution Function (CDF), of a real-valued random
variable X, evaluated at x, is the probability function that X will take a
value less than or equal to x. It is used to describe the probability
distribution of random variables in a table.
• And with the help of these data, we can easily create a CDF plot in an
excel sheet.
[Link] Singh
Cumulative distribution function
• In Probability and Statistics, the Cumulative Distribution Function (CDF) of
a real-valued random variable, say “X”, which is evaluated at x, is the
probability that X takes a value less than or equal to the x.
• A random variable is a variable that defines the possible
outcome values of an unexpected phenomenon. It is defined for both
discrete and random variables. It is also used to specify the distribution of
the multivariate random variables.
• If the random variable is above a particular level, it is known as tail
distribution or the Complementary Cumulative Distribution Function
(CCDF).
[Link] Singh
CDF for a Discrete Random Variable:
• The CDF defined for a discrete random variable and is given as
• This gives the probability that the random variable X takes on a value less
than or equal to x.
• The variable X lies within a semi-closed interval (a,b],where a<b.
• Therefore the probability within the interval is written as
P(a < X ≤ b) = FX(b) – FX(a)
[Link] Singh
CDF for a Continuous Random Variable:
The CDF defined for a continuous random variable is given as
where
is the probability density function (PDF) of the continuous random variable X.
2. The CDF represents the area under the PDF curve from −∞ to x.
3. For an interval (a,b], the probability is:
P(a < X ≤ b) = FX(b) – FX(a)
Note:(Same as in the discrete case, but calculated via integration.)
[Link] Singh
Examples of CDF for a Discrete Random Variable:
[Link] Singh
[Link] Singh
[Link] Singh
Example
• Example for CDF which is given by rolling a fair six-sided die, where X is the
random variable
• We know that the probability of getting an outcome by rolling a six-sided die is
given as:
• Probability of getting 1 = P(X≤ 1 ) = 1 / 6
• Probability of getting 2 = P(X≤ 2 ) = 2 / 6
• Probability of getting 3 = P(X≤ 3 ) = 3 / 6
• Probability of getting 4 = P(X≤ 4 ) = 4 / 6
• Probability of getting 5 = P(X≤ 5 ) = 5 / 6
• Probability of getting 6 = P(X≤ 6 ) = 6 / 6 = 1
• From this, it is noted that the probability value always lies between 0 and 1
and it is non-decreasing and right continuous in nature.
[Link] Singh
Examples of CDF for a Continuous Random Variable
[Link] Singh
[Link] Singh
[Link] Singh
[Link] Singh
[Link] Singh
[Link] Singh
Empirical cumulative distribution function
• Use an empirical cumulative distribution function plot to display the data points in
your sample from lowest to highest against their percentiles.
• These graphs require continuous variables and allow you to derive percentiles and
other distribution properties. This function is also known as the empirical CDF or
ECDF.
[Link] Singh
• If you measure the same characteristic in multiple samples, you can use
empirical CDF plots to compare the sample distributions.
• Optionally, your software can display the fitted cumulative distribution
function so you can compare how well the empirical distribution follows
the fitted distribution.
• The fitted distribution uses parameters estimated from your data.
• Unlike a Q-Q plot, your statistical software does not transform the axes to
create a straight line for a cumulative distribution function.
[Link] Singh
[Link] Singh
[Link] Singh
[Link] Singh
[Link] Singh
[Link] Singh
[Link] Singh
[Link] Singh
[Link] Singh
[Link] Singh
[Link] Singh
[Link] Singh
The blue stepped line is the empirical CDF function and the red
curve is the fitted CDF for the normal distribution.
[Link] Singh
The Survival Function
[Link] Singh
[Link] Singh
What is survival analysis?
• Survival analysis (also called time-to-event analysis or duration analysis) is a
branch of statistics aimed at analyzing the expected duration of time until one or
more events happen, called survival times or duration times.
• In survival analysis, we are interested in a certain event and want to analyze the
time until the event happens.
• While the event of interest is often death (in this case we study the time to death
for patients having a specific disease) or recurrence (in this case we study the time
to relapse of a certain disease).
[Link] Singh
Example
We may also analyze the time until:
• Getting cured from a certain disease
• Finding a new job after a period of unemployment
• Being arrested again after being released from jail
• The first pregnancy
• The failure of a mechanical system or a machine
• A bank or a company goes bankrupt
• A customer buys a new product or stops its current subscription
• A letter is delivered
• A taxi picks you up after having called the taxi company
• An employee leaves the company
etc.
[Link] Singh
• Survival data, also referred as time-to-event data, requires a special set of statistical methods for three
main reasons:
• Duration times are always positive: the time until an event of interest takes place cannot be less than 0.
• Different measures are of interest depending on the research question, context, etc. For instance, we
could be interested in:
– The probability that a cancer patient survives longer than 5 years?
– The typical waiting time for a cab to arrive after having called the taxi company?
– How many, out of 100 unemployed people, are expected to have a job again after 2 months?
• Censoring is almost always an issue:
– Most of the time, the event occurred before the end of the study and the survival duration is known.
– However, sometimes, the event is not yet observed at the end of the study. Suppose that we study
the time until death of patients with breast cancer. Luckily, some patients will not die before the end
of the study.
– Other times, another event takes place before the event of interest. For example, a cancer patient
may die from a car accident.
– And even sometimes the patient withdraws from the study or moves to another country so we
cannot observe her survival time (this is known as lost to follow up or drop out).
– Censoring can be seen, in some sense, as a type of missing data.
– For these reasons, many “standard” descriptive statistics, hypothesis tests and regression models are
not appropriate for this kind of data.
[Link] Singh
[Link] Singh
[Link] Singh
[Link] Singh
[Link] Singh
[Link] Singh
[Link] Singh
[Link] Singh
[Link] Singh
[Link] Singh
[Link] Singh
[Link] Singh
[Link] Singh
[Link] Singh
[Link] Singh
Quantile Function
• The Quantile Function, in probability and statistics, is the inverse of a
distribution function.
• It maps a probability value to the corresponding value of a random
variable within a given distribution.
• Essentially, it tells you the value below which a given percentage of
observations fall.
• For example, the 0.95 quantile (or 95th percentile) is the value below
which 95% of the data points lie.
[Link] Singh
Quantile Function
• What is the Quantile Function?
• The quantile function is the inverse of the cumulative distribution
function (CDF).
[Link] Singh
Quantile Function
[Link] Singh
[Link] Singh
[Link] Singh
[Link] Singh
[Link] Singh
[Link] Singh
[Link] Singh
Quantile Function
[Link] Singh
[Link] Singh
[Link] Singh
[Link] Singh
[Link] Singh
[Link] Singh
[Link] Singh
[Link] Singh
[Link] Singh
[Link] Singh
[Link] Singh
[Link] Singh
[Link] Singh
[Link] Singh
[Link] Singh
[Link] Singh
[Link] Singh
[Link] Singh
[Link] Singh
[Link] Singh
[Link] Singh
[Link] Singh
[Link] Singh
[Link] Singh
Quantile-Quantile (Q-Q) plot
• What is a Q-Q Plot?
• A Quantile-Quantile (Q-Q) plot is a graphical tool used to
assess whether a dataset follows a specific theoretical
distribution — typically, the normal distribution.
[Link] Singh
[Link] Singh
Q-Q plot
• Q-Q plots are also known as Quantile-Quantile plots. As the
name suggests, they plot the quantiles of a sample
distribution against quantiles of a theoretical distribution.
• Doing this helps us to determine if a dataset follows any
particular type of probability distribution like normal, uniform,
exponential.
[Link] Singh
• Q Q Plots (Quantile-Quantile plots) are plots of two quantiles against each
other.
• A quantile is a fraction where certain values fall below that quantile.
• For example, we mean the fraction (or percent) of points below the given
value. That is, the 0.3 (or 30%) quantile is the point at which 30% percent of
the data fall below and 70% fall above that value.
• A 45-degree reference line is also plotted. If the two sets come from a
population with the same distribution, the points should fall approximately
along this reference line. The greater the departure from this reference line,
the greater the evidence for the conclusion that the two data sets have
come from populations with different distributions.
[Link] Singh
How to Make a Q Q Plot
A Q-Q plot is used to compare the quantiles of a
data distribution with the quantiles of a
theoretical distribution (e.g., normal). It helps
you assess whether your data follows that
distribution.
[Link] Singh
[Link] Singh
[Link] Singh
[Link] Singh
[Link] Singh
Steps:
• Step 1: Sort the Data
[Link] Singh
Step 2: Determine Quantile Positions
[Link] Singh
[Link] Singh
[Link] Singh
How to Make a Q Q Plot
Question: Do the following values come from a normal
distribution?
7.19, 6.31, 5.89, 4.5, 3.77, 4.25, 5.19, 5.79, 6.79.
[Link] Singh
Step 1: Order the items from smallest to largest.
• 3.77
• 4.25
• 4.50
• 5.19
• 5.79
• 5.89
• 6.31
• 6.79
• 7.19
[Link] Singh
[Link] Singh
Step 3: Find the z-value (cut-off point) for each segment in Step 3. These
segments are areas, so refer to a z-table to get a z-value for each
segment.
The z-values are:
• 10% = -1.28
• 20% = -0.84
• 30% = -0.52
• 40% = -0.25
• 50% = 0
• 60% = 0.25
• 70% = 0.52
• 80% = 0.84
• 90% = 1.28
• 100% = 3.0
[Link] Singh
[Link] Singh
Measures of Location
• Measures of Location (also known as
measures of central tendency) describe the
central point or typical value of a dataset.
They help summarize a large set of data with a
single representative value.
[Link] Singh
[Link] Singh
[Link] Singh
Density Curves
• A density curve is a curve on a graph that represents the
distribution of values in a dataset. It’s useful for three reasons:
1. A density curve gives us a good idea of the “shape” of a distribution,
including whether or not a distribution has one or more “peaks” of
frequently occurring values and whether or not the distribution is skewed
to the left or the right.
2. A density curve lets us visually see where the mean and the median of a
distribution are located.
3. A density curve lets us visually see what percentage of observations in
a dataset fall between different values.
[Link] most famous density curve is the bell-shaped curve that represents
the normal distribution.
[Link] Singh
Example: Creating & Interpreting a Density Curve
• Suppose we have the following dataset that
shows the height of 20 different plants (in
inches) in a certain field:
• 4, 5, 5, 6, 6, 6, 6, 7, 7, 7, 7, 7, 8, 8, 8, 9, 9, 9, 2,
2
• If we created a simple histogram to display the
relative frequencies of each value,
[Link] Singh
And if we created a density curve to capture
the “shape” of this distribution, it would look
like this:
The x-axis shows the data value
and the y-axis shows the relative
frequency (e.g. the value “7” The curve is highest near the center of the
occurs 5 times out of 20 total distribution because that’s where the
values in the dataset, thus it has a most values are located. It’s also lowest
relative frequency of 25% or 0.25. near the ends of the distribution because
fewer plants take on those values (e.g. a
height of 4 inches or 10 inches).
[Link] Singh
• Density curves come in all shapes and sizes
and they allow us to gain a quick visual
understanding of the distribution of values in
a given dataset. In particular, they’re useful for
helping us visualize:
• Density curves allow us to quickly see whether
or not a graph is left skewed, right skewed, or
has no skew:
[Link] Singh
1. Skewness
[Link] Singh
2. The location of the mean & median
• Depending on the skewness of a density curve, we can quickly know
whether the mean or median is larger in a given distribution. In particular:
• If a density curve is left skewed, then the mean is less than the median.
• If a density curve is right skewed, then the mean is greater than the
median.
• If a density curve has no skew, then the mean is equal to the median.
[Link] Singh
3. Number of Peaks
• Density curves also allow us to quickly see how
many “peaks” there are in a given distribution. In
each of the examples above, the distributions only
had one peak, so we would describe those
distributions as unimodal.
• However, some distributions can have two peaks
which we call bimodal distributions. And in rare
cases we can also
have multimodal distributions that have two or
more peaks.
[Link] Singh
What are Stem and Leaf Plots?
• A stem and leaf plot, also known as a stem and leaf diagram, is
a way to arrange and represent data so that it is simple to see
how frequently various data values occur. It is a plot that
displays ordered numerical data.
• A stem and leaf plot is shown as a special table where the digits
of a data value are divided into a stem (first few digits) and a
leaf (usually the last digit). The symbol ‘|’ is used to split and
illustrate the stem and leaf values.
• For instance, 105 is written as 10 on the stem and 5 on the leaf.
This can be written as 10 | 5. Here, 10 | 5 = 105 is called the
key. The key depicts the data value a stem and leaf represent.
[Link] Singh
How do we Construct a Stem and Leaf Plot?
Step 1: Classify the data values in terms of the number of digits in each value,
such as 2 digit numbers or 3 digit numbers.
Step 2: Fix the key for the stem and leaf plot. For example, 2 | 5 = 25, 3 | 2 =
3.2 or 19 | 2 is 192.
Step 3: Consider the first digits as stems and the last digit as leaves.
Step 4: Find the range of the data, that is the lowest and the highest values
among the data.
[Link] Singh
Step 5: Draw a vertical line. Place the stem on the left and the
leaf on the right of the vertical line.
Step 6: List the stems in the stem column. Sort them in
ascending order.
Step 7: List the leaf values in the column against the stem from
lowest to the highest horizontally.
[Link] Singh
[Link] Singh
Example 1 :
1. 26, 37, 48, 33, 49, 26, 19, 26, 48
• Solution :
• Arrange the given data set in ascending order.
• 19, 26, 26, 26, 33, 37, 48, 48, 49
[Link] Singh
Example
2. A test score out of 60 marks is recorded for a group of 45
students :
[Link] Singh
[Link] Singh
Solution
[Link] Singh
[Link] Singh
[Link] Singh
[Link] Singh
• The table below shows the duration of calls that Rosy
makes each day. Represent the given data using a
stem and leaf plot.
[Link] Singh
Step 1: Sort the data (number of minutes).
• 2, 3, 5, 6, 10, 14, 19, 23, 23, 30, 36, 56
Step 2: Choose the stems and the leaves. Just because the
data values range from 2 to 56, use the tens digit for the
stem and the ones digit for the leaf. Also, include the key.
Step 3: Write down the stems on the left of the vertical
line.
Step 4: Write down the leaves for each stem on the right of
the vertical line.
[Link] Singh
[Link] Singh
[Link] Singh
[Link] Singh
Thank You
(poojasingh@[Link])
[Link] Singh