Module – 1

Introduction to Statistics & Data

1) What are Statistics?
• Statistics is the science of collecting, organizing, summarizing, and drawing
conclusions from data.

• Two broad branches:

o Descriptive statistics: Organize and summarize data (tables, graphs, measures

like mean/median, range/SD).

o Inferential statistics: Use a sample to make conclusions about a population

(confidence intervals, hypothesis tests, correlation).

2) Population vs. Sample; Parameter vs. Statistic

• Population: Entire group you want to study (e.g., all customers this year).

• Sample: Subset taken from the population (e.g., 200 customers).

• Parameter: A numerical summary of a population (unknown in practice; e.g., true

average spending).

• Statistic: A numerical summary of a sample (computed; e.g., average of 200

customers).

Goal: Use statistics (sample measures) to learn about parameters (population measures).

3) Variables and Types of Data

A variable is any characteristic that varies across individuals or items.

(A) Qualitative / Categorical

• Nominal: Labels only; no natural order.

Examples: Gender, blood group, city name, product category.

• Ordinal: Categories with a meaningful order, but gaps not equal.

Examples: Star ratings (★–★★★★★), education levels (Primary < Secondary <
Graduate).

(B) Quantitative / Numerical

• Discrete: Countable values (usually integers).

Examples: Number of calls received, defect counts, goals scored.
 • Continuous: Measurable on a continuum; can take any value in an interval.
Examples: Height, weight, time, temperature (as measurement).

4) Scales of Measurement (how data are recorded)

• Nominal: Names/labels only (colour, brand).
Allowed operations: =, ≠; mode is meaningful; bar/pie charts.

• Ordinal: Order matters, but differences aren’t equal (rankings).

Allowed operations: comparisons (<, >); median often used.

• Interval: Ordered with equal differences; no true zero.

Example: Temperature in °C/°F. Can add/subtract; mean/SD OK; ratios (twice as hot)
are not meaningful.

• Ratio: Interval + true zero; ratios are meaningful.

Examples: height, weight, time, sales amount. All arithmetic valid; geometric mean
possible.

5) Data Collection (brief, theory)

• Survey (questionnaires, interviews): Gets opinions/attributes; risk of response bias.

• Observational study: Observe without intervention; shows association, not causation.

• Experiment: Researcher assigns treatments; best for establishing causation with

controls and randomization.

6) Organizing Data
• Raw data → Frequency table: value counts.

• Grouped data (for continuous variables): class intervals, class width, midpoints.

• Relative frequency: proportion; cumulative frequency: running total.

7) Common Graphs (what to use for which data)

• Bar chart: Categorical (nominal/ordinal). Bars separated.

• Pie chart: Categorical shares of a whole (use sparingly).

• Histogram: Quantitative (continuous/discrete with many values). Bars touch; shows

shape (skew, modal peaks).

• Frequency polygon: Connect midpoints of histogram bars; compares distributions.

• Ogive (cumulative curve): Cumulative frequencies; useful for medians/percentiles.

• Box plot: Shows median, quartiles, possible outliers (quick shape + spread view).

Quick distinction: Bar (categories, gaps) vs Histogram (numeric bins, no gaps).

Measures of Central Tendency (Averages)
1) What “central tendency” means
• Central tendency describes the typical value around which the data cluster.

• Main measures (textbook standard):

o Arithmetic Mean (Mean)

o Median (positional average)

o Mode (most frequent value)

Why it matters: Picking the right average makes summaries fair and readable, especially when
data are skewed or contain outliers.

2) Mean (Arithmetic Mean)

2.1 Idea (plain language)

• Add up all values and divide by how many values.

• Uses every observation → very informative, but sensitive to outliers.

2.2 Where it’s appropriate

• Data are roughly symmetric, with no extreme outliers.

• Interval/ratio scale data (true numeric).

2.3 Variants you must know

• Simple mean: for raw data.

• Weighted mean: when each value has a weight or frequency (e.g., class sizes differ).

• Combined mean: when merging two or more groups (use group totals & sizes).

• Trimmed mean (mention only): drop small % of extreme values before averaging
(reduces outlier impact).

2.4 Merits / Demerits

• Merits: uses all data; algebra-friendly; good for further analysis.

• Demerits: pulled by skew/outliers; not suitable for open-ended class intervals without
care; can be misleading for ordinal data.

3) Median (Positional Average)

3.1 Idea

• The middle value when data are sorted.

 • Splits data into two equal halves.

• Robust to outliers/skew.

3.2 Where it’s appropriate

• Skewed distributions (e.g., incomes, waiting times).

• When outliers are present.

• For ordinal data (order matters).

3.3 Textbook cases to recognize

• Ungrouped data (odd/even n): middle item / average of the two middle items.

• Discrete grouped data (by frequency): locate the (n/2)th item in cumulative counts.

• Continuous grouped data: identify the median class (where cumulative frequency
crosses n/2) and use the standard median formula (you’ll apply it when numericals
are allowed).

3.4 Merits / Demerits

• Merits: not affected much by outliers; easy to explain; works with open-ended
intervals (with care).

• Demerits: ignores exact distances between all values; less algebraic use than mean.

3.5 Boxplot cue (median highlighted)

4) Mode (Most Frequent Value)

4.1 Idea
 • The value (or class interval) that occurs most often.

4.2 Where it’s appropriate

• Categorical data (most common category).

• Discrete data with a clear peak.

• Grouped continuous data → the modal class is the one with the highest frequency
bar in the histogram.

4.3 Merits / Demerits

• Merits: the only average definable for nominal data; intuitive (“most typical”).

• Demerits: may be ill-defined (no clear single peak or multiple modes); unstable in
small samples.

5) Choosing the right average (decision guide)

Recommended
Situation Reason
measure

Symmetric numeric data, no

Mean (± SD) Uses all data; efficient
outliers

Skewed numeric data / outliers Median (± IQR) Robust to extremes

Categorical/nominal data Mode / Proportions “Most common” category

Honors order, avoids false

Ordinal data (ranks, Likert) Median / Percentiles
precision

Open-ended class intervals Median / Mode Mean may be unreliable

Accounts for different group

Need fair weight by size Weighted Mean
sizes

Golden rule: Right-skewed or outlier-heavy → Median & IQR.

Mode is essential for categorical summaries.

6) Relationship among mean, median, mode (empirical note)

• Symmetric distribution: mean ≈ median ≈ mode.

• Right-skewed: mean > median > mode.

• Left-skewed: mean < median < mode.

 • (Classical empirical relation sometimes stated as Mode ≈ 3×Median − 2×Mean for
moderately skewed data—use as a rough guide only.)

7) Reporting guidelines (how to write in notes/reports)

• Always name the measure you’re reporting and why it suits the data shape.

• Pair the average with a spread:

o Mean → SD; Median → IQR.

• If data are transformed (e.g., log scale), say so: “median (log-scale).”

• If categories are unequal in size, report proportions along with mode.

8) Common pitfalls (and what to avoid)

• Reporting mean for heavily skewed data (can mislead).
→ Use median & IQR instead.

• Using mode to summarize continuous data with no clear peak.

→ Prefer median/mean depending on shape.

• Ignoring units and context (e.g., minutes vs hours).

• Forgetting to check data quality before summarizing (outliers, wrong codes).

9) Tiny recap (memory lines)

• Mean: informative but sensitive (use when symmetric, no big outliers).

• Median: robust (use when skewed or outliers present).

• Mode: most common (especially for categorical).

• Pair center with spread: Mean+SD or Median+IQR.

Measures of Dispersion (Variability)

1) Why dispersion matters
Dispersion tells you how spread out the data are around the center. Two datasets can
share the same mean/median but feel very different if one is tightly clustered and the
other is widely scattered.

Same mean, different spread (visuals):

2) Core measures
(a) Range

• Definition: max − min

• Note: Quick but very sensitive to extremes.

(b) Interquartile Range (IQR)

• Definition: Q3 − Q1 (spread of the middle 50%).

• Why use: Robust to skew/outliers; pairs with median.

Boxplot view of median, IQR, outliers:

IQR outlier rule (potential outliers):

Low fence = Q1 − 1.5×IQR; High fence = Q3 + 1.5×IQR
 (c) Variance & Standard Deviation (SD)

• Idea: Average squared deviation from the mean (variance); SD is its square root in
original units.

• Character: Sensitive to outliers; best with roughly symmetric data.

• Interpretation: Larger SD ⇒ observations typically lie farther from the mean.

(d) Coefficient of Variation (CV)

• Definition: SD ÷ mean (unit-free).

• Use: Compare variability across different measurement scales.

(e) MAD (Median Absolute Deviation) — mention

• Definition: Median of |value − median|.

• Character: Very robust spread measure; pairs well with the median.

3) Choosing the right spread

Recommended Pair with
Situation Reason
spread center

Symmetric, no big Standard Deviation Uses all deviations;

Mean
outliers (SD) efficient

Skewed or outlier- Robust against

IQR (± MAD) Median
prone extremes

Comparing across Mean Unit-free

CV
units/scales (context) comparison

Fast but very

Quick total extent Range —
sensitive

Rule of thumb: Median → IQR, Mean → SD.

Skewness & Shape of Distributions

1) What is “shape” and “skewness”?
• Shape describes the overall form of a distribution: symmetry vs skewness, number of
peaks (modality), and tail behavior.

• Skewness indicates asymmetry:

o Right-skewed (positively skewed): long tail to the right.

 o Left-skewed (negatively skewed): long tail to the left.

o Symmetric: left and right sides mirror roughly.

Visuals (copy/paste PNGs):

• Symmetric:

• Right-skewed:

• Left-skewed:
2) Mean–Median–Mode relationship
• Symmetric: mean ≈ median ≈ mode

• Right-skewed: mean > median > mode

• Left-skewed: mean < median < mode

3) What to report for different shapes

Center to Spread to
Shape Why
report report

Symmetric Mean SD Uses all data; efficient when tails are balanced

Right-/Left-
Median IQR Robust to outliers/long tails
skewed

Median IQR (plus Single mean can hide multiple peaks; show
Multi-modal
(often) range) the shape clearly

4) Boxplots for skew

• In a right-skew, the upper whisker and upper half of the box stretch longer; in a left-
skew, the lower side stretches.

• Outlying dots indicate potential outliers (investigate; don’t delete blindly).

Boxplot comparison (three shapes):

Exploring Binary & Categorical Data
Binary Data

Overview

• Variables with exactly two outcomes (e.g., Yes/No, Buy/No-Buy, Pass/Fail).

• EDA goal: report the rate (e.g., Yes%) and make fair comparisons across segments (city,
device, section).

Summaries

• Frequency of each outcome (Yes, No).

• Percentage (proportion) for a consistent base (overall or per group).

Prefer percentages over counts when group sizes differ.

Visuals

• Bar chart of Yes% (or both Yes%/No%).

• 100% stacked bars to compare rates across groups on the same 0–100% scale.

Fair Comparisons (Base Selection)

• Row % (within each group): compare rates across groups (“Within Device A, Yes = …%;
within Device B, Yes = …%”).

• Column % (within an outcome): describe who makes up the outcome (“Among

Buyers, …% are A and …% are B”).
Reporting

• “Overall Yes rate is …%; within City A it is …%, vs …% in City B.”

• “Among buyers, …% are Device B and …% are Device A.”

Quality Checks

• Consistent labels (e.g., Yes/No not mixed with Y/N).

• Very small groups → percentages unstable (note this or combine tiny strata).

• Show Missing/Unknown as a category if informative.

Pitfalls

• Comparing counts across unequal groups (use %).

• Mixing row% and column% in one statement (always name the base).

• Using numeric plots (e.g., histogram) for binary data.

Categorical Data (Nominal & Ordinal)

Overview

• Nominal: labels with no order (Payment: UPI/Card/Cash; City A/B/C).

• Ordinal: categories with a natural order (Satisfaction: Low < Medium < High; Likert 1–
5).

Summaries

• Frequencies and Percentages per category.

• Mode (most common) is a valid “average” for categorical variables.

• For ordinal, you may add order-aware summaries (e.g., median category, cumulative
Visuals

• Bar chart for counts or percentages (bars are separate).

• Ordered bar chart for ordinal levels (keep Low → Medium → High).

• 100% stacked bars to compare composition across groups on the same 0–100% scale.

• Pie chart only when a clear “parts-of-whole” view is needed; bars are usually clearer.

Ordinal example (keep natural order):

Comparing Across Groups

• Use percentages when groups differ in size (not raw counts).

 • Keep the y-axis consistent across panels/plots.

• Many tiny categories → combine into Other (state this).

Reporting

• “UPI is most common (…%), followed by Card (…%) and Cash (…%).”

• “In Region East, Medium satisfaction is most frequent (…%), with High at …%.”

Quality Checks

• Unify spellings/case (“Card” vs “card”).

• Show Missing/Unknown if informative.

• Do not reorder ordinal levels alphabetically—preserve the logical order.

Pitfalls

• Using means/SD for categories (use proportions/mode instead).

• Using histograms (numeric) for categories (use bar charts).

• Mixing counts in one group with percentages in another (choose one basis and keep it
consistent).

Exploring Two or More Variables:

Numeric ↔ Numeric (Scatter & Correlation)

• Purpose: See the direction (positive/negative), form (linear/curved), strength

(tight/loose), and outliers between two numeric variables.

• When correlation (Pearson’s r) is meaningful: only when the pattern looks roughly
linear and not dominated by outliers. For monotonic but nonlinear relations or ordinal
data, prefer a rank correlation (e.g., Spearman) conceptually.

• Overplotting fixes: with very dense data, use hexbin (darker = denser) or add point
transparency.

Figures :

• Positive linear relationship —

• Curved/nonlinear relationship —

• Outlier influencing correlation —

 • Dense data handled via hexbin —

Points to remember:

• “The scatter shows a positive linear pattern; correlation is appropriate.”

• “The pattern is curved; Pearson’s r would understate the association—use a non-linear

summary or a rank-based view.”

• “One high-leverage outlier pulls the line/correlation—investigate the point.”

• “With many points, a hexbin reveals structure hidden by overplotting.”

Categorical ↔ Numeric (Distributions across groups)

• Purpose: Compare a numeric outcome across categories (e.g., marks across classes).

• Display: Grouped boxplots (or violins) to compare medians, IQRs, and outliers side-
by-side.

• Interpretation: Talk about center (median) and spread (IQR) first; note outliers; avoid
claiming causation.

Figure :
 • Grouped boxplots —

• “Group B has a higher median and wider spread than Group A.”

• “Outliers in Group A—possible data issues or genuine extremes; note and investigate.”

Categorical ↔ Categorical (Contingency & Percentages)

• Purpose: Describe the association between two categorical variables.

• Tool: Contingency table of counts, then convert to row % (within-group rates) or

column % (within-outcome composition) for fair statements.

• Visual: 100% stacked bars (each bar scaled to 100%) for quick comparison on the same
scale.

• Language: “Within City A, x% purchase vs y% in City B (row %). Among buyers, p% are
from City B (column %).”

Good practice & cautions (apply across pairs)

• Check form before quoting r: Use correlation only if the scatter is roughly linear.

• Be robust: Outliers and skew suggest using median/IQR comparisons (boxplots) or

rank ideas.

• State the base for percentages: row % vs column % to avoid mixing interpretations.

• No causation claims in EDA: We describe associations; causality needs design or

modeling beyond EDA.
Formula Table:
Name Formula Notes
𝑛
1
Mean (raw data) 𝑥ˉ = ∑ 𝑥𝑖 𝑛=number of values
𝑛
𝑖=1
Weighted / ∑𝑤𝑖 𝑥𝑖 ∑𝑓𝑖 𝑚𝑖 𝑤𝑖 =weights; 𝑓𝑖 =freq;
𝑥ˉ = or 𝑥ˉ =
Grouped mean ∑𝑤𝑖 ∑𝑓𝑖 𝑚𝑖 =class midpoint
𝐿=lower boundary of
𝑁
Median (grouped, ( 2 − 𝑐𝑓 ) median class; 𝑁=total freq;
continuous) 𝑥̃ = 𝐿 + ℎ 𝑐𝑓 =cum. freq before class;
𝑓𝑐
𝑓𝑐 =class freq; ℎ=class width
(𝑓1 − 𝑓0 ) 𝐿=lower boundary of modal
Mode (grouped) Mode = 𝐿 + ℎ class; 𝑓1 =modal freq;
(2𝑓1 − 𝑓0 − 𝑓2 )
𝑓0 , 𝑓2 =adjacent freqs
Range Range = max⁡ −min⁡ Very sensitive to extremes
Interquartile
IQR = 𝑄3 − 𝑄1 Middle 50% spread
Range
∑(𝑥𝑖 − 𝑥ˉ)2
Sample variance 𝑠 = 2 With sample mean
𝑛−1

Sample SD 𝑠 = √𝑠 2 Same units as 𝑥

Population ∑(𝑥𝑖 − 𝜇)2
𝜎2 = , 𝜎 = √𝜎 2 Using population mean 𝜇
variance/SD 𝑁
Coefficient of 𝑠
CV = Unit-free
Variation 𝑥ˉ
Proportion / 𝑥
𝑝 = , % = 100𝑝 𝑥=count, 𝑛=total
Percentage 𝑛
𝑟
Pearson 𝑛∑𝑥𝑦 − (∑𝑥)(∑𝑦)
= For tabulated totals
correlation
√[𝑛∑𝑥 2 − (∑𝑥)2 ][𝑛∑𝑦 2 − (∑𝑦)2 ]