Lecture1

basic concepts

1
 Types of Data

Type On Disk In Python

tabular CSV DataFrame
hierarchical JSON dict
textual plaintext string
geospatial ??? ???

2
 W hat Does Data Look
Like?
variables
observational
units

quantitative variables

categorical variables

Data like this, that can be stored in a spreadsheet, is called

tabular data.

3
 How is Tabular Data Represented on
Disk?

⇓
name,pclass,survived,sex,age,sibsp,parch,ticket,fare,cabin,embarked,boat,body,[Link]
"Allen, Miss. Elisabeth Walton",1,1,female,29,0,0,24160,211.3375,B5,S,2,,"St Louis, MO"
"Allison, Master. Hudson Trevor",1,1,male,0.9167,1,2,113781,151.5500,C22 C26,S,11,,"Montreal, PQ / Cheste
"Allison, Miss. Helen Loraine",1,0,female,2,1,2,113781,151.5500,C22 C26,S,,,"Montreal, PQ / Chesterville,
"Allison, Mr. Hudson Joshua Creighton",1,0,male,30,1,2,113781,151.5500,C22 C26,S,,135,"Montreal, PQ / Che
"Allison, Mrs. Hudson J C (Bessie Waldo Daniels)",1,0,female,25,1,2,113781,151.5500,C22 C26,S,,,"Montreal
"Anderson, Mr. Harry",1,1,male,48,0,0,19952,26.5500,E12,S,3,,"New York, NY"
"Andrews, Miss. Kornelia Theodosia",1,1,female,63,1,0,13502,77.9583,D7,S,10,,"Hudson, NY"
"Andrews, Mr. Thomas Jr",1,0,male,39,0,0,112050,0.0000,A36,S,,,"Belfast, NI"
"Appleton, Mrs. Edward Dale (Charlotte Lamson)",1,1,female,53,2,0,11769,51.4792,C101,S,D,,"Bayside, Queen
"Artagaveytia, Mr. Ramon",1,0,male,71,0,0,PC 17609,49.5042,,C,,22,"Montevideo, Uruguay"
"Astor, Col. John Jacob",1,0,male,47,1,0,PC 17757,227.5250,C62 C64,C,,124,"New York, NY"

Comma-Separated Values (CSV) format

4
 How is Tabular Data Represented in
Python?

⇓
DataFrame

Let’s interact with this data using Python in a notebook.

All of our code will be written in Colab notebooks like this one.

5
 Review: Categorical Variables
Tosummarize a categorical variable, we report the counts of each
possible category.
df["pclass"].value_counts()
3 709
1 323
2 277
Name: pclass, dtype: int64

To visualize a categorical variable, we make a bar plot.

df["pclass"].value_counts().[Link]()

Hmm...why are the

classes o u t o f order?
6
 Review: Categorical Variables
Tosummarize a categorical variable, we report the counts of each
possible category.
df["pclass"].value_counts()
3 709
1 323
2 277
Name: pclass, dtype: int64

To visualize a categorical variable, we make a bar plot.

df["pclass"].value_counts().sort_index().[Link]()

Notice that w e can

chain methods, o n e
a f t e r the other.

7
 Selecting Columns
Note that we selected a single column by passing the column name as a
key to the DataFrame.
df["pclass"]
0 1
1 1
2 1
.. The result is a one-dimensional
1306
1307
3
3 pandas object called a Series .
1308 3
Name: p c l a s s , Lengt h: 1309, dt ype: int 64

We can select multiple columns by passing a l i s t of column names.

d f [ [ " p c l as s " , "survived"]]

The result is two-dimensional, an-

other smaller DataFrame.
How do we make sense of multiple
variables at once?

8
 Summarizing Multiple Categorical Variables

To summarize multiple categorical variables, we report the counts

of every possible combination of categories.

We can use the .value_counts() method of DataFrame.

d f [ [ " p c l as s " , "survived"]].value_counts()
pclass survived
3 0 528
1 1 200
3 1 181
2 0 158
1 0 123
2 1 119
dtype: int64
Note that the result is a Series , with a multi-level index, one for
each variable!

9
 Summarizing Multiple Categorical Variables
pclass survived
3 0 528
1 1 200
3 1 181
2 0 158
1 0 123
2 1 119
dtype: int64
Let’s make this information easier to read by arranging one
variable along the rows and the other along the columns.

( d f [ [ " pc l a s s" , "survived"]].value_counts().

unstack())

This representation is called a

two-way table or a crosstab
(short for “cross-tabulation”).

10
 Visualizing Multiple Categorical Variables

From a crosstab, we can make a

bar plot to visualize the data.

( d f [ [ " pc l a s s" , "survived"]].value_counts().

unstack().
[Link]())

This is called a grouped

bar plot.

11
 Marginal Counts
How do we recover the counts for each individual variable from a
crosstab?
crosstab = d f [ [ " p c l as s " , "survived"]].value_counts().unstack()
crosstab

We could sum over the columns (across each row) to obtain the
counts for pclass...
[Link](axis="columns")
p c la s s
1 323
2 277
3 709
d t yp e: i n t 6 4
12
 Marginal Counts
How do we recover the counts for each individual variable from a
crosstab?
crosstab = d f [ [ " p c lass" , "survived"]].value_counts().unstack()
crosstab

...or sum over the rows (down each column) to obtain the counts
for survived.
[Link](axis="rows")
survived
0 809
1 500
d t yp e: i n t 6 4

13
 Proportions

Instead of counts, it can be useful to report proportions, where

we normalize by the total.

count
proportion = .
total

For example, the proportions of the three passenger classes are:

df["pclass"].value_counts() / len(df)
3 0.541635 Notice that the values
1 0.246753
2 0.211612 in a distribution add up
Name: pclass, dtype: float64 t o 1.0!

Together, the proportions of a categorical variable are called the

distribution of the variable pclass.

14
 Probabilities

What does it mean to say, “The proportion of passengers in 3rd

class is 0.541635?”
One interpretation is as a probability.
“If we were to pick a passenger on the Titanic at random, the
probability that they are in 3rd class is 0.541635.”

We notate this as

P (3rd class) = 0.541635

or, if we want to be explicit about the variable and the category,

P (pclass = 3) = 0.541635

15
 Vectorization
Let’s take a closer look at the code for calculating the proportions.
df["pclass"].value_counts() / len(df)

Notice that we divided a Series by a number! Is that even legal?

In pandas, operations are vectorized. A Series behaves like a
vector.
Vectors in pandas work like vectors in math!
Math Review

To multiply a vector

⃗v = (v1, v2, ..., vn)

by a scalar (a.k.a. number) a,

a⃗v = (av1, av2, ..., avn),

we multiply each component of

the vector by a.
16
1 Review

2 Two (or More) Categorical Variables

3 Proportions and Probabilities

4 Joint and Conditional Distributions

17
 Joint Distributions

We can also calculate the distribution of multiple variables, called

a joint distribution.

d f [ [ " p c l as s " , "survived"]].value_counts().unstack() / len(df)

Notice that the values

in the joint distribution
also sum t o 1.0!

18
 Visualizing Joint Distributions

How would this bar plot change if we plotted the joint

distribution instead of the counts?
The scale on the y-axis changes, but the shape is the
same.

To appreciate the power of proportions, we need to look at

conditional distributions.

19
 Conditional Distributions
To compare survival across the classes, we should normalize by the
total in each class.

pclass
1 323
2 277
3 709
dtype: int64

crosstab [Link](axis="columns")
Next, we divide the crosstab by the total in each class.
pclass_totals = [Link](axis="columns")
[Link](pclass_totals, axis="rows")

These are the conditional distributions

of survived given pclass.

20
 Visualizing Conditional Distributions
To visualize a conditional distribution, we could make a grouped
bar plot...

([Link](pclass_totals, axis="rows").
[Link]())

21
 Visualizing Conditional Distributions
To visualize a conditional distribution, we could make a grouped
bar plot...

([Link](pclass_totals, axis="rows").
[Link](stacked=True))

...but it is better to make a stacked bar plot.

22
 Conditional Probabilities

What does it mean to say, “The conditional proportion of survival

given 3rd class is 0.255289”?

One interpretation is as a conditional probability.

“If we were to pick a 3rd class passenger on the Titanic at
random, the probability that they survived is 0.255289.”

We notate this as

P (survived|3rd class) = 0.255289

or, if we want to be explicit about the variable and the category,

P (survived = 1|pclass = 3) = 0.255289

23
 Quantitative Variables
We have analyzed a quantitative variable already. Where?
In the Colombia COVID data!
df_CO = pd.read_csv(url + "colombia_2020-[Link]")
df_CO

This example will motivate our discussion of quantitative

variables today!
2
Visualizing One Quantitative Variable
To visualize the age variable, we did the following:
quantitative binning categorical make a
variable variable “bar plot”

df_CO["age"] = [Link](
df_CO["Edad"],
bins=[0, 10, 20, 30, 40, 50, 60, 70, 80, 120],
labels=["0-9", "10-19", "20-29", "30-39", "40-49", "50-59", "60-69
right=False)
df_CO["age"].value_counts(sort=False).[Link]()

This is the idea behind a

visualization called the
histogram.

2
 Histograms
Pandas provides a built-in method for constructing histograms:
[Link]().

df_CO["Edad"].[Link]()

How does this differ from the manual histogram from earlier?

• There are no spaces

between the bars.
• The x-axis is just numbers,
rather than bins.
2
 Distributions
Recall the distribution of a categorical variable.
The distribution of a quantitative variable is similar. The counts
are scaled so that the total area is 1.0 (or 100%).

df_CO["Edad"].[Link](density=True)

How does this differ from the manual histogram from earlier?

• Only the y-axis changes.

• The shape is the same!

27
Summarizing a Quantitative Variable
If you had to summarize this data using a single number, what
number would you pick?

spread

center

If you had to summarize this data using two numbers, what

number would you pick second?
28
 Summaries of Center: M ean
One summary of the center of a quantitative variable is the mean.
To calculate the mean of a quantitative variable x with values
x 1 , x 2 , x 3 , ..., x n , we use the formula:
Σ n
xi
x = mean(x) = i=1
n

You can calculate it manually...

df_CO["Edad"].sum() / len(df_CO)
39.04742568792872

...or using a built-in Python function.

df_CO["Edad"].mean()
39.04742568792872

29
 Summaries of Center: M ean
Don’t be fooled by the humble mean,
Σ n
i=1 xi
x = mean(x) = .
n
It is not at all obvious that this formula should give a summary of
center!

Let’s investigate one reason in a Colab.

If I have seen further [than others],

it is by standing on the shoulders of
giants.
– Isaac Newton

30
 Summaries of Center:
M edian
Another summary of center is the median, which is the “middle”
of the sorted values.
To calculate the median of a quantitative variable x with values
x 1 , x 2 , x 3 , ..., x n , we do the following steps:
1 Sort the values from smallest to largest:

x (1) , x (2) , x (3) , ..., x (n) .

Statisticians call the sorted values the order statistics.

2 The “middle” value depends on whether we have an odd or
an even number of observations.
• If n is odd, then the middle value is x (n + 1 ).
2
• If n is even, then there are two middle values, x ( n ) and
2
x ( n2 +1) . It is conventional to report the mean of the two
values (but you can actually pick any value between them).

31
 Summaries of Center:
M edian

We can implement these steps in Python code manually. I asked

ChatGPT, and it generated this code:

But it’s easier to use the built-in Python function.

df_CO["Edad"].median()
37.0

32
 Summaries of Center: M ean vs.
M edian
We now have two summaries of center. How do they compare?

median = 37.0

mean = 39.0

How would we summarize spread now?

33
 Summaries of Spread: Variance
One measure of spread is the variance.
The variance of a variable x whose values are x 1 , x 2 , x 3 , ..., x n is
calculated using the formula
Σ n
(x i − 2
var(x) = i=1
nx)
−1

You can implement this formula manually...

(((df_CO["Edad"] - df_CO["Edad"].mean()) ** 2).sum() /
(len(df_CO) - 1))
348.0870469898451

...or using a built-in Python function.

df_CO["Edad"].var()
348.0870469898451

What are the units? years2

15
 Summaries of Spread: Standard
To fixDeviation
the units, we take the square root to get the standard
deviation:
√
sd(x) = var(x)
years years 2
We can calculate it using the built-in Pandas method [Link]:
df_CO["Edad"].std()
18.65709106452142

spread: ±18.7 years

center: 39.0 years

35