Analyzing Quantitative Data

Dr. Andrii Matviienko

 Is A better than B?

30.06.22 Łódź HCI Summer School 2022 | Quantitative Data Analysis | Andrii Matviienko 5
 Three main methods
§ Hypothesis testing
§ P-values
§ Confidence intervals
§ Adds a lower and upper bound on the true value
§ Bayesian methods
§ Prior probabilities (before the experiment)
§ prior probability that A & B are the same: 80%
§ prior probability that B is better: 20%
§ Posterior probabilities (after the experiment)
§ posterior probability that A & B are the same: 25%
§ posterior probability that B is better: 75%
§ Bayes factor (odds ratio), e.g., 3:1 = B is 3 times better than A

30.06.22 Łódź HCI Summer School 2022 | Quantitative Data Analysis | Andrii Matviienko 6
 Comparing Groups

Design # of IVs # of Levels Parametric Test Non-Parametric Test

2 Independend-samples t-test Mann-Whitney-U-Test
1
Between-groups 3+ One-way ANOVA Kruskal–Wallis test
2+ 2+ Factorial (n-way) ANOVA Scheirer-Ray-Hare test
2 Paired-samples t-test Wilcoxon test
1
Within-groups 3+ Friedman test
Repeated-Measures ANOVA
2+ 2+ Aligned Rank Transform
mixed 2+ 2+ Split-Pot ANOVA ANOVA

30.06.22 Łódź HCI Summer School 2022 | Quantitative Data Analysis | Andrii Matviienko 7
 Statistics 101
A (very!) short (and incomplete!) introduction to statistics

30.06.22 Łódź HCI Summer School 2022 | Quantitative Data Analysis | Andrii Matviienko 8
 Probability
§ What are the chances of a fair coin coming up heads 10 times in a row?
§ If I roll two six sided dice, how likely is it that I’ll roll two sixes?
§ How likely is it that five cards drawn from a perfectly shuffled deck will all be
hearts?
§ What are the chances that I’ll win the lottery?

predict

Model Data

30.06.22 Łódź HCI Summer School 2022 | Quantitative Data Analysis | Andrii Matviienko 9
 Statistics
§ If my friend flips a coin 10 times and gets 10 heads, are they playing a trick
on me?
§ If five cards off the top of the deck are all hearts, how likely is it that the
deck was shuffled?
§ If the lottery commissioner’s spouse wins the lottery, how likely is it that the
lottery was rigged?
infer

Model Data

30.06.22 Łódź HCI Summer School 2022 | Quantitative Data Analysis | Andrii Matviienko 10
 Probability vs. Statistics

Probability
predict

Model Data

Statistics
infer

Model Data

30.06.22 Łódź HCI Summer School 2022 | Quantitative Data Analysis | Andrii Matviienko 11
 So how do we come from data to a model?
§ Let‘s imagine we measure the size of two
individuals.
§ Andrii: 182cm
§ Florian: 186cm
§ We can easily conclude that Florian is
taller than Andrii.
§ Even if we measure several times, we may
find some small measurement inaccuracies,
but the basic conclusion will remain the
same.
§ This is easy.

30.06.22 Łódź HCI Summer School 2022 | Quantitative Data Analysis | Andrii Matviienko 12
 Sampling (I)
§ Typically, you don't want to make statements about individuals but larger
populations.
§ You are not interested in individuals but in entire populations.
§ However, we usually cannot test the entire population
§ “Kids are faster using system a” -> We would need to test the system with all kids on
the planet.
§ That’s why we need to sample parts of the population.
§ We use this sampled group to perform tests.
§ Based on the results, we make inferences about the underlying population.
§ Significance tests allow us to determine how confident we are that the results
observed from the sampling population can be generalized to the entire population.

30.06.22 Łódź HCI Summer School 2022 | Quantitative Data Analysis | Andrii Matviienko 13
 Sampling (II)

30.06.22 Łódź HCI Summer School 2022 | Quantitative Data Analysis | Andrii Matviienko 14
 Descriptives (I)
§ Now that we have collected data from our sample, we want to understand
that data. The first step is descriptive statistics.
§ Probably you know all of these measures from school, so we’ll keep it short.

Suppose you want to investigate whether the use of specific word-prediction on a smartphone has an impact
on typing speed.

30.06.22 Łódź HCI Summer School 2022 | Quantitative Data Analysis | Andrii Matviienko 15
 Descriptives (II)
§ We measured the typing time for the same
text in two groups, one using word
prediction and one without.
§ Now let‘s have a look at some descriptives:

30.06.22 Łódź HCI Summer School 2022 | Quantitative Data Analysis | Andrii Matviienko 16
 Mean
§ The mean is the average value of a distribution.
§ Easy: add up all the numbers, then divide by how many numbers there are
Given [13, 18, 13, 14, 13, 16, 14, 21, 13], the mean is
(13 + 18 + 13 + 14 + 13 + 16 + 14 + 21 + 13) ÷ 9 = 15

30.06.22 Łódź HCI Summer School 2022 | Quantitative Data Analysis | Andrii Matviienko 17
 Descriptives (II)
§ We measured the typing time for the same
text in two groups, one using word
prediction and one without.
§ Now let‘s have a look at some descriptives:

30.06.22 Łódź HCI Summer School 2022 | Quantitative Data Analysis | Andrii Matviienko 18
 Median
§ The median is the middle value of the data
§ Also easy: Sort data in numerical order, then pick the element in the middle.
Given [13, 18, 13, 14, 13, 16, 14, 21, 13], the median is
13, 13, 13, 13, 14, 14, 16, 18, 21

30.06.22 Łódź HCI Summer School 2022 | Quantitative Data Analysis | Andrii Matviienko 19
 Descriptives (II)
§ We measured the typing time for the same
text in two groups, one using word
prediction and one without.
§ Now let‘s have a look at some descriptives:

30.06.22 Łódź HCI Summer School 2022 | Quantitative Data Analysis | Andrii Matviienko 20
 Standard Deviation and Variance
§ The standard deviation and the variance are a measure of how spread-out
numbers are
§ Variance: The average of the squared differences from the mean
§ We calculate the mean
§ calculate the difference of each value to the mean
§ square these differences
§ and average them
§ Standard Deviation: Square root of variance.

Standard Deviation
30.06.22 Łódź HCI Summer School 2022 | Quantitative Data Analysis | Andrii Matviienko 21
 Descriptives (II)
§ We measured the typing time for the same
text in two groups, one using word
prediction and one without.
§ Now let‘s have a look at some descriptives:

30.06.22 Łódź HCI Summer School 2022 | Quantitative Data Analysis | Andrii Matviienko 22
 Descriptives in R

30.06.22 Łódź HCI Summer School 2022 | Quantitative Data Analysis | Andrii Matviienko 23
 Okay… So let‘s answer our question…
§ So based on the gathered data and
the descriptives: Is there actually a
difference between the groups?
§ The descriptives suggest that there
really is a difference here (yay!).
§ But can we be sure?
§ No! But we can quantify how high the
chance is that it is just coincidence.

30.06.22 Łódź HCI Summer School 2022 | Quantitative Data Analysis | Andrii Matviienko 24
 Statistical significance testing and p (I)
§ Significance testing allows us to quantify the chance that an event occurred
just by random chance.
§ We speak of statistical significance when it is very unlikely to have occurred
given H0.
§ For us (most of the time): H0: All samples originate from the same
population
§ That means: There is no difference in the data.
§ Remember: We can only reject hypothesis, not confirm them
§ If we reject H0 based on our analysis, we say: The probability that all of this data that
we have seen belongs to the same underlying population is very low, therefore we
assume that there is a difference between the groups.

30.06.22 Łódź HCI Summer School 2022 | Quantitative Data Analysis | Andrii Matviienko 25
 Statistical significance testing and p (II)
§ We express this probability as p.
§ There are MANY misunderstandings and misinterpretations of p even in otherwise
really good textbooks.
§ The correct interpretation is: Under the assumption that H0 is true, p indicates the
probability to see results that are at least this extreme.
§ Assuming that there are no differences in reality, p gives the probability of seeing this effect in the
data by chance.

30.06.22 Łódź HCI Summer School 2022 | Quantitative Data Analysis | Andrii Matviienko 26
 Errors
§ As we are dealing with probabilities, there is the chance of making mistakes.
§ Think of it as a court hearing.
§ H0: The defendant is innocent.

30.06.22 Łódź HCI Summer School 2022 | Quantitative Data Analysis | Andrii Matviienko 27
 Type I Errors
§ A Type I error occurs when the jury decides that the defendant is guilty
when he is actually innocent, meaning that the null hypothesis is rejected
when it is true.
§ More general: A Type I error (also called an α error or a “false positive”)
refers to the mistake of rejecting the null hypothesis when it is true and
should not be rejected.

30.06.22 Łódź HCI Summer School 2022 | Quantitative Data Analysis | Andrii Matviienko 28
 Errors
§ As we are dealing with probabilities, there is the chance of making mistakes.
§ Think of it as a court hearing.

30.06.22 Łódź HCI Summer School 2022 | Quantitative Data Analysis | Andrii Matviienko 29
 Type II Errors
§ A Type II error occurs when the jury decides that the defendant is innocent
when he is actually guilty, meaning that the null hypothesis is accepted
when it is false.
§ More general: A Type II error (also called a β error or a “false negative”)
refers to the mistake of not rejecting the null hypothesis when it is false and
should be rejected

30.06.22 Łódź HCI Summer School 2022 | Quantitative Data Analysis | Andrii Matviienko 30
 Errors
§ As we are dealing with probabilities, there is the chance of making mistakes.
§ Think of it as a court hearing.

§ Type I error: rejecting the null hypothesis when it is true.

§ Type II error: not rejecting the null hypothesis when it is false.
30.06.22 Łódź HCI Summer School 2022 | Quantitative Data Analysis | Andrii Matviienko 31
 How to calculate that „p“ Value?
§ You can select the correct significance
test to analyze your data based on
§ The number of independend variables
(factors)
§ The number of conditions in each factor
(levels)
§ The type of experiment (between or
within groups. Quantitative
Data

§ The type of your dependend variable

Categorical Numerical

Nominal Ordinal Interval Ratio

30.06.22 Łódź HCI Summer School 2022 | Quantitative Data Analysis | Andrii Matviienko 32
 Parametric vs. Non-Parametric Quantitative
Data

§ There are two basic groups of tests. Categorical Numerical

Parametric tests and non-parametric tests.

§ Parametric tests have higher power (i.e., it Nominal Ordinal Interval Ratio

is easier to obtain significant results), but

they have more rigorous requirements for
the data (“assumptions”).

30.06.22 Łódź HCI Summer School 2022 | Quantitative Data Analysis | Andrii Matviienko 33
 Assumptions

30.06.22 Łódź HCI Summer School 2022 | Quantitative Data Analysis | Andrii Matviienko 34
 Types of (Quantitative) Data

Quantitative
Data

Categorical Numerical

Nominal Ordinal Interval Ratio

+ natural order + same interval + defined 0

30.06.22 Łódź HCI Summer School 2022 | Quantitative Data Analysis | Andrii Matviienko 35
 Assumption of the Independence of Cases
§ Avoid connections (“dependencies”) between data points you collect.
If 2 different persons conduct the experiment, make sure that they introduce the task to the participants
in the same way. Different instructions can lead to dependencies.
• “The performance depends on what the instructor told the participant”

§ We cannot tell if this assumption is fulfilled from the data!

§ Solution: when designing and conducting your measurements, make sure to
avoid dependencies wherever possible.

30.06.22 Łódź HCI Summer School 2022 | Quantitative Data Analysis | Andrii Matviienko 37
 Assumption of Normality
§ The data needs to be approximately
normally distributed.
§ This rules out data that are not at least on
interval scale!
§ If your data are on an interval scale,
you also need to check whether these
data are approximately normally
distributed.
§ Most easy approach (which is fine for this
lecture): Check the histogram of data.
§ There are also statistical tests to do this.
Most common are
§ Shapiro-Wilk’s test
§ Kolmogorov-Smirnov test
§ These tests are offered by most statistical
software.
30.06.22 Łódź HCI Summer School 2022 | Quantitative Data Analysis | Andrii Matviienko 38
 Assumption of the Homogeneity of Variances
§ Important: Only for between-group
designs!
§ Parametric tests require the variances
of each group to be approximately
equal.
§ How to test:
§ Use Levene's test
§ Offered by most statistical tools (we’ll look
at this later)
§ If Levene’s test is significant (p<.05), use a
non-parametric test instead.

30.06.22 Łódź HCI Summer School 2022 | Quantitative Data Analysis | Andrii Matviienko 39
 Assumption of Sphericity
§ Important: Only for within-groups designs!
§ The assumption of sphericity is closely related to the assumption of
homogeneity of variances:
§ For between-subjects designs, we require that the variances between the groups are
equal.
§ For within-subject designs, we require that the variances of the differences within
each group are equal.
§ How to test:
§ Use Mauchly's test
§ Offered by most statistical tools
§ We’ll look at this later
§ If Mauchly's test is significant (p<.05), you can correct the results of the test.
§ We’ll look at this later

30.06.22 Łódź HCI Summer School 2022 | Quantitative Data Analysis | Andrii Matviienko 40
 Parametric vs. Non-Parametric
Quantitative
§ There are two basic groups of tests. Parametric Data

tests and non-parametric tests.

Categorical Numerical
§ Parametric tests have higher power (i.e., it is
easier to obtain significant results), but they
have more rigorous requirements for the data Nominal Ordinal Interval Ratio

(“assumptions”).
§ Assumptions:
§ Independence of cases
§ Normality
§ Homogeneity of Variances (between-subjects)
§ Sphericity (within-subjects)
§ Rule of thumb: Whenever possible: Use
parametric tests. If one of the assumptions is
not fulfilled, then switch to the non-parametric
equivalent.
30.06.22 Łódź HCI Summer School 2022 | Quantitative Data Analysis | Andrii Matviienko 41
 How to calculate that „p“ Value?
§ You can select the correct significance
test to analyze your data based on
§ The number of independend variables
(factors)
§ The number of conditions in each factor
(levels)
§ The type of experiment (between or
within groups. Quantitative
Data

§ The type of your dependend variable

Categorical Numerical

Nominal Ordinal Interval Ratio

30.06.22 Łódź HCI Summer School 2022 | Quantitative Data Analysis | Andrii Matviienko 42
 Comparing Groups

Design # of IVs # of Levels Parametric Test Non-Parametric Test

2 Independend-samples t-test Mann-Whitney-U-Test
1
Between-groups 3+ One-way ANOVA Kruskal–Wallis test
2+ 2+ Factorial (n-way) ANOVA Scheirer-Ray-Hare test
2 Paired-samples t-test Wilcoxon test
1
Within-groups 3+ Friedman test
Repeated-Measures ANOVA
2+ 2+ Aligned Rank Transform
mixed 2+ 2+ Split-Pot ANOVA ANOVA

30.06.22 Łódź HCI Summer School 2022 | Quantitative Data Analysis | Andrii Matviienko 43
 Between-Groups Design with 1 IV with 2 Levels
§ Suppose you want to investigate whether
the use of specific word-prediction on a
smartphone has an impact on typing speed.

H0: There is no difference in the task completion time

(TCT) between individuals who use the word-
prediction software and those who do not use the
software.

§ We collect data from 16 participants who

either type the text with or without
prediction support.

30.06.22 Łódź HCI Summer School 2022 | Quantitative Data Analysis | Andrii Matviienko 44
 Comparing Means
§ Looking at the mean values, we can
see a difference.
§ But can we be sure that this
difference is not just a matter of
chance?
§ No! But we can quantify how high the
chance is that it is just coincidence.

30.06.22 Łódź HCI Summer School 2022 | Quantitative Data Analysis | Andrii Matviienko 45
 Independend-samples t-test
§ As we have one IV (support system) with two levels (prediction, no
prediction) in a between-groups experiment, we can use the independed
samples t-test.
§ There are three things that we see
here (that we‘ll also encounter for
different tests):
§ The degrees of freedom (df)
§ The statistics value (Statistic)
§ The p value

30.06.22 Łódź HCI Summer School 2022 | Quantitative Data Analysis | Andrii Matviienko 46
 Statistics value (t Value)
§ The statistics value is calculated based on
the data
§ incorporates both the sample size and the
variability in the data.
§ We are not going into details on the
calculation.
§ A higher t-value means that there is more
heterogeneity in the data.
§ Larger difference of the mean values -> higher t
value.
§ Lower variance of the data -> higher t-value
§ But at what t-value do we reject H0?

30.06.22 Łódź HCI Summer School 2022 | Quantitative Data Analysis | Andrii Matviienko 47
 t distribution
§ If we collect multiple random samples (i.e.,
samples without an effect!) of
§ the same size
§ from the same population
§ and performed the same t-test
§ we would obtain many t-values and we
could plot a probability distribution of
them.
§ That is: The probability to see specific t-values
when there is no effect in the data!
§ For any given t-value, this distribution tells
us the probability to obtain this value just
by chance.

30.06.22 Łódź HCI Summer School 2022 | Quantitative Data Analysis | Andrii Matviienko 48
 Degrees of Freedom
§ The degrees of freedom (df) are the number of observations that are free to
vary when estimating statistical parameters.
§ For independend samples t-tests, the number is n – 2
§ While we are comparing two cells, all other (n-2) cells are free to vary.
§ We are not going into details here as degrees of freedom are not something
you need to understand to perform a statistical analysis.

30.06.22 Łódź HCI Summer School 2022 | Quantitative Data Analysis | Andrii Matviienko 49
 p value (I)
Df = 14

§ Remember the definition of a p value?

§ “… p indicates the probability to see results
that are at lest this extreme.“
§ As we know the t-distribution (based on
the df of our experiment), we can check
whats the probability of a t-value that is at
least this extreme.

In our example: p(|t| > 2.17) = 0.0477

So for our example: The probability that the

experiment showed a difference at least this extreme
is < 5%.

30.06.22 Łódź HCI Summer School 2022 | Quantitative Data Analysis | Andrii Matviienko 50
 p value (II)
§ Our goal was to be able to reject H0.
H0: There is no significant difference in the task completion time (TCT) between individuals who use the
word-prediction software and those who do not use the software.

§ p gives us the probability that the difference we found is just random.

§ So we need a threshold at which we can reject H0.
§ We consider a probability < 5% as statistically significant.
§ Note that this value is not mathematically justified. It is an agreement of the
community, what we consider as significant results and what not.
§ We report p in three groups
§ p < 0.05
§ p < 0.01
§ p < 0.001
30.06.22 Łódź HCI Summer School 2022 | Quantitative Data Analysis | Andrii Matviienko 51
 Independend-samples t-test (III)
§ As we have one IV (support system) with two levels (prediction, no
prediction) in a between-groups experiment, we can use the independed
samples t-test.
§ There are three things that we see
here (that we‘ll also encounter for
different tests):
§ The degrees of freedom (df)
§ The statistics value (Statistic)
§ The p value
§ How to report: t(14) = -2.17, p < .05

30.06.22 Łódź HCI Summer School 2022 | Quantitative Data Analysis | Andrii Matviienko 52
 Comparing Groups

Design # of IVs # of Levels Parametric Test Non-Parametric Test

2 Independend-samples t-test Mann-Whitney-U-Test
1
Between-groups 3+ One-way ANOVA Kruskal–Wallis test
2+ 2+ Factorial (n-way) ANOVA Scheirer-Ray-Hare test
2 Paired-samples t-test Wilcoxon test
1
Within-groups 3+ Friedman test
Repeated-Measures ANOVA
2+ 2+ Aligned Rank Transform
mixed 2+ 2+ Split-Pot ANOVA ANOVA

30.06.22 Łódź HCI Summer School 2022 | Quantitative Data Analysis | Andrii Matviienko 54
 Let‘s add another Level
§ Let's extend our study with a second
prediction model.
§ We now compare three levels (prediction1,
prediction2, no_prediction)

30.06.22 Łódź HCI Summer School 2022 | Quantitative Data Analysis | Andrii Matviienko 55
 Multiple Comparisons
§ First idea: Let’s compare the 3 variables pairwise to see if there are
significant differences between them.
§ prediction1 vs prediction2
§ prediction1 vs no_prediction
§ prediction2 vs no_prediction
§ Remember: We have established that we accept a 5% chance of rejecting
H0 even if its true by random chance (Type I errors).
§ This happens with every comparison
§ The total chance of making a mistake for n comparisons is 1-(1-0.05)^n
§ For our example (n=3), that‘s already a chance of ~14% of a Type I error!

30.06.22 Łódź HCI Summer School 2022 | Quantitative Data Analysis | Andrii Matviienko 56
 One-way ANOVA (I)
§ Let's extend our study with a second
prediction model.
§ We now compare three levels (prediction1,
prediction2, no_prediction)
§ ANOVA (Analysis of Variances) controls for
Type I errors and allows us to compare 3+
group means simultaneously while
keeping our Type I error rate <5%.

30.06.22 Łódź HCI Summer School 2022 | Quantitative Data Analysis | Andrii Matviienko 57
 One-way ANOVA (II)
§ Basic idea: compare the average variation
between groups to the average variation
within groups.
§ Between-groups variation: The differences
of the means of the groups.
§ Within-group variation: The mean of the
variances of the groups.
§ Based on these two values, we calculate F:
§ F = between groups variation / within groups
variation
§ A larger between-groups variation increases F
§ A lower within-groups variation increases F

30.06.22 Łódź HCI Summer School 2022 | Quantitative Data Analysis | Andrii Matviienko 58
 One-way ANOVA (III)
§ Now we do the same thing as with the t-test
§ If we were to repeat this test multiple times with random sample data without an
effect, we could again plot a distribution of F values based on the degrees of freedom
of the between (df_between) and within (df_within) variances.
§ For k = number of groups and n = number of participants,
§ df_between = k-1
§ df_within = n-k
For our example (16 participants, 3 groups), this probability distribution looks like this:

30.06.22 Łódź HCI Summer School 2022 | Quantitative Data Analysis | Andrii Matviienko 59
 One-way ANOVA (IV)
§ Using this probability distribution, we can
now check the probability to see a result at
least this extreme.
§ For our example:

§ How to report: F(2, 21) = 7.28, p<0.01

30.06.22 Łódź HCI Summer School 2022 | Quantitative Data Analysis | Andrii Matviienko 60
 Post-hoc tests (I)
§ The ANOVA is a so-called omnibus („for
all“) test
§ It tells us that (given a 5% error level) there is
indeed a difference between the groups of
our data.
§ But: The test does not tell us between which
groups.
§ In order to check between which groups
truly significant differences exist, we
conduct so-called post-hoc tests.

30.06.22 Łódź HCI Summer School 2022 | Quantitative Data Analysis | Andrii Matviienko 61
 Post-hoc tests (II)
§ How we do it? We do pairwise comparisons between the groups.
§ Wait… didn‘t we do all this ANOVA stuff because we cannot do pairwise
comparisons?
§ Yes! Pairwise comparisons inflate the Type I error.
§ But thanks to the omnibus tests (ANOVA), we know that there are significant
differences within a 5% error level.
§ After significant omnibus test, we can do pairwise comparisons
§ However, we still have to correct for multiple comparisons to avoid an increased Type
I error for the individual comparisons.

30.06.22 Łódź HCI Summer School 2022 | Quantitative Data Analysis | Andrii Matviienko 62
 Post-hoc tests (III)
§ Most post-hoc tests are based on the t-test we have seen before
§ But: Not the individual variance of the data of the respective group is used for the
test but a pooled variance of all data (all groups).
§ Additionally, there are correction methods for multipe comparisons.
§ Most simple and most popular is the Bonferroni method: Multiply p by the total
number of comparisons.
In our example, we have 3 comparisons, so each of the p values it 3 times higher!

30.06.22 Łódź HCI Summer School 2022 | Quantitative Data Analysis | Andrii Matviienko 63
 Comparing Groups

Design # of IVs # of Levels Parametric Test Non-Parametric Test

2 Independend-samples t-test Mann-Whitney-U-Test
1
Between-groups 3+ One-way ANOVA Kruskal–Wallis test
2+ 2+ Factorial (n-way) ANOVA Scheirer-Ray-Hare test
2 Paired-samples t-test Wilcoxon test
1
Within-groups 3+ Friedman test
Repeated-Measures ANOVA
2+ 2+ Aligned Rank Transform
mixed 2+ 2+ Split-Pot ANOVA ANOVA

30.06.22 Łódź HCI Summer School 2022 | Quantitative Data Analysis | Andrii Matviienko 65
 So we can do multiple Levels now. What about
multiple IVs?
§ Up to now, we have analyzed the influence of
one IV (or one factor) on the dependend
variable.
§ Let‘s imagine we extend our experiment and
additionally divide the groups by how
experienced the participants are in typing.

Each group now consists of a combination of support

system (no_prediction, prediction1, prediction2) and
experience (inexperienced, experienced).

30.06.22 Łódź HCI Summer School 2022 | Quantitative Data Analysis | Andrii Matviienko 66
 Factorial ANOVA (II)
§ We won‘t go into details on how the factorial ANOVA works.
§ But the basic ideas is the same.
§ The result looks similar. But there are a few interesting things here:
§ We get F statistics and p values for the individual influence of factors.
§ We additionally get F statistics and a p value for the interaction effect between the
two factors.

30.06.22 Łódź HCI Summer School 2022 | Quantitative Data Analysis | Andrii Matviienko 67
 Main Effects
§ The main effect is what we also analyzed in the one-way ANOVA: The direct
influence of a factor.
§ Main effects help us formulate simple relations like experienced users type faster
than inexperienced users.

30.06.22 Łódź HCI Summer School 2022 | Quantitative Data Analysis | Andrii Matviienko 68
 Interaction Effects (I)
§ In the real world (which is what we are measuring!), there are also more
complicated relationships than "if a gets bigger, then b gets smaller".
§ Think of interaction effects as „it depends“.
§ Prediction tools help inexperienced users
§ For experienced users, however, the
suggestions have no influence!
§ So the answer to the question: "Do
suggestions help users with their typing
speed?" is: It depends.
§ For inexperienced users: Yes!
§ For experienced users: Probably not!

30.06.22 Łódź HCI Summer School 2022 | Quantitative Data Analysis | Andrii Matviienko 69
 Interaction Effects (II)
§ As for the main effects, we perform comparisons for the interaction effects.
§ We pairwise compare all combinations of all factors.
§ For our example, that results in 15 comparisons

30.06.22 Łódź HCI Summer School 2022 | Quantitative Data Analysis | Andrii Matviienko 70
 Comparing Groups

Design # of IVs # of Levels Parametric Test Non-Parametric Test

2 Independend-samples t-test Mann-Whitney-U-Test
1
Between-groups 3+ One-way ANOVA Kruskal–Wallis test
2+ 2+ Factorial (n-way) ANOVA Scheirer-Ray-Hare test
2 Paired-samples t-test Wilcoxon test
1
Within-groups 3+ Friedman test
Repeated-Measures ANOVA
2+ 2+ Aligned Rank Transform
mixed 2+ 2+ Split-Pot ANOVA ANOVA

30.06.22 Łódź HCI Summer School 2022 | Quantitative Data Analysis | Andrii Matviienko 72
 Within-Subjects Analysis
§ For all the techniques we have just discussed, there are within-subjects
equivalents.
§ The basic idea, execution and interpretation of these methods is equivalent
to their between-groups counterparts.
§ All of them are supported in Jamovi

30.06.22 Łódź HCI Summer School 2022 | Quantitative Data Analysis | Andrii Matviienko 73
 Comparing Groups

Design # of IVs # of Levels Parametric Test Non-Parametric Test

2 Independend-samples t-test Mann-Whitney-U-Test
1
Between-groups 3+ One-way ANOVA Kruskal–Wallis test
2+ 2+ Factorial (n-way) ANOVA Scheirer-Ray-Hare test
2 Paired-samples t-test Wilcoxon test
1
Within-groups 3+ Friedman test
Repeated-Measures ANOVA
2+ 2+ Aligned Rank Transform
mixed 2+ 2+ Split-Pot ANOVA ANOVA

30.06.22 Łódź HCI Summer School 2022 | Quantitative Data Analysis | Andrii Matviienko 74
 Analyzing mixed designs
§ So far, now we have looked at statistical analysis tools for between-groups
and within-groups designs.
§ However, the distinction between-groups or within-groups does not have to refer to
a complete experiment, but can be defined individually per factor.
§ So how do we analyze data from an experiment that contains both,
between and within-groups variables?
§ Turns out: it works very similarly.

30.06.22 Łódź HCI Summer School 2022 | Quantitative Data Analysis | Andrii Matviienko 75
 Split-Pot Anova – Omnibus Test
§ As with „normal“ factorial designs,
we get
§ main effects
§ Interaction effects
§ If we discover significant effects, we
perform post-hoc comparisons
§ for the main effects
§ for the interaction effects

30.06.22 Łódź HCI Summer School 2022 | Quantitative Data Analysis | Andrii Matviienko 76
 Split-Pot ANOVA in Jamovi
§ For a Split-Pot ANOVA in Jamovi, use the repeated measures analysis!
§ Don‘t ask me, I‘m not the one to blame…

Within-subject factors here

Between-subjects factors here

30.06.22 Łódź HCI Summer School 2022 | Quantitative Data Analysis | Andrii Matviienko 77
 Comparing Groups

Design # of IVs # of Levels Parametric Test Non-Parametric Test

2 Independent-samples t-test Mann-Whitney-U-Test
1
Between-groups 3+ One-way ANOVA Kruskal–Wallis test
2+ 2+ Factorial (n-way) ANOVA Scheirer-Ray-Hare test
2 Paired-samples t-test Wilcoxon test
1
Within-groups 3+ Friedman test
Repeated-Measures ANOVA
2+ 2+ Aligned Rank Transform
mixed 2+ 2+ Split-Pot ANOVA ANOVA

30.06.22 Łódź HCI Summer School 2022 | Quantitative Data Analysis | Andrii Matviienko 79
 Non-Paraemtric Tests
§ All types of tests discussed also have non-parametric equivalences that you
can use if the assumptions of the parametric test are not met.
§ e.g., normal distribution
§ These tests have less statistical power
§ In simple words, you need more difference in the data for the test to give you a
significant result.
§ Therefore, always prefer the parametric test if applicable.
§ The procedure is equivalent to the parametric alternatives:
§ If you are comparing more than two groups, perform the appropriate omnibus test.
§ After a significant omnibus test, perform appropriate post-hoc tests.

30.06.22 Łódź HCI Summer School 2022 | Quantitative Data Analysis | Andrii Matviienko 80
 Non-Paraemtric Tests in Jamovi
§ The non-parametric equivalents are somewhat hidden in Jamovi.

30.06.22 Łódź HCI Summer School 2022 | Quantitative Data Analysis | Andrii Matviienko 81
 Comparing Groups without Jamovi
§ Most of the tests are supported currently by Jamovi
§ The other tests have quite good R packages
Scheirer–Ray–Hare Test ART ANOVA

See [Link] See [Link]

30.06.22 Łódź HCI Summer School 2022 | Quantitative Data Analysis | Andrii Matviienko 82
 Comparing Groups

Design # of IVs # of Levels Parametric Test Non-Parametric Test

2 Independend-samples t-test Mann-Whitney-U-Test
1
Between-groups 3+ One-way ANOVA Kruskal–Wallis test
2+ 2+ Factorial (n-way) ANOVA Scheirer-Ray-Hare test
2 Paired-samples t-test Wilcoxon test
1
Within-groups 3+ Friedman test
Repeated-Measures ANOVA
2+ 2+ Aligned Rank Transform
mixed 2+ 2+ Split-Pot ANOVA ANOVA

30.06.22 Łódź HCI Summer School 2022 | Quantitative Data Analysis | Andrii Matviienko 83