SOCI2030: Quantitative

Research Methods
Lecture 4: Variables
Lai Wei
Feb 25

*I thank Satoshi Araki for sharing slides

Last Week
• Literature Review
• Locate the relevant literature
• Professional association
• Annual Review
• High Impact Journals
• Stay in dialogue with the literature
• Gap
• Debate
• Writing tips
• Motivation

• Research hypothesis
• Definition of research hypothesis
• Testable statements motivated from literature
• Competing hypotheses
• Make it a fair fight
This Week
• Collecting Variables from Survey

• Transforming hypotheses to variables

• Types of Survey Questions
• Common Pitfalls of Asking Questions
• Asking Sensitive Questions
• Survey Experiment
From Hypotheses to Variables
From Hypotheses to Variable
• Hypothesis → Variable → Question

• Hypotheses, variables, and questions should be consistent.

• Variables should be quantifiable by questions.
• In addition to independent variables (predictors) and dependent variables
(outcomes), it is important to take account of control variables (controls).
• An analytic framework (i.e., a figure summarising the relationship among
variables) is useful to organise concepts.
Example from Last Week
• Research Question:
• Does social media affect mental well-being?
• Competing Hypotheses:
• H1: Social media harms one’s mental well-being
• H2: Social media benefits one’s mental well-being
• What variables do we need to measure?
• Independent variable: social media usage
• Dependent variable: mental well-being
• Control variables: common causes of independent and dependent variables
• Age, gender, ethnicity, family background, levels of education, etc.
• Mediator (optional): sleep quality
Example from Last Week
• To put in graphical format
• Analytical Framework

Age, gender, ethnicity, family background, levels of education, etc.

Social Media Usage Sleep Quality Mental Well-Being

Some Notes on Control Variables
• Remember that a control variable is a common cause of independent and
dependent variables
• Logically, it has to happen before the independent variable
• What happens after the independent variable is called mediator
• Why do we need to collect control variables?
• We want to control for them in our analysis
• Otherwise the relationship we get between independent and depent vars is confounded
• In all surveys, we usually start with demographic information, the most
essential information about an individual
• Age, gender, ethnicity, family background, levels of education, etc.
• They often serve as control variables in the analyses
Types of Survey Questions
• So how exactly can we measure social media usage and mental well-being?
• Types of survey questions (a non-exhaustive list):
• Continuous
• Discrete
• Binary
• Nominal
• Ordinal
• Open-ended
Open-Ended Questions
• Free-form (open text format) to answer comments/words.

• Examples:
• How do you evaluate the current economic situation in Hong Kong?
• What do you think about online learning at the University?
• Describe to us your mental well-being state recently.

• Pros:
• Elicit useful information unconsidered by the researcher

• Cons:
• Hard to analyze directly with quantitative methods
• Quantitative analysis requires either further processing/NLP techniques

• In general, helpful for qualitative analysis and future survey design, not
recommended for projects in this class
Continuous Questions/Variables
• Question where answers are exact numbers (usually include many possible values)

• Example:
• Please answer your age.
• What is your height?
• Please write your GPA.
• How many hours of social media do you use everyday?

• Collects the most comprehensive information, recommend a continuous measure,

instead of a discrete one, when possible
Discrete Questions/Variables
• Question where answers are limitedly many
• Binary question

• Example:
• Are you a local HKer or not?
• Have you used social media yesterday?
• Are you feeling stressed out recently?

• Easy to answer
• But information collected can be crude
Discrete Questions/Variables
• Nominal question: Category without specific meanings in numbers.

• Please answer your gender.

• 1. Man, 2. Woman, 3. Other, 4. Prefer not to say etc.
• What is your employment status?
• 1. Full-time employee, 2. Part-time employee, 3. Student,
• 4. Housewife/husband, 5. Unemployed etc.
• What is your nationality?
• 1. Afghanistan, 2. Albania, 3. Algeria, etc

• The numbers associated with each option is purely symbolic

• There is no ordering between the options
Discrete Questions/Variables
• Ordinal question: Categories with meaningful ordering.

• Examples:
• How would you rate the liveability of Hong Kong out of 10? (1 being not at all, 10 being
perfect)
• How likely are you to recommend your friends to take this course? (0 being never, 5
being absolutely)
• Overall, how do you evaluate the food quality of Alfafa? (0 being awful, 10 being perfect)
• How would you rate the frequency of your social media usage? (0 no use; 1 somewhat
frequent; 2 frequent; 3 very frequent)

• Compared with nominal questions, the numbers associated with options here contain
meanings
• Common choices are 5 or 7 point scale, so there is a “middle choice”
• Sometimes scholars intentionally choose even numbers (like 4) to force a preference
Some Notes on Question Types
• Collect to the most detailed level, when possible
• For instance, collect number of hours of social media usage, rather than “whether you
used social media yesterday or not”
• Why? A more crude measurement can be recovered from a more granular
measurement
• From hours of usage, we automatically know whether the person used it or not
• From “strongly disagree”, “disagree”, “agree”, “strongly agree”, we can collapse the categories
into “disagree” and “disagree”

• However, collecting more information comes at a cost

• Harder for the respondent to answer – more cognitive burdern
• Answer a question on a scale of 1-3 is easier than a question on a scale of 1-10
• Do you know exactly how many hours of social media you used?
• Privacy: people may not want to give away too detailed information
• Time: more detailed questions often take longer to complete
• Sometimes we are interested in people’s subjective evaluation

• To sum up, choose the measurement that your research question needs!
Back to our example
• Social media usage:
• Continuous
• How many hours of social media have you used yesterday?
• Binary
• Have you used social media yesterday?
• Ordinal
• How would you rate the frequency of your social media usage? (0 no use; 1 somewhat
frequent; 2 frequent; 3 very frequent)
• Nominal (NA)

• Mental well-being:
• Continuous (NA)
• Binary
• Have you felt stressed yesterday?
• Ordinal
• How would you rate the frequency of your social media usage? (0 no use; 1 somewhat
frequent; 2 frequent; 3 very frequent)
• Nominal (NA)

• What if I am not sure what measurement to use?

• Check the literature!
• Make sure that your measurement has appeared in past academic research
An exercise
• Does part-time gig work affect worker’s life satisfaction?
• Come up with some measurements!

• Participation in gig work

• Continuous
• Binary - Particpation in gig work
-> continuous: hours worked per week (no. of hours) - exact number
• Nominal -> binary: are you currently engaged in gig work? yes/no - particpation status
• Ordinal -> nominal: type of gig work? eg. 1. freelance, 2. delivery ….
-> ordinal: frequency of gig work engagement (0 never; 1 occasionally; 2 frequently; 3 always)

• Life Satisfaction - Life statisfaction

-> continuous: life statisfaction on a scale from 1 to 10
• Continuous -> binary: life statisfaction? satisfied/ not satisfied
-> nominal: sources of life statisfaction? 1. family, 2. work, 3. social life …
• Binary -> ordinal: self-reported life statisfaction (1 very dissatisfied, 2 dissatisfied, 3 neutral, 4 satisfied, 4 very satisfied)
• Nominal
• Ordinal
Pitfalls in Survey Question Design: Wording
• We want questions that are answerable
• Respondents know what the questions means and know what their answer is

• Some common pitfalls that lead to problematic questions

• Wording
• Vague question
• Vague option
• Confusing wording
• Ordering
• Priming
• Carryover
• Anchoring
Pitfalls in Survey Question Design: Wording
• Vague question
• Not clear in meaning.
• ⇒ Specify the definition/situation so that respondents clearly understand what they are
asked.

• Bad Example:
• Do you think that adolescents are affected by social media?

• More Specific Example:

• Do you think that mental health of university students in Hong Kong are hindered by
excessive use of Instagram?
Pitfalls in Survey Question Design: Wording
• Vague option

• Example:
• How often do you attend classes at the University?
• 1. Very often 2. Quite often 3. Not often 4. Never

• Problem: Definition of each choice might significantly differ across respondents.

• Prepare more objective choices.

• * Yet, subjective choices are sometimes useful.

• Clearer example:
• On average, how often do you attend lectures at the University over the last 6 months?
1. Everyday 2. 3-5 days a week 3. 1-2 days a week … …
• On average, how many days do you attend lectures every month at the University?
__________days / month
Pitfalls in Survey Question Design: Wording
• Confusing wording

• Double-barreled question:
• In general, do you think the government’s response to the pandemic was poor and
therefore people are dissatisfied with their current situation?
• Include one topic in one question!

• Overlapped choices:
• What type of device are you currently using? (choose one)
1. Personal Computer 2. University-owned Computer 3. Tablet
4. Smartphone 5. Other ________________

Related: do not forget “no opinion”, “do not know”, “no choice” options
Pitfalls in Survey Question Design: Ordering
• Priming: Exposure to a stimulus activates a concept in memory that is then given
increased weight in subsequent judgment tasks

• If you show people a news story first..

• And then asks people a question:

• What is the most important agenda in relation to the
• COVID-19 pandemic?
• 1. Healthcare
• 2. Economy
• 3. Social relationship
• 4. Education

• However, this can be useful in survey experiments if we are interested in the priming
effect!
• We may want to know whether reading news stories affect people’s opinion on COVID
Pitfalls in Survey Question Design: Ordering
• Priming:

• Another example:

• If the following question is asked the first…

• How much debt are you in right now?

• And then…

• How stressed are you right now?

• Again, this can be useful in survey experiments!

• Suppose for the other half of respondents, we switch the order between two questions…

• That gets us the extent to which thinking about debt affects stress level
Pitfalls in Survey Question Design: Ordering
• Carryover: Respondents perceive two questions as related and therefore use similar
considerations in answering both.

Q1. Some studies show watching anime has a negative impact on child development. Do
you agree?

Q2. How many hours did you spend to watch anime last week?
Pitfalls in Survey Question Design: Ordering
• Anchoring: Early questions set a standard to which later questions are compared.

• Q1. On average, about 60% of university students positively evaluate online learning
worldwide. Do you think it is high, low, or neither?

• Q2. In Hong Kong, about 30% of university students positively evaluate online learning.
Do you think it is high, low, or neither?

• Q3. How do you evaluate your experience of online learning during the pandemic?
Choose one figure between 0 (not at all) and 10 (completely satisfied)
Asking Sensitive Questions
• More often than you may think, people do not want to answer your survey questions
• Income
• Substance abuse
• Academic integrity
• Criminal and other illegal activity history
• etc.
• Causes of insincerity: social desirability bias, shame, fear of legal repercussions

• To get this information at the individual level is difficult

• When they do not want to tell, they do not tell

• Some remedies:
• Anonymity/confidentiality assurance
• Use of vignettes (do not ask about the person him/herself):
• “How common do you think drug use is among college students?”
• “Among your friends, what proportion of them do you think have cheated in assignments?
Asking Sensitive Questions
• We cannot get personal-level response, but can get population prevalence
• There are two ways to know population prevalence without knowing any individual response: list
experiment and randomized response method

• List experiment: Randomly assign all respondents to two groups, for each group ask a list question
with one difference in the listed items:

• You can get the population prevalence by calculating the mean difference between the two groups!

• Group 1: how many of the following have you done in the last week?
• A. Went to a class
• B. Have lunch with a friend
• C. Watch Netflix
• Group 2: how many of the following have you done in the last week?
• A. Went to a class
• B. Have lunch with a friend
• C. Watch Netflix
• D. Use marijuana
Asking Sensitive Questions
• Randomized response method:
• Ask a sensitive question in the authentic form “have you used drug last week”, “have you
engaged in any illegal activity”, etc.
• BUT
• Before the respondent answers, ask the respondent to flip a coin/roll a dice/flip a card, etc. And
only ask the respondent with a certain realized outcome to answer truthfully, and others just
answer “yes”.
• Therefore, even if they answered “yes”, there is no way to know whether they committed the
sin or not.

• For example, if it is a coin flip example, if no one used drug, we expect those who answer “yes”
to be 50 percent. While the rest 50 percent who answer truthfully would say “no”
• If it turns out to be 60 percent, for instance, it means that among the 50 percent who are
expected to answer truthfully, 10 percent said “yes”, so 20 percent of people have used drugs.
Survey Experiments
• Survey experiment is the form where a survey and an experiment is combined
• Two types of survey experiments: sensitive topic-related (list experiment etc;) causal effect
orientated

• Among the causal effect oriented survey experiments, two common strategies
• Priming: randomly split respondents into groups, each exposing to different materials (typically texts,
can be figures/videos as well)
• Vignette/factorial: imagine a hypothetical person, and ask the respondent what would the
hypothetical person do

• Why do we want to do experiment at all?

Survey Experiments
• Benefits of an experiment:
• Suppose we still want to study social media effect
• We split people randomly into two groups
• Also suppose that we have some kind of control over people’s social media usage, and we force one
group to use social media, and the other not use social media today (perfect compliance)
• Then because of randomization, we do not need any control variable!
• A simple comparison between treatment and control group gives credible causal effect
• Why? Because social media usage is randomly assigned, nothing causes it, and therefore there is
absolutely no confounder

Age, gender, ethnicity, family background, levels of education, etc.

Social Media Usage Sleep Quality Mental Well-Being

Survey Experiments
• Example: priming
• Suppose we want to study the effect of exposure to violence on political opinion
• …which is heavily confounded

• We can run an experiment:

 Survey Experiments
Treatment Text (Exposure to Violence):
Imagine the following scenario:
Last night, a violent confrontation erupted in the downtown area of your city. A group of armed individuals clashed with law
enforcement officers during a protest that quickly turned into a riot. The scene was chaotic, with sounds of gunfire, shattering
glass, and people screaming in panic. Several buildings were set on fire, and numerous cars were vandalized. Emergency services
struggled to control the situation as the violence escalated. By the end of the night, multiple people were injured, and the city
center was left in ruins. Authorities are still investigating the cause of the violence and working to restore order.
Control Text (Neutral Event):
Imagine the following scenario:
Last night, a community event took place in the downtown area of your city. Residents gathered for a peaceful evening of music,
food, and entertainment. The local park was filled with families enjoying picnics, children playing games, and live performances
by local artists. Food vendors lined the streets, offering a variety of delicious treats, and community groups set up booths to
share information about their activities. The event concluded with a beautiful fireworks display, and everyone left the park
feeling happy and content. Authorities reported that the event went smoothly, with no incidents or disturbances.
Survey Questions (Post-Text):
After reading the text, please answer the following questions:
[Link] what extent do you support increasing funding for law enforcement? (Scale: 1 - Strongly oppose to 7 - Strongly support)
 Survey Experiments
• Example: vignette (adjust one condition at a time); factorial (adjust multiple conditions)
• Suppose we want to estimate the effect of income on fertility intention
• Difficult because the relation between income and fertility intention is confounded
• We can instead do a vignette/factorial experiment
• Randomly partition respondents into groups and for each group, give them a different combination of
treatment (highlighted texts)

You are 30 years old, living in a small apartment. You completed high school and did not pursue further
education. Currently, you are unemployed and actively looking for work. Your household income is around
$200,000 per year.

Given this scenario, please answer the following questions:

1. How many children do you ideally want to have in your lifetime?
2. How likely are you to plan for a child in the next two years? (Scale: 1 - Very Unlikely to 7 - Very Likely)
Survey Experiments
• Pros:
• Yields immediate causal estimate. No worries for confounding. Hooray!
• Cons:
• Can only study very small treatments that we can impose during a survey
• In a survey we cannot control for people’s social media for a week
• Maybe ask the respondent to browse Instagram for one minute
• The treatment in a survey experiment is usually only one approximation of the actual treatment in
reality that we care about
• For priming: We want to know the effect of exposure to violence on political attitude, we cannot
actually expose people to violence but can ask respondent to read coverage of a violent event
• For vignette: We cannot assign people’s income, but we can ask a person to imagine what could
happen if he/she has higher income; validity of result depends on people’s accuracy of self-
prediction
This Week
• Transforming hypotheses to variables
• Types of Survey Questions
• Continuous
• Discrete: binary, nominal, ordinal
• Open-ended
• Common Pitfalls of Asking Questions
• Vague question
• Vague option
• Confusing wording
• Asking Sensitive Questions
• List experiment
• Randomized response
• Survey Experiment
• Priming
• Vignette/Factorials
Optional exercise
For each table, decide on an independent and a dependent variable

How would you measure them?

What are the confounders?

Can the sensitive question techniques be used?

Can the survey experiment technique be used (to estimate the effect of at least
a dimension of the independent variable)?