Chapter 5

Measuring Variables and Sampling

Learning Objectives

 Explain the meaning of measurement.

 Compare and contrast Steven’s four scales of measurement.
 Explain the difference between reliability and validity.
 Describe the different types of reliability.
 Describe the different types of validity evidence and the strategies used to obtain
evidence of validity.
 Explain the meaning of sampling and its terminology.
 Describe each of the random sampling techniques, including their strengths and
weaknesses.
 Describe each of the nonrandom sampling techniques, including their strengths and
weaknesses.
 Explain the difference between random selection and random assignment.
 Describe the considerations involved in determining the appropriate sample size.
 Describe the sampling approaches used in qualitative research.

Chapter Outline

Variable and Measurement

• Variable
– a condition or characteristic that can take on different values or categories
– e.g., gender, reaction time
• Measurement
– the assignment of symbols or numbers to something according to a set of rules
– gender – male/female
– reaction time – minutes or seconds
Scales of Measurement
• Stevens (1946)
– measurement can be categorized by the type of information that is communicated by the
symbols assigned to the variables of interest
• Nominal scale
– use of symbols to classify or categorize
– e.g., gender, ethnicity, religion, major in college
• Ordinal scale
– rank-order scale of measurement
– equal distances on scale not necessarily equal on dimension being measured
– e.g., finishing order in a race, letter grades (ABCDF), SES (low, medium, high)
• Interval scale
– equal distances between adjacent numbers
– e.g., temperature on Fahrenheit or Celsius scale
• Ratio scale
– highest scale of measurement
– same properties of other scales plus absolute zero point
– e.g., weight, height, number grades, temperature on Kelvin scale, reaction time, length
Psychometric Properties of Good Measurement
• Reliability
– refers to the consistency or stability of the scores of your measurement instrument
• Validity
– refers to the extent to which your measurement procedure is measuring what you think it is
measuring and whether you have interpreted your scores correctly
• A measure must be reliable in order to be valid, but a reliable measure is not necessarily valid
Types of Reliability
• Reliability
– refers to the consistency or stability of the scores of your test, assessment, instrument, or
raters
• Test-retest reliability
– consistency of individual scores over time
– same test administered to individuals two times
– correlate scores to determine reliability
– how long to wait between tests
 typically an increase in time between testings will decrease reliability
• Equivalent-forms reliability
– consistency of scores on two versions of test
– each version of test given to the same group of individuals
– e.g., SAT, GRE, IQ
• Internal consistency reliability
– consistency with which items on a test measure a single construct
– e.g., learning, extraversion
– involves comparing individual items within a single test
– coefficient alpha (Cronbach’s alpha) is common index
 should be +0.70 or higher
 multidimentional tests will generate multiple coefficient alphas
• Interrater reliability
– degree of agreement between two or more observers (raters)
– how is interobserver agreement calculated
– nominal or ordinal scale
 the percentage of times different raters agree
– interval or ratio scale
 correlation coefficient
Validity
• The accuracy of the inferences, interpretations, or actions made on the basis of any
measurement
• Construct validity
– involves the measurement of constructs
– e.g., intelligence, happiness, self-efficacy
– do operational definitions accurately represent construct we are interested in
– operationalization is a never ending process
Methods Used to Collect Evidence of Validity
• Content validity
– validity assessed by experts
 do items appear to measure construct of interest? (face validity)
 were any important content areas omitted?
 were any unnecessary items included?
• Internal structure
– how well do individual items relate to the overall test score or other items on the test
– uni- vs. multi-dimentional constructs
– factor analysis
 statistical procedure used to determine the number of dimensions present in a set of
items
– homogeneity
 item to total correlation (coefficient alpha)
• Relations to uther variables
– criterion-related validity
 criterion
– the standard or benchmark that you want to correlation with or predict accurately on
the basis of your test scores
 predictive validity
– using scores obtained at one time to predict the scores on a criterion at a later time
– e.g., GRE and graduate school GPA, LSAT and law school GPA, MCAT and
medical school GPA
 concurrent validity
– degree to which scores obtained at one time correctly relate to the scores on a known
criterion obtained at the same time
– e.g., new depression scale and Beck Depression Inventory
– convergent validity
 extent to which test scores relate to other measures of the same construct
 e.g., same as predictive and concurrent validity
– discriminant validity
 extent to which your test scores do not relate to other test scores measuring different
constructs
 e.g., happiness and depression, depression and IQ
– known groups validity evidence
 extent to which groups that are known to be different from one another actually differ
on the construct being developed
 e.g., females high on femininity and males high on masculinity
Using Reliability and Validity Information
• Norming group
– the reference group upon which reported reliability and validity evidence is based
• Sources of Information about tests
– Mental Measurements Yearbook
– tests in print
– PsycINFO and PsycARTICLES
Sampling Methods
• Sample
– a set of elements selected from a population
• Population
– the full set of elements or people from which the sample was selected
• Sampling
– process of drawing elements from population to form a sample
• Representative sample
– a sample that resembles the population
• Equal probability method of selection method (EPSEM)
– each individual element has an equal probability of selection into the sample
• Statistic
– a numerical characteristic of sample data
– e.g, sample mean, sample standard deviation
• Parameter
– a numerical characteristic of population data
– e.g., population mean, population standard deviation
• Sampling error
– the difference between the value of the sample statistic and the value of the population
parameter
• Sampling frame
– a list of all the elements in a population
• Response rate
– the percentage of individuals selected to be in the sample who actually participate in the
study
Sampling Techniques
• Biased sample
– a non-representative sample
• Proximal similarity
– generalization to people, places, settings, and contexts that are similar to those described in
the study
Random Sampling Techniques
• Simple random sampling
– choosing a sample in a manner in which everyone has an equal chance of being selected
(EPSEM)
– sampling “without replacement” is preferred
 – random numbers generators simplify the process
 [Link]
 [Link]
• Stratified random sampling
– random samples drawn from different groups or strata within the population
 groups should be mutually exclusive
 strata can be categorical (nominal or ordinal) or quantitative (interval or ratio)
 proportional stratified sampling
– involves insuring that each subgroup in sample is proportional to the subgroups in the
population
• Stratified random sampling example (proportional)
– strata – gender (males/females)
– population – presidents of APA, N = 122
 14 female presidents (11%)
 108 male presidents (89%)
– sample – n = 100
 11 female presidents drawn randomly
 89 male presidents drawn randomly
• Cluster random sampling
– involves random selection of groups of individuals
– clusters
 a collective type of unit that includes multiple elements (has more than one unit in it)
 e.g., neighborhoods, families, schools, classrooms
– one-stage cluster sampling
 randomly select clusters and using all individuals within
 e.g., randomly select 15 psychology classrooms using all individuals in each classroom
– two-stage cluster
 randomly select clusters AND
 randomly choosing individuals within each chosen cluster
 e.g., randomly select 30 psychology classrooms, then randomly select 10 students from
each of those classrooms
• Systematic sampling
– involves three steps
 determine the sampling interval (k)
 population size divided by desired sample size
 randomly select a number between 1 and k, and include that person in your sample
 also include each kth element in your sample
 periodicity
 potential but uncommon problem
 problematic situation in systematic sampling that can occur if there is a cyclical
pattern in the sampling frame
• Systematic sampling example
– population N = 100
– sample n = 10
– k = 10
– randomly select a number between 1 and 10
 e.g., 5
– the 5th person in the population will be included in the sample along with person #15, 25,
35, 45, 55, 65, 75, 85, and 95
Nonrandom Sampling Techniques
• Generally produce biased, non-representative samples
• Convenience sampling
– using research participants that are readily available
– e.g., college students
• Quota sampling
– identifying quotas for individual groups and then using convenience sampling to select
participants within each group
– e.g., gender – 25 males and 25 females
– e.g., year in school – 15 freshman, 15 sophomores, 15 juniors, and 15 seniors
• Purposive sampling
– involves identifying a group of individuals with specific characteristics
– e.g., college freshmen who have been diagnosed with ADHD
• Snowball sampling
– technique in which research participants identify other potential participants
– particularly useful in identifying participants from a difficult to find population
– e.g., Spanish speaking ESL students, parents of children with autism
Random Selection and Random Assignment
• Random selection
– involves selecting participants for research from the population to be included in the
sample
– purpose is to obtain a representative sample
• Random assignment
– involves how participants are assigned to conditions within the research
– purpose is to create equivalent groups to allow for investigation of causality
– e.g., 20 college students sign up to be participants in a study. Each is randomly assigned to
the treatment or control group of the study
Determining Sample Size
• Five simple rules for determining sample size
• if less than 100, use entire population
• larger sample sizes make it easier to detect an effect or relationship in the population
• compare to other research studies in area by doing a literature review
• use Table 5.3 in book for a rough estimate
• use a sample size calculator (e.g., G-Power)
• Larger sample sizes are needed if population is
– heterogeneous
 composed of widely different kinds of people
– you want to break down the sample into multiple subcategories
 e.g., look at males and females separately
– if you want to obtain a narrow or more precise confidence interval
– when you expect a small effect or weak relationship
– when you use less efficient methods of sampling
 e.g., cluster sampling
– for some statistical techniques
– if you expect a low response rate
Sampling in Qualitative Research
• Qualitative research focuses on in-depth study of one or a few cases
• Several different sampling methods are available. It is common to mix several different
methods

Multiple choice questions

1. In the context of an experiment, a variable is

* a. any factor that can vary across participants or situations.
b. any phenomenon or characteristic that can be measured.
c. any phenomenon or characteristic of a participant or situation that has a specific value.
d. the unknown quantity that the experiment will determine.

2. ___________ is the assignment of symbols or numbers to something according to a set of

rules.

a. Variable
* b. Measurement
c. Validity
d. Reliability

3. ___________ is the simplest scale of measurement.

a. Ordinal
* b. Nominal
c. Ratio
d. Interval

4. Which of the following measurement scales is accurately paired with an example?

a. Interval—rankings of tennis players
b. Ratio—zip codes
c. Nominal—test scores on an exam
* d. Ordinal—a professor listing his students from the best to worst
5. What differentiates interval from ratio scales of measurement?
a. Interval scales use rank order; ratio scales do not
b. In a ratio scale equal distance on the dimension represent equal distance on the
dimension being measured; this is not true for interval
c. Scores of zero are not possible on interval scales
* d. Ratio scales include an absolute zero point – indicating the absence of what is being
measured

6. Temperature on a Kelvin scale is an example of

a. nominal measurement.
b. ordinal measurement.
c. interval measurement.
* d. ratio measurement.

7. Which of the following would represent scores on a nominal scale?

a. Attractiveness ratings on a scale of 1-5
* b. Coding religion as protestant = 1; catholic = 2, etc.
c. Temperature on a Celsius scale
d. Exam scores

8. ______________ refers to the consistency of results and ____________ is the extent to which
you are measuring what you think you are measuring.
a. Reliability; periodicity
b. Validity; reliability
* c. Reliability; validity
d. Convergence; divergence

9. Which of the following is TRUE?

a. A reliable measure is always valid

b. A valid measure is never reliable.
* c. A valid measure is always reliable.
d. A reliable measure is never valid.

10. In order to establish the reliability of a measure of intelligence, Kevin administers two forms
of the test to a group of students. Which of the following reliability coefficient values would
indicate the most reliability for the test?
a. 0.35
* b. 0.85
c. -0.85
d. 2.20
11. Jenna would like to establish the reliability of a new measure of self-esteem but she doesn’t
have enough time to administer her test more than once. Which of the following methods of
establishing reliability would you suggest to Jenna?
a. Equivalent forms
* b. Internal consistency
c. Multidimensional
d. Concurrent

12. If we include items assessing memory, logic, and verbal comprehension on an intelligence
test – as opposed to food preferences or shoe size – then we have satisfied which of the following
types of validity?
a. Discriminant
b. Convergent
* c. Face
d. Internal

13. Construct validity

a. is not needed if you use a good operational definition.
* b. is supported when similar results are obtained from different operationalizations of the
dependent variable.
c. is not needed if your measure if reliable.
d. is determined by replicating the results of your experiment.

14. Which of the following illustrates reliability?

a. Dean takes an IQ test and scores at the 60th percentile.
b. Scores on a new test of reading comprehension correlate highly with scores on well
established reading comprehension tests.
c. Fred scores poorly on one school's entrance exam but does better on another.
* d. Jacquie takes three practice GRE verbal exams and scores 300, 299, and 301.

15. A variable shows reliability when

a. enough experimenters decide to use it in their research.
b. it is accepted in the Encyclopedia of Psychology.
c. other researchers demonstrate that it does measure what it is supposed to measure.
* d. similar results are obtained each time it is measured.

16. The measurement of a variable has validity when

a. the same results are obtained each time it is measured.
b. it becomes an accepted variable in a given area of research.
c. it can be measured quantitatively.
* d. the inferences that are made from the measurement are accurate.
17. When conducting psychological research we want the research to be valid. Reliability and
validity are necessary ingredients of valid research. The relationship between validity and
reliability is that
a. if the research is reliable you can be certain that it is valid.
b. reliability causes validity.
* c. the research must be reliable for it to be valid but a reliable research measurement is
not necessarily valid.
d. the research must be valid for it to be reliable

18. Suppose you have created a new method of diagnosing anxiety disorders. How could you
demonstrate that your method is construct valid?
a. teach several licensed clinicians to use and evaluate your method.
b. use your method to diagnose the same group of participants repeatedly over the course
of several years, and see if you consistently arrive at the same diagnosis for a given
individual
c. have several specialists read about your new technique and invite their opinions
* d. use your method to diagnose a group of participants, then see if your diagnoses match
with diagnoses taken from other, established methods

19. Tom wanted to assess the reliability of his measure of anxiety so he had a group of
introductory psychology students complete the measure of anxiety on March 3rd and again on
March 25th. He then compared the scores that the students made on the two testing occasions
using a statistical technique called correlation. He used this quantitative index as his measure of
reliability. Tom used what method to assess reliability?
* a. test-retest
b. equivalent forms
c. split-half
d. Cronbach’s alpha

20. Eduardo decided to assess the reliability of the carbohydrate craving inventory he
constructed. He had constructed two identical versions of the inventory and a group of 50 people
took both versions. Then Eduardo compared the responses of these 50 people on the two
versions of the craving inventory for his assessment of the reliability of the inventory. Eduardo
used what method to assess reliability?
a. test-retest
* b. equivalent forms
c. split-half
d. Cronbach’s alpha
21. Jacqueline wanted to assess the reliability of ratings made of children’s aggressive behavior
so she had two students rate the degree of aggression displayed by each of 50 children while
engaged in play. She then compared the ratings made by these two students and computed the
degree of agreement between them. Jacqueline used what method of assessing reliability?
a. split-half reliability
* b. interrater reliability
c. internal consistency reliability
d. test-retest reliability
22. Afiya wanted to assess the reliability of students’ observations of children’s aggressive
behavior so she had two students observe 100 behaviors displayed by each of 50 children while
engaged in play. After viewing each behavior the students recorded the behavior as being
aggressive or nonaggresive. Afiya then computed the percentage of times the two students
agreed on their assessment of each behavior. Afiya used what method of assessing reliability?
* a. interobserver agreement
b. Cronbach’s alpha
c. internal consistency reliability
d. test-retest reliability

23. Owen wanted to assess the internal consistency of his measure of anxiety so he measured
reliability estimates by comparing items within his test. He reported a reliability estimate of .80.
What measure of reliability is Owen using?
* a. Cronbach’s alpha
b. path analysis
c. factor analysis
d. Riesen’s reliability estimate

24. Dr. Smarsh creates a new intelligence test and wants to assess its reliablility. She administers
her test to the same sample of individuals on two separate occasions one week apart. What type
of reliability is Dr. Smarsh using?

a. internal consistency reliability

b. equivalent forms reliability
* c. test retest reliability
d. interrater reliability

25. The publisher of the SAT can assess this type of reliability when individuals take the SAT
more than one time with different versions of the SAT being given each time. Which type of
reliability can be assessed?

* a. equivalent forms reliability

b. internal consistency reliability
c. Cronbach’s reliability
d. interrater reliability

26. Interobserver agreement is assessed with nominal or ordinal variables by calculating

* a. percentage of agreement
b. correlation coefficient
c. t-test
d. coefficient alpha
27. Gerald is developing a measure of shyness and he determines that students scoring high on
the measure also score high for introversion on a well-known introversion-extraversion scale.
The outcome best illustrates
a. face validity.
* b. concurrent validity.
c. predictive validity.
d. discriminant validity.

28. Cronbach’s alpha is a measure of

a. face validity.
* b. internal consistency.
c. predictive validity.
d. concurrent validity.

29. Students sometimes complain that scores on the Graduate Record Exam (GRE) are not
related to how well students perform in graduate school. Essentially the students are saying
that the GRE does not have
a. reliability.
b. internal consistency.
* c. predictive validity.
d. discriminant validity.

30. Discriminant validity refers to

* a. the degree to which the measure does not correlate with measures of different constructs.
b. the strong correlation the measure has with measures of similar constructs.
c. the degree to which the measure discriminates between different components of the
construct.
d. the degree to which the test measures a single construct.

31. Convergent validity refers to

a. the degree to which the measure does not correlate with measures of different constructs.
* b. the degree to which the measure does correlate with measures of similar constructs.
c. the degree to which the test measures multiple constructs.
d. the degree to which the test measures a single construct.

32. One way to assess construct validity is to establish that scores on the test in question do NOT
correlate with established scales that are dissimilar or conceptually unrelated concepts. (e.g, a
scale to measure depression would likely not correlate with scales designed to measure
happiness). This type of validity is called
a. concurrent validity.
b. convergent validity.
* c. discriminant validity.
d. predictive validity.
33. A study examines scores on an employment test and job performance six months later. This
study is most likely attempting to establish
* a. criterion validity.
b. face validity.
c. reliability.
d. construct validity.

34. Construct validity is most related to what other concept discussed in research methods?

a. reliability
* b. operational definitions
c. falsifiability
d. measurement

35. ________ validity is validity that is assessed by experts.

a. Predictive
b. Discriminant
c. internal
* d. Content

36. The extent to which groups that are known to diverge from one another actually differ on the
construct being developed is known as

a. discriminant validity
b. predictive validity evidence
* c. known groups validity evidence
d. criterion validity

37. When evaluating reliability and validity information of a published measure, it is important
to note the __________ upon which the information was gathered.
a. predictive sample
* b. norming group
c. estimate group
d. peer group

38. Some psychological tests are designed to measure more than one construct, they are
multidimensional. __________ is a statistical technique that can be used to determine the number
of dimensions that a particular measure is testing.
a. Cronbach’s alpha
c. Path analysis
c. Operationalization
* d. Factor analysis

39. The Mental Measurements Yearbook is a good place to find

 a. unpublished (but probably useful) tests.
* b. established standardized tests.
c. biographies of important people in test development.
d. a list of journal articles that use psychological tests.

40. ___________ refers to any sampling method in which each individual has an equal chance of
being selected for the sample.
* a. Equal probability selection method (EPSEM)
b. Convenience sampling
c. Stratified sampling
d. Homogeneous sampling selection

41. A(n) __________ is a list of all members of a population.

a. parameter
b. norming group
* c. sampling frame
d. equal probability selection method

42. A ____________ is the full set of all individuals of interest and is typically hard to assess
fully.

a. sample
* b. population
c. sampling error
d. statistic

43. Dr. Saucer is in the process of drawing individuals from a population to be included in a
sample. This process is called

a. sampling frame
b. parameter
* c. sampling
d. proximal similarity

44. Dr. Konrad contacts the registrar at his university to get a complete list of students enrolled.
He wants to draw a simple random sample from the entire population. He does this to make sure
he is creating a

a. framed sample
* b. representative sample
c. stratified sample
d. cluster sample
45. When using simple random sampling it is suggested that you _________ because it will lead
to a more representative sample.
a. sample with replacement
* b. sample without replacement
c. sample half with replacement and half without replacement
d. use a larger sample than if using a nonrandom method

46. A major advantage of randomly selecting participants from a population is that

a. it allows you to do your study with fewer participants and still find statistical
significance in your results.
b. you can be more confident that it was the manipulation of the independent variable that
caused the changes observed in the dependent variable.
* c. you can be more confident that your sample is representative of the population.
d. it is more likely that your sample will have the characteristics you need it to have.

47. In a truly random sample from a population,

a. all participants will be matched on important characteristics.
* b. all members of the population have an equal chance of being selected.
c. gender distribution should be 50% male and 50% female.
d. every member of the population has a 50:50 chance of being selected.

48. Suppose you wish to test a representative sample of people in your psychology class on
attitudes toward on-line courses. There are 40 people in the class, 30 females and 10 males.
What would be the most efficient strategy to ensure that your sample reflects the distribution
of males and females in the classroom population?
a. simple random sample
b. cluster sample
* c. proportional stratified sample
d. convenience sample

49. ____________ is to ____________ as population is to parameter.

* a. Sample; statistic
b. Statistic; sample
c. Sample; element
d. Random sampling; nonrandom sampling

50. A subset of data drawn from the larger population of interest is a

a. population.
* b. sample.
c. parameter.
d. quota.

51. Although statistics are calculated to represent parameters, we know that there can be a
difference between the calculated statistic and population parameters. This difference is called
* a. sampling error
b. sampling frame
c. sampling difference
d. parameter difference

52. Dr. Monroe is interested in surveying college students at her university who are first
generation college students. She contacts the registrar’s office and finds out that there are 3500
first generation college students total. She wants to include 350 of those students in her sample.
She then divided the population by the sample size and gets k = 10. She then picks a random
number between 1 and 10 which is 8. This determines that every 8th person in the population will
be included in the sample. What type of sampling method is this?

a. cluster sampling.
b. stratified sampling
c. random sampling
* d. systematic sampling

53. To study career aspirations among high school students in Alabama, a researcher randomly
selects 5% of the state’s school districts and gives all the students in each district a survey
designed to measure career goals. What sampling procedure is being used here?
a. quota
b. stratified
* c. cluster
d. EPSEM

54. Non-random sampling techniques typically produce ______ samples.

a. random
b. representative
* c. biased
d. systematic

55. Jim is conducting a survey to learn about student attitudes toward abortion. He passes out his
survey to the first 100 students that enter the cafeteria. What sampling method is Jim using?
* a. convenience
b. cluster
c. stratified
d. EPSEM

56. Which of the following is NOT a nonrandom sampling technique?

a. convenience
* b. cluster
 c. snowball
d. quota

57. Which of the following nonrandom sampling techniques would be most similar to stratified
random sampling?
a. convenience
* b. quota
c. cluster
d. purposive

58. If you are interested in participants in your study with very specific characteristics (e.g.
English as a second language) you may conduct ________ sampling and once you contact some
individuals who want to participate, you may then conduct ________ sampling in which
participants identify other potential participants.

a. snowball; purposive
b. matched; snowball
c. random; non-random
* d. purposive; snowball

59. ________________ of participants is done to obtain a representative sample, and

__________ of the participants is done to improve the experimental design of the study.
* a. Random selection; random assignment
b. Random selection; random sampling
c. Random assignment; random selection
d. Random assignment; matching

60. Random assignment of participants to the various groups in an experiment

a. makes it more likely that extraneous variables will impact the experiment.
* b. increases the probability that the groups are equivalent.
c. is essential if you want to generalize your results to the population.
d. is very difficult to do and is therefore not commonly done.

61. Your textbook authors suggest that if your population has fewer than 100 people you should
a. use a stratified sampling method.
b. use simple random selection.
c. find a larger population
* d. test everyone in the population.

62. Which of the following situations would NOT necessitate a larger sample size?
a. if your population is heterogeneous
b. if you plan to use multiple categories
c. if you expect a weak effect
* d. if you use proportional stratified sampling

63. Which of the following was NOT offered as a type of sampling method used for qualitative
research?
* a. purposive sampling
b. extreme case sampling
c. homogeneous sample selection
d. maximum variation sampling

Vocabulary

Define the following in psychological terms:

biased sample element

census equal probability of selection method (EPSEM)
cluster cluster random sampling
equivalent-forms reliability coefficient alpha
face validity concurrent validity
factor analysis content-related evidence or content validity
homogeneity internal consistency reliability
convenience sampling interobserver agreement
convergent validity evidence interrater reliability
criterion-related validity interval scale
cronbach’s alpha known groups validity evidence
discriminant validity evidence measurement
disproportional stratified sampling mixed sampling
multidimensional construct reliability coefficient
nominal scale representative sample
norming group response rate
one-stage cluster sampling sample
operationalization sample size calculator
ordinal scale sampling
parameter sampling error
periodicity sampling frame
population sampling interval
predictive validity simple random sampling
proportional stratified sampling snowball sampling
statistic proximal similarity
stratification variable purpose of random assignment
stratified random sampling purpose of random selection
systematic sampling purposive sampling
test-retest reliability quota sampling
two-stage cluster sampling random assignment
validation random selection
validity ratio scale
validity coefficient reliability
 variable

Essay questions

1) Define variable and measurement. Describe the difference between them and give an
example for each.

2) List the four scales of measurement from least complex to most complex – provide an
example of each.

3) Describe the difference between random selection of participants and random

assignment of participants to groups. What are the implications of inadequate random
selection of participants? What are the implications of lack of random assignment of
participants to groups?

4) Using an example, explain the difference between reliability and validity. Explain
what it means to say that a reliable measure is not always valid, but a valid measure is
always reliable.

5) Describe three different methods of assessing reliability.

6) Define interrater reliability and describe how interobserver agreement is calculated .

7) Describe how operational definitions and operationalization is important for our

understanding of construct validity. Use an example to assist in answering.

8) Discuss content validity and criterion-related validity (be sure to include a discussion
of predictive and concurrent validity).

9) What is factor analysis and how would the results be different if the assessment tool
was uni-dimentional compared to multi-dimensional?

10) Assume that you have created a new test to measure depression. Explain how you
could use converging evidence and discriminant evidence to establish the validity of your
instrument.

11) What is known groups validity evidence? Give an example to help explain your
answer.

12) Describe, in general terms, the difference between random sampling methods and
nonrandom sampling methods. Briefly describe the four random sampling techniques
presented in the text.

13) Describe the four nonrandom sampling techniques presented in your text. If, as
indicated in your text, these techniques are “weaker sampling methods” why would
researchers use them?
 14) Discuss the factors that are important for researchers to consider when determining
sample size.

15) Briefly summarize some of the sampling methods used in qualitative research.

Classroom Exercise Suggestions

1) Your students may already be familiar with the four scales of measurement but a simple way
to personalize your lecture is to provide them with a brief (anonymous) survey to complete at the
beginning of class. A sample survey is included below that includes examples of all four
measurement scales. After students complete the survey ask them to identify the items
representing each scale of measurement. You might also consider collecting the surveys and
using the data later in the semester to illustrate some simple statistical techniques.

DATA COLLECTION SHEET

(Do not write your name on this sheet.)

What is your age? _______

What is your gender? _______
What is your attitude toward abortion? (circle one below)
I’m pro-choice I’m pro-life I’m unsure or it depends
How many people were in your high school graduating class? _______
How many different states have you spent at least one night in? ______
What is your academic classification? FR SO JR SR
How many miles to your hometown? _______
Estimate how often you go home each semester ________
Do you own a cell phone? ___________
If you own a cell phone, how many minutes do you talk on your cell phone per day?
_______; how many text messages do you send and receive (total number): _______
Using the 7 point scale at the bottom of the page, rate your attitudes toward:
Britney Spears _______ Children _______
This class _______ Classical music _______
Beer _______ The Twilight Series _______
Thai food _________ Tattoos ________

1 2 3 4 5 6 7
Very Neutral Very
Negative Positive

2) To help students understand the concepts of reliability and validity, you might consider the
exercise proposed by Miserandino (2006) who used an Internet-based test as a fun way to
reinforce these topics. This assignment has the added benefit of reminding students, once again,
of the importance of critically evaluating information that they encounter on the Internet.

Miserandino, M. (2006). I scream, you scream: Teaching validity and reliability via the
ice cream personality test. Teaching of Psychology, 33 (4), 265-268.

3) Some of your students may believe that psychological research isn’t valid unless research
participants are randomly selected. Students should be aware that because of large populations of
interest, it may not be feasible to use random selection. Furthermore, you should point out that
because much research in psychology is “basic” – attempting to establish general relationships
between variables – random selection is less important. On the other hand, random assignment is
crucial in helping to eliminate extraneous variables and allowing an unambiguous interpretation
of our results. A good discussion of this distinction can be found in the Stanovich (2009) book
referenced below. The Enders et al. (2006) article describes a very simple classroom
demonstration of random assignment using a deck of playing cards.

Enders, C., Laurenceau, J., & Stuetzle, R. (2006). Teaching random assignment: A
classroom demonstration using a deck of playing cards. Teaching of
Psychology, 33(4), 239-242.
Stanovich, K.E. (2009). “But it’s not like real life!” The artificiality criticism and
psychology. In How to Think Straight About Psychology. Boston, MA: Allyn &
Bacon.

4) The Web site Clips for Class ([Link]) has an extensive list of videos for use
in many different psychology classes. Under the Statistics tab there are several videos that could
help students in this chapter. They are listed below with descriptions found on the Web site and
suggested questions for students after watching each clip.
 Nominal Variables Video
 This video explains nominal variables. How do these differ from ordinal variables?
 Sesame Street: Neil Patrick Harris Has Telly’s New Shoes
 In this brief song, characters sing and dance about shoes. Is the list of shoes given a
nominal scale, ordinal scale, interval scale, or ratio scale? What kind of descriptions
of shoes could be given to change it to a different kind of scale?

5) The Web site Clips for Class ([Link]) has an extensive list of videos for use
in many different psychology classes. Under the Research tab there are several videos that could
help students in this chapter. They are listed below with descriptions found on the Web site and
suggested questions for students after watching each clip.
 Reliability
 The headless professor discusses the concept of reliability and some of the more
common types of reliability measures. He presents a 2 x 2 contingency table to
demonstrate reliability as having more agreements than disagreements. What do
measures of reliability determine? Why did the headless professor present the 2 x 2
table to explain reliability? How is reliability defined in that instance? Why is a type
of interrater reliability important for clinicians?
 Validity
  The headless professor defines validity and how it is established by way of a 2 x 2
contingency table. He also discusses different types of validity such that in clinical
psychology (diagnosis and prognosis) and industrial/organizational psychology
(performance). Define validity in your own words. Why is it important to establish
validity in clinical psychology and industrial/organizational psychology? Why did the
headless professor present the 2 x 2 table to explain validity?
 Samples
 The headless professor explains the difference between subjects (participants),
groups, samples, and populations. He offers a clinical example and a marketing one,
and provides a diagram showing the relationship between a population, group,
sample, and participants. What is the difference between subjects/participants,
groups, samples, and populations? Develop a research question. Are you able to
identify your participants, relevant groups, secure a sample, and estimate your
population of interest?

6) Raechel Soicher created several videos demonstrating different sampling techniques using
Jing. Here are the links to those videos.

 Simple Probability Sampling ([Link]

 Systematic Probability Sampling ([Link]
 Stratified Probability Sampling ([Link]
 Cluster Sampling ([Link]