Research Methods 2

Process and Examples

Khalid Alemerien
Tafila Technical University
Spring 2024/2025
 Research Problems
 Literature Reviews: Review Process, Review guidelines,
Validity threats, Review decisions
 Review Examples
 Introduction to Quantitative Research
 Study Designs
 Survey Research
 Survey Research Examples
 Controlled Experiments (Elements and Methods)
 Controlled Experiment Examples
 Case Studies Examples

2
3
 Source of information
◦ Proposals by academic staff or departments
◦ Past projects
◦ Brainstorming
◦ Your own goals and learning objectives
◦ Reading about / working in the subject area
◦ Systematic analysis of the subject area
 Research Territory Maps
 Show how topics related to each other
 Relevance Trees
 Break down a particular subject or research question into lower
and lower levels of detail
 Spider Diagrams
 Combines features of Research Territory Maps with those of
Relevance Trees

4
5
6
7
 A problem statement is an explanation in research that describes the
issue that is in need of study.
 It is expected to be brief and concise, and should not include the
findings of the research or detailed data.
 It is going to define the problem, which can be thought of as a gap in
the information base.
 Its purpose is to summarize the current information and where a lack
of knowledge may be presenting a problem that needs to be
investigated.
 It defines the problem and proposes a way to research a solution, or
demonstrates why further information is needed in order for a
solution to become possible.

8
 How will your research contribute to the existing knowledge base in your
field of study?
 How is it significant?
 Why does it matter?
 What is the context?
 Example:
We propose to design and implement self-contained games-themed
programming assignment modules based on the Microsoft XNA framework
for CS1/2 courses taught in C#.
 Context For Our Project: Survey of Related Work
◦ Games development classes.
◦ Games programming classes.
◦ Games client classes.

9
 A literature review is an account of
what has been published on a topic
by accredited scholars and
researchers.
 Literature Process Steps
 Step 1:
1. Select a topic to complete your
project.
2. Establish your research questions
and organize your literature into
logical categories around the
subject/ topic areas of your
questions.
3. Make sure you understand the
concept of ‘broader’ and
‘narrower’ terms.

10
 Step 2:
1. Use a variety of resources - locate
books, journals, and documents
that contain useful information
and ideas on your topic.
2. Internet sites, theses &
dissertations, conference papers,
ePrints and government or
industry reports.
3. Reference sources such as
dictionaries can assist in defining
terminology, and encyclopaedias
may provide useful introductions
to your topic by experts in the
field and will list key references.

11
 Step 3:
1. a number of databases that
provide full text access to articles,
that allow you to refine your
search to ‘peer reviewed’
journals.
2. Use the And, Or, Not operators,
Wildcards and Logical Brackets
when searching in the databases.
For instance, you can use And to
narrow your search while the
operator OR expands your search.
Not, on the other hand, helps to
exclude irrelevant information
from your search results.

12
 Step 4:
 Group the sources into the themes and
sub-themes of your topic. As you read
widely but selectively in your topic
area, consider what themes or issues
connect your sources together.
◦ Do they present one or different
solutions?
◦ Is there an aspect of the field that is
missing?
◦ How well do they present the
material and do they portray it
according to an appropriate theory?
◦ Do they reveal a trend in the field?
◦ Pick one of these themes to focus
the organization of your review.

13
 Step 5:
 You can organize the review in many
ways; for example,
1. you can center the review
historically (how the topic has
been dealt with over time);
2. or center it on the theoretical
positions surrounding your topic
(those for a position vs. those
against, for example);
3. or you can focus on how each of
your sources contributes to your
understanding of your project.
 Summary (table)

14
 Surveys are designed to collect information from a small number of people to be
representative of a larger number of people to be studied, for instance, the
information about their...
◦ Attributes
◦ Behavior
◦ Preferences
◦ Attitudes
◦ Opinions
 Systematically draw information from a certain population in order to:
◦ Describe demographic information (e.g., age, gender, school year, affiliation)
◦ Draw patterns from the population studied
◦ Explain trends out of phenomenon
 There are several types of surveys, including:
◦ Phone
◦ Face-to-Face
◦ Paper and Pencil
◦ Online (Web-based): e.g., SurveyMonkey, Qualtrics

15
 There are two basic formats:
◦ Closed-ended – recommended
◦ Open-ended.
◦ A combination of both in a survey depending on your goal.
 Recommendations
◦ Invitation:
Creating a good invitation to participate your survey is important. In your survey
invitation (email) letter, try to include 1-2 sentences describing a purpose or goal of
your survey.
◦ Disclosure:
 Mention length of survey
 Ensure that responses are confidential
 Ensure that participation is voluntary
 Provide contact information in the cases where participants have questions
about survey
◦ Pretesting:
Before administering a survey, make sure you test your survey in advance. Survey
pretesting will help you determine the effectiveness of your survey.
16
17
18
19
20
 A case study is a detailed study of
a specific subject, such as a person,
group, place, event, organization,
or phenomenon. Case studies are
commonly used in social,
educational, clinical, and business
research.

 A case study research design

usually involves qualitative
methods, but quantitative
methods are sometimes also used.
Case studies are good for
describing, comparing, evaluating
and understanding different
aspects of a research problem.

21
22
23
24
 Origin in the natural sciences - scientific method
 Systematic empirical investigation of quantitative properties and phenomena
and their relationships.
 The goal is develop models, theories, and hypotheses pertaining to natural
phenomena (how it works)
 The research is generally driven by hypotheses, which are formulated and
tested rigorously.
 Measurement is fundamental since it gives the connection between
observation and the formalization of the model, theory and hypothesis
 Repeatability of the experiments and testing of hypotheses are vital to the
reliability of the results, since they offer multiple opportunities for scrutinising
the findings.
 Quantitative research is based more directly on its original plans and its
results are more readily analysed and interpreted.
 Quantitative research is concerned with the collection and analysis of data in
numeric form. It tends to emphasize relatively large-scale and representative
sets of data, and is often, falsely in our view, presented or perceived as being
about the gathering of `facts'.
25
 Quantitative Research - Key Characteristics
◦ Control: This is the most important element because
 it enables the scientist to identify the causes of his or her observations.
 Experiments are conducted in an attempt to answer certain questions.
They represent attempts to identify why something happens, what
causes some event, or under what conditions an event does occur.
 Control is necessary in order to provide unambiguous answers to such
questions.
◦ Operational Definition: This means that terms must be defined by the
steps or operations used to measure them. Such a procedure is necessary
to eliminate any confusion in meaning and communication.
◦ Replication: To be replicable, the data obtained in an experiment must be
reliable; that is, the same result must be found if the study is repeated.
◦ Hypothesis Testing: The systematic creation of a hypothesis and subjecting
it to an empirical test.

26
 Strengths
◦ Precision - through quantitative and reliable measurement
◦ Control - through sampling and design
◦ Ability to produce causality statements, through the use of controlled
experiments
◦ Statistical techniques allow for sophisticated analyses
◦ Replicable
 Limitations
◦ The complexity of human experience.
◦ The human agency people do not all respond in the same ways.
◦ It leads to the assumption that facts are true and the same for all people all
of the time.
◦ Quantitative research often produces banal and trivial findings of little
consequence due to the restriction on and the controlling of variables.
◦ It is not totally objective because the researcher is subjectively involved in
the very choice of a problem as worthy of investigation and in the
interpretation of the results.

27
 Experimental research consists of three main components: the sample,
independent and dependent variables

 The sample consists of a number of test subjects randomly chosen from the
population of available subjects.
 Independent variables are the conditions which may influence the behaviour of
the subjects in the sample.
 Values of dependent variables may change as a result of a change in the
independent variables. The changes are measured and submitted to statistical
analysis to describe relationships between the variables. Typically laboratory
experiments are conducted in a more controlled manner than the field variety.

 Methods of control
◦ Random assignment
◦ Control group
◦ Masking (Blinding)

28
29
 In experiments,
researchers manipulate
independent variables
to test their effects on
dependent variables.

 In a controlled
experiment, all variables
other than the
independent variable
are controlled or held
constant so they don’t
influence the dependent
variable.

30
  Internal validity refers to the degree of confidence that the causal relationship
being tested is trustworthy and not influenced by other factors or variables.
 Threats to internal validity
There are eight factors that can threaten the internal validity of your research.

Threat Explanation Example

History Unanticipated events change A new (better) manager starts during the
the conditions of the study study, which improves job satisfaction.
and influence the outcome.
Maturation The passage of time During the six-month experiment, employees
influences the dependent become more experienced and better at their
variable (job satisfaction). jobs. Therefore, job satisfaction may improve.
Testing The pre-test (used to Employees feel the need to be consistent in
establish a baseline) affects their answers in the pre-test and post-test.
the results of the post-test.
Instrumentation There is a change in how the The questionnaire in the post test contains
dependent variable is extra questions compared to the one used for
measured during the study. the pre-test.
31
 Threat Explanation Example
Participant Participants in the control and Instead of a randomly assigning employees to
selection experimental group differ one of two groups, employees can volunteer to
substantially and can thus not be participate in an experiment to improve job
compared. satisfaction. The experimental group now
consists of more engaged (more satisfied)
employees to begin with.
Social Interaction between participants The group of employees with fixed working
interaction from different groups influences hours are resentful of the group with flexible
the outcome. working hours, and their job satisfaction
decreases as a result.
Attrition Over the course of a (longer) Really dissatisfied employees quit their job
study, participants may drop during the study. The average job satisfaction
out. If the drop out is caused by will now improve, not because the “treatment”
the experimental treatment (as worked, but because the dissatisfied employees
opposed to coincidence) it can are not included in the post-test.
threaten the internal validity.
Regression Extreme scores tend to be closer Employees who score extremely low in the first
towards to the average on a second job satisfaction survey probably show greater
mean measurement. gain in job satisfaction than employees who
scored average. 32
  External validity refers to the extent to which results from a study can
be applied (generalized) to other situations, groups or events.
 There are three main factors that might threaten the external validity
of our study example.
Threat Explanation Example
Testing Participation in the pre-test The questionnaire about job satisfaction used
influences the reaction to the in the pre-test triggers employees to start
‘treatment’. thinking more consciously about their job
satisfaction.
Sampling bias Participants of the study Employees participating in the experiment are
differ substantially from the significantly younger than employees in other
population. departments, so the results can’t be
generalized.
Hawthorne Participants change their The employees make an extra effort in their
effect behavior because they know jobs and feel greater job satisfaction because
they are being studied. they know they are participating in an
experiment.
33
 Construct validity
◦ concerns the extent to which your test or measure accurately assesses
what it’s supposed to.
◦ To achieve construct validity, you have to ensure that your indicators and
measurements are carefully developed based on relevant existing
knowledge.
◦ The most common threats are:
 Poor operationalization
 Experimenter expectancies
 Subject bias

 To produce valid results, the content of a test, survey or measurement

method must cover all relevant parts of the subject it aims to measure. If
some aspects are missing from the measurement (or if irrelevant aspects are
included), the validity is threatened.

34
- [Link]
problem-statement-examples/
- [Link]
sal/[Link]
- [Link]

- [Link]

- Streefkerk, R. (2022, October 10). Internal vs. External Validity |

Understanding Differences & Threats. Scribbr. Retrieved November 3, 2022,
from [Link]
- [Link]

- Ullrich Hustadt, Research Methods in Computer Science, University of

Liverpool.

35