CHAPTER 3
RESEARCH METHODOLOGY
This chapter describes the method used to conduct the current study. The chapter
starts by presenting the research questions and research approach. Next, the background
information of the participants and the description of research instruments are described
in detail. The procedure of gathering and analyzing the data is then presented.
3.1. Research questions
To explore university EFL teachers’ perceptions of holistic and analytic rubrics in
assessing speaking, this study addresses two questions:
Research question 1: What are teachers’ perceptions of holistic and analytic
rubrics in assessing speaking?
Research question 2: How are teachers’ perceptions reflected in their reported
practice of using holistic and analytic rubrics in assessing speaking?
3.2. Research approach
This study is a descriptive study. Both quantitative and qualitative data were
collected and analysed to understand fully the research problem and strengthen the results
(Gay, Mills & Airasian, 2011). As Creswell (2003) stated that a study could be
considered to have a mixed method when it incorporates both quantitative and qualitative
data. In this study, quantitative data were collected through questionnaires to solve the
first research question, i.e., to find out teachers’ perceptions of holistic and analytic
rubrics in assessing speaking. Besides that, to have a deeper understanding about
university EFL teachers’ perceptions of holistic and analytic rubrics in assessing
speaking, the qualitative data were employed under the interviews. In addition, to find out
how teachers’ perceptions are reflected in their reported practice of using holistic and
analytic rubrics in assessing speaking, qualitative data were also collected through the
interview.
To investigate university EFL teachers’ perceptions of holistic and analytic
rubrics in assessing speaking, the questionnaire was first employed to collect data.
Subsequently, the interview was carried out in order to provide a deeper understanding of
the teachers’ thoughts based on their perceptions from the quantitative data. Cohen,
Manion and Morrison (2000) stated that the quantitative data might not give a full picture
of the particular slice of the reality she was investigating. Or the researcher might feel
less confident to generate the collected data from one specific method. Consequently,
with the combination of questionnaire and interview, this descriptive study is believed to
be approriate to obtain the data so as to find out the answers to the research questions.
3.3. Participants
The setting of this current study was at the universities located in Can Tho city. In
this study, the participants’ responses were collected with a convenient sampling method.
Forty university EFL teachers at the age of 22 to 43 in Can Tho city were invited to
participate in the study. Most of them were university graduates and postgraduates
majoring in English studies or English language teaching. Moreover, they received
professional training in teaching methods. Among forty participants, fifteen of them
earned Master degree while others held Bachelor of Education. In addition, they were
working at the university under the role of English lecturers. They were lecturing and
assessing the student’s performance in English listening and speaking classes. For the
gender of teachers, 26 of them were female and 14 of them were male.
Concerning the teaching background, the participants in this study had at least 2
years of teaching experience and assessing speaking skill. All of them were familiar with
applying holistic and and analytic rubrics in assessing. They regularly assessed the
students’ performance via holistic and analytic rubrics in their classes.
3.4. Research methods
The research instruments, the questionnaire and the interview were employed to
address the research questions. According to Fraenkel, Wallen & Huyn (1993), the
questionnaire can be employed via mail, or delivered to a substantial number of people at
the same time while the interviews are able to clarify any unclear points or to ask
interviewees for more extended responses. Two types of instruments used to collect the
data were displayed in the following table:
Table 3.1. Research instrument
Research questions instruments
What are teachers’ perceptions of holistic and analytic Questionnaire
1
rubrics in assessing speaking? Interview
How are teachers’ perceptions reflected in their
Interview
2 reported practice of using holistic and analytic rubrics
in assessing speaking?
3.4.1. Questionnaires
The questionnaire was designed in both Vietnamese and English version. It was
based on the theoretical framework described in the literature review. In particular, the
focus was on how the teachers perceived about holistic and analytic rubrics in assessing
speaking, especially on their understanding, awareness about holistic and analytic rubrics
in assessing speaking skill. The questionnaire consists of approximately 49 questions
with 4 clusters.
In particular, the first cluster was formulated with 7 questions to ask for
background information of participants including name, age, professional qualifications,
teaching experience, and attended EFL training courses/ teaching workshops. The second
cluster consisted of 7 questions to investigate teachers’ perceptions of the siginificance of
using rubrics in assessing speaking skill. For the next cluster, it comprised of 17
questions of the teachers’ perceptions of holistic rubrics in assessing speaking. The last
cluster included 18 questions to explore teachers’ perceptions of analytic rubrics. These
questions were set up by Likert’s scale of agreement with five points (strongly disagree,
disagree, neutral, agree, strongly agree). The Vietnamese version of the questionnaire
was given to all 40 university EFL teachers to prevent them from misunderstanding the
purpose of each item in the questionnaire (see Appendix B for full form of
questionnaire).
3.4.2. Interviews
The interview designed both in Vietnamese and in English was carried out for two
purposes: To acquire in-depth understandings about teachers’ perceptions of holistic and
analytic rubrics in assessing speaking and to find out how teachers’ perceptions are
reflected in their reported practice of using holistic and analytic rubrics in assessing the
student’s speaking performance.
To gain deeper insights into teachers’ perceptions and their reflective practice of
using holistic and analytic rubrics in assessing speaking, the interviews were conducted
with nine participants. They were randomly selected for the interview. Nine participants
were invited for the interview. Then, in order to obtain the expected information from the
participants, semi-structured interviews were employed. Semi-structured interview is the
combination of structured and unstructured interview which most of content and
procedure of interviewing have been planned to help the interviewer focus on the
intention of asking questions in the interviews while unstructured interview requires the
interviewer to be flexible in interviewees’ specific situation (Cohen, Manion and
Morrison, 2000). Therefore, basically each teacher participating in the interview of the
present study was asked the same questions through the same sequence. Thanks to the
gained information from semi-structured interviews, it was possible to compare and
contrast the information, which made it possible to interpret and draw the conclusion
from the findings.
Regarding to the questions for semi-structured interviews, a four-open-ended
question interview was developed by the researcher in accordance with the supervisor’s
suggestions focused on 2 main points: the reason why teachers had a preference for a
certain type of rubric and how teachers’ perceptions are reflected in their reported
practice of using holistic and analytic rubrics in assessing speaking skill.
Based on the questions from the questionnaire, 2 major questions were designed in
order to explore the reason why teachers have preference of certain types of rubrics. (see
Appendix C for all major interview questions).
Due to the fact that the scoring process appears in the raters’ mind, it is impossible
to see how they practice it (Fulcher & Davidson, 2007). The most approachable way to
see their practice is conducting the interviews. University EFL teachers’ reflective
practice was revealed by means of thought-provoking questions from the researcher.
Thanks to the interview questions, the teachers were asked to recall the procedure in
which they scored the students’ speaking performance. Through this reflective practice,
the reasons why teachers had such practice were also revealed, which shed lights on their
perceptions.
Consequently, another 2 interview questions were employed to find out how
teachers’s perceptions are reflected in their reported practice of using holistic and analytic
rubrics in assessing speaking skill. The interview questions were designed in order to
help teachers retell the scoring procedure in which they assessed the students’ speaking
performance. Each question aimed to discover the procedure in which teachers practiced
holistic rubrics and analytic rubrics. In this procedure, the teachers reported every
detailed step of using a certain type of rubric. Finally, all of the participants’ responses
were transcripted and analysed to find out their perceptions through the reflective practice
(see Appendix C for all major interview questions).
All questions were asked in Vietnamese to make the respondents feel convenient
and self-confident in providing as much information as they could. All of the interview
sessions were recorded by means of MP3 recordings.
3.5. Data collection procedure
3.5.1. Piloting the questionnaire
After designing the questionnaire with the assistance of supervisor, the researcher
piloted the questionnaire with 29 teachers who shared similar teaching context with the
target participants. The researcher asked for their permission to take part in the survey.
Then, the questionnaire was sent to them in two ways: one via Email by using Google
Form and the other via handouts. After the reseacher had got the responses from the
participants, the data were run on SPSS version 20 (Statistical Package for the Social
Sciences) to test the reliability of the instrument. The reliability coefficient of the piloted
questionnaire was relatively high (α= .791). Then, nine teachers randomly selected were
asked to attend the interview session.
During the piloting phase, the participants made some suggestions on the
questionnaire. The instruction for each part of the questionnaire should be made clearer to
the interviewees. For example, the instruction for cluster 1 of the questionnaire after the
suggestions was “Please choose the most approriate answer and provide relevent
information about you for each question below”. Moreover, the participants suggested
that researcher shouldn’t mix the major and level of education of the participants into the
same question. Therefore, it was separated into 2 different questions: One for the
participants’ major, the other for the participants’ level. What is more, with the second
cluster of the questionnaire, the researcher should put them into more detailed clusters for
further convenience of future analysis. Based on the suggestions from the participants,
the second cluster of the questionnaire was separated into 3 sub-clusters: One sub-cluster
for the teachers’ perceptions of the significance of using rubrics in assessing speaking,
another for the teachers’ perceptions of holistic rubrics in assessing speaking, the other
for the teachers’ perceptions of analytic rubrics in assessing speaking.
In addition, four questions were added to the questionnaire in order to see
teachers’ perceptions of the significance of using rubrics in assessing speaking. The main
content of these added questions is about the significance of using rubrics in language
assessment, the effects of rubrics on communicative development.
3.5.2. Administering the questionnaire
The questionnaire was administered to forty participants in two weeks after the
piloting phase. Twenty nine questionnaires were sent via email by using Google Form
while the rest was delivered directly to participants. All of the participants completed the
questionnaire and sent it back to the researcher.
3.5.3. Piloting the interview
After analysing the questionnaire, the researcher selected 4 teachers randomly. All
of the interview sessions were recorded and transcribed.
4 selected teachers were invited to take part in the interview. The aim of the first
two interview questions was to find deeper insights into using holistic and analytic
rubrics in assessing speaking. The interviewees explained the reason why they prefered
certain types of rubrics. The rest of the interview questions was about the scoring process
via which the teachers scored the students’ speaking performance. Teachers’ perceptions
were reflected through this practice.
3.5.4. Administering the interview
The interview was progressed with 9 participants and individual meetings were set up
with their agreement. Each interview for each part lasted about 15 minutes. After the
purposes of interview were carefully explained to the participants, all teachers allowed
the researcher to record their responses by audio recordings for later transcribing.
3.6. Data analysis
3.6.1. Analysis of the questionnaire
Quantitative data were subjected to SPSS program version 20. The participants’
responses for the questionnaires were coded into numbers. Firstly, the Scale Test was run
to check the reliability of the questionnaire. Next, descriptive Statistic Test was then
calculated to observe the mean scores, the maximum, the minimun and standard
deviations of the whole group. Finally, the One Sample T- Test was conducted to
evaluate whether mean scores of the teachers’ perceptions of holistic and analytic rubrics
in assessing speaking were statistically different from a test value.
3.6.2. Analysis of the interview data
Participants’ responses from the interviews were transcribed, read and analysed in
order to gain the in-depth understandings of the topic under this investigation. Besides
that, the interviews which showed teachers’ reflective practice of using holistic and
analytic in assessing speaking were analysed and presented in the paper. The final stage
of this process was interpreting the collected data.
3.7. Summary
This chapter has presented the methodology of conducting the study. Forty
university EFL teachers ranging from the age of 22 to 43 were invited to take part in the
study. The research instruments, the questionnaire and interview were employed to figure
out teachers’ perceptions of holistic and analytic rubrics in assessing speaking. The
questionnaire was delivered to the participants via Email by using Google Form and via
the handouts within 2 weeks. After getting the reponses from the participants, the
researcher piloted the questionnaire to check the reliability of the questionnaire. Next, 40
responses from the participants were analysed. 9 selected teachers were invited to
participate in the interview sessions. All of the data were subjected to SPSS program
version 20.
In the next chapter, the results of the study will be presented.