LECTURE: TOOLS AND CRITERIA DEVELOPMENT FOR FOREIGN LANGUAGE

ASSESSMENT
Learning outcomes:
- Explain the role of criteria, descriptors, and scales in criteria-referenced (criterion-based) assessment.
- Use the Common European Framework of Reference for Languages (CEFR) to design performance
descriptors and rating scales.
- Differentiate between analytic and holistic rubrics and select the appropriate type for different
assessment purposes.
- Create effective rubrics for speaking and writing tasks, including performance level descriptors and
weighting.
- Apply basic quality checks: clarity, observability, alignment to constructs, and rater consistency.

Plan:
1. Why criteria matter: fairness, transparency, and learning impact
2. From constructs to criteria: what exactly are we assessing?
3. Descriptors and scales: definitions, writing principles, common pitfalls
4. CEFR as a baseline: proficiency levels, descriptor banks, and alignment logic
5. Rubrics as the main tool of criterion-based assessment
6. Types of rubrics: analytic vs holistic (with examples)
7. Moderation, standardization, and reliability: making rubrics work in real classrooms
8. Summary + reflective questions
Key terms
Construct: The ability or skill the assessment intends to measure (e.g., interactional speaking ability).
Criterion: A feature of performance that is judged (e.g., task achievement, grammatical control,
coherence).
Descriptor: A short statement describing observable performance (e.g., ‘Can narrate a past event with
some detail’).
Scale: An ordered set of levels with descriptors showing progression (e.g., A2 → B1 → B2).
Rubric: A scoring tool that lists criteria and performance levels (analytic or holistic).

1. Criteria-referenced assessment in foreign language education

In foreign language education, assessment is not only a measurement activity; it shapes teaching
priorities, learner motivation, and the perceived meaning of success. Criterion-referenced (also called
criterion-based) assessment evaluates learner performance against clearly defined criteria rather than
ranking learners against each other (norm-referenced assessment). The shift toward competence-based
curricula has reinforced the need for transparent criteria and performance standards.
Language assessment scholarship emphasizes that assessment quality depends on how well an
assessment instrument captures the intended construct (validity), how consistently results can be
reproduced (reliability), and how the assessment affects teaching and learning (washback). Classical and
modern references include Bachman & Palmer’s work on language testing and usefulness, Alderson’s
contributions to evaluation and washback, and Brown’s practical frameworks for classroom assessment.
In the methodology tradition, Russian and post-Soviet scholarship has long highlighted the
pedagogical function of assessment in language teaching. Influential voices such as E. I. Passov and N. D.
Galskova stress that assessment should support communicative development, not merely check discrete
knowledge. Within Kazakhstan’s context, the modernization of foreign language education is strongly
associated with the scientific and methodological school led by S. S. Kunanbayeva, which foregrounds
competence-based outcomes and system-level alignment.
If teachers can articulate criteria, learners can understand what counts as quality performance and
how to improve. Without explicit criteria, scores risk becoming opaque and potentially unfair.

Think of a recent speaking or writing task you assessed. What were your criteria? Were they
written down? Could learners predict your score?
 2. Descriptors and rating scales: how to design them
Descriptors translate an abstract construct into observable language. A well-written descriptor is
learner-centered, positive (what the learner can do), concise, and anchored in observable behavior. Rating
scales group descriptors into ordered levels that represent progression. In language assessment, scales
may be general (broad proficiency) or task-specific (performance on a particular kind of task).
Brian North’s research on descriptor scaling (including work supporting CEFR descriptor
development) highlights that descriptors need to be calibrated and interpreted consistently by raters. In
classroom settings, teachers often build local scales for specific tasks, but alignment to external standards
(such as CEFR) helps maintain comparability.
2.1 Principles for writing effective descriptors
1. Use ‘can do’ wording where appropriate (especially for proficiency-based descriptors).
2. Make the descriptor observable: avoid vague adjectives like ‘good’ or ‘nice’ unless
defined.
3. Keep one main idea per descriptor; avoid multiple skills in one sentence.
4. Control the scope: specify conditions (e.g., ‘in familiar contexts’, ‘with some support’).
5. Use consistent terminology across levels (e.g., ‘some errors’ → ‘occasional errors’ → ‘rare
errors’).
6. Avoid comparing to other learners; describe performance, not rank.
7. Include boundaries: what the learner can do and typical limitations at that level.
2.2 Common problems and how to fix them
Problem: descriptors are too general
 Fix: add observable indicators (e.g., ‘uses past forms with some control in short
narratives’).
Problem: multiple criteria mixed together
 Fix: split into separate descriptors (e.g., fluency vs accuracy vs interaction).
Problem: negative phrasing (‘cannot…’)
 Fix: focus on achieved capability and state limitations neutrally (‘can… but may…’).
Problem: levels are not clearly different
 Fix: increase contrast using frequency, complexity, range, and autonomy as progress
markers.
Problem: task descriptors are copied from exams without adapting
 Fix: tailor descriptors to the local task, curriculum, and learner profile.
Mini-activity: rewriting descriptors
Rewrite the vague descriptor below into an observable one:
• ‘The learner speaks well and uses good vocabulary.’
Possible rewrite:
• ‘Uses a sufficient range of topic-related vocabulary to express main ideas and some detail; searches for
words at times but meaning is generally clear.’
 3. Using CEFR as a basis for developing criteria
The Common European Framework of Reference for Languages (CEFR), developed by the
Council of Europe, provides a widely used proficiency framework for describing language ability across
levels (A1 to C2). CEFR’s influence is especially strong in curriculum design, textbook development, and
assessment reform. In many contexts, including Kazakhstan, CEFR-alignment supports comparability
across institutions and contributes to clearer learning targets.
The CEFR is not a test; it is a descriptive framework. Its value for assessment lies in offering: (1)
common reference levels, (2) illustrative ‘can do’ descriptors, (3) a functional-communicative orientation,
and (4) guidance for linking assessments to proficiency standards. For teachers, CEFR becomes a starting
point for local criteria, which must be adapted to tasks, learners, and curricula.
3.1 CEFR levels in brief (what changes across levels?)
Progression across CEFR levels can be expressed using recurring dimensions:
• Range (lexical and grammatical repertoire)
• Accuracy (control of forms and frequency of errors)
• Fluency (speed, pausing, and smoothness)
• Interaction (turn-taking, repair, responding)
• Coherence (organization, linking, discourse control)
• Task complexity (from simple routine tasks to complex, abstract topics)
Table 1. Example progression for spoken interaction (illustrative)
Level Typical Strengths Typical
interaction limitations
A2 Short Can Needs
exchanges on ask/answer simple repetition; limited
familiar topics questions; uses basic range; frequent
strategies pauses
B1 Sustained Can narrate, May struggle
conversation on explain, and react; with precision;
everyday and some can keep occasional
abstract topics conversation going breakdowns
B2 Effective Can argue a May lack
interaction in a wide viewpoint; uses idiomaticity;
range of situations repair strategies; occasional
good discourse inaccuracies in
control complex structures
Note: Table 1 is an illustrative simplification. For operational use, consult CEFR descriptors
and/or validated descriptor banks.
3.2 Linking classroom criteria to CEFR: a practical workflow
8. Step 1: Define the assessment purpose (diagnostic, formative, summative) and decision to
be made.
9. Step 2: Define the construct and task: What language ability does the task elicit (e.g.,
argumentation, narration, interaction)?
10. Step 3: Select relevant CEFR descriptor categories (e.g., ‘Overall spoken production’,
‘Coherence and cohesion’, ‘Range’, ‘Accuracy’).
11. Step 4: Adapt descriptors to the task and context (topic, time limits, support, genre).
12. Step 5: Decide the number of levels (e.g., 4 levels for classroom rubric, mapped roughly to
A2/B1/B2).
13. Step 6: Draft descriptors for each criterion across levels; ensure observable differences.
14. Step 7: Pilot with student samples; revise for clarity and discriminating power.
 15. Step 8: Conduct rater calibration (teacher team) and moderation (double marking,
discussion).
Kazakhstani context note
In Kazakhstan, CEFR-aligned reforms and level-based teaching have been discussed in research
and policy-oriented reports, including analyses of assessment practices and teacher development needs.
Methodological scholarship associated with S. S. Kunanbayeva’s school emphasizes systemic
modernization and competence-based outcomes, which aligns naturally with the CEFR logic of
describing performance and progression.
 4. Rubrics as the core tool of criterion-based assessment
A rubric is a scoring guide that communicates what quality looks like. In foreign language
assessment, rubrics are used for productive skills (speaking and writing) and increasingly for interactive
tasks (discussion, presentation, role-play) as well as for integrated tasks (listening-then-speaking, reading-
then-writing).
Educational assessment authors such as Susan Brookhart and Heidi Andrade emphasize that
rubrics can serve both assessment and instruction: when learners see criteria and quality levels, they can
plan, monitor, and revise their work. From the perspective of classroom language assessment (e.g.,
Brown; Fulcher; Harding), rubrics also support reliability by guiding raters toward consistent judgments.
4.1 Functions of rubrics
 Transparency: learners understand how performance is judged.
 Consistency: multiple teachers (or the same teacher over time) apply similar standards.
 Feedback: criteria-level comments reveal strengths and next steps.
 Self- and peer-assessment: learners use the rubric to evaluate and improve work.
 Alignment: rubrics connect tasks to curriculum outcomes and CEFR-oriented descriptors.
4.2 Types of rubrics: analytic vs holistic
Analytic rubrics separate performance into multiple criteria (e.g., task achievement, coherence,
range, accuracy, pronunciation). Each criterion receives its own score. Holistic rubrics provide a single
overall score based on an overall description of performance.
Analytic rubrics: when they are best
 When detailed feedback is needed (formative assessment).
 When different criteria matter differently and may progress at different rates.
 When teacher teams need shared standards (moderation).
 When teaching aims to target specific subskills (e.g., cohesion, interaction).
Holistic rubrics: when they are best
 When quick scoring is required (large groups, limited time).
 When the construct is best captured as an integrated performance (overall communicative
effectiveness).
 When decision-making requires a single score (placement, broad grading).
4.3 Strengths and limitations
Analytic rubrics increase diagnostic value but may reduce scoring speed and sometimes create an
illusion of precision if descriptors are not clear. Holistic rubrics are efficient but provide less actionable
feedback. A practical compromise is to use an analytic rubric for teaching cycles and a holistic scale for
end-of-unit decisions, while keeping both aligned.
 5. Technology for designing effective rubrics (step-by-step)
This section provides a practical procedure (a ‘technology’) for building rubrics that are
instructionally useful, CEFR-aligned, and feasible for real classrooms. The steps below integrate ideas
from language testing (Bachman & Palmer; Fulcher), educational measurement (Popham; Brookhart), and
methodology traditions in foreign language education.
5.1 Step 1: Specify the task and the construct
Start from the task: what exactly will learners do? Then specify the construct: what ability does
successful performance demonstrate? For example, a debate task may require (a) interactional
competence, (b) argumentation, (c) discourse management, and (d) language control. If the construct is
unclear, the rubric becomes inconsistent and unfair.
Checklist:
 Task type and genre (presentation, interview, email, narrative, report)
 Conditions (time, preparation, prompts, support, resources)
 Target level(s) (e.g., B1/B2 transition)
 Communicative purpose (inform, persuade, narrate, negotiate)
5.2 Step 2: Select criteria (usually 3–6)
Choose a manageable number of criteria. Too many criteria overload raters and confuse learners.
For productive skills, common criteria families include: Task achievement/content,
Organization/coherence, Lexical range/appropriacy, Grammatical control, Fluency and pronunciation
(speaking), Mechanics (writing), and Interaction (for dialogic tasks).
5.3 Step 3: Define performance levels
Decide how many performance levels you will use. Four levels are often practical for classroom
rubrics (e.g., Beginning / Developing / Proficient / Strong). If you need CEFR linkage, label levels with
approximate alignment (e.g., A2, B1, B2) or with local labels mapped to these. Ensure that the rubric is
appropriate for your learners; a six-level rubric may be too detailed for everyday classroom use.
5.4 Step 4: Write descriptors for each criterion at each level
Write descriptors so that the difference between levels is meaningful and observable. Use progress
markers such as: range (basic → wider), accuracy (frequent → occasional errors), autonomy (needs
support → independent), and discourse control (simple linking → varied cohesive devices). Avoid relying
only on ‘more/less’ without specifying what exactly changes.
5.5 Step 5: Decide weighting and scoring rules
Not all criteria are equally important for every task. For example, in a persuasive speech,
argumentation and organization may be weighted more than grammatical accuracy. Weighting should
reflect the learning goals and the construct. Keep rules simple: learners should understand how the final
score is produced.
5.6 Step 6: Pilot, calibrate, and revise
A rubric draft is not final until it is tested. Pilot the rubric with sample student performances. Ask:
Do the descriptors separate weak, medium, and strong performances? Do different raters give similar
scores? Use moderation meetings to discuss borderline cases and refine wording. Calibration practices are
common in high-stakes exams and are also valuable in schools.
5.7 Quality checklist for rubrics
 Alignment: criteria reflect the task and learning outcomes (and CEFR categories if used).
 Clarity: descriptors are understandable to teachers and (ideally) to learners.
 Observability: descriptors reference evidence in performance, not vague impressions.
 Discrimination: levels show real progression and are not repetitive.
 Feasibility: the rubric can be applied within the time available.
 Consistency: similar wording patterns across criteria/levels; no hidden criteria.
 Fairness: language is unbiased; accommodations are considered where needed.
 6. Sample rubrics (ready-to-use templates)
The samples below can be adapted to your syllabus and learner level. They illustrate CEFR-
inspired wording without claiming formal standard-setting. For high-stakes use, you would need formal
linking procedures.
6.1 Sample analytic rubric for a B1-level speaking task (monologue: ‘Describe an
experience’)
Task: 2–3 minute talk about a memorable event. Follow-up questions: 1–2 minutes.
Criterion Level 4 Level 3 Level 2 Level 1
(Strong) (Proficient) (Developing) (Beginning)
Task Address Covers Partly Does not
achievement & es the prompt the prompt; addresses the address the
relevance fully; adds includes some prompt; limited prompt; very
relevant detail; detail; minor detail; frequent limited content;
stays on topic; digressions; digressions; cannot handle
responds responds follow-up follow-up
confidently to adequately to answers short or without major
follow-up. follow-up. unclear. support.
Organiza Clear Generall Limited No clear
tion & structure y organized; organization; organization;
coherence (beginning– sequencing ideas may be disconnected
middle–end); mostly clear; hard to follow; sentences;
effective uses some few linking listener must
sequencing; linking devices devices; abrupt infer
uses linking though transitions. relationships.
devices to guide occasionally
the listener. repetitive.
Lexical Uses a Adequat Limited Very
range & good range of e range for the range; repeated limited
appropriacy topic topic; words; frequent vocabulary;
vocabulary; occasional searching; frequent
chooses words inappropriate paraphrasing is breakdown due
appropriately; choices; can weak. to missing
paraphrases paraphrase words.
effectively. simple gaps.
Gramma Good Control Frequent Persisten
tical control control of of basic errors even in t errors;
common structures; some basic forms; meaning often
structures; errors in more meaning unclear; relies
errors are complex forms; sometimes on isolated
occasional and meaning unclear; limited words or
rarely impede generally clear. complex memorized
meaning. structures. phrases.
Fluency, Mostly Some Frequent Very
pronunciation & smooth hesitation; pace pauses and self- hesitant; long
interaction delivery; natural varies; correction; pace pauses;
pace; pronunciation slow; pronunciation
pronunciation generally pronunciation often prevents
and intonation intelligible; sometimes understanding;
 generally clear; occasional hinders cannot maintain
uses repair repair. understanding. interaction.
strategies.
Scoring suggestion: 5 criteria × 4 points = 20 points total. Optional weighting: Task achievement
×2 for presentations.
Feedback note: Provide 1–2 ‘next steps’ aligned to the lowest-scored criterion.
6.2 Sample holistic rubric for speaking (overall communicative effectiveness)
Use when quick scoring is needed (e.g., short interviews or role-plays).
Band Overall description
4 Communicates effectively; message
clear and well supported; minor language
issues do not affect understanding.
3 Communicates main message;
generally clear though with noticeable
hesitation or errors; occasional
misunderstanding.
2 Communicates limited message;
frequent hesitation and errors; understanding
requires effort; limited support/detail.
1 Cannot communicate message
beyond isolated words/phrases; frequent
breakdown; understanding is very difficult.

6.3 Sample analytic rubric for B1–B2 writing (opinion paragraph, 180–220 words)
Task: Write an opinion paragraph on a school/community issue. Include reasons and one example.
Criterion Level 4 Level 3 Level 2 Level 1
(Strong) (Proficient) (Developing) (Beginning)
Task Clear Position Position No clear
achievement position; well- clear; reasons somewhat position;
developed adequate; unclear or minimal
reasons; example present reasons content;
relevant but may be underdeveloped requirements
example; fully brief; meets ; example weak not met.
meets length most or missing;
and genre. requirements. requirements
partly met.
Coheren Logical Generall Organiza Disorgan
ce & cohesion progression; y logical; some tion uneven; ized; very
effective paragraphing; limited linking; limited
paragraphing; a uses linking transitions cohesion;
variety of devices though sometimes difficult to
linking devices sometimes unclear. follow.
used repetitive.
appropriately.
Lexical Good Adequat Limited Very
resource range; e range; register range; frequent limited
 appropriate mostly repetition; vocabulary;
register; appropriate; several frequent misuse;
occasional some word- inappropriate meaning often
minor choice issues. choices. unclear.
collocation
issues.
Gramma Uses a Mostly Frequent Persisten
tical range & range of accurate in errors; limited t errors;
accuracy structures; common complex meaning often
errors are structures; structures; unclear.
occasional and errors in meaning
do not impede complex sometimes
meaning. structures; unclear.
meaning
generally clear.
Mechani Minor Some Frequent Very
cs (spelling & errors only; errors; errors; frequent errors;
punctuation) does not affect readability readability hard to read.
readability. mostly affected.
maintained.
Optional CEFR mapping: Level 2 ≈ B1 threshold; Level 3 ≈ solid B1; Level 4 ≈ B2 tendencies
(task dependent).
 7. Implementing rubrics in classroom practice
A rubric is only effective if it is integrated into instruction. Research on formative assessment
(e.g., Black & Wiliam) shows that criteria and feedback improve learning when learners understand
targets and can act on feedback. In foreign language classrooms, rubric use supports self-regulated
learning and peer feedback, particularly in writing and speaking cycles.
7.1 Before the task: making criteria visible
 Share the rubric in advance and explain each criterion with short examples.
 Show 1–2 anonymized samples (or teacher-made samples) and score them together.
 Ask learners to paraphrase criteria in their own words.
 Convert rubric criteria into a planning checklist (especially for writing).
7.2 During the task: support and observation
 For speaking, use simplified ‘live’ rubrics with fewer criteria for real-time scoring.
 Record performances (if allowed) for moderation and feedback.
 Use observation notes aligned to criteria (evidence-based comments).
7.3 After the task: feedback, self- and peer-assessment
 Provide criterion-level feedback (one strength, one next step per criterion).
 Ask learners to self-assess and justify with evidence (quote their own sentence; point to an
interaction moment).
 Use peer-assessment with training and ‘comment stems’ (e.g., ‘Your organization is clear
because…’).
 Encourage revision and resubmission (especially for writing).
7.4 Reliability and moderation (teacher team practices)
Even classroom assessment benefits from basic reliability practices. Fulcher and Harding discuss
the importance of rater training and standardization in performance assessment. Practical steps include:
 Agree on benchmark samples for each level (anchor performances).
 Conduct short calibration sessions: score a sample independently, then compare and
discuss.
 Clarify ambiguous descriptors and add examples of evidence.
 Use double marking for a small sample to check consistency.
 8. Workshop tasks (optional, 30–45 minutes)
Choose one task below for group work. Output: a draft rubric + short justification of criteria and
level wording.
 Task A (Speaking): Pair discussion (3–4 minutes) on ‘Should school uniforms be
mandatory?’ Build a rubric with interaction as a separate criterion.
 Task B (Writing): Email of complaint (150–180 words). Build a rubric emphasizing
register and pragmatic appropriacy.
 Task C (Integrated): Listen to a short talk and summarize it orally. Build criteria that
capture comprehension evidence and speaking quality.
Reflection questions
16. Which criteria best reflect communicative competence for your context? Why?
17. What is the risk of overemphasizing grammatical accuracy in communicative tasks?
18. How can CEFR descriptors be adapted without ‘copy-paste’ misuse?
19. What moderation practice could your department realistically implement next month?

Summary
Criteria and rubrics make assessment transparent and instructionally useful when they are aligned
to a clear construct, expressed through observable descriptors, and applied consistently. CEFR offers a
shared language for describing proficiency and can guide local rubric design, but teachers must adapt
descriptors to real tasks and learners. Analytic rubrics support diagnostic feedback; holistic rubrics
support efficiency. In all cases, piloting and moderation are essential for fairness and consistent
interpretation.
 References (selected, APA style)
 Alderson, J. C. (2005). Diagnosing foreign language proficiency: The interface between
learning and assessment. Continuum.
 Andrade, H. (2000). Using rubrics to promote thinking and learning. Educational
Leadership, 57(5), 13–18.
 Bachman, L. F., & Palmer, A. S. (1996). Language testing in practice. Oxford University
Press.
 Black, P., & Wiliam, D. (1998). Assessment and classroom learning. Assessment in
Education, 5(1), 7–74.
 Brookhart, S. M. (2013). How to create and use rubrics for formative assessment and
grading. ASCD.
 Brown, H. D. (2004). Language assessment: Principles and classroom practices. Pearson
Education.
 Council of Europe. (2001). Common European Framework of Reference for Languages:
Learning, teaching, assessment. Cambridge University Press.
 Council of Europe. (2020). CEFR Companion Volume with new descriptors. Council of
Europe Publishing.
 Fulcher, G. (2010). Practical language testing. Hodder Education.
 Harding, L. (2015). Language assessment. Oxford University Press.
 Kunanbayeva, S. S. (Kazakhstan). Works on modernization and methodological
foundations of foreign language education (scientific and methodological school).
 Luoma, S. (2004). Assessing speaking. Cambridge University Press.
 McNamara, T. (2000). Language testing. Oxford University Press.
 Moskal, B. M. (2000). Scoring rubrics: What, when and how? Practical Assessment,
Research & Evaluation, 7(3).
 North, B. (2000). The development of a common framework scale of language
proficiency. Peter Lang.
 Popham, W. J. (1997). What’s wrong - and what’s right - with rubrics. Educational
Leadership, 55(2), 72–75.
 Weigle, S. C. (2002). Assessing writing. Cambridge University Press.
 Passov, E. I. (Russia). Communicative method and methodological foundations of foreign
language teaching (selected works).
 Galskova, N. D. (Russia). Methodology of foreign language teaching and educational
goals (selected works).
 Solovova, E. N. (Russia). Modern approaches to teaching and assessment in foreign
language education (selected works).