See discussions, stats, and author profiles for this publication at: [Link]

net/publication/305811243

Classification of Writing Patterns Using Keystroke Logs

Chapter · January 2016

DOI: 10.1007/978-3-319-38759-8_23

CITATIONS READS
10 442

4 authors, including:

Mo Zhang Chen Li
Wuhan University Educational Testing Service
15 PUBLICATIONS 110 CITATIONS 8 PUBLICATIONS 35 CITATIONS

SEE PROFILE SEE PROFILE

Paul Douglas Deane

Educational Testing Service
104 PUBLICATIONS 1,516 CITATIONS

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Syntax, Island Constraints, Grammar in Mind and Brain View project

Writing Process Analysis Using Keystroke Logs View project

All content following this page was uploaded by Paul Douglas Deane on 27 May 2018.

The user has requested enhancement of the downloaded file.

Running Head: CLASSIFICATION OF WRITING PATTERNS 1

Classification of Writing Patterns Using Keystroke Logs

Mo Zhang, Jiangang Hao, Chen Li, and Paul Deane

Educational Testing Service, Princeton, NJ, USA

Author Information:

Mo Zhang (Corresponding author), E-18, Educational Testing Service, Princeton, NJ,

08541, MZhang@[Link]

Jiangang Hao, T-02, Educational Testing Service, Princeton, NJ, 08541, JHao@[Link]

Chen Li, T-20, Educational Testing Service, Princeton, NJ, 08541, CLi@[Link]

Paul Deane, R-11, Educational Testing Service, Princeton, NJ, 08541, PDeane@[Link]
CLASSIFICATION OF WRITING PATTERNS 1

Abstract

Keystroke logs are a valuable tool for writing research. Using large samples of student

responses to two prompts targeting different writing purposes, we analyzed the longest 25

inter-word intervals in each keystroke log. The logs were extracted using the ETS

keystroke logging engine. We found two distinct patterns of student writing processes

associated with stronger and weaker writers, and an overall moderate association between

the inter-word interval information and the quality of final product. The results suggest

promise for the use of keystroke log analysis as a tool for describing patterns or styles of

student writing processes.

Keywords: keystroke logs, writing processes, writing pattern, inter-word interval

CLASSIFICATION OF WRITING PATTERNS 2

Classification of Writing Patterns Using Keystroke Logs

Introduction

Keystroke logs (KL) are a valuable tool for writing research (Leijten & van Waes,

2013). A keystroke logging program essentially records the mechanical processes of one’s

writing and the temporal information associated with an action (e.g., insert, delete).

Information recorded and extracted from KLs can provide rich evidence on one’s writing

practice, writing proficiency, linguistic skills, as well as underlying cognitive processes

(e.g., Leijten, Macken, Hoste, van Horenbeeck, & van Waes, 2012). There is an extensive

research literature on writing using KLs. For example, van Waes, Leijten, & van Weijen

(2009) showed that KLs can be used for conducting empirical research on comparing

writers with different ability levels, on studying cognitive processes during writing, and

on examining learning styles. Many other published studies have used KLs for studying

writing strategies (Xu & Ding, 2014), genre effects (Beauvais, Olive, & Passerault, 2011),

keyboarding skills (Grabowski, 2008), and writing skills for native speakers vs. non-

native speakers (Miller, 2000; Roca de Larios, Manchon, Murphy, & Marin, 2008). Lastly,

studies have used KLs for writing research in various contexts, such as spontaneous

communication (Chukharev-Hudilanein, 2014), professional writing (Leijten, van Waes,

Schriver, & Hayes, 2014), educational assessment (Deane, 2014), language translation

(Dragsted & Carl, 2013), and cognitive modeling (Almond, Deane, Quinlan, & Wagner,

2012). This literature indicates that stronger writers tend to write more fluently and

persistently, spending more time on task, writing in longer "bursts" of uninterrupted text

production, and pausing primarily at locations such as sentence and clause boundaries that

indicate an emphasis on sentence and discourse-level planning. These results have been
CLASSIFICATION OF WRITING PATTERNS 3

confirmed in an assessment context in our own studies (Deane, 2014; Deane & Zhang,

2015; Zhang & Deane, 2015).

From an assessment perspective, evaluating writing processes can be valuable for

at least two reasons. One is to give teachers and students (diagnostic) feedback to improve

writing practice. As a simple example, a limited time on task might suggest a lack of

persistence. Similarly, an absence of editing behavior, in combination with disjoint text

(e.g., containing substantial amount of grammatical and mechanical errors interfering the

flow and meaning of the text), might prompt attention to teaching revision strategies

(Zhang & Deane, 2015). A second potential value is to characterize differences among

subpopulations beyond the quality of the final product, such as mechanical characteristics

of text production (e.g., typing speed, extent of editing) and writing patterns and styles

(e.g., lack of planning). It is this latter idea that is the focus of this study.

We used a KL engine developed at Educational Testing Service, which can produce

a large number of features such as the length of a within-word pause (Deane, 2014). In this

study, we focused on the inter-word interval (IWI) feature, or pauses between two adjacent

words, extracted from the KL engine in order to address two research questions: (1) Can

we distinguish writing patterns using this information? (2) How does this information relate

to the human ratings of text-production skills (e.g., grammar, organization)?

Previous research suggests that IWIs tend to be associated with such cognitive

activities for word and sentence planning and deliberation, especially when lengthy and at

sentence, clause, or discourse boundaries (Baaijen, Galbraith, & de Glopper, 2012;

Banerjee, Feng, Kang, & Choi, 2014; Chenoweth & Hayes, 2001; Chukharev-Hudilainen,

2014; Gould, 1980; Xu & Ding, 2014), though IWIs can also indicate difficulties in
CLASSIFICATION OF WRITING PATTERNS 4

keyboarding and lexical access such as in spelling. Shorter IWIs, in contrast, are more

likely to reflect basic keyboarding fluency (e.g., Alves, Castro, & de Sousa, 2007).

Figure 1 provides a dendrogram visualization of the IWI duration using one essay

as an example. The original essay reads as below, with bolded emphasis added by the

authors.

“I think the community should build a swimming pool because i believe

you can I maintain a healthier lifestyle and have fun while doing it It has

more pottential to benefit more people than than a foreign exchange. The

whole community can benefit the pool but only a select few would benefit

from the exchange program.”

At the bottom of the figure are the words produced. The height of the horizontal

lines that connect two words (i.e., the ones with the shortest distance on the time dimension

from two clusters) represents the IWI duration in seconds. We note that, in this example,

the three longest between-word pauses all happen at the phrasal and sentence boundaries,

which naturally separate the text production into four bursts. Specifically, the pause

between “exchange” and “The” is the longest with more than 70 seconds, with “exchange”

on the end of the previous sentence and “The” being the first word of the following

sentence. It is reasonable to assume that the student spent a fair amount of time planning

the second sentence or reading and evaluating the source materials.

CLASSIFICATION OF WRITING PATTERNS 5

IWI (in Sec.)

Figure 1. Visualizing the IWI Duration

The pauses between the words “I” and “maintain” (where “I” is likely to be a typo), and

the words “it” and “It,” both occur at reasonable junctures between long chunks, each pause

lasting for about 10 seconds. A sensible explanation for these relatively long pauses is that

the student was deliberating on the choice of words and sentence structure. All the other

IWIs in this example were less than 2 or 3 seconds.

CLASSIFICATION OF WRITING PATTERNS 6

An Approach to Comparing Keystroke logs

Even though the IWI can provide temporal evidence about the essay composition

process, we need to quantify and summarize this information in a way that can be used to

make meaningful distinctions among groups of students. A practical challenge is how to

align the IWIs from different keystroke logs so that we can compare overall writing

patterns despite significant differences in the time individual students spend on writing and

the number of words they ultimately produce.

In this study, we explore one novel method of comparison, which attempts to

preserve information about the longer pauses (likely to reflect strategic processing) and to

interpret them in terms of their temporal position in the student's writing process. A

software implementation of this method can be found in Hao, Smith, Mislevy, von Davier,

& Bauer (in press). Our attempt in this study focused on the longer between-word pauses

(IWIs). First, for each log, we rank the IWIs from the longest to the shortest. Then, we

choose the length (or duration) of IWI and its normalized median time point along the time

axis as two indicative variables to represent each keystroke log. By placing these indicative

variables one after another based on the rank ordering of the IWI duration, we form a vector

of IWI for each keystroke log. Subsequently, for all vectors (from different logs) to have

the same length, we need to truncate each vector by introducing a cut-off in the rank

ordering. In our implementation, we choose 25 longest IWIs as the cut-off rank, for which

rationale is to be explained shortly.

Through this aligning procedure, to this point, each keystroke log is represented by

an IWI vector with 50 elements that capture the duration and temporal location (i.e., median

time-points.) of the 25 longest IWIs We can then compare the similarity or difference
CLASSIFICATION OF WRITING PATTERNS 7

among logs using the IWI vectors. Finally, it is worth mentioning that, to align keystroke

logs of different length in time, we normalized the median time-points of the IWIs by the

total writing time to the range between zero and one.

The decision to restrict attention to the longest 25 IWIs is motivated by the overall

distribution of IWIs, which follows a heavily skewed distribution in which almost all IWIs

are very short (less than half of a second). The psycholinguistic literature referenced above

suggests rather different interpretations for the main part of the distribution (which reflects

fluent text processing), and the tail consisting of very long pauses (which are more likely

to reflect strategic processes such as discourse-level and sentence-level planning). Also,

the cut-off rank of 25 is the largest number that essays can reach in our dataset before cases

need to be excluded.

Figure 2 gives two examples of different students’ writing processes, each

represented by the longest 25 IWIs and their median time-points.

IWI (in sec)
IWI (in sec)

Median Time-Point [Normalized]

CLASSIFICATION OF WRITING PATTERNS 8

Figure 2. Visualizing the Top 25 IWI Durations and Median Time-Points for Two Essays

Written by Different Students

The vertical axis indicates the length of IWIs in seconds, and horizontal axis gives

the corresponding median time-point of an IWI. The median time-point is computed as the

mid-point of the absolute time between two adjacent words. For example, if the last

character of the first word is typed in minute 1 second 30 and the first character in the

following word is typed in minute 1 second 40, the median time-point of the IWI is minute

1 second 35. Also provided in the graphs are the total number of words in the final product

and the human score on a rubric of basic writing skills for each essay.

As can be seen, the two individual KL examples in Figure 2 are drastically different.

The top panel shows that this student has made few relatively long inter-word pauses, with

four instances standing out as particularly extensive (i.e., greater than 20 seconds). The

lower panel shows a very different pattern of IWI duration and median time-point derived

from a different essay. Compared to the example above, the longest 25 IWIs in the lower

panel are somewhat more evenly distributed throughout the composition and the length of

the durations seems to vary less across the 25 IWIs. By normalizing the median time-point

by the total writing time, we removed the confounding factor of time on task (which also

relates to the essay length). Figure 2 suggests that the IWI information provides a different

type of evidence about writing proficiency from the essay length and human scores on

essay quality.
CLASSIFICATION OF WRITING PATTERNS 9

Method

Instrument

We used two summative ELA (English Language Arts) writing assessments

developed by the Cognitively Based Assessment of, for, and as Learning (CBAL™)

research initiative at Educational Testing Service (ETS®) for this investigation (Bennett,

2010). The CBAL scenario-based assessments are designed to have a progression from

three sections of lead-in tasks to a culminating essay task. Deane et al. (2015) describe the

theoretical foundation for the design of this test structure in detail.

In this study, we used one assessment from each of two writing purposes: Policy

Recommendation and Argumentation. The targeted level was grade 7 for policy

recommendation and grade 8 for argumentation. For our investigation, we focused on the

essay task only. Specifically, the policy recommendation essay asks students to evaluate

two proposals (i.e., on how to spend a generous monetary donation to the school) and write

an essay recommending one policy over the other. The argumentation essay asks students

to take a position using reasons and evidence as to whether students should be rewarded

for getting good grades. Students are provided with a planning tool and can access three

source materials at any time (that were used in the lead-in tasks). Students are encouraged

to utilize the examples and viewpoints given in the source materials (with appropriate

quoting or paraphrasing).

The essays were graded on two scoring rubrics: writing fundamentals and higher-

level skills targeted at the specific writing purpose. The human score scale is integer 0 to 5

for both rubrics, where 0 indicates some unusualness in the essay response (e.g., empty,

off topic). Responses were graded by two independent teacher raters who were trained by
CLASSIFICATION OF WRITING PATTERNS 10

ETS assessment specialists, with a possible third rater for adjudication. The first and second

rater agreements were a quadratically weighted kappa of 0.63, exact percent agreement of

51, and one-point adjacent percent agreement of 97 for the policy recommendation essay.1

The comparable values were 0.67, 52, and 97 for the argumentation essay. For our analyses,

we used adjudicated human scores as the criterion variable (to address Research Question

2). That means when the first two human scores were within one point apart, we used the

average of the two human scores. When the first two human scores were discrepant by

more than one point, we used the average of the two closest scores. If all three scores were

two points apart, we used the middle score.

Data Set

The data set was collected as part of a larger study conducted in multiple US states

in 2013. Included in this study were 831 essay responses from the policy recommendation

assessment, and 902 essay responses from the argumentation assessment. For the policy

recommendation assessment, 36% of the participants were male, 35% female, and the

remaining 29% unreported. As for ethnicity, nearly half of the participants (49%) were

Caucasian; 16% were Hispanic; 6% were Asian, African American, Middle Eastern,

mixed, or others; and 29% did not indicate. Finally, most of the participants in this

assessment were 7th graders (52%), 9% were 6th graders, 10% were 8th graders, and the

1 Consider the two human scores (x and y) form a 2-by-2 matrix. The quadratically weighted kappa (QWK) is
computed as κ = 1 –∑xywxyOxy/∑xywxyExy, where wxy is the weighting matrix, Oxy is the observed pair of rating Exy is
expected pair of rating. Further, the weight follows a quadratic function of the differences between the two scores; that
is, the weight wxy equals to 0, 1, 4, 9, and 16 for score differences of 0, 1, 2, 3, and 4, respectively (Fleiss & Cohen,
1973). QWK can range from 0 to 1 with 0 indicating no association between the two ratings and 1 indicating perfect
association between the two ratings. The exact percent agreement is the sum of the percentages of the pair of ratings
where the ratings are exact the same (i.e., on the diagonal in the cross-tab table). The one-point adjacent percent agree
is the sum of percentages of the pair of ratings where the two human scores are within one score point. Both exact and
one-point adjacent percent agreement can range from 0 to 100, with 0 indicating no agreement between the two ratings
and 100 indicating perfect association between the two ratings.
CLASSIFICATION OF WRITING PATTERNS 11

remaining 29% provided no information. For the argumentation assessment, 42% of the

participants were female, 38% male, and 20% did not report on their gender. For ethnicity,

60% were Caucasian, 13% Hispanic, 7% were Asian, African American, Middle Eastern,

mixed, or others, and 20% were unreported. Finally, 38% were in grade 8, 20% grade 7,

28% grade 9, and 14% unreported.

Keystroke logging was part of the data collection by design, from which the

duration and median time-point of the longest 25 IWIs were extracted. We also obtained

several summary process features, including the total number of word inserts and total

effective writing time for supplementary analyses.

Data Analyses

For the first research question, we conducted a hierarchical cluster analyses based

on the cosine distance between the IWI vectors using complete linkage (Johnson, 1967).

The cosine distance measure is used to decide whether the clusters are adequately different

from one another. The cosine distance is mathematically equivalent as (1-r), where r is the

Pearson correlation coefficient (Jones, Oliphant, & Peterson, 2014). The Cohen’s (1968)

rules for evaluating the magnitude of Pearson correlation coefficient r are: no to small

association (0 - 0.30), moderate association (0.31 - 0.50), and large association (0.51 -

1.00). We transformed these criteria suggested by Cohen (1968) (for evaluating the size of

the correlation coefficient) to the cosine distance measure’s scale. Hence, we considered a

cosine distance value of equal to or greater than 0.50 to indicate a strong separation between

clusters, a value in between 0.30 and 0.50 to indicate a moderate separation, and a value of

less than 0.30 to indicate a weak separation.

CLASSIFICATION OF WRITING PATTERNS 12

After the number of clusters was determined, we examined and compared the

clusters on several aspects. One aspect was the IWI duration and median time-point pattern.

For this analysis, we evenly divided the time axis into 10 bins, with each bin accounting

for one tenth of the normalized total writing time.2 Second, we computed the mean of the

logarithm-transformed IWI duration values in each bin separately for each cluster. This

statistical transformation was undertaken in order to make the distribution of the IWI

duration more similar to a normal distribution (Kalbfleisch & Prentice, 2002). Third,

separately for each of the 20 bins (i.e., 10 bins in essay prompt), we conducted a two-

sample independent t-test between the clusters on the mean log IWI values.

A second aspect we used to compare clusters was the density distribution of the

IWI duration and median time-point. We generated heat maps of the IWI duration by

median time-point to visually compare the probability of IWIs of certain lengths occurring

at certain times in the writing process between clusters. To generate the density graphs, we

evenly divided the x axis (normalized median time-point ranging from 0 to 1) into 100

steps, and evenly divided the y axis (IWI duration ranging from 0 to 16 seconds) into 400

steps, which resulted in 100×400 blocks.3 We then calculated the density of each block and

produced a normalized density distribution in the form of a heat map for each cluster.

Finally, we compared the clusters based on the human ratings on text production

skills, total number of words produced, and total writing time. We computed the effect

sizes and conducted two-sample independent t-tests on the means of those measures

between the clusters. Of note is that the total number of words produced is different from

2 On one hand, the bin size needs to be large enough so that there are enough keystrokes in each bin. On the
other hand, it needs to be small enough to show variations across bins. After a number of experiments, we found
that 10 bins are optimal.
3 The observed maximum IWI duration is much greater than 16 seconds; however, the overwhelming majority

of the information falls under the lower part of each density plot.
CLASSIFICATION OF WRITING PATTERNS 13

the traditional essay length. That is, words that were produced but deleted during the

processes (and not appearing in the final product) were counted for this measure. Further,

for total writing time, we used active writing time, which excluded pre-writing (i.e., pause

before typing the first character), in order to be consistent with the timing data used for

IWI analyses. Finally, due to the highly skewed distribution in human response time, we

used logarithm-transformed values for these analyses (Ulrich & Miller, 1993; van der

Linden, 2006).

For the second research question, we examined how the IWI information related to

human scores. We used leave-one-out multiple linear regression of human scores on the

IWI vector (i.e., top 25 IWI durations and median time-points), and computed the

correlation coefficient of observed and predicted human scores. We hypothesized a

moderate positive association between IWI information and human score because the

processes indexed by the location and duration of the IWI should theoretically contribute

to the quality of the final product. In addition to using all samples available for each essay

task, to investigate whether there was a difference in association between IWI information

and human scores between the clusters, we conducted the regression analyses separately

for each cluster.

Results

Results for Research Question 1: Writing Patterns

Based on our criteria, we identified two clusters for both essay tasks, as can be seen

in Figure 3. The cosine distance between the two clusters is 0.63 for the policy

recommendation essay and 0.62 for the argumentation essay. Based on the transformed

Cohen (1968) criteria, this result indicates that the two clusters are weakly correlated (i.e.,
CLASSIFICATION OF WRITING PATTERNS 14

strongly separated). For both essays tasks, there are more essays in cluster 1: for the policy

recommendation essay, n = 636 in cluster 1, n = 196 in cluster 2; for the argumentation

essay, n = 591 in cluster 1, n = 311 in cluster 2.

Policy Recommendation Essay Argumentation Essay

Figure 3. Clustering of Keystroke Logs

Note. Green: Cluster 1. Red: Cluster 2. The height of horizontal lines connecting two

clusters indicates the cosine distance between two clusters.

Next, we examined the qualitative differences between the two clusters. Figures 4

and 5 show the mean logarithm-transformed IWI duration at different median time-points

in the writing process. Based on the t-test results, all pairs of comparisons suggested

statistically significant differences (at a p < 0.05) between the two clusters. Specifically,

for both writing tasks, IWIs in cluster 1 were significantly longer than the ones in cluster 2

consistently throughout the composition. The differences appear to be smaller around the

middle than at the two ends of the writing process. Additionally, it is worth noting that, for

both essays, cluster 1 shows a more even straight-line pattern across median time-points

than cluster 2, which appears to have a discernable drop in the IWI duration at the

beginning and end of the composition process.

CLASSIFICATION OF WRITING PATTERNS 15

2.5

Mean Log(IWI) [Sec]

2

1.5
Cluster 1
Cluster 2
1

0.5
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
Median Time-Point [Normalized]

Figure 4. Mean Log(IWI) Pattern (Policy Recommendation Essay)

2.5
Mean Log(IWI) [Sec]

1.5
Cluster 1
Cluster 2
1

0.5
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
Median Time-Point [Normalized]

Figure 5. Mean Log(IWI) Pattern (Argumentation Essay)

To further visually examine how the distribution of the IWIs from the two clusters

differs, we plotted the density of IWI and the corresponding median time-points (Figures

6 and 7). The generation of the density graphs was described previously in the Method

section. For cluster 1 on the policy recommendation essay, we found that the density of

IWIs appears to hover around 3-to-4 seconds throughout the text production process, with

density shorter than two seconds fairly low. In contrast, the density pattern in cluster 2 is
CLASSIFICATION OF WRITING PATTERNS 16

notably different, where we find higher density with shorter IWIs at the beginning and the

end of the composition process.

Figure 6. Visualizing the Density of IWI (Policy Recommendation Essay)

Figure 7. Visualizing the Density of IWI (Argumentation Essay)

The contrast in IWI density pattern between clusters 1 and 2 is even more dramatic

for the argumentation essay. Cluster 1 shows more evenly distributed IWIs hovering

around 2 seconds throughout the composition, whereas cluster 2 exhibits more frequent

and shorter pauses at the two ends of the writing process. These results of the contrasting
CLASSIFICATION OF WRITING PATTERNS 17

IWI patterns between the two clusters also agree with the previous analyses presented in

Figures 4 and 5.

Table 1 shows additional differences between the clusters. For both essays, cluster

1 writers were more able than cluster 2 writers evidenced by the higher human scores on

the essay quality. Based on the t-test results, the average human scores for cluster 1 were

both statistically significantly higher than cluster 2, and practically significant in terms of

effect size. Cluster 1 also shows similarities to previous research results in which students

who scored higher spent more time on task, wrote longer essays, and were more fluent in

executing critical writing processes (e.g., Deane & Zhang, 2015).

Table 1

Characteristics of essay responses by cluster

Policy Recommendation Essay

Human Score Word Insert Writing Time

Cluster N
Mean(SD) Mean(SD) Mean(SD)

1 635 2.58 (0.80) 230 (123) 459 (239)

2 196 2.27 (0.81) 181 (111) 382 (247)

Effect Size ES = 0.39 ES = 0.49 ES = 0.37

t-value 4.73 (p < 0.0001) 5.99 (p < 0.0001) 4.59 (p < 0.0001)

Argumentation Essay

Human Score Word Insert Writing Time

Cluster N
Mean(SD) Mean(SD) Mean(SD)

1 591 2.87 (0.85) 233 (110) 353 (203)

CLASSIFICATION OF WRITING PATTERNS 18

2 311 2.57 (0.90) 178 (97) 293 (189)

Effect Size ES = 0.35 ES = 0.57 ES = 0.32

t-value 4.94 (p < 0.0001) 8.10 (p < 0.0001) 4.54 (p < 0.0001)
Note. Word insert: sum of words that an author inserts which may or may not appear in

the final product. Writing time (in sec.): active composition time excluding pause time

before the first keystroke. Effect size and t-value for the writing-time measure are based

on log-transformed values.

Results for Research Question 2: Relation to Human Ratings

The second research question examines the association of IWI duration and median

time-point with human scores; that is, the correlation between information extracted from

the writing processes with the quality of the final product. In particular, we conducted

regression analyses to predict writing scores from the vectors for the top 25 IWIs, and

correlated the resulting predicted scores with human ratings.

Ignoring cluster membership, the correlation coefficient between predicted and

observed human scores was 0.46 for the policy recommendation essay (n = 831) and 0.48

for the argumentation essay (n = 902). However, when cluster membership was taken into

account, we found notable differences between the clusters. The cluster-specific regression

models resulted in correlation coefficients of 0.47 for cluster 1 and 0.65 for cluster 2 for

the policy recommendation essay. Similar results were observed for the argumentation

essay, where cluster 1 also yielded a considerably lower correlation coefficient (0.46) than

cluster 2 (0.51).
CLASSIFICATION OF WRITING PATTERNS 19

Discussion

Our analysis focused on the 25 longest inter-word pauses in each essay and

indicated that student response patterns fell into two major groups. In pattern 1, the 25

longest pauses were distributed relatively evenly throughout the composition. Essays that

exemplify pattern 1 received higher mean human scores, contained more words, and were

composed over a longer active writing time. In pattern 2, the longest 25 pauses were

somewhat shorter than in pattern 1, and were concentrated at the beginning and end of the

composition. Essays that exemplify pattern 2 received lower mean human scores, had

fewer words, and were written over a shorter active composition time. We replicated these

findings across two writing prompts, each focused on a different writing purpose and

administered to different student samples. It is worth stressing that the results of writing

patterns should be interpreted at the group level; that is, the patterns do not reflect the

distribution of the 25 longest IWIs of any individual KL.

In the literature, pauses of more than two seconds are generally described as

terminating 'bursts' of text production (Chenoweth & Hayes, 2001), and tend to be

interpreted in think-aloud protocols as occasions for sentence-level planning (Baaijen,

Galbraith & de Glopper, 2012). This cognitive interpretation can readily be applied to

pattern 1. As can be observed in Figures 7 and 8, the longest pauses produced by pattern-1

writers fell most often between 1.5 and 4 seconds in duration, typical of pauses between

bursts. This interpretation is strengthened by the fact that pattern-1 pauses were evenly

distributed across the entire text. The resulting rhythm – a regular series of bursts of fast

production delimited by pauses of 2 seconds or more – may be emblematic of fluent text

CLASSIFICATION OF WRITING PATTERNS 20

production, in which the writer pauses primarily to plan the next major grammatical or

textual unit.

The striking feature of pattern 2 is the presence of a second kind of pause, mostly

shorter than the pauses observed in pattern 1, concentrated near the beginning and the end

of the composition. These time points are arguably where a writer who has difficulty

generating text is most likely to experience difficulty, consistent with Baaijen, Galbraith &

de Glopper (2012)'s observation that certain kinds of behavioral events, such as text

production followed by revision, are associated with shorter pauses. It is thus possible,

though by no means certain, that the higher frequency of short pauses concentrated at the

beginning and ends of pattern 2 essays reflects difficulties in text generation, leading to

false starts and interruptions instead of fluent text production at the beginning of an essay

(when the writer is under the most stress to plan content), and at the end of an essay (when

the writer may be running out of ideas, and thus once more experiencing higher levels of

uncertainty about what to write next.) We conducted a post-hoc qualitative analysis of a

small subset of logs from this dataset, and found that some weaker writers did produce a

small amount of text – a few words, or even part of a word – and then delete it after a short

pause, only to proceed to another false start. It is thus possible that pattern 2 involves this

kind of hesitation, although we cannot confirm it without further analysis in which we

correlate the distribution of IWIs with the distribution of deletions and edits.

Conclusion

In this study, we propose a new way to compare the temporal sequence of IWIs

across different students using a vector representation. This approach enables us to describe

global patterns in pausing behavior, which may correspond to different cognitive strategies
CLASSIFICATION OF WRITING PATTERNS 21

or styles of writing. This study represents an initial attempt, using a specific keystroke log

feature (IWIs) and a specific similarity metric, to explore ways to represent and directly

compare KLs, analyze the resulting classification patterns, and pose cognitive accounts for

the identified patterns in the context of writing done for standardized tests. Overall, our

analysis indicates that there do appear to be qualitative differences among groups of writers

in the time-course of text production, some of which differences can be detected from a

very small sample of events (e.g., only the 25 longest inter-word intervals).

However, it should be note that the method we employed in this study represents

our starting point to explore better representations and similarity measures for the KLs.

Based on the current methodological scheme, we observed some clear pattern differences

in students’ writing processes, which held across two prompts. However, a different scale

transformation, for example, on the IWI time-point, will change the similarity matrix

structure and affect the clustering results. In our future investigations, we will experiment

with other similarity measures (e.g., Euclidean or Mahalanobis types of distance measures)

and representations such as matched filtering, which might be more robust than the current

approach.

It is also important to note that the decision to target the 25 longest IWIs represents

two levels of abstraction: first, by restricting attention to IWIs, and second, by excluding

shorter IWIs from the analysis. These decisions provided a useful lens with which to

examine the data, since the literature provides strong reasons to suspect that the longest

IWIs will reflect global differences in writing patterns and strategies. The decision to

standardize to the 25 longest IWIs also made it easier to compare essays of different lengths

(and which were composed over shorter or longer time periods), but it does represent a
CLASSIFICATION OF WRITING PATTERNS 22

small portion of the total data; hence, it will be useful to extend the scope of future analysis

to include all IWIs.

Deane (2014) provides evidence that many keystroke features are not particularly

stable across changes in prompt, genre, and/or topic. Therefore, caution should be

exercised in generalizing the results. Further studies are needed to determine the extent to

which these results reflect prompt-specific or general differences in student writing

behaviors, which will require studying students from other grade levels and writing

prompts targeting other writing purposes (e.g., narrative writing).

Finally, it might be valuable to enrich the representation to include information

about the context of such writing actions as IWI. For example, some IWIs happen between

words in a long burst of text production; others, in the context of other actions, such as

edits or deletions. We would interpret the second cluster, in which most pauses were near

the beginning and end of student essays, very differently if they were associated with

editing and deletion, than we would if they were associated with uninterrupted text

production. Thus, it would be of particular value to enrich the current approach by

undertaking analyses that identify qualitative, linguistic or behavioral differences and that

would allow us to relate those findings to the differences in writing patterns observed here.

References

Almond, R., Deane, P., Quinlan, T., & Wagner, M. (2012). A preliminary analysis of

keystroke log data from a timed writing task (RR-12-23). ETS Research Report:

Princeton, NJ.

Alves, R. A., Castro, S. L., & de Sousa, L. (2007). Influence of typing skill on pause–

execution cycles in written composition. In Rijlaarsdam, G. (Series Ed.); M.

CLASSIFICATION OF WRITING PATTERNS 23

Torrance, L. van Waes, & D. Galbraith (Volume Eds.), Writing and cognition:

Research and applications (Studies in Writing, Vol. 20, pp. 55-65). Amsterdam:

Elsevier.

Baaijen, V. M., Galbraith, D. & de Glopper, K. (2012). Keystroke analysis: Reflections

on procedures and measures. Written Communications, 29, 246-277.

Banerjee, R., Feng, S., Kang, J., S., & Choi, Y. (2014). Keystroke patterns as prosody in

digital writings: A case study with deceptive reviews and essays. Proceedings of

the 2014 Conference on Empirical Methods in Natural Language Processing, Doha,

Qatar.

Beauvais, C., Olive, T., & Passerault, J. (2011). Why are some texts good and others not?

Relationship between text quality and management of the writing processes.

Journal of Educational Psychology, 103, 415-428.

Bennett, R. E. (2010). Cognitively Based Assessment of, for, and as Learning (CBAL): A

preliminary theory of action for summative and formative assessment.

Measurement, 8, 70-91.

Chenoweth, N. A., & Hayes, J. R. (2001). Fluency in writing: Generating text in L1 and

L2. Written Communication, 18, 80-98.

Chukharev-Hudilainen, E. (2014). Pauses in spontaneous written communication: A

keystroke logging study. Journal of Writing Research, 6, 61-84.

Cohen, J. (1968). Weighted kappa: Nominal scale agreement with provision for scaled

disagreement or partial credit. Psychological Bulletin, 70, 213-220.

CLASSIFICATION OF WRITING PATTERNS 24

Deane, P. (2014). Using writing process and product features to assess writing quality and

explore how those features relate to other literacy tasks (RR-14-03). ETS Research

Report: Princeton, New Jersey.

Deane, P., Sabatini, J. S., Feng, G., Sparks, J., Song, Y., Fowles, M., … Foley, C. (2015).

Key practices in the English Language Arts (ELA): Linking learning theory,

assessment, and instruction (RR-15-17). ETS Research Report: Princeton, NJ.

Deane, P., & Zhang, M. (2015). Exploring the feasibility of using writing process features

to assess text production skills (RR-15-26). ETS Research Report: Princeton, New

Jersey.

Dragsted, B., & Carl, M. (2013). Towards a classification of translation styles based on

eye-tracking and keylogging data. Journal of Writing Research, 5, 133-158.

Fleiss, J. L., & Cohen, J. (1973). The equivalence of weighted kappa and the intraclass

correlation coefficient as measures of reliability. Educational and Psychological

Measurement, 33, 613–619.

Gould, J. D. (1980). Experiments on composing letters: Some facts, some myths, and some

observations. In L. Gregg and E. Steinberg (Eds.), Cognitive processes in writing

(pp. 97-127). Hillsdale, NJ: Lawrence Erlbaum.

Grabowski, J. (2008). The internal structure of university students’ keyboard skills. Journal

of Writing Research, 1, 27-52.

Hao, J., Smith L., Mislevy, R., von Davier, A., & Bauer, M. (in press). Taming log files

from game and simulation-based assessment: Data model and data analysis tool.

ETS Research Report: Princeton, New Jersey.

CLASSIFICATION OF WRITING PATTERNS 25

Hinkle, D. E., Wiersma, W., & Jurs, S. G. (2002). Applied statistics for the behavioral

sciences (5th Ed.). Wadsworth Publishing.

Johnson, S. C. (1967). Hierarchical clustering schemes. Psychometrika, 32, 241-254.

Jones, E., Oliphant, T., & Peterson P. (2014). SciPy: Open source scientific tools for

Python [Computer software]. Retrieved from [Link]

Kalbfleisch, J. D., & Prentice, R.L. (2002). The statistical analysis of failure time data (2nd

ed.). Hoboken, New Jersey: John Wiley and Sons.

Leijten, M., Macken, L., Hoste, V., van Horenbeeck, E., & van Waes, L. (2012). From

character to word level: Enabling the linguistic analyses of Inputlog process data.

Proceedings of the EACL 2012 Workshop on Computational Linguistics and

Writing, Avignon, France.

Leijten, M., & van Waes, L. (2013). Keystroke logging in writing research using Inputlog

to analyze and visualize writing processes. Written Communication, 30, 358-392.

Leijten, M., van Waes, L., Schriver, K., & Hayes, J. R. (2014). Writing in the workplace:

Constructing documents using multiple digital sources. Journal of Writing

Research, 5, 285-377.

Miller, K. S. (2000). Academic writers on-line: Investigating pausing in the production of

text. Language Teaching Research, 4, 123-148.

Roca de Lario, J., Manchon, R., Murphy, L., & Marin, J. (2008). The foreign language

writer’s strategic behavior in the allocation of time to writing processes. Journal of

Second Language Writing, 17, 30-47.

Ulrich, R., & Miller, J. (1993). Information processing models generating lognormally

distributed reaction times. Journal of Mathematical Psychology, 37, 513–525.

 CLASSIFICATION OF WRITING PATTERNS 26

van der Linden, W. (2006). A lognormal model for response times on test items. Journal

of Educational and Behavioral Statistics, 31, 181-204.

van Waes, L. Leijten, M., & van Weijen, D. (2009). Keystroke logging in writing research:

Observing writing processes with Inputlog. GFI-Journal, No 2-3.

Xu, X., & Ding, Y. (2014). An Exploratory study of pauses in computer-assisted EFL

writing. Language Learning & Technology, 18, 80-96.

Zhang, M., & Deane, P. (2015). Process features in writing: Internal structure and

incremental value over product features (RR-15-27). ETS Research Report:

Princeton, New Jersey.

Acknowledgements

We would like to thank Marie Wiberg, Don Powers, Gary Feng, Tanner Jackson,

and Andre Rupp for their technical and editorial suggestions for this manuscript, thank

Randy Bennett for his support of the study, and thank Shelby Haberman for his advice on

the statistical analyses in this study.

View publication stats