Archives of Clinical Neuropsychology, Vol. 14, No. 5, pp.

419–432, 1999
Copyright © 1999 National Academy of Neuropsychology
Printed in the USA. All rights reserved
0887-6177/99 $–see front matter

PII S0887-6177(98)00033-X

Expanded Scoring Criteria for the Design

Fluency Test: Reliability and Validity in
Neuropsychological and College Samples

Stephanie Lewis Harter and Christine C. Hart

Texas Tech University

Gregory W. Harter

Health South Hospital

In spite of sensitivity to right frontal lobe dysfunction, the Design Fluency Test (DFT) has been
limited by one global score with little psychometric data. This study developed an expanded
scoring system with standardized instructions for multiple dimensions of design performance
and provided reliability and validity data in a college (n 5 64) and diverse neuropsychological
sample (n 5 165). The scoring system allowed reliable scoring of number of novel designs, com-
plexity of designs, variations in designs, and concrete, frankly perseverative, and scribbled re-
sponses. Performances for a college sample were relatively stable at 1-month retest, but showed
practice effects for number of novel designs and complexity. Two principal components showed
modest, expected relationships to other neuropsychological measures in the clinical sample.
Clinical subjects with a history of closed-head injury or dementing disorder showed impaired
DFT performances, with complexity the most sensitive indicator of impairment. © 1999 Na-
tional Academy of Neuropsychology. Published by Elsevier Science Ltd

Jones-Gotman and Milner’s (1977) Design Fluency Test (DFT) showed initial potential
for specific sensitivity to right frontal lobe lesions, which often elude neuropsychological
tests (cf., Ruff, Light, & Evans, 1987; Stuss & Benson, 1984). Their original study found
patients with primarily right frontal or right frontal-central cortical excisions to be most
impaired in design fluency, followed by those with right temporal or left frontal exci-
sions. All groups of patients with unilateral cortical excisions were impaired relative to
matched, normal controls, except for a group with small left posterior lesions. A subse-
quent study, including larger groups of patients with more focal lesions, suggested that
impairment on the DFT was specific to patients with excisions involving the right frontal
lobes, and further suggested that the right central region was particularly important in
design fluency. In contrast, those with left central excisions showed the most severe im-
pairments in verbal fluency (Jones-Gotman, 1991a, 1991b). Subsequent studies using the

Address correspondence to Stephanie Lewis Harter, Department of Psychology, Box 42051, Texas Tech Univer-
sity, Lubbock, TX 79409-2051; E-mail: steph@[Link]

419
420 S. L. Harter, C. C. Hart, and G. W. Harter

DFT have also demonstrated impairments in patients with presumed bilateral frontal
lobe dysfunction, including those with closed-head injury (Varney et al., 1996), schizo-
phrenia (Kolb & Whishaw, 1983; Stolar, Berenbaum, Banich, & Barch, 1994), and Par-
kinson’s Disease (Taylor, Saint-Cyr, & Lang, 1986). Design fluency impairments have
also been observed in those with Alzheimer’s, multi-infarct, and alcohol-related demen-
tia (Bigler, 1995; Bigler et al., 1988). Further, decreases in design fluency, related to im-
pairments in verbal fluency and other frontal lobe-dependent tasks, have been observed
with normal aging (Axelrod, 1989; Daigneault, Braun, & Whitaker, 1992; Mittenberg,
Seidenberg, O’Leary, & DiGiulio, 1989).
In spite of this initial promise, the DFT has been criticized for overly vague, global
scoring procedures and a lack of psychometric data (Ruff, Evans, & Marshall, 1986). In-
dividual differences in approach to the task, including complex or simplified designs and
completely unique or slightly varied designs (Hanks, Allen, Ricker, & Deshpande, 1996;
Ruff et al., 1987), have also led to difficulties in interpreting the meaning of individual
global scores for number of designs produced. Perhaps because of such reservations, the
DFT has failed to achieve the widespread use of analogous verbal fluency measures, par-
ticularly in clinical applications (Varney et al., 1996). One strategy to improve the reli-
ability of the DFT has been to increase structure in the nature of the task (e.g., Ruff et
al., 1987). However, the unstructured nature of the DFT, while complicating scoring,
may increase sensitivity to frontal-lobe–mediated executive functions by increasing the
initiation and organization required of the patient. The original DFT procedures specify
both an unstructured “free” condition and a more structured “fixed” condition, in which
exactly four lines must be used for each design. The free condition of the DFT appears
generally more sensitive than the fixed, although the fixed may be particularly sensitive
to Binswanger’s disease (Lezak, 1995; Jones-Gotman, cited in Spreen & Strauss, 1998).
Previous studies have generally provided normative and reliability data for only one
global score from the DFT, representing number of different, nonnameable novel de-
signs produced in 5 minutes. However, researchers have also observed qualitative differ-
ences in the performances of groups based on site of and etiology of lesion. Jones-Gotman
and Milner found patients with right and right frontal-central lobe excisions to have the
highest percentages of perseverative responses, scoring repetitions of similar designs, vari-
ations on a theme, and scribbled responses as perseverations (Jones-Gotman, 1991a,
1991b; Jones-Gotman & Milner, 1977). Bigler (1995) also noted increased perseverative
errors among demented patients. Increases in perseverative responses, without a reduc-
tion in novel design production, have also been noted with aging (Daigneault et al., 1992).
Jones-Gotman and Milner (1977) found nameable designs (e.g., pictures of concrete ob-
jects) to be associated with right temporal and to a lessor extent right frontal lobe excisions.
Previous indications that the more unstructured condition of the DFT is the more
sensitive, suggested that increasing standardization in scoring criteria, rather than in the
task itself, would best increase reliability while preserving the sensitivity of the test.
Thus, this study attempted to increase structure, specificity, and multidimensionality in
the scoring procedures for the DFT while preserving the unstructured nature of the task.
Extensions and modifications of the original scoring system were developed in order to
allow measurement of complexity of designs; differentiate between “variations on a
theme,” scored as perseverations by Jones-Gotman and Milner (1977), and more frankly
perseverative responses; and allow reliable scoring of more qualitative aspects of design
production. Specific instructions were developed for scoring (a) number of novel ab-
stract designs, (b) complexity, (c) “variations on a theme,” (d) drawing quality, (e) per-
severative responses, (f) concrete responses, and (g) random, scribbled responses. The
DFT was administered to a college sample and to a diverse clinical sample referred for
 Design Fluency Test 421

neuropsychological assessment. Scoring dimensions were evaluated for interrater and

test-retest reliability, intercorrelations, relation to other neuropsychological measures,
and ability to identify expected impairments in clinical groups. In particular, it was pre-
dicted that persons with presumed bilateral frontal lobe damage from closed-head injury
would have difficulty on the DFT, in relation to the severity of their head injury. Patients
with generalized dementia were also expected to perform poorly on the DFT.

METHODS

Subjects
College participants volunteered to participant in research for extra credit in their In-
troductory Psychology classes. They were given the DFT on two occasions, approxi-
mately 4 weeks apart, as well as a brief medical history interview to screen for any his-
tory of neurological injury or illness. College participants were primarily female (91%);
White, non-Hispanic (89%), and right-handed (86%). Their ages ranged from 17 to 58,
with an average of 20. Eighty-one percent of college participants reported no history
suggesting neurological injury, illness, or other dysfunction. Of the remaining 19%, 3
participants reported a history of striking their head in a motor-vehicle accident (MVA)
resulting in loss of consciousness and/or posttraumatic amnesia and 8 participants re-
ported a history of some other blow to the head resulting in a loss of consciousness or
posttraumatic amnesia. Only 1 of the participants reporting a history of possible head in-
jury reported continued problems related to the injury. These were headaches and re-
lated reading comprehension problems. Three other participants reported recent signs
of possible neurological dysfunction, of unknown origin. These included seizures, dizzi-
ness, and fainting, and memory problems.
Clinical participants included 165 neurological and psychiatric patients. Ages ranged
from 13 to 84, with a mean of 37. Years of formal education ranged from 2 to 20, with a
mean of 11. Fifty-three percent were female, 85% were right-handed, and 91% were
White, non-Hispanic. These included all referrals to the neuropsychological services of a
rural, regional hospital over a 2-year period who received the DFT as a part of the assess-
ment battery. Participants included both inpatients and outpatients and were referred
due to suspected cognitive impairments or to personality changes plausibly related to
some neurological injury or disease.
Diagnostic and history information were drawn from reports by the referring physi-
cians and from extensive medical history and clinical interviews done by the neuropsy-
chologists. Patients often presented with multiple possible etiologies of neuropsycholog-
ical dysfunction. Head injury appeared to be the primary etiological factor in 37% of the
cases. Ninety-five percent of these cases had sustained their head injury more than a
year previously. The time since injury ranged from 0 to 507 months, with a mean of 62
and a median of 20. An additional 21% of clinical participants had a history of head in-
jury, although there was another, more primary reason for referral. Other primary diag-
nostic groups included depressive disorders (12.7%), alcohol and drug abuse (8.5%),
schizophrenia (4.8%), neurotoxic exposure (4.2%), bipolar disorder mixed or manic
(4.2%), and a variety of other less frequent neurological and psychiatric disorders.

Measures
Design Fluency Test (DFT). The DFT was administered according to procedures drawn
from Jones-Gotman and Milner (1977), using standard written and oral instructions, in-
422 S. L. Harter, C. C. Hart, and G. W. Harter

cluding acceptable and nonacceptable examples. Participants were asked to draw as

“many, different” designs as possible in a 5-minute period, avoiding “scribbling” and any
drawing that could be “named.” Protocols were scored in a blind manner by a neuropsy-
chologist or by a doctoral student specializing in clinical neuropsychology using the stan-
dardized scoring instructions prepared for this study (available from the authors). Both
of these raters scored all protocols from the college sample and the first 40 protocols
from the clinical sample to establish interrater reliability (see Results section). In these
cases, scores by the neuropsychologist were used in all analyses following the reliability
analyses.

Other neuropsychological measures. Clinical participants originally took the DFT as a

part of a neuropsychological battery. There was variation among participants as to which
other measures were administered, depending upon the referral question. At least 80%
of patients completed the Wechsler Adult Intelligence Scale-Revised (n 5 156; Wechs-
ler, 1981), the Memory Assessment Scales (n 5 95; Williams, 1991), the Controlled Oral
Word Association Test (COWAT, n 5 98; Benton & Hamsher, 1989), the Trailmaking
Test (Trails A & B, n 5 160; Reitan, 1958), the Short Category Test (n 5 82; Wetzel &
Boll, 1987), Finger Oscillation (Tapping R & L, n 5 147; Reitan & Wolfson, 1993), the
Hand Dynamometer (n 5 132; Reitan & Wolfson, 1993), and the Dichotic Listening
Test (n 5 132; Roberts, Varney, Paulsen & Richardson, 1990). Thus, these measures
were compared to performances on the DFT.

RESULTS

Psychometric Properties of the DFT

Variations on a theme, perseverative responses, and scribbled responses occurred too
rarely in the college sample to accurately estimate reliability. Other DFT scores were an-
alyzed using repeated measures analysis of variance with rater and time of testing as in-
dependent variables. Means and standard deviations of the DFT scoring dimensions are
summarized in Table 1. Main effects for time of testing were significant for number of
novel designs, F (1, 63) 5 12.11, p , .001, and complexity of the protocol, F (1, 63) 5
9.07, p , .005, reflecting practice effects on both dimensions. No practice effects were

TABLE 1
Means (M) and Standard Deviations (SD) of Interrater and Test-
Retest Reliability Data for the College Sample (n 5 64)

Initial Testing Retesting

Rater 1 Rater 2 Rater 1 Rater 2

Number of novel designs

M 9.70 9.62 11.80 11.73
SD 5.73 5.83 6.46 6.52
Complexity
M 163.03 183.95 160.03 174.81
SD 70.87 73.88 69.56 70.99
Number of concrete responses
M .20 .17 .27 .30
SD .78 .92 1.01 1.08
Drawing quality
M 1.75 1.80 1.87 1.81
SD .53 .54 .42 .50
 Design Fluency Test 423

noted for concrete responses or drawing quality. Rater effects were noted only for com-
plexity of the protocol, with the less experienced rater tending to score protocols as
more complex, F (1, 63) 5 10.33, p , .005. This may have been due to differences in
scoring repetitious elements in designs (e.g., zigzag lines and other repeated patterns),
which are given limited credit to avoid inflating scores.
Reliability estimates were calculated from these analyses of variance using intraclass
correlations as discussed by Winer (1971) and Shrout and Fleiss (1979) and are summa-
rized in Table 2. These estimates were very close to reliability estimates calculated using
Pearson correlations. As expected, interrater reliabilities are higher than test-retest reli-
abilities, since they do not involve changes due to participant’s state at the time of testing
(e.g., fatigue, motivation). Interrater consistency and test-retest reliabilities for drawing
quality are less than desirable, but fair for a rating task requiring subjective clinical
judgement (cf., Cicchetti & Sparrow, 1981). All other interrater consistencies are at least
in the range recommended for research use by Nunnally (1978), as are test-retest reli-
abilities for complexity of the protocol and number of concrete responses. Test-retest
reliability for number of designs closely approaches the recommended range of r . .70.
Interrater consistencies for number of novel designs, complexity of the protocol, perse-
verative responses, and scribbled responses are particularly strong, achieving desirable
standards for tests intended for clinical application (Nunnally, 1978).
Distributions of DFT scores in the college and neuropsychological samples are sum-
marized in Table 3. No variations on a theme or perseverative responses were noted in
the college sample. Concrete and scribbled responses occurred only rarely in the college
sample, with only 6 cases offering 1 or more concrete responses (9.3%) and only 3 cases
offering a scribbled response (4.7%). Variations on a theme and perseverative, concrete,
and scribbled responses were few, but less infrequent, in the neuropsychological sample,
appearing in 25.5%, 4.2%, 32.7%, and 15.8% of protocols, respectively.
Numbers of perseverations and scribbled responses deviated markedly from normal
distribution in the clinical sample. Examination of participants’ scores on these variables

TABLE 2
Reliability of the Expanded Scoring System for the
Design Fluency Test

Intraclass Correlations

Test-Retest
Scoring Dimensions Interrater (4 weeks)

College students (n 5 64)

Number of novel designs 1.00 .69
Complexity .97 .77
Number of concrete responses .90 .91
Drawing quality .59 .51
Clinical subjects (n 5 40)
Number of novel designs .98
Complexity .98
Number of variations .77
Number of preseverations 1.00
Number of concrete responses .81
Number of scribbles 1.00
Drawing quality .46

Note. Reliability for variations, perseverations, and scribbles could not

be computed for the college student sample, due to the rare occurrence
of those responses in that sample. Retest measures were not available
for the clinical sample.
424 S. L. Harter, C. C. Hart, and G. W. Harter

revealed three participants with extreme responses that contributed to the nonnormality
of distribution on these scores. One participant with a long history of chronic undifferen-
tiated schizophrenia with generalized impairments on neurocognitive testing produced
206 simple, scribbled responses that also resulted in scoring of 205 perseverative re-
sponses, since the responses were virtually identical to each other. Another subject with
mild to moderate mental retardation produced 16 perseverative responses. A third par-
ticipant with a comorbid history of personality disorder, alcohol abuse, and learning dis-
ability and a remote history of a mild head injury produced 17 scribbled responses. In order
to prevent these outliers from unduly influencing analyses, their scores were transformed
to reduce the extremity of responses while preserving rank order, as suggested by
Tabachnick and Fidell (1989). Thus, 16 perseverations was transformed to 4, one higher
than the next highest number of 3 perseverative responses, and 205 perseverations was
transformed to 5. Seventeen scribbled responses was transformed to 7, one higher than
the next highest number of 6 scribbled responses, and 206 scribbled responses was trans-
formed to 8. As indicated in Table 3, these transformations increased the normality of
distribution for these variables. Group comparisons of diagnostic groups discussed be-
low did not include these three participants because of their comorbid diagnoses and/or
because of insufficient participants sharing their primary diagnosis.
Intercorrelations of DFT scoring dimensions in both the college and clinical samples
suggested that it is a multidimensional task. Principal components analyses with oblique
rotations were used to summarize the major dimensions reflected in scores. In the col-
lege sample, perseverative responses and variations on a theme were not included in
analyses since they did not occur on the initial testing. Analyses of the other scoring di-
mensions yielded 2 principal components with eigenvalues greater than 1.00 (1.79 and
1.16, respectively) accounting for 59% of the variance. Results of the analyses are sum-

TABLE 3
Distribution of Design Fluency Test Measures

Scoring Dimension M SD Skew Kurtosis

College sample (n 5 64)a

Number of novel designs 9.70 5.73 1.20 2.21
Complexity 163.03 70.87 .02 2.33
Number of variations 0 0
Number of perseverations 0 0
Number of concrete responses .20 .78 4.80 25.22
Number of scribbled responses .05 .21 4.39 17.84
Drawing qualityb 1.75 .53 2.16 2.26
Clinical sample (n 5 165)
Number of novel designs 10.19 8.39 2.87 15.65
Complexity 109.01 64.48 0.54 20.11
Number of variations 0.90 2.48 5.03 34.23
Number of perseverations 1.39 15.00 12.73 162.84
(with transformations)c (.08) (.51) (7.48) (60.02)
Number of concrete responses 0.90 2.31 4.60 26.83
Number of scribbled responses 1.70 16.09 12.64 161.39
(With transformations)c (.33) (1.14) (4.39) (20.87)
Drawing Qualityb 1.57 0.59 0.28 20.59
aInitialtesting, neuropsychologist’s scores.
bRatings of drawing quality: 0 scribbles only, 1 crude, 2 average, 3 excel-
lent, precise.
cScores on two extreme outliers for each of these variables were re-

duced to 1 and 2 above the next highest scores, preserving the order of
scores.
 Design Fluency Test 425

marized in Table 4. Correlations of variables with factors suggest that the first compo-
nent reflects quantity of abstract design production, reflecting number and complexity of
novel designs, and the second component reflects scribbled and poorly drawn responses.
In the clinical sample, three components were extracted with eigenvalues greater than
one (eigenvalues of 2.17, 1.61, and 1.07) accounting for 69% of the variance. The first
component appeared to reflect quantity of abstract design production, correlating most
highly with number of novel designs (r 5 .83), variations on a theme (r 5 .77), and com-
plexity of the protocol (r 5 .75). The second component appeared to reflect scribbled,
perseverative responses, correlating most highly with scribbled responses (r 5 .85) and
perseverative responses (r 5 .62) and inversely with drawing quality (r 5 2.72). The third
component appeared to reflect concreteness. It was most highly correlated with concrete
responses (r 5 .89), but was also correlated with perseverative responses (r 5 .59) and
was inversely related to complexity (r 5 2.42). The third component appeared to be less
distinctly and reliably measured, since only number of concrete responses had its highest
correlation with this component. Thus, a second principal components analysis of the
clinical sample extracted two components, allowing more direct comparison with the
structure of scores in the college sample. Results of this analysis are also summarized in
Table 4. The first two components accounted for 54% of the variance. Similar to the col-
lege sample, the first component appeared to reflect quantity of abstract design produc-
tion, while the second component appeared to reflect scribbled and perseverative re-
sponses. Concrete responses were inversely related to the first component and directly
related to the second component. These components had a low inverse relationship in the
clinical sample (r 5 213), but were relatively independent in the college sample (r 5 2.06).

Relation to Demographic Variables

The DFT appeared relatively independent of demographic factors. Men and women
did not differ in complexity of the protocol, design quality, or numbers of variations, per-
severations, concrete responses, or scribbled responses in either the college or clinical
samples. Men and women did not differ in number of novel designs in the clinical sam-

TABLE 4
Principal Components Analyses of the Design Fluency Test in
College and Clinical Samples

Structure Matrix

Abstract Scribbling/
Scoring Dimension Design Productivity Perseveration

College sample (n 5 64)

Complexity .874 2.186
Number of novel designs .817 .040
Number of concrete responses 2.442 .013
Number of scribbles .183 .806
Drawing quality .335 2.718
Clinical Sample (n 5 165)
Number of novel designs .826 2.087
Complexity .799 2.359
Number of variations .713 .140
Number of perseverations 2.013 .758
Number of scribbles .062 .705
Drawing quality .234 2.688
Number of concrete responses 2.337 .382
426 S. L. Harter, C. C. Hart, and G. W. Harter

ple, but men did produce significantly more designs in the college sample (males: M 5
15.17, SD 5 8.70, n 5 6; females: M 5 9.14, SD 5 5.12, n 5 58; p , .05). However, the
small number of men in the college sample (few men volunteered for the study) raised
questions about the representativeness of that subgroup. Education was unrelated to all
scoring dimensions in the clinical sample. There was a nonsignificant trend for more ad-
vanced students to produce more novel designs in the college sample (r 5 .23, p 5 .06).
Age was unrelated to all scoring dimensions in the college sample and was only slightly
related to an increased number of variations on a theme in the clinical sample (r 5 .17,
p , .05).
Hand dominance was related to some of the more qualitative scoring dimensions.
Left-handed participants had poorer drawing quality in the clinical sample (right-handed:
M 5 1.62, SD 5 .58; left-handed: M 5 1.29, SD 5 .55; p , .05), more perseverations
(right-handed: M 5 .02, SD 5 .19; left-handed: M 5 .62, SD 5 1.41; p , .001), and more
scribbled responses (right-handed: M 5 .28, SD 5 .89; left handed: M 5 2.54, SD 5 .52;
p , .001). They also tended to have more variations on a theme (right-handed: M 5 .75,
SD 5 1.83; left-handed: M 5 1.79, SD 5 4.74; p 5 .06). Left-handed participants in the
college sample also tended to have poorer drawing quality (right-handed: M 5 1.80, SD 5
.52; left-handed: M 5 1.44, SD 5 .53; p 5 .06). As discussed previously perseverative re-
sponses and variations on a theme did not occur in initial testing in the college sample.

Relation to Other Neuropsychological Measures

Other neuropsychological measures were not available for the college sample. Princi-
pal component scores from the two-factor analysis of the clinical sample were compared
to performances on other neuropsychological measures completed by at least 80% of
the clinical sample (see Methods section). Component 1, abstract design productivity,
was weakly related to performance IQ (r 5 .18, p , .02) and tended to be related to ver-
bal IQ (r 5 .15, p 5 .06). It had a low relation to verbal fluency on the COWAT (r 5 .23,
p , .005) and tended to be related to tapping for the right hand (r 5 .16, p 5 .05). It was
inversely related errors on the Short Category test (r 5 2.18, p , .05) and to seconds re-
quired to complete Trails A (r 5 2.18, p , .05) and Trails B (r 5 2.24, p , .01). Com-
ponent 2, scribbled, perseverative responses, was inversely related to verbal IQ (r 5
2.31, p , .001) and performance IQ (r 5 2.34, p , .001), verbal fluency (r 5 0.26, p ,
.001), verbal (r 5 2.31, p , .001) and visual (r 5 2.24, p , .005) memory on the MAS,
dichotic listening performance for the right (r 5 2.33, p , .001) and left (r 5 2.27, p ,
.001) ears, and tapping right (r 5 2.30, p , .001) and left (r 5 2.19, p , .05). It was also
related to seconds required to complete Trails B (r 5 .34, p , .001) and errors on the
Short Category test (r 5 .25, p , .005). These relationships to other neurocognitive mea-
sures were in the expected directions, but their modest magnitude suggested that the
DFT is not redundant to other measures in a neuropsychological battery.

Discrimination Between Clinical Groups and the College Comparison Group

DFT measures were included in a multivariate analysis of variance (MANOVA),
comparing various clinical diagnostic groups to the nonclinical, college student compari-
son group. The 12 students reporting a possible history of neurological injury or other
dysfunction were excluded from the college group. Six patients with a primary diagnosis
of minor closed head injury, involving no loss of consciousness (LOC) or posttraumatic
amnesia (PTA) were excluded from analyses, due to inadequate group size. The remain-
ing participants with a primary diagnosis of closed-head injury were divided into mild to
 Design Fluency Test 427

moderate (LOC and PTA less than 2 hours) and severe (LOC and PTA greater than 2
hours) groups. Additional clinical groups included those with primary diagnoses of a de-
menting disorder, alcohol or drug abuse or dependency, and unipolar depressive disor-
der. Other diagnostic categories were excluded from analyses due to small subsample
size. Dependent variables included number of novel designs, complexity, drawing qual-
ity, variations on a theme, and perseverative, scribbled, and concrete responses. See Ta-
ble 5 for a summary of DFT performances for each diagnostic group.
Diagnostic groups differed significantly in multivariate comparison, F (Wilks crite-
rion, 35, 570) 5 2.78, p , .001. Univariate comparisons were significant for complexity,
concrete responses, scribbled responses, number of novel designs, and variations on a
theme, in order of decreasing effect size (.18, .11, .08, and .08, respectively). Planned
comparisons of each diagnostic group to the college comparison group (with significance
level at p , .05) on variables significant in univariate analyses indicated that the mild to
moderate head injury group had more scribbled responses and variations on a theme
than the college group. The severe head injury group had less complex protocols with
more concrete responses than the college group. The dementing disorders group had
less complex protocols and more scribbled responses and variations than the college
group. The depressive disorders group produced more novel designs, but less complex
protocols with more concrete responses than the college group. They also had more vari-
ations on a theme. Planned comparisons of the mild-moderate and severe head injury
groups and dementing disorders group to other clinical groups on variables with overall

TABLE 5
Means (M) and Standard Deviations (SD) of Diagnostic Groups and a Nonclinical College Student
Comparison Group on Dimensions of Design Fluency Test Performance

Number of
Novel Drawing Concrete Scribbled
Diagnostic Group Designs Complexity Quality Variations Perseverations Responses Responses

Mild–moderately
head injured
(n 5 35)
M 9.49 129.57 1.66 1.06 .06 .54 .54
SD 7.07 67.03 .59 3.76 .24 1.87 1.17
Severely head injured
(n 5 20)
M 6.50 69.05 1.45 .10 .15 1.70 .35
SD 4.91 48.34 .60 .45 .67 2.77 1.14
Dementia
(n 5 9)
M 10.11 87.00 1.22 1.78 .00 .89 .67
SD 8.24 62.73 .44 3.60 .00 1.17 1.32
Etoh/drug abuse
(n 5 14)
M 10.50 116.43 1.43 .71 .00 .29 .00
SD 8.65 79.30 .51 1.38 .00 .47 .00
Unipolar depression
(n 5 17)
M 15.29 102.94 1.53 1.71 .00 1.35 .06
SD 16.15 57.88 .51 3.08 .00 2.40 .24
Nonclinical college
(n 5 52)
M 8.92 155.50 1.71 .00 .00 .13 .06
SD 5.03 72.04 .54 .00 .00 .52 .23
428 S. L. Harter, C. C. Hart, and G. W. Harter

univariate significance were also conducted. The mild to moderate head injury group
had significantly more complex protocols than the severe head injury group. They had
more scribbled responses and fewer novel designs than the depressive disorders group.
They also had more scribbled designs than the alcohol/drug abuse group. In addition to
the previously mentioned differences from the mild-moderate head injury group, the se-
vere head injury group had fewer novel designs than the depressive disorders group.
They also had less complex protocols and more concrete responses than the alcohol/
drug abuse group. The dementing disorders group did not differ significantly from any of
the other clinical groups on number of novel designs, complexity, variations, concrete re-
sponses, or scribbled responses. However, the small size of the dementing disorders
group limited power for these comparisons.

Impaired Performances
For DFT dimensions that significantly discriminated diagnostic groups in the above
analyses, percent of participants in each diagnostic group with impaired performances
was calculated to further assess clinical relevance of the DFT scoring dimensions. Num-
ber of novel designs and complexity were considered impaired if the participant per-
formed more than 1.5 standard deviations below the college comparison group on that
variable (1 or no novel designs, 56 or less for complexity of the protocol). Variations on
a theme, scribbled responses, and concrete responses rarely or never occurred in the col-
lege comparison samples. Thus, any response of this type was considered to indicate pos-
sible impairment. Percentages of each diagnostic group exhibiting potentially impaired
performances are summarized in Table 6.
Of the 52 college comparison participants, only 10 (19.2%) had a potential impair-
ment on any dimension of the DFT. These included 4 with reduced complexity (7.7%), 4
with 1 to 3 concrete designs (7.7%), and 3 with 1 scribbled response (5.8%). All of the
participants with dementing disorders had some impaired performance. From 63 to 65%
of the other diagnostic groups had some impaired performance. Reduced complexity
and concrete responses were particularly characteristic of the severe head injury group
and the dementing disorders group. The dementing disorders group was also character-
ized by variations on a theme and scribbled responses. However, variations on a theme

TABLE 6
Percent of Each Diagnostic Group with Impaired Performances

Number of Concrete Scribbled Any

Novel Designs Complexity Variations Responses Responses Impairment
Diagnostic Group (%) (%) (%) (%) (%) (%)

Mild–moderately
head injured
(n 5 35) 2.9 11.4 25.7 25.7 22.9 62.9
Severely head injured
(n 5 20) 20.0 45.0 5.0 50.0 15.0 65.0
Dementia
(n 5 9) 11.1 44.4 44.4 44.4 33.3 100
Etoh/drug abuse
(n 5 14) 7.1 28.6 35.7 28.6 0 64.3
Unipolar depression
(n 5 17) 0 29.4 29.4 47.1 5.9 64.7
Nonclinical college
(n 5 52) 0 7.7 0 7.7 5.8 19.2
 Design Fluency Test 429

and scribbled responses were not reliably associated with severity of head injury, since
they were more likely to occur in the mild to moderate than the severe head injury
groups.

DISCUSSION

Results provide initial reliability and validity data for an expanded scoring system for
the DFT. Performances remained relatively stable on 1-month retesting, especially con-
crete responses (r 5 .91) and complexity of the protocol (r 5 .77), filling a previous need
for published test-retest reliability estimates. There were modest practice effects for
complexity of the protocol, which increased by 11%, and for novel designs, which in-
creased by 22% (two designs).
Previous studies have reported interrater reliabilities for number of novel designs
ranging from .64 to .87 (Jones-Gotman, 1991a; Ross, Axelrod, Hanks, Kotasek, & Whit-
man, cited in Spreen & Strauss, 1998; Woodard, Axelrod, & Henry, 1992), in contrast to
the current reliabilities of 1.00 in the college sample and .98 in the clinical sample. The
expanded instructions also allow reliable scoring of the complexity of designs in the pro-
tocol (r 5 .97–.98), providing more comprehensive and sensitive measurement of design
production. The expanded scoring instructions partition rule violations into persevera-
tions, concrete responses, and scribbled responses. The reliability of rule violations has
rarely been addressed in previous studies, since in the original scoring procedures rule
violations were subtracted from total number of designs to produce one global score for
number of novel designs. Two studies giving reliabilities for rule violations have re-
ported inter-rater reliabilities of .21 to .74, depending upon the nature of the violation
(Ross et al., cited in Spreen & Strauss, 1998; Woodard et al., 1992), in contrast to present
reliabilities of .81 to 1.00. Previous scoring systems have not allowed separate examina-
tion of variations on a theme, which were combined with frank perseverations. The
present scoring system allows variations to be scored reliably enough (r 5 .77) to allow
investigation of whether such responses represent perseverative repetitions or strategic
variations to increase design production (cf., Vik & Ruff, 1988). Attempts to provide
qualitative ratings of the skill and precision of drawing yielded lower reliabilities than
the more quantitative scoring dimensions.
Performance on the DFT appeared to be multidimensional in both college and clini-
cal samples. The primary and secondary components in both samples included abstract
design productivity and scribbled responses. There also appeared to be a smaller third
component in the clinical sample reflecting concrete responses that did not emerge in
the college sample due to the reduced number of scoring dimension relevant in that sam-
ple. Modest inverse relationships were found between scribbled, perseverative re-
sponses and verbal and performance IQ, verbal fluency, and other neuropsychological
tests dependent upon integrity of the frontal and temporal lobes. Generally smaller rela-
tionships were observed between abstract design productivity and performance IQ, ver-
bal fluency, and other measures related to frontal lobe functioning. The pattern and
magnitude of these relationships is consistent with previous findings for number of novel
designs (Varney et al., 1996) in suggesting that Design Fluency is not a redundant mea-
sure in a neuropsychological battery, that it is related in the expected manner to other
tests of frontal lobe functioning, and that poor performance is not simply a matter of im-
paired motor speed or visual-motor integration.
Results are consistent with previous research (Varney et al., 1996) in suggesting im-
paired DFT performance in those with a history of closed-head injury. The expanded
430 S. L. Harter, C. C. Hart, and G. W. Harter

scoring system allows more complete assessment of dimensions of impairment. Reduced

complexity of responses in the protocol and concrete responses were particularly related
to severity of head injury.
Results are also consistent with previous research in suggesting impaired DFT perfor-
mance in patients with generalized cognitive deficits associated with dementing disor-
ders (Bigler, 1995; Bigler et al., 1988). All dementing patients were impaired on some
dimension of the DFT. Performances of dementing patients were particularly character-
ized by reduced complexity and increased scribbled responses and variations on a theme
in contrast to the college comparison sample. High percentages of this sample also
showed concrete responses.
Overall, complexity of the protocol appeared to be the most sensitive indicator of ab-
stract design productivity, more effectively discriminating groups with presumed frontal
lobe impairment and generalized dementia. However, the present failure of number of
novel designs to discriminate head-injured and dementing groups from the college com-
parison sample may be partially due to characteristics of the college sample. College stu-
dents had the highest average complexity of designs, which necessarily limited the num-
ber of designs that they could produce within the time limit. Number of novel designs in
the college comparison group was lower than that of older normal controls in previous
studies (Jones-Gotman & Milner, 1977; Varney et al., 1996; Woodard et al., 1992), in-
cluding a normal aging sample (Axelrod, 1989). The complexity of designs in those sam-
ples is not known. However, the current college sample did not differ in number of novel
designs and complexity from a larger, gender-balanced college sample used in a study of
test order effects (Hart, Harter, Miller, & Harter, 1996). College students may be differ-
ently motivated than older adult volunteers and more inclined to produce complex de-
signs. The current college sample was also predominately female, unlike the clinical sam-
ple. Males in the college sample produced more designs than females. However, there
were no gender differences for DFT performances in the present clinical sample, in the
previous college sample (Hart et al., 1996), or in Varney et al. (1996) control and clinical
samples. Overall, comparison to previous studies suggests that use of a college compari-
son sample may have yielded an underestimate of impairments in number of novel de-
signs for the clinical groups. However, further comparisons of normal groups on the
DFT would be necessary to confirm this speculation.
In contrast to the college comparison group, the depressive disorders group produced
a higher number of novel designs, more similar to comparison groups used in previous
studies. Both head-injured groups in the current study produced fewer novel designs
than the depressive disorder’s group.
All clinical subjects in the current study were referred due to suspected cognitive
problems or complaints or personality changes subsequent to neurological insult. Thus,
it is unclear that the depressive disorders or the alcohol and drug abuse groups were free
of neurological impairment. In addition to lack of an age-matched normal comparison
sample, the current study is also limited by relatively small size of some clinical groups.
For instance, the small number of patients with dementing disorders limited power for
comparisons to other groups and precluded examination of different types of dementia.
Also, the very small number of patients with focal or clearly lateralized lesions pre-
vented assessment of sensitivity of the DFT to laterality.
The expanded scoring system offers the opportunity to further study qualitative dif-
ferences in DFT performance related to locus and etiology of neurological injury. The
current patterns of performance for head-injured versus demented patients are sugges-
tive in this regard and should be cross-validated with a larger sample. It would addition-
ally be useful in future studies to investigate the specificity of the DFT to neurological
 Design Fluency Test 431

disorders, in comparison to psychiatric comparison groups that do not include patients

with suspected neurocognitive impairment.

REFERENCES

Axelrod, B. N. (1989). Frontal lobe functioning in normal aging. Unpublished doctoral dissertation, Wayne
State University, Detroit, MI.
Benton, A. L., & Hamsher, K. de S. (1989). Multilingual Aphasia Examination. Iowa City, IA: AJA Associ-
ates.
Bigler, E. D. (1995). Design fluency in dementia of Alzheimer’s type, multi-infarct dementia and dementia
associated with alcoholism. Applied Neuropsychology, 2, 7–14.
Bigler, E. D., Schultz, R., Grant, M., Knight, G., Lucas, J., Roman, M., Hall, S., & Sullivan, M. (1988). Design
fluency in dementia of the Alzheimer’s type: Preliminary findings. Neuropsychology, 2, 127–133.
Cicchetti, D. V., & Sparrow, S. A. (1981). Developing criteria for establishing interrater reliability of specific
items: Applications to assessment of adaptive behavior. American Journal of Mental Deficiency, 86, 127–
137.
Daigneault, S., Braun, C. M. J., & Whitaker, H. A. (1992). Early effects of normal aging on perseverative and
non-perseverative prefrontal measures. Developmental Neuropsychology, 8, 99–114.
Hanks, R. A., Allen, J. B., Ricker, J. H., & Deshpande, S. A. (1996). Normative data on a measure of design
fluency: The Make a Figure Test. Assessment, 3, 459–466.
Hart, C. C., Harter, S. L., Miller, S. A., & Harter, G. W. (1996). Design Fluency within a cognitive battery: A
study of test order effects. Paper presented at the National Academy of Neuropsychology, 16th Annual
Meeting, New Orleans, LA.
Jones-Gotman, M. (1991a). Localization of lesions by neuropsychological testing. Epilepsia, 32, 541–552.
Jones-Gotman, M. (1991b). Presurgical psychological assessment in children: Special tests. Journal of Epi-
lepsy, 3 (Suppl. 1), 93–102.
Jones-Gotman, M., & Milner, B. (1977). Design Fluency: The invention of nonsense drawings after focal cor-
tical lesions. Neuropsychologia, 15, 653–64.
Kolb, B., & Whishaw, I. Q. (1983). Performance of schizophrenic patients on tests sensitive to left or right
frontal, temporal, or parietal function in neurological patients. Journal of Nervous and Mental Disease,
171, 435–443.
Lezak, M. D. (1995). Neuropsychological assessment. New York: Oxford University Press.
Mittenberg, W., Seidenberg, M., O’Leary, D. S., & DiGiulio, D. V. (1989). Changes in cerebral functioning
associated with normal aging. Journal of Clinical and Experimental Neuropsychology, 11, 918–932.
Nunnally, J. C. (1978). Psychometric theory (2nd ed.). New York: McGraw-Hill.
Reitan, R. M. (1958). Validity of the Trail Making Test as an indicator of organic brain damage. Perceptual
and Motor Skills, 8, 271–276.
Reitan, R. M., & Wolfson, D. (1993). The Halstead-Reitan Neuropsychological Test Battery: Theory and clin-
ical interpretation. Tucson, AZ: Neuropsychology Press.
Roberts, R., Varney, N. R., Paulsen, J., & Richardson, E. (1990). Dichotic listening and partial seizures. Jour-
nal of Clinical and Experimental Neuropsychology, 12, 448–458.
Ruff, R. M., Evans, R., & Marshall, L. F. (1986). Impaired verbal and figural fluency after head injury.
Archives of Clinical Neuropsychology, 1, 87–101.
Ruff, R. M., Light, R. H., & Evans, R. W. (1987). The Ruff Figural Fluency Test: A normative study with
adults. Developmental Neuropsychology, 3, 37–51.
Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological
Bulletin, 86, 420–428.
Spreen, D., & Strauss, E. (1998). A compendium of neuropsychological tests (2nd ed.). New York: Oxford
University Press.
Stolar, N., Berenbaum, H., Banich, M. T., & Barch, D. (1994). Neuropsychological correlates of alogia and
affective flattening in schizophrenia. Biological Psychiatry, 35, 164–172.
Stuss, D. T., & Benson, D. F. (1984). Neuropsychological studies of the frontal lobes. Psychological Bulletin,
95, 3–28.
Tabachnick, B. G., & Fidell, L. S. (1989). Using multivariate statistics (2nd ed.). New York: Harper Collins.
Taylor, A. E., Saint-Cyr, J. A., & Lang, A. E. (1986). Frontal lobe dysfunction in Parkinson’s disease: The
cortical focus of neostriatal outflow. Brain, 109, 845–883.
Varney, N. R., Roberts, R. J., Struchen, M. A., Hanson, T. V., Franzen, K. M., & Connell, S. K. (1996). Design
432 S. L. Harter, C. C. Hart, and G. W. Harter

Fluency among normals and patients with closed head injury. Archives of Clinical Neuropsychology, 11,
345–353.
Vik, P., & Ruff, R. R. (1988). Children’s figural fluency performance: Development of strategy use. Develop-
mental Neuropsychology, 4, 63–74.
Wechsler, D. (1981). WAIS-R Manual. New York: The Psychological Corporation.
Wetzel, L., & Boll, T. J. (1987). Short Category Test, Booklet Format. Los Angeles, CA: Western Psychologi-
cal Services.
Williams, J. M. (1991). Memory Assessment Scales. Odessa, FL: Psychological Assessment Resources.
Winer, B. J. (1971). Statistical principles in experimental design (2nd ed.). New York: McGraw Hill.
Woodard, J. L., Axelrod, B. N., & Henry, R. R. (1992). Interrater reliability of scoring parameters for the
Design Fluency Test. Neuropsychology, 6, 173–178.