Appeared in Proceedings of VL 2000: IEEE International Symposium on Visual Languages.

Seattle, WA: IEEE Computer Society, September 10-13 2000, pp. 157-164.

Tabular and Textual Methods for Selecting Objects from a Group

John F. Pane Brad A. Myers

Computer Science Department Human Computer Interaction Institute
Carnegie Mellon University Carnegie Mellon University
Pittsburgh, PA 15213 USA Pittsburgh, PA 15213 USA
+1 412 268 8078 +1 412 268 5150
pane+vl2000@[Link] bam+@[Link]

Abstract ming languages is the accurate specification of boolean

expressions [1]. In addition to programming languages, this
The accurate formulation of boolean expressions is a same problem also appears in the task of formulating que-
notorious problem in programming languages and data- ries for common end-user activities such as web searching,
base query tools. This paper studies the ways that untrained library catalog searching, and other database retrieval tasks
users naturally express and interpret queries, revealing [2]. Despite the great difficulty that users have demon-
some of the underlying reasons why this task is so difficult. strated with using the boolean operators AND , OR , and
Among the study's findings are: people interpret the word NOT to construct these expressions, no universally better
AND to mean either conjunction or disjunction depending alternatives have been discovered, so most programming
on context, the scope to which they attribute the word NOT languages continue to rely on them, including many visual
depends on whether the subsequent operator is AND or OR, and forms-based languages (e.g., [3, 4]). Early web search
and they often ignore parenthesis. Therefore, relying on engines also used these operators, although many have
these words and symbols for query formulation will result turned to less expressive query languages (for example, the
in poor usability. A tabular query form is proposed that plus and minus unary operators for inclusion and exclu-
avoids the need to name the operators, provides a clear dis- sion). Newsweek reports that even with these simplifica-
tinction between conjunction and disjunction, and makes tions, most web users are dissatisfied with search engines,
grouping more explicit. Comparing the tabular language and less than 6% manage to use these operators in their
with textual boolean expressions, the study finds that searches [5].
untrained users perform better when they express their que- The problems with boolean queries are exemplified in
ries in the tabular language, and about equally well when studies of non-programmers writing solutions to program-
interpreting queries written in either language. We con- ming problems in their own words [6, 7]. For example, in
clude that systems may benefit by adopting a tabular nota- these studies it was very common for participants to use the
tion for query formulation. word AND where the word OR is the correct boolean opera-
tor. Instead of saying something like “count the cars with
license plates from Georgia or Louisiana” they would say
“count the cars with license plates from Georgia and Loui-
siana.” The latter version refers to an empty set of license
1. Introduction plates when interpreted according to boolean logic, but in
English it is usually interpreted to mean the union of the
We are applying human-computer interaction techniques two states’ license plates.1 It was also noted that the words
to the design of new programming language features. OR and NOT rarely appeared, suggesting that boolean
Design decisions are resolved by looking to prior research expressions are not a natural way to formulate these state-
for warnings about potential problems, suggestions for ments. The participants often used other words and sen-
design alternatives, and guidance in selecting among vari- tence structures to specify their queries accurately. For
ous potential solutions. When necessary, we perform new example, rather than saying “if I get up late and I’m not
user studies to investigate questions that are not fully very hungry I skip breakfast,” they might say “if I get up
addressed by prior research. late I skip breakfast unless I’m very hungry.” This latter
We are currently using this method to design a primarily construction avoids both the AND and NOT operators.
textual programming system for children to use in creating In the new study reported here, we investigated several
interactive simulations and games. One of the challenges of of these alternative formulations to see whether they were
this effort is to craft the features of the system to address more accurate than traditional boolean expressions. In addi-
the problems observed in the prior research. While this is
straightforward in some cases, it is quite difficult in others. 1. This ambiguity in how to interpret the word that means “and” also
One of the more notorious problem areas in program- appears in many other natural languages, according to our informal poll.
tion, because prior research suggests that non-textual query [16]), but studies of use of natural language for program-
languages may be more effective than textual syntaxes [8], ming are less common (e.g. [17, 18]).
the study compared these textual alternatives against a pro- These problems have led researchers to develop graphi-
posed new query language that uses tabular forms that cal interfaces for queries. For example, truth tables and
would integrate well into a primarily textual language. Venn diagrams have been shown to be effective for specify-
The study used a grid of nine colored shapes, where a ing simple queries [12, 19, 20]. Another system used tiles in
subset of the shapes could be marked. The participants were a two-dimensional grid, where one dimension represented
given two kinds of problems: code generation problems, union and the other represented intersection, although these
where some shapes were already marked and they had to implicit semantics were found to be confusing [21]. A sys-
formulate a query to select them; and code interpretation tem that used the graphical metaphor of water flowing
problems, where they were shown a query and had to mark through filters was found to be superior to boolean expres-
the shapes selected by the query. They solved all of these sions [8], however the screen space required for this tool
problems twice, once using purely textual queries, and once might limit its effectiveness in a larger context such as a
using the proposed tabular forms. programming language.
The results suggest that a tabular language for specifying
boolean expressions can improve the usability of a pro- 3. Design alternatives for Boolean queries
gramming or query language. On code generation tasks, the
participants performed significantly better using the tabular Our new programming language for children will be pri-
form, while on code interpretation tasks they performed marily textual. In reviewing the prior research we found
about equally in the textual and tabular conditions. The that although the problem of boolean queries is notorious,
study also uncovered systematic patterns in the ways partic- there are few prescriptions for how solve it effectively. For
ipants interpreted boolean expressions, which contradict the example, the prior work suggests that we should avoid
typical rules of evaluation used by programming languages. using the words AND, OR, and NOT, but there is no hard
These observations help to explain some of the underlying evidence that a different textual query language would be
reasons why boolean expressions are so difficult for people any better.
to use accurately, and suggest that refining the vocabulary Earlier studies have analyzed the natural language solu-
and rules of evaluation might improve the learnability and tions that non-programmers provided to solve program-
usability of textual query languages. A general awareness ming problems, and identified some common trends in the
of these contradictions can help designers of future query ways that boolean queries were expressed [6, 7]. The
systems adhere to one of the basic human-computer inter- vocabulary and syntax of these solutions were uncon-
action principles: to speak the user’s language [9]. strained, so they provide insight into how people prefer to
express their answers. We speculated that a programming
2. Related work language that closely matches these natural preferences
would be more usable than one that requires users to trans-
There is a large body of prior research that identifies late their natural solutions into a less natural form. With this
usability issues in the design of programming languages, in mind, we proposed several alternate ways to express tex-
and can provide a basis for making design decisions. For tual queries and compared them in this study. In addition,
the language we are currently designing for children, a sur- we also proposed a tabular format for queries.
vey of the prior research that applies to beginners is espe-
cially relevant [10]. 3.1. Tabular query forms
Many researchers have noted that boolean query lan-
guages using the AND, OR, and NOT operators are not very Although some graphical query methods had been
effective in programming languages or database retrieval shown to be more effective than boolean expressions, many
(e.g. [1, 2]). Several researchers have noted that the com- of them were limited to expressing very simple queries. We
mon usage of these operators in natural language causes wanted a solution that is fully expressive. Also, many of the
errors in queries, such as the substitution of AND for OR graphical systems would not integrate well into a program-
[11, 12]. It has also been noted that the intended scope of ming language, where the entire computer screen cannot be
the NOT operator is ambiguous in natural language [13]. devoted to this one subtask of the programming process.
The difficulties of boolean expressions are intensified We required a format that is compact and readable in the
when several operators must be combined to form the query context of a larger program. With these points in mind, we
[14]. Parenthesis improved performance in that study, but designed a tabular form that is fully expressive and compat-
other studies have shown that beginners have difficulty with ible with the programming language we are developing.
parentheses, especially if they are nested [11, 12]. Since our new programming language will represent
Replacing the boolean query language with a different data on cards containing attribute-value pairs, we designed
subset of natural language, using other words for the opera- the query form to also use a card metaphor. For the pur-
tors, is still likely to be inadequate [15]. Many systems that poses of this study, we simplified the forms by leaving out
permit unrestricted natural language queries have been the attribute names, and limiting the number of terms to
shown to be effective for information retrieval tasks (e.g. three. We called these match forms (see Figure 1). Criteria
are placed into the slots, one term per slot. All of the terms express a negated term without using the NOT operator.
on a single form implicitly form a conjunction. Negation is
specified by prefacing a term with the NOT operator. Dis- 4.3. Location of Unless
junction is specified by including an additional match form
adjacent to the first one. Hypothesis 3: Users will interpret an Unless clause
more accurately when it appears at the very end of the
statement.
Although in English it may be natural to say “if a then ...
unless b,” in programming languages those ellipsis (...) may
be filled with a large block of code. If the Unless clause will
appear at the very end of the IF statement, it will be far
removed from the part of the query that is specified in the
Figure 1. Match forms expressing the query: IF clause. Because this violates the principle of locality
(blue and not square) or (circle and not green) [22], it may reduce usability. While the principle of locality
could not be tested directly with our simple stimuli, we
This two-dimensional layout is similar to the grid of tiles wanted to investigate whether the Unless clause is sensitive
described by Anick et al. [21] – one dimension implements to its placement within the query.
intersection and the other implements union. However,
match forms provide cues to help users remember which 4.4. Context-dependent interpretation of AND
operator uses each dimension, such as the text in the form
heading and the visual grouping. In addition, the scope of Hypothesis 4: Users will interpret AND as boolean con-
the NOT operator is made explicit by confining it to a single junction in some contexts but not in other contexts.
term. This proposed query language can express arbitrarily People often use AND in places where the correct bool-
complex queries, although some queries have to be formu- ean operator is OR. This may be because interpretation of
lated in a less concise way than pure boolean expressions AND in the English language depends on its context. In
would allow. To relieve this somewhat, the forms in our some cases it is interpreted to be a further restriction on a
proposed language will also allow an entire form to be query (boolean conjunction or set intersection), while in
negated (“objects that do not match ...”), but that feature is other cases it is interpreted to expand the query (boolean
not used in this study. disjunction or set union). For example, these two statements
are usually interpreted differently: “pick up the boxes that
4. Hypotheses are blue and green” vs. “pick up the boxes that are blue and
the boxes that are green.” We attempted to demonstrate this
The study tests nine hypotheses. The first seven hypothe- context-sensitive interpretation of the AND operator.
ses examine various textual alternatives to traditional bool-
ean expressions, and the last two hypotheses examine the 4.5. Verbose AND vs. OR
tabular design alternative.
Hypothesis 5: Users will interpret a verbose AND
4.1. AND vs. nested IF expression as boolean disjunction more accurately than an
OR expression.
Hypothesis 1: Users will interpret nested IF statements If Hypothesis 4 is confirmed, it would be useful to char-
more accurately than a boolean expression using AND. acterize the contexts in which AND is interpreted as a bool-
In the prior studies, people frequently nested an IF state- ean disjunction instead of conjunction. If certain
ment inside another IF statement. Instead of saying, “if a constructions consistently lead to disjunctive interpreta-
and b then ... ,” they would say, “if a then if b then ... .” The tions, perhaps they can reliably replace the rarely-used OR
use of nested IF s may be easier to use and understand operator. We hypothesized that a more verbose expression
because it avoids using the confusing AND operator for that restates part of the query is more likely to induce a dis-
conjunction, and keeps the boolean expression simpler. junctive interpretation (see the example in Section 4.4.).

4.2. NOT vs. Unless 4.6. Operator precedence of NOT

Hypothesis 2: Users will interpret an Unless clause Hypothesis 6: Users will interpret the NOT operator
more accurately than a boolean expression that uses AND with lower precedence than the other boolean operators.
and NOT. People often interpret the NOT operator with lower pre-
In the prior studies, people often wrote a simple condi- cedence than the other boolean operators. This is opposite
tional statement and then stated an exception at the end. For to the rules of interpretation in most programming lan-
example, they would write, “if a then ... unless b” This is an guages, where NOT has higher precedence than the other
alternative to “if a and not b then ... .” In addition to avoid- boolean operators. That is, in “not a and b,” programming
ing the AND operator, the Unless clause permits the user to languages associate the NOT tightly with the “a”, while we
expect people to first interpret the expression “a and b” and assigned to a path through the questions. The paths were
then apply the NOT operator to the result. constructed so that, for every pair of questions we intended
to compare, the number of times that either question
4.7. Parenthesis for expression grouping appeared first was balanced.
The eleven RF questions were constructed by translating
Hypothesis 7: Users will misinterpret parenthesized the RT questions into the tabular language. So, each partici-
expressions. pant answered the same question twice, once with text and
Regardless of the precedences chosen for the boolean again with match forms. Comparing the performance in
operators, a mechanism is required for the user to clarify or these two conditions is the basis for testing hypothesis 8.
override them. Programming languages typically use paren- By random assignment, half the participants solved the RF
thesis to explicitly group sub-expressions, but research has questions first, and the other half solved the RT questions
shown that beginners have difficulty with parenthesis. first, to control for any effect of presentation order.
There were a total of 32 questions in the four reading
4.8. Tabular vs. textual and writing sections. After this, participants answered a sur-
vey of seven preference questions. Each of these showed a
Hypothesis 8: Users will interpret queries that use query result along with two or more queries that would cor-
match forms more accurately than equivalent textual que- rectly generate the result. The participants were asked to
ries. select the one they liked the best.
Hypothesis 9: Users will generate more accurate queries
using match forms than they generate using text. 5.1. Participants
We investigated the relative usability of match forms
compared with text on both interpretation and generation of In addition to examining these hypotheses with children
queries. We expected match forms to be effective because who are the target audience of our programming language,
they eliminate many of the problems with text that are dis- we were interested in how the results would generalize to
cussed above. By avoiding the words AND and OR, any other ages. So, we recruited both children and adults to par-
confusion with the meaning of these words in English is ticipate in the study.
avoided. Also, the precedence or grouping of the operators 33 volunteers participated, 13 children (ages 10-14), and
becomes less ambiguous. 20 adults (ages 18-46). 14 were male and 19 were female.
All but two were native speakers of English. 7 participants
5. Method reported that they had written computer programs (4 adults,
3 children). 27 reported that they had some experience with
Before beginning the study, participants filled out a web search engines, and 18 had used advanced searching
questionnaire that collected basic demographic informa- features (such as AND, OR, NOT, +, -, etc.). Two adults
tion. Then they answered a set of problems that were were experienced with the SQL database query language.
divided into four sections. In two of the sections, the writ-
ing sections, the participants generated queries to match a 5.2. Materials
result that we supplied. In the other two sections, the read-
ing sections, participants interpreted queries that we sup- The 32 problems were presented in a web browser, one
plied. We label the two writing sections WT (writing text) problem per web page. Each of the problem groups was
and WF (writing forms), and the two reading sections RT preceded by an instruction page explaining how the query
(reading text) and RF (reading forms). language or query forms work and introducing the format
There were five WT questions and five identical WF of the exercises. The WT and WF instruction pages were
questions. Comparing the performance across these two constructed to be as similar to each other as possible, as
conditions is the basis for testing hypothesis 9. By random were the RT and RF pages. The web server managed the
assignment, half of the participants answered the WT ques- random assignment of participants to a path through the
tions first, and the other half answered the WF questions problems, the presentation of the problems in the order
first, to control for any effect of presentation order. All of determined by that path, and the collection of the data
the writing questions were presented before any of the read- anonymously. Figure 2 contains an example problem from
ing questions, so that the queries that were displayed in the each of the four problem groups.
reading sections would not bias their responses in the writ-
ing sections. 5.3. Procedure
There were eleven RT questions, forming the basis for
testing hypotheses 1-7. The first five hypotheses can be Participants began on the demographic questionnaire
evaluated by comparing relative student accuracy across a page and proceeded at their own pace through the materials.
pair of questions. Hypotheses 6 and 7 can be evaluated by They were instructed to be as accurate as possible and were
examining which of two interpretations the participants told that there was no time limit. When they submitted an
used in answering a single question. To control for any answer, the server recorded it and presented them with the
effect of presentation order, participants were randomly next page in the sequence. The server performed some
 WF WT

RF RT

Figure 2. An example problem from each of the four problem groups (WF, WT, RF, and RT) before the
answers are filled in. The color of each object is red, green or blue on the computer screen.

basic syntactic checks (for example, it made sure the user any incorrect replacement of one color or shape with
provided a boolean operator on the WT tasks, and that they another one that did appear in the study (e.g. blue for green)
didn’t put multiple criteria into a single slot in the WF was marked as incorrect. Except where otherwise noted,
tasks). If this check failed, an error message asked the par- textual answers were interpreted the way a programming
ticipant to go back and fix the answer. Any time participants language would interpret them. Invented shorthand nota-
returned to a previous page to revise an answer, we tions were marked as incorrect. Redundant or overly com-
recorded all of the answers but only used the final one for plex answers were scored as correct if they resulted in the
the results presented here. correct selection. Finally, on the problems where we gave
Each participant’s answer was scored as correct or incor- special instructions, answers that did not follow the instruc-
rect according to the following policy. Spelling errors were tions were marked as incorrect even if they resulted in a
tolerated, as were additional words such as an, a, or the. correct selection (e.g. one of the questions in WF and WT
Plural and singular forms of all words were accepted. Con- asked the users include the word NOT in their answers).
sistent use of an incorrect color name that did not actually Because we simplified the match forms for this study,
appear in the study (e.g. orange for red) was tolerated. But, some of the problems became more complex when they
were translated from RT to RF. For example, the lack of a Hypothesis 5 is disconfirmed.
% dis-
way to negate a whole match form causes the expression Users will interpret a verbose AND expression as boolean disjunc- junction
tion more accurately than an OR expression.
“not (a and b)” to be translated into the tabular equivalent of
select the objects that match blue and the objects that match circle 55
“(not a) or (not b).” These question pairs were discarded
select the objects that match square or green 82
from the comparison of RT to RF in testing hypothesis 8.
p<.05

6. Results Hypothesis 6 is disconfirmed for NOT with AND.

Users will interpret the NOT operator with lower precedence than
the other boolean operators.
No significant differences were detected between chil- select the objects that match not red and square %
dren and adults, between males and females, or between precedence of NOT is higher than AND 64
programmers and non-programmers, so the results are interpreted as: (not red) and square

aggregated across all of the participants. The numbers precedence of NOT is lower than AND 9
interpreted as: not (red and square)
shown are percentages.
p<.001
In evaluating hypotheses 1-5, we performed within sub-
ject comparisons on pairs of questions from the RT problem Hypothesis 6 is confirmed for NOT with OR.
Users will interpret the NOT operator with lower precedence than
group. Statistical significance in these comparisons was the other boolean operators.
evaluated with a non-parametric sign test. To test hypothe- select the objects that match not triangle or green %
ses 6 and 7, we examined which of two interpretations par- precedence of NOT is higher than OR 9
ticipants used in answering a single question. Statistical interpreted as: (not triangle) or green

significance in these comparisons was evaluated with a precedence of NOT is lower than OR 67
interpreted as: not (triangle or green)
binomial test. In evaluating hypotheses 8 and 9, we com- p<.001
pared pairs of questions between RT & RF and between WT
& WF, respectively. These comparisons were within sub- Hypothesis 7 is confirmed.
ject, and statistical significance was evaluated with a non- Users will misinterpret parenthesized expressions.
%
select the objects that match (not circle) or blue
parametric sign test. In all of the statistical tests, p<.05 was ignore parenthesis, NOT has low precedence 39
used as the threshold for significance. interpreted as: not (circle or blue)
Hypothesis 1 is not confirmed. observe parenthesis 12
Users will interpret nested IF statements more accurately than a interpreted as: (not circle) or blue
boolean expression using AND. % correct
p<.05
Nested IF 94
select the objects that match red, if the objects match triangle As mentioned above, three of the RF problems were not
AND 85 well-matched to the corresponding RT problems, so these
select the objects that match blue and circle pairs were discarded in analyzing hypothesis 8. This left
not significant eight pairs of reading problems to test hypothesis 8. All five
Hypothesis 2 is not confirmed.
pairs of writing problems were used to test hypothesis 9.
Users will interpret an Unless clause more accurately than a bool-
ean expression that uses AND and NOT. %correct Hypothesis 8 is not confirmed.
Unless 97 Users will interpret queries that use match forms more accurately
% correct
than equivalent textual queries.
select the objects that match blue, unless the objects match square
Match Forms (RF) 71
AND NOT 94
select the objects that match square and not red Text (RT) 74
not significant not significant

Hypothesis 3 is confirmed. Hypothesis 9 is confirmed.

Users will interpret an Unless clause more accurately when it Users will generate more accurate queries using match forms than
appears at the very end of the statement. % correct they generate using text. % correct
Unless at end 97 Match Forms (WF) 94
select the objects that match blue, unless the objects match square Text (WT) 85
Unless earlier 76 p<.0001
unless the objects match green, select the objects that match circle The following table breaks down the individual prob-
p<.05 lems in WF vs. WT, showing the percent correct. The prob-
Hypothesis 4 is confirmed. lems are labeled with our canonical text solutions.
Users will interpret AND as boolean conjunction in some contexts % con-
but not in other contexts. junction
select the objects that match blue and circle 85
select the objects that match blue and the objects that match circle 36
(55% of the participants interpreted this as boolean disjunction)
p<.0001
 red and square (blue and cir- circle square The hypothesis was disconfirmed when comparing the pre-
triangle and not
red
cle) or (red
and triangle)
or blue and not
reda
cedence of NOT with AND. 64% of the participants treated
NOT with higher precedence than AND, matching the com-
Match Forms 94 73 91 42 33
(WF) mon usage in programming languages. However, the
hypothesis was confirmed when comparing the precedence
Text (WT) 94 64 12 18 21
n.s. n.s. p<.0001 p<.01 n.s.
of NOT with OR. In this case, 67% of the participants
treated NOT with lower precedence than OR. Since consis-
a. The word NOT was required in the solution to this problem tency is an important human-computer interaction principle
[9], this reversal in the natural interpretation of precedence
7. Discussion suggests that it is unwise to rely on implicit precedence
rules.
Although the results help to explain some of the reasons Hypothesis 7 was confirmed. Users ignored parenthesis
why boolean queries using AND, OR, and NOT are so diffi- significantly more often than they observed them. The
cult, the textual alternatives that we proposed did not query was, “select the objects that match (not circle) or
improve performance. On the other hand, the proposed tab- blue.” The results on hypothesis 6 suggest that without the
ular query forms did improve performance on writing tasks, parenthesis, most participants would have applied the NOT
while performing about the same on reading tasks. operator to the expression “circle or blue.” The parenthesis
were not able to override this tendency.
7.1. Textual query variations
7.2. Match forms vs. text
Hypothesis 1 was not confirmed. The participants per-
formed about the same using nested IF statements as they Hypothesis 8 was not confirmed. On reading tasks, the
did using a boolean expression with the AND operator. On participants performed about as well with match forms as
the preferences survey, the majority of the participants pre- they did with text. However, hypothesis 9 was confirmed.
ferred the boolean expression. On writing tasks, the participants performed significantly
Hypothesis 2 was not confirmed. The participants per- better with the match forms than they did with text. This
formed about the same using an Unless clause as they did disparity, where a positive effect is stronger on generation
using a boolean expression with the AND and NOT opera- tasks than interpretation tasks, has also been observed in
tors. On the preferences survey, the majority of the partici- other systems (e.g. [23]). On the preferences survey, which
pants preferred the boolean expression. was a reading task, the participants’ choices were about
Hypothesis 3 was confirmed. The participants performed equally divided between text and match forms.
significantly better with the Unless clause at the end than Match forms were not superior for code interpretation,
they did with the Unless clause earlier in the statement. but they did not have a detrimental impact on that task.
However, given the result of Hypothesis 2, the importance Thus the overall effect of using match forms should be pos-
of this result is questionable. Also, the very simple prob- itive due to the strong gains on generation tasks, despite the
lems used in this study did not provide a good way to test lack of an effect on interpretation tasks.
the situation where we speculated that the Unless would The breakdown of individual questions in the query gen-
violate the principle of locality. On the preferences survey, eration task shows that the participants performed about the
most of the participants preferred the Unless at the end. same in the two conditions when the queries were simpler,
Hypothesis 4 was confirmed. Two slightly different que- but as the queries became more complex, the differences in
ries using AND resulted in significantly different interpreta- favor of the match forms increased. While the trend in favor
tions. 85% of the participants interpreted the AND in “select of match forms was present in all cases, only the queries
the objects that match blue and circle” as a conjunction that involved disjunction revealed significant differences
operator. But only 36% of them interpreted it that way in between match forms and text. As expected, the most com-
“select the objects that match blue and the objects that mon error on these problems was the substitution of AND
match circle.” Instead, 55% of them interpreted the AND in where OR was required.
the second statement as a disjunction operator. This result The three problems that were excluded from the reading
helps to explain the frequently observed error where users comparison were among the more complex queries. Since
incorrectly use AND instead of OR. the advantage of query forms is stronger on more complex
Hypothesis 5 was disconfirmed. Despite the fact that the queries, excluding this data may have reduced any positive
majority of the participants interpreted AND as a disjunc- effect of match forms on the reading task, making it less
tion operator in “select the objects that match blue and cir- likely that our study would be able to confirm hypothesis 8.
cle,” they are significantly more accurate in interpreting Further research into this question is warranted.
disjunction if the OR operator is used. On the preferences The strong effect of match forms came with very little
survey, the majority of the participants preferred OR over a training. It is unlikely that the participants had used an
verbose AND statement to express disjunction. equivalent tabular query language before, and they only
In the surprising results of hypothesis 6, we measured viewed a brief instruction page with a few examples before
reliable effects in opposite directions depending on context. beginning to solve the problems. While the instructions for
the textual problems were similarly brief, the participants tation and Evaluation,” Journal of American Society for
brought knowledge from a lifetime using the words AND, Information Science, vol. 44, pp. 327-339, 1993.
[9] J. Nielsen, “Heuristic Evaluation,” in Usability Inspection
OR, and NOT in English. This may have interfered with the Methods, J. Nielsen and R. L. Mack, Eds. New York: John
programming language interpretation, or made them less Wiley & Sons, 1994, pp. 25-62.
careful in reading the instructions. [10] J. F. Pane and B. A. Myers, “Usability Issues in the Design of
Novice Programming Systems,” Carnegie Mellon University,
8. Conclusions Pittsburgh, PA, School of Computer Science Technical
Report CMU-CS-96-132, August 1996.
[11] S. L. Greene, S. J. Devlin, P. E. Cannata, and L. M. Gomez,
Based on the results of this study, we can make the fol- “No IFs, ANDs, or ORs: A Study of Database Querying,”
lowing recommendations to designers of programming lan- International Journal of Man-Machine Studies, vol. 32, pp.
guages, scripting tools and search engines that incorporate 303-326, 1990.
[12] A. Michard, “Graphical Presentation of Boolean Expressions
query mechanisms: in a Database Query Language: Design Notes and an Ergo-
• Do not use the word AND. nomic Evaluation,” Behaviour and Information Technology,
• Do not rely on parenthesis for grouping. vol. 1, pp. 279-288, 1982.
• Do not rely on implicit operator precedence rules. [13] A. McQuire and C. M. Eastman, “Ambiguity of Negation in
Natural Language Queries,” in Proceedings of the Eighteenth
• Consider using tabular query forms instead of pure text. Annual International ACM SIGIR Conference on Research
This study of what is natural for untrained users provides and Development in Information Retrieval, Posters:
a scientific basis for choosing among design alternatives in Abstracts, 1995, pp. 373.
[14] P. J. M. D. Essens, C. A. McCann, and M. A. Hartevelt, “An
query tools for beginners. We will use these results in the Experimental Study of the Interpretation of Logical Opera-
design of a new programming system for children, expect- tors in Database Querying,” in Cognitive Ergonomics: Con-
ing that this strategy will yield a language that is easier to tributions from Experimental Psychology, Database
learn and use than other languages. In addition to these spe- Interrogation, G. C. v. d. Veer, S. Bagnara, and G. A. M.
cific recommendations, the strategy employed here can be Kempen, Eds. Amsterdam: North-Holland, Elsevier Science
Publishers, 1992, pp. 201-225.
used by developers to assist in the design of other kinds of [15] A. Kohl and W. Rupietta, “The Natural Language Metaphor:
tools. An Approach to Avoid Misleading Expectations,” in Pro-
ceedings of IFIP INTERACT'87: Human-Computer Interac-
Acknowledgments tion, 1987, pp. 555-560.
[16] H. Turtle, “Natural Language vs. Boolean Query Evaluation:
A Comparison of Retrieval Performance,” in Proceedings of
We would like to thank Albert Corbett, Bernita Myers, the Seventeenth Annual International ACM SIGIR Confer-
Barbara Pane, and the anonymous participants. This ence on Research and Development in Information Retrieval,
research is funded in part by the National Science Founda- Performance Evaluation, 1994, pp. 212-220.
tion under Grant No. IRI-9900452. Any opinions, findings [17] A. W. Biermann, B. W. Ballard, and A. H. Sigmon, “An
Experimental Study of Natural Language Programming,”
and conclusions or recommendations expressed in this International Journal of Man-Machine Studies, vol. 18, pp.
material are those of the authors and do not necessarily 71-87, 1983.
reflect those of the National Science Foundation. [18] A. Bruckman and E. Edwards, “Should We Leverage Natu-
ral-Language Knowledge? An Analysis of User Errors in a
Natural-Language-Style Programming Language,” in Pro-
References ceedings of the 1999 Conference on Human Factors in Com-
puting Systems. Pittsburgh, PA: ACM Press, 1999, pp. 207-
[1] J.-M. Hoc, “Do We Really Have Conditional Statements in 214.
Our Brains?,” in Studying the Novice Programmer, E. Solo- [19] J. Thomas and J. Gould, “A Psychological Study of Query by
way and J. C. Spohrer, Eds. Hillsdale, NJ: Lawrence Example,” in National Computer Conference, vol. 44. Ana-
Erlbaum Associates, 1989, pp. 179-90. heim, CA: AFIPS, 1975.
[2] C. Hildreth, “Intelligent Interfaces and Retrieval methods for [20] S. Jones, “Graphical Query Specification and Dynamic
Subject Search in Bibliographic Retrieval Systems,” in Result Previews for a Digital Library,” in Proceedings of the
Research, Education, Analysis & Design. Springfield, IL, ACM Symposium on User Interface Software and Technol-
1988. ogy, Enabling Architectures, 1998, pp. 143-151.
[3] Prograph CPX User Guide. Halifax, Nova Scotia: Pictorius [21] P. G. Anick, J. D. Brennan, R. A. Flynn, D. R. Hanssen, B.
Incorporated, 1996. Alvey, and J. M. Robbins, “A Direct Manipulation Interface
[4] J. G. Hays and M. M. Burnett, “A Guided Tour of Forms/3,” , for Boolean Information Retrieval via Natural Language
Oregon State University, Dept. of Computer Science Techni- Query,” in Proceedings of the Thirteenth Annual Interna-
cal Report 95-60-6, June 1995. tional ACM SIGIR Conference on Research and Develop-
[5] J. Tanaka, “The Perfect Search,” in Newsweek, vol. 134, ment in Information Retrieval, User Interfaces. Brussels,
1999, pp. 71. Belgium, 1990, pp. 135-150.
[6] L. A. Miller, “Natural Language Programming: Styles, Strat- [22] J. R. Cordy, “Hints on the Design of User Interface Language
egies, and Contrasts,” IBM Systems Journal, vol. 20, pp. 184- Features – Lessons from the Design of Turing,” in Languages
215, 1981. for Developing User Interfaces, B. A. Myers, Ed. Boston:
[7] J. F. Pane, C. A. Ratanamahatana, and B. A. Myers, “Study- Jones and Bartlett, 1992, pp. 329-340.
ing the Language and Structure in Non-Programmers’ Solu- [23] F. Modugno, A. T. Corbett, and B. A. Myers, “Evaluating
tions to Programming Problems,” International Journal on Program Representation in a Visual Shell,” in Empirical
Human-Computer Studies, to appear, 2000. Studies of Programmers: Sixth Workshop, W. D. Gray and D.
[8] D. Young and B. Shneiderman, “A Graphical Filter/Flow A. Boehm-Davis, Eds. Norwood, NJ: Ablex Publishing Cor-
Representation of Boolean Queries: A Prototype Implemen- poration, 1996, pp. 131-146.