Test & Analyze

Testing Without a
You dont always need to wait for complete specifications to start your testing effort.
BY MICHAEL BOLTON

Map
SOMETIMES WHEN FACED WITH AN UNFAMILIAR APPLI-

cation and a directive to test it, it can feel as if youve been asked to chart a course through unknown waters. Youre not sure what youll find on the voyage, but youve got experience, some notions of what to look for, and an idea of where youd like to go. If you see something surprising or unexpected, youll take note of it. Those are exploratory skills, and sometimes theyre all you need to begin. Some testing advocates suggest that you should never test without a complete written specification. Thats unrealistic advice. First, there are plenty of contexts in which you may be asked to test without a formal specification: when youre evaluating a piece of commercial software to see if it suits your companys needs; when your organizations production code is so old and so heavily patched that the original specification would be meaninglesseven if it could be found; or when work-

[Link]

JANUARY 2005

BETTER SOFTWARE

25

Test & Analyze

ing on Agile projects. Second, completeness is entirely dependent upon perspective and context. Even so-called complete specifications contain much that is implicit. In fact, the more explicit the document, the longer and more ponderous it isand the less likely that someone will read it in its entirety. Finally, certain kinds of specifications might be supplied to you through some means other than a formal documentconversation, email, or your own inferences. A quick meeting with the boss, combined with your skills at identifying value, risks, and problems, might give you all the charter you need to begin supplying useful information to management. An effective tester can always obtain valuable information by exploration, even if the sole purpose of exploring is to gather information for a more detailed test strategy.

History: The features or functions current behavior should be consistent with its past behavior, assuming that there is no good reason for it to change. This heuristic is especially useful when testing a new version of an existing program. Image: The products look and behavior

The Product itself: The behavior of a given function should be consistent with the behavior of comparable functions or functional patterns within the same product unless there is a specific reason for it not to be consistent.
Purpose: The behavior of a feature, func-

should be consistent with an image that the development organization wants to project to its customers or to its internal users. A product that looks shoddy often is shoddy.
Comparable products: We may be able to use other products as a rough, de facto standard against which our own can be compared. Claims: The product should behave the

tion, or product should be consistent with its apparent purpose.

Statutes: The product should behave in compliance with legal or regulatory requirements.

Prepare for theJourney

When the explorers of old set sail for uncharted waters, they did not set out unequipped. They knew the sun and the stars, and they carried tools such as compasses, sextants, and clocks, not only for navigation but also for mapmaking. More importantly, they ventured out with extensive background knowledge and skills, which included deduced reckoning, celestial navigation, and horse sense. For exploratory testers, knowledge is often represented in two terms that youll use as a testing expert: oracles and heuristics. An oracle is a principle or mechanism by which we can tell if the software is working according to someones criteria; an oracle provides a right answeraccording to somebody. A heuristic is a provisional and fallible guide by which we investigate or solve a problem; its also a method by which learning takes place as a result of discoveries informed by exploration. James Bach has given us a helpful set of oracle heuristics, to which Ive added one, in the form of this mnemonic: HICCUPPS. The idea is that a product should be consistent with:
26
BETTER SOFTWARE JANUARY 2005

way some document, artifact, or person says it should. The claim might be made in a specification, a Help file, an advertisement, an email message, or a hallway conversation, and the person or agency making the claim has to carry some degree of authority to make the claim stick.
Users expectations: A feature or func-

Remember: Heuristics are guidelines, not edicts; theyre fallible. They arent universalthere are plenty of other ways by which we can decide whether a product is acceptable or unacceptable. There is some conceptual overlap between some of the pointsbut to an explorer, features of the new territory overlap, too.

Discover

Explore and

tion should behave in a way that is consistent with our understanding of what users want, as well as with their reasonable expectations.

Armed with these tools, lets imagine that Im working for a small start-up, and that my company is going to be releasing its software on CDs. The company doesnt have a lot of money, and every dollar

Figure 1: The main screen of Nero Cover Designer, shown immediately after startup.

[Link]

Test & Analyze

counts, so my boss has asked me to evaluate the program that comes with the CD burner: the popular Nero CD recording software. Printing CD covers is a requirement, so she asks me to have a look at Neros Cover Designer, a subset of the CD recording package, to see whether the company should use it. Instead of writing up an elaborate test plan, Ill just plunge in, quit when I have more information, and then (and only then) make some decisions about how to proceed. Figure 1 shows what I see on the main screen just after I start the program. Yogi Berra was right: you can observe a lot just by looking. One of the first things I note is that there appears to be a default setting for the font: Arial for the face and 16 for the point size. My boss doesnt need to tell me to test fonts; I have the consistency with purpose heuristic in my head to tell me that, if the task is to print CD covers, graphics and textand therefore fontsare likely to be part of that task. Do I care about the accuracy of the point sizes and color depth of the graphics? Maybe, but I can ask about those things later, after Ive run some other tests. I make a note to ask questions about accuracy and move on. Im going to need to put something on my CD cover, so I choose to insert a new object. I click Object, Insert, Text Box. Then I double-click the new object that appears, and the dialog shown in Figure 2 pops up. Something already feels funny. In the new font properties area, the name of the font has disappeared and the point size now appears to be 8. In accordance with the consistency within the product heuristic, one would think that the font properties should be the same on both the main screen and the new dialog. Do we have our first bug? Id say yes, but perhaps we should do some checking. Ill note it.

about what the program can do. So lets press the F1 key. Why F1? Windows users have a heuristic that F1 should trigger the Help file, courtesy of the Windows User Interface Guidelines. (See this issues StickyNotes for more information.) Cover Designer is a Windows program, and a programs behavior should be consistent with programs like it. If there is a compelling reason for your program to behave differently, then it might be worthwhile to depart from de facto UI standards. Otherwise, consistency with other products is a favor to the user, saving her the time and trouble of learning a different way of doing things. When we press F1, a tooltip appears at the hot spot on the mouse pointer:

Hmmm . . . theres nothing here that looks like a reference to text. In fact, theres nothing here that seems to refer to anything in the Cover Designer. Lets go to the Index and look for the words Cover Designer. All I see is Help for the Nero CD-ROM burning software. That means that either there is no Help for Cover Designer, or if there is a Help file, its not coming up from inside Cover Designer. Thats a problem based on the consistency with user expectations heuristica user could reasonably expect that a Help file summoned from within an application should be that applications Help file. Well, it seems as though Ill have to give up on Help, and thats noteworthy. Lets return to the first presumed bug and do some more investigation. I dont know exactly what my company is going to put on the CD cover, but the specifics dont matter, so Ill put in some text that

Investigate

NewFindings

We dont have a specification, but Windows programs typically come with a Help file. A program should be consistent with claims it makes about itself, and the Help file is usually full of claims

The tooltip says, Lets you modify the contents of the text. Shouldnt that say, Lets you modify the contents of the text box? That might be a quibble, but in some contexts Id be willing to call it a second bug. If this were my program, I might find the imprecise English a little embarrassing, which would violate the consistency with Figure 2: In the textbox properties box, the font name is image heuristic: A pro- missing. Could this be a bug? gram should be consistent with the image that a company wishes to present. And another thing: shouldnt F1 display the Help dialog instead of a tooltip? By Windows conventions, a tooltip should appear when you hover over an item with the mouse. Ill write a couple more notes about these Help issues; they might represent another bug or two. I want to open the Help file. Theres another way to do thatI can click the Help button to open it, click the Find tab, and find all the references to text box. (See Figure 3.) Figure 3: The Help dialog isnt helpful.
[Link]
JANUARY 2005 BETTER SOFTWARE

27

Test & Analyze

tations heuristic suggests that typing some text should not change the selected font unless Ive asked to do so. Even though this rectifies the problem I noted as the first bug, it does so in a way that gives me pause, and this is arguably yet another bug; Ill write that down. Ill highlight the text that Ive entered and choose a different typeface and size; again, specifics dont matter. Ill select Comic Sans MS and 26 points. (See Figure 5.) Then, Ill press OK to close the dialog. Now, Ill immediately click the text box to open the dialog again. (See Figure 6.) Presto! The typeface is back to Arial, and the size is 16. This violates the consistency Figure 4: After typing in the text box, the font name with purpose heuristic. appears. Surely the purpose of pressing OK (rather than Cancel) on an object is to retain the properties that Ive selected until I explicitly change them: A feature or function should be consistent with its apparent purpose. Another bug to note. Note that in this dialog there are tabs for pen, brush, and image as well as text. I try this out, and I find that Figure 5: The text is highlighted, and the typeface and size every time I try to reare changed. open the text box to modify one of these attributes, the font information disappearsan inconvenience and an annoyance and, even without a specification, manifestly a bug. Im disappointed because I seem to remember this feature working in a previous version of Nero Cover Designer. Thats a violation of the consistency with history heurisFigure 6: The typeface and size did not save with the text. reflects the way that I might use the program. (See Figure 4.) When I type Beatles Compilation in fact, immediately after striking the B keythe font size of 8 turns to 16, and the formerly blank drop-down for the typeface is suddenly set to Arial. The consistency with the users reasonable expec28
BETTER SOFTWARE JANUARY 2005

tic: A program should behave in a manner consistent with its own history or previous versions of the product. The bugs in this program have been pretty easy to find, and this last one is so troublesome that I have some grave doubts about the rest of the program. After five minutes of testing, Ill be able to tell the boss that she should not rely on this product to produce the companys CD coversand thank goodness I didnt waste time preparing an elaborate test plan based on some incomplete specification that some programmer apparently didnt read.

The Journey

Ends

This was a particularly egregious example, but if youre still adamant that you need a written specification before you can begin testing, consider what youve just read in the context of two questions. First, did we need a written specification to provide important, credible, timely information to management? Second, would the cost of researching and preparing a specificationand waiting for it to be preparedadd significantly to the value of our report? As you can see, in many contexts its not only perfectly OK but also entirely desirable to test without using a specification. My background knowledge of GUIs on Windows helped me recognize several problems, and my ability to put myself in a users shoes helped too. A few minutes of exploration, wandering through one feature of the program and looking through the spyglass of those exploratory testing heuristics, has helped me not only to find bugs but also to identify credibly why I think theyre bugs, even though I had nothing like a complete, formal, written specification. Although I didnt have a map, I was certainly able to explore and compile one along the way. {end} Michael Bolton lives in Toronto and teaches heuristics and exploratory testing in Canada, the United States, and other countries as part of James Bachs Rapid Software Testing course. Contact Michael at mb@[Link].

[Link]