Test Automation 20 Questions
Table 10.2 Questions to ask testing tool vendors.
1.
What does this tool do? What are the benefits of using this tool?
2.
How can this tool solve my problems? Which of my current problems can and can't this
tool help with? (You need to know what your own problems are.)
3.
Will you demonstrate your tool using our tests on our site? What environment is needed to
run this tool? (version numbers of operating systems, RAM, network configuration,
browser etc.) How much disk space does it use?
4.
What does this tool not do? What future enhancements are planned for the tool? (Note that
this also indicates current limitations of the tool.) What influence do tool users have on
future development of the tool?
5.
Can tests be re-run in debug mode within the tool? Can the tests be run as a background
task, so I can do other work while it is running the tests? (if this matters).
6.
What is the market share of this tool? How do you define market share? (Every vendor will
have a different definition, which makes their tool look best.)
7.
Why is this tool better than other similar tools?
8.
What proportion of tools sold are providing real benefits to the organisations which bought
them? For those which have not succeeded, why haven't they? What will I have to do to be
sure to succeed with this tool?
9.
What support is available, e.g. training, consultancy, help desk, Service Level Agreement
for resolution of problems, technical expertise in our area? What effort is needed in-house
by us to support the tool?
10. What test planning standards, test processes, test structuring scheme, etc. need to be in
place to gain real benefits from using this tool? (Ask if the vendor has read Fewster and
Graham.)
11. What features are included to ease the learning curve for the tool?
12. How do other sites usually work with the tool?
13. Is there a user group, and can I attend the next meeting?
14. Can you give me the names of reference sites, and can I meet with at least two users who
are achieving real benefits using this tool?
15. How many versions have been released in the past year? How is release management
handled? Do you release a defect list with the product?
16. How many known defects are there in the tool currently? (If they say none, be wary!)
17. What are your QA and test processes? What testing is done on the tool itself? (What does
the vendor know about testing in general?) Is the tool used to test itself? (This doesnt
necessary mean a lot.)
18. What kind of tailoring / customisation is possible for this tool? What extensions / add-ons /
in-house routines have other users built? (May indicate tool drawbacks or additional work
that you will need to do to achieve the best benefit from the tool.)
19. How does this tool integrate with other tools, e.g. other testing tools, project management
tools, configuration management tools, etc.? Ask specifically about tools you already have.
20. How long will it take to achieve real payback from this tool? Can you give me case
histories of financial benefits from other users? (This is surprisingly difficult - not many
users actually track it - make sure you plan to!) Can you help me estimate how much effort
will be involved in implementing this tool in my organisation?
Thanks for contributions from Paul Herzlich, Peter Oakley, Peter Herriott & Matthew Flack
Edited from Software Test Automation, Mark Fewster & Dorothy Graham, Addison Wesley, 1999
Page 1 of 2
Test Automation 20 Questions
Table 10.3 Questions to ask other testing tool users.
1.
How long have you been using this tool? Are you basically happy with it?
2.
How many copies/licenses do you have? How many users can be supported? What hardware and
software platforms are you using?
3.
How did you evaluate and decide on this tool? Which other tools did you consider when
purchasing this tool?
4.
How does the tool perform? Are there any bottlenecks?
5.
What is your impression of the vendor (commercial professionalism, on-going level of support,
documentation and training)?
6.
How many users actually use the tool? If the tool is not widely used, why not? E.g. technical
problems with the tool, lack of management support to go through the learning curve, problems
maintaining test scripts when software changes, resistance, lack of training, too much time
pressure, etc.
If the tool is currently shelf-ware, i.e. not used, skip to question 18.
7.
How much space does the tool-related test data take up, and how is this controlled? (This can be
a hidden cost.)
8.
Does the tool perform your entire set of tests or are there pre and post test activities that have to
be performed manually? (If so, why?)
9.
How easy is it to interpret the results of the automated tests?
10. What is your assessment of the quality of your own internal testing practices prior to acquiring
the tool? How did the use of the tool affect the quality of the testing?
11. Were there any non-technical problems in your organisation through introducing the tool, and
how were they overcome?
12. Is the tool now integrated into your work processes and standard procedures? If so, how much
effort and how long did this take?
13. Were there any other benefits or problems in using the tool, which were not anticipated?
14. Do you feel the tool gives you value for money?
15. Have you saved any money by using this tool? Can you quantify the savings or improvements?
16. How long did it take you to achieve real benefits? What are the critical factors for achieving
payback?
17. What improvements are needed in the tool? Is the vendor responsive to requested
enhancements?
18. What were your objectives or success criteria when buying the tool? (e.g. run more tests, more
consistent regression tests, improvement in meeting release deadlines, improved productivity,
capacity planning, performance assessment)
19. Have your objectives been achieved? Were they the right objectives? If not, what should they
have been?
20. If you were doing it over again now, would you still purchase this tool? What would you do
differently?
Thanks to Peter Oakley for additional ideas.
Edited from Software Test Automation, Mark Fewster & Dorothy Graham, Addison Wesley, 1999
Page 2 of 2