9 Evaluating the Results tebe Design 4 [ se | x Development Implementation eeea ane The Evaluation Step of the ADDIE Model Check What You Know The HR director told Boris and Marietta that he and the CEO were pleased with the e-mail program and looking forward to its launch with the first group of learners. “There’s one more thing,” he said. (Continued) Designing and Deolping Teaining Programs: efor Essential Guise to Training Basis Copyright ©2010 by John Wiley & Sons, ne. Reproduced by permission of Plier, an Imprint of Wiley wwverPeifer om, 237“Once we get the program up and running, the CEO will want to know how well it's working, so welll need a way to evaluate the results.” “No problem,’ they replied.’ We've already been thinking about the evaluation.” How do you think Boris and Marietta will go about evaluating the e-mail program? What questions should they ask? What criteria should they use? Looking back on my school years, [remember tests, tests, and more tests. Trying to figure out math problems and guessing at which circles to fill in on the electronic scoring sheets. Frantically trying to finish “blue book” essays before the time was up. Reports. Presentations. All in the service of trying to prove that I had learned something or was able to do something so I could obtain a coveted grade and move on. I suspect that most of those tests were conducted to find out about me— how carefully I'd listened during class, and how well I'd studied—not about how well the course itself helped me to learn. I know that things are changing in education and that there is now more focus on evaluating the teacher’s performance, as well as the students’. But what's the function of evaluation when it comes to training? Who and what are being evaluated, and why? That's what this chapter is about. In this chapter, you'll learn: The function of the program evaluation The questions an evaluation needs to answer How to conduct an evaluation Devgan Dewi Dining Prgms: ie Eset dest Paning Basis ‘Copyright © 2010 by John Wiley & Son Inc Reproduced by permission of Piifr, an Imprint of Wiley. wwwPieffercom 238 Designing and Developing Training ProgramsThe various levels of evaluation How to determine what levels to use Evaluation doesn’t deserve to be listed last in the ADDIE model because it takes place in every element and surrounds the instructional design process. Evaluation is a constant guard at the gate of failure. Chuck Hodell, ISD From the Ground Up: ‘A No-Nonsense Approach to Instructional Design (2nd ed.) 1. The Function of Evaluation Check What You Know What is the basis for evaluating the success of a training program? What are the criteria for measuring success? When are those criteria established? As the quote above explains, don’t be misled by the placement of “Evaluation” at the end of the ADDIE acronym. Evaluation starts much earlier in the process and continues after the training has been completed. It’s an Designing and Deting Ting Programs: fife Eesti Gules to Dining asks. Copyright © 2010 by John Wiley & Sons ne Reproduce by permision of Peer, an imprint Wiley. wwe Pefercom Evaluating the Results : 239ongoing question: How will /do we know whether this program has been successful? What changes can we make to improve it? ‘The basis for evaluation is established when you identify the desired outcome—what the program is intended to achieve—and write the learning objectives that establish the criteria for measuring success. As the program is. designed and developed, a continual review of the delivery system, content, structure, learning activities, and materials helps you keep the goal in mind while responding to new information and changes in the situation. Testing the program before it is launched provides a check to see whether it seems to work as designed. Finally, post-training evaluations measure how well the program is working and identify any changes that may need to be made. Evaluation provides critical information that organizations need to decide whether to continue a program and, if so, whether to change it in any way. ‘The evaluation process takes time, attention, and expertise; for those reasons, it is often neglected, done haphazardly, or skipped altogether. But without evaluation, a program can take on a life of its own, continuing to run month after month, or year after year, without anyone ever stopping to ask whether it actually helps people learn, whether they are using what they learned, or whether there is any evidence that the training has resulted in a change that ‘was worth the organization's investment, 2. Questions an Evaluation Needs to Answer Check What You Know Suppose you were designing the evaluation for the job application training program that was used as an example earlier in this book. What are some of the questions that the evaluation would need to answer? ‘Dasgnig ant Decioping Tring Programs: Pfeifer Essential Guides to Taning Basis Copyright© 2010 by Joka Wiley & Sons, In Reproduced by permission of Pfeifer, an imprint of Wily. www [Link] 240 Designing and Developing Training Programs‘When a doctor evaluates the success of a treatment, she needs to ask lots of questions: To what degree has a change in the patient's condition taken place? To what degree have the symptoms been alleviated? Is the change likely to be temporary or permanent? Was the change due to the treatment or to something else? If the condition still exists, why? What side-effects, if any, have there been? Are any changes needed in the treatment regime? Should treatment be continued? Evaluating the success of training also requires asking a lot of questions, such as: How did the participants feel about the training? Did they find the program interesting, enjoyable, and useful? What did participants learn? How well did participants achieve the learning objectives? To what degree has change taken place in the learners’ performance, behavior, and/or attitudes? What unforeseen effects, if any, has training had? To what degree has the expected outcome been achieved? How well has the outcome of the program met the organization’s business goals? Does the program provide a reasonable return on the organization's investment? If the participants have not achieved the learning objectives, and/or the expected outcome has not been achieved, what are the reasons? Should this program be repeated? If so, what changes, if any, are needed to improve it? Just as the doctor collects information to evaluate the success of treatment by observing the patient, asking questions, and running tests, evaluating the success of a training program requires gathering information by asking questions, observing behavior and the results of behavior, and perhaps, by administering tests. signing and Developing Taning Programs: Peifer Essential Guest ning Bass, Copyright © 2010 by Joka Wiley & Sons, ine. Reproduced by permission of Pfr, an Imprint of Wiley worwPfeffercom Evaluating the Results 241[Link] to Conduct an Evaluation There are many different methodologies for evaluating training success, and many books have been written on the evaluation process. Perhaps the most commonly used methodology is some version of Professor Donald Kirkpatrick’s Four Levels of Evaluation, described in his book, Evaluating Training Programs, which is now in its third edition. This method is based on four key questio The Levels of Evaluation The first four levels (Kirkpatrick): + Level 1: What participants thought about the training program Level 2: What participants learned from the training program Level 3: The extent to which participants are applying what they learned Level 4: The degree to which the program achieved the desired results ‘The fifth level (Phillips): + Level 5: The organization's return on its investment in the program How satisfied were the participants? Did they find the program interesting? Useful? Enjoyable? How much did participants learn? To what extent did they gain knowledge, learn new skills, and/or change their attitudes as a result of the training? What change in behavior occurred as a result of the program? To what degree are participants applying what they learned? What were the results of training? ‘To what degree was the desired outcome achieved? Dr. Jack Phillips, a well-respected author and expert on accountability, measurement, and evaluation, has added a fifth item to evaluate that asks, in essence: Were the benefits of this training worth the cost? What was the organization’s return on its investment? Designing an Deceoping Training Programs: Refer Essential Gusts to Taming Basics Copyright © 2010 by John Wiley & Sons, Inc. [Reproduced by permission of Pieife an Imprint of Wiley. wwwPfelfercom 242 Designing and Developing Training ProgramsCheck What You Know Which of the statements below are true? 1, ____ Positive participant responses to training are the best indication of how well it has succeeded. 2. ____People like to be asked for their feedback, but their responses are usually too subjective to be of much use. 3, ___ Participant responses are useful for identifying changes to improve training for future groups. 4, ___ The most common way of evaluating training is to ask participants how well they liked the trainer. Level 1:Participants’ Reaction to Training A department store where I shop often hands out little questionnaires entitled “How Was the Service Today?” along with the credit card receipt. A few days after I purchased a new computer printer, I received an e-mail that promised to enter my name in a drawing if would fill out a survey. It seems as if all the people I do business with want to know how I feel about my experience with their service and products. Those questionnaires and surveys are what training professionals commonly call “smile sheets.” On the Kirkpatrick model, they are Level 1 evaluations, which answer the question, “How satisfied were participants with their training experience?” Level 1 evaluations are typically conducted by asking participants to fill out a questionnaire or engage in an informal discussion (What worked? What didn’t?”) at the end of the training session or to complete an online survey right after training is over. They are more rarely conducted by interviewing participants in person or on the phone. The questions focus on how well participants thought the program met their needs, how they thought the trainer did, how well the time was used, how well the activities helped them learn, and what changes might improve the program for future groups. Designing and Dewloping Tuning Programs: elf Esential Gude to Training Bass Copyright © 2010 by John Wiley &eSons, Ine [Reproduced by permission of Pfeifer an Imprint of Wiley www Pfeifer com Evaluating the Results 243Level 1 evaluations provide helpful information, but it’s important to remember that they are very subjective. Participants are usually responding Level 1 Questions Whether you use a questionnaire or sutvey, group discussion, or individual interviews to gather information fora Level 1 evaluation, you will probably be trying to answer these kinds of questions: How well did participants think that the program met their needs? IF training was delivered in a workshop, how did the trainer do? How well did the activities help them learn? Did any time seem to be wasted? Did any information or activities appear to be unnecessary? + Did anything seem to be missing? What changes might improve the program? more to the trainer’s style and personality than to the substance of the program, and they may be reluctant to make negative comments because they do not want to hurt the trainer's feelings. It's like asking visitors how they like the way you decorated your office—those who think it looks great will say so enthusiastically, while those who think you have questionable taste are likely to smile, say something noncommittal, and keep their real opinions to themselves. And it's important to remember that, although satisfaction with the learning experience can affect the motivation to learn, people can have a great time at a workshop without learning much of anything. It's not surprising that Level 1 evaluations are by far the most common methods of assessing the success of training—they are the easiest and least costly to administer. ‘Measuring participants’ responses to the training experience is only part of the process, however. To truly evaluate a program’s success, you need to know what participants actually learned, the degree to which they are applying what they learned, and the business results of the training program. The sample “smile sheet” evaluation at the end of the chapter is typical of those that are routinely administered right after a workshop has been completed. Similar questions can be asked electronically, using online survey programs such as SurveyMonkey. Designing and Developing Taning Programs: Fer Essent Gusts to Tretning Bases. (Copyright© 2010 by John Wiley & Sens, nc Reproduced by permission of Pfs, an Impeint of Wiley: www: Peilfercom 244 Designing and Developing Training ProgramsCheck What You Know Which statements accurately describe the purpose of a Level 2 evaluation? 1, ___To find out what participants learned in the workshop. 2, ___Tofind out what people have told others about the training. 3. ___To find out how well people are able to apply what they learned ‘on the job. 4, ___ Tofind out the extent to which participants achieved the learning objectives. Level 2: Participants’ Learning Level 2 evaluations seek to answer the questions, “What have participants learned? To what extent have they achieved the learning objectives?” You learned earlier that well-written learning objectives contain within them the criteria for measuring success. Here's an analogy: Suppose you order a suit froma tailor in another city. You provide the tailor with certain criteria: style, measurements, fabric, and quality of workmanship. The finished suit can then be evaluated to see how well it meets those criteria. Similarly, if a learning objective specifies that a loan officer be able to list with 95 percent accuracy the questions that need to be asked of a loan applicant, the loan officer's learning can be evaluated by asking her to list those questions. There are many methods for evaluating how well someone has learned, including written and performance tests; questionnaires, assessments, and surveys; observation of performance and/or the results of performance; and interviews, The method or methods you use depend on what you are measuring. Awritten test might be the best way to evaluate how well someone has learned facts; a performance test might be an effective way of determining whether someone can follow a procedure; and a survey might demonstrate whether someone has changed his or her attitudes. Desging and Deoing ining Programs: fer Esl Guide to Ting Bass Copyright ©2010 by Joba Wiley & Sons, nc. Reproduced by permission of Pieifer, an imprint of Wiley. wwwu:Plifercom, Evaluating the Results 245Level 2 evaluations are more time-consuming and costly to develop and administer than Level 1 evaluations. To determine what change has taken place as a result of training, you need a baseline: what people knew or could do, or what their attitudes were, before training. Establishing that baseline usually requires administering some kind of a pretest or conducting some kind of pre-training assessment, which is not always easy to do. Performance tests, which require observing people perform in a real or simulated situation, are necessarily subjective and may require a great deal of time. And all tests, assessments, and interview questions must be carefully designed to produce valid results. But perhaps the most important thing to remember about Level 2 evaluations is that they measure what people have learned—not necessarily the extent to which they can or will apply what they learn. Being able to describe the steps involved in solving a problem doesn’t mean that someone can actually use those steps to solve a problem. Being able to give performance feedback in a simulated situation doesn’t necessarily mean that the learner will be able to give that same kind of performance feedback to an employee when he is back on the job. Check What You Know Check the statements that accurately describe Level 3 evaluations. 1. The primary purpose of a Level 3 evaluation is to measure the extent to which participants: __ remember what they learned are applying what they learned 2. Acommon reason that Level 3 evaluations are seldom done is that: ___ they require time-intensive activities ___ participants are seldom willing to cooperate 3. Results of a Level 3 evaluation that indicate no significant change in the learners’ performance: ___Ieads to the conclusion that training was unsuccessful in helping people learn ___could mean that people have had no opportunity to use what they learned Designing and Developing Trining Prgreme: Pfifr Esvertal Cudas to Tsing Basis Copyright © 2010 by John Wiley &e Son, Ine Reproduced by permission of Plier, an Imprint of Wiley. woewPiefiercom 246 Designing and Developing Training ProgramsLevel 3: Participants’ Behavior One of the most important questions to answer about training is, "To what extent are people applying what they learned?” Remember, the primary purpose of training is to change or improve performance in the workplace. So it stands to reason that the most valid evaluation method is an assessment of how well participants apply what they learned. This level of evaluation is rarely carried out because it requires time- intensive activities such as directly observing on-the-job performance or the results of performance; interviewing managers, colleagues, customers, and others; and/or detailed surveys. Unlike the other evaluation levels, which can be done right after training has been Questions for Level 5 Evaluations + To what extent are participants applying what they learned? What changes have there been in their performance? Where has their behavior shown the most improvement? The least improvement? To what extent have participants had opportunities to use what they learned? What, if any, obstacles have kept them from using the learning? What, if any, changes have there been in the situation that might be impacting the participants’ ability to apply the learning? completed, Level 3 evaluations can be done only after the participants have had the time—and opportunity—to apply what they learned. To complicate matters, if it appears that people have not changed or improved their performance, it can be very difficult to identify the reasons. They may not have had an opportunity to apply what they learned. They might not want to change their behavior. Their managers might not support changes. They might lack the necessary equipment. Changes in organizational structure or procedures might mean they no longer need to use what they learned. For example, the suit the tailor made you might fit just fine and look great on you, but you never wear it because you've left your corporate job to become an artist. If a training project is to have long-lasting value, then it should be connected to specific business results that you can measure. .. . Training projects can’t solve every business problem, Designing and Developing Tning Programs: Pele Essential Gies to Training Bass Copyright© 2010 by ohn Wiley & Sons, Ine Reproduced by permission of Pir an Imprint of Wiley worwPleiter om Evaluating the Results 247but a good training project should be able to articulate its goals in ways that can be measured. “The ADDIE Instructional Design Model,” www [Link] Check What You Know Which statements are true of Level 4 evaluations? i If the budget allows, you should doa Level 4 evaluation of every training program. 2. ___ The purpose of a Level 4 evaluation is to measure the degree to which the training achieved the desired outcome identified at the beginning of the project. 3, ____Level 4 evaluations are complicated because itis difficult to separate out which results were due to training and which to other factors. 4, ___You can't plan a Level 4 evaluation until well after the trai program has been completed. Designing and Developing Toning Programs: fer Essential Cds to Trning Basis Copsright © 2010 by Joka Wiley &e Sons, Ine ‘Reproduced by permission of Pfeifer, an Imprint of Wiley. [Link] 248 Designing and Developing Training Program:Level 4: Results of Training The design of a training program begins with an examination of the reason that training is needed—sales are slow, there are too many accidents or product defects, customers are complaining, valuable employees are leaving, people are not using meeting time productively. Level 4 evaluations measure how well the training achieved the desired outcome that was identified at the beginning of the project. To what degree has the training improved the company’s ability to retain valuable employees or customers? To what extent has employee productivity increased? Has the safety record improved? Are the desired sales targets being met? Are employees more satisfied with their jobs? There are good reasons why Level 4 evaluations are seldom done or done more or less haphazardly. It's difficult to collect information—even to determine ‘what information should be gathered; there is seldom a clear baseline against ‘which to compare the results; and it’s usually very hard to separate out which results, ot lack of results, were due to training and which to other factors. Still, today’s organizations are more and more focused on results. They want to be sure that they are using scarce resources where they will do the most good, and training programs increasingly need to provide evidence of their value. As instructional designers, we can’t wait to start thinking about how ‘we can measure results until after the program is over. It's too late by then to. gather the baseline information we'll need. We must think about all the stages of evaluation right from the beginning of our training project by asking, and asking again: “How will we know whether this program has been a success?” Level 5: Return on Investment (ROI) Today's organizations increasingly examine the impact of every activity on business results. But when it comes to training, it can be very difficult to evaluate the actual return the organization receives from its investment. Evaluating the ROI requires calculating all the costs of designing, delivering, and following up the training program—including the salaries and benefits of everyone involved at any stage; consultant fees; facility rental or usage costs; equipment, materials, and supplies; and the cost of participants’ time away from their jobs—and then quantifying the program benefits, such as how much time or money is saved by increasing productivity, reducing errors or accidents, retaining customers, and reducing turnover. In fact, according to Bob Pike, evaluating the ROT is not suitable for all learning programs, and only 5 to 10 percent of all programs are evaluated at this level. Designing and Developing Tuning Programs: Peer Essent Guides to Training Bass, Copyright © 2010 by John Wiley & Sons, Ine Reproduced by permission of Peer, an imprint of Wiley. www [Link] Evaluating the Results 249Check What You Know Which of the following statements about deciding which level of evaluation to use are accurate? 1. ____An important factor to consider when deciding whether to evaluate a program beyond Level 1 is whether there is enough information to provide a baseline against which to measure the results. 2, ___One way to determine which levels of evaluation to use is to figure out whether the information provided by the evaluation is likely to be worth the cost of collecting and analyzing it. 3. ___ It’s usually worthwhile to conduct at least the first three levels of, evaluation for every program,even if the program is to be conducted for only one group. 4, ___ Few trainers have the expertise needed to conduct all five levels of evaluation. How to Determine Which Level(s) to Use As mentioned earlier, the evaluation process can be lengthy and costly, and it often requires special expertise. At the same time, the information provided by a thorough evaluation can save the organization money in the long run. It’s a matter of deciding which level or levels of evaluation are necessary and cost- effective in a given situation. To make that decision, the organization needs to consider these kinds of questions: Whether the program is likely to be repeated. Obviously, if the program is “special occasion” training that will be delivered only once to only one group of participants, extensive evaluation is unlikely to be worth the time and trouble. A Level 1 evaluation can probably provide enough useful information to help improve the design and delivery of future programs on other topics. But for programs that are to be ongoing or repeated with large numbers of people, more extensive evaluations may be important to ensure that the training is achieving its goals, not wasting everybody's time. estgong and Decelopong Trang Programs: Peer Eset Gules to Tating Basics Copyright ©2010 by John Wiley &eSons, rc Reproduced by permission of Pieifer, an imprint of Wiley. www Peiffercom 250 Designing and Developing Training ProgramsQuestions for Determining Which Level to Use + Isthe program likely to be repeated? How often? For how many people? + Isthere a baseline against which to measure what people have learned? + Isthe information provided by the evaluation likely to be worth the cost? + Isthere sufficient expertise to conduct evaluations beyond Level 1? Boris and Marietta’s Progress Boris and Marietta were ready to design the evaluation of their e-mail program because they had started thinking about it way backat the beginning of the design process. They reviewed the criteria for success—the learning objectives the program was designed to help participants achieve—and discussed ways in which they could gather the information they would need. “Let's be clear about what we need to know,” Marietta said."| think there are afew key questions: Have the learners’ attitudes toward e-mail changed? Are they using their e-mail time more productively? Writing e-mail that meets the criteria they learned in the workshop?” “Those are good questions,” Boris said."And we should also find out how they felt about the workshop itself. Did the activities work for them? Did they think it covered the right content? Have the right structure? How about the trainer’s presentation?” “We can use an online survey the day after the workshop to get that, information,” Marietta said.”But how are we going to know whether their attitudes and skills have changed, whether they are using their e-mail time more productively, and whether their e-mail now meets the criteria?” Boris thought for a moment."We'll need some kind of baseline. How about sending out a questionnaire before the workshop, asking them about their use of e-mail, and then sending the same questionnaire four weeks after training. We can compare the results.” “And when we send out the questionnaires, we could ask them to send us a few samples of the e-mail they send to clients. You know—before and after.A comparison of the writing samples would give us a good idea about whether they are applying what they learned.” Desig and Deng Ting Progra Pfr xen Gus o Ting Bas Copyright© 2010 by Jo Wey & Sons Ine Reproduced by prison of ei an impo Wey wor ecco Evaluating the Results 251Whether there is a baseline against which to measure the results of training. To determine the extent to which a situation has changed as a result of training, you need detailed information about what it was before training: what people knew, how they were behaving, what attitudes they had, and so on. Baseline information comes from statistics, surveys, tests and assessments, and other sources. That information can be difficult to gather, but it’s necessary for accurately determining how successful the program has been. Whether the information provided by the evaluation is worth the cost. It costs essentially nothing to conduct a Level 1 evaluation; if the baseline information exists, a Level 2 evaluation may be relatively easy and inexpensive, But the higher levels of evaluation may be worth doing only if the program is to be delivered to large numbers of people, is vital to support the organization's business goals or strategic initiatives, and/or is likely to have significant business impacts. Whether there is sufficient expertise to conduct evaluations beyond Level 1. Even Level 2 evaluations may require expertise in developing and administering valid assessments, and conducting evaluations at the higher levels may be beyond the capabilities of an organization’s trainers. For those levels of evaluation, an organization might need to hire an outside consultant who has the necessary expertise. Quick Quiz List the three to five key learning points from this chapter that will be most helpful to you. Designing and Developing Txbing Programs: Peer Essential Gules to Training Bass. Copyright © 2010 by John Wiley de Sons, Inc Reproduced by permission of Plier, an Imprint of Wiley urn Peiffercom 252 Designing and Developing Training ProgramsWhat's Next? You have learned the basic principles that guide the design and development ofa training program. In the last chapter of this book, you will learn about the ways in which those principles are applied to the rapidly growing field of distance learning. Apply What You Learn Which evaluation method or methods would be best for your training program? Why? What resources would be needed? Level 1, participants’ responses to training Level 2, what participants learned from the training program Level 3, the extent to which participants are applying what they learned Level 4, the degree to which the program achieved the desired results RO, the return the organization received on its investment Your rationale: The resources that would be needed: Designing and Developing ening Prograns: for Esontial Cues to Training Basis Copyright © 2010by John Wiley &Sons, In. Reproduced by permission of Pfr an Imprint of Wiley. [Link] Evaluating the Results 253Answers to Exercises Check What You Know Which of the statements below are true? 1. __ Positive participant responses to training are the best indication of how well it has succeeded. 2. __ People like to be asked for their feedback, but their responses are Usually too subjective to be of much use. _X_Participant responses are useful for identifying changes to improve training for future groups. 4, _X_The most common way of evaluating training is to ask participants how well they liked the trainer. Check What You Know Which statements accurately describe the purpose of a Level 2 evaluation? 1. _X_To find out what participants learned in the workshop. 2. __Tofind out what people have told others about the training. 3. __Tofind out how well people are able to apply what they learned on the job. 4, _X_To find out the extent to which participants achieved the learning objectives. Designing and Developing Training Programs: Peer Essential Guest Training Basics. ‘Copyright © 2010 by John Wiley & Sons nc Reproduced by permission of Pile an Imprint of Wiley. www [Link] 254 Designing and Developing Training Program:Check What You Know Check the statements that accurately describe Level 3 evaluations. 1. The primary purpose of a Level 3 evaluation is to measure the extent to which participants: __ remember what they learned _X__ are applying what they learned 2. Acommon reason that Level 3 evaluations are seldom done is that: _X__ they require time-intensive activities ___ participants are seldom willing to cooperate 3. Results of a Level 3 evaluation that indicate no significant change in the learners’ performance: _—__ leads to the conclusion that training was unsuccessful in helping people learn _X__ could mean that people have had no opportunity to use what they learned Check What You Know Which statements are true of Level 4 evaluations? 1. ___Ifthe budget allows, you should do a Level 4 evaluation of every training program. 2. _X_ The purpose of a Level 4 evaluation is to measure the degree to which the training achieved the desired outcome identified at the beginning of the project. 3. _X__Level 4 evaluations are complicated because its difficult to separate ‘out which results were due to training and which to other factors. 4, ___You can't plan a Level 4 evaluation until well after the training program has been completed. Designing and Decloping Tining Programs: eer Essential Guites to raining Basis ‘Copyright © 2010by Joha Wiley & Sons, Inc. Reproduced by permission of Pier, an Imprint of Wiley. worwePitfercom Evaluating the Results 255Check What You Know Which of the following statements about deciding which level of evaluation to use are accurate? 1. _X_An important factor to consider when deciding whether to evaluate a program beyond Level 1 is whether there is enough information to provide a baseline against which to measure the results. 2. _X_Oneway to determine which levels of evaluation to use is to figure out whether the information provided by the evaluation is likely to be worth the cost of collecting and analyzing it. __ It usually worthwhile to conduct at least the first three levels of evaluation for every program, even if the program is to be conducted for only one group. 4, _X_Few trainers have the expertise needed to conduct all five levels of evaluation. Designing and Developing Tranny Programs: PeifrExsertal Gules to Ting Basic. Copyright © 2010by John Wily & Sone, Inc. Reproduced by permission of Pieifer an Imprint of Wily. www [Link] 256 Designing and Developing Training ProgramsSample “Smile Sheet” Evaluation Title of program: Date(s) of program: Trainer(s): Please check your primary reason for attending this program: Increase my knowledge/skills Required by my company or manager Other (please specify): Overall, how would you rate this program? Please circle the appropriate number: Excellent Average Poor 5 4 3 a 1 The pace of this program was (please check one): O too fast just right Otoo slow The program provided (please check one): too much information O the right amount of information not enough information How relevant was the program to your work? Very relevant Not relevant 5 4 3 2 1 Designing and Developing Teting Programs: Pfr Essential Cues to Training Basis. ‘Copyright © 2010 by John Wile & Sons, In: Reproduced by permission of Plier an Imprint of Wie: wwrwPflflercom Evaluating the Results 257How useful will what you learned in the program be? Very useful Not useful 5 4 3 2 1 How well did this program meet your expectations? Very well Not well 5 4 3 2 1 Please rate the trainer on the following: Agree Do Not Agree The trainer was well prepared. 5 4 3 2 4 The trainer was knowledgeable. Gata too tn The trainer was responsive to participants’ needs, questions, and concerns. S 2 5.2 4 The trainer presented the material clearly. bp 4a 2a The trainer kept the workshop on track. 5 43 2 1 The trainer gave everyone opportunities to participate. bearer Please give us your specific comments below: What did you like most about the program? What did you find most valuable? Desgning an Deveopong Taming Programs: Rfifer Essential Guides to Training Basis. Copyright© 2010 by Joha Wiley & Sons, Inc [Reproduced by permission of Pfeifer, an Imprint of Wiley. [Link] 258 Designing and Developing Training ProgramsWhat changes would you suggest to improve the program? Other: Designing and Developing Tning Programs: Defer Essential Gules to raining Basis, (Copyright© 2010 by John Wiley & Sons, Ine. Reproduced by permission of Pfr an Imprint of Wiley. wwwwPeiflercom Evaluating the Results 259