Evaluation and Improvement

As a state leader or program staffer, you can maximize the benefits of evaluation by following a few basic guidelines:

  • Invest heavily in planning: Invest time and effort in deciding what you want to learn from your evaluation. This is the most important step in the process. Consider what is most important to discover about your program and its impact on participants, and use this information to guide your evaluation planning.
  • Integrate the evaluation into the program’s ongoing activities: Evaluation is not something that an outsider does to a program after it is over or an activity tacked on merely to please funders. Program leaders and staff can gain knowledge and improve practice from an evaluation more quickly when it is integrated into the program.
  • Ensure that leadership participates in the evaluation and show program staff that it is highly valuable work: An evaluation needs the participation of leadership to succeed. Even if an outside evaluator is hired to conduct the evaluation, leadership must be full partners in the evaluation process. An outside evaluator or in-house expert cannot do it alone. Leadership can outline the program’s big-picture vision, objectives, and key participants for the evaluator. Also, program leads and staff will value the evaluation if leadership values it. Talk about the evaluation with leads and staff individually and in meetings. If you hire an outside evaluator to conduct the evaluation, be sure that this individual attends staff meetings and gives presentations on the status of the evaluation. Leadership involvement will encourage a sense of ownership of and responsibility for the evaluation among program staff.
  • Involve as many program leads and staff as possible, as much and as early as possible: Project staff have a considerable stake in the success of the evaluation, and involving them early in the process will enhance the evaluation’s effectiveness. Staff will have questions and issues that the evaluation can address and are usually pleased when the evaluation validates their own hunches about what does and does not work in the program. Because of their experience and expertise, program staff can ensure that the evaluation questions, design, and methodology are appropriate for the program’s participants. Further, early involvement of staff will promote their willingness to participate in data collection and other evaluation-related tasks.
  • Engage partners, communities, and stakeholders in evaluation planning and implementation:[1] Engage partners, communities, and those who have a stake or vested interest in the evaluation findings. Stakeholders may include the intended users who can most directly benefit from the evaluation, as well as others who have a direct or indirect interest in program implementation. Engaging stakeholders in the evaluation enhances intended users’ understanding and acceptance of the usefulness of evaluation information. Stakeholders are much more likely to buy into and support the evaluation if they are involved in the evaluation process from the beginning. Work with the people who will be using evaluation information throughout the entire process so that the information collected, analyzed, and reported best meets their needs.
  • Be realistic about the burden on leadership and staff: Evaluations are work. Even if your evaluation calls for an outside evaluator to do most of the data collection, it takes time to arrange for the evaluator to have access to records, administer questionnaires, or conduct interviews. It is common for agencies and evaluators to underestimate how much additional effort this involves. When program leads and staff brainstorm about the questions they want answered, they often produce a very long list. This process can result in an evaluation that is too complicated. Focus on the main purpose of the evaluation and key questions that it should address.
  • Be aware of ethical and cultural issues: This guideline is very important. When evaluating a program that provides services or training, you must always consider the responsibilities to the participants and the community. You must ensure that the evaluation is relevant to and respectful of the cultural backgrounds and individuality of participants. Evaluation instruments and methods of data collection must also be culturally sensitive and appropriate for your participants. Participants must be informed that they are taking part in an evaluation and that they have the right to refuse to participate in this activity without jeopardizing their participation in the program. Also, ensure that confidentiality of participant information will be maintained. Knowing when to use an Institutional Review Board is also important. This is an administrative body established to protect the rights and welfare of human research subjects who are recruited to participate in research activities that are conducted under the sponsorship of the institution with which it is affiliated.
 

[1] Centers for Disease Control and Prevention, Office on Smoking and Health & Division of Nutrition, Physical Activity, and Obesity. (2011). Developing an effective evaluation plan: Setting the course for effective program implementation. Atlanta: Authors. (p. 7).