Taking the Bogeyman Out of Program Evaluation

Careful, systematic planning is essential for producing useful and informative program evaluations.  A well-conceived evaluation process is built on clear, thorough communication between the program administrators and the evaluator about components, constraints, and goals of both the program and the evaluation. All parties must be certain they understand each other’s needs completely before collecting the first piece of data. Such a full exchange of information makes it easier to plan an evaluation that covers all parts of a program and controls for any biases that might color the interpretation of outcomes.
This site provides a range of resources to help you build solid program evaluation plans and judge the worthiness of existing evaluations.

  • A white paper describes different kinds of program evaluations, discusses the importance of using rigorous research techniques, and synthesizes advice and information about design and implementation.
  • The Resources for Practitioners page provides links to guides and other publications offering how-to advice for planning and executing an evaluation.
  • A Glossary defines common terms in program evaluation planning, design, and implementation.

Because science and medicine have used rigorous evaluation methods for years, this site includes several examples from those fields. While those examples have structural elements that apply to education programs (e.g. using rigorous research methods, ensuring that statistical models include sufficient controls, etc.), you may want to skip details that do not relate to your needs.
Before you start to work, here’s some advice from experts to help you avoid serious mistakes:

  • Start planning the program evaluation as early as possible and document details, including budgets for money and time, in the full project plan. Involve the person doing the external evaluation in planning.
  • Formulate the evaluation questions carefully, based on the project’s goals, delivery methods, and stakeholders. And don’t skip the logic-model stage. Both steps provide a foundation for designing a strong program evaluation.
  • Identify data sources early and contact the owners to make sure needed data will be available. If the evaluation will involve participation by program stakeholders, involve them early to make sure they understand and agree to their expected role.
  • Devise plans for assessing whether the program is being enacted as designed so that adjustments can be made to improve fidelity or change elements that are difficult to implement, if necessary.
  • View evaluation as an integral, multi-stage part of the program that continues through the program’s lifetime. That approach to evaluation will require you to plan timetables and procedures for ongoing data collection and formative processes for using the growing store of data – and the information it reveals – to fine tune the program and institute major changes if necessary.

Use rigorous methods. Rigorous program evaluation is a critical component of a continuous improvement process. A well-designed evaluation can save money if it reveals that a crucial part of a program is not working and you correct the problem or discontinue the component.