Q: What is implementation evaluation?
A: Also known as process evaluation, an implementation evaluation is designed to determine whether a program is being delivered as intended. It documents what happens (if activities are conducted as planned and according to schedule); the frequency and intensity of the activities; and the extent to which the participants were reached. Implementation evaluation requires close monitoring of implementation activities and processes. This type of information can be used to adjust activities throughout a program’s lifecycle.
Q: Why do we need to conduct an implementation evaluation? Why can’t we just use the results from an outcome evaluation?
A: Program staff may use information from an implementation evaluation to make improvements and adjust activities to meet desired outcomes. This is especially important for programs that are still undergoing development or that are replicating an existing model in a new setting or with a different population.
Documenting implementation is directly relevant to understanding outcomes. Without knowing exactly what program components have been implemented or to what degree they are implemented, you cannot make use of outcome evaluation results to understand what worked or didn’t work, and why.
Q: What are common questions used in an implementation evaluation?
A: In general, an implementation evaluation answers: How is the program being implemented? Are all participants being reached as intended? What is the quality and extent of implementation?
Other questions that may be relevant for your program: What aspects of your original program model were implemented as planned and what had to be changed? What changes were made and why? What aspects of the program were felt to work particularly well, and why? Is there evidence that any unintended outcomes occurred, either positive or negative, for either the program, its staff, or for participants? Did you confront any barriers that were not anticipated? What next steps will you take/do you recommend to further revise the model and why?
Q: What is dosage? Why do I need to measure quality as well as dosage in an implementation evaluation?
A: Dosage refers to how much of program activity was done, how many people were involved, and how much of each activity was administered to each participant, classroom, or school over a specified length of time. Dosage is an important concept in evaluating program effectiveness since a diluted dose may produce the same results as no dose at all.
At the same time, it is important to measure the quality, not just the quantity of implementation activities whenever possible. For example, you may want to assess the extent to which a particular science lesson plan connects with state standards, requires the use of critical thinking skills, and/or integrates mathematics in addition to counting the number of lessons or hours of science within a magnet curriculum unit.
Q: Can I rely on self-reported data in my implementation evaluation?
A: To measure changes in classroom practice or student behavior and attitudes, evaluators often rely on survey instruments and self-reported logs.
Research points out that subjects tend to report what they believe the researcher expects to see, or report what reflects positively on their own abilities, knowledge, beliefs, or opinions. Research also has shown that human memory isn’t necessarily reliable.
While it may be too expensive or work-intensive to conduct frequent observations and in-depth interviews over time to corroborate self-reported data, given the lack of reliability of self-reported data, you should build in checkpoints and make note of large discrepancies as you collect data. It may also be appropriate to require documents or more specific information as part of self-reported data (e.g., titles of lessons that were taught, sign-in logs for meetings).
Q: Should we use pre-existing instruments to help us document implementation?
A: Using pre-existing evaluation instruments can often save you time in gathering high-quality data since they come with established validity and reliability. You may be able to compare data from your program with data from other programs using the same instrument. However, you should consider the cost of using such instruments, as copyrighted surveys may have to be purchased. In addition, since pre-existing instruments are not developed specifically for your program, the data may not be of the best quality since they may not be appropriate for your program’s activities or in addressing the evaluation’s specific concerns.
