Extra Resources for MSAP Rigorous Evaluation

Conducting an MSAP rigorous evaluation provides a unique opportunity to gauge the impact of your program on student achievement and minority group isolation. There are some extra considerations the evaluation team should think about in setting up a rigorous evaluation. The additional resources provided here and for other practices will help you determine if this type of evaluation is appropriate for your district and, if so, to design and carry out the most rigorous evaluation possible.

Surface assumptions about how your program is supposed to work

• Expect probing questions from your rigorous evaluator. To design an effective evaluation that measures a program’s impact, your evaluator must first understand the assumptions behind your program and how it is supposed to work.

• Don’t assume that magnet program implementation looks the same across grade levels and themes. A common pitfall is assuming homogeneity of magnet activities to determine a districtwide effect of the program without accounting for major differences between, for example, STEM and art themes, or elementary and high schools.

• A good timeline for conducting an experimental or quasi-experimental evaluation reflects the reality that full implementation of program activities often doesn’t occur until year 2 of the grant cycle. Build in checkpoints for testing your assumptions related to implementation at the end of year 1 and adjust your logic model accordingly.

Create a logic model to explicitly connect your activities and goals

• If you are funded by an MSAP grant to conduct a rigorous evaluation, you are required to submit a logic model as part of your initial, interim, and final reports.

• A good logic model articulates what the new experimental program is supposed to do when implemented appropriately, how that will impact teacher practice and student experience and, ultimately, how that will lead to your desired outcomes.

• When different partners are conducting the MSAP regular evaluation and rigorous evaluation, use the logic model as a communication tool to bridge your studies of program implementation and impact.

Use your logic model to develop relevant evaluation questions

Review these sample evaluation questions from MSAP grantees to see how impact evaluation questions are framed in various contexts.

• Do students attending magnet schools make greater achievement gains than similar students attending conventional schools? If any differences are found, what is the magnitude of the difference in achievement gains between magnet and non-magnet students?

• Do magnet schools produce greater achievement benefits for NCLB-defined subpopulations of students than non-magnet schools?

• Does the magnet program result in closing the performance gaps between students from minority and non-minority backgrounds at a faster rate than at non-magnet comparison schools?