Extra Resources for MSAP Rigorous Evaluation

Conducting an MSAP rigorous evaluation provides a unique opportunity to gauge the impact of your program on student achievement and minority group isolation. There are some extra considerations the evaluation team should think about in setting up a rigorous evaluation. The additional resources provided here and for other practices will help you determine if this type of evaluation is appropriate for your district and, if so, to design and carry out the most rigorous evaluation possible.

Manage logistics and communications for data collection

• Try to avoid setting up a data collection plan that puts schools on data overload. If your district is conducting multiple evaluations (e.g., Title I, English language learners, technology), some comparison school sites may be inundated with data requests without having a direct use for the findings.

• Be clear and up front with control or comparison schools about your data needs as well as the purpose of a rigorous evaluation. Make sure you make time for communication and relationship building, not just data requests.

• Start early with getting procedures approved. Go through the appropriate channels in your district (e.g., Institutional Review Board, superintendent’s office) to get buy-in and the official approval for the evaluation and data collection plan.

SAMPLE MATERIAL Successfully Collecting Data From Comparison Groups (.pdf 262.2 KB)

Review these tips for building the cooperation and trust you need from comparison groups to ensure you collect high-quality data.

---

SAMPLE MATERIAL Communication With Families About Experimental Design (.pdf 238 KB)

Consider how this letter to families helps prevent misunderstandings while garnering support for an evaluation’s data needs.

---

Troubleshoot data issues as they emerge

• By design, rigorous evaluations need data from an additional control or comparison group to determine the impact of a program. This means there is another set of issues to anticipate when planning data collection for these types of evaluations.

• Make sure you receive data from most (if not all) of the participants in the magnet program, and the control or comparison sites. Uneven attrition from either group compromises the results of a rigorous evaluation. You may want to anticipate higher than average mobility rates for students when figuring out the number of participants you need to make the study work.

TOOL Anticipating Data Collection Issues for Rigorous Evaluation (.doc 172.5 KB)

Learn how to address specific issues that may negatively impact findings of experimental or quasi-experimental evaluations.

---

VIGNETTE Challenges of an MSAP Rigorous Evaluation for an Interdistrict Program (.pdf 175.9 KB)

Reflect on how one interdistrict evaluation team addressed challenges in conducting rigorous evaluation design.

---

Use techniques to ensure valid and reliable data

• Before you begin your data collection process, familiarize yourself with what constitutes valid and reliable data in the context of a rigorous evaluation.

TOOL Getting Key Items Right When You Measure Program Outcomes (.doc 89 KB)

Check if your evaluation uses key data collection approaches that produce valid evidence of program impact for a rigorous evaluation.

---