Q: What are “validity” and “reliability” in terms of evaluation data?
A: Validity refers to the extent to which appropriate and accurate information is used for a particular purpose. It is sometimes viewed as trustworthiness or believability. In terms of an instrument, validity is the degree to which it measures what it is intended to measure. In terms of an evaluation study, validity refers to the degree to which it uses sound measures, analyzes data correctly, and bases its inferences on the study’s findings.
Reliability means the extent to which an instrument, test, or procedure produces the same results on repeated trials.
Getting high-quality data for an evaluation is a reference to getting the most valid and reliable data possible.
Q: When do issues of quality data usually emerge?
A: Attention to data quality should be paid throughout the evaluation process. You can take proactive measures early on in your evaluation process to ensure you collect valid and reliable data. During the data collection process, there are techniques you can use to ensure data quality. It is also common to have problems surface when data are prepared for analysis. For this reason, you should build in frequent checkpoints and allocate sufficient time and resources to handle data issues during the collection and analysis stages of evaluation.
Q: What is data cleaning and why is it an important part of data collection?
A: Data cleaning is the process of going through submitted data and looking for inaccuracies, inconsistencies, duplications, and anomalies to ensure that the data are useful and functional. It is important to build in time to check for mistakes caused by changes to the data collection process or human error. Data cleaning often involves eliminating inaccuracies and redundancies, checking for consistency and completeness, and providing feedback for improvements in the future. While data cleaning can be intricate and time consuming, it is critical that your evaluator get high-quality data to use for analysis.
Q: What is an Institutional Review Board and how does an IRB impact data collection plans?
A: An IRB is a committee or organization charged with reviewing and approving the use of human participants in research and evaluation projects. The IRB serves as a compliance committee and is responsible for reviewing reported instances of regulatory noncompliance related to the use of human participants in research.
If the evaluation uses federal funds, then the IRB must approve the evaluation protocols and designs before data collection begins. This includes the necessary content in letters requesting informed consent as well as plans for maintaining the anonymity and/or confidentiality of participants.
Q: Who handles the planning and implementation of data collection processes? I thought we hired an evaluator to handle the data.
A: Your evaluator will have experience in designing a data collection plan that yields valid and reliable data, but your input and support is necessary to ensure the plan is feasible. Knowledge about the district context, existing data, and the capacity of district staff are all elements that impact data collection. Your relationships with personnel and school staff are also useful in terms of getting the data you need. When combined with the expertise in IRB processes, data management, and data analysis your evaluator brings, your support ensures effective implementation of the data collection plan.
