What About Ethics?
To help schools and school districts execute high quality program evaluations, the Education Innovation Institute, at the University of Northern Colorado, has created a multi-faceted training program to help you in, Evaluating, What, Works.
The Ethics of Random Assignment
Randomized control trials, or RCTs, are the most effective method for determining if a new program is causing a positive outcome for students. But many people have voiced ethical and political concerns about randomly assigning children to a new program. They ask: Isn’t it wrong to withhold the program from some students? If the program is expected to be beneficial, shouldn’t every student receive it?
These are important considerations when designing an RCT. However, randomization is not in and of itself unethical and can even be more fair than other approaches when rolling out a new program.
Assuming a Positive Impact
To address these concerns, we can examine a common assumption people make when thinking about RCTs, which is that a proposed program will have a positive impact on students. However, out of the thousands of programs available to schools, only a small sliver have been evaluated for causality. Correlational research - studies that suggest a program has benefits but does not prove that the program causes benefits - are much more common. Even for the few cases when cause has been shown, the benefits may not translate to a new setting. For example, a curriculum that dramatically raises reading scores in inner city schools might not have the same impact in the suburbs, and an algebra curriculum evaluated for students with low math scores may not have a similar effect among higher-scoring students. So, even when a program shows promise through anecdotes and correlational studies, a randomized evaluation should be carefully considered before a new program is taken system-wide.
Will Programming be Withheld?
There is also concern that randomization takes programming away from some students who would otherwise get it. But even when a new program is expected to become universal, limited resources can confine its initial implementation to a few hand-picked or volunteer teachers. Those in the initial sampling often have more enthusiasm or special skills for the intended program than non-participating teachers. Therefore, early programmatic success will not necessarily translate to all teachers when the program is expanded. Thus, this non-randomized roll-out approach could result in misleading outcome indicators. Further roll-out could unfairly burden teachers and students into the future.
The Risk of Ongoing Burden
Committing resources to a new program that has not been tested with an RCT can be expensive. There are the upfront implementation costs, but more importantly, once a new program is put in place, it has a tendency to live on. Even when the program is not having the intended impact, it continues to take up time and money—sometimes for years to come—that could be better used elsewhere.
Discussions with Stakeholders
Because the value of a proposed RCT is not always well understood, those who will be affected may initially react negatively to the idea. Sufficient planning time is a fundamental element to the success of RCT’s and can provide the opportunity to discuss stakeholders’ concerns. These discussions can include such things as: an appropriate level for randomization, who will be included in the pool of randomized participants, and what impact on student outcomes can be reasonably expected during the evaluation’s timeframe.
Using Results to Make Decisions
It’s also important to consider how the evaluation results-- positive, null, or negative-- will affect future decision-making. As educators feel more and more overwhelmed by the multiple initiatives that are available to them, they are likely to welcome an evaluation that helps the district prioritize time and resources effectively.
Evaluating What Works
Ethical and political objections to a proposed RCT, such as these, are hurdles that can be overcome. With sufficient planning, thoughtful randomization, and a well-planned roll-out, stakeholder concerns can usually be addressed. And the Evaluating What Works program is here to help.