Fact or Fiction
Making decisions based on data is a popular trend, but make sure the facts are real and not simply strongly held opinions masquerading as reality.
By Douglas B. Reeves
Fall/Winter 2009
The subject of this column is “decision-based data-making.” This is not a misprint.
Educational leaders and policy- makers pride themselves on “data- based decision-making,” but discern- ing school board members must eval- uate carefully which came first—the data or the decision. With that, I’d like for us to consider different levels of claims that you likely will encounter and provide some of the most recent and best sources of educational research.
LEVEL 1: ‘I BELIEVE IT’
Education policy discussions are rife with Level 1 claims. Silicon Valley entrepreneur James Barksdale was fond of saying, “Everybody’s entitled to their own opinion; they’re just not entitled to their own facts.” Often, policymakers must listen to sincerely held beliefs about the virtues of corporal punishment or claims that children will learn to read and write by coloring, but we need not confuse respectful listening with acknowledgement of a fact.
LEVEL 2: ‘IT WORKED FOR MY STUDENTS AND MY CHILDREN’
Memoirs sometimes masquerade as research. With heartrending detail, a writer or speaker will recall the details of their experiences with their students or families. “It worked for me!” they enthuse, or more frequently, “I tried it in my classroom and it didn’t work.” In both cases, the implied claim is that personal experience can be generalized to all students and all schools.
LEVEL 3: ‘MY COLLEAGUES HAVE SIMILAR EXPERIENCES’
School improvement washes upon the shoals of the determined opposition of the vocal few. When the “I” becomes “we” in discussions of reform ideas, then leaders become preoccupied with the need for popularity rather than effectiveness. They succumb to the idea that “buy in” is the prerequisite for change, failing to acknowledge that effective change requires that people sacrifice time and energy—and pre-existing beliefs.
Wise leaders do not conduct an endless search for “buy in,” but acknowledge the truth—change is difficult and always involves opposition. They say, “I understand and respect that you do not agree with me and that you do not like my proposal for improved student achievement. But I’ve got great news: I’m not asking you to like it—I’m just asking that you do it, give it a try, and together let’s evaluate the impact on achievement. If it doesn’t work, we’ll stop. If it works, we’ll continue. But I’m not going to fail to give a promising new idea a chance just because you’ve had some bad experiences in the past.”
LEVEL 4: SYSTEMATIC EXAMINATION OF AUTHENTIC CASES
Fortunately, board members do not have to settle for claims at levels 1, 2, or 3. Leaders can and must demand a higher standard. Researchers such as Heather Zavadsky (Bringing School Reform to Scale, 2009) and Karin Chenoweth (It’s Being Done, 2007) provide compelling cases of sustained school improvement.
Case studies are not necessarily isolated anecdotes, as Ben Levin’s splendid How to Change 5,000 Schools (2008) makes clear. Case studies aren’t random samples and not necessarily generalizable to other schools, but systematic examination of effective schools allows researchers, leaders, and policymakers to consider in detail the specific actions of teachers and leaders who are associated with improved achievement.
LEVEL 5: LOOKING FOR ‘PREPONDERANCE OF EVIDENCE’
Imagine you are on a jury. There is conflicting testimony by persuasive witnesses on both sides. Experts, stories, and statistical tables seem to support both sides. What do you do?
In a criminal case, you must find evidence “beyond a reasonable doubt” to return a verdict in favor of the prosecution. In a civil case—the closest anal- ogy to the decision standards faced by education policymakers—you must consider the “preponderance of the evi- dence.” The evidence is not perfect, and neither side has a monopoly on the truth. But at the end of the day, you find that one side has made a sufficient case for public policy.
You know that exceptions test the rule—just because my grandmother smoked like a chimney and lived into her 90s, you nevertheless ban smoking in schools. Just because you received corporal punishment and turned out just fine, you decide to consider the evidence and ban the beating of children in the schools you govern. Fortunately, some excellent examples of educational research meet this most demanding standard.
Ken Leithwood and his colleagues (Learning from Leadership, 2010) used a rigorous methodology to link specific leadership practices to student achievement. Andy Hargreaves and his colleagues (The Fourth Way, 2009) bring together decades of research and on-site observations to identify specific teaching and leader- ship variables that are most effective. Robert Marzano (The Art and Science of Teaching, 2007) synthesizes the work of many other researchers in a meta-analysis—a technique that brought together the work of more than 1,000 studies. And in a stunning tour de force in research, John Hattie (Visible Learning, 2009) published a meta-analysis of more than 800 meta- analyses, providing clear guidance for leaders and policymakers about the most—and least—effective education- al practices.
EVIDENCE-BASED DECISIONS
When we focus on Level 5 research, we induce a degree of humility in any researcher. My work is nothing more than a pebble on the mountain of research. Beliefs and anecdotes, no matter how compelling, cannot compete with evidence.
The problem, of course, is that heartfelt beliefs and tear-jerking anecdotes can trump the best evidence during the public comment period of a board meeting. In “Queen for a Day,” a reality television show that aired in the 1950s, the audience members determined the winner—the woman with the saddest story—by the volume of their applause. But in the 21st century, board members are often called upon to select not the most popular but the most effective winner of the debate.
You can make this difficult choice when you first identify the five levels of claims that you will encounter, and limit your decisions to those based on the best evidence.