What baseball stats can teach us about standardized tests
As I was reading Keith Law's book “Smart Baseball,” it struck me that this is why we at Michigan Future Inc. have turned against standardized tests as the measure of school, educator and student performance. I'm sure many of you are asking, "What do baseball statistics have to do with standardized tests?" The subtitle of the book provides a clue to the answer: “The story behind the old stats that are ruining the game, the new ones that are running it, and the right way to think about baseball.”
Law's book is an explanation of the new statistics that are being used by baseball teams and, increasingly, the media to measure baseball players. But the book starts with an explanation of why the old statistics needed to be replaced. The bottom line: They do a lousy job of measuring the contribution each player makes to winning baseball games. So, you had for nearly a century a set of stable statistics that as the subtitle says were "ruining the game" because the statistics were not aligned with what every team wanted: to win.
Law argues the adherence to traditional measures of player value leads to bad personnel decisions — which players are signed and how much they are paid — and maybe even more importantly, how the game is played. And both reduce the chance of a team winning. Not smart!
Anyone who watches baseball today can see that better understanding of what contributes most to winning and having measures that are aligned with what contributes most to winning is fundamentally changing how the game is played. From using pitchers far differently, to how players are positioned on defense, to what hitters are trying to accomplish at the plate, to which players get to play and which players are paid the most.
Just as the traditional measures of baseball were not aligned with the desired outcomes, so it is with standardized test scores and desired student outcomes. We have been unable to find any evidence that standardized test scores are good predictors of the outcomes we want from K-12 education: improved life outcomes of students.
There is compelling research that concludes standardized tests scores are not good predictors of either college success or life outcomes.
The book “Crossing the Finish Line” used a large, nationally representative set of student data to analyze what truly predicts success in college. And what they found was that students’ high school GPA — made up of grades given by individual teachers across four years of high school — was far more predictive of eventual college graduation than their SAT/ACT score. And the students’ GPA was predictive regardless of high school attended; whether students went to a “good” high school or a “bad” high school, a good GPA predicted college success.
Why? Because while test scores measure a student’s ability on a narrow band of math and reading skills, GPA measures a diverse set of capacities, encompassing academic habits, content knowledge and noncognitive skills exhibited day after day across four years of high school.
Research from Northwestern economist C. Kirabo Jackson came to a similar conclusion. Using a large set of student data from North Carolina, Jackson found that a “noncognitive” index of grades, attendance and disciplinary records was more predictive of long-term success than test scores. And he also found the set of teachers who were able to improve this index was an entirely different set of teachers than those who were adept at raising test scores.
The message from both cases: When we focus only on test scores, we miss the really important stuff.
Just like in baseball, the adherence to the wrong high-stakes measures in schools leads to bad design. And that’s both in curriculum and pedagogy; both what is taught and how it is taught. Narrowing curriculum to the content-specific skills that are on the test rather than a focus on broad 6Cs skills that includes content — but far more collaboration, communication, critical thinking, creativity and confidence — is making it harder for schools to move away from teachers imparting content to learning through projects.
What baseball has figured out — and so far education policymakers have not — is you can do real harm by an adherence to traditional measures. The right way to construct measures is by working back from the outcomes you want. The lesson “Smart Baseball” teaches us is if the measure does not help predict achieving the desired outcome, get rid of it and find measures that do.
Lou Glazer is president of Michigan Future Inc.