As someone who has been publicly critical of learning outcomes assessment for a long time, one of the questions I am often asked is: “If you are so opposed to assessment, what would you replace it with?” By way of an answer I have started resorting to this fable:
Imagine that you live in a Bronze Age village. You and everyone else in the village depend for your livelihood on subsistence farming, so you have a keen interest in the success of your crops. Because of that, you have developed a good sense of when to plant what crop, what types of soil work best with specific crops, when to weed, when to harvest and so on. It’s not scientific knowledge but it works pretty well. Still, you are always on the look out for ways of improving your yields.
One day a group of priests show up. They hold an information session at which they advise you that in the imperial capital (which, of course, taxes your crops), the authorities are concerned that the crop yields in the villages (and thus the tax revenues) are not what they could be. They would like to see that change.
The priests point out that most of the other villages have begun sacrificing children to Moloch in order to improve their yields. Reluctantly, your village agrees to start sacrificing children to Moloch in the hope that he will reward you with better harvests.
At first, it seems to be working. Soon the village has a new temple to Moloch and a clutch of recently hired associate and assistant priests. But after a couple of harvests your yields have reverted to the mean.
A delegation goes to the priests and asks what the problem is.
Moloch, you are told, does not bestow his favors on those who just go through the motions of sacrificing children. You need to buy in to the process, to commit, to really have faith, or it won’t work.
So you stop making the most junior farmers do all the sacrificing, you start spending more time going to tedious ceremonies devoted to Moloch, and really try to commit.
Still, the crop yields don’t change. You go back to talk to the priests. You note that that there is now a new wing on the building and the temple complex is starting take up good arable land that you once used to produce food.
You tell them that your crop yields seem about the same, but the relentless expansion of the temple complex is cutting into the total production of food.
They respond that Moloch worship was never really about crop yields. It’s more about the conversations it sparks about how much we care about our crops and how it lets us reflect on what we and Moloch value about our children. And don’t forget, the imperial tax collector, and the Solaria Foundation that advises him, feel very strongly about the importance of Moloch worship. And anyway, what would you replace Moloch worship with if we did stop? Don’t you care about the crops?
So how would we replace Moloch worship? Golden Calves to Ba’al? Cats to Isis? Bulls to Mithras?
Or, might it be a positive step just to stop sacrificing children to Moloch? There would be fewer resources squandered on the temple and the priests. The land the temple sits on could be put back in cultivation and even if your yields remained the same, at least you would have a little more land in production.
This is pretty much where we are now with assessment. Assessment is just as effective as and no more grounded in evidence than the cult of Moloch. No one can show that 30 years of assessment has caused improvements in student learning. The arguments for assessment are more faith-based than scientific. That it has not yielded results is attributed to the faculty’s lack of faith (aka “buy in”). Conveniently this is an unfalsifiable argument in that faith is hard to measure, so assessors can always fall back on the “not enough buy in” argument any time someone points out that assessment can’t be shown to improve student learning. And just making that observation is seen as an example of people like me not buying in to assessment.
So, as with Moloch worship, just stopping would bring benefits: fewer wasted resources, many fewer meetings, and less administrative intrusion into the curriculum.
We can probably all agree that Moloch does not affect agricultural yields. But it’s worth thinking briefly about what has actually caused changes in agricultural yields over the centuries. What might our Bronze Age villagers have done to increase the productivity of their crops?
The increase in crop yields just since World War Two has been staggering and the contrast would be even more dramatic if you compared Bronze Age yields to modern food crop productivity. These increases are largely attributable to scientific agriculture. Careful, well-designed, controlled, scientific experiments conducted by experts have produced improved, higher yielding crops and determined the best practices for growing them. If there is an unsung hero of the 20th century it is Norman Borlaug, the plant scientist whose work is partly responsible for making it possible for the planet to support over 7 billion people.
Borlaug did not ask farmers to change their practices continually based on their own amateur assessments of the year’s crop. Meaningful study of crop yields and performance involves controlling for myriad external variables. There is rainfall, sunlight, temperature, and soil chemistry, all of which have to be systematically taken into account to determine whether a cultivar or practice is in fact producing the increased yields you are seeking.
Student learning takes place in an equally complex and variable social, physical, and financial environment. Assessment takes none of this into account and asks instructors to continually change their practices based on bogus data. The Green Revolution worked; assessment has not delivered on its promises.
For whatever reason, the study of higher education does not attract the resources and talent that, say, the study of genetics, biochemistry or economics does, so we have yet to find our Norman Borlaug.
Maybe the 21st century will see an educational revolution that equals the Green Revolution of the 20th century. But while we wait, let’s stop sacrificing children to Moloch.
excellent
If only assessing learning was as simple as measuring crop yields. Farmers can effectively measure both inputs and outputs. Educators have no such luxury. We can measure inputs to some degree, but the measurement of outputs/outcomes is fraught, to say the least. How do we get around that problem in a way that leads to meaningful assessment of learning?
I like the farmer analogy, but it fails to apply correctly in this case. Let me rework it so it matches the true state of affairs.
For decades and decades, farmers went out in their fields and worked to produce the best crop they could. Unfortunately, few farmers had any data on the quality and quantity of their output other than anecdotal data. “I once had an ear of corn,” said one farmer, “who went to Oxford!” Another farmer points out, “Nearly all of my corn gets eaten right away! We must be doing a great job as farmers.”
Although the farmers repeatedly assured the government and the community members that they were providing a good service – just “trust us,” they would tell them – the rising cost of grain, which pushed many families deep into debt, and the tax credits, used to prop up farmers in years of bad weather, were receiving increased ire from the government and taxpayers. So naturally, the government and the community members began demanding that the farmers prove that they were providing a good benefit to the community.
The farmers were appalled, insulted, and irritated that anyone would dare to doubt their expertise in farming. They had, in fact, studied farming intensely and worked long hours in difficult conditions out in the fields. The farmers also pointed out that they used farming practices that were widely accepted and commonly known to be effective. Yet, this was not enough for the government and community members – they wanted to see evidence that the crops produced high quality crops of sufficient quantity to feed their hungry community.
At the same time, some farmers had long suspected that they could grow more and better crops by modifying their farming practices. They knew there were differences in rain, and sunlight, and pests each year, but did not have good data on the relationship between those experiences and the quality and quantity of output of crops. So these farmers developed grading scales to evaluate the quality of the crop produced and insisted on using baskets that were similarly sized to gauge the quantity of crops produced each year. Now, the baskets were sometimes a bit different sized, and the grading scales were sometimes inaccurately applied, but it still gave farmers a better idea of what they were producing than without using such measures.
Farmers tried lots of different things to improve the quality and quantity of their crops. Some prayed at the temple, some sacrificed children, some altered their planting dates, some added additional water in dry months, and others tried other activities. But since the farmers were collecting information about the quality and quantity of yield, it was easy to determine, over time, which of these practices were making a difference. Some farmers changed nothing about their farming practices and complained that the act of measuring the quality and quantity of their crops did not lead directly to better crops. Of course these farmers were also correct – by doing nothing differently, other than measuring their output, they had little reason to expect that their results would change.
Assessment charts a similar path. For too long our responses to concerns about the quality and value of our degrees and programs was simply “trust us.” As a result, government, parents, taxpayers, and others began demanding that we provide some evidence that we were actually producing something of value. Grade data (which can be easily manipulated by instructors) and anecdotes simply weren’t sufficient (for every 1 positive anecdote it was easy to find 3 negative anecdotes). At the same time, there was a growing recognition that degree program quality could be improved by using some of the same information provided for accountability purposes. Assessment is not perfect by any means and isn’t always effective – but neither does measuring the output of farmers always produce gains in quality and quantity.
I have hears requests for “proof” that assessment produces gains in learning. To me this is a bit of a nonsensical question. It’s like asking to compare the output between a farmer who is measuring the quality and quantity of his output to a farmer who isn’t (to what, then, do you compare? You can’t compare something to the null.). Or, let’s say as a the researcher you volunteer to measure the output of the 2nd farmer. This improves your research methodology but does little to really get at the value of assessment – that is to say, if the 1st farmer is simply measuring the quantity and quality of his output and isn’t taking any improvement actions in response, than why would we expect there to be any differences?
In any case, I doubt I will change your mind (I note with irritation the cheap shot you take at the end of your piece on educational researchers which represents little more than your own arrogance and ignorance) but I write to provide a different perspective on an important issue and challenge you to be open to seeing this work from a different perspective.