Comments

I like the farmer analogy, but it fails to apply correctly in this case. Let me rework it so it matches the true state of affairs.

For decades and decades, farmers went out in their fields and worked to produce the best crop they could. Unfortunately, few farmers had any data on the quality and quantity of their output other than anecdotal data. “I once had an ear of corn,” said one farmer, “who went to Oxford!” Another farmer points out, “Nearly all of my corn gets eaten right away! We must be doing a great job as farmers.”

Although the farmers repeatedly assured the government and the community members that they were providing a good service – just “trust us,” they would tell them – the rising cost of grain, which pushed many families deep into debt, and the tax credits, used to prop up farmers in years of bad weather, were receiving increased ire from the government and taxpayers. So naturally, the government and the community members began demanding that the farmers prove that they were providing a good benefit to the community.

The farmers were appalled, insulted, and irritated that anyone would dare to doubt their expertise in farming. They had, in fact, studied farming intensely and worked long hours in difficult conditions out in the fields. The farmers also pointed out that they used farming practices that were widely accepted and commonly known to be effective. Yet, this was not enough for the government and community members – they wanted to see evidence that the crops produced high quality crops of sufficient quantity to feed their hungry community.

At the same time, some farmers had long suspected that they could grow more and better crops by modifying their farming practices. They knew there were differences in rain, and sunlight, and pests each year, but did not have good data on the relationship between those experiences and the quality and quantity of output of crops. So these farmers developed grading scales to evaluate the quality of the crop produced and insisted on using baskets that were similarly sized to gauge the quantity of crops produced each year. Now, the baskets were sometimes a bit different sized, and the grading scales were sometimes inaccurately applied, but it still gave farmers a better idea of what they were producing than without using such measures.

Farmers tried lots of different things to improve the quality and quantity of their crops. Some prayed at the temple, some sacrificed children, some altered their planting dates, some added additional water in dry months, and others tried other activities. But since the farmers were collecting information about the quality and quantity of yield, it was easy to determine, over time, which of these practices were making a difference. Some farmers changed nothing about their farming practices and complained that the act of measuring the quality and quantity of their crops did not lead directly to better crops. Of course these farmers were also correct – by doing nothing differently, other than measuring their output, they had little reason to expect that their results would change.

Assessment charts a similar path. For too long our responses to concerns about the quality and value of our degrees and programs was simply “trust us.” As a result, government, parents, taxpayers, and others began demanding that we provide some evidence that we were actually producing something of value. Grade data (which can be easily manipulated by instructors) and anecdotes simply weren’t sufficient (for every 1 positive anecdote it was easy to find 3 negative anecdotes). At the same time, there was a growing recognition that degree program quality could be improved by using some of the same information provided for accountability purposes. Assessment is not perfect by any means and isn’t always effective – but neither does measuring the output of farmers always produce gains in quality and quantity.

I have hears requests for “proof” that assessment produces gains in learning. To me this is a bit of a nonsensical question. It’s like asking to compare the output between a farmer who is measuring the quality and quantity of his output to a farmer who isn’t (to what, then, do you compare? You can’t compare something to the null.). Or, let’s say as a the researcher you volunteer to measure the output of the 2nd farmer. This improves your research methodology but does little to really get at the value of assessment – that is to say, if the 1st farmer is simply measuring the quantity and quality of his output and isn’t taking any improvement actions in response, than why would we expect there to be any differences?

In any case, I doubt I will change your mind (I note with irritation the cheap shot you take at the end of your piece on educational researchers which represents little more than your own arrogance and ignorance) but I write to provide a different perspective on an important issue and challenge you to be open to seeing this work from a different perspective.

If only assessing learning was as simple as measuring crop yields. Farmers can effectively measure both inputs and outputs. Educators have no such luxury. We can measure inputs to some degree, but the measurement of outputs/outcomes is fraught, to say the least. How do we get around that problem in a way that leads to meaningful assessment of learning?

excellent