I have had some time to reflect on the three most recent letters (there was an earlier one) that were sent to the Chronicle in response to my essay “An Insider’s Take on Assessment.”
One of them, from Josie Welsh at Missouri Southern, is largely consonant with my view on assessment. She asks why assessors ought to “trouble faculty to collect junk data when others have labored to produce empirically sound findings we can apply to the classroom?” I have long said that if people want to do real research on student learning, that’s a good thing and faculty should seriously consider the results of careful, well-design research on what works in the classroom.
The other two letters, one from NILOA people and the other from a group associated with AALHE and something called the Virginia Assessment Group, make an effort to refute the arguments I set out in the Chronicle. The results are fairly predictable and entirely in keeping with Upton Sinclair’s observation about how hard it is to convince people of something when their salaries depend upon them not being convinced.
Neither letter challenges the central point of everything that I have ever written about assessment: there is no evidence in the scholarly literature that all the time and money that has gone into assessment has resulted in improvements in student learning.
One of the letters acknowledges this directly:
“published research articles that empirically demonstrate that assessment of learning by itself leads to student learning improvement are few.”
Neither article attempts to refute Eubanks’ arguments about the poor data quality of virtually all real-world assessment.
Rather we learn that:
“assessment is not about measurement alone. Such a view is both limiting and outdated. Also, a focus on measurement sidelines conversations about the organization and structure of assessment efforts, as well as a more complex view of the use of results.”
This is utterly at odds with the lived experience of anyone who works in a university, but it also sounds like the authors are trying to redefine assessment as something that is not about measuring student learning but rather about more nebulous things like “creat[ing] engaging spaces and experiences for student learning.”
Jankowski and Marshall of NILOA attempt to refute my argument that assessment discounts disciplinary expertise and knowledge by citing a study that shows that the most common form of assessment involves “disciplinary assignments.” I don’t know what this means, but my guess is that this does not mean trusting that a faculty member’s grade on a term paper or final exam can be used for assessment. Further they observe that a survey of provosts indicates that provosts find “classroom-based performance assessment” to be the most useful form of assessment to “inform teaching and learning.” Perception studies are perhaps the lamest and laziest form of research there is. Do we have reason to believe that provosts are right about this? Do we have research that shows that student learning improves when we engage in “classroom-based performance assessment”?
One interesting thing I gleaned from the report they cited is that the larger and more selective a school is, the less assessment it does and the less use of assessment data it makes. That people are willing to pay more to go to schools that do less assessment and less to go to schools that do more assessment (selective schools tend to be pricier that less selective ones), tells us that market forces see no value in assessment as it is currently practiced.
I could go on, but I’ll conclude with this. Assessors have been telling us that our professionally informed instincts about student learning are not trustworthy and that they need to be replaced with the rigorous use of real measurement of concrete learning outcomes. When confronted with a demand that they provide concrete, measurable evidence that assessment does what it claims to do, they fall back on anecdotes, perceptions and attempts to redefine assessment itself.