Bad Assessment

Back in 2007 when assessment had just started to really intrude into academic life, Laurie Fendrich wrote the first (and still the best) statement I have read about assessment. Her references to May Day parades,  Mao, and forced self-criticism, presciently captured the Taylorism meets Orwell quality of assessment.  The assessment industry has finally conceded that it has gone down an  “unproductive path.”  They can’t claim that they had not been warned over a decade ago about what they were doing.

This is just the first paragraph.  Read the whole thing here. Unfortunately it is so good that 12 years after its publication, the Chronicle still has it paywalled.

 

Outcomes-assessment practices in higher education are grotesque, unintentional parodies of both social science and “accountability.” No matter how much they purport to be about “standards” or student “needs,” they are in fact scams run by bloodless bureaucrats who, steeped in jargon like “mapping learning goals” and “closing the loop,” do not understand the holistic nature of a good college education. For all the highfalutin pronouncements accompanying the current May Day parade of outcomes assessment, in the end they boil down to a wholesale abandonment of the very idea of higher education. Whatever their purpose, outcomes-assessment practices force-march professors to a Maoist countryside where they are made to dig onions until they are exhausted, and then compelled to spend the rest of their waking hours confessing how much they’ve learned by digging onions. The mentor-protégé model of a college education is gone. We now confront the robot model, in which knowledge is reduced to what Nietzsche called “knowledge stones” — bits of information that administrators can count and students can digest without thinking.

Kara Moloney wrote a comment challenging the accuracy of the IHE’s reporting on the events at WASC.  Among other things, she was put out that Bob Shireman was quoted at length despite being an audience member rather than a panelist.  John Ahearn had this response.  If anyone knows him, please buy him a beer for his service to academia.

John Ahearn Kara Moloney • 8 hours ago

WASC met last week and the theme of the conference was “Provocative Questions…Courageous Answers.”  IHE has this article about what sounds like a pretty thoughtful panel on why assessment is not working.

“It’s not just that faculty members are crabby and hate change … There are good reasons why faculty hate it. It’s real and it’s earned,” Jankowski said. (An Inside Higher Ed survey of faculty members last year, for instance, found that 59 percent of respondents agreed that assessment efforts “seem primarily focused on satisfying outside groups such as accreditors or politicians,” rather than serving students.) Essays like this also reflect faculty disdain.

There is a certain irony to having the head of NILOA lament the state assessment, but it’s a step in the right direction.

The latest entry in the debate as to whether the value of college lies in what you learn while you are there or in what it signals about you comes from an article in the National Review Online.  James Piereson and Naomi Schaefer Riley build on an article that appeared in the Chronicle: “Why Thousands of College Grads Start their Careers at a Rental-Car Company.” (paywalled)

This is what economists such as Ohio University’s Richard Vedder and George Mason’s Bryan Caplan have been arguing for years: College degrees are simply a signifier — an easy way of telling an employer that you have a basic grasp of the English language, some rudimentary math skills, and the ability to show up on time in clean clothes. On those measures, is a graduate of the University of Michigan any different from a graduate of Michigan State or Northern Michigan University? Not really. Does a 3.8 GPA predict that you will do better or worse at managing a car-rental office than someone with a 2.8 GPA? Probably not. Does majoring in business predict that you will do a better job than an English major or a sociology major or a physics major? It’s unlikely.

The management at Enterprise are saying aloud what many employers know to be true. Bosses who require a college degree are taking advantage of a system that does the sorting for them. They understand that a bachelor’s degree is not really necessary for doing an entry-level job, and that whatever your educational background, you will require significant training to do well in that particular position.

I am not convinced that signaling is the only reason college is valuable, but it does make one wonder whether our efforts at increasing retention and putting programs online in ways that no longer require that students “show up on time in clean clothes” is undermining a central and important aspect of college.

My favorite academic novel is Kingsley Amis’ Lucky Jim. I reread it every couple of years.  My wife can always tell that I have succumbed to the urge revisit the book because it is the only book that makes me giggle out loud.

Each time I reread it, I notice something new.  This year it was my spring break travel book and it was this conversation between the book’s protagonist Jim Dixon (a first year history lecturer) and his friend Beesley (lecturer in English) that stuck out.

‘It’s the same everywhere you look; not only this place, but all the provincial universities are going the same way…go to most places and try to get someone turfed out merely because he’s too stupid to pass his exams—it’d be easier to sack a prof.  That’s the trouble with having so many people here on Education Authority grants, you see.’

‘How do you mean? The students have to get their money from somewhere.’

‘Well, you know, Jim. You can see the Authorities’ point in a way. “We pay for John Smith to enter College here and now you tell us, after seven years, that he’ll never get a degree. You’re wasting our money.” If we institute an entrance exam to keep out the ones who can’t read or write, the entry goes down by half, and half of us lose our jobs.  And then the other demand: “We want two hundred teachers this year and we mean to have them.”  All right, we’ll lower the pass mark to twenty percent and give you the quantity you want, but for God’s sake don’t start complaining in two years’ time that your schools are full of teachers who couldn’t pass the General Certificate themselves, much less teach anyone to pass it.  It’s a wonderful position, isn’t it?’

One of the conceits of the various reform movements, which of course includes assessment, is that there was once a time when grades were not inflated, when students were serious, engaged and well-prepared, and they learned lots of stuff in College.  Lucky Jim was written right after the war when Amis was a lecturer at provincial red brick much like the unnamed university where Dixon teaches.  Clearly the same types of concerns that surround standards, grades, completion rates, and student preparation were just as prevalent in 1953 as they are now.  No doubt if someone were to look though Peter Abelard’s personal diary medieval scholastics would be found to have had all the same concerns.

 

 

 

Huffpost has a long and interesting article on the threat posed to higher education by the proliferation of Online Program Managers. OPMs help colleges develop, manage and market online programs, usually master’s-level programs. In exchange for those services they typically pocket 50 to 60% of the tuition. The article is by Kevin Carey of New America and it includes a section on Bob Shireman’s efforts to regulate for-profits while he was in the Department of Ed.

The short version of the article is that the OPMs take the now discredited for-profit model and use their connections to legitimate non-profits to create a cloak of repeatability around their work.  They started by working with prestigious schools but have shifted to partnerships with financially troubled smaller privates and state schools.  In some these partnerships it’s getting harder and harder to tell whether the OPM or the college is the senior partner.

It’s a good article, although one of the OPMs has disputed some of it’s claims.  Carey is right that the OPMs prefer master’s programs because they are more opaque in their admissions practices than undergraduate programs, so those programs can be more “accessible” than undergraduate programs without affecting a school’s published admissions standards.  But more than that, my sense is that they prefer master’s programs that confer a credential the quality of which is irrelevant.  This is especially the case with master’s in education.  For a teacher to get the pay bump that comes from obtaining a master’s degree it does not matter whether the degree was earned in residence in a face-to-face to program that required lab time and real research or just knocked out online with the cheapest and least rigorous program available.  You still get the same raise whether you earned your degree at Cal Tech or from De Vry.

I wrote about this (Automatic Pay Raises for Teachers Create Perversive Incentives in Graduate Education)a while back for the James G. Martin Center.

 

 

 

 

 

An article in the March/April issue of Change: the Magazine of Higher Learning begins with this tidbit:

The assessment of student learning in higher education has been headed down an unproductive path for too long.  Not enough faculty and administrators engage in an assessment process that fosters cognitive and affective learning for all their students.

Yes, really.  And this is coming from authors associated with AAC&U, NILOA and AALHE.  People representing the exact organizations that have done the most to impose the bureaucratic form of assessment that is now the norm in higher education have belatedly recognized that assessment is unproductive.  But it’s not them, it’s you.  The good news is that they have a new plan and this time it will work.  Trust them. After all they have only been leading us down an unproductive path for twenty years now, so surely they are just the people to get the bumbling faculty back on track. What’s the new plan? We need to shift our thinking about assessment so that we focus less on “assessment of learning” and more on “assessment for learning.”  I guess now that we have the verbs in our learning outcomes sorted out it’s time to start thinking about prepositions.

How is that anyone still takes this seriously?

 

The Chronicle has posted two letters in response to my most recent article.  One is from Dan Sullivan and Kate McConnell who wrote because they felt I had misunderstood an article they had written.  I have previously posted a critique of the article in question.  The second and more recent letter is from Catherine Wehlburg.  She takes the argmentum ad lapidem approach.  I am wrong because my argument is just “silly,”  so there is no need to deploy evidence to counter it.

I also take issue with the statement that “learning isn’t the most important element of a college education.” Now, that’s just silly. Of course learning is the most important part of a college education. That is why students attend college — it isn’t just to collect course credits (though some students might think that it is). The point is to learn — new skills, behaviors, knowledge, and even new perspectives. And there is a great deal of data on what students are learning and how that learning is helping them be successful in the workforce and in life.

This just reinforces my sense that the arguments for assessment are ultimately based on people’s intuition about what college is about or ought to be about rather than thoughtful or critical reflection.

Below is a repost of 2018’s most read entry in the blog.  A highly revised version of it should appear soon in Cultural and Pedagogical Inquiry.

 

As someone who has been publicly critical of learning outcomes assessment for a long time, one of the questions I am often asked is: “If you are so opposed to assessment, what would you replace it with?” By way of an answer I have started resorting to this fable:

 

Imagine that you live in a Bronze Age village. You and everyone else in the village depend for your livelihood on subsistence farming, so you have a keen interest in the success of your crops. Because of that, you have developed a good sense of when to plant what crop, what types of soil work best with specific crops, when to weed, when to harvest and so on. It’s not scientific knowledge but it works pretty well. Still, you are always on the look out for ways of improving your yields. Continue reading “Repost: Stop Sacrificing Children to Moloch”

This a guest post by Deborah Borman of the Univeristy of Arkansas at Little Rock.  The resistance is strong in the Natural State.

 

I read your article with great interest, as it is the rare voice that stands up against the onslaught of rubrics in higher education.

I am unabashedly anti-rubrics. In my article, Degrading Assessment: Rejecting Rubrics in Favor of Authentic Analysis, 41 Seattle L. Rev. 713 (2018), I advocate to end the trend of using rubrics for assessment of legal analytical writing. As developed through the college admissions process and later integrated into the grading process, standardization in assessment in all education is historically discriminatory. While there are some andragogical uses for rubrics, the potential for discriminatory application and the underlying discriminatory history further underscore the potential damaging effects of using rubrics. Rubrics fail to promote either good learning or good writing. At the legal education level, law schools misconstrue the requirements of American Bar Association (ABA) curriculum standards by using rubrics to implement outcome assessment. The various standards for outcome and assessment cannot be fulfilled without implementing holistic evaluation.

https://digitalcommons.law.seattleu.edu/cgi/viewcontent.cgi?article=2523&context=sulr