Bad Assessment

A couple of weeks ago I wrote an article for the Chronicle (“College Finances are Being Eaten from the Inside“) that argued that OPMs are taking over some of the colleges with which they work. My article derived in a part from a recent report by Stephanie Hall at the Century Foundation, called “Invasion of the College Snatchers,” that resulted from some open records requests they made at about a dozen schools known to be involved with OPMs. (The report is awesome in that it sustains the “Invasion of the Body Snatchers” metaphor through to the end.)

In response to my article and to Kevin Carey’s earlier article in the Huffington Post, Kellie Otter at Georgetown, wrote an article called “Online Program Managers are not the Enemy.”

I found her response unpersuasive.

Just last night the Chronicle published my letter in response.

I hope to do more with this, especially as the Century Foundation is still making more open records requests, some of which are being resisted with astonishing determination.

I am just now embarking on a new project that has me reading about weights and measurement and their relationship to scientific epistemology. I just ran into this:

“When you cannot measure, your knowledge is meager and unsatisfactory.”

Lord Kelvin

“If you cannot measure, measure anyhow”

Frank Knight’s translation of Kelvin for the social sciences.

Somehow this is the best short summation of the assessment problem I have encountered.

Karin Brown, a philosopher at San Jose State University, has just published an excellent piece in the AAUP journal Academe. In it she argues that bureaucratically mandated assessment is fundamentally anti-intellectual.

Sample quotation:

The consequence of this bureaucracy means that I am forced to put SLOs on my syllabus that I did not write and that I find neither meaningful nor helpful. To make matters worse, I am then forced to write a report proving that I met these goals. In other words, I am forced to adopt a pedagogy that I consider anti-intellectual and fill out a report on it explaining how this anti-intellectual assessment requirement helped me to improve my course. Could this really be happening in a university?

She is also the author of an early critique of assessment that had not previously come to my attention.

If there is one thing I have learned from assessment it’s that it’s always important to use the right metric. In learning outcomes assessment that means that the thing you choose to measure must be easily measurable and sure to shine a positive light on your program or course.

Likewise, it’s important for both individual and institutional self-esteem to always use the right comparisons to your peers. When I meet an academic peer who has a better publication record than I do, I take comfort in the near certainty that on bike ride I could leave them far behind and gasping for air. On the other hand, among my fellow cyclists, I am the best published.

Occasionally this strategy fails when you encounter someone like this guy (teaches at Harvard and holds a world record for most pull-ups in a minute), but in general it a winning plan. As long as you set the goalposts just right, you too can be above average.

At the institutional level this works too. As long as you define excellence narrowly enough, your university is sure to be highly ranked in something.

You may rank 873rd for medical school admissions, but maybe some obscure website ranked you top 10 in affordable online degrees for veterans. Put it on the landing page.

I got to thinking about this after reading a parody on this subject at Michael Morris’ University Life site.

Best of the parody rankings:

—  Oberlin College:  Ranked #5 in percentage of philosophy majors who can distinguish between post-modernism, Post Raisin Bran, and the U.S. Postal Service (Foucault Institute Journal)

—  DePaul University:  Ranked #4 in percentage of deceased alumni who enter the Catholic section of heaven without a forced layover in Purgatory; ranked #6 in percentage of undergraduates whose prayers to God for assistance during final exams are answered directly by a member of the Holy Trinity rather than a low-ranking angel (Vatican Daily)

The Chronicle of Higher Education just published my latest article, “A Reason to be Skeptical of ‘College for All.'”

As is always to the case, the published version is shorter than what I submitted. The editors at the Chronicle are quite good and the article is surely better for having been cut a bit.

What got left out was a long paragraph that referenced Michael Lind’s work and linked to this article in The Bellows. It was reading his piece a couple of months ago that gave me the idea for the article.

The paragraph that was cut is here:

Michael Lind has argued that our fraught political climate is driven by conflict between two elements of the “overclass.”  On one side are the college-educated professionals and on the other is the small business elite (who also tend to have college degrees but whose incomes are less dependent on educational credentials).  Both groups’ politics are shaped by a legitimate fear of downward mobility and proletarianization.  The small business elite seek to reduce regulation and to minimize taxation of their businesses, while the college-educated elite favor social welfare policies like subsidized child care (which benefits two-income households), college debt forgiveness, and free college, all of which benefit college-educated professionals.  But the other thing that the college educated elite push for is the expansion of those parts of the economy that employ the college educated.  Lind observes that when protestors (most of whom have college degrees) talk about defunding of the police, they are calling for public sector spending to shift way from unionized, working-class police, who typically lack college degrees and toward non-unionized, college-educated counselors, social workers, and social services professionals.  So their proposed solution to the problem of police violence also dovetails neatly with their class interests.

It was Lind’s observation about the college-educated elites’ desire to expand the parts of the economy that employ them that got me thinking about the extent to which self-interest might shape the calls for College-for-All when they come from people who work in higher education.

It’s an excellent article and I urge you to read it. I have not read the book yet, but I expect it’s good too.

My last post featured a critique of the rush to move K-12 education online. The article in question was in The Bellows which calls itself Marxist. I noted that the I was struck by the conservative tone of that article. So today I have just run into a very similar article, though this one is more higher ed oriented, in the National Review.

When the Marxists and movement conservatives are all singing from the same hymnal, you know we have reached a rare bipartisan agreement: online education is a poor substitute for the real thing.

The Bellows (an online journal that claims to be Marxist, but seems deeply conservative in tone) recently published an article about California’s COVID-driven shift to online instruction for K-12 students. The author, a teacher, argues that what seems like a crisis to most of the people experiencing it, is also, for the tech industry and capitalism in general, an opportunity.

He notes that the Silicon Valley has contributed lavishly to support the tech needs of California schools shifting to online delivery.

However:

The tech industry has not made massive donations to medical and therapy services, which low-income students often receive through community schools. Likewise, there is no private backing for the state’s free grab-and-go meals program. Distance learning is a sleight of hand. Framed as a panacea, online education is actually the vehicle for a long-desired economic restructuring. 

Online schooling will generate a treasure trove of data tech firms can buy and sell. Free meals will not.

The digitization of schools is an initial step toward digitization of society as a whole. Just as the school bell schedule was designed around the factory model, so the current model of virtual learning is training affluent students for a life of self-directed work at home. It is training low-income students for a life of no work at all.

Online education is not just preparing students for the tech industry’s economy of the future. It’s laying the ground work for education without teachers.

When online curricula are fully built and courses can run themselves there will be no need for teachers’ labor. California teachers have become the Uber drivers of education: providing a temporary service until the technology that can replace them is ready for the market.

All of this is equally applicable to higher education. Online and remote education seems tailor-made to exacerbate inequality for students and to marginalize academic labor.

In a recent article in The Atlantic, Ian Bogost of Georgia Tech observes that there has been a near universal rejection of remotely-delivered college in the US. Students, parents, state legislators, university administrators, college football fans, and the president, have all demanded that college campuses reopen. Many people have, of course, noted this and most have taken it as an indictment of remote learning. Me for example. Bogost’s take is different and interesting. He says that our unwillingness to accept remote classes as an acceptable substitute for the traditional college experience is a sign that college has never been primarily about education.

From the article:

An education may take place at college, but that’s not what colleges principally provide. Higher education survived a civil war, two world wars, the Great Depression, and the 1918 Spanish flu, the worst pandemic the U.S. has ever faced. American colleges will outlast this crisis, too, whether or not they are safe, whether or not they are affordable, and whether or not you or your children actually attend them. The pandemic offered an invitation to construe college as an education alone, because it was too dangerous to embrace it as an experience. Nobody was interested. They probably never will be.

I suspect he is right about this. Unfortunately, I’d be willing to bet that over at NILOA they are reading this and thinking,”We better double down on co-curricular assessment if that’s what college is really about.”

I have been critical of dual enrollment (also called concurrent enrollment in some states) for a while. Education Next published a long and interesting report on the subject last week. It seems that my concerns about the rigor of the courses, the level of preparation of the students involved, and the unequal availability of these courses are more widely shared than I thought. Texas, which has been at the forefront of the dual enrollment movement is starting to have second thoughts.

Perhaps the most interesting part of the article concerned the low quality of most of the research on the subject.

From the article:

Studies in Texas and elsewhere have found that dual enrollment boosts key student outcomes, including high school graduation, college enrollment, college persistence and completion and time to degree.

Still, most of the evidence for dual credit has come from small-scale, single-state studies that haven’t controlled for selection bias: the possibility that students who choose the classes are more motivated, and more skilled, than those who don’t. In a 2017 literature review,
the Institute of Education Sciences found only two studies that met its standards for rigor, and both focused on early-college high schools.

In Texas, a pair of recent studies has ignited a debate over the benefits of dual credit. One, by the University of Texas system, found that dual credit students had higher GPAs than students who came in without college credits, and were three times as likely to graduate in four years.

The other, commissioned by the Texas Higher Education Coordinating Board, found that dual credit increased the odds of graduating for traditionally advantaged students only; for lower income students, it actually hurt their chances of completing college.

One surprising finding in the system’s study was that dual credit did not reduce borrowing for the majority of participants. While students surveyed by the researchers cited “saving time and money” as a major motivator for taking dual credit courses, only those who entered college with 60 or more credits— the number required for an associate’s degree—saw any significant reduction in student loan debt.

The study apparently does not offer an explanation for why dual enrollment might hinder lower income students from finishing college, but I wonder if going directly into upper-level courses because you have already done most of your general education in high school, gives vulnerable students less time find their sea legs.

Scott Newstok, How to Think like Shakespeare: Lessons from a Renaissance Education, Princeton, NJ: Princeton University Press, 2020.

This is a book that does not fit neatly into any established genre.  Above all it’s about how to think and how to teach people to think, but it’s not a how to manual. At a time when higher education is stampeding toward everything shiny, new, and up to date, this book is deliberately backward looking.  It looks unapologetically to the past for ideas, models, and habits of mind that Newstok contends are just as relevant now as they were in Shakespeare’s time.  Even the form the book takes, that of a commonplace book (though it has more narrative and commentary from the author than is typical of the genre), is deliberately anachronistic.  

For our readers who are not in the humanities, a commonplace book is a collection of excerpts of other people’s work.  They go back to antiquity, but became intensely popular during the Renaissance and only faded from use in the last century or so.  Many were made for personal use only—a type of private intellectual scrapbook of compelling ideas.  Other commonplace books were less personal; they were complied with readers in mind and were published. 

Because we are not accustomed to the commonplace book genre anymore, the book takes a little getting used to. At first, I did not realize that the italicized passages were quotations.  In fact, in the first couple of pages I found myself thinking, “Man, he’s kind of overdoing it with the all the italics. This looks like a book that an overzealous undergrad went nuts on with a highlighter.”  Because there is so much quoted material (it’s the nature of the genre), all of which is cited in footnotes, your eye is dragged to the bottom of the page constantly.  It’s not the sort of thing we are accustomed to reading. After a while you get used to it, but at first I found it a bit jarring.

The whole premise of the commonplace book is that the ideas of others—established, time-tested ideas—have value.  In fact you could call the book derivative. It is consciously and intentionally builds on a scaffolding of ideas erected by others.  That notion, the idea that there is value and wisdom in the past, is central to this book.  In fact the book seems to be reacting against almost every new direction in higher education.  But it is not just a conservative complaint about the dangers of new stuff; instead it celebrates the value of traditional education and shows how the habits of thought that a real liberal education offers continue to have value and relevance in a changing world.

Readers of Bad Assessment will like Chapter 2, which is a critique of the utilitarian view of education.  In it, Newstok describes asking his daughter, aged seven, if she had learned any new words at school:

This otherwise vivacious child contemplated a moment, looked at us coldly, and whispered: assessment.

In this same chapter he points out that Shakespeare’s limited formal education, which consisted mostly of drilling on Latin grammar, would seem to be the epitome of useless learning, even in his time.  Nonetheless, it managed to prepare the young Shakespeare for a career as a professional playwright, a job that did not even exist at the time he was in school.  Paul Banksley, the satirical advocate of 22^nd century skills, should take note. 

Newstok’s point is well taken.  We tend to assume that our problems are new ones, but as he shows us, people in the past have had to deal with very similar issues and dilemmas.  Thoughtful people in earlier ages produced ideas and ways of thinking that we can use now to address our own problems.

Inevitably in reading a book like this one thinks of quotations that ought to have been included but were not. For me it was Confucius’s statement that: “The accomplished scholar is not a utensil.”

So, read the book. Then send me the quotation that you think should have been in it, but was not. If I get enough, I’ll create a “commonplace page.”

Note: This is the second book review that has appeared in Bad Assessment. If anyone has ideas about books that ought to be reviewed or would like to review a book for us, let me know.