Sullivan and McConnell, the VALUE Rubrics and the Future of your Assignments

I recently published an article (paywalled) in the Chronicle that uses data generated by an article in Change: The Magazine of Higher Learning by Dan Sullivan and Kate McConnell.  The original draft I sent to the Chronicle included a critique of that article and another article in the next month’s issue of the same journal.  My editor asked me to cut that part of the article.  Below is some of the material that was edited out:

Something of value may have just emerged from the VALUE project. A research project using the VALUE rubrics has produced results that call into question the utility of rubrics in general and challenge the foundational assumptions on which the learning outcomes industrial complex is built.

VALUE stands for “Valid Assessment of Learning in Undergraduate Education” and it’s a project of the American Association of Colleges and Universities (AACU). The name itself suggests a tacit admission that what passes for assessment at most institutions, the every-program-every-year-mandatory-loop-closing-based-on-fragmentary-data-derived-from-small-samples, is not really valid. The VALUE project looks like an honest attempt to address that shortcoming. The central and best known feature of the VALUE project has been the creation of sixteen rubrics that are meant to address outcomes that the architects of the project consider that “all students need for success in work, citizenship, and life.” These outcomes are things like critical thinking, ethical reasoning, quantitative literacy, problem solving and so on.   The rubrics are, in the language of the creators, “transdisciplinary” in that the critical thinking rubric should be able to assess that quality in a student’s work whether that work was done in an English, philosophy, or physics class.

One could easily argue with the particular outcomes they are looking at or the way they define the qualities they are looking for or even their assertion that every student needs these sixteen abilities to succeed in work and life, but the fundamental idea of the VALUE rubrics makes sense. If we must do assessment, rather than doing ad hoc fake research projects every semester, it would be much better to sample actual student work using carefully developed and tested rubrics that would allow comparisons across programs and institutions.

So I was interested to see an early example of the use of data generated by the VALUE project in an article by Daniel Sullivan and Kate McConnell in Change: The Magazine of Higher Learning. In this they describe a project that involved looking at nearly 4000 pieces of student work from 14 institutions. They used the critical thinking and written communication rubrics each to score about 2000 pieces of student work, so despite the transdisciplinary nature of the rubrics they seem to have used one rubric for half the items and the other for the other half. The rubrics produce scores on a 1 to 4 scale that ranges from benchmark (1) to capstone (4). The scores generated by the whole exercise are pretty much what you would expect. Most student work falls in the middle of the scale. For critical thinking 12% got 4s and 13% got 1s and the rest were in the middle. Written communication skewed a bit higher with 23% getting 4s and the rest in the middle.

Things get interesting though when they break the scores out by credits earned. If you separate the scores by the year of the students whose work was scored, you find that mean growth in their scores is only 0.2. For written communication it is only 0.18. Perversely seniors score slightly higher than first year students but lower than sophomores and juniors.

This is a problem. If the rubrics really measure students’ ability to think critically or write effectively and students get better at these things while they are in college, scores should go up over time. But here they don’t.

The authors then bring into play the difficulty of the assignments.   Some, but not all, of the samples were rated for their difficulty by the instructors who submitted them to the project. The authors find that assignments that were rated as more difficult elicited higher scores than easier assignments and that if you look only at the hardest of the critical thinking assignments you can show modest growth of 1.14 points from the first year to senior year. But the easier assignments show barely detectable growth. For written communication the pattern is reversed and the hardest assignments produce the least growth and the easiest the most, and even those only show half a point of growth. My guess is that this part of the study contains more noise than signal. The samples sizes are much smaller because so few assignments were rated for difficulty and the 8-point scale used to rate the difficulty of the assignments seems even more subjective than the rubrics.

[I am no expert on quantitative research, but this looks a bit like p-hacking to my untrained eye.  It’s not clear that the original intent of the study was to look at assignment difficulty and that difficulty only came into play once the ambiguous and probably disappointing results of the broader use of the VALUE rubrics became apparent.]

Despite results that seem ambiguous at best about whether the VALUE rubrics have effectively detected any learning and nothing that even hints at what might have caused any of the learning that might have been measured (why do none of these studies every try to use as a control a group of non-college students of the same age?  A cohort of military recruits seems like an ideal comparison), the authors conclude that this is evidence that more challenging assignments lead to more learning. This may be more sophisticated than the typical assessment project, but it’s still grounded in assessment culture, so the authors have identified some easy and cheap fixes that can be imposed on faculty. (Assessment seems never to identify problems that are structural and might require money to fix.) Thus the statement that, “Insisting that faculty give assignments that are both appropriately demanding and intentional about higher-order learning goals as well as disciplinary content learning goals…is not only effective at improving students’ higher-order learning, it also represents an affordable, reasonable strategy for enhancing student learning.”

My guess is that assignments are the next big thing in the learning industry and that Sullivan and McConnell are thus primed to see their project’s data as evidence of the importance of assignments. In fact the next month’s issue of Change features an article with the title “Fertile Ground: The Movement to Build more Effective Assignments.” It features an info graphic with a flashlight labeled “assignments” which is illuminating a bunch of bubbles with terms like “scaffolding” and “evaluative criteria” in them. I have no idea what this supposed to tell me, but I am confident that it will appear in hundreds of PowerPoints at faculty conferences next fall as assessment directors explain the urgent need for faculty to create “intentionally designed assignments” and to invite “additional stakeholder groups (for instance student affairs professionals and local employers) into discussions of assignment design…”  Yes, it really says this.

I don’t think that Sullivan and McConnell’s data are telling us anything meaningful about the value of certain types of assignments, but I do expect that we will be hearing more about assignments in the future.