Campbell’s Law and Community College Completion

There is an interesting and delightfully cordial debate going on at AEIdeas (a blog at American Enterprise Institute) about CUNY’s attempts to provide programs that help community college students complete in a timely manner.  One participant Angela Rachidi has looked at CUNY’s programs and sees their apparent success as an example of how community colleges an develop programs that help at risk students have improved outcomes.

CUNY ASAP doubled graduation rates when compared to a control group, 42 percent vs. 22 percent after 3 years — similarly impressive.

The other interesting feature of this is that CUNY actually used random assignment of students to the ASAP and START (another CUNY remediation program) programs so unlike learning outcomes assessment there was actually a control group of students who were not in either program. Rachidi wants to seem more of this:

Finally, more programs targeting low-income people need to be rigorously evaluated using random assignment. Prior analysis of data from CUNY ASAP suggested that students were benefitting, but no one was sure whether these gains were due to the program or something else. Similar concerns would exist had CUNY Start not been rigorously evaluated.

Another group of AEI education types are not convinced and see Campbell’s Law at work.

Campbell’s law is:

“The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.” Put simply: “When a measure becomes a target, it ceases to be a good measure.”

Source: Frederick Hess in Education Next

They point out that despite the random assignments of students, there is a critical variable that skews the outcome:

CUNY’s Accelerated Study in Associate Programs (“ASAP”) provides intensive advising, tutoring, and financial support to students. All good, but one key ASAP requirement is that students remain enrolled full-time. Well, full-time students typically complete at much higher rates than part-time students (who may struggle with family or work obligations). As MDRC notes, ASAP thus selects for students who are more likely to make it through. Indeed, program requirements geared to such students could well wind up narrowing the opportunities available to students who can’t afford to take time out of the workforce or enroll full-time.

They conclude with this caution:

If we have learned anything from No Child Left Behind and the more recent high school graduation rate scandals in K-12, it’s that Campbell’s Law applies with a vengeance when it comes to education. If educators know they’re supposed to move completion rates, they’ll find ways (good and bad) to move them.

That means caution is warranted. These programs may prompt institutions to lower academic standards, find ways to pass students along by pretending they’re learning (what K-12 types call “social promotion”), or wind up enrolling students they view as more likely to complete. We’ll be best served if we tackle the college completion problem with our eyes wide open, and avoid the temptations to celebrate promised elixirs until we know more about potential side-effects.