Indifference to evidence

IHE has an article today looking at a study on how universities make technology purchases.  The short version is they seem to rely more on gut instinct and hype than on actual research on what works and what does not.

From the article:

…researchers found a wide range of approaches to selecting new technology, but few made use of strong scientific evidence to show whether that technology has a beneficial outcome.

Peer-reviewed external research was mentioned by just a fifth of interviewees.

Incorporating externally produced research into decision-making processes in higher ed is “difficult,” the authors said.

Fortunately, as with assessment,  bad, internally produced research is cheap and easy to come by.

“Externally produced, rigorous research, such as randomized controlled trials (RCTs), is often expensive, may take too long to inform pressing decisions, and is often difficult to generalize to a decision maker’s context,” they said. On the other hand, “locally relevant, internal research, such as faculty and student surveys or pilot studies, may be more feasible to implement and may provide more timely information” but may be “less reliable for providing solid answers to questions about effectiveness for improving academic outcomes.”

Hmm…sounds like technology purchasing is using what is known in the assessment business as “actionable data.”  It may not be valid or reliable or meaningful, but it’s available and lets you do what you wanted to do anyway which was buy some cool new software or make faculty fill out forms (why not both?).

“I was expecting to find more rational decision-making processes,” said Fiona Hollands, associate director of the Center for Benefit-Cost Studies of Education at Teachers College, who co-wrote the study. “I thought more institutions would start with a need or a problem and then figure out the solution, rather than starting with solutions and finding problems to solve with them.”

“There are places that literally scan the market looking for new innovative technologies, bring it in, play with it in their technology units and then try to find a use for it on campus. I find that a bit absurd,” said Hollands.

I have actually spent some time thinking about software purchases and why they seem to be the go-to solution for all problems on campus.  I have concluded that most of the big problems on campus (recruitment, retention, showing that students are learning) are so complex and involve so many externalities, that they are virtually insoluble without real structural change.  Most administrators know this, but also have to appear to be addressing these problems.

Buying new software is the easiest way to signal that you are taking a problem seriously.  That technology purchases  create multiple opportunities to add lines  to CVs might be a factor too.  Someone gets to lead the town meetings and workshops where “stakeholders” discuss the problem.  Then someone gets to be the person who oversees that actual purchase of the software. This is the best job because it involves being wined and dined by vendors.  Then once the software has been purchased someone gets to oversee the migration, implementation and, of course, training.

At this point the software has already done its job.  If anyone asks about retention or assessment or whatever, the vice president of whatever can say that steps have been taken.   He and a bunch of other people will have new lines on their CVs.  Best of all,  if they bought the right software they may have shifted most of the burden of work associated with the issue to people outside their unit who can now enter all the retention or assessment documnetation directly into the new software themselves.

So maybe it’s not that surprising that universities are not going to great lengths to test whether the technology they purchase solves the problems it purports to solve.  That may not have been the point of the purchase in the first place.