Confessions of an Assessment Skeptic

At a recent AACU conference and President Obama's State of the Union address, there was more talk about assessment (okay the president did not really talk about it directly) and how we need more information to evaluate educational institutions and determine which ones are good and deserving of more public money.

I must admit being ambivalent about assessment. Although I don't rely on quantitative data or analysis in my research, I think that most proponents of assessment oversell the value of this data/analysis and underestimate the real cost of and effort in producing this information. I must confess, I am an assessment skeptic! Despite being pretty engaged with assessment over the past decade, I am fairly cautions about what it can accomplish and how it can and will be deployed.

I am sure I am missing something important here, but I think the general argument is:

If we government, employers, prospective students, and the parents of prospective students have more information about outcomes, then these people will make better choices about their education.

This argument seems to rely on a number of unspoken assumptions:

(1) students, employers , and governments make rational decisions related to college.
(2) academic achievement (as opposed to convenience, potential earning power, location, athletics, campus social life, etc) are the main factors in shaping decisions about college.
(3)ll institutions (and/or majors) have similar educational goals and thus can be usefully compared.
(4 the primary concern of campus administrators is academic quality (as opposed to budgets, fundraising, enrollment growth, etc)

There is a part of me that really wants to believe in assessment and the promise of a rational conversation about student learning outcomes (after all I am a notorious idealist).

However, my pragmatic side looks at the rest of our capitalist society and thinks about the general irrationality of our stock market and our capitalist system (e.g. the housing and dot.com bubbles - not to mention CEO pay) and wonders why these other areas of our economy have failed to adopt the kind of assessment models proposed for colleges and universities. Just as an aside, I wonder what Walmart (or Target, Kohl's, or K-mart) would be like if consumers applied some sort of assessment measurement tool to its products. I think that most consumers value efficiency and economy over actual achievement - isn't that the whole business discussion about price points and convenience? My guess is that most consumers of higher education rely on their own formula that takes into account a series of factors from convenience to social life to locale to major to reputation to actual quality.

OK, I have probably gotten off track here. It seems to me that the call for assessment is really a an acknowledgement of the failure of the employment market and the behavior of college students (and their families). Neither employers nor college students make good choices and the government is left paying an increasing amount of money to more colleges each year.

I have come to the very tentative conclusion that the assessment movement is failing for a pretty classic fallacy: confusing a means for an end. Assessment proponents appear to believe to have a transformed the process of assessment (or some kind of rational rating or evaluation system) into goal in and of itself. The problem is that the mere existence of information does not mean people will act on in it, they really want that data, or that academic excellence is the main point of college. Moreover, like any market, some sellers focus on low-quality, low-cost education while other sellers will inevitably focus on a different market segment that desires high-quality, high cost "luxury" goods.

My second tentative conclusion is that assessment proponents have failed to understand the nature of the scientific method and statistical analysis: (1) statistical data must be interpreted, (2) the results are only as strong as the inputted data, (3) they frequently give you only probabilities, not eternal verities, and (4) past performance (a la the stock market) does not determine future results, especially as colleges and universities respond to all sorts of situations. Consider how economists continue to struggle to understand and the predict the economy despite a bevy of statistical models for evaluating the health and future of the economy.

I don't think assessment is necessarily a bad thing. (In fact, I have improved my teaching and/or learned stuff about what was happneing in my clas). I just don't thing it is a silver bullet that can cure everything that ails higher education. Moreover, it is a pretty convenient whipping boy or straw man because who wants to defend academic mediocrity or a second-class education? However, the underlying premise is that if we only had more data, then folks would stop making bad choices about higher education. If this were true, then how come consumers continue to make so many unhealthy and inefficient choices when it comes to everything to fast-food to cigarettes?

I really believe that higher education can improve. I am just not sure if the assessment movement is the always the right way to promote quality! It also takes faculty away from what they do best: profess their fields.

I really wish I had the answers to the questions that the assessment movment raises, but I don't. And neither do they.