Meaningful assessment & avoiding black holes

Working off of a couple of other blog posts, notably this one by Historiann, Dr. Crazy argues for a view of assessment in higher education that gets out from under the usual pitched battles between those who see a need for “accountability” and those who want faculty to retain/reclaim control over curriculum. Both sides agree that students are not learning as well as they should be, but disagree over the nature of the problem and the role that external constituencies should have in shaping the direction of higher education and how you assess or determine whether learning is happening or not. For the former group, appeasing external constituencies, boards, legislators, accreditors, taxpayers is paramount, and for the latter, it is an unwanted interference, at least at the level of determining what students should be learning.

Historiann counters in the first comment to Crazy’s piece that assessment in many places is already happening, but that the data is generally ignored, especially when it is inconvenient, i.e., when it suggests that students would be better served by additional faculty hires, smaller class sizes, etc., or when the only function it serves is to show external constituencies that an institution is “serious” about holding faculty accountable. In that case, all you really need is a file cabinet, in an office or on a server, that you can point to when someone starts asking questions. With so many institutions understaffed, and looking at increasing enrollments, there are more serious things to worry about than “accountability” and “assessment”.

While I admire Dr. Crazy’s will to find ways to make assessment work for faculty, and most importantly, students, I am more in agreement with Historiann, at least in the sense that I think that, in too many cases, the whole process has been corrupted by having been associated with satisfying external constituencies, and from a faculty perspective, administrative imperatives tied to those constituencies, and not with actually improving student learning.

That point, that assessment needs to be about students, both in the sense of seeing them as active agents in their own learning and in making their learning the point of the process, is Dr. Crazy’s strongest argument for faculty taking a constructive role in crafting assessment regimes, and not just chafing under the pressure to take part so that administrators can look shiny for boards, accreditors, legislators, and the public.

As far as I can tell, in situations where faculty have initiated programs to assess study learning within their courses, majors, and minors, there is value for students in terms of curriculum reforms that help to address deficiencies and student needs in relationship to what faculty think students should be learning in their fields. I’ve seen this on my own campus where certain departments, for whatever reason, were out in front of the current mania for assessment.

The problem is that for other departments the assessment discussion has been started not as a faculty-student issue, but as a administration to faculty matter. What you end up with in terms of instruments, ends, and what counts as data looks very different when the conversation begins that way. Even more problematic are those circumstances where a faculty have tools for assessment in place, but is dismissed because it doesn’t fit into the right kinds of boxes for administrative review. In our last round of accreditation, every department in Social Science, even those with theses and senior capstones, and where faculty could document how those experiences had been used to improve curriculum for students, was told that they lacked adequate mechanisms for assessing student learning.

This is where Historiann is right: many faculty already do assessments of student learning. Where assessment becomes “makework” is where we have to turn that into some kind of report or document that serves administrative goals related to “accountability”. I have my own experiences with the Black Hole of Information, sometimes never even getting to the point of reporting the results of an assessment activity, but simply submitting the instrument used without ever being asked for the data.

(And one side issue here is how the terms of assessment are constantly changing. I can look at my syllabi and track the shift from “objectives” to “outcomes” and now accreditors in our region want every institution to have “core themes”, and have changed both the accrediting standards and the schedule for visits. As a consequence, as faculty, we get an ever evolving, and not entirely rational, set of requests from our Dean for this or that kind of assessment using this or that kind of form, most frequently, with great desperation near the end Spring term.

Which just goes to Crazy’s point that this should be about students, and it isn’t).

I was thinking the other day how my class blogs, especially the one I have for my intro classes, are treasure troves for assessment of student learning. I mine a lot of information out of those discussions regarding what my students are and are not getting out of the course, what is working for them and what is not, and combined with in-class assessment I do on a weekly basis, I am continually generating notes on how to change or improve the course.

To the best of my knowledge, though, I cannot satisfy administrative requests for data by pointing them to the blogs. And I don’t have the time or energy to digest the useful stuff for others, let alone in some kind of format or medium that is bureaucratically useful. Does my university have a person or department who could or would do that kind of work? Of course not.

What we do have is a software service that will pull together all manner of standardized and quantitative data for interested constituencies to peer at. The extent to which assessment has become an industry also plays a role in the corruption that has infected this issue, perhaps irreparably, at least when it comes to getting us to where Crazy would like everyone to be.

One of the other commenters at Reassigned Time 2.0, Susan, remarks that one area where formalized assessment activities of the kind currently required of faculty has been useful is in thinking about curriculum in a programmatic way – what do we want our students to be learning and what role do different courses play in achieving those outcomes? That’s a useful conversation to have.

But I also agree with Earnest English that, while statements of learning outcomes, the usual first step towards assessment in the current parlance, is useful on a macro level for programs, and for individual courses, on a day-to-day, student-to-student basis, I don’t know how much this process matters, or the extent to which it obscures more than it reveals. As this commenter notes, what any individual “gets” out of a class is going to be highly variable, maybe it corresponds neatly with the predetermined outcome and maybe it does not. What I do know is that I rarely think about these outcomes after I’ve finished writing a syallbus, and am in the thick of actually teaching actual students.

Of course, I can always take what happens in the course of actual teaching and make it conform to statements of outcomes if I want or need to. I can make this easier by seeding my assignments and class materials with keywords taken from the outcomes, but, at a basic level, what does that provide evidence of except my ability to manipulate language? I think what Earnest English is getting at is that it is very difficult to account for what kinds of knowledge students will take and make from their classes, and, however useful it might be to set a few marks to meet as far as what we think our students ought to be learning, it would be a mistake to treat learning as a simple matter of hitting those marks. More to the point, it is a mistake for faculty to give into a system of assessment that rests on holding faculty “accountable” to the kinds of simple statements of outcomes with which we litter our catalogs and syllabi.

In the spirit of Dr. Crazy’s appeal to get out of the morass of faculty vs. admin vs. everyone else, to me the only helpful thing would be for a time out on all official demands for assessment so that faculty, who need to catch up to bureaucratic curve, can do so in a way where we get to think about students first and admin and everyone else second, or even as afterthoughts. And then admin and everyone else can come up with the means by which that work is translated into something that meets their needs.

My university is in the same situation as Historiann’s right now. We have skyrocketing enrollment, but with a tenure track faculty that hasn’t grown significantly in a decade or more. In my division, at least, adjunct hires have contracted over time, even to replace faculty on sabbatical or with course releases. We have administrators scrutinizing registration for low enrolling upper division courses to cancel so that faculty can be reassigned to teach higher enrolling lower division offerings. This is how we are trying to deal with our increased enrollments instead of hiring new faculty, a solution that undercuts one of the university’s other goals, which is to have students complete their degrees in four years.

That software service I mentioned costs money. Doing all of the “makework” to fulfill system and accreditor demands for data costs money or faculty time that, as above, we don’t have. Like Historiann, I think that most faculty have more to worry about than doing assessment in the properly bureaucratic manner, or to generate data and instruments that simply disappear into a black hole in the Dean’s office.