At least a dozen people this week sent me a link to a new web site which attempts to synthesize data about the “completion rate” of massive open online courses (MOOCs) across platforms and institutions, and to correlate these data points with characteristics of the courses. Given the large amount of higher ed press highlighting this type of MOOC data, it’s hardly surprising to see this kind of effort.  Since I assume I am the source of some of the data points in the chart (at least indirectly through previous blog posts), I wanted to weigh in on the limits of this approach to defining best practices for MOOCs or measuring quality (which seem to be the two implied goals of the analysis).

Why is it not reasonable to infer that if a student enrolls in a MOOC and doesn’t finish, that something is wrong with either (a) the student or (b) the course (which is basically what we would usually assume in the analogous situation in a campus course)?  There are lots of reasons, but here are my top 5.

1. They never started the course in the first place. Based on data about Duke’s Coursera courses, anywhere from 1/3 to 1/2 of students who enroll in our MOOCs never come back and log in after the course begins. To quote Sophocles Antigone, “When I have tried and failed, I shall have failed.”
   
2. They never intended to finish. MOOCs are a new phenomenon, so it’s not surprising that they attract a lot of people who are interested in learning (or in Duke, or in the faculty member, or in the topic) but aren’t very concerned about completing the course. General curiosity about online courses is an important reason for enrolling for about 1/3 of our MOOC students. In one of our courses, Earning the Statement of Accomplishment was rated as very or slightly unimportant by at least half of the students, and in that same course, was very important to only about 10% of them.
3. Anyone can sign up. Of course this is the point of MOOCs, but I think we all agree that there are a lot of things that narrow the pool of students before a campus course begins including (but not limited to) a secondary school education,  the admissions office, the bursar’s office, whether or not they’ve passed the prerequisites as defined on our campus, and the number of seats in the classroom.
4. The incentives for finishing are unclear…  The jury is definitely out on what kind of value a certificate or Statement of Accomplishment per se from most MOOCs has for a student in the short and long term. I still like having mine posted on my office wall, but I don’t list the fact of having completed any MOOC on my CV (yet). By contrast, students taking a campus course have quite a few incentives to finish (and definitely some consequences for not finishing, both financial and academic).
5. …And a lot of the students are really busy with more important things. In our largest course, about 2/3 say they work either full or part time, with full time  outnumbering part time 2:1. About 1/3 are currently students (including pre-college, undergrad and grad). And quite a few are students who work.

None of this means that it doesn’t matter how many students finish, or that we shouldn’t investigate whether there are MOOCs that are more effective than others in retaining students who want to complete them. It’s exciting to see data from across platforms and institutions about the MOOC phenomenon. And I love the creative crowdsourced approach to getting it, since many of these courses haven’t made data readily available about their outcomes. But if we’re going to gather metrics, let’s find some that seem to have more meaning behind them. Here are a couple of suggestions for how we can talk about the outcomes of a MOOC. If you have others, please add your comments.

1. What proportion of students are satisfied with the learning achieved relative to the effort/time invested?
2. What is the completion rate for the course as a proportion of those who signal that they intend to complete AND begin doing the graded work?
3. What is the sustained participation rate in the course – how many students log in every week and participate in the course by watching videos, taking quizzes, and/or participating in the online community forums?
4. Last, but by no means least, what kind of learning outcomes are achieved?