Summative Evaluations
Last week on the MOOC Research Google Group the following post came up by Alan Selig (reproduced here with permission):
While I don't have the time right now for an immediate answer, I intend on coming back to this later on this week. I also wanted to get it out there because I think that some form of evaluation is important - both learner assessments if you are accrediting people in your course and course assessments to make sure goals were realized. I am interested in what other people think about this issue :-)
I just got around to listening to the presentation by Tony Bates. Toward the end Stephen made reference to the surveys that many institutions give to students at the point of graduation. Not surprisingly these are usually very favorable, as the respondents have self-selected according to their favorable feelings toward the school.
It made me wonder (again) how we get evaluations in a MOOC from those who only participate at certain points in the overall schedule. Especially since we want to affirm that sporadic or episodic participation is a successful approach to MOOCs, how do we get summative evaluations when the end point for an individual participant can be anywhere during the course of a MOOC, or event beyond the official end of it?
While I don't have the time right now for an immediate answer, I intend on coming back to this later on this week. I also wanted to get it out there because I think that some form of evaluation is important - both learner assessments if you are accrediting people in your course and course assessments to make sure goals were realized. I am interested in what other people think about this issue :-)
Comments
Organizers would be interested to know how people used a MOOC, what made it easy or difficult, how they set their goals, organized their work, interacted with people and materials and so on. This information could come from a variety of sources including data mining, students self reporting, more organized or formalized "cluster based" reporting including peer evaluations.
You give two external reasons for wanting to evaluate students: course development and management and accreditation.
As a course developer I'd be interested to know about usability, coherence, personal distance, engagement and so on. I'd also like to have some personal metrics: learning styles, personality profiles, comfort with technology etc. Learner maintained records such as blogs or electronic portfolios would also be useful, though many students would probably need to be trained to us them effectively and that may render these tools less "authentic."
Accreditation is much more complex, given current structures. We could revert to an older form of accreditation based on an apprenticeship model. This relies more on personal than institutional authority and would be more acceptable in some cultures than others. Unfortunately, Western culture deprecates personal authority in favor of institutions, which then concatenate into huge corporate structures leaving little value other than raw authority. Getting institutional cultures to accept the notion of apprenticed accreditation may be difficult.
I don't see it as a "problem" yet, as OER and MOOCs are still young and we are all experimenting. And before we jump to solve a problem I think we need to understand a lot more about the overall ecology in which the MOOCs is growing. That said, I don't deny that discourse on the topic is warranted. And one way to discuss it is from the view of the stakeholders. (learners, facilitators, institutions, employers...)
I suspect that there are already a number of ways learners are dealing with assessment. After all learners may be motivated to document or legitimize the time they have spent engaged in this enterprise. I myself have taken two MOOCs "for credit," PLENK2010 and Change11. In both cases I am paying my institution for the credit, I proposed an independent study to my advisor, and _I_ proposed the outcomes and criteria by which my outcomes would be measured.
Do you know if George or Stephen have any students that are taking the course for credit? I understand that in other courses (Alec Couros's for instance) a small number of students are paying customers, with a predetermined set of objectives and assessment strategies. This technique may for the time being meet the institutional needs but in the long run may not serve the student's emergent learning goals.
And what of employers? What will learners use to convey their skill and knowledge in the absence of diplomas and grade reports as more and more of our learning occurs in informal, independent or non-traditional online environments? Let's see where the McArthur Foundation digital badge project takes us.
http://www.macfound.org/site/c.lkLXJ8MQKrH/b.4196225/apps/s/content.asp?ct=11221065