Evaluations in MOOCs
Since Week 8 of Change11 has yet to start (materials seem no where to be found) I thought it would be worth going back and commenting on my previous post on MOOC summative evaluations. The question posed by Alan Selig was how to get summative evaluations from MOOC participants when you have dip-in-jump-out model for most MOOCs; I say most because at least the language MOOC I will be designing won't be so much of a dip-in-jump-out -people could, but if you are a true n00b in the language you might have problems if you skip a week since language is cumulative.
In any case, my first step would be to ask do we need an assessment and why? Now, just to be clear, I am not saying that we don't need assessments in MOOCs, but rather we need to rethink the assessment (as both Mark and Jane commented on in my previous post). What do we aim to accomplish with an assessment? Is it for us (organizers) or for the them (the participants)? What form of assessment is best suited for the pedagogical design of our MOOC and the types of participants we have?
Once those things are sorted out, then we can start thinking about specifics. As Jane said there would probably be two "tracks" of assessment, those taking the MOOC for credit, and those who do not (like me). My feeling is that those who take the course for credit have slightly different goals and objectives than those who do not. Those differing goals will be evident in what each type of participant is looking for in a MOOC. Jane brought up the idea of Open Badge, which I think is a wonderful idea; personally as a MOOC participant I would like to have a little something, a momento in addition to my blog posts, that I participated in a MOOC. MobiMOOC had that, Inge did send out (digital) certificates of participation to those who participated above a certain threshold - I thought that was pretty cool! I wasn't participating in MobiMOOC just for the certificate - I was doing it for the knowledge, but it was a nice additional motivating factor.
Of course a potential issue with badges is exactly what I came across in my other blog post on open door policies and analytics. If you specify that you need to be active in X-many sessions and [insert participation criterion and quantity] to get our cool badge people might cry out that the bar is unfairly high. The devil's advocate of course might say "well if you are intrinsically motivated to only participate (as we have defined participation) in fewer than [quantity], then what does the badge matter?" I honestly have not answer for that - it's part of a longer discussion and debate that I am sure I will continue to have as this MOOC goes on and in future MOOCs :-)
So, back to assessment. At the risk of death by assessment perhaps one way of assessing the modules in a MOOC would be a weekly assessment that takes place one week after the module is done - so "week 6" would not be assessed at the end of "week 6" but rather at the end of "week 7," this would give people enough time to get through week 6 and digest what has occurred on that topic over the past couple of weeks. This is sort of similar to a taking of the pulse that one of my professors did at the end of each weekly module. She would have us do a short 5 question assessment on to module to see what we thought about it and how it could improve. This might be an option for MOOCs - performing formative assessment.
I think summative assessment is a bit of a problem if you are talking about non-credit MOOCs simply because people will dip-in-jump-out. At the same time, what are you assessing at the end of the course? At my institution summative assessments seem to be more faculty assessments, in other words "how did this professor do?" In a MOOC this isn't an issue because each week we have a new facilitator or guest. The other use for formative assessments at my campus is to see how this course fits in with the rest of your course of study or how it fits within the organization's published goals (like being inclusive and civic minded for example). Well, a MOOC will not be part of a MOOC curriculum (but that could happen!), and as far as I know MOOCs aren't offered by just one organization, so there is no overall goals or mission to adhere to. So we go back to the drawing table and ask: what are we assessing? what is our goal? and who are we asking? The goals and motivations of the participants will determine what the feedback will be, so what sort of data can we collect to get a good idea about our MOOC?
As far as learner assessment goes (not course assessment) Accreditation is something that came up in both Mark's and Jane's responses - and even then there are different types of accreditation. What future employers look for is one type of accreditation, and what regional or national higher education institutional accreditors look for is another. Mark mentions something like an apprenticeship model. I actually like the idea of apprenticeship in academia but one thing comes to mind: in order to be an apprentice you need a master - someone to both keep you under their wing and show you the ropes, but also someone who can vouch for you. I don't think that MOOCs are quite there yet with the pairing of apprentice-master and thus they are not there with the vouching (if that is important to people). Perhaps a shorter term "solution" to learner assessment might be an ePortfolio: each participant who wants to "prove" that they learned something in a MOOC can create an ePortfolio that they maintain artifacts of their learning throughout all MOOCs that they have participated.
I think in the end, the big question (in all assessment, both course and learner assessments) ais what do you want to accomplish with this assessment? I think once that's answered at the very least you can experiment with different methods to figure out what method works best :-)
In any case, my first step would be to ask do we need an assessment and why? Now, just to be clear, I am not saying that we don't need assessments in MOOCs, but rather we need to rethink the assessment (as both Mark and Jane commented on in my previous post). What do we aim to accomplish with an assessment? Is it for us (organizers) or for the them (the participants)? What form of assessment is best suited for the pedagogical design of our MOOC and the types of participants we have?
Once those things are sorted out, then we can start thinking about specifics. As Jane said there would probably be two "tracks" of assessment, those taking the MOOC for credit, and those who do not (like me). My feeling is that those who take the course for credit have slightly different goals and objectives than those who do not. Those differing goals will be evident in what each type of participant is looking for in a MOOC. Jane brought up the idea of Open Badge, which I think is a wonderful idea; personally as a MOOC participant I would like to have a little something, a momento in addition to my blog posts, that I participated in a MOOC. MobiMOOC had that, Inge did send out (digital) certificates of participation to those who participated above a certain threshold - I thought that was pretty cool! I wasn't participating in MobiMOOC just for the certificate - I was doing it for the knowledge, but it was a nice additional motivating factor.
Of course a potential issue with badges is exactly what I came across in my other blog post on open door policies and analytics. If you specify that you need to be active in X-many sessions and [insert participation criterion and quantity] to get our cool badge people might cry out that the bar is unfairly high. The devil's advocate of course might say "well if you are intrinsically motivated to only participate (as we have defined participation) in fewer than [quantity], then what does the badge matter?" I honestly have not answer for that - it's part of a longer discussion and debate that I am sure I will continue to have as this MOOC goes on and in future MOOCs :-)
So, back to assessment. At the risk of death by assessment perhaps one way of assessing the modules in a MOOC would be a weekly assessment that takes place one week after the module is done - so "week 6" would not be assessed at the end of "week 6" but rather at the end of "week 7," this would give people enough time to get through week 6 and digest what has occurred on that topic over the past couple of weeks. This is sort of similar to a taking of the pulse that one of my professors did at the end of each weekly module. She would have us do a short 5 question assessment on to module to see what we thought about it and how it could improve. This might be an option for MOOCs - performing formative assessment.
I think summative assessment is a bit of a problem if you are talking about non-credit MOOCs simply because people will dip-in-jump-out. At the same time, what are you assessing at the end of the course? At my institution summative assessments seem to be more faculty assessments, in other words "how did this professor do?" In a MOOC this isn't an issue because each week we have a new facilitator or guest. The other use for formative assessments at my campus is to see how this course fits in with the rest of your course of study or how it fits within the organization's published goals (like being inclusive and civic minded for example). Well, a MOOC will not be part of a MOOC curriculum (but that could happen!), and as far as I know MOOCs aren't offered by just one organization, so there is no overall goals or mission to adhere to. So we go back to the drawing table and ask: what are we assessing? what is our goal? and who are we asking? The goals and motivations of the participants will determine what the feedback will be, so what sort of data can we collect to get a good idea about our MOOC?
As far as learner assessment goes (not course assessment) Accreditation is something that came up in both Mark's and Jane's responses - and even then there are different types of accreditation. What future employers look for is one type of accreditation, and what regional or national higher education institutional accreditors look for is another. Mark mentions something like an apprenticeship model. I actually like the idea of apprenticeship in academia but one thing comes to mind: in order to be an apprentice you need a master - someone to both keep you under their wing and show you the ropes, but also someone who can vouch for you. I don't think that MOOCs are quite there yet with the pairing of apprentice-master and thus they are not there with the vouching (if that is important to people). Perhaps a shorter term "solution" to learner assessment might be an ePortfolio: each participant who wants to "prove" that they learned something in a MOOC can create an ePortfolio that they maintain artifacts of their learning throughout all MOOCs that they have participated.
I think in the end, the big question (in all assessment, both course and learner assessments) ais what do you want to accomplish with this assessment? I think once that's answered at the very least you can experiment with different methods to figure out what method works best :-)
Comments
Juxtaposed,connectivism as branch of learning is extremely powerful and useful for lifelong learners. Persons learn cooperatively and contribute collaboratively.The complexity of executing an assessment presents a logistic nightmare- people jump into this courses when time and other factors allow. Which format will the assessment take a criterion based or normative assessment,better still of what value will it add to this community of adult learners who are motivated by other factors.
Summative of formative assessment? a decision has to be reached whether the assessment is formative or assessment.I can say without fear of contradictions that self assessment is the way forward- this are adults that are motivated by other factors- other than a nominal label of pass or fail.