Monday, October 28, 2013

Let’s Learn from MOOCs and Recapture the Microphone

Quite a few topics have been twirling in my mind these days but nothing was really solidifying until I read the following three blog posts in my pocket account in at the same time:
They are all interesting reads, so I highly recommend that you read them and think about them as a collection. One of the things that was discussed in this collection is the (mistaken) grouping of MOOC and online courses of the "traditional" sort. This is something I've discussed in the past both on this blog and in person. Hogue sees a silver lining: the MOOC craze is raising awareness of traditional online courses and there are more players interested in looking into offerings in the traditional online space. However, as she writes, the other side of this is:
An unfortunate side effect of equating online learning to MOOCs is that those who are new to online learning and are being asked to convert face-to-face courses to online think that their online courses need to look like MOOCs – some of which are quite poorly designed!
This is something that I worry a bit about because there are faculty groups out there such as CFHE (no relation to the 2012 MOOC as far as I know) that not only lump MOOCs with online learning and poorly designed MOOC, or even conscientiously designed ones that are experimental, lump their failures in with tried and true, and researched, traditional online courses. Furthermore, while 2000 is not that far into our past, it's more than a decade away, but for-profit school misconduct is still something that is shoved in our faces without critical analysis of the differences between institutional practices and rigor of courses.
Hogue continues on to write a bit more about the silver lining in MOOCs in that they are potshot good sources of OER. I have to say that this is something I've thought about because I am thinking about researching and co-designing a course on gamification and games in education. There are at least 3 MOOCs out the that have some nice videos that discuss principles that my co-designer and I can look into and use as part of th course, depending on the direction we want to take. Of course, this potentially poses an issue of design. Hogue ponders the following:
So, the MOOC affect on online courses is a mixed blessing. Learners may very well benefit from the increase in the availability of online learning options, organizations will benefit from branding and perhaps lower cost for higher quality online course content materials, but online instructional designers and instructors won't see any increase in compensation, but will see an increase in expectations of quality from both learners and their organizations. Do you agree? Am I sounding too pessimistic here?
I have to say that I am not that pessimistic. We are actually due for a major reality check in our industry (instructional design). People who hire instructional designers (if we judge from job postings) want a master of all trades. Of course, this means that you won't be a master of any of them, so you won't be creating awesome videos, or animations, or simulation environments if you are a generalist. This types of MOOC endeavors require a team, and a team is not something that is available in just any run of the mill course, as hogue points out. People can demand that all their online courses are MOOC style in quality of materials, but unless they want to sink in the cost of people, they won't get it. As designers and instructors we need to drive this point home to people
One last thing from the Hogue post:

A good online course does not need the high quality production that an xMOOC has, it isn't about marketing the university or boosting the ego of the presenting professor. The focus should be more on creating meaningful experiences for the learners – which doesn't require high production costs – rather it requires thoughtful learning design, which is mindful of the medium, but also mindful of the learner circumstances.
I would actually argue that xMOOCs are over-produced. Even xMOOCs don't require that level of production. You can develop a really great xMOOC and not need a documentary sized budget to make things happen. That said, I agree, the design considerations vary on a lot of things, and how you design an online course will vary substantially from course to course based on they're factors.
Next up we have MacGregor. One of the things that stood out to me was the following:

An annex to the main report published last week, The Maturing of the MOOC – Literature review of massive open online courses and other forms of online distance learning, finds that opinions on the role of MOOCs in development have polarised. While many identify MOOCs as providing direct access to global high quality education, others “detect a new form of cultural imperialism.”
She writes about combating the "cultural imperialism" that is identified the the report by [by the way, it would have been nice to have a link to that report :-) ] by having MOOCs offered in other languages. I would say that you need to go a step beyond that and think about what MOOC pedagogy means beyond language usage. There were examples of different MOOCs in different languages across Europe, but there is something to be said about he cultural hegemony that exists in they creation of software and the underlying assumptions that go into that software design. I am a firm believer that you can't just take the xMOOC approach, or the cMOOC approach and have those formats translated into Chinese, Korean, French or Greek. The cultural and educational philosophies of those countries and their history play a large part in how education has been shaped. MOOCs, at least the cMOOCs that I have been exposed to, don't seek to repeat or copy the existing higher education structures, but they do seek to further understand and expand on them. When you do this you start from a specific base and move outward from there. When we have three of the major platforms for xMOOCs coming from North America, the imposed underlying assumptions and foundations are those of North American higher education, not European, and not Asian. It's time to rethink that paradigm.
Finally, we have Cillay. This is where the blog post title comes from for this post. There were a number of interesting things in this post, but I think that the educational hegemony and reinforcement of existing power structures in Highered education is something that really needs discussing. Here are two interesting excerpts discussing MOOC participants (no sources are cited in his blog post by the way):

Those students who have “experienced” MOOCs are often older, knowledgeable, and credentialed and, even so, many MOOCs offer an unenviable student success rate, some estimate that rate to be around 5%. Is it fair to expect a typical 18-year-old to thrive in an environment that requires academic confidence, preparation, and self-discipline? We could look at the San Jose State University experiment with Udacity to provide a partial answer–where we see a younger population of students in need of remediation, not performing well in their MOOC.

If that’s so, (cue the really uncomfortable question) who are those students ill-prepared to thrive in a MOOC? Might we envision a population that lacks access to effective high schools, who have had limited success in formal learning, and from a lineage unfamiliar with higher education? Is it prudent to create a system that requires the most disadvantaged students to pay more? And is it right? This may be a question that each university will need to answer independently.
This is true, and I've discussed it elsewhere. MOOCs, the way they are currently designed (limited support structures from the institution) are designed for the self-starter, the student who already knows a little something about the topic or can create connections without a lot of assistance. They are not designed with the novice, or newbie, in mind - which is why it's perplexing to me that all these "elite" institutions are treating their MOOC offerings like a gift from God. Students who are undergraduates don't, by and large, have those skills needed to be successful MOOC learners. This is why most people who participate in these MOOCs do so in a way that is comparable to participating in public lectures by public academics and then taking their participation either home with them or go to the digital equivalent of an intellectual salon. There is definitely a disconnect between what MOOCs from elite institutions claim they want to do, with what they are actually doing. I am not saying that we can't bring more of these topic to the people, but I really doubt that we can MOOCify undergraduate education like this. Even graduate education can't be don't in whole like this.
Another part that Cillay discusses is the cost of free. He writes:

Then there is the cost of these free courses. Not to the student, but to the university. The cost to a university to play with edX is substantial. To offer a course through edX, is $50,000. If a university wants edX assistance building an online course, that’s $250,000, plus $50,000 every time it offers the course. I wonder at the sustainability of this type of investment in “free” courses.

So, what is the price of free? If you had asked this back in the cMOOC days, I would have said "very little" because everything was distributed. Blogs, wikis, twitter, mendeley and so on. All you really needed is some software to collect and disseminate the day's MOOC work in a newsletter. So what's that? Maybe $10 per day on an amazon virtual server. This is certainly nothing like what coursera, EdX,and Udacity have as their costs. Make no mistakes, the for profits see this as a way to enter the, currently entrenched, LMS market. We've most certainly have gone way off the rails with this from the early edupunk days of MOOC. I think it's time to reassess the mega productions, the LMS, and everything that has gone into the xMOOCs. If you are spending that much money on something you are giving away for free, you are doing something wrong. You really need some tangible, and I would argue ethically derived, ROI to justify that kind of expenditure.
Finally, the question is posed about what is the student experience in MOOC? Cillay writes that the student rxperience is often forgotten in MOOCs. I would argue that perhaps this is the case I with some MOOCs, but not with all MOOCs. Just because we have, as Hogue puts it, Content-Learner interactions it doesn't mean that we've forgotten about the experience of the learner. I would counter-argue that we forget about the learners experience in our face to face courses when we have auditoria full of people in intro level courses. I would also argue this when you have one instructor teaching a section of 50 online. What is the learner experience there? In MOOCs we may do a poor job of learner engagement at this point because we certainly don't have the knowledge of how to do this well in massive environments, but what is the excuse for current face to face on traditional online practices?

Thursday, October 24, 2013

Critique of Making your own Quasi-MOOC

With three MOOCs done (only undertaking one now), I have a little more time to go through and read what has been piling up in my Pocket account.  Now, over the past couple of years there have been a number of articles on building your own MOOC, from a variety of people.  Some in publications like Learning Solutions Magazine, some in eBook form, some in in Blog form.

One of the blog-form posts comes by way the blog "Managing eLearning" and the title is How to make your own MOOC. I was quite curious to see what the author had to write about the topic, but I was seriously disappointed when I read it.  My main issue with the article is that it ascribes to  a very centralized xMOOC, offered by an "elite" University.  I don't think that the "elites" have it right.  I applaud the exploratory spirit of some "elite" Universities, but they get many things wrong. So, building on this xMOOC model seems just wrong to me.  In this article there are 6 principles, or key ingredients to build you own MOOC. The article is really basic, and the headlines can possibly apply to almost any online course.  That said, let me deconstruct the areas that I have an issue with:

Underlying assumption of the author is that "massive" means "tens of thousands of users, ensuring that there is someone out there who is able and willing to answer almost any given question." and that this type of Massive is developed by "Brands. Educational brands. Big brands" like Harvard, MIT, Berkley, and so on.  I completely disagree. We had MOOCs before these "elite" universities decided to jump in with their interpretation of a MOOC.  The universities, or entities, doing this were not big brands, but people still went to them and had a learning experience.  Massive, as I have written before, is not a static amount.  Massive can vary depending on the subject at hand.  An introductory level Algebra course will be more "massive" than a graduate master's level advanced course in biomimicry. They can both be MOOCs, but the underlying requirements for the course will determine how many people actually sign up.  If a regular biomimicry course enrolls 8-15 students in a semester, then 150 students is actually massive for that course.

Next up, let's look at the author's "ingredients" for a MOOC.

Ingredient 0 --> An LMS: The author writes "If you don't already have one, you will need one. Social features, especially discussion forums, are a must." I honestly completely disagree.  An LMS is not a requirement for a MOOC, especially if we are considering discussion forums.  In a MOOC, the LMS discussion forum doesn't work well, let's face it.  We can work with the technology we have, but what it boils down to, and what we've seen thus far in the last 2 years of xMOOCs is that forums aren't well suited for this, at least in their current incarnation.  An LMS also does not address the design decisions of a distributed MOOC, where the LMS is a bit antithetical to that way of thinking about a course.

Ingredient 1--> Synchronous design: The author uses a slightly modified understanding of what synchronous is, so that's something to keep in mind. What the author suggests is that all learners need to move in lock-step. While building an online learning community is important, I disagree that MOOC learners need to keep in lock-step, moving through the same type of materials and activities.  I can see some people ahead of the curve, and some straggling.  The key ingredient is that community, not the synchronicity of the material.

Ingredient 2--> Short Learning Activities: Here we have a suggestion that we work on bite-sized learning activities, like Khan Academy.  While I agree that short videos are a good (compared to hour long cognitively overtaxing alternatives), but I disagree that Khan Academy style videos are "it" for MOOCs.  We are back to a didacting sage-on-the-stage approach that isn't really helpful. Sure, some elements of this might work, but it's not something that you can generalize across the board for all MOOCs, across all disciplines and across all levels.

Ingredient 3--> Require Peer Review: I think peer review is great.  Peer review for a grade, however, not so much.  Peers, even more knowledgeable (MKO) peers, aren't the subject "expert".  I can learn a lot from peer to peer scaffolding, however at the end of the day, my peers aren't necessarily qualified to grade me, and have that be my final "grade" for the assignment.  Additionally, while I do think that peers can learn a lot from one another, forcing peer review, in an open course seems antithetical to the open ethos of the course.  If I don't want to share my work, that should be fine.  If I want to share my work, that should be fine too.  Self-selecting, self-appointed peer review groups are preferable to forcing everyone to be in a peer review if they want a certificate of participation.  A better way to deal with this, in my opinion, is what OLDS MOOC did with badges for peer review.

Ingredient 4--> Required Group Work: While this may be good in "regular" online courses, I think that in the MOOC front it's too contrived and too antithetical to the find your own path.  In smaller, "regular" online courses where you have fewer students, requiring group work is necessary because people may not self-organized in ways that encourage active community engagement.  In MOOCs, however, this is much less of a problem because you will always have people there that are able to kindle the flame of community sufficiently for others to jump in. Adding required group work, for a course that gives students no formal credit at the end, for me, adds a barrier to entry.

Ingredient 5--> Teaching Assistants I've seen quite a few MOOCs run without TAs.  Remember, just because 100,000 people registered for your MOOC, not everyone is serious about participating.  You may just have a ton of people on your hands that are window-shoppers and never bothered to unregister when they decided that the MOOC was not for them.  TAs seem a bit of an overkill.  What would be better would be a core "team" of subject experts, each tackling a week of the upcoming topics, and they all contribute to the kindling, along with other MKO peers.

Tuesday, October 22, 2013

1 week, 3 completed MOOCs, 1 MOOC Experience Reflection

Online Games & Narrative Course Logo
Last summer, when I signed up for these things, I really didn't  keep proper timing of the courses I signed up for, because I was signed up for three concurrent MOOCs, while working a full time job, and messing around with other interesting things (MOOC related).  In any case, after several PACKED weeks, three MOOCs are done, and I have some thoughts about MOOC design and MOOC Process to go along with them from my own personal experiences.

The first MOOC was on Coursera, and it was Online Games: Literature, New Media, and Narrative with Jay Clayton  of Vanderbilt University. The thing that attracted me to this course was the aspect of online gaming and how it tied into other media. The theme was Lord of the Rings, which I am sort of lukewarm about. It's fine, but it's not the type of literature, or game for that matter, that I would spend a ton of time on.  The nice thing about the course was that the people talking about the materials were real geeks about it.  The material wasn't dry, and the enthusiasm about the subject really came alive on the screen (at least for me).  The course did fall a bit short of me in the assessment area.  I did partake in the quizzes, which were good enough for formative assessment, in other words it helped me make sure I was on the right track, but beyond that, I really didn't have an incentive to really participate in the forums. The combination of being a bit "m'eh" about the subject and the fact that forums in MOOCs just don't work that well made me avoid the forums for this course.  The other thing that was a bit of an eyebrow raiser was the "distinction track" of the course.  Now that I have seen all three peer-reviewed assignments, I really don't see the distinctiveness of the distinction track.  Yes, it requires more work, but at the end of the day it's peer reviewed, and that peer reviewer grade doesn't necessarily do justice to evaluate any work I would have contributed. The two redeeming thing about the Distinction Track assignments was that they (1) didn't limit the amount of words you could use for text-submissions, and (2) you could actually use a variety of media (papers, game-making, videos) to submit your work.  As an assignment it was interesting, as an evaluation of learning it was not.

The second MOOC was on Coursesites, and you this was the Mozilla Open Badges MOOC that you've seen me write about on this blog. This MOOC had weekly live streamed sessions (as well as recordings from them), and Open Labs for badges.  In addition there were discussions forums and weekly challenge assignments that could award you badges. I have to say, that the awarding of badges was motivational for me because the assignments required enough time and thought, that I don't know if I would have bothered to put pen to paper to hash out some ideas if there wasn't some external award for these.  Now, this is part of my PhD brainstorm, so I would have written something in my PhD ideas notebook, but I wouldn't necessarily have gone into this amount of detail.

So, as far as process for this MOOC:  I liked the live sessions. It kept a degree or regularity in the course that allowed me to attend to these live videos on Mondays, think about the content for a few days, jot down some notes, and on Saturdays write-up a little something for the weekly challenge and submit it.   The challenges were interesting, but some were a bit out of my domain, so I used some assumptions to complete them.  Probably 1/3 of what I submitted was returned to me for improvement, and I resubmitted it.  I actually got feedback on what I submitted which was awesome.  This was something that continued my motivation to participate in the MOOC.  As far as discussion forums go....well, I did make an attempt to participate in the forums, but I didn't participate as much as I had intended to. The nice thing is that there were separate areas (using the Groups tool) to discuss Badges in different contexts, such as Badges for MOOCs or Badges for Higher Ed courses. Unfortunately there didn't seem to be many participants in the MOOC (or at least in some groups), so while the "intimate" feeling was nice, it also meant that it didn't really fit with the type of participation I wanted,which was 70% read, 30% write.  All things considered, this wasn't bad, and the discussions made much more sense.  Still not optimal, but good enough.

Finally, the last MOOC, that in theory is concluding this week, but I am already done with, is the MOOC on Pragmatics at the Virtual Linguistics Campus.  My motivation here was to fill in some knowledge from the time that I took an introduction to linguistics course and we only did a few weeks on Pragmatics.  The course, like before was set-up like a self-paced eLearning course, with automated testing, self-paced multimedia, and lectures.  Everything was available at the beginning of the course, so there was no need to wait until someone released a new module for you.

This MOOC was a bit of a hit or miss. I think I definitely enjoyed the Phonetics and Transcription MOOC that they had last spring more than I did this one.  The recorded lectures were fine, and the self-paced eLearning materials were fine as well.  There seemed to be less attention given to some of the assessments (multiple choice quizzes) this time in that some assessments in some chapters were just one question! If you get it right, you pass the module with 100%.  The other issue was that learner evaluations in Module X references things that learners would learn in Module X+3, so some people were confused by this.  Luckily I was not since I had already covered some of this through my Master's in Applied Linguistics. This seems like an oversight, but attention to those finer details is something that, for me, can make or break a MOOC.

Discussions were used mostly as a way to troubleshoot, for me anyway, even though some participants used it as a way to disambiguate, especially in those evaluations from Module X where things from Module X+3 were mentioned and taken as previous knowledge. Just like the Spring Phonetics & Transcription course, I didn't spend a lot of time in the forum. The other thing that was different this time around was the lack of reading materials.  Last spring, each week, there was a scan from a book chapter (different books each time). This was pretty nice because each week, in addition to the self-paced eLearning materials, I read something from a book. This year, they cut that out, presumably for copyright reasons. In either case, the course this time around seemed more bare.  It lacked a level of detail that I had come to expect.

Finally, it's still left to be seen what the Statement of Participation looks like, but I hope that they didn't overlook design issues because they were looking to make some money from certified, graded, Certificates of Participation.  Last spring, the certificate of participation listed all modules taken, and the final percentage grade.  This year, it seems that you will need to pay to get that level or reporting (and have your grade registered at the University of Marburg), and people who don't pay just get a Coursera-style certificate of participation.  Let's wait and see.

I am now enrolled in some other MOOCs. Let's see how those pan out.  In the mean-time, back to reading about MOOCs in the press, and writing more about them.

Tuesday, October 15, 2013

Badge MOOC Challenge 6: Building a Successful Badge System

Trust Network Badge
Well, this is it!  We are in the final week of  the #OpenBadgesMOOC, and this is the last post (for badge purposes anyway) from Mozilla's #OpenBadgesMOOC. As with previous blog posts in this series I am brainstorming about including badges in an #ESLMOOC that I am thinking of designing, developing, implementing and them studying for a potential PhD.  With this week's materials we are tackling the Badge System.  Since this brainstorming is all theoretical and planning, I will most likely have some assumptions that underlie this brainstorming session.  As with previous weeks, we have the prompt (from the MOOC site) followed by my brainstorming on the topic.

Challenge Assignment 6: Building a Successful Badge System
  • Verification
  • Authentication
In order for Open Badges to gain full acceptance, extra precautions must be in place to ensure transparency in and confidence about the badging process.  This involves authenticating that the badge holder is indeed the one who earned the badge, and that the badges displayed by a badge holder are verified as coming from an authorized source. These “official” steps can be technologically addressed in your badge system implementation. In addition, the open badge ecosystem is evolving to include reputation systems evaluating learning providers and assessors as well as endorsements offered by employers and standards organizations.

If you’ve gotten to this challenge, you’ve invested an enormous amount of thought and work in the prior five challenges. Don’t let it go to waste: Now is the time to actually implement a badge system. Draft a project plan, find collaborators, and see it through.
  1. What stakeholders at your institution or company need to review/approve the badge system? Do you have the right materials and explanations to help them make informed decisions?
  2. Can the badge system be used to address existing goals and thereby strengthen its purposes?
  3. Are peers or partners or consortial institutions implementing badge systems, perhaps providing collaboration opportunities? Do competitive pressures strengthen your reasons for implementing a badge system?
  4. Are your learners demanding more authentic and targeted learning opportunities? Are you delivering the value your learners expect?
  5. What are the next steps for your badge system? What other resources do you need? Build a roadmap/workplan for this badge system.

BRAINSTORMING for this week:
So, coming to the end here, this may actually be one of the easier challenges to respond to. The nice thing about building something from the ground up, and working on a MOOC of my own interests is that there is no institution to review or approve my badge system. This means that I can "bake in" badges into the course and that they won't be an afterthought.  I suppose that if I want some institutional resources I may have to seek some buy-in from the institution, but since they seem eager to experiment with badges it may not be such a hard sell.

Since the MOOC has not yet begun its development cycle (I am currently in an Analysis stage), this also lends itself to barges being tied into goals, meaningfully, as goals for #ESLMOOC get decided on, and as meaningful MOOC assessments are thought out for this MOOC. Thus, I am hoping that baking in a badge system will strengthen the MOOC outcomes.

In terms of peers, if this #ESLMOOC is dissertation material, I guess I have to do most of the original preparation alone, but it would be great to get input for the design and implementation of this MOOC from instructors who do ESL as their day-job. Going forward, it would also be interesting to connect with others who are interested in examining the efficacy of badges for MOOCs, either at a regional or international level. There are no competitive pressures to implement a badge, unless you think of research and publishing a competitive pressure. I just tend to think of a badge system as a "good idea," that can help motivate learners, and give them a tangible item that shows the fruits of their academic labor.

In terms of the learners for the #ESLMOOC, I'd have to go out on a limb and make some initial assumptions about them.  In the first blog post I described a couple of sample students. Now, since the MOOC is international, I foresee that the learners will have many different motivations for joining and many different expectations for completing the MOOC. They probably have expectations of themselves about how often they participate in the course.  As we've seen in many different posts on, the as well as many blogs from various MOOC participants, we see that people join and continue (or discontinue) their participation in a MOOC for many reasons, and some don't have to do with the MOOC itself.  The interesting nut to crack will be a real proper tracking of learners in MOOCs, and getting feedback from people who want to continue, but the barrier to continuation is just a little too much to overcome.  If the barrier is course related, it will be interesting to see how a MOOC can be modified to help those learners.  Badges may end up being something that keeps people going.  I know that, for me, badges can be motivating to continue the course, if there is something else in the course that is of interest to me as a learner.

As far as delivering value goes, while a MOOC is free, learners to pay for it by putting in the time, effort and brainpower to complete the MOOC, so they probably want to see something back from it.  Realistically speaking, in a 6-8 week MOOC someone's language isn't going to improve 100% in all areas, so the MOOC can provide value to the learners, and so can a badge system, but at the end of the day, the value that learners get out of the MOOC will in part be based on how much they put into it.  I know from personal experience MOOCing, that what I get out of MOOCs greatly depends on how much I engage in them.  I expect a language MOOC to be even more so.

So, the next steps in this project are as follows:  I expect to do a whole lot of research into what the course ought to be (in other words the Analysis phase of instructional design).  Concurrently, I plan on researching a research on MOOCs (the little that exists), but also educational research on discussions forums, twitter, wikis, blogs, social bookmarking and so on to see what current research says about these things.  I am a few years behind on this at this point :-).   Once this is done, I will begin with assessments in mind, and that also means badges.  I will most likely be using Purdue's Open Passport platform since it ties into Backpack, and it's ridiculously easy to create badges on that system.  Considering I am not in a PhD program yet, I am not in any rush to complete this project tomorrow.  I am thinking about this as a long term project (next 18-24 months) since I am working on it on my own, on my spare time. I think that as I am developing assessments and badges I will seek the feedback from peers who are ESL instructors to see what they think.  I am wondering if anyone teaches ESL online - that would be of immense help in terms of sounding boards.

So, finally, let's end with a user story. Remember Stella D'Agostino? We met Stella in our first challenge blog post.  She is an Italian Professor at the University of Milan where the language of instruction is slowly changing from Italian to English.  She had completed her education in English, so she wanted to not be stuck in unnecessary classes.  She was signed up for #ESLMOOC to see what all the fuss was about.  She saw that badges would be something that would be available to learners who have demonstrated proficiency in the language.

As she looked at the modules for the six week #ESLMOOC she saw that she didn't really need to learn much from certain modules (2/3 the MOOC to be precise), so she submitted her assessments for those early, so that the ESL instructors could assess her work.  She passed those with flying colors and got her badges of mastery for those weeks, however her involvement did not end there. Since she had mastered those weeks early she became a peer mentor (another badge she could earn) for those weeks helping fellow classmates, some of whom were at the same University as she was.  For the weeks that she hasn't mastered before, she was able to participate fully, submit evaluation materials and get mastery badges for those as well.  At the end, she not only had all mastery badges for the MOOC, but she also had some additional (let's call them "rare") badges that showed her ability to be a peer mentor.  This helped her with work in that she had additional opportunities at work to excel and help peers in an offline (non MOOC) way.

OK, this is a bit down the road - but I think that these future (post-badge implementation) stories should be inspirational.  It's not just enough to get a badge - is it? :-)

Your thoughts?

Friday, October 11, 2013

Badge MOOC Challenge 5: Authentic Assessment and Evidence for a Badge Ecosystem

The real badge?
Alright!  The penultimate week in #OpenBadgesMOOC, brought to us by Mozilla and Coursesites.  Continuing this week is the exploration of how badges can be incorporated into this #ESLMOOC that I've been thinking about designing, implementing, and hopefully collecting some data for some interesting analysis.  Dissertation-wise it seems like a good topic, but considering the University I was considering applying to has suspended operations due to Austerity Measures in Greece...well, I guess I'll keep looking at other programs while Greece sorts its issues out :-)

So, as with previous Badge Challenges, the Prompt comes before my brainstorming.

Challenge Assignment 5: Authentic Assessment and Evidence for a Badge Ecosystem
Badge system design acknowledges that not all learners are the same, not all learning situations are alike, and not all ways of ascertaining learning accomplishments and skills attainment are the same. Badges offer learners cum job seekers not just more flexibility in how they learn but also in how they prove that they have the skills/competencies represented by a badge.

Assessors have a multi-partite role in the badge ecosystem. They must devise strategies that, as much as possible, push assessment activities into the world where actual performance will occur. They must validate both performance and performer. They must provide a robust set of metadata for each assessment that communicates that validation in total.

In a healthy badge ecosystem, learners demonstrate their competencies in authentic learning environments, capture evidence of their achievements, and have valid assessment to back up the earned badge. Just as badges open the field for innovative learning providers, they simultaneously stimulate rethinking how learning opportunities are provided and assessed. Based on the badge system you described in the prior challenge, describe the learning and assessment frameworks that are needed.
  1. How can learners’ needs best be addressed?
  2. Will traditional learning contexts and methodologies suffice, or can/should they be reworked? Are entirely new methodologies needed? Will new/different staff be required?
  3. How do competencies map to the learning activities and assessments?
  4. What types of evidence and assessment are valued and/or required by employers?
  5. Do the methodologies support the validation frameworks the learning provider needs?
  6. Write one or more “after badges” user stories depicting the value of the badge ecosystem for learner personas. What opportunities do badges provide for your personas? What challenges must be overcome in order to optimize the value of badges? Who do they share badges with? What goals do they have for using their badges?

BRAINSTORMING for this week:
It was interesting to see Kyle Bowen at the weekly live session this week in #OpenBadgesMOOC.  I have met him in person at Campus Technology conferences and I have seen some of what Purdue is doing, which is pretty nifty!  After one of his presentations I created an account on Open Passport to mess around with badge creation.  This was a pretty easy system to create badges - if you know what badges you want to create that is!  Just like ePortfolios, the technology is not (generally) the issue, but rather the underlying objectives and learning that you are attempting to assess.

For this week I think that we're back to familiar waters, with fewer assumptions than previous weeks since some decisions in the last couple of weeks were above my "paygrade."  I do think that since I am thinking about Badges in relation to MOOCs it is a little easier. Applying badge considerations to MOOCs is easier, as far as I am concerned, because, in theory, you aren't tied down by the institutional baggage of how assessment has been done for the last decade or century :-)

When thinking of learners, and learner's needs, it's best to do some preliminary needs analysis, however this is hard even when thinking about "traditional" higher education courses, let alone a MOOC where you potentially have learners from all over the world with many, and competing, interests.  That said, when designing the MOOC it's best to think about the target demographic for your MOOC and then make sure that those people who sign up are the learners for which the MOOC is designed. I do believe that in the weeks leading up to the MOOC, when learners sign up, they can potentially fill out a survey that allows the MOOC facilitators to keep certain things in mind as the MOOC is facilitated, and as additional or supplemental material is discovered, evaluated, and rolled into the MOOC. This should help address the needs of specific learners who are signed up for the MOOC.  The big thing to think about is whether or not facilitators and designers of the MOOC keep addressing the needs to non-participating students in the MOOC, in other words, students who signed up, and might be reading along, but are not visibly participating in some way, shape, or form.

As far as staffing and methodologies are concerned, I would say that staff is definitely needed. A MOOC isn't a one-man show, but rather it's a team effort to design, develop, and implement the MOOC.  In my initial thinking, I am thinking of designing and developing the MOOC on my own, simply because it's probably something that is going to be part of a dissertation.  That said, when implementation-time comes along, I would like to recruit some ESL instructors from my institution, maybe on a volunteer basis, or through some sort of grant support to help pay them a small stipend, to help facilitate the course with me and to provide resources for course material evaluation, learner outreach, and in the end, of the trial group - some sort of evaluation support.

As far as methodologies go, as with most situations where instruction changes from one medium to another, there will be some change in methodology because there isn't a 1:1 correspondence between existing face to face language learning courses and the MOOC format.  I suppose you could shoe-horn the face to face approach into a MOOC, but it just won't be successful. The commonality between the MOOC approach to language teaching and existing face to face teaching is that I plan on using a Communicative Language Teaching (CLT) framework for designing, developing and implementing this #ESLMOOC. While CLT can be the underlying method in both approaches, the tools users, and specific methodological approaches will vary between the two mediums. Thus, I foresee that in a MOOC environment we will be using a variety of  Web 2.0 applications like Blogs, Twitter, YouTube/Vimeo videos that learners can use to process and engage with the material, but also to use in the assessment of the learner's acquired or improved skills as the MOOC progresses. The methodology chosen to design and run this MOOC should  be supportive of the frameworks for validating learning.

As far as competencies and their mappings to learning activities and assessments go, that's a subject to be determined - mostly because the competencies themselves need to be determined.  Right now, the CEFR has broad levels of competence.  In order to make those broad levels more meaningful, but also as a way to map badges for those broad competencies, supporting competencies need devising. At the very least there are three supporting competencies, including hearing, speaking and writing. Those in turn would need some supporting competencies of their own.  All of those would have considerations for activities and assessments.  As far as assessments go, I don't plan on having many assessments. There will be two levels of assessment for #ESLMOOC.  The first level would be basic assessment where the assessment is relatively easy and it doesn't need many assessors, or lots of assessor hours, to complete. This is probably where badges fit in.  The second level of assessments is a more in-depth assessment of learners, perhaps in a smaller cohort, that will serve as part of the data gathering process for the PhD. If #ESLMOOC runs again, and if the second stage isn't as time consuming as I am envisioning it now, then those will be "badge-ified" as well.

When thinking about employers, going back to the original use case of this #ESLMOOC to prepare instructors of higher education in non-North American institution to teach their subjects in English, I would say that there are two broad elements of assessment that they need. There may be more, but those are subject to some sort of needs analysis.  The two broad elements I can see at the moment are:

  • Clarity of speaking (speaking with less of an accent, being more comprehensible)
  • Clarity of written feedback to students

Finally, in terms of thinking of some after badges stories let's revisit some of the personas and their colleagues.

We have Professor Tomas.  He is teaching at the University of Milan. Since he is going to be required to soon be teaching in English for his graduate courses, he decides to take part in #ESLMOOC to hone his skills in the English Language.  Also, since he was willing to be a guinea pig for the researchers, he was assessed more than other learners in the MOOC.  At the end of the MOOC he "graduated" the course with a collections of competency based badges, including one that gave him credentials as someone who could communicate orally at the C2 level (CEFR framework).  With this set of badges he is able to pass the requirements at his own institution since his institution signed onto #ESLMOOC as a sponsor and was able to vet some of the outcomes of the MOOC.  Professor Tomas' only issue is that he also moonlights at other institutions, and since they weren't part of the conceptualization process of #ESLMOOC, and they are no familiar with badges, they are not sure what to make of his accomplishments.  Still, he has published his badges from #ESLMOOC on a variety of spaces, including his LinkedIn account, his account, and on his professional website via an embed code that he got from his backpack.  For him, the overall goal for this particular set of badges is to certify his English skills.  Using digital badges he can take care of the immediate needs of his home institution to demonstrate competency in English, but he can also advertise his skills via badges, as a way to also get noticed for any part-time work in institutions abroad, including the US.

That's all for badges for this week.  Thoughts?  One more week to go in #OpenBadgesMOOC

Saturday, October 5, 2013

Badge MOOC Challenge 4: Accreditation and Validation Frameworks for a Badge Ecosystem

Value Map Badge
It's Saturday, so it must be #OpenBadgesMOOC time :)  The thing that I just noticed about these badges on the #OpenBadgesMOOC is that if you look closely enough they look stitched.  Maybe there is an easter egg hidden somewhere, whereby if you earn all #OpenBadgeMOOC badges they send you a sash with all of them stitched on - LOL :)

In any case, it's the end of Week 4 on the MOOC (2 more weeks to go) and this week we are talking about validation.  It's interesting.  One of the things that comes to mind as I progress through these is that the initial "levels" were a little easier to articulate, at least for my #ESLMOOC project, but as the weeks progress it's getting harder since I don't have all of the information. This also draws a parallel to Kirkpatrick's Level's of Evaluation where Levels 1 and 2 are easier to measure, at least in the short term, but Levels 3 and 4 (and if you look at Philip's 5th level) it gets harder.

In any case, we trot on.  As with weeks before, the prompt comes first, and the brainstorming follows.  Any assumptions will be footnoted (I wish blogger had some good footnoting tools)

Challenge Assignment 4: Accreditation and Validation Frameworks for a Badge Ecosystem
At the next level of complexity, we add an additional component:
  • Validation
Because badges are agnostic as to the mode of learning that learners employed to gain competencies, learning providers can innovate with new ways to engage learners. While these new strategies may be as effective, if not more so, than more traditional approaches, postsecondary institutions seeking and maintaining accreditation are concerned about whether these new strategies conform to accreditation standards. Accreditors are themselves catching up to what rapidly evolving innovations in teaching and learning mean.

Learning providers have traditionally relied on academic accreditation and reputation as validation of the value of their targeted learning outcomes. Badge systems open the field for non-traditional learning providers as well as innovative learning methodologies in traditional institutions. With this opening up comes the need for clear, transparent validation of learning providers and their methodologies; without it, the value of their badges is questionable. Based on the badge system you described in the prior challenge, describe the validation framework that’s needed.
  1. Will traditional accreditation or other existing frameworks suffice?
  2. Are there relevant standards bodies that can provide review and validation?
  3. What types of reputation frameworks are already in place, and do they apply to the badge system?
  4. What peer organizational and institutional networks can be leveraged for peer review, evaluation, and endorsement?
  5. Will new research or evidence be needed, for example evaluation of employment outcomes (e.g, how many job seekers get jobs)?
  6. Write one or more “after badges” user stories depicting the value of the badge ecosystem for learning provider personas.

BRAINSTORMING for this week:
The live session this past Monday was pretty interesting, especially since there was a presenter from NEASC, who happens to accredit the school I work for.  Of course, that being said, since the #ESLMOOC is going to be more of a pet project (something that hopefully will lead to a dissertation - fingers crossed), NEASC won't have a ton to do with this type of accreditation.

Now, thinking of this week, I honestly can't say that existing "accreditation" will suffice for the #ESLMOOC.  What we have, in terms of "evaluation" for English Language Learners is standardized testing by the TOEFL by way of ETS, the CPE by way of Cambridge University ESOL, the MET by way of CaMLA,  and finally the IELTS test. At least these are the four big ones that people seem to know about.  From a framework perspective, we have the the CEFR developed by the council of Europe, and national tests tend to map onto this framework.  That said, the framework, while a good start, seems quite broad to me. I wouldn't be able to say with a high degree of certainty that two individuals, both at the C1 level (penultimate level) have the same exact skill set with the language. I am sure that they can adapt, but they won't necessarily be a perfect fit from the start.

Knowing this, I think that the council of Europe and the ACTFL, on the broad level, as well as national standards bodies for language would be able to come up with validation criteria for specific badges if tasked with this undertaking - the question is: are the on-board with badges? The other thing that comes to mind is perhaps a concern about over-regulation of badges. For example, in a Higher Education classroom, if a colleague wants to institute badges for both demonstrable knowledge (artefact based) and demonstrated behavior (no visible artefacts from the learner to be digitized and available to the accreditor), how much latitude does this person have from their department, college, university or accreditor to determine what sort of behavioral badges are appropriate for his classroom? I guess there is a Goldilocks Zone here, in badging,  as well.

Now, as far as reputation frameworks go, I am actually not really sure what already exists. I suppose, from people I know, the reputation is the fact that they have passed x-exam with y-grade.  For instance, last summer when I was talking to a number of friends of friends, or friends of family and they told me with pride that they had passed the "Lower" exam, or had passed their "Proficiency" exam.  This was usually tied to discussions about work, and working abroad, so for them this was a credential.  When I was asked if I had passed any of these exams, I was a bit perplexed, because even though I grew up in Greece, I also spent the last 20 years in the US, and have completed all of my tertiary education here. So, it would seem to me that these exams have a built-in reputation system because it is something that people can compare apples-to-apples to; whereas when they were asking me about my exams, since I had none (but I had alternate experiences), they did not know how to compare their achievements to mine.

Thinking about Peer Assessment and Endorsement, I would conceive this as three concentric circles.  In hte inner most circle, I would say that ESL departments in campuses around the New England area (NEASC's territory) could get together to peer review each other.  It seems that ESL, at least in my institution, is non-credit.  The #ESLMOOC, similarly, would also be non-credit; after all, it will be open, online, and free as MOOCs are.   The next level up is a circle at the national, or North American level.  Perhaps ESL departments from around North America can get together and peer review these badging system so determine efficacy, validity and fidelity.  Finally, the top-level circle would be standards bodies like the European Council and ACTFL looking at the broad strokes and reviewing this work.  So, the outer levels of this circle are more broad, and the closer you get to the core, the more details you get.

As far as research goes, I think that some research will be needed, but I don't think, at this point in time, that employment is the right rubric to use.  While the #ESLMOOC is mostly going to be targeted toward learners in higher education settings (maybe those who are looking toward switching the language of instruction to English), and there is some aspect of employability, I think that there are probably other areas that can be examined through research.  One area that really comes to mind, for me, is tying into Learning and Learner Analytics when it comes to classroom performance.  I think that as badges become more central (and assuming that there is no grade inflation with badges), it would be interesting to see how students with different sets of badges do in different classes.  Right now courses have generic pre-requisites for upper level courses such as "COURSE 601, 605, 612" or the nebulous "permission of instructor." The idea of pre-requisites is that you have gained, supposedly, some knowledge in those previous courses that is a foundation block for this upper level course.  Well, what about transfer credit students.  Sometimes transfer students meet 80% of the course requirements so they get to waive that intro course, but are they missing a certain key skill or knowledge that they will need to succeed in an upper level course?  With badges we can get more granular as to what students know, and can do, and therefore can better place them in courses.

With that, let me lead you into an "after" badges scenario.  In this scenario we have two actors: Councilor Troy.  She works at the University Advising Office at Big State University (BSU).  She is one of the people responsible for advising students for course sign-up each semester.  Based on student interests, courses completed, and University policies, Troy tries to help students sign up for the courses they need in order to get to their desired goals.

Right across from her is Jonas Quinn, who has recently transferred in from Regional Community College (RCC) after completing his Associates Degree in Computing Technology.  Jonas now had taken some Sociology courses while he was completing his AS degree and wants to major in Sociology now that he is a Big State University.  All of his general education requirements transferred in, so he is hopeful that he doesn't have to repeat any courses that he has already taken in order to pursue a degree in Sociology.

In years past, Troy's job was considerably harder, in that upper level courses, like SOC315 only had other SOC courses as pre-requisites, like SOC215. In turn, SOC201 had a pre-requisite of SOC101 and 115. Since Troy wasn't a member of the sociology department, it was at times hard to figure out what skills were needed to be successful in upper level courses.  Now that badges had been implemented regionally, Jonas had come in with a variety of badges from the sociology courses he had taken.  As it turns out, BSU and RCC use some common badges in their respective sociology departments.

It seems that Jonas has 9 out of 10 pre-requisite badges for his first BSU sociology course.  Since Jonas doesn't have all 10 necessary to join the course, but is over-qualified for lower courses, she send him to Professor Smith, department chair of Sociology, to seek permission to join the course, and to also see what he (Jonas) can do to demonstrate knowledge or ability for that 10th badge.

** Addendum Section Based on Feedback Received **
1. How does the badge ecosystem accredit and validate the learning provider and/or assessor and it's program(s)?
The badge ecosystem for #ESLMOOC has been envisioned to go along the same lines as accreditation of higher education programs in the US. Since #ESLMOOC is most likely going to be offered with a University partnership, any sort of accreditation stemming from a peer reviewer entity such as NEASC will be used to accredit the #ESLMOOC.  Also, #ESLMOOC badges should be interoperable with the CEFR that I have mentioned before.  This combination of regional accreditation as well as the European Framework (which is known at a much broader level) will serve as a way to accredit the learning and/or assessing in this MOOC.  That said, I don't think NEASC, or the Council of Europe have had to deal with badges yet, so foundations will have to be laid in order to get this Accreditor/Standards Body/Learning Organization partnership going :)

2. How can a learner/badge seeker determine the legitimacy of the provider, or the suitability of the provider's methodology to the learner's needs?
I have to say, this is quite a difficult thing to think about because even "accredited" programs (traditional courses and programs that is) seem to be peer reviewed through regional bodies that have some sort of endowment from the government. Just because a program is accredited in the country-x it doesn't mean that their degree is valid in country-y without some sort of additional work (or having to do the whole thing over again).  Thus, I would say that the criteria for determining the legitimacy of the learning provider would be the same as determining the legitimacy of an accredited program in the US since #ESLMOOC will run through a regular college program. If the school is accredited, then the course is accredited to some extend. It would be great to have the council of Europe backing it too (at least as a test-case), but it is not highly likely.

As far as the methodology goes - the MOOC format is really something that still has a lot of unknowns, which is why it's also an interesting topic of study for a dissertation. I don't think badges can really tell you much about the methodology used, at least if we go for badges for achievement. So, I don't see a concrete link between badge and methodology.  That said, since the MOOC format is so new, it would be really useful to provide some learning scaffolds and some up front information to let the learners know what they can expect in a MOOC.

3. In your ecosystem, could a well-structured reputation framework be a supplement or stand-in for peer review?
I think that Peer Review is important when learning a language.  Some learners will pick up on things that others will not. Therefore there is this collective scaffolding that goes on, and learners help each other fill the gaps in their knowledge and capabilities.  That said, peer review isn't the only thing that's important.  I know in xMOOCs (Edx, Coursera, et al) Peer Review seems to be the main form of assessment (along with machine graded quizzes), but this type of peer review isn't taken seriously, so I wouldn't give badges just for doing peer review.  I think that an instructor would have to review  the peer review comments and then badges can be provided to peer reviewers.  The better your reviews towards peer's work, the more you can "level up" in the badges you get.

4. How you think badges might result in grade inflation (which pre-supposes that badges exist along side grades, rather than in place of grades?
The #ESLMOOC won't be graded - unless of course the sponsoring organization wants to also give people the option for credit.  That said, graded assignments are not going into the design of the course.  Badges, however, are planned to be part of the design.  My concern with grade inflation is the subjectivity of the assessor.  In theory letter and numeric grades should be based on grading rubrics.  Sometimes people are more generous than others in awarding a grade or marking the rubric than others. If we look at this from a qualitative research perspective, if we draw parallels between grading and coding qualitative data, I am essentially worried about interrater reliability - or in my case "intergrader reliability".  Does this make sense? :)

** End of Addendum **

That's all for this week on badges.  Tune in next week for more ;-)  Your ideas for #ESLMOOC badges?