Friday, January 24, 2020

Academic Facepalm (evaluation edition)

Back in December, I was searching for the #tenure hashtag on twitter.   There was some discussion (probably stated by Jesse Stommel 😜) which prompted me to search for this #hashtag out of curiosity to see what was tagged.  Along with heartwarming stories of people who've just earned tenure (a nice perk right before the winter break!), there was this wonderful tweet specimen...



I'm not gonna lie.  IT BUGS ME.

It bugs me as a learner.  I've always completed course evaluations and I tried to give honest feedback to the professor.  If the course was easy, hard, just right, I wanted them to know.  If I was appreciative, I wanted them to know.  Yes, sometimes I've half-assed it and just completed the Likert scale with a "loved the course" comment at the end, but many times I try to be more concrete about the feedback.

It bugs me as a program manager. I am the individual who sets up, collects, and often reminds students, about the course evaluations.  My colleague is in charge of making sure things like these get into personnel files and maintains department records, and also seems to manager tenure and promotion paperwork for our department (among her other duties).  Faculty committees spend time discussing this each year for merit increases.  So. much. wasted. effort! 

It bugs me as an adjunct.  Yes, I teach for the fun of it. I like helping new instructional designers find their footing.  As an adjunct, if my course evaluations are bad I could be no hired again just for that.  There are no protections.  And, then you've got this tenured individual who openly flaunts their privilege.

Now, don't get me wrong.  I know that Level 1 evaluations are flawed.  They don't measure learning, they measure reactions to the learning event.  But they are feedback nevertheless.   If you don't give a bleep about what students say about your course, one day, despite your tenure, you might not have any students left...

As an aside, I feel like tenure is an outdated institution.  I'd advocate for strong unions over tenure any day of the week.

Your thoughts?

Wednesday, January 22, 2020

MOOC Completion...according to whom?


The other day I had an interesting (but brief exchange) with Kelvin Bentley on twitter about MOOC completion.  This isn't really a topic that I come back to often, given that completion-rates for MOOCs, as a topic, seems to have kind of died down, but it is fun to come back to it. To my knowledge, no one has come up with some sort of taxonomy of the different degrees of completion of a MOOC†.

But let me rewind for a second.  How did we get to the topic of MOOC completion?  Well, I've been attempting to make my extended CV more accessible (to me).  In the past, I used a WYSIWYG HTML publishing platform to manage my extended CV‡.  The idea was that I could easily export it and just push it on the web.  In practice, I never did this, and when I changed computers it became a hassle to maintain. So, I moved everything over to google docs for cleanup (and easier updates).  In cleaning up my CV sections (I am not done, btw!), I did make a startling self-discovery. In the time-period 2013-2016, I binged on a lot of xMOOCs!😅  The most notable platforms were Coursera, Edx, Udacity, but there were others such as the now-defunct Janux (Oklahoma University) and Open2Study (Australia Open University), as well as overseas platforms like MiriadaX and FutureLearn.  In the time period 2011-2012 I didn't have a lot of MOOCs, mostly because during this period a lot were cMOOCs and xMOOCs hadn't really spread like wildfire.

This realization now begs the question: "How many did you complete?" (and you guessed it, Kelvin asked it...).  My answer comes in the form of a question "based on whose metrics and measures?".  When you sign up for a paid course (e.g., professional development seminar, college course, certification prep course, etc.) I think that there is an unspoken assumption that the goals of the course mirror, to a greater or lesser extent, the goals of the learner♠.  Can this assumption be something that transfers over into the world of a free MOOC?  I personally don't think so.  I've long said that the course completion metric (as measured by completing all assignments with a passing grade) is a poor metric.  One very obvious reason to me was that people simply window-shop; and since there is no disincentive to unenroll, people don't take that extra step to leave the course formally, as they would with a paid course where they could receive a refund. I've been saying this since xMOOC completion rates were touted as an issue, but few people listened. Luckily it seems that people are changing their minds about that (or just don't care 😜). I guess George Siemen's advice to Dave Cormier holds true for my own rantings and ravings: publish those thoughts in a peer-reviewed journal or they don't exist 🤪 (paraphrased from a recent podcast interview with Dave).

Assuming that we exclude window-shoppers from our list of completion categories♣, what remains?  Well, instead of thinking of distinct categories (which might give us a giant list), let's think of completion in terms of whose perspective we are examining.  On the one extreme, we have the learner's perspective.  The extreme learner's perspective is characterized by total control by the learner as to what the goals are. In this perspective, the learner can be in a course and complete a certain percentage of what's there and still consider the course as done. Why?  The learner might have prior knowledge, and what they are looking for is to supplement what they already know without going through the hoops of any or all assessments in the course. They've probably evaluated the materials in the course, but if they already know something, why spent a lot of time on something already known? Or, an item that should be done to obtain 100% completion is only available in the paid version (some FutureLearn courses are like this), and are inaccessible to learners on the free tier.

On the other extreme, we have the perspective of the course designer. This is the perspective that most research studies on completion seem to adopt. The course designer is working with an abstracted learner population, with abstracted goals.  The outcomes of the course might be based on actual research into a learner group, they might be based on the intuition of the course designer, or they might just be whatever the course designer has an interest in preparing (sort of like the Chef's soup of the day, it's there, you can have it, but it doesn't mean that this is what you came into the restaurant for).  In a traditional course (the ones you pay and get credentialed for) it makes sense that a learner could simply go along for the (educational) ride because they are paying and (presumably) they've done some research about the course, and it meets their goals. In a free offering, why would a learner conform to the designer's assumptions as to what the learner needs? Especially when a free offering can (and probably does) gather the interest of not just aspiring professionals, but people in the profession (who presumably have some additional or previous knowledge), as well as hobbyists who are free-range learning?

Given those two extremes of the spectrum, I would say that there is a mid-point.  The mid-point is where the power dynamic between the learner and the designer is at equilibrium.  The educational goals (and what hoops the learner is willing to jump through) 100% coincide with what the designer designed. Both parties are entering the teaching/learning relationship on equal footing.  If you lean over a little to one side (learner side), the designer might consider the course incomplete, and if you lean over to the other side (the designer side) the learner might start to feel a bit annoyed because they have to jump through hoops that they feel are not worth their while. Some might begrudgingly do it, others not, it really depends on what the carrot is at the end of that hoop.  For me, a free certificate or badge did the trick most times. The threat of being marked as a non-completer (or more recently the threat of losing access to the course altogether 😭) however does not motivate me to "complete" the course on the designer's terms.

That said, what about my experience?  Well... my own behaviors have changed a bit over the years.  When xMOOCs first hit the scene I was willing to go through and jump through all the hoops for the official completion mark.  I did get a certificate at the end; and even though it didn't really carry much (or any?) weight, it was a nice memento of the learning experience. Badges were custom made (if there were badges), and the certificates were each unique to the MOOC that offered them.   Back in the day, Coursera had certificates of completion (you earned the minimum grade to pass), and certificates of completion with distinction (you basically earned an "A").  It was motivating to strive for that, even though it didn't mean much. It was also encouraging when MOOC content was available beyond the course's official end, so you could go back and review, re-experience, or even start a bit late.  As we know, things in the MOOC world changed over the years.  Certificates became something you had to pay for.  Sometimes even the assessment itself was something you had to pay for - you can see it in the MOOC but you can't access it.  Peer essay grading on coursera wasn't something that I found particularly useful, but I was willing to jump through the hoops if it meant a free moment at the end of the course (achievement, badge, certificate, whatever). Once things started having definitive start- and end- dates♪ , and content disappeared after that when certificates (which still we're worth much to the broader world) started costing money, the jumping through the same silly hoops (AES, CPR, MCEs, etc.) it just didn't feel worthwhile to go above my own learning goals and jump through someone else's hoops.

So, did I complete all those MOOCs?  Yup, but based on my own metrics, needs, and values.

What are your thoughts on MOOC completion?  Do you have a different scale? Or perhaps defined categories?





Marginalia:
† There may be some article there somewhere that I've missed, but in my mission to read all of the MOOC literature that I can get access to, I haven't found anything.

‡ What's an extended CV?  It's something that contains everything and the kitchen sink.  That workshop I did back in 1999 for that defunct software?  Yup, that's there...because I did it, and I need a way to remember it. It's not necessarily about the individual workshops, but about the documenting of the learning journey.  The regular CV is somewhat cleaner.

♠ Maybe this assumption on my part is wrong, but I can't really picture very many reasons (other than "secret shopper") that someone would pay money to sign-up for a course that doesn't meet their goals.

♣ Window-shoppers I define as people who enroll to have a look around, but either have no specific educational goals they are trying to meet (e.g., lookie-loos), or have goals to meet, but they deem the MOOC to not meet them (e.g., "thanks, but not what I am looking for"). Either way, they don't learn anything from the content or peers in the MOOC, but at the same time, they don't unenroll since there is no incentive to do so (e.g., a refund of the course course).

♪ e.g., module tests deactivating after the week was over and you couldn't take them - AT ALL if you missed that window

Wednesday, January 1, 2020

A decade in review...onward to 2020!


I didn't quite expect this, but it seems like everywhere you turn you see "a decade in review" news stories (radio and TV), "the internet" (in general) and blog posts, twitter threads, and Instagram stories (more specifically).  I hadn't really thought about doing one of these posts, but what the hay, why not join in? 😜 . The last decade has certainly been eventful.  I kicked off the decade by completing my last 2 master's programs, changing jobs (3 departments and 4 titles in the last 10 years), starting to teach, and participating in research.

I absolutely loved Audrey Watter's 100 debacles of Ed-Tech, so I decided to pick a few and structure my post around this since most of these made an impact on my work-life, and some for my leisure. I am not going to pick through every one of those items, but I'll pick a few (and maybe add some of my own).

New Media Consortium (Horizon report #100)
This one was a shocker for me. The way the NMC just ceased to exist was something I'd expect only from a VC-funded start-up. In the last decade, I was able to attend both conferences that were offered in Boston by the NMC.  I enjoyed both, and I made quite a few interesting contacts via those conferences. I also used the Horizon Report as something in the courses I teach.  Not necessarily as something ultra definitive, but something to hone the critical skills of students in my courses (and have some fun prognosticating). While the Horizon Report has been picked up by Educause there is something distinctly different about the feel of Educause as compared to the NMC. Speaking of conferences that went bust:  Campus Technology.  I used to attend CampusTech every year.  It was held in Boston which made it super convenient, it had exhibition floor passes for free (which also meant that you could just attend the presentations if you snuck in), and they had a pretty liberal press pass policy which allowed me to attend for free as being affiliated with my school's paper, and later the CIEE journal. It was also co-located with AAEEBL which basically meant two conferences for one low price of free (for me).  Good times! They will be sorely missed.  I did learn a lot (even if you factor in the amount of hype).


Ning (#98)
Ning is something I came across while I was an MEd student in Instructional Design. Ning, along with SocialGO and Elgg are white label social networks which allow anyone to fairly easily build communities.  SocialGO was never free (it has a free trial), and Elgg is open source - which while great it does require the user to provide a fair amount of their own IT maintenance.  Not being in a position to do that, Ning hit the sweet spot of free basic hosting (up to 200 users for free?) and no server update and maintenance hassles.  Using Ning I built two networks, one for my MEd program (UMassID.com), and one for the Applied Linguistics Department (this is before I started working for them).  After Ning eliminated their free model I did garner enough support for UMassID for a few years, but each year I felt like I was looking for champions to pay for the $200 annual fee.  Most times I was successful, but at some point, I just felt like I didn't want to keep poking the champions for money any longer. We still use Ning for our department's portal, which makes it easy to post information for students, but also keep our alumni in the loop.  With all the changes happening in terms of who owns the platform, I fear that I might need to think of migrating at some point in the future.  I wonder how successful I would be in convincing my university to adopt Elgg, sort of like AU has with their Landing.


Badges (#86)
Open Badges are something that is was really pumped for. I am not sure I am all that disappointed that they haven't taken off like wildfire.  Any long-term change in credentialling does take time to have something accepted and endorsed. As a gamer, I liked badges because they are very much like achievements.  You can have smaller achievements to push you along, and you can have larger achievements (or stackable badges) that allow for much more descriptive information about what someone is capable of. Over the last 9 years we've had the Mozilla backpack, Badgr.io, credly, and Purdue's Passport; I am sure that there are more but those are the ones I've dabbled with.   Now, what I am disappointed in are two things:   (1) everything closing up, and (2) the fragility of badges.  Over the past year, Mozilla stepped out of this arena and migrated things to Badgr.  Credly is shuttering their free version (which allowed folks to create and distribute their own badges like I did for my classes) and replacing it with a paid version (Acclaim).  I was able to download all of my badges from my backpack and upload them to Badgr (which seems to have the capability to freely create badges), but this brings me to the second problem:  badge fragility.  A number of the badges I've earned over the past 8 years are not importable into my backpack because URLs are no longer accessible (and hence not verifiable).  Now, I know that I have those badges, I can post them on this blog or website, but it does pose a problem for the long-term viability of badges.  If someone wants to verify my diplomas they can contact the Registrar's office at my university and they can confirm that I've completed certain areas of study.  With badges, this is currently an issue.


Google Reader (#54)
**sigh** OK this still stings.  Damn you google! Google Reader wasn't just an RSS reader, it was a community.  I could subscribe to RSS feeds of my favorite blogs.  I could follow people from my contacts, and I could follow other people on Reader.  I could upvote RSS posts, and I could share and basically create an RSS feed of my shared items.  While I can (and do) use another reader now (Feedly) it's just not the same.  I end up sharing stuff I read on Feedly on my twitter accounts, but that ends up seeming like a lot of spam (because I read and share).  I feel like this change on the web has also made changes to sites.  Whereas in previous years with Reader I could get the entire news post in my feed now most sites give you a meager 3-5 line previous and you have to click to go to the site for the full thing.🙄.  There just isn't a satisfactory replacement for Reader.  The Old Reader is pretty close in terms of usability but it really lacks the network which made Google reader amazing!  Damn you google! 


Google Glass (#26)
In this past decade, I started traveling again after (what seemed to be) a long hiatus from such activities.  I love traveling. I love seeing new things and experiencing something different.  One of my travels brought me to Italy, to Herculaneum and Pompeii.  The experience was amazing, and I loved how preserved these cities were.  No matter the level of preservation they are still ruined.  As I was walking through the streets of these ancient cities I was thinking was an awesome use case for google glass.  One could use augmented reality as they walked through the streets to see buildings in their full glory. They could see Romans walking through the street while being busy with day to day life. They could hear the sounds of a dead language being spoken again all around them. Put in some QR codes and some geotagged locations and you've got educational pop-ups!  This sounded like a great vision of the future. I was thinking in terms of tourism, but it could easily apply to learning.  Sadly this is not the case...


MOOCs (#4)
Well, technically Audrey's #4 is the phrase "year of MOOCs", but I think that items like "promise of free" (#99) and "UC Berkeley Deletes Its Online Lectures" (#67) could fit in here. MOOCs may be seen as flops, and perhaps for some things they may be. But, as Siemens recently wrote in a twitter thread, MOOCs aren't out yet (OK, paraphrasing here).  Just because we (in North America) are "done" with them, it doesn't mean that others are done with them.  They may make a monumental comeback depending on who they evolve outside of our continent.  I still think there is a lot of promise for MOOCs (and I look forward to the regenesis of the cMOOC), but there are attributes of xMOOCs that really have bugged me over the past few years.  When there were only one or two MOOCs happening at a time, it was perfectly manageable for me (as a learner) to jump in and participate.  When options for providers and topics exploded, it became hard.  I chose which MOOCs to attend in real-time, and which ones to do as a self-paced learner.  Well, it seems like self-paced is not really much of an option the way that things have evolved in the xMOOC world any longer.  Once the course is over, unless you've paid for it, it becomes locked and unavailable.  xMOOCs have embraced a freemium model that takes away agency on the part of learners.   Now, I hope MOOCs survive because they are a part of my own balanced learning diet, but I do hope that providers and designers keep tweaking the recipe.  The freemium model doesn't really work all that well for things that you aren't credentialing people for.  Maybe in the coming decade, we'll see a resurgence of the cMOOC 😀.


I don't want to close out the blog post with only negatives, so I think I should mention some positives.  This past decade was about networks! Through networks, I "met" a lot of interesting and intelligent people who have positively impacted my life.  These are people I've learned within MOOCs.  These are people with whom I've conducted research.  These are people with whom I've virtually connected  (and in some cases even met face to face!).  These are people on twitter chats, on DMs, and at conferences. And also people in my doctoral cohort(s).  Even though technology might not always work for us, the people involved have made the last decade on the web a supportive learning environment for me, so thank you all for the MOOCs, the mLearning, the Rhizos, the lurking, the critical ID, the book and article recommendations, the conference crashing (like wedding crashers, but for conferences), the dissertation encouragement, and so much more! I hope the learning continues in the decades to come 😎