Analyzing the Synthetic Syllabus

Bing Image Creator: A Syllabus

Wow...it's been almost two months since I started this post! It's hard to believe that it took this long to return to this thought experiment.  Just to remind the diligent reader of this blog, this came out of not one, not two, but multiple places on the web (including professional development conferences!) whereby instructional designers (and other professionals) were demonstrating the use of GPT to put together quick and dirty course outlines for the busy adjunct. While I've got issues with this framing, I'll put those aside for now.  I thought that it might be interesting to actually go through the process to create a course outline and syllabus for a course that I used to teach often before I started my doctoral journey. The course is INSDSG 684: The Design and Instruction of Online Courses, a graduate course in the MEd program of Instructional Design at UMass Boston.

I think it's important to start with a course that you know so that you can both critique the design "decisions" that GPT makes and also leave yourself open to being pleasantly surprised by what an LLM produces.  The three AI services I tested were SMARTIE, ChatGPT 3.5, and Gamma. I won't comment on SMARTIE extensively because I chose to run another course that I used to teach frequently (INSDSG 601: Foundations in Instructional Design and Learning Technology) through it so I could do a compare/contrast, but I got a bit bogged down enough to just junk that attempt. If you're curious as to how far I got, here's the unannotated Google doc of the SMARTIE 601 syllabus/outline.

So, back to 684.  Here's the ChatGPT syllabus with my comments, and here's the Gamma PowerPoint output (which is still in beta). Feel free to go through and make your own annotations.

Some big-picture things I've noticed:

  • It took me over 90 minutes to prompt CGPT for everything, copy it into the Google Doc, and format it. This doesn't delete duplicates, critically analyze the syllabus, and address duplications and inconsistencies.  If I were called upon to teach this course again, with no current syllabus or list of learning objectives from the department, it would take less time than prompting CGPT to produce something for me.
  • CGPT produced a list of required and optional texts for the course, and while most were legitimate (i.e., they exist), it really did incorporate some made-up stuff in that list, which means that anyone using this really, really, really needs to fact-check those books and articles.
    • It's worthwhile noting that GAMMA did the same thing with the course text. It totally made up a textbook. How do I know? I couldn't find it on Google, on IGI's website, and it's not listed on the faculty profile of the supposed author.
  • CGPT produced a plan which made for both an uneven, and an unreasonable workload for learners. For example, just for Week 1, the assigned readings were around 450 pages in two assigned books. This pattern continued through Week 10, while Week 11 only had 2 articles (maybe 40 pages between both). Alignment between learning objectives for the course, the weekly topic, and the readings was there in a way, but boy was it a lot!
  • Speaking of additional resources, CGPT recommended videos, podcasts, and articles (at my prompting), but didn't include any URLs.  I know that this is an issue, but when specific resources are given, resources that have defined URLs and DOIs (e.g., Teaching in Higher Ed Podcast and Sal Khan's TED talk - Let's Use Video to Reinvent Education) should be no brainers
  • It seems like CGPT has a weird fascination with pairs.  If you skip down to the Activities section of the syllabus, you'll see that every week basically has two activities for students to complete. While CGPT gets points for keeping things interesting and varied, I am having a hard time seeing how the learning community gels if they're jumping from task to task.
  • The criteria points for graded assignments were basic and repetitive for all assignments.  For what it's worth, I felt the same way about rubrics produced in SMARTIE for the other class.
  • Going back to student overload for a moment, I don't think the activities and the course deliverables were really all that integrated (or at least it seems like CGPT didn't consider what had come before from past prompts when it provided me with activities) because the grading breakdown for the course not only didn't consider the activities and assessments that came before, it added to them!
  • In terms of course policies, was pleasantly surprised that the camera policy for live sessions is "camera optional"
  • Also, in policies, I think that many things that were presented as a "policy" seemed more like classroom norms to discuss and engage with learners about
  • GAMMA did produce some headscratchers - such as "screen recording" being used in the design of the course. I don't know how germane that is to learners in the course.
  • Finally, with the exception of a hiccup in CGPT, it seems like a lot of final projects for this course are an online course designs, which is to be expected.  In fact, I think both all 5 instructors for this course (going back to 2008) have had this as a final deliverable.
Overall, I think that if you're a subject expert, and have taught the course many times, LLMs won't really save you that much time. You're better off going with your gut, or asking a colleague who teaches a similar course to see if you can build off their syllabus - if you're really in a crunch.

Your thoughts?

Comments

Post a Comment

Popular posts from this blog

Latour: Third Source of Uncertainty - Objects have agency too!

Image
Continuing on my exploration of ANT, and asynchronous and indirect dialogue with Latour - this blog post will cover the third source of uncertainty, which according to Latour, is that Objects have agency too! As with the previous blog posts, I've pulled out quotes from the book that seemed interesting, or that I reacted to in some way, and I am responding to them here. no tie can be said to be durable and made of social stuff (p. 66) This quote seems to continue Latour's assertion that there is no such thing as "social" or "social stuff" and that "social", or the meaning of needs to be negotiated and better understood.  It also continues the thought that social can only be seen from the actions of its actors, the traces they leave behind, and that these bonds are not durable because they need continuous reinforcement. I guess Social is a perishable item. Left to its own devices, a power relationship that mobilizes nothing but social skil
Read more

MOOC participation - open door policy and analytics

The other day I was reading ZML Didaktik on the topic of MOOC participants. In MOOCs, one of the big questions is why are people lurking and not participating? If more than 500 people join a MOOC, why are only 10% contributing with any amount of regularity?  On the same blog, in a previous blog post, I had commented (it was an open stream of thought really) that perhaps there should be an open enrollment period, and then if the system sees that certain participants are bellow a threshold of activity, the system may give them the option to self-identify as a lurker, or un-register from the class. This line of thought went along a view that compared to the traditional classroom; in a traditional classroom students register for the course, they can attend classes for a week and then decide whether or not they want to stay with the course or not. If they do stay with the course they know that there is a certain amount of "lurking" that they can get away with, but they do have
Read more

You've been punk'd! However, that was an educational experience

Image
It's now the end of Teaching Goes Massive: New Skills Required (aka #massiveteaching) on coursera. Well, almost, we still have a couple of days left. I guess that the lesson here is that we were (the "learners") were punk'd† by Paul-Olivier Dehaye of the Univerisity of Zurich. After that last blog post (and subsequent pickup of the post by George Siemens and others) Inside Higher Education and the Chronicle wanted to chat with me about this MOOC experience. I only had time for one of the two before things went to press, so my 15 minutes of fame went to IHE.  Both IHE and the C hronicle have written about the topic, and have received some information from Coursera on the incident.  Others have also written about it (see here , here and here ) It's surprising to me that the University, and Coursera, waited until after this thing was a big issue in order to respond in the class, and clue people into what was happening. On Wednesday (day 3 of Week 3/3 of the
Read more