Tuesday, November 25, 2014

DALMOOC episode 7: Look into your crystal ball

Whooooa! What is all this?

Alright, we're in Week six of DALMOOC, but as usual I am posting a week behind.  In previous weeks I was having a top of fun playing with Gephi and Tableau. Even thought the source material wasn't that meaningful to me I was having fun exploring the potential of these tools for analytics. This week we got our hands on Rapidminer a free(mium) piece of software that provides an environment for machine learning, data mining and predictive analysis. 

Sounds pretty cool, doesn't it?  I do have to say that the drag and drop aspect of the application does make it ridiculously easy quickly put together some blocks to analyze a chunk of data. The caveat is that you need to know what the heck you are doing (and obviously I didn't ;-) ).  I was having loads of issues navigating the application, and I somehow managed to not get some windows that I needed in order to input information to, and I couldn't find where to find the functions that I needed...  Luckily one of my colleagues was visiting who is actually working on machine learning and was able to give me a quick primer on Rapidminder - crisis averted.  I did end up attempting the assignment on my own, but I wasn't getting the right answer.  With other things to do, I gave up on the optional assignment ;-)

With that software experience this past week, what is the use of prediction modeling in education? Well (if you can get your software working ;--)  ), the goal is to develop (and presumably use) a model which can infer something (a predicted variable) from some combination of other aspects of data that you have on hand (a.k.a. predictor variables).  Sometimes this is used to predict the future, and sometimes it is used to make inferences about the here and now. An example of this might be using a learner's previous grades in courses as predictors for future success.  To some extent this is what SATs and GREs are (and I've got my own issues with these types of tests - perhaps something for another post).  The key thing here is that there are so many variables in predicting future success. It is not just about past grades, so take that one with a grain of salt.

Something that goes along with modeling is Regression: You use this when there is something you want to predict and it is numerical in nature. Examples of this might be number of student help requests, how long it takes to answer questions, how much of an article was read by a learner, prediction of test scores, etc. A regressor is a number that predicts another number.  A training model is when you use data that you already know the answers from and try to build a model to teach the algorithm.

There are different types of regressions.  A linear regression is flexible (surprisingly so according to video), and it's a speedster.  It's often more accurate than more complex models (especially ones you cross-validate). It's feasible to understand your model (with some caveats).

In watching the videos last week, some examples of regression algorithms I got conceptually from a logic perspective, but some just seem to go right over my head.  I guess I need a little more experience here to really "get it" (at least from an applied sense)

Another way to create a model is Classification: You use this when there is something you want to predict (label) and that prediction is categorical, in other words it is not a number, but a category such as right and wrong; or will drop, or persevere through course. Regardless of the model you create, you always need to cross validate the model you are using for the level you are using it in (e.g. new students? new schools? new demographics?) otherwise your model might not be giving you the information you think it's giving you.

This week, for me, was yet another reminder that I am not a maths person.  Don't get me wrong, I appreciate the elegance of mathematics, but I honestly don't care about optimizing my algorithms through maths.  I'd like to just know that these certain x-algorithms work for these y-scenarios, and I want easy ways to use them :)  Anything beyond that, for me, is overkill.  This is probably why I didn't like my undergraduate education as much as I've enjoyed my graduate education:  I wanted to build things, but my program was focusing on the nitty gritty and engine performance :)

  • Alternative episode title: Outlook hazy, try again later
  • Neural Networks have not been successful methods (hmmm...no one has told this to scifi writers ;-) sounds cool, even though they are inconsistent in their results)

Monday, November 24, 2014

Designing in the Open (and in connected ways)

Wow, hard to believe, but we've reached the final module of Connected Courses (and boy is my brain tired!).  I found out last week that there may be a slim chance of me being able to teach Introduction to Instructional Design (INSDSG 601, a graduate course) at some point in the new future. This is something that was offered to me a couple of summers ago, but being away on vacation at the time (with questionable internet access) it didn't seem like a good idea to be teaching an online course.

I've been poking around the course shell, here and there, over the past couple of years (even since teaching this course was a remote possibility) to get ideas about how to teach the course.  The previous instructor, who had been teaching this course for the past 10 years but recently refocused on other things, did a good job with the visual design of the course. It's easy to know what you are are supposed to do each week.  Then again, from the design of the course I can see that the the focus of the course each week seems to center around the instructor (each week has lectures in addition to chapter readings), and we saw in the cited literature in #dalmooc that this isn't pedagogically effective.  This is something I've been wanting to change.  The other thing that I don't like is the reliance on the Dick & Carey textbook. Granted, this textbook seems to be a seminal book in the field, but it is not the easiest thing to read for a novice learner (who is also figuring other things out about the ID field) and in my experience most learners read it, but don't really get the fine grain elements. This book, in my opinion, is a good reference book, but not necessarily a good instruction book†. The thing that really convinced me to scrap this course and start from scratch with a new design is that the assignments seem to all be assignments (50% of final grade) that built on top of one another culminating in a final project (the other 50% of final grade) are all taking place in the forums.  The project-based aspect I like, and I also like the peer review aspect.  However, I don't like this double-counting of points, and the closed nature of the course (everything happening in an LMS). So, here we go with a re-design (if I know I am teaching the course)!

The learning objectives (that I can't really mess with) are as follows:
  • State the reason for using an Instructional Design Model. 
  • Identify and describe the purpose of each component of the Dick and Carey Model of Instructional Design. 
  • Develop instructional (performance) objectives that include behavior, condition and criteria.
  • Develop an assessment strategy for an instructional event. 
  • Develop assessment items that map to instructional objectives. 
  • Develop an instructional strategy that maps to learner needs and performance objectives. 
  • Plan a formative evaluation strategy to assess instructional materials. 
  • Compare the Dick & Carey ISD model with other models
Since this is an intro course, my own additional objectives for this course are to (1) setup learners to be able to find and retrieve sources from our academic library, and (2) begin creating their own repository (aka "toolbox") of resources that they can make reference to not only as they progress through the program, but also as they become working professionals.

I have some ideas for assignments to reach these goals, however I am a bit stuck.  I want my course design to be 100% (or at least 90% if I can't reach 100%) open access materials.  Students would be free to go and find and retrieve textbooks, articles, and resources from pay-walled sources, but the materials I provide need to be 100% open access. This means I need a new textbook (or an un-textbook).  What would you recommend for resources for an introductory course in instructional design as far as open resources go?  Dick & Carey are having me do some mental gymnastics (ADDIE seems to have more free/open resources on the web than D&C).

As far as lectures go, I am thinking that lectures in the course are automatically out.  The current lectures all start with "Hello everyone, I am Dr. so-and-so". Since I am not Dr. so-and-so, this is an unnecessary cognitive barrier for learners, and in all honesty I don't want to sit down and do 13 weeks worth of lectures. I think there are much more fun ways to spend my time, and help my learners navigate the subject, than 30-45 minute lectures each week.  If I had enough buy-in I'd love to get onto a Google Hangout and have a recorded discussions with some of the great minds, and leaders, in instructional design to discuss topics of ID including mobile learning, distance education, corporate training, and so on  - you know, things that will get the learners thinking about how to structure the remainder of their studies, pick areas to focus on, and what they might want to be lifelong learners in.

So, initial brainstorming post - open resources!  What do you think kind reader?

In subsequent posts (if this goes forward) I think I am going to focus on activities, other materials, and flow of the course.  If you want me to write about other subjects as well leave a comment :)

†other faculty of instructional design please feel free to chime in! I what to know what you think about Dick & Carey.

Thursday, November 20, 2014

Attack of the untext - my own stumbling blocks

It's been a while since Rhizo14 ended, but the community is going strong! Facebook may not be as active (or maybe facebook is  hiding most Rhizo posts from my timeline...that could be it...anyway), but we are still chugging along with the collaborative *graphy. I can't call it an ethnography, or autoethnography because variables have changed.  Some of us decided to get together and write an article for Hybrid Pedagogy on why the Collaborative *graphy article is taking so long (a meta-article if you will) but we got stuck there too (or it seems as though we are stuck).  I think others have written about their own personal views on this on their own blogs, so I've been working out what my own stumbling blocks are with this project. I think I have a way to explain things now!

So, when working collaboratively in previous collaborative work situations your final product feel unified.  The main analogy that I can give give is the main root of one plant which looks like this:

In this example (going with the rhizome metaphor) you have one main path, and all the side paths are footnotes, citations and references, and end-note commentary.  The coloring is all the same because regardless of where you have one author or many authors the final product sounds like a unified voice.  Many ideas have come into, and go out of, this main line (see expansion roots in the model), but at the end of the day those side roots don't overtake the main idea or argument.

The original project started as a collaborative autoethnography (CollabAE).  This eventually became an issue because some people stepped back from the  project and thus is was no longer an autoethnography for the entire MOOC, but rather an multi-author ethnography (MAE) of the MOOC. We could use other people's anonimized data, assuming that we had their permission. At that point it wasn't introspective (auto-ethnography) but rather analytic - but this seemed to lack the rhizomatic aspect (to some extent anyway) that made the CAE unique in this aspect, and there were issues of silencing voices (or inadvertantly silencing voices since some people didn't want to be authors, or weren't comfortable with their views being part of this analysis). Things got busy with school, work, and others pursuits that I lost track of the CAE.

The CAE, at least the way we collected data, looks like the image above.  Each color represents a different author, and each other has (probably) a main point and certain supporting literature, tangents, side-points and so on that they made in their original write up. Some authors connect to other author's writings, and this is visualized above as roots crossing through other root's paths.  As chaotic as this may look, it does make sense. I think the closest analogy for this would be George Veletsiano's Student Experiences of MOOCs eBook. To some extend (being a bunch of experimental academics ;-) ) we may have over-thought this CAE.  In hindsight, I would say that this should be a multi-step process.  Perhaps the first step in the process, with a deliverable, would be an eBook, similar to Veletsiano's style, of our Rhizo experiences.  Here people can write anonymously or  eponymously.  Submitted chapters could go through peer review, but not the traditional academic peer review - the peer review that aims to disambiguate and seeks to grow those side-roots a bit in case eventual readers want to do down the paths.  There could be a foreword (Dave Cormier perhaps?) but the reader would be left to read, process, and make sense of each individual story.  As such this could be not a collaborative AE  but a cooperative AE (CoopAE), people working concurrently, but not necessarily together to get this done.  One big, overall, document, but each chapter can stand on its own.

So since the CollabAE wasn't going far, a couple of people thought we could sit down and write an article about what's up with this process.  Why are things taking so long?   The visual for this untext looks something like this (according to me anyway).

Whereas the CollabAE has separate, but distinct, stories where others commented on, but didn't necessarily write-over the text, in our meta-analysis I am seeing both original (concurrent) threads emerging (two or more people writing at the same time but not about the same message). This is represented by different color main-roots.  Then I am also seeing people expanding on those main-roots (different color sub-roots) by either adding onto the document, or having side conversations.  I have to admit that this is fascinating as a brainstorming piece, and it could be considered by some as a performance piece or something alternative like #remixthediss.

That said, however, the problem is that we don't have an audience.  A document as chaotic as this one is helpful to us as authors to help us better understand our own positions on things, and to better help us understand or analyze our own lived experiences in the MOOC.  However, I am not convinced that this is geared toward a reading audience. It's not necessarily something that they expect, and I am not sure how a reader will process this chaos.  For me, at the end of the day, I go back to my goal.  What is the goal of the *graphy project (decided to change it's name since CollabAE and CoopAE seem to not describe it)?  What is the goal of the untext about the *graphy project? Is the goal something for the internal constituents? Something for the public?  If it's for both, what's the overlap space where the final published product would be useful (and comprehensible) to both?  Good questions.  I've got my own answers, but as a group...I don't know :)

As a side note, this seems like an interesting application of co-learning (see connected courses for more details)

Monday, November 17, 2014

DALMOOC episode 6: Armchair Analyst

Week 6 CCK11 blog connections
I was trying for a smarter title for this episode of #dalmooc thoughts, but I guess I have to go with Armchair Analyst since I ended up not spending a ton of time with either Gephi or Tableau last week. So, the reflection for week 4 is mostly on theoretical grounds; things I've been thinking about (with regard to learning analytics) and "a ha" moments from the videos posted.

I think week 3 and week 4 blend together for me.  For example, in looking at analytics the advice, or recommendation, given is that an exploration of a chunk of data should be question driven rather than data-driven.  Just because you have the data it doesn't necessarily mean that you'll get something out of it.  I agree with this in principle, and many times I think that this is true.  For instance, looking back at one of our previous weeks, we saw the analytics cycle.  We see that questions we want to ask (and hopefully answer) inform what sort of data we collect and potentially how we go about collecting it.  Just having data doesn't mean that you have the data that you need in order to answer specific questions.

On the other hand, I do think that there are perfectly good use cases where you might be given a data-dump and not have any questions.  Granted, this makes analysis a bit hard, like it did for me the last couple of weeks.  This data (anonymized CCK11 data, and sample evaluation data for Tableau) didn't really mean much to me, so it was hard to come up with questions to ask.  On another level I've been disconnected from the data, so it's not personally meaningful as a learner (CCK11 data was anonymized), and since I didn't have a hand in organizing, offering, and running CCK11 it's not as useful for me as a designer.  However as a researchers, I could use this data dump to get some initial hypotheses going.  Why do things look the way they look?  What sort of additional, or different, data do I need to go out and test my hypothesis?  How might I analyze this new data?  As such, a data-driven approach might not be useful for answering specific questions, however it might be a spark to catalyze subsequent inquiry into something we think might be happening; Thus helping us formulate questions and go out and collect what we need to collect to do our work.

So, for example, I have just started my EdD program at Athabasca University.  I have a lot of ideas running through my head at the moment as to what I can research for a dissertation in 3 years†. As I keep reading, I keep changing and modifying my thoughts as to what do to.  I may be able to employ learning analytics as a tool in a case study research approach.  For instance, I teach the same course each spring and fall semester, an online graduate instructional design course on the design and instruction of online courses (very meta, I know). The current method of teaching is quite scaffolded, and as Dragan was describing last week (or this week?) I tend to be the central node in the first few weeks, but my aim is to just blend in as another node as the semester progresses. This process is facilitated through the use of the Conrad & Donaldson Phases of Engagement Model (PDF).

So, one semester I can use this model to teach the course and another semester I might create an Open Online Course based on the principles of connectivism and run the course like that. I'd have to make some changes to the content to make sure that most course content be Open Access content, that way I would be eliminating some variables, but let's assume I've done this and I'm just testing Connectivism vs "regular" graduate teaching (whatever that is).  I can then use SNA, as one of my tools, to see what's happening in these two course iterations. I can see how people are clustering together, forming communities (or not), how they are engaging with one another and so on. This analysis could be an element of the study of efficacy of connectivism as employed in a graduate course‡.

On the other side of things, if I were to just stick with my traditional online course, I could still use SNA to improve my course.  One of the things that I notice is that some groups tend to form and stay together early on in the semester.  These seem to be informal groups (person X commenting on person Y's posts throughout the semester more than they do for person Z). Since the semester is 13 weeks long, a JIT dashboard of course connections would be useful to both encourage people to find study groups, but also to engage more with people that they don't normally engage in.  People who usually post late in the forums (at least in my experience) don't often get many responses to their posts, which is a real pity since they often bring some interesting thoughts to the discussion.

A good example of this is the image above, the CCK11 blogs from Week 6.  I see a number of disconnected blogs.  Were these blogs never read (as measured by the click-through rate on the gRSShopper Daily)? Were they never commented on by anyone? Some of the blogs may not speak to anyone in the course, but in a course of 1131 participants (citation), assuming a 80% drop off by week 6, that's still around 200 people active in the MOOC, why is not one connecting with these posts, and can we do anything to spur participation?  Maybe an adopt a blog post campaign?  This is also where the quantitative aspects of SNA mesh with the qualitative aspects of research. Here we could also do an analysis of what gets picked up (those connected nodes) to what doesn't get picked up, and do an analysis of the text. This might help us see patterns that we can't see with SNA alone.

That's it for week 4.  And now I am all caught up.  Welcome to week 5!  Your thoughts on Week 4?

† The more I think about this, the more I am learning toward a pragmatic dissertation rather than a "blow your mind" dissertation. I see it more as an exercise that will add some knowledge in this world, but given that doctoral dissertations are rarely cited, I am less interested in going all out, and more interested in demonstrating pragmatics of research through a topic of interest. Thoughts on this? I definitely don't want to stick around in dissertation purgatory.
‡ I'm pretty sure that someone (or quite a few) have written about this, especially with regard to CCK08, but let's just roll with this example.

Thursday, November 13, 2014

DALMOOC episode5: Fun with Gephi

CCK11 Tweet visualization
Alright, after a few days of being sidelined with a seasonal cold, I'm back on #dalmooc.  Still catching up, but I have a feeling I am getting closer to being at the same pace as the rest of the MOOC ;-)  In any case, this is a reflection on week 3 where we started messing around with social network analysis (SNA).  This is cool because it's something that I had started doing on another MOOC on coursera, with Gephi, so it was an opportunity to get back on and messing with the tool.

So, what is SNA?  SNA is the use of network theory to analyze social networks.  Each person in this network is represented by a node (or edge), and nodes can be connected to other nodes with a vertex (or many vertices). These connections can indicate a variety of things (depending on what you are examing), however for my usage in educational contexts I am thinking of vertices as indicators of message flow, who sends messages to whom in a network, and also who refers to whom in a network. I think this latter one is interesting from an academic citation point of view as well.

As was pointed out in week 3, SNA can help discover patterns of interaction in online learning environments. I think that it can also help up discover patterns in physical environments, however this is harder because we don't have big brother watching the physical environments as much as we can collect data about patterns of participation in virtual environments. It's much more labor intensive to keep accurate track in a physical environment.

An interesting application of SNA is its use in understanding learning design (Lockyer et al - in video). We can use SNA to analyze patterns of interaction in courses that we design an implement, thus we can (to some extent) see how our designs are affecting the learner's patterns of participation.  While this is feasible, I think that it's also hard to keep the variables pinned down so that there are no confounding errors. If you've designed an online course (that is NOT self-paced) you can see the same course design taught different ways if you put different faculty in the driver's seat.  As such I think that in studies using SNA to analyze course design (and/or teaching methods) it's important to account for all variables.

Other interesting things from Week 3:

An Instructor-centered network is one where the instructor is central node in network. These are recognized in literature as only leading to lower levels of knowledge construction (see Bloom's taxonomy). Related to this type of network is having one learner have a dominant role in a course, thus the instructor is replaced (or shares the spotlight) with a dominant learner.  This is also not desirable from a pedagogical point of view. One can start with an instructor-centered environment and facilitate the change to a P2P interaction. Students will need scaffolding in order to reach that P2P network.

Sense of community is predictor of success in educational endeavors. A common way of collecting this type of data is questionnaires, and I think that in education this can be both in-class as part of a mid-term temperature check in the course, but also in the final course evaluation.  I am wondering, however, how accurate this self-reporting is. Is this just an affective measure? Or can learners feel like they are lacking a sense of community but in reality have it but not get as much as they feel they need?

Network brokers are nodes that connect two or more communities in a network and have a high degree of centrality.  These network brokers can see information across many different communities, and as such can have access to many different ideas flow through them. Network brokers are associated with high levels of achievement and creativity. So, in an educational setting it's good to be a network broker.

Cross-Class networks are latent ties by attending the same events, so even though I am not connected with many people in #dalmooc (at least on twitter I don't retweet or write to many others in the MOOC - maybe I should...) I am connected to other people through the course hashtag and by attending the same event. In a traditional learning setting this could be likened to participating in a learner community such as UMassID.com (our instructional design community) or the Athabasca University Landing network.

CCK11 Blogs, week 6
Next up, the Gephi portion of this post.  I've been messing around with Gephi data from CCK11. I was quite excited to get my hands on the CCK11 data to mess around with in Gephi until I remembered that I didn't tweet all that much in CCK11...D'oh! I was curious to see where I was  in the network of connections.  Even if I were active I don't think I'd be able to see myself there because the data appears to be anonymized (and rightfully so).

I did run some analysis of the blog connections in CCK11 using Gephi again (part of the data dump available in #dalmooc) and here was a place where I expected to see myself and see who I was connecting to, however, again, the data was anonymized. My question entering into this analysis was more about analyzing my own patterns of interaction.  I was new to MOOCs back in 2011 and CCK11 was the MOOC where I really started learning about connecting with others for academic purposes. Thus, I wanted to see what my initial and developing connected literacies pointed to. Alas, this was not to be :-)

As Dragan mentioned in one of the videos of this week, analytics should be question-driven, not data-driven. Just because you have data, it doesn't mean that you should use it, or that you will find anything meaningful in it.  This was the case with me and this data. There were some interesting visualizations, but I wanted to have a look at the people involved, who connected to whom, and look more at the qualitative connections: what posts, what ideas, what types of know-how got distributed throughout the network and by whom. It's a little hard to do this with anonymized data, so you really need to abstract and think at higher levels when working with this.  If we had data from other MOOCs, this type of anonymized data could be useful to compare patterns of participation of one MOOC to another.

Thus concludes my week 3 experiences.  What were your thoughts?