Badge MOOC Challenge 5: Authentic Assessment and Evidence for a Badge Ecosystem
 |
| The real badge? |
So, as with previous Badge Challenges, the Prompt comes before my brainstorming.
Prompt:
Challenge Assignment 5: Authentic Assessment and Evidence for a Badge Ecosystem
Badge system design acknowledges that not all learners are the same, not all learning situations are alike, and not all ways of ascertaining learning accomplishments and skills attainment are the same. Badges offer learners cum job seekers not just more flexibility in how they learn but also in how they prove that they have the skills/competencies represented by a badge.
Assessors have a multi-partite role in the badge ecosystem. They must devise strategies that, as much as possible, push assessment activities into the world where actual performance will occur. They must validate both performance and performer. They must provide a robust set of metadata for each assessment that communicates that validation in total.
In a healthy badge ecosystem, learners demonstrate their competencies in authentic learning environments, capture evidence of their achievements, and have valid assessment to back up the earned badge. Just as badges open the field for innovative learning providers, they simultaneously stimulate rethinking how learning opportunities are provided and assessed. Based on the badge system you described in the prior challenge, describe the learning and assessment frameworks that are needed.
- How can learners’ needs best be addressed?
- Will traditional learning contexts and methodologies suffice, or can/should they be reworked? Are entirely new methodologies needed? Will new/different staff be required?
- How do competencies map to the learning activities and assessments?
- What types of evidence and assessment are valued and/or required by employers?
- Do the methodologies support the validation frameworks the learning provider needs?
- Write one or more “after badges” user stories depicting the value of the badge ecosystem for learner personas. What opportunities do badges provide for your personas? What challenges must be overcome in order to optimize the value of badges? Who do they share badges with? What goals do they have for using their badges?
BRAINSTORMING for this week:
It was interesting to see Kyle Bowen at the weekly live session this week in #OpenBadgesMOOC. I have met him in person at Campus Technology conferences and I have seen some of what Purdue is doing, which is pretty nifty! After one of his presentations I created an account on Open Passport to mess around with badge creation. This was a pretty easy system to create badges - if you know what badges you want to create that is! Just like ePortfolios, the technology is not (generally) the issue, but rather the underlying objectives and learning that you are attempting to assess.
For this week I think that we're back to familiar waters, with fewer assumptions than previous weeks since some decisions in the last couple of weeks were above my "paygrade." I do think that since I am thinking about Badges in relation to MOOCs it is a little easier. Applying badge considerations to MOOCs is easier, as far as I am concerned, because, in theory, you aren't tied down by the institutional baggage of how assessment has been done for the last decade or century :-)
When thinking of learners, and learner's needs, it's best to do some preliminary needs analysis, however this is hard even when thinking about "traditional" higher education courses, let alone a MOOC where you potentially have learners from all over the world with many, and competing, interests. That said, when designing the MOOC it's best to think about the target demographic for your MOOC and then make sure that those people who sign up are the learners for which the MOOC is designed. I do believe that in the weeks leading up to the MOOC, when learners sign up, they can potentially fill out a survey that allows the MOOC facilitators to keep certain things in mind as the MOOC is facilitated, and as additional or supplemental material is discovered, evaluated, and rolled into the MOOC. This should help address the needs of specific learners who are signed up for the MOOC. The big thing to think about is whether or not facilitators and designers of the MOOC keep addressing the needs to non-participating students in the MOOC, in other words, students who signed up, and might be reading along, but are not visibly participating in some way, shape, or form.
As far as staffing and methodologies are concerned, I would say that staff is definitely needed. A MOOC isn't a one-man show, but rather it's a team effort to design, develop, and implement the MOOC. In my initial thinking, I am thinking of designing and developing the MOOC on my own, simply because it's probably something that is going to be part of a dissertation. That said, when implementation-time comes along, I would like to recruit some ESL instructors from my institution, maybe on a volunteer basis, or through some sort of grant support to help pay them a small stipend, to help facilitate the course with me and to provide resources for course material evaluation, learner outreach, and in the end, of the trial group - some sort of evaluation support.
As far as methodologies go, as with most situations where instruction changes from one medium to another, there will be some change in methodology because there isn't a 1:1 correspondence between existing face to face language learning courses and the MOOC format. I suppose you could shoe-horn the face to face approach into a MOOC, but it just won't be successful. The commonality between the MOOC approach to language teaching and existing face to face teaching is that I plan on using a Communicative Language Teaching (CLT) framework for designing, developing and implementing this #ESLMOOC. While CLT can be the underlying method in both approaches, the tools users, and specific methodological approaches will vary between the two mediums. Thus, I foresee that in a MOOC environment we will be using a variety of Web 2.0 applications like Blogs, Twitter, YouTube/Vimeo videos that learners can use to process and engage with the material, but also to use in the assessment of the learner's acquired or improved skills as the MOOC progresses. The methodology chosen to design and run this MOOC should be supportive of the frameworks for validating learning.
As far as competencies and their mappings to learning activities and assessments go, that's a subject to be determined - mostly because the competencies themselves need to be determined. Right now, the CEFR has broad levels of competence. In order to make those broad levels more meaningful, but also as a way to map badges for those broad competencies, supporting competencies need devising. At the very least there are three supporting competencies, including hearing, speaking and writing. Those in turn would need some supporting competencies of their own. All of those would have considerations for activities and assessments. As far as assessments go, I don't plan on having many assessments. There will be two levels of assessment for #ESLMOOC. The first level would be basic assessment where the assessment is relatively easy and it doesn't need many assessors, or lots of assessor hours, to complete. This is probably where badges fit in. The second level of assessments is a more in-depth assessment of learners, perhaps in a smaller cohort, that will serve as part of the data gathering process for the PhD. If #ESLMOOC runs again, and if the second stage isn't as time consuming as I am envisioning it now, then those will be "badge-ified" as well.
When thinking about employers, going back to the original use case of this #ESLMOOC to prepare instructors of higher education in non-North American institution to teach their subjects in English, I would say that there are two broad elements of assessment that they need. There may be more, but those are subject to some sort of needs analysis. The two broad elements I can see at the moment are:
Finally, in terms of thinking of some after badges stories let's revisit some of the personas and their colleagues.
We have Professor Tomas. He is teaching at the University of Milan. Since he is going to be required to soon be teaching in English for his graduate courses, he decides to take part in #ESLMOOC to hone his skills in the English Language. Also, since he was willing to be a guinea pig for the researchers, he was assessed more than other learners in the MOOC. At the end of the MOOC he "graduated" the course with a collections of competency based badges, including one that gave him credentials as someone who could communicate orally at the C2 level (CEFR framework). With this set of badges he is able to pass the requirements at his own institution since his institution signed onto #ESLMOOC as a sponsor and was able to vet some of the outcomes of the MOOC. Professor Tomas' only issue is that he also moonlights at other institutions, and since they weren't part of the conceptualization process of #ESLMOOC, and they are no familiar with badges, they are not sure what to make of his accomplishments. Still, he has published his badges from #ESLMOOC on a variety of spaces, including his LinkedIn account, his Academia.edu account, and on his professional website via an embed code that he got from his backpack. For him, the overall goal for this particular set of badges is to certify his English skills. Using digital badges he can take care of the immediate needs of his home institution to demonstrate competency in English, but he can also advertise his skills via badges, as a way to also get noticed for any part-time work in institutions abroad, including the US.
That's all for badges for this week. Thoughts? One more week to go in #OpenBadgesMOOC
For this week I think that we're back to familiar waters, with fewer assumptions than previous weeks since some decisions in the last couple of weeks were above my "paygrade." I do think that since I am thinking about Badges in relation to MOOCs it is a little easier. Applying badge considerations to MOOCs is easier, as far as I am concerned, because, in theory, you aren't tied down by the institutional baggage of how assessment has been done for the last decade or century :-)
When thinking of learners, and learner's needs, it's best to do some preliminary needs analysis, however this is hard even when thinking about "traditional" higher education courses, let alone a MOOC where you potentially have learners from all over the world with many, and competing, interests. That said, when designing the MOOC it's best to think about the target demographic for your MOOC and then make sure that those people who sign up are the learners for which the MOOC is designed. I do believe that in the weeks leading up to the MOOC, when learners sign up, they can potentially fill out a survey that allows the MOOC facilitators to keep certain things in mind as the MOOC is facilitated, and as additional or supplemental material is discovered, evaluated, and rolled into the MOOC. This should help address the needs of specific learners who are signed up for the MOOC. The big thing to think about is whether or not facilitators and designers of the MOOC keep addressing the needs to non-participating students in the MOOC, in other words, students who signed up, and might be reading along, but are not visibly participating in some way, shape, or form.
As far as staffing and methodologies are concerned, I would say that staff is definitely needed. A MOOC isn't a one-man show, but rather it's a team effort to design, develop, and implement the MOOC. In my initial thinking, I am thinking of designing and developing the MOOC on my own, simply because it's probably something that is going to be part of a dissertation. That said, when implementation-time comes along, I would like to recruit some ESL instructors from my institution, maybe on a volunteer basis, or through some sort of grant support to help pay them a small stipend, to help facilitate the course with me and to provide resources for course material evaluation, learner outreach, and in the end, of the trial group - some sort of evaluation support.
As far as methodologies go, as with most situations where instruction changes from one medium to another, there will be some change in methodology because there isn't a 1:1 correspondence between existing face to face language learning courses and the MOOC format. I suppose you could shoe-horn the face to face approach into a MOOC, but it just won't be successful. The commonality between the MOOC approach to language teaching and existing face to face teaching is that I plan on using a Communicative Language Teaching (CLT) framework for designing, developing and implementing this #ESLMOOC. While CLT can be the underlying method in both approaches, the tools users, and specific methodological approaches will vary between the two mediums. Thus, I foresee that in a MOOC environment we will be using a variety of Web 2.0 applications like Blogs, Twitter, YouTube/Vimeo videos that learners can use to process and engage with the material, but also to use in the assessment of the learner's acquired or improved skills as the MOOC progresses. The methodology chosen to design and run this MOOC should be supportive of the frameworks for validating learning.
As far as competencies and their mappings to learning activities and assessments go, that's a subject to be determined - mostly because the competencies themselves need to be determined. Right now, the CEFR has broad levels of competence. In order to make those broad levels more meaningful, but also as a way to map badges for those broad competencies, supporting competencies need devising. At the very least there are three supporting competencies, including hearing, speaking and writing. Those in turn would need some supporting competencies of their own. All of those would have considerations for activities and assessments. As far as assessments go, I don't plan on having many assessments. There will be two levels of assessment for #ESLMOOC. The first level would be basic assessment where the assessment is relatively easy and it doesn't need many assessors, or lots of assessor hours, to complete. This is probably where badges fit in. The second level of assessments is a more in-depth assessment of learners, perhaps in a smaller cohort, that will serve as part of the data gathering process for the PhD. If #ESLMOOC runs again, and if the second stage isn't as time consuming as I am envisioning it now, then those will be "badge-ified" as well.
When thinking about employers, going back to the original use case of this #ESLMOOC to prepare instructors of higher education in non-North American institution to teach their subjects in English, I would say that there are two broad elements of assessment that they need. There may be more, but those are subject to some sort of needs analysis. The two broad elements I can see at the moment are:
- Clarity of speaking (speaking with less of an accent, being more comprehensible)
- Clarity of written feedback to students
Finally, in terms of thinking of some after badges stories let's revisit some of the personas and their colleagues.
We have Professor Tomas. He is teaching at the University of Milan. Since he is going to be required to soon be teaching in English for his graduate courses, he decides to take part in #ESLMOOC to hone his skills in the English Language. Also, since he was willing to be a guinea pig for the researchers, he was assessed more than other learners in the MOOC. At the end of the MOOC he "graduated" the course with a collections of competency based badges, including one that gave him credentials as someone who could communicate orally at the C2 level (CEFR framework). With this set of badges he is able to pass the requirements at his own institution since his institution signed onto #ESLMOOC as a sponsor and was able to vet some of the outcomes of the MOOC. Professor Tomas' only issue is that he also moonlights at other institutions, and since they weren't part of the conceptualization process of #ESLMOOC, and they are no familiar with badges, they are not sure what to make of his accomplishments. Still, he has published his badges from #ESLMOOC on a variety of spaces, including his LinkedIn account, his Academia.edu account, and on his professional website via an embed code that he got from his backpack. For him, the overall goal for this particular set of badges is to certify his English skills. Using digital badges he can take care of the immediate needs of his home institution to demonstrate competency in English, but he can also advertise his skills via badges, as a way to also get noticed for any part-time work in institutions abroad, including the US.
That's all for badges for this week. Thoughts? One more week to go in #OpenBadgesMOOC
Comments