Badge MOOC Challenge 4: Accreditation and Validation Frameworks for a Badge Ecosystem
 |
| Value Map Badge |
In any case, it's the end of Week 4 on the MOOC (2 more weeks to go) and this week we are talking about validation. It's interesting. One of the things that comes to mind as I progress through these is that the initial "levels" were a little easier to articulate, at least for my #ESLMOOC project, but as the weeks progress it's getting harder since I don't have all of the information. This also draws a parallel to Kirkpatrick's Level's of Evaluation where Levels 1 and 2 are easier to measure, at least in the short term, but Levels 3 and 4 (and if you look at Philip's 5th level) it gets harder.
In any case, we trot on. As with weeks before, the prompt comes first, and the brainstorming follows. Any assumptions will be footnoted (I wish blogger had some good footnoting tools)
Prompt:
Challenge Assignment 4: Accreditation and Validation Frameworks for a Badge Ecosystem
At the next level of complexity, we add an additional component:
- Validation
Learning providers have traditionally relied on academic accreditation and reputation as validation of the value of their targeted learning outcomes. Badge systems open the field for non-traditional learning providers as well as innovative learning methodologies in traditional institutions. With this opening up comes the need for clear, transparent validation of learning providers and their methodologies; without it, the value of their badges is questionable. Based on the badge system you described in the prior challenge, describe the validation framework that’s needed.
- Will traditional accreditation or other existing frameworks suffice?
- Are there relevant standards bodies that can provide review and validation?
- What types of reputation frameworks are already in place, and do they apply to the badge system?
- What peer organizational and institutional networks can be leveraged for peer review, evaluation, and endorsement?
- Will new research or evidence be needed, for example evaluation of employment outcomes (e.g, how many job seekers get jobs)?
- Write one or more “after badges” user stories depicting the value of the badge ecosystem for learning provider personas.
BRAINSTORMING for this week:
The live session this past Monday was pretty interesting, especially since there was a presenter from NEASC, who happens to accredit the school I work for. Of course, that being said, since the #ESLMOOC is going to be more of a pet project (something that hopefully will lead to a dissertation - fingers crossed), NEASC won't have a ton to do with this type of accreditation.
Now, thinking of this week, I honestly can't say that existing "accreditation" will suffice for the #ESLMOOC. What we have, in terms of "evaluation" for English Language Learners is standardized testing by the TOEFL by way of ETS, the CPE by way of Cambridge University ESOL, the MET by way of CaMLA, and finally the IELTS test. At least these are the four big ones that people seem to know about. From a framework perspective, we have the the CEFR developed by the council of Europe, and national tests tend to map onto this framework. That said, the framework, while a good start, seems quite broad to me. I wouldn't be able to say with a high degree of certainty that two individuals, both at the C1 level (penultimate level) have the same exact skill set with the language. I am sure that they can adapt, but they won't necessarily be a perfect fit from the start.
Knowing this, I think that the council of Europe and the ACTFL, on the broad level, as well as national standards bodies for language would be able to come up with validation criteria for specific badges if tasked with this undertaking - the question is: are the on-board with badges? The other thing that comes to mind is perhaps a concern about over-regulation of badges. For example, in a Higher Education classroom, if a colleague wants to institute badges for both demonstrable knowledge (artefact based) and demonstrated behavior (no visible artefacts from the learner to be digitized and available to the accreditor), how much latitude does this person have from their department, college, university or accreditor to determine what sort of behavioral badges are appropriate for his classroom? I guess there is a Goldilocks Zone here, in badging, as well.
Now, as far as reputation frameworks go, I am actually not really sure what already exists. I suppose, from people I know, the reputation is the fact that they have passed x-exam with y-grade. For instance, last summer when I was talking to a number of friends of friends, or friends of family and they told me with pride that they had passed the "Lower" exam, or had passed their "Proficiency" exam. This was usually tied to discussions about work, and working abroad, so for them this was a credential. When I was asked if I had passed any of these exams, I was a bit perplexed, because even though I grew up in Greece, I also spent the last 20 years in the US, and have completed all of my tertiary education here. So, it would seem to me that these exams have a built-in reputation system because it is something that people can compare apples-to-apples to; whereas when they were asking me about my exams, since I had none (but I had alternate experiences), they did not know how to compare their achievements to mine.
Thinking about Peer Assessment and Endorsement, I would conceive this as three concentric circles. In hte inner most circle, I would say that ESL departments in campuses around the New England area (NEASC's territory) could get together to peer review each other. It seems that ESL, at least in my institution, is non-credit. The #ESLMOOC, similarly, would also be non-credit; after all, it will be open, online, and free as MOOCs are. The next level up is a circle at the national, or North American level. Perhaps ESL departments from around North America can get together and peer review these badging system so determine efficacy, validity and fidelity. Finally, the top-level circle would be standards bodies like the European Council and ACTFL looking at the broad strokes and reviewing this work. So, the outer levels of this circle are more broad, and the closer you get to the core, the more details you get.
As far as research goes, I think that some research will be needed, but I don't think, at this point in time, that employment is the right rubric to use. While the #ESLMOOC is mostly going to be targeted toward learners in higher education settings (maybe those who are looking toward switching the language of instruction to English), and there is some aspect of employability, I think that there are probably other areas that can be examined through research. One area that really comes to mind, for me, is tying into Learning and Learner Analytics when it comes to classroom performance. I think that as badges become more central (and assuming that there is no grade inflation with badges), it would be interesting to see how students with different sets of badges do in different classes. Right now courses have generic pre-requisites for upper level courses such as "COURSE 601, 605, 612" or the nebulous "permission of instructor." The idea of pre-requisites is that you have gained, supposedly, some knowledge in those previous courses that is a foundation block for this upper level course. Well, what about transfer credit students. Sometimes transfer students meet 80% of the course requirements so they get to waive that intro course, but are they missing a certain key skill or knowledge that they will need to succeed in an upper level course? With badges we can get more granular as to what students know, and can do, and therefore can better place them in courses.
With that, let me lead you into an "after" badges scenario. In this scenario we have two actors: Councilor Troy. She works at the University Advising Office at Big State University (BSU). She is one of the people responsible for advising students for course sign-up each semester. Based on student interests, courses completed, and University policies, Troy tries to help students sign up for the courses they need in order to get to their desired goals.
Right across from her is Jonas Quinn, who has recently transferred in from Regional Community College (RCC) after completing his Associates Degree in Computing Technology. Jonas now had taken some Sociology courses while he was completing his AS degree and wants to major in Sociology now that he is a Big State University. All of his general education requirements transferred in, so he is hopeful that he doesn't have to repeat any courses that he has already taken in order to pursue a degree in Sociology.
In years past, Troy's job was considerably harder, in that upper level courses, like SOC315 only had other SOC courses as pre-requisites, like SOC215. In turn, SOC201 had a pre-requisite of SOC101 and 115. Since Troy wasn't a member of the sociology department, it was at times hard to figure out what skills were needed to be successful in upper level courses. Now that badges had been implemented regionally, Jonas had come in with a variety of badges from the sociology courses he had taken. As it turns out, BSU and RCC use some common badges in their respective sociology departments.
It seems that Jonas has 9 out of 10 pre-requisite badges for his first BSU sociology course. Since Jonas doesn't have all 10 necessary to join the course, but is over-qualified for lower courses, she send him to Professor Smith, department chair of Sociology, to seek permission to join the course, and to also see what he (Jonas) can do to demonstrate knowledge or ability for that 10th badge.
** Addendum Section Based on Feedback Received **
1. How does the badge ecosystem accredit and validate the learning provider and/or assessor and it's program(s)?
The badge ecosystem for #ESLMOOC has been envisioned to go along the same lines as accreditation of higher education programs in the US. Since #ESLMOOC is most likely going to be offered with a University partnership, any sort of accreditation stemming from a peer reviewer entity such as NEASC will be used to accredit the #ESLMOOC. Also, #ESLMOOC badges should be interoperable with the CEFR that I have mentioned before. This combination of regional accreditation as well as the European Framework (which is known at a much broader level) will serve as a way to accredit the learning and/or assessing in this MOOC. That said, I don't think NEASC, or the Council of Europe have had to deal with badges yet, so foundations will have to be laid in order to get this Accreditor/Standards Body/Learning Organization partnership going :)
2. How can a learner/badge seeker determine the legitimacy of the provider, or the suitability of the provider's methodology to the learner's needs?
I have to say, this is quite a difficult thing to think about because even "accredited" programs (traditional courses and programs that is) seem to be peer reviewed through regional bodies that have some sort of endowment from the government. Just because a program is accredited in the country-x it doesn't mean that their degree is valid in country-y without some sort of additional work (or having to do the whole thing over again). Thus, I would say that the criteria for determining the legitimacy of the learning provider would be the same as determining the legitimacy of an accredited program in the US since #ESLMOOC will run through a regular college program. If the school is accredited, then the course is accredited to some extend. It would be great to have the council of Europe backing it too (at least as a test-case), but it is not highly likely.
As far as the methodology goes - the MOOC format is really something that still has a lot of unknowns, which is why it's also an interesting topic of study for a dissertation. I don't think badges can really tell you much about the methodology used, at least if we go for badges for achievement. So, I don't see a concrete link between badge and methodology. That said, since the MOOC format is so new, it would be really useful to provide some learning scaffolds and some up front information to let the learners know what they can expect in a MOOC.
3. In your ecosystem, could a well-structured reputation framework be a supplement or stand-in for peer review?
I think that Peer Review is important when learning a language. Some learners will pick up on things that others will not. Therefore there is this collective scaffolding that goes on, and learners help each other fill the gaps in their knowledge and capabilities. That said, peer review isn't the only thing that's important. I know in xMOOCs (Edx, Coursera, et al) Peer Review seems to be the main form of assessment (along with machine graded quizzes), but this type of peer review isn't taken seriously, so I wouldn't give badges just for doing peer review. I think that an instructor would have to review the peer review comments and then badges can be provided to peer reviewers. The better your reviews towards peer's work, the more you can "level up" in the badges you get.
4. How you think badges might result in grade inflation (which pre-supposes that badges exist along side grades, rather than in place of grades?
The #ESLMOOC won't be graded - unless of course the sponsoring organization wants to also give people the option for credit. That said, graded assignments are not going into the design of the course. Badges, however, are planned to be part of the design. My concern with grade inflation is the subjectivity of the assessor. In theory letter and numeric grades should be based on grading rubrics. Sometimes people are more generous than others in awarding a grade or marking the rubric than others. If we look at this from a qualitative research perspective, if we draw parallels between grading and coding qualitative data, I am essentially worried about interrater reliability - or in my case "intergrader reliability". Does this make sense? :)
** End of Addendum **
That's all for this week on badges. Tune in next week for more ;-) Your ideas for #ESLMOOC badges?
Now, thinking of this week, I honestly can't say that existing "accreditation" will suffice for the #ESLMOOC. What we have, in terms of "evaluation" for English Language Learners is standardized testing by the TOEFL by way of ETS, the CPE by way of Cambridge University ESOL, the MET by way of CaMLA, and finally the IELTS test. At least these are the four big ones that people seem to know about. From a framework perspective, we have the the CEFR developed by the council of Europe, and national tests tend to map onto this framework. That said, the framework, while a good start, seems quite broad to me. I wouldn't be able to say with a high degree of certainty that two individuals, both at the C1 level (penultimate level) have the same exact skill set with the language. I am sure that they can adapt, but they won't necessarily be a perfect fit from the start.
Knowing this, I think that the council of Europe and the ACTFL, on the broad level, as well as national standards bodies for language would be able to come up with validation criteria for specific badges if tasked with this undertaking - the question is: are the on-board with badges? The other thing that comes to mind is perhaps a concern about over-regulation of badges. For example, in a Higher Education classroom, if a colleague wants to institute badges for both demonstrable knowledge (artefact based) and demonstrated behavior (no visible artefacts from the learner to be digitized and available to the accreditor), how much latitude does this person have from their department, college, university or accreditor to determine what sort of behavioral badges are appropriate for his classroom? I guess there is a Goldilocks Zone here, in badging, as well.
Now, as far as reputation frameworks go, I am actually not really sure what already exists. I suppose, from people I know, the reputation is the fact that they have passed x-exam with y-grade. For instance, last summer when I was talking to a number of friends of friends, or friends of family and they told me with pride that they had passed the "Lower" exam, or had passed their "Proficiency" exam. This was usually tied to discussions about work, and working abroad, so for them this was a credential. When I was asked if I had passed any of these exams, I was a bit perplexed, because even though I grew up in Greece, I also spent the last 20 years in the US, and have completed all of my tertiary education here. So, it would seem to me that these exams have a built-in reputation system because it is something that people can compare apples-to-apples to; whereas when they were asking me about my exams, since I had none (but I had alternate experiences), they did not know how to compare their achievements to mine.
Thinking about Peer Assessment and Endorsement, I would conceive this as three concentric circles. In hte inner most circle, I would say that ESL departments in campuses around the New England area (NEASC's territory) could get together to peer review each other. It seems that ESL, at least in my institution, is non-credit. The #ESLMOOC, similarly, would also be non-credit; after all, it will be open, online, and free as MOOCs are. The next level up is a circle at the national, or North American level. Perhaps ESL departments from around North America can get together and peer review these badging system so determine efficacy, validity and fidelity. Finally, the top-level circle would be standards bodies like the European Council and ACTFL looking at the broad strokes and reviewing this work. So, the outer levels of this circle are more broad, and the closer you get to the core, the more details you get.
As far as research goes, I think that some research will be needed, but I don't think, at this point in time, that employment is the right rubric to use. While the #ESLMOOC is mostly going to be targeted toward learners in higher education settings (maybe those who are looking toward switching the language of instruction to English), and there is some aspect of employability, I think that there are probably other areas that can be examined through research. One area that really comes to mind, for me, is tying into Learning and Learner Analytics when it comes to classroom performance. I think that as badges become more central (and assuming that there is no grade inflation with badges), it would be interesting to see how students with different sets of badges do in different classes. Right now courses have generic pre-requisites for upper level courses such as "COURSE 601, 605, 612" or the nebulous "permission of instructor." The idea of pre-requisites is that you have gained, supposedly, some knowledge in those previous courses that is a foundation block for this upper level course. Well, what about transfer credit students. Sometimes transfer students meet 80% of the course requirements so they get to waive that intro course, but are they missing a certain key skill or knowledge that they will need to succeed in an upper level course? With badges we can get more granular as to what students know, and can do, and therefore can better place them in courses.
With that, let me lead you into an "after" badges scenario. In this scenario we have two actors: Councilor Troy. She works at the University Advising Office at Big State University (BSU). She is one of the people responsible for advising students for course sign-up each semester. Based on student interests, courses completed, and University policies, Troy tries to help students sign up for the courses they need in order to get to their desired goals.
Right across from her is Jonas Quinn, who has recently transferred in from Regional Community College (RCC) after completing his Associates Degree in Computing Technology. Jonas now had taken some Sociology courses while he was completing his AS degree and wants to major in Sociology now that he is a Big State University. All of his general education requirements transferred in, so he is hopeful that he doesn't have to repeat any courses that he has already taken in order to pursue a degree in Sociology.
In years past, Troy's job was considerably harder, in that upper level courses, like SOC315 only had other SOC courses as pre-requisites, like SOC215. In turn, SOC201 had a pre-requisite of SOC101 and 115. Since Troy wasn't a member of the sociology department, it was at times hard to figure out what skills were needed to be successful in upper level courses. Now that badges had been implemented regionally, Jonas had come in with a variety of badges from the sociology courses he had taken. As it turns out, BSU and RCC use some common badges in their respective sociology departments.
It seems that Jonas has 9 out of 10 pre-requisite badges for his first BSU sociology course. Since Jonas doesn't have all 10 necessary to join the course, but is over-qualified for lower courses, she send him to Professor Smith, department chair of Sociology, to seek permission to join the course, and to also see what he (Jonas) can do to demonstrate knowledge or ability for that 10th badge.
** Addendum Section Based on Feedback Received **
1. How does the badge ecosystem accredit and validate the learning provider and/or assessor and it's program(s)?
The badge ecosystem for #ESLMOOC has been envisioned to go along the same lines as accreditation of higher education programs in the US. Since #ESLMOOC is most likely going to be offered with a University partnership, any sort of accreditation stemming from a peer reviewer entity such as NEASC will be used to accredit the #ESLMOOC. Also, #ESLMOOC badges should be interoperable with the CEFR that I have mentioned before. This combination of regional accreditation as well as the European Framework (which is known at a much broader level) will serve as a way to accredit the learning and/or assessing in this MOOC. That said, I don't think NEASC, or the Council of Europe have had to deal with badges yet, so foundations will have to be laid in order to get this Accreditor/Standards Body/Learning Organization partnership going :)
2. How can a learner/badge seeker determine the legitimacy of the provider, or the suitability of the provider's methodology to the learner's needs?
I have to say, this is quite a difficult thing to think about because even "accredited" programs (traditional courses and programs that is) seem to be peer reviewed through regional bodies that have some sort of endowment from the government. Just because a program is accredited in the country-x it doesn't mean that their degree is valid in country-y without some sort of additional work (or having to do the whole thing over again). Thus, I would say that the criteria for determining the legitimacy of the learning provider would be the same as determining the legitimacy of an accredited program in the US since #ESLMOOC will run through a regular college program. If the school is accredited, then the course is accredited to some extend. It would be great to have the council of Europe backing it too (at least as a test-case), but it is not highly likely.
As far as the methodology goes - the MOOC format is really something that still has a lot of unknowns, which is why it's also an interesting topic of study for a dissertation. I don't think badges can really tell you much about the methodology used, at least if we go for badges for achievement. So, I don't see a concrete link between badge and methodology. That said, since the MOOC format is so new, it would be really useful to provide some learning scaffolds and some up front information to let the learners know what they can expect in a MOOC.
3. In your ecosystem, could a well-structured reputation framework be a supplement or stand-in for peer review?
I think that Peer Review is important when learning a language. Some learners will pick up on things that others will not. Therefore there is this collective scaffolding that goes on, and learners help each other fill the gaps in their knowledge and capabilities. That said, peer review isn't the only thing that's important. I know in xMOOCs (Edx, Coursera, et al) Peer Review seems to be the main form of assessment (along with machine graded quizzes), but this type of peer review isn't taken seriously, so I wouldn't give badges just for doing peer review. I think that an instructor would have to review the peer review comments and then badges can be provided to peer reviewers. The better your reviews towards peer's work, the more you can "level up" in the badges you get.
4. How you think badges might result in grade inflation (which pre-supposes that badges exist along side grades, rather than in place of grades?
The #ESLMOOC won't be graded - unless of course the sponsoring organization wants to also give people the option for credit. That said, graded assignments are not going into the design of the course. Badges, however, are planned to be part of the design. My concern with grade inflation is the subjectivity of the assessor. In theory letter and numeric grades should be based on grading rubrics. Sometimes people are more generous than others in awarding a grade or marking the rubric than others. If we look at this from a qualitative research perspective, if we draw parallels between grading and coding qualitative data, I am essentially worried about interrater reliability - or in my case "intergrader reliability". Does this make sense? :)
** End of Addendum **
That's all for this week on badges. Tune in next week for more ;-) Your ideas for #ESLMOOC badges?
Comments
Good luck.
All the best, David
Good luck, David