Assessment and serious games

 

Here's a quote from the serious games introduction in Vista:

Almost a decade ago, a Ministry of Education wanted to commission one of your instructors (David Vogt) to create a completely immersive online graduating exam for grade 12 Physics. Students would encounter virtual problems to solve to demonstrate their understanding of physical concepts, rather than just mathematical equations. It’s a compelling idea that wasn’t feasible then, but soon will be. In fact, the notion of a “final exam” could disappear in an environment like this if adaptive learning techniques were coupled with embedded assessment tools to provide a continuously formative educational experience.

 

That brings up some interesting points:

1. How would assessment occur in a serious game?
2. How would assessment differ from K-12 to post-secondary to corporate/government education?

 

We'd love to hear your thoughts - please answer by editing this page.

 

1. How would assessment occur in a serious game?

 

from Kyle:

The question of assessment makes drawing a distinction between serious games and subtitles for Chitrageet a little easier. Or perhaps that is an unfair comparison; Chitrageet was designed to be an almost incidental, learner-led educational experience; in that sense it differs from formal education. Designs in formal education mean the term is almost another way of saying "formal evaluation."

There is very little in the way of assessment for a program like Chitrageet. For a game like Civilization, however, where the object is to out-develop one's competitor civilizations, assessment is built into every level of the game. Even games like Warcraft have "replay" features which would enable learners and their instructors to review choices made and compare the success of different students' strategies. Assessment can be handled in a number of ways. Broadly speaking, it could be "this child ranks 8th in the class in terms of word-munchers scores" ... or in a less peer-competitive (and thus better learning environment), "this child has progressed from through the difficulty levels--or made significant personal score increases--at a satisfactory rate" ... or it could focus in on the details of a child's improvements to specific in-game competencies. We have standardized tests based on pooled achievement level data; we could do the same for students' mastery at Serious Games.

 

from Paul: Yes, I agree with Kyle. The types of assessment that I prefer are ones that monitor personal progress, and then present that info back to the student. It's not that I'm against healthy, positive competition, but sometimes the competitive element can overshadow the skills aquisition. I teach Grade 2, and there are always some students that just want to get to the end of the 'adventure' etc as soon as possible without fully consolidating their skills. It's also helpful when the software has different difficulty levels. 'Early finishers' can then repeat the whole adventure at a higher level, thus really putting their new-found skills into practice. An example of this type of software that we use is The Crystal Rainforest, an adventure that introduces the LOGO programming language.

 

from Ada:

I'm questioning whether skills mastery is the most important element we would like to assess in serious games played in the K-12 setting. To me, I guess it isn't as significant whether a student has progressed in decision making and thus advanced levels in a game, but it is more beneficial in knowing what kinds of transformed thinking and reflections the student has gone through during the process of playing the game. So I would rather the student explore the different strategies and results in playing the same game and reflecting on that through discussion, written response, and/or an assignment as opposed to playing to win. In corporate/government education settings however, I guess reflections may be less valued than the end results to prove the employee has made the 'right' decisions, so in those cases skills mastery would be an important assessment element.

 

 

From Micah

 

Ada, I think that you really hit the nail on the head with your thinking regarding assessment in regard to serious games. I could not agree more that it is very important to explore the transformation of thinking and reflection that has occurred. A game such as SimCity can demonstrate to the students that events such as earthquakes have catastrophic effects on communities. It is possible that students would then be able to visualize the effects in a different manner than if they were to simply hear a lesson about earthquakes. Possibly after playing the game, the teacher could ask some pointed questions to test for how thinking has changed.

 

from Richard:

Yes, I agree with your latter points, Ada. The idea of reflecting and discussing is obviously a sound idea for learning in all grades. Would the written response provide the evidence of learning, of seeing the connections? At the K-8 level, one has in mind this business of reporting to parents/caregivers. One could easily envision a student who's a wonderchild at the game and yet, for a variety of reasons, be unable to describe in words the nature of what happened. Practically, I wonder, whether kids (adults too?) who are asked to do a written reflection would balk at the suggestion: "that's not fun," and the fun aspect can't be overemphasized with the game. Writing, discussing has the connotation of work. Or maybe I'm simply thinking of what I would be like. Internally generated feedback of the sort Kyle suggested has promise in this regard.

 

from Shari:

In an earlier post on a dif. Wiki page, I commented on the idea that gaming students make greater connections / learn more by being involved in a serious game -- I pointed out that some learners are visual in that they prefer to write things down or see things in written form, to retain the knowledge. I would like to go further and add to Ada's comment above and link it to last week's discussion on Collaborative Learning Environments. The rise of the use of CE in employee workplaces gives credence to the fact that today, our 'recorded knowledge' gets out of date so fast that the only way we can avoid our knowledge or expertise from going 'stale' is to be present and engaged in the discussions taking place before each of us, in this collaborative environment. What we are now all doing is synthesizing shared information to 'create new knowledge'. I think this post here is proving the point!

 

From Tracey:

I would definitely welcome some form of computer generated assessment tools for Grade 6 students (or any intermediate aged student)....there are a variety of learning styles to assess and I think the students would be more relaxed in a serious game evaluation tool in comparison to a pencil paper assessment. I will marvel at the day that a "final exam" for high school students would be in the form of a serious game - I have doubts but am all for it. My son, at this very moment is studying for a unit quiz for Principals Math 11.....I KNOW he would prefer a more realistic version of what he knows with a serious game format - more real life problem solving as opposed to the pages and pages of formulas and equations that need to be worked out on paper. This topic reminds me of the Intelligent Tutoring Systems concept whereby the programs have been criticized because the students would be able to figure out how to manipulate their scores based on clicking a certain button over and over. I'm just glad I don't have to design the game for assessment - very tricky in terms of hard assessment data?? Just some thoughts.

 

From Richard:

Great thoughts on assessment, Tracey. It seems so often, at least in the school system, to come back to the evidence of learning, the assessment. Parents have all been through the education system and all have their own ideas of what a "test" is. Despite the I think awesome potential of serious games to really engage students in subject-specific matter and more--lots of integration, synthesis going on--hard assessment data will come from isolated pencil and paper tests, at least for the next decade or three.

 

From Julie:

I am with Tracey. Wouldn't it be great to be accessed on what level you are at in the game to what mark you get in class. The higher the level, the greated the accomplishment and therefore the better grade.

 

From David:

This is a great discussion - I'm enjoying it! I thought I'd add a personal note to anchor the history and viability of assessment with serious games. Where in the World is Carmen Sandiego?, mentioned a few times here, was originally released in 1985. This was enormously successful (commercially and pedagogically) on the earliest (pre-Internet) personal computer platforms. It was fascinating for me this week, during meetings in Ottawa, to meet with a Canadian serious games company that is being contracted by the current owners of the Carmen Sandiego IP to produce a new online 3D version of Carmen. The fascinating part is that the timing of this redevelopment is geared to the fond memories of the original users, who are now parents. A true 'second generation' serious game.

 

From my own experience, I believe my Brainium company was the first to launch serious games onto the Internet (game-based curriculum for middle school science) in 1995. We built the first prototypes for a browser called Mosaic, pre-dating Netscape. We did lots of embedded assessment and adaptive learning R&D in the late 1990s, and could demonstrate better motivation, test results, etc. But without dedicated longitudinal studies it is very difficult to rule out wildcard factors such as "novelty" being the prevalent to learning success. Therefore reliable assessment methodologies embedded in serious games require solid coupling with traditional academic research programs. But there are a number of problems in getting there.

 

Some of the original and best research in the application of games for learning was done at UBC by Maria Klawe (then Dean of Science) within a program partnered with Electronic Arts called EGEMS (Electronic Games for Education in Math and Science) in the late 1990's. It's interesting that EA stated fervently that "nobody is ever going to make money from educational games", which they continue to believe despite their purchase of properties such as the Sim-City suite of products. In 2003 Canada also established a national centre of excellence in this area based out of SFU called SAGE for Learning (Simulations and Advanced Game Environments for Learning), which I helped to create.

 

Being an academic, it is interesting to me that it took so long for serious games research to become a legitimate "field" - it became acceptable and an academic darling only about four years ago. This can be generalized much more broadly: only very rarely can an EVA or entrepreneur rely on any form of academic research to validate the learning values or potentials of an emerging technology. In fact, even when such research begins relatively concurrently with such an emergence, it can't keep pace - the study and publication cycle is such that technology has almost inevitably moved dramatically onwards before the traditional research is complete. That's why most ventures with a sincere R&D focus (they aren't common) almost always rely on rapid prototyping and applied research methodologies.

 

So getting back to assessment, we always need "humans in the loop" - dedicated educators who are able to independently and concurrently verify (most likely through active sampling) that whatever assessments are being generated have useful meaning. In my experience the learning technologies market is flooded with snake oil salespeople who market (often very sincerely) the unprovable learning credentials and assessment capabilities of their products and services. And, of course, this was also true long before digital learning technologies came along...

 

From Adam:

I'm interested in what you say David about traditional academic research not being able to keep pace with developments in technology, so that there's a reliance on rapid prototyping and applied research methodologies for those who develop serious games. Would this field be well-suited to use of design-based research? I'm still getting to grips with what DBR is, but when comparing it to 'traditional' academic research as applied to e-learning technologies some go as far as saying that much of this research is socially irresponsible in comparing one form of media with another or with earlier educational interventions to see which is 'better'. I like the way that DBR starts with a significant educational problem that needs solving (others have already identified 'educational' games that are cool enough but don't really seem to answer any particular need), and from then on there are a number of cycles in which the best available theory or design principles initially available are used to guide development of a solution. By iteratively developing/evaluating/reflecting, the educational problem, solution and design principles are progressively refined and shared for the benefit of the wider community of practice. Maybe this sharing of design principles is what would make DBR unappealing for most in the serious games industry, except those who are intrapreneurs or looking to develop open source projects?

 

From Phil:

 

There are different approaches to assessment and serious gaming

 

You have learning outcomes such as what the student is supposed to learn or derive from the game on an individual basis. Assessment there compares the students skills to the desired outcomes. A lot of instructors use rubrics or metrics to allow them to better assess individual components of a skill. I don't see that process being any different when applied to the assessment of serious games. I think the fairest approach there would be individual improvement. Did the student demonstrate improvement within the context of the game?

 

I don't think comparing one kids performance to another is very productive. That "military style" assessment was in fashion at the start of the last century. Even computer adaptive testing which is still used seems to be a little old fashioned and I am surprised that tests like the GMAT still use it. Comparing the levels at which a student can reach doesn't really test learning outomes ... it is more of a comparison metric. I think games are just a different spin on testing improvement against learning outcomes. Did student X improve their score an acceptable level? That is far superior to ... did student X achieve a higher score that student Y? What does that realy prove in an educational environment?

 

You also have tools to extract certain outcomes in order to progress students through the material. If you require students to learn certain things you can embed those things in the game so that students have to learn them or find them in order to progress. You could bury in "bonus points" or activities for students who want to go above and beyond ... but the core part of the game should be able to be completed by everyone ... it should not become a devisive tool. Why not have areas where teams can help one another ... and add grop work into the assessment? Especially in a social game that would be a key assessment metric.

 

Lastly I would point out that there are millions of diffrent types of games. There are puzzles or mysteries which test problem solving, team based games whch test teamwork, strategy games which test resource allocation and tactics, action games that test reflexes and all out competition games where the winner takes all. Each one of those games can be assesed differently and used for different purposes depending on the circumstance. The choice of game is just as important as the choice of assessment metric.

 

 

Assessment Discussion Summary

From the discussion, here is a brief summary of ways of assessing serious games:

• Which student finishes the task(s) first or the quickest.
• Personal progress – Evaluate how well the student acquired the skills, rather than how quickly.
• Determine how playing the game has transformed the student’s thinking and if the student engaged in a reflective process.
• Embedded assessment tools – Some skepticism exists around how accurate these tools could be, and the consensus was that human input was required in the assessment process as well.

There was not very much discussion about how using and assessing serious games would differ between K-12, post-secondary and corporate education. From the corporate perspective, serious games assessment would be formative; the end results would hopefully be recognizable in an improvement in the employee’s perspective. Or, in the case of medical or military employees, the employee should be made to demonstrate or discuss the skill learned from the game.

But, we would have liked to hear more about K-12 and post-secondary education assessment could work since in those environments grades are assigned.