Evaluating and monitoring the reaction to your game after launch is nothing new. Indeed, companies like Zynga or Popcap have made analytics and close examination of user behaviours a fundamental part of their business plan, using them to work out how to make their games as addictive as possible (and how to milk their users for money in the most efficient way). I’m sure anyone launching a game these days is at least checking user numbers using some sort of analytics software, but perhaps not to the same degree.
Not that I’m suggesting everyone should be Zynga (definitely not), but there is something that makers of educational games can learn from this approach: that evaluation of your game should be a key part of your business model. In fact, for anyone who makes a game intended to have a educational impact or to effect behavioural change, this is even more important. These sorts of games aren’t just about making money; they are trying to do even more than that, and it’s simply not good enough to assume that they are having the intended effect without testing this assumption.
I can’t count the number of times I’ve heard about a new learning game that makes this mistake, and every time my heart sinks. You may have started with clear learning objectives and conscientiously built a really great game around them, but that doesn’t necessarily mean that your message or learning is getting across. All kinds of things can get in the way of this happening: simple misunderstanding of the in-game copy, different interpretations of the intended message, cultural differences, the context of the platform and the way it gets used in practice (in the classroom, for example).
Testing a game
Testing with the target audience during development is vital to iron out some of these issues at an early stage and should continue right through the process. However, evaluation after release is also hugely important. Now your game is not just being trialled by your select group, it is in the wild, open to a much bigger potential audience and you have much less control over how it’s being used and perceived. You could be surprised by the actual audience reaction to your game.
Case study: an unexpected response
We found exactly this at Wellcome Collection when we evaluated High Tea, a game about the 19th Century Opium Wars between Britain and China. You can read the whole evaluation online, but two key findings demonstrate that players’ perceptions vary and might not be what you expect. In the game, you play as a British trader smuggling opium into China, very much against the wishes of the Chinese authorities. It puts you in a position where you are not just complicit, but in fact actively taking part in illegal and controversial behaviour. We expected people to be shocked that this is what the British had been up to, and many were, but about 10% of players felt more positively towards the British Empire after playing. On further examination, it turned out that by putting people in the position of the British trader, they had come to empathise with them. They felt that they were just trying to make a living and they could kind of understand that, even if they didn’t agree with it.
An unexpected reaction to the High Tea game by Wellcome Collection
Pretty deep for a game in which you have no real character of your own and interact through a map-based interface. Another surprise was that interpretation of the game’s message varied to a considerable degree. We were thinking about it in historical terms, and expected it to spark debate about the behaviour of the British Empire. A tiny minority thought we were glorifying the actions of the British, which we expected, but others thought it was about the economics of the situation, for example, and focussed on that.
Methods of evaluation
The good news is that summative (post launch) evaluation of your game doesn’t have to be difficult. For High Tea and other games at Wellcome Collection, we looked at detailed analytics that tracked different aspects of user behaviour, we put up a survey (with a prize draw incentive), followed up the survey with user interviews and a focus group, and analysed the commentary found on blogs, game comments, youtube and elsewhere. We found it to be a really fascinating process, and one we learned a lot from. There are other evaluation methodologies out there that you could also look at where suitable, such as undirected observation of users.
Our game had informal learning aims, and we mostly hoped that people would learn a little and be interested enough to give it a bit of thought afterward. It’s perhaps more difficult if your game is intended to change behaviour, or improve student grades. A self-reporting survey might not be sufficient evidence for this, although it’s a start, and the evaluation might need to be more long-term and more targeted.
The Adventure Begins from SuperBetter on Vimeo.
For example, I was impressed to see that Jane McGonigal’s Superbetter is testing its quite grand claims about being able to increase mental resilience in players by running clinical trails (results not out yet), which is pretty hardcore evaluation. This is going to be expensive and time-consuming, of course, but that appears to have been the plan from the start, so was presumably factored into planning and budgets. It would be great to see everyone else making games with purpose building in some time and money for evaluation, even if it doesn’t go quite as far as full-on randomised controlled trials.
After all, don’t you want to know what’s really going on if your game doesn’t do as well as expected? Or even why it might be doing well, so you can build on that? It’s likely to be too late to bolt an evaluation on if you don’t have a plan in place from the beginning (plus you’ll find yourself scrabbling around for the funds if you haven’t allocated them upfront).
Share your findings
For those of you who are already evaluating your games in this way, great work! However, I also feel that if you’re evaluating your game and keeping the results to yourself, that isn’t quite good enough either. Educational games makers are a community, and there are lots of challenges for us to face that are better faced together. I strongly believe that sharing what we learn as we make these games will benefit all of us, and we shouldn’t think of it as helping the “competition”.
So test and evaluate, please, it will be worth it, but don’t just leave it there: share it.
Have you done some evaluation work you’d like to share? Get in touch with us at edugameshub.org, we’d love to publish it!
Comments are closed.