作者：Ben Lewis-Evans

【这是Ben Lewis-Evans关于游戏用户研究方法系列文章的第二部分，第一部分主要阐述焦点小组、启发法、问卷调查等内容，本文将讨论访谈、观察法（其中包括“自言自语”法和实境调查法）、游戏参数和生物统计学等内容。】

访谈

与问卷调查一样，它的主要目标也是搜集主观数据。但面对面的访谈意味着你可以在互动过程中搜集数据，如果执行得当，就可以获得丰富的数据。但这种方法也非常费时，很难分析和量化你最终所得到的数据。

你在访谈中所搜集到的数据质量，很大程度上取决于你自身的采访水平，以下是一些访谈技巧：

采访前

这里有一些基本内容要注意。首先，要选择一个合适的环境，一般来说最好是干预因素较少并且令人舒适的环境。

另外，最好一次只采访一个人，这样可以避免受访者的看法受到他人观点的影响，但有时候也可以一次采访2个人，因为这样有助于受访者相互启发对方——这一点对讨论协作型或多人模式的游戏尤其适用。

很显然，在受访者自己所处的游戏环境中访问他们会更困难，但如果非如此不可的话，就要尽是减少周围的干扰因素（例如，有礼貌地询问他们是否可以把门关上等等）。

可以在访谈过程中录音，但并不一定要录相，最重要的是记录下访谈内容。因为你过后未必记得所有的内容，就算你很擅长作笔记，也最好能够利用录音资料回顾访谈内容。最好是持续记录，不要让中断的动作让受访者觉得你评估的是他本人而非游戏。

你得告知受访者你们会录音记录，并获得他们的同意才能执行这种操作。另外得记得向他们说明，你希望他们谈论哪些内容，并解释采访目的并非评价他们的游戏水准，而是观察他们的游戏体验，以及游戏运行表现。

你事先要拟一个采访提纲，例如写下你想提的问题，并对其进行排序。另外也要准备提出补充问题，但要注意这些问题不可带有挑衅意味。受访者可能对你的游戏存在非议，但你的表现要潇洒一点。

撰写这些问题时可以参照上篇文章提到的调查问卷的多数设计原则。换句话说，这些问题要简洁扼要，不可具有明显的诱导性，不可累赘，并且只有一个意图。另外要回避一些只能答“是”或“不是”的问题，这类问题比较合适出现在调查问卷上。

采访中

开头先说些简单的事情，这和问卷调查一样，主要是为了暖场和热身，让受访者自然过渡到接受访谈的状态。切入正题后要记住，一次只能提一个问题，确保受访者结果回答后再进入下一题。你应该掌握移情的倾听技巧，用点头或者回答“嗯”之类的动作回应对方，以示你确实在听他们说话。

如果他们离题了，也没有什么关系，因为这是访谈过程中常见的情况。但如果他们扯得太远了（例如大谈自己优秀的游戏设计想法），或者在某个问题上纠缠过久，那就得想法缓慢地让他们重回正轨。

要尽量保持中立态度。可以给受访者一点建议，以示你之前听过这些说法。不要表现出你很不耐烦的样子，不要对他们所说的内容表现出过度反应。这听起来似乎有违移情倾听技巧的做法，但是如果操作得当，就可以完美地表现出你在认真倾听而非暗示你在判断他们所说的话。

另外，在访谈结束前不妨给一点时间，让对方补充一些你没问而他们想说的事情。

采访后

整理访谈记录。注意这项工作会花掉相当一段时间。如果你执行多次访谈，要注意找到受访者所谈的共同话题。访谈的一大好处（这也是观察游戏玩法的好处）就是它可以为开发团队（甚至是管理层）带来一些富有启发的内容。

优势

*主观印象的数据来源很丰富

*可以追问其他问题

弊端

*不易量化数据

*费时

*不够客观

观察研究

观察研究（有时也称为行为观察或现场观察）就是让一些人玩你的游戏，然后进行录相以便之后进行观察，或者直接在现场观察。这可能是传统用户研究测试法中最有价值的方法，你自己亲自看用户玩游戏无疑更有说服力。因为此时你可以观察他们如何在游戏中取得进展，如何应对挑战，他们在哪受阻或失去兴趣。

你还可以注意他们的肢体动作和表情，以便探测他们的情绪。但要注意，只能记录下你所看到的内容，不要妄自揣测他们的心理活动。观察者或组织者应该保持中立和不干预态度，要是想知道玩家在哪跌倒或犯错，若非不得已的情况，就不要轻易施予援手。

如果你有一个合适的实验场地，就可以展开大规模的观察。但最好是进行一对一的观察，这样可以让玩家免受彼此干扰，而观察者也不会因测试玩家太多而分散注意力。

执行观察研究时还有一些事项需要注意，以下是我总结的一些步骤：

步骤1：设计好观察流程

要明确在哪测试，是否需要拍摄记录，要测试游戏的哪些环节。最理想的情况是全面测试，但由于现代游戏具有非线性和突发性特点，要做到这一点实非易事。另外，应该挑选游戏最具代表性的内容进行测试，并确保其核心机制足够有趣。

执行观察测试时最好是创建和使用测试脚本。测试脚本就像是电影脚本，美味糕点的食谱一样，它会安排好测试流程，包括事件的发生顺序，事件内容，甚至是供测试者使用的脚本对话。

脚本内容要尽量详细。要考虑到用户进入房间后的情形，例如是否应该向他们提供咖啡？他们要坐在哪里？是否有告诉他们卫生间和紧急出口在哪？在测试过程中你是否随时都和他们交谈，或者在特定节点才交谈？你怎样呈现游戏环节？

这听起来有点过于严谨了，也有人认为这种方法不适合展开调查研究，因为它会吓坏玩家，让他们坐立难安，觉得自己才是检测对象。这种说法不无道理，但如果你想获得更有效的数据，最好还是事先安排好测试流程。优秀的调查人员可以做到这一点，而且不会让操作过程显得太生硬。

最后要记住，如果一开始是愉快地欢迎测试用户，不停地同他们聊些无关话题，后面又发现时间透支，只能匆匆接待下一个测试者，这也并不是很好的做法。因为你后面会发现，用户评价的并非你的游戏，而是你本人。

良好的测试脚本应该包括接待参与者、管理人员，介绍流程（例如测试步骤，时间等）。记住，此时的态度可以随意和放松一点，让大家知道接受评估的是游戏而非玩家。

脚本应该清楚描述玩家需执行的任务，例如“从这一关开始玩到下一关”，重要的是明确起点和终点。因为这可以让你准确把握这一过程中所发生的特定问题。

你还应该明确自己希望看到哪类行为（可以是游戏中或游戏外的行为），这样你才会清楚应该注意哪些地方。假如许多人观察到的行为都和你所写下的观察结果一致，那么就再好不过了——双跳看起来到底应该是怎样？跳跃失败又如何呢？这种大家都认同的行为描述列表就是所谓的行为详述，它有助于减少多人一起观察并讨论结果时所产生的困扰。

或以借助一些软件解决这个问题（游戏邦注：例如Morae或Observer等软件），这类工具可以记录屏幕和参与者画面，为测试计时，还可以让你在观察过程中（或者之后查看记录视频时）进行快速的预编码。如果你无法录相或使用优质软件工具，那就要制作一个良好的行为详述表以写下观察结果（最好坐在用户看不到你的地方），这样也有助于生成有用的观察结果。

一切准备就绪之后，你应该彻底检验一下自己的安排，确保各个环节流畅运转，并拥有明确的指令。这也是出色的调查人员所应获取的观察经验。

在展开测试之前，你得招募测试者。在第一部分中我们已经讨论过这个话题，但这里还是要强调一点，最重要的是引进具有代表性的用户。如果你的游戏锁定的是5-8岁的女孩，那么找隔壁宿舍的男生来参与测试就没有什么意义了。而如果你要找孩子来测试游戏，那就有另一堆问题需要考虑（游戏邦注：例如得到家长同意，安排群组测试和单人测试，以及沟通交流等问题）。

步骤2：执行观察流程

展开观察后要做些什么？当然就是观察玩家行为。要查看这些参与测试者的反应，并记录下他们在游戏中，在游戏外的行为，以及他们所说的话。可以借助上文提到的软件完成这项工具，记录玩家的游戏过程有助于你过后重新回顾。另外，录相资料还可以作为说服他人哪些地方需要修改的证据。

要注意，不但要记录玩家的行为，还要记录他们某个操作发生的时间和情形，这种情形对诠释数据来说也很重要。

在观察过程中，要注意只能记录自己的所见所闻，例如要写“玩家看到过场动画时发笑了”而不是“玩家很开心”，因为后者的描述只是你自己对玩家心理状态的臆断。

这里看起来又有于严谨了一点，但从长期来看，你的臆断越少就越好，并且准确地描述玩家行为也有助于你同他人探讨这些问题。

除了你所看到的行为之外，你还应该记录玩家在游戏中的表现。例如，玩家通过特定步骤所费的时间，游戏中取得的分数，破坏数量和使用的弹药数量等等。如果你并非通过游戏衡量系统追踪这些变量，那就真的很有必要详细记录内容。

要注意在玩家体验游戏过程中，不要传递你的反馈意见。当你看到玩家卡在某个环节而却没找到“显而易见”的解决方法时，这确实会让你纠结，但这也正是你需要看到的结果。这意味玩家开始测试时，你就不可以再跟他们聊天，并且要礼貌地请求他们不要跟你谈天。可以让他们带一个耳机，这样可以制造一种独立空间之感，也有助于他们投入心理状态。

但这并不代表你不可以插入提问，或者真的出现问题时仍然坐视不理。无论如何，最好要记录下你感到哪里不对劲的阶段，测试结束后再问玩家他们当时的感受。

步骤3：结束测试

这时要感谢玩家参与测试，也许还可以借此让他们填写一份最终的调查问卷，甚至是对其进行简短的采访。这个简短采访可以让你借机针对自己观察到的有趣情况提出问题，以便让自己释疑解惑，得到一些有用的反馈。再次强调，一定要用非对抗性或中立的态度进行采访，不要咄咄逼人地指出他们在第三关时应该如使用喷气机脱离那个深洞。

最后，假如你进行多次测试，建议你腾出15-30分钟时间重新整理房间，在下一拨用户进来前理清头绪。长时间的观察可能让你精疲力尽，所以安排片刻休息可以让自己放松一下，也可以让你重新审视自己的观察结果。

实境调查

另一种观察研究法就是实境调查。这实际上就是一种现场堪察工作，你要在玩家的自然状态中进行观察。也就是说，你可以让他们在自己正常的游戏环境中观察他们体验游戏的习惯，并从中获得在你自己的测试过程中无法得到的结果。

在这种情况下，我之前提到的测试脚本就没有多大用处了，因为此时的关键是观察玩家如何在自己的日常生活中体验你的游戏。不过提前准备自己来访时想说的内容，自己将如何进行观察，这一点仍然很有必要。

实境调查更有助于提升现有游戏质量，你也可以通过这种方法观察玩家如何在现实生活中体验与自己的产品类似的游戏。在这种情形中，测试游戏原型或者正处于开发过程中的游戏，就没有多大意义了，因为玩家无法对这种内容形成一种游戏习惯。不过，在这种过程中观察自己的游戏如何被玩家使用（或者滥用）仍是一个不错的主意。

观察方法的优劣

观察研究法最大的优势之一就是人可以真正看到新玩家如何体验你的游戏。在我看来，这就是极有价值的信息。这种方法也有助于让你了解意想不到的情况。

不过，观察研究很显然非常耗时，如果你有闻必录的话那就更是如此。另外，成为优秀的观察者也需要投入一定精力。正如我之前所言，优秀的观察者必须保持中立态度，尽量不要干扰玩家体验游戏。另外，他们还必须清楚自己要了解哪些内容，何时补充提问等。

先进行实践，并列出你希望看到的典型行为，这有助于培养你的观察能力。然后你就会知道在展开测试前需要了解哪些内容。记住你首要观察的是他们在游戏中的行为，所以要无设想他们将执行的动作。例如，观察他们是否在同一个地方跳跃多次？或者一直墥墙？

此外，不要用自己的想法来判断你所看到的情形。不要臆断玩家心理状态，只要写下自己看到的东西即可，如果真想知道他们在想什么，过后再问他们。

优势

*可提供客观数据。你可以看到玩家真正的行为，而不是他们所说的操作。

*玩家的行为可能出乎你的意料。

*此时记录设备所搜集的信息，也许还有其他用途。

弊端

*费时，尤其是你需要详细记录内容的时候。

*要掌握足够的经验才能成为出色的观察者。

*进入测试环境可能让玩家感到不安，或者他们得知自己被监控，可能会表现出一些完全不自然的行为。

“自言自语”法（think out loud)

这是观察法的一个扩展，也就是你可以采用上述提到的观察法，然后在玩家体验游戏时询问他们当时的感受。然后你一边观察他们的行为，一边记录下这些内容。这种方法可以增加有关玩家心理状态的信息，让你了解他们对游戏的假设性看法。注意，此时你可能会发现他们所说的情况是错误的，或者并非你喜欢听到的答案，但一定要克制这种纠正他们的冲动。

这种方法可以让你了解意想不到的情况，并且这些是其他方法所无法提供的信息，至少可以让你更为了解用户在心理上与游戏的交互情况。但是，它也有一个极大的弊端，那就是让人们一边玩一边说下自己的感受，这种做法非常不自然。这可能会改变玩家实际上的游戏操作行为，影响他们的游戏乐趣。

与此同时，你可能还会发现，有些人一开口就没完了，而有些人却几乎难以启齿，结果你就会发现这种方法更多地是反映人们之间的差别而非游戏本身的问题。

优势

*允许你搜集客观（行为）数据的同时，获知玩家内心的想法和感受。

*可能获得意想不到的结果。

弊端

*耗时。

*不自然。

*人们所说的仍是主观信息。

玩法参数

还有一个观察法补充内容就是玩法参数。它实际上就是游戏本身搜集的统计资料，可以让你明白玩家所执行的操作。例如，《光晕》中的统计系统、《极品飞车》中的日志系统。

这些玩法参数可以提供有关游戏玩法的可靠、主观而统计性的事实。要在游戏中植入这些数据搜集系统，如果采用这种做法，你就会发现自己手头上掌握了大量数据来源。想知道在地图上哪种武器最常让玩家毙命？玩家使用哪些升级道具？他们最常丧命的地方？这些都可以从玩法参数中找到答案。优秀的参数触发器可以成为一种十分有效的辅助观察手段，可以减轻你记录玩家表现细节的担子。

（参数的力量：上图左是玩家在《光晕：致远星》中以某种武器杀敌时所站的地点，右图则是该武器的杀敌数。你只具备一点基本的FPS游戏知识，也可以大概推测出这是哪类武器，并了解这个关卡的提升空间。）

不过搜集玩法参数也非常费时，并且你需要一定规模的玩家参与测试才能有效搜集数据。此外，这种参数缺乏与情感，以及玩家个人情况有关的主观信息——它只是一堆数据而已。

另外，如果你无法仔细分析自己搜集到的数据，你就会面临数据超载的情况。例如，在《光晕：致远星》中载入任何一个热图，打开所有武器造成的死亡数量，将其与所有武器的杀敌数进行比较。这可能会让你了解到在该玩家区域的杀敌数与死亡数的分布情况，但却无助于分析玩家体验的细节。

优势

*可以远程和分离地收集客观数据。

*可以查看趋势，并为其他观察法提供数据支持。

*可以在不影响玩家的情况下持续搜集数据。

弊端

*费时。

*缺乏主观反馈。

*需要大量样本才能有效搜集数据。

*同时也会让你获得过多数据。

生物计量法

我之前已经探讨过这个话题，所以不在本文赘述详细内容。生物计量法是指通过记录玩家肢体信号（例如通过EEG获取脑波，通过眼球追踪技术查看玩家寻找的内容）从而获知玩家内部的生理状态。它和自言自语法以及玩法参数一样，可以做为一种观察测试的补充。

与观察和玩法参数一样，生物计量法的妙用在于它可以提供一些关于玩家状况的主观数据。它可以在不影响和打断玩家的情况下，持续搜集与玩家情感和兴奋感有关的数据。

但这种搜集方法目前仍十分具有干扰性，因为你得将用户与测量仪器连接起来，并且这些设备价格十分昂贵，而且你还得学习这些设备的使用方法，并且学会分析它们所搜集的数据。

此外，这种方法也可能让你搜集到错误的数据信号，并且行业对于这种方法所搜集到的数据可靠性尚未达到一致意见。例如，你的游戏中有些环节可能导致玩家心率提高，这可能意味着游戏令人兴奋，也可能意味着它让人感到恐怖或者不快，或者玩家实际上觉得很无聊，已经神游到其他令人兴奋的事情上了，甚至也有可能是因为玩家当时只是做了个深呼吸（这也可能提高人的心率）。

以上问题意味着你需要将生物计量数据与其他数据来源（游戏邦注：包括观察、采访、调查问卷）相结合，这样才能弄懂你所搜集到的生理信号究竟代表什么情况。这也导致有些人怀疑这个方法的可用性，他们认为这非但不能提供有效的补充信息，还可能产生许多额外工作量。

不过生物计量法的倡导者认为，生物计量数据的优势在于，它可以让你获知玩家自己不知道的情感和生理反应。最后要注意，它也只是一种补充工具，要不要使用这种方法要取决于你的判断。

优势

*提供客观的量化数据。

*可在不干扰玩家的情况下持续搜集数据。

弊端

*具有干扰性。

*耗时耗财。

*因特异性、推理和有效性等问题而难以诠译数据信息。

总结

我这两篇文章只能算是粗略的引言，所述内容也并不全面。如果你有兴趣深入研究这个话题，我推荐阅读Valve成员Mike Ambinder的有关著作，以及Mike同Bungie成员John Hopson在这一问题上的讨论内容。

最后要说的是，上述方法只是辅助开发游戏的手段，它们也许能搜集到大量有趣的数据，但这些数据并不能主导设计。要用你的创意想法引导游戏开发，要让游戏用户研究搜集到的数据帮助开发游戏，而非约束你的创造力。（本文为游戏邦/gamerboom.com编译，拒绝任何不保留版权的转载，如需转载请联系：游戏邦）

Finding Out What They Think: A Rough Primer To User Research, Part 2

by Ben Lewis-Evans

[The following is the second of two articles by college professor and researcher Ben Lewis-Evans on games user research methodology (see Part 1, which covered focus groups, heuristics, and questionnaires, as well as giving a grounding in the topic of user research in general. In this article, Lewis-Evans covers interviews, observational methods (including think out loud and contextual inquiry), game metrics, and biometrics.]

Interviews

Much like a questionnaire — a topic covered in the last installment — an interview is for collecting subjective data. However, the face-to-face nature of an interview means that you can be more interactive in your data collection, which if done correctly, can lead to very rich data. However, it is also obviously quite time-consuming, and it is harder to analyze and quantify the data you get at the end.

The quality of what you get out of an interview will also depend greatly on your own skill as an interviewer, so here are some tips.

The Setup

There are some basic things here. First, choose a good setting. Generally speaking, it should be comfortable, with as few distractions as possible.

Also, plan to generally only interview one (or perhaps two) people at a time. One is usually best, so as to avoid one person’s opinion dominating, but sometimes two people can have a nice dynamic and prompt each other — particularly if discussing a cooperative or multiplayer game.

Obviously, if you are going to be interviewing people in their own play environment, this gets harder, but still do you best to make sure extra distractions are not around (e.g. ask politely if the door can be closed, etc.).

Also it is useful to set up a way to record the interview; this doesn’t have to be video, but it is important that you at least record what is said. You will not remember everything, and even if you take great notes it is a good idea to have the recording to refer back to. This is also preferable to making notes constantly, as this tends to distract the person you are interviewing and can create a feeling that they are being assessed personally — rather than the game.

You will have to alert the person you are interviewing to the recording and get their permission. You will also have to explain to them what you are interested in asking them about, and also as with all of these methods make it clear that this interview is not about evaluating their performance, but to look at their experience of, and the performance of, the game.

You should also make an interview script; i.e. write down the questions you want to ask, and the order in which you want to ask them. However you should also be prepared to ask follow-up questions, although take care and don’t be confrontational when you do. It may be your game they are badmouthing, but you need to play it cool.

When coming up with these questions many of the same rules creating questionnaires apply (see part 1 of this series). In other words, try to be clear and precise with your questions, and check they are not leading, loaded, and only have one meaning. Also take care to avoid questions that can be answered by a simple yes or no; if you are after that kind of information use a questionnaire instead.

During the Interview

Start off with some easy stuff. Much like with a questionnaire, this warms them up, and will get them in the mood to talk to you. Once you get going also take care to only ask one question at a time, and usually only move on when you are sure the person you are talking to is finished. You should practice listening empathically and encourage responding by doing things like nodding your head and saying “mm-hmm” to acknowledge you are listening to them.

It is fine if they go off on a tangent; that is part of interviewing. However, if they go too far off (say, talking too much about a great design idea they have) or spend too long, be prepared to gently nudge them back into talking about things you are interested in.

Also, try to be as neutral as possible. One bit of advice given to interviewers is to act as if you have heard it all before. Not as in you are bored, but as in that you are not shocked or overly reactive to what they are saying. This may sound like it is against the idea of empathic listening, but if done correctly, it is perfectly possible to signal that you are listening carefully without also implying you are judging.

Finally, it can be useful to finish up the interview with a short period of time where your interviewees can add anything that you may not have asked about.

After the Interview

Make a transcript. Be aware this will take some time! Then, if you conducted multiple interviews, look for common themes and threads in what people are saying. One good thing about interviews (which is also a good thing about gameplay observation and think out loud methods) is that they can generate great sound bites or quotes that can be quite convincing when given to others on the development (or even management) team.

Pros

•Rich data source for subjective impressions

•Can ask follow up questions

Cons

•Not easy to quantify

•Time-consuming

•Not objective

Observational Studies

Observational studies (also sometimes called Behavioral Observation or Ethnographic Observation) are where you get some people to play your game and either record them doing so to watch later, or observe them directly. This technique is probably the most valuable of the traditional user research testing methods because nothing beats actually watching someone play your game. You can see how they progress, how they deal with challenges, and where they appear to become stuck or unmotivated.

You can also attempt to pay attention to their body posture and expression in order to try and gauge their emotions. Be careful, however, and only write down what you actually see — don’t try and infer any inner thought processes. Also the observer or facilitator should be as neutral and hands-off as possible. You want to see where people get stuck or make mistakes, so don’t help them unless you really, really have to.

Observation can be done in large groups if you have the correct lab space. But it is probably better done on a one-on-one basis so that players do not distract each other and the observer is also not distracted.

Again, there are several things to remember when setting up an observational study, and I will try and go through some of the steps below:

Step 1 – Designing the Observation Session

Work out where you will be testing, if you will be recording the session or not, and what part or parts of the game you will be testing. It would be ideal if you could test exhaustively, however with modern non-linear and emergent games this is obviously difficult. Still, you should plan to test at the very least large representative chunks of your game, and certainly make sure that the core mechanics are fun.

One useful thing to do when running an observational type test is to create and use test scripts. Test scripts, like a movie script, or the recipe for a delicious cake, lay out how the test will go and typically include the order of events, what the events are, and even scripted dialog for the tester to use.

Be specific with this. What happens when the user comes into the room? Do you offer them coffee? Where do they sit? Do you tell them where the bathroom and emergency exits are? Do you talk to them at any time during the test, or only at set points? What build do you load up? In what order to you present the game portions you are looking at?

This may sound overly scientific, and in fact some argue that it is a bad way to run research, as it scares the player and gets them in a mindset that makes them feel tested. This is a valid point. At the same time, however, if you want the best data, it is good to have at least some consistency, and certainly you need to have some plan for the session. Good researchers can do this and still not come across as a robot.

Ultimately, it is no good if at the start of the day you welcome a user happily, chat about their ride over to the lab as you offer them a coffee, and then later in the day you just rush the next user into the room and try and get things done as fast as possible. Then you may find that it is not your game they are rating, but you.

A good test script should include welcoming the participants, administrative stuff, and some introduction to the procedure (what will happen, how long it will take, etc.) Remember, again, at this point it is good to be casual and relaxed to make it clear that the game is being evaluated, not the player.

The script should also clearly describe the tasks they are going to be doing during the observation session, this can be as simple as “start the level and then get to the next level” but it is important that they have clear defined start and end points. This is because you want to know exactly when in this process certain issues or behaviors occurred.

You should also think about what kind of behaviors (in the game, and perhaps outside of it, depending on the game) that you expect to see, just so you know what to look out for. It can be helpful if many people are observing to agree on a clear sentence (or two) that outlines the behaviors you are observing — so what exactly does a double jump look like? What about a failed jump? A list of such agreed-upon descriptions of behaviors is called an ethogram, and it helps to cut down on confusion when discussing what was observed with others.

There are quite a few software packages out there aimed at helping out at this point (for example Morae or Observer, although there are many others) which can handle the recording of the screen and the participant, the timing of the test and also allow you to carry out quick predefined coding as you observe (or when you go back later to watch recorded video). This said, if you can’t afford to record and use fancy software then having a nice well-defined ethogram and somewhere to write down observations (sit somewhere the user can’t see you — even if that is sitting behind them) can still produce useful results.

Once you have everything ready to go, you should test out the setup you have to make sure everything runs smoothly and the instructions are clear. This is also important as the only way to become a good observer is get experience observing.

Finally before you can start you need to look into recruiting people to test. I covered this in part 1, but just to be clear, the most important thing at this point is that you want representative users. It’s no good testing it with the dudebro in the dorm next-door if your target market is young girls five to eight years of age. And if you are going to be testing with children you are opening up whole other box of issues (parental permission, testing in groups vs. alone, communication, and so on).

Step 2 – Running the Observation Sessions

What do you do once the session is running? Well, observe, of course. Watch the participants and take notes on what they do in the game, in real life, and what they say. As mentioned above, there is software that can help you do this, and certainly recording the play session will help you later if you can go back and review it. Plus the recordings can be great material to show other folks who might need convincing that something needs to change.

Be careful to note not only what the player is doing, but also the time at which they do it, and the situation in which it occurs. This context is important for interpreting the data.

Also when observing, write down only what you see and hear. So, write: “the player is laughing at the cutscene and smiling” not “the player is happy” as this latter description makes assumptions about the user’s internal states.

Again this may seems overly scientific, but in the long run the fewer assumptions you make, the better, and it helps when communicating with others what you found if you stick to clear descriptions of behavior.

In addition to the behaviors you see, you should also record down performance measures from in the game. In other words, you should note things like the time taken to clear a particular stage, the in-game score, the amount of damage taken and ammo used, and so on. This is especially important if you are not tracking these variables through a gameplay metrics system.

Finally, I should note that during this time it is good if the player is isolated from others (including you) and can concentrate on the game. This can be as advanced as an observation room at Microsoft or just you simply sitting in the same room out of the player’s line of sight.

The important thing is that you let the player play without feedback from you. This can be frustrating when you see someone getting stuck in a place where the solution is “obvious”, but this is exactly the type of thing you want to see. This means that you shouldn’t really talk to the player once the session has begun, and you should politely ask them not to talk to you either. Having them wear headphones can help as it adds to the sense of a self-contained environment, and can help put the player in the right frame of mind.

This doesn’t mean you can’t step in and ask a question, or help if things are going really wrong. However, generally speaking, it is better to note down periods where you are not really sure what was going on and then ask the player about them later, after the test is done.

Step 3 – End of the Session

This part is pretty straight forward. Thank the player for taking part, and perhaps take the opportunity to give them a final questionnaire or even conduct a quick interview with them.

This interview/debrief gives you the chance to ask about any interesting observations you made and try and get some feedback on what was going on. Again, be careful to do this in a non-confrontational and neutral manner; don’t jump down their throat about how they should have just used the jetpack to get over that hole they kept falling in during level 3.

Finally, if you are running multiple sessions, it is usually advisable to give yourself about 15 to 30 minutes at this point to reset the room, and gather your thoughts before the next player/group of players. Observing for long periods can be very draining, so these little breaks can help recharge your batteries a little, and also allow you to think about and perhaps review the observations you have made so far.

An Aside: Contextual Inquiry

Another type of observation study is a contextual inquiry. This is essentially fieldwork where you play David Attenborough and go out and observe your players in their natural habitat. The idea being that you can observe them in their normal play setting going about their normal routines when using your game, and therefore get insights that may not be available in your own test situation.

In this case the test scripts I mentioned above are somewhat less useful, as the point here is to see how players use your game in their normal routine. However being prepared in terms of what you will say when you show up and how you will carry out the observation is still important.

Contextual inquiries also tend to be more useful for improving existing games, or observing similar games to yours to learn about how people play them in the real world. As such, testing prototypes or games in development in this fashion can be less useful, as players haven’t had an opportunity to develop a routine with them. Still, it is certainly nice to see how your game will be used (and abused) in the wild.

Pros and Cons of Observational Methods

That is all I will cover as far as observations go (with the exception of think out loud, which I will discuss in a bit). The biggest advantage of observation studies is that you really get to see what new players do when playing your game. This is invaluable and makes observation studies the best methodology, in my opinion. This method can also give you unexpected insight, and reveal issues that you may have never seen without it.

However, observations are obviously quite time-consuming, especially if you are recording everything. Also, it can take quite some work to be a good observer. As I already said, a good observer has to be as neutral and non-intrusive as possible. But also they have to know what to look for, and when to ask follow-up questions.

What typically helps is first practicing, and making a list of typical behaviors you would expect to see. Then you know what to look for before you go in to a real session. Remember you want to primarily be looking for their behavior in the game, so think about the types of things they will be doing. So for example do you notice them jumping repeatedly in one place? Or running into walls?

Also, avoid coloring what you see with your own expectations. Don’t try to infer internal states; just write down what you observe, and then if you want to know what was going on in their head, ask them later.

Pros

•Provides you with objective data. You are seeing what players actually do, not what they say they would do/have done

•Players may surprise you

•When recording equipment is used, the material collected can be used for other purposes

Cons

•Time-consuming, especially if you record and go back through everything carefully

•It takes experience to be a good observer

•Going into a test environment can make people nervous or at least aware that they are being monitored, and therefore they may not produce completely natural behavior

Think Out Loud Protocol

An expansion of the observation method, this is where you carry out observations as detailed above, but ask the players to talk about what they are thinking as they go along. You then record this while observing what they are doing. This adds information about the internal states of the player, and gives you an idea of what assumptions they are making based on the game. Again, you may hear things that are wrong, or that you don’t like, but resist the urge to correct the player.

This method can lead to unexpected insights that wouldn’t be gained otherwise, or at the very least gives you a little more insight into how people are mentally interacting with your game. However, it does have one big downside, and that is that it is quite unnatural to talk about what you are doing and feeling as you do it. This might change how people will actually play the game and affect how much or little fun they are having.

Related to this, you may find that some people talk non-stop, whereas others hardly say a word, a finding that may reflect more the differences in people than in your game.

Pros

•Allows for the collection of objective (behavioral) data while also getting access what players are thinking and feeling
•Can result in unexpected insights
Cons

•Time-consuming

•Unnatural

•What people say is still subjective

Gameplay Metrics

Similar to, and a great addition to, observational methods are gameplay metrics. This is basically is statistics generated by the game itself. which tell you about what the players did. For example think of the statistics systems that Bungie and 343 Studios have for Halo, or the Autolog system in Need for Speed.

Such gameplay metrics give you hard, objective, statistical facts about how the game plays. The collection of this data has to be built into the game, but once it is, you have a very large data source at your fingertips. Want to know what weapons take down players most on a map? What power-ups they use? Where they tend to die? Gameplay metrics make that possible. Good metric triggers and hooks can also really compliment observations as they take some of the load off you in terms of recording the in-game details about the player’s performance.

The power of metrics: on the left is where people are standing when they make kills with a weapon and on the right is deaths by this weapon in Halo Reach. With just a basic knowledge of FPS games, you can still probably work exactly what kind of weapon this is and where the elevated and the open spaces are in the level.

However, metrics can be time-consuming to collect and you need a good number of players to really take advantage of them. Plus metrics lack subjective information about emotion and what is going on with the players — it is just a mass of data.

Also, you can end up with data overload if you don’t analyze what you collect carefully. For example, load up any heat map in Halo Reach and turn on deaths by all weapons, and compare it to kills by all weapons. The result will generally give you a little information in that the kills and deaths seem pretty well spread out over the player area, but is not particularly useful for working out finer details of the player experience.

Pros

•Objective data that can be collected remotely and discretely

•Can be used to see trends and provide data that supports other methods

•Allows for continuous data recording without interrupting the player

Cons

•Time-consuming

•No subjective feedback

•Needs large sample sizes to get meaningful data

•However, at the same time, you can get too much data

Biometrics

I will finish up by covering biometrics. I have already done a primer on this topic, so I am not going to go into too much detail here. Biometrics are basically when you record signals from the body, such as brain waves via EEG, or where people are looking via eye tracking, to try and give us a clue as to your player’s internal body states. In this way they are again like think out loud is, and like metrics can be: an add-on to observation-based testing.

Much like observation and gameplay metrics, biometric measures can be useful because they give you objective data on what is going on. They also allow for data on emotions and excitement to be collected continuously during play without having to stop and ask questions about how people are feeling.

However, the ways you can collect biometrics are currently quite invasive in that you have to wire people up, and they cost quite a bit of money and time to use, not to mention you have to be trained in how to use them — as well as how to analyze the data that you get.

Also they have problems in that you can get artifacts or fake signals in your data, and that there is no real solid agreement on what you are measuring actually means. For example, heart rate may go up in a certain part of your game, which could mean it is exciting, but it could also mean it is scary and unpleasant, or that your player is so bored they are now thinking about last night in bed with their partner, or even just that the person you are recording took a deep breath (which also increases your heart rate).

The above issues mean that you have to combine biometric data with other sources of data (observations, interviews, questionnaires) in order to work out what is going on with the physiological signals you are collecting. This has led some people to doubt its usefulness, arguing that it doesn’t add anything that you couldn’t already get but does add a bunch of complications and extra work (see an excellent discussion on this point, and on games user research in general, by Bungie’s John Hopson and Valve’s Mike Ambinder here).

Proponents of biometrics, on the other hand, suggest that the strength of biometric data is that it gives you access to emotions and reactions that the players may not know they are having, and therefore helps you look at other data sources in a way you haven’t before. Ultimately, however, it is just another tool and you will have to judge if it is appropriate for you or not.

Pros

•Gives objective quantifiable data

•Allows for continuous data recording without interrupting the player

Cons

•Invasive

•Costs a lot of time and money

•Problems with specificity, artifacts, inference and validity can make it difficult to interpret

Final Words

As I stated in the introduction these two articles are intended as rough primers, and are in no way comprehensive. I am sure that others will disagree with what I say, and perhaps others will be happy to add more or provide material of their own. However, I do hope though that this document can be of some use.

If you are interested in further reading on this topic then I can recommend checking out this excellent document from Mike Ambinder at Valve and watching this discussion between Mike and John Hopson from Bungie. As mentioned in the last article, you may also consider checking out the IGDA Special Interest Group for Games User Research on LinkedIn.

Furthermore a wiki-based games user research primer is in the works. This primer will arise out of a workshop at the CHI 2012 conference and is planned to be an evolving wiki based site that can provide the community with up to date information on game research methodologies. So watch that space.

Finally, please remember that the above methods are here you help you develop your games. The data they give can be rich and interesting, but it should not be domineering. In the end, it is your creative vision that guides the game; the data that can be collected via games user research is there to help you, NOT limit you.（source:gamasutra）