作者：Andre Dittrich

建立稳定性对于我们来说一直都是一个重要的议题，但是当游戏真正进入开发阶段时，游戏和工具的稳定性就变成了技术团队应该考虑的重要议题之一。因为在这时候将会有最多人参与进游戏开发环节，等待着游戏中潜在的漏洞或遗漏道具的出现，同时这也意味着需要投入更多资金加以修复。所以，保持游戏结构的稳定性是一项非常重要的任务。

遗憾的是，我现在并没有“能够解决所有问题”的处方。我想在此与你们分享我们曾经使用过的一些措施。

validate build stability(from javeriana.edu)

迭代次数规则

拥有一个稳定的游戏结构真的很重要，但是你也不能因此破坏了整个团队的迭代次数。游戏关卡设计者经常会提出一个小功能的改变请求，或者只是需要负责部门快速修改一个主要的漏洞等，并且要求在下一个问题出现前完成这个任务。而这时你也不想看到他花了几周的时间等待这些改变的完成吧。

如果你的编程团队同时拥有10或20名甚至更多工程师，而你却仍未采取任何措施去维持游戏结构的稳定，那么你的游戏将会面临至少一个的游戏漏洞，而因此阻碍游戏的发行。

主要问题便是，应对措施需要你挪出更多的日常费用，而这也成为造成缓慢迭代次数的关键原因。所以现在你能做的便是维持日常开支和游戏结构稳定性相平衡。

自动化结构系统

你需要持续集成服务器（CIS）！当“真正的”漏洞扰乱了你的游戏结构已经算很糟糕了，但是更糟糕的是一些小小的漏洞都能够轻易毁了你的游戏。你是否曾经在一天早上走进办公室却发现不能编译游戏？因为一些排印错误，或者还未收到相关文件，还是隐藏错误的合并信息？有多少人因为这样那样的错误白白浪费了一个上午的时间？很显然，这些漏洞都是可以避免的。

CIS的主要功能是让你在检查变化过程中持续保持引擎的正常使用，确保引擎和工具的正常编制。当然了，我们还有一些简单快捷的冒烟测试，能够帮助你明确什么时候能够打开引擎进行操作。

但是你能做的还有更多。白天，你需要随时关注着引擎结构的运行，让它们保持越快的速度，同时你还需要进行冒烟测试。晚上，我们能做的就更多了。我们通过运行自动化测试，以获得测试阶段和游戏阶段中的内存使用和性能状况的相关数据。这些数据主要用于支撑内部网站的图表描绘。这些图表将能够帮助你观察性能和储存状况的突然变化，以及平缓的开发过程。

适当核对一些评论，能让你在游戏遇到问题前适时阻止最坏情况的发生，或者，至少你可以最快并最有效地意识到问题的发生（而不需要技术援助或其他程序员浪费大量时间去寻找问题的根源）。

既然谈到了自动化测试，我们就来说说单元测试。我曾经体验过单元测试，尽管我不得不承认这充其量只是一种教我们如何去做的测试方法。我们将单元测试框架在游戏开发的早期整合进Unreal Engine中的Unreal Script和kismet功能中。我们之所以选择用单元测试编写AI编码，是因为这种测试能让我们自行编写而无需要依靠中间件代码做辅助。

而我们的主要错误便在于我们尝试了一些综合测试，并浪费了大量时间于这些测试过程中。有一段时间，我们甚至将“单元测试”贯穿于游戏的每一个功能中。当因为其系统的变化（而非测试编码的漏洞）导致测试失败后，我们便不得不花费更多时间于修补这些测试漏洞，并最后终止单元测试。

在我们的下一款游戏中，我想通过一些单元测试以测试我们编码中的重要部分。综合测试主要适用于那些已完成且不容易改变的功能，所以意味着你不得不将这种方法置于开发过程的后期。

同事审核

这可以说是改善游戏结构稳定性的最好方法了。它不仅能够帮助你真正改善游戏结构稳定性，同时也能够促进你们团队成员间以及与发行者之间的关系，达到一种三赢局面。

这个方法很简单：当有人想要核对改变时，他需要得到其他同事的审核才能继续进行。当然了，这种方法只有在很正规的执行过程中才有效。同时，审核人员还应该真正理解那个改变是什么，它是如何变化的，并且为什么会发生变化等等。

审核中不存在所谓的“愚蠢的问题”。如果你在审核过程中有任何不理解的问题都必须提出来。特别是针对那些高层或领导人而言，因为他们经常认为自己不应该低头去咨询一些愚蠢的问题。需要的话，你还可以向他人咨询相关意见。你也可以批评类型或一些细节问题。只要你认为额外或有利于改善的审核有帮助，那么就果断地执行这一方法把。它不仅能够帮助你判断改变是否有效，同时也能够让你与同事共享观点和相关知识。

所以，你在最后得到了什么？通过审核你只能发现一些显而易见的问题，并想出相应的解决方法。但是你甚少能够发现那些真正复杂的漏洞或者相关副作用。所以，那些未被发现的问题将会在后来的自动化系统，质量评价中被发现，更糟糕的情况是，它们会一直维持到用户发现为止。

因为人们总是会对一些通常看不到的东西特别在意，所以这个过程也提供给他们相互学习的机会。至少这两个人都知道相关改变的细节，因此有了一定的存在感，而不会轻易离开公司。

在你的团队中，队员们能够谈论你的工作，并确保这项工作能够在被发现漏洞之前得到完善（游戏邦注：如果你的审核者发现了那一段你认为无误的代码是错的，那么这个场面将非常尴尬）。团队成员能够交谈，他们具有共同的语言，他们理解每一位成员的优势和劣势。

在同事审核过程中需要记住的一些事：

耗时：确保每个人都知道这是一件花时间的工作，估算出所耗时间。

每次登记都应该接受审核：很多“简单”或“小型”的登记也会造成很多错误。

同事愿意做审核：如果没人有时间接受审核，那么这一工作看起来真的很烦人，所以你应该针对次问题想出相关解决方法。

阐明谁来做审核工作：你应该更加重视这一环节，如果你发现一个由登记引起的漏洞，那么你应该确保这时候有两个人能够帮助你解决这一问题。

登记评论

刚开始我并不清楚登记评论为何能够改善游戏结构的稳定性，因为一旦漏洞被登记在案，那么便不能改变了。好的登记评论能够让我们更容易找到问题所在。结构化格式评论更有效。想象一下你坐在电脑屏幕面前打算从100条登记评论中找出引起AI漏洞的根源，真的要抓狂了吧。最快最简单的方法便是，尽可能地选取那些有帮助的信息。我们正是采取这种方法找到很多“难以发现”的漏洞。

但是事实上，如果你能够合理使用登记评论这一方法，它确实能为你带来不少好处。你可以使用资源控制系统以获取相关区域每天登记人员的自动化邮件，以及这些登记人员的资料。这一系统不仅有利于获取登记评论，同时也能够让程序员，QA人员以及开发者了解相关登记信息。

结构测试

这是一项困难的任务，并且需要内部质量评价资源和一些额外工具作辅助。基本问题很简单：在登记改变之前，如果你不确定这个改变是否准确，那么就测试看看。

我想每个人都了解这种糟糕的感觉吧，即当你按照一种早期的编程方法做出改变，而这种编程同时也触及了很多其它编程内容（假如那个最初的编程人员已经去世了，而你将不得不在你的中间件中改变这一编程）。你并不知道这么做有何副作用，而且你也找不到任何自动化测试方法，而这时唯一的方法就是——测试。

最适合测试这项工作的莫过于QA人员了。所以，你需要创建一个游戏的局部结构（或者具有代表性的一部分），并将其发给QA团队以测试改变。在等待期间你也可以检验你的改变，或者继续寻找其它漏洞。

为了尽可能实现这一举措，你需要采取一些有力的辅助工具。我们正在使用Unreal Engine的结构工具。使用Unreal Frontend能帮助我们针对各种平台创造出更好的游戏结构。这个工具能够按照你的想法面向不同平台加工你的游戏。如果没有这个工具，我们便不得不使用中央处理器（支柱服务器）来维持我们的游戏结构。

QA人员通过使用简单的网页界面将游戏结构复制到他们的电脑上或XBOX（微软游戏机）上。所以你可以创建一个游戏结构，并将其复制到网络文件夹上，发电子邮件给QA人员说明是在哪里发现这一结构。但是当整个过程变得更简单时，人们会更青睐于这个方法，并且不会轻易放弃它。为了让整个过程更加顺畅，我们同样也建立了一系列严格工作流程。

但是，即使你完美地按照上述方法进行改善，也不能保证你的游戏完全没有漏洞，但是这么做却能让你花更多时间去检查并发现其它重要且有趣的漏洞，最重要的是，能够更好地完善你的游戏结构稳定性。（本文为游戏邦/gamerboom.com编译，如需转载请联系：游戏邦）

Opinion: How To Improve Build Stability

by Andre Dittrich

Build stability is always an important topic for us, but once a game production has entered the production phase in earnest, the stability of the game and the tools becomes one of the more important aspects for the tech team. The simple reason for this is that the number of people relying on this is highest at that point and any time these people have to wait for a bugfix or missing tools potentially means a lot of money wasted. So, keeping your build as stable as possible is important.

And now for the bad news: I do not have the “This Solves All Our Problems” recipe. I want to share some of the measures we have applied in our projects. If you have other measures you have taken to ensure build stability, please tell me. I am always interested in doing more.

Iteration Time Rules

Having a stable build is very important – yes, but you cannot ruin the iteration time for your team. There will always be that level designer that requests a small feature, a small change or simply needs a critical bug fix really fast (usually yesterday) to finish the mission for the next milestone. You do not want him to wait for a week for that change.

With 10, 20 or even more engineers working on your code base at the same time, the chance is high that there is always at least one that has added a bug that makes it impossible to release the next engine version to the team at least if you do not take some measures that help to keep the build stable.

The problem of course is that the measures you take cannot add so much overhead that they become a reason for slow iteration times. So, everything you do needs to strike a balance between overhead and improved build stability.

Automated Build Systems

CIS – continuous integration server: you need this! It is bad enough if “real” bugs trouble your build – it is far worse if simple bugs destroy it. Ever come into the office in the morning to find out that you cannot compile the game? A typo, a file that had not been checked in, a bad merge? How many people lost how much time during this one morning? This is totally avoidable.

The main function of our CIS is to continuously build the engine whenever somebody checks in a change. This makes sure that the engine and tools at least compile. Of course, we also run a few easy and fast smoke tests that also make sure that you can at least start the engine.

But you can do even more. During the day, the focus is on getting the engine build as fast as possible and running smoke tests. During the night, we can do a lot more. We run automated tests to get statistics for memory usage and performance in test levels and game levels. These statistics are made available as graphs on an internal website. These graphs are an enormous help to recognize and track down sudden jumps in either performance or memory as well as gradual development.

Together with good check in comments (see below), you can prevent this from breaking the game before it actually becomes a problem, or you at least recognize the problem very fast and efficient (without TAs or programmers spending time to find out why MissionXY is not running any more).

When I talk about automated tests, I guess I have to talk about unit tests as well. I have some experience with it, though I have to admit that most of it is about how not to do it. We integrated a unit testing framework into the Unreal Engine on the Unreal Script and Kismet level pretty early in the production process. We started to use it for the AI code mostly as this was mainly written by us and not relying too much on middleware code (except pathfinding).

The main mistake we made was that we ended up with actually doing integration tests and maintaining those takes a lot of time. For some time, we even made it part of the process to have “unit tests” for every feature we did. At some point we started spending more and more time on fixing the tests which were failing because of changes in other systems and not because of bugs in the tested code – we stopped doing it.

For our next projects, I want to do actual unit tests to test critical parts of our code. Integration tests are something that should be used for finished features that are not very likely to change a lot, and I guess that means you have to keep that for a later time in the production. If you have experience with successfully applying either, I would like to hear about your experiences.

Peer Reviews

This is one of the best tools in our belt to improve build stability. It not only gives you a substantial improvement in build stability, it also fosters communication within the team and distributes knowledge (win – win – win).

The idea is pretty simple: Whenever someone wants to check in a change, he needs to get this change reviewed by one of his colleagues. Of course, this will only work if it is taken serious. The goal of a review should be that the reviewer has a good understanding of what the change is, how and why it was done.

There are no dumb questions during a review. If you do not understand something while you do a review, ask. This goes especially towards seniors or leads that sometimes might feel they should not ask dumb questions. If you think you even need someone else’s opinion get it. You may and should criticize style and details. Ask for additional or improved comments if you think they might help. This is not only about making sure the change works; it is about sharing ideas and knowledge as well.

So, what do you get in the end? Reviews will easily spot obvious issues or problems with the idea of how to solve the issue at hand. They will rarely spot really intricate bugs or side effects. By that, it will remove quite a number of bugs that would have been found later by the automated systems, by the QA, or even worse, by somebody trying to use a broken tool.

What you also will get is people learning from each other, people looking into parts of the system they would not see usually. At least two people know the change that has been made in detail, so people getting sick or leaving the company becomes less of an issue.

You get a culture of talking about your work and making sure work is actually done before the check-in (it is pretty embarrassing if obvious flaws are discovered by your reviewer in a piece of code that you actually considered worth checking in). People in your team talk, they develop a common language, they understand weaknesses and strengths of the team members.

A few things to keep in mind to make peer reviews work:

it costs time – make sure everybody knows that this is time well spent and factor it into your estimates

every checkin is reviewed – a lot of mistakes are made with “easy” or “small” checkins

people should be available for a review – nothing is as annoying as not being able to check in just because nobody has time, therefore you should have a damn good reason to refuse a review

add the information about who did the review to the check-in comment – reviews will be taken a lot more serious that way, and if you hunt a bug caused by a check-in, you know the two guys you should talk to to help you

Check-in Comments

It might not be very clear initially how check-in comments can improve build stability because once the bug is checked in, it is in. Good check-in comments make it a lot easier to track down an issue. Applying a structured format makes it even easier. Just imagine sitting in front of the screen scanning through a list of 100 check-in comments to find out which change could cause your AI getting stuck while trying to vault over a cover. The easier it is to read the information and the better the information is in it, the faster you will be. We fixed quite a lot of our “hard” bugs that way.

But actually check-in comments (if they are well done) have even more uses. You can subscribe to your source control system (we use perforce) to get an automated mail for every check-in in areas you are interested in and stay up to date with what is checked in by whom. This is not only a useful tool for a lead; it is also interesting for other programmers, QA, or producers to know what actually is checked in.

Test Builds

This is something that is not easy to do and it requires substantial in-house QA resources and some additional tool support. The basic idea is again pretty simple: Before you check in a change that you are not so sure about – test it.

I guess everybody knows this bad feeling when he is changing something in a very old part of the code, and this code also touches a lot of other code (maybe the guy who originally wrote it is not even there anymore, or you have to change code in your middleware). You are just not sure about the side effects and yes there is no automated testing around that part of the code. Basically the only way to find out what your change does besides what you intend it to do, is testing it.

The best people you have for testing are QA people (some of our QA guys find the strangest bugs and more importantly reliable repros for really hard ones – amazing). So, the idea is to create a local build of the game (or representative part of it), and send that to the QA team to test your change. While you are waiting, you can shelf your change and continue with something else.

To make this a viable option, you need really great tools to make the whole process as easy as possible. We are using the Unreal Engine with their build tools. It is easy to create a local build of the game for any platform using the Unreal Frontend. This tool is used to cook the game for the platform you need it for. Out of this tool, we can push a build on to a central server (the prop server).

The QA can get this build by using a simple web frontend and have it copied to their PC or Xbox. Yes, you could cook a build, copy it into some network folder, and write a mail to the QA on where to find it. But the easier the whole process is, the more likely it is that people are actually using it and do not find excuses to not do it or get frustrated because they have to. We also established a bit of a strict workflow around it to make the whole process even smoother.

Even applying all of the things I explained above perfectly will not give you zero bugs, but it will allow you to spend your time on the important and interesting bugs and what is even more important – adding cool shit to your game.（source:gamasutra）