Is automation really the future of game QA?
By Gaming IQ’s Arran Oakes
Automation has been heralded as one of the greatest leaps forward in QA, but is it really all it’s cracked up to be?
For game QA, automation generally means more confidence in a build’s status earlier, and significant time – and frustration – saved on extremely repetitive tasks.
Whether applied to general testing or specific cases, if dev teams can work with QA to write unit tests in the initial stages, it can save QA from having to run into regressions on builds. The more upfront automation you have on the builds before they hit QA, the more time the teams can spend finding bugs proactively and using different approaches.
REDUCING REPETITIVE TESTING
By running tests in parallel, a test lead can understand which areas of a test case are generating the most fails on a particular build. This information can be used to direct manual testing, and avoid time expended on testing incomplete features on a build. Sony is using automated tests to ensure that their manual test teams are freed up to be more effective – ideally using the automation tools to wrap up a lot of the tedious or costly test cases.
No more tying up the whole test team on smoke-testing multiplayer builds, herding bored testers towards headshots or tasks along the lines of ‘kill 1,000 enemies with a toothpick’. In the case of multiplayer load testing, it’s not uncommon to have automated kits spend 700-plus hours running in-game tests, increasing the chance that low frequency issues will be detected much faster.
As an added bonus, the automated kits also generate their own debug information, video capture, automatic core file linking and translation and developer trace.
With so many potential advantages for QA, it may seem surprising that automation has only recently started gaining momentum for all but the largest developers and publishers. There are several common reasons for this: financial, technical and, often the most difficult to overcome, cultural.
Automation is not cheap. Doing it on any significant scale requires investment in software, potentially hardware, and almost certainly skilled people. The latter is often the most challenging, as people versed in the art of automation are hard to come by. That said, it would be well worth setting your best developers and system architects to work on test automation frameworks now.
Technically, feature or design changes tend to be prevalent with some studios, so you’d need to ensure that your frameworks are as adaptable and modular as possible. This enables you to update a test suite as quickly as possible when design changes occur. Additionally, actually doing higher level automation and getting it to run reliably – let alone with consistent results – is not easy.
As with any significant investment of time or money, buy-in from senior stakeholders and the wider production teams is required.
It is not uncommon to see pushback until you can demonstrate tangible results.
THE CULTURE BARRIER
At the end of the day though, no automation framework is as adaptable as a real tester, so it usually requires substantial buy-in from development teams.
The earlier you can start working with a specific team, the better the results you’ll likely see. With each success you gain traction to change the culture. In some cases, Sony is working with developers to design in-test frameworks before development even starts.
Try to resist the temptation to start on only high-level integration tests such as UI or screenshot comparisons. These may be conceptually easy to understand and author, but they are also very fragile and not easily scalable. This could hold you back while you just lose time and money on maintenance.
A WORLD OF BENEFITS
Nevertheless, the potential benefits of automation are huge. For example, Unity’s ability to develop and release across so many platforms is almost solely attributable to automation. It enables them to more easily verify features across 15 to 20 platforms with a massive number of test cases run on every single commit.
Quite simply, automation is the latest, and potentially greatest, tool in the QA arsenal. Any mid-sized developer would be well advised to invest time in it as soon as possible.（source：develop-online）