游戏邦在:
杂志专栏:
gamerboom.com订阅到鲜果订阅到抓虾google reader订阅到有道订阅到QQ邮箱订阅到帮看

分析主流Android开发商的应用测试方式

发布时间:2012-06-04 17:51:28 Tags:,,,

作者:Kim-Mai Cutler

几周前,我曾谈到香港开发商Animoca如何测试其Android游戏。该公司目前拥有7000多万次的下载量,在400款左右的设备上测试他们的各款应用作品。下图来自其总部,是该公司运用的所有Android手机和平板电脑概览。

animoca android from techcrunch.com

animoca android from techcrunch.com

毋庸置疑,上篇文章惹恼了不少Android支持者。有些评论者表示,它吓跑潜在开发者,他们因Android的分裂现象及需要支持众多设备、屏幕尺寸、密度和OS版本而心生畏惧。

因此我调查了其他手机游戏开发者是如何进行Android平台的质量控制测试。下面是所获结果:

life is crime from techcrunch.com

life is crime from techcrunch.com

Red Robot Labs:(由基准资本公司支持,由来自EA、Playdom和Crowdstar的元老团队创建。公司有350多万次的下载量。目前有款游戏在Google Play的高营收游戏榜单中排名第27)

Red Robot内部采用12种设备,设立一个由2人构成的质量控制小组。他们通过英国公司Testology进一步覆盖35款手机。

公司联合创始人Pete Hawley(游戏邦注:他来自EA,涉足游戏领域超过15年)表示,“我采用常识性筛选方法。”他遵循80/20法则,旨在确定能够覆盖最多用户的最少设备数量。他们从来自谷歌的基本数据着手,数据显示不同版本Android及屏幕尺寸密度的整体分布情况。然后他们进行深入分析,旨在找出运用最广泛的设备。最后,他们会查看用户请求和支持问题。

他表示,最好有选择性地确定所支持的设备,尤其是目前市面上存在各式各样由亚洲涌入的低端手机。

他表示,“排除采用小型、劣质、陈旧OS的用户也非常重要。总地来说,了解所有手机、运营商、OS及OS运营商并没我想的那么困难。争取80%的覆盖率并不需要投入很多工作。”

下图是Red Robot去年秋天设备分布的大致情况。

android install from techcrunch.com

android install from techcrunch.com

Pocket Gems:(由Sequoia Capital和Redpoint Ventures支持。有超过7000万次的下载量。刚涉猎Android平台不久,据苹果iTunes Rewind显示,去年他们有两款作品入驻高营收iOS游戏榜单前10名。有一款游戏在Google Play高营收游戏榜单中排名第35。)

所以Pocket Gems的QA测试其实是由前空军上校Ray Vizzone完成。他们采用40多款设备,在矩阵中进行评估。他们确保将平板电脑和手机,然后是高分辨率和低分辨率的设备都纳入在内,还有就是5个主要的图像处理单元(GPU)(游戏邦注:包括Adreno、PowerVR、Tegra、Mali和Vivante)。

他们的QA过程以快速为目标,因为游戏行业过去几年来已发生根本性变化。就如Zynga给社交游戏领域所带来的变化,如今的手机游戏更像是服务,而非你从货架上取出的制成品。它们需要每隔几天就持续更新新鲜内容。

对于这一旧金山初创公司而言,质量保证测试是个全天过程,由美国团队及海外团队共同完成。美国团队白天设计和完成测试后,他们会将内容转交给海外团队,他们手头持有同样的40种Android设备。这一团队会通宵进行额外的兼容性测试,将所有漏洞存储至缺陷追踪系统,这最终又在次日上午回到美国团队手中。

Pocket Gems分3个阶段测试各功能。包括:1)新功能测试;2)整合测试;3)候选发行版本测试。当开发者给游戏设计新功能时,Pocket Gems的QA团队已开始设计相应的测试方案,这样他们就能够在时机成熟时着手测试工作。当所有这些功能稳定后,它们就会被植入游戏当中,再次继续测试。

公司联合创始人Harlan Crystal表示,“随着漏洞在整合测试阶段逐步显现及顺利修复,产品经理和测试主管就会开始着手进行风险评估,确定何时冻结准备中的代码库,开始发行工作。一旦做出决策,完成回归(RC)测试就会启动。”

最后一个过程包含全套测试,主要检测内存、性能和设备兼容性。他表示,“如果我们没有在RC测试阶段发现任何新漏洞,那么我们就会将内容发行。”

Storm8:(超过3亿次的下载量。完全自食其力。有4款作品出现在Android前50高营收游戏榜单中。创始人是早期的Facebook成员)

Storm8利用30-50款设备,他们将此划分成高端、中端和低端设备。他们有意购买各类型的设备。游戏发行后,他们会让应用将不同KPI(关键绩效指标)送回公司的服务器。

公司首席执行官Perry Tam表示,“这样,我们就能够判断是否需要就特定类型的设备或者甚至是特定设备做出进一步调整,以尽可能提高设备的性能。”

storm8 testing from techcrunch.com

storm8 testing from techcrunch.com

Animoca:(超过7000万次的下载量。由IDG资本和英特尔资本支持)

继我发表上篇文章后,Animoca撰文陈述为何他们要在如此多的设备上进行质量保证测试。主要原因在于,该公司在中国大陆及亚洲其他地区拥有庞大的用户基础,这些地区有大量的低端和非兼容性Android设备(游戏邦注:它们采用没有资格运行谷歌应用或官方Android应用商店的Android操作系统)。

公司表示,“如果我们认为90%的兼容性就完全足够,那么将会流失700万次下载量。有数百万用户将因我们的决策而获得糟糕体验。我们的应用收益将减少10%。”

记住Android并非一家年轻公司。它是有着10多年历史的Outblaze手机游戏业务部门,Outblaze投身数字媒体和应用许多年。所以他们在进行兼容性和质量保证测试方面有丰富的经验。

该公司首席执行官Yat Siu表示,其综合性也是他们在平台取得不俗表现的部分原因,他们在Android平台的年收益高达上千万美元。目前Animoca没有任何作品出现在美国的50大高营收游戏榜单中,但他们在亚洲市场的排名非常靠前,此外他们的应用年发行量也非常惊人。

animoca worldwide from techcrunch.com

animoca worldwide from techcrunch.com

screen shot from techcrunch.com

screen shot from techcrunch.com

总结:如果这依然让你望而生畏,那么记住,这在功能手机时代远要糟糕得多(至少,Rovio首席营销官 Peter Vesterbacka是这么认为的。Rovio表示,相比J2ME/Brew时代,Android其实非常简单!他们在推出轰动巨作《愤怒的小鸟》前制作做了50多款作品)。

为说明过去的艰难处境,以下是两个来自JAMDAT 2005年原始IPO资料的幻灯片。JAMDAT在功能手机时代被EA以6.8亿美元收购。该公司花费了5年时间同全球40个国家的90多家运营商建立关系,当时应用通常需要支持400种设备左右。

screen shot 02 from techcrunch.com

screen shot 02 from techcrunch.com

screen-shot 03 from techcrunch.com

screen-shot 03 from techcrunch.com

所以虽然Android分裂现象有些令人头疼,但你父亲时代的手机应用开发商在QA测试中更是举步维艰,他们需要应对400种设备及来自100家运营商的业务开发人员。

如今出现Testology、uTest之类的专业服务帮助展开手机QA测试后,事情更是简单许多。也就是说,大型开发商依然能够在内部完成多数工作。(本文为游戏邦/gamerboom.com编译,拒绝任何不保留版权的转载,如需转载请联系:游戏邦

How Do Top Android Developers QA Test Their Apps?

By Kim-Mai Cutler

A couple weeks ago I ran this post showing how one Hong Kong developer, Animoca, tests its Android games. The company, which has had more than 70 million downloads, tests every one of their apps on about 400 different devices. The photo above is from their headquarters and is just a taste of all the Android phones and tablets they use.

Needless to say, that post pissed Android supporters off. Some commenters said it intimidated would-be developers, who might get scared off by Android fragmentation and the perception that you have to support hundreds of devices, screen sizes and densities and versions of the OS.

So, I asked around to see how other mobile game developers do quality assurance testing for Android. This is what I got:

Red Robot Labs: (Backed by Benchmark Capital. Veteran founding team from EA, Playdom and Crowdstar. More than 3.5 million downloads. They currently have the #27 top-grossing game in the Google Play store.)

Red Robot uses about 12 devices in-house and has a quality assurance team of two people. They then use a U.K.-based company called Testology to get further coverage with 35 handsets.

“I applied a common sense filter,” says co-founder Pete Hawley, who hails from EA and has more than 15 years’ experience in the gaming industry. He goes by an 80/20 rule in trying to identify a low number of devices that will cover the widest amount of users. They start with the basic data from Google that shows overall distribution of different versions of Android and screen size densities. Then they look at their analytics to find which devices are most widely used by their players. Finally, they’ll look at player requests and support tickets.

He says it’s good to be selective about which devices to support, especially with all sorts of lower-end handsets coming in from Asia.

“Saying no to players with small, poor, outdated phones or old OSs is important too,” he says. ”Overall, I’d say the process of staying on top of all the handsets, carriers, OS’s and carriers wasn’t as hard as I expected. It’s not a great deal of work to keep the 80 percent well-covered.”

Here’s a snapshot of how Red Robot’s device distribution looked last fall. (It’s a very fragmented pie!)

Pocket Gems: (Backed by Sequoia Capital, Redpoint Ventures. More than 70 million downloads. Newer to Android, but they had two of the top 10 grossing iOS games for all of last year according to Apple’s iTunes Rewind. #35 top-grossing game in Google Play.)

So Pocket Gems’ QA testing is actually run by a former Air Force colonel(!) named Ray Vizzone. They use a little more than 40 devices evaluated in a matrix they explain in the video below. They make sure they include both tablets and phones and then high-resolution and low-resolution devices. They also make sure to include all five major graphic processing units (GPUs) including Adreno, PowerVR, Tegra, Mali and Vivante.

Their QA process is designed to be hyper-speedy as the gaming industry has changed in some fundamental ways over the last few years. Like what Zynga has done in the social gaming industry, today’s mobile games are more like services rather than finished products you pick up off the shelf. So they require constant updates with fresh content every few days.

For the San Francisco-based startup, quality assurance testing is a 24-7 process that involves teams both in the U.S. and abroad. After the U.S. team designs and performs tests during the day, they hand their work to an offshore team that has all of the exact same 40 or so Android devices. This team does extra compatibility testing overnight and files all of the bugs into a defect tracking system, which go back to the U.S. team in the morning.

Pocket Gems tests all features in three phases. They have 1) new features testing 2) integration testing and 3) release candidate testing. Even as developers design new features for their games, Pocket Gems’ QA teams are already at work designing tests for them so they can be checked the moment they’re ready. Once those features are stabilized, they’re integrated into the games and tested a second time.

“As the bugs are found and fixed during integration testing, the product managers and test leads begin their risk assessment as to when to freeze the code base in preparation for shipping,” co-founder Harlan Crystal explains. “Once this decision is made, a full regression test pass is started.”

That final pass involves a full suit of tests that examine memory, performance and device compatibility. “If we don’t find any new or critical bugs during this RC test pass, we bless the bits and ship it!” he says.

Storm8: (More than 300 million downloads. Totally bootstrapped. Four games in Android’s top-grossing 50. Founders are early Facebook alums.)

Storm8 uses between 30 and 50 devices, which they divide into groups of high-end, mid-range and low-end devices.  They intentionally buy devices for each category. After they launch games, they have the apps send back different KPIs (key performance indicators) back to the company’s servers.

“This way, we can tell if we need to further fine-tune a certain class of devices, or even specific devices, to squeeze the last bit of performance from the devices,” says chief executive Perry Tam.

Animoca: (More then 70 million downloads. Backed by IDG-Accel and Intel Capital).

After the original post ran, Animoca ran a longer piece explaining why it does quality assurance testing with so many devices. The main reason is because the company has a huge user base in mainland China and other parts of Asia where there is a plethora of lower-end and non-compatible Android devices (meaning phones that are based on the OS but aren’t certified to run Google applications or the official Android app store).

“If we had taken the approach that 90 percent compatibility is good enough, we’d be lacking support for 7 million of [our] downloads,” the company explains. “Several millions of consumers would have had a bad experience as a result of our decision, and our app revenues would probably be short by around 10 percent.”

Keep in mind that Animoca is not exactly a young company. It’s a mobile gaming-centric arm of a more than 10-year-old company called Outblaze that has focused on digital media and apps for years. So they have lots of experience in doing compatibility and quality assurance testing.

The company’s chief executive Yat Siu feels that their comprehensiveness is a part of why they perform decently on the platform, with “double-digit” millions of dollars in revenue per year from Android. Animoca doesn’t have any games in the top-grossing 50 right now in the U.S., but they make up for it with high rankings in Asian markets and in the sheer number of apps they publish per year.

Conclusion: If this still freaks you out, just remember that it was way worse in the days of feature phones. (At least, that’s what Rovio’s Peter Vesterbacka tells us. Rovio says compared to the J2ME/Brew era, Android is actually easy! They had to make more than 50 games before they created uber-hit Angry Birds.)

Just for reminders about how hard it was then, here are two slides from JAMDAT’s original IPO slidedeck in 2005. JAMDAT was the seminal mobile gaming acquisition of the feature phone era when they were bought by Electronic Arts for $680 million. The company had to spend five years building relationships with more than 90 carriers in about 40 countries and it was standard to support about 400 devices.

So while Android fragmentation seems like a headache, your dad’s mobile app maker was trudging seven miles uphill in the snow QA testing with 400 different phones and dealing with business development people from a hundred carriers.

It’s also easier now with specialty shops handle mobile QA testing now like Testology, which Red Robot uses, and uTest. That said, the very biggest developers still want to do most everything in-house.(Source:techcrunch


上一篇:

下一篇: