游戏邦在:
杂志专栏:
gamerboom.com订阅到鲜果订阅到抓虾google reader订阅到有道订阅到QQ邮箱订阅到帮看

详细分析《Vessel》向PS3移植的过程

发布时间:2014-04-01 14:13:46 Tags:,,,,

作者:Tony Albrech

2013年1月,我开始着眼于《Vessel》。我会在“空闲”时间里着眼于这款游戏是因为John和Martin(游戏邦注:Strange Loop Games的创始人也是《Vessel》的创造者)让我考虑将这款游戏移植到PS3上。从在Pandemic Studios一起工作时我们就认识了彼此,在那里我与Martin一起致力于引擎团队而John则是作为一名AI程序员。我很想要进行游戏移植,但我也想在开始工作前明确需要为此付出多少努力。

Vessel(from overbyte)

Vessel(from overbyte)

我发现移植这样的游戏有点让人怯步——它具有定制的渲染引擎,一个带有主要游戏代码(基于Lua进行编写)部分的定制且流畅的实体引擎,但我也觉得可以试着去接受这样的挑战。我非常尊敬John和Martin——他们都是非常出色的程序员,我也期盼着与他们再次展开合作。我想要做这件事,但却不希望让他们失望。他们将成为我的客户,但同时也还是朋友。

我开始着眼于他们的代码——为了引用一个移植过程,我需要让游戏运行起来,如此我才能了解它。《Vessel》的部分内容已经经由第三方被移植到PS3上了,但当我接手时却还未进行编译。所以我花了几周的夜晚去修改FileIO,模版的编译错误(GCC和MSVS不同意模版的一些细节),并注释一些我认为并不相干的内容,从而让游戏能够运行起来。一旦它开始工作,我便会在游戏中运行我所信赖的分析器并尝试着猜测它需要多少时间才能将这个“小孩”发展到可发行的状态。我所发现的结果有点惊人。

《Vessel》运行的主线程有3条。游戏线运行着所有的Lua脚本,这控制着主要的游戏逻辑和游戏对象,渲染线程将输出游戏线程并将其转换到一个能够供应给SPU的格式中,然后创造一个图像需求缓存,并将其平行供应给GPU,还有实体线程管理着所有的实体处理—-流体动力学,刚体碰撞等等。John已经独自创造了实体系统,这真的很出色(也很复杂)。如果要从头开始了解它太困难了,所以我能做的便是着眼于游戏性能中一些较糟糕的关卡,并看看自己需要加快多少速度。

基于PS3结构我仔细审查了一些关卡,并选择了2个我认为最糟糕的。从中我知道实体线程是引起怠工的主要原因,所以我将自己的分析专注于该线程。基于我测试的最糟糕的关卡,游戏的运行大约是每帧60毫秒—-大概是每秒15至20帧。通过与Strange Loop的讨论,我知道实体线程是基于60帧每秒运行,这便意味着我需要加速4倍。根据以往的经验,我知道自己可以做到这点。游戏线程主要是基于每帧30毫秒而运行,渲染线程将基于每帧5毫秒进行记录。SPU渲染非常快,我也清楚这并不是个多大的问题。

经过几个月的时间,我们最终在6月份签订了合同。所以我只能在这种情境下做我能做的事—-我花了2周时间致力于《Vessel》然后与家人到英国度假6周。

在等待《Vessel》的合同期间我雇佣了一名资深程序员Ben Driehuis。Ben具有很高的知名度并曾致力于《生化奇兵》等系列游戏,更重要的是他与我住在同一座城市。在花了6年的时间进行远距离合作后,我发现及时递交游戏的最佳方法便是确保工作团队是在同一个地方工作。最终我选择了Ben。在接受雇佣后他致力于自己之前从未面对过的平台和引擎。然后当我离开前往英国时,他还必须独自处理这些工作,应对一款知名游戏的执行合同然后基于《Vessel》开始工作。可怜的家伙。

我同样有个朋友对QA这方面的内容很感兴趣,也就是James,所以他负责一遍又一遍地玩《Vessel》并向我们报告漏洞,告诉我们之前所做出的哪部分性能完善不起作用等等。James同时也帮助我们测试TRC(技术要求清单——一个大型的测试列表,为了将游戏递交给索尼就需要进行这样的测试)以及真正的游戏递交内容。

当我从英国回来后我便搬到了新办公室并开始致力于《Vessel》的移植。我们的目标是在8月末完成《Vessel》——我们拥有4个人工月的时间去完成工作,我仔细计算了移植的每一部分需要花费多少时间。并确保这些时间是否足够。

Ben和我划分了各自的工作任务:我致力于实体的优化而Ben负责其它内容,包括音频受损,一些没有生气的游戏对象,需要进行优化的加载,需要减少的内存使用,游戏保存,整体的PS3特定功能的执行等等。Ben已经在移植过程中编写了一些很有趣的内容。

从最初的代码检查来看,我注意到了许多内存分配的出现,所以我对削减(或删除)这些内容然后节省大量的帧时间充满信心。我同样也明确了重新安排内存的使用从而提高可缓存性而去帮助PlayStation 3的主处理机,从而让我可以暂时不去管实体代码的核心内容。我想要做的最后一件事便是移植所有的实体到SPU上。

三周后我开始移植实体到SPU。我所做出的改变不断改善性能,但是我发现如果不做出较大的改变我们便不可能达到想要的速度。移植代码到SPU与从一个平台移植代码到另一个平台是完全不同的情况。你可以基于C++编写SPU,但它们具有非常有限的内存—-每个只有256千字节。所以为了移植任何主要的复杂任务,你需要理解任务如何使用内存——它在哪里,它阅读的顺序是什么,它在哪里进行编写—-大多数使用高级程序语言的程序员都会认为这是理所当然了。除此之外,你还必须考虑任务的平行放置。在一个SPU上运行很棒,但在6个SPU上运行就太可怕了。

我所创建的第一个SPU任务是寻找在一套对象周围的掉落物。我的第一枪的帧时间是大约2毫秒——我们所考虑的功能是从最高的2.4毫秒变成0.6毫秒!这意味着为了每帧达到16毫秒,我只需要这么做21次便可!

我和Ben不断致力于代码中。在8月的第一个周末,我已经在实体线程创造了15毫秒帧时间,然而线程却仍是一个瓶颈。Ben腾出了超过200兆的内存,我们接近运行于一个零售的PS3上。但我们已经开始注意到一些事了——游戏中的一些杠杆和触发器并不能有效运行,流体着色器也是如此,在某些情况下我们达到了200毫秒,但是游戏却因为杠杆而不能运行,而关节轴承的AI也似乎遭到了破坏。

显然在这种情况下我们并不能满足8月的时间限制。我和Ben还需要花时间去参与其它项目的工作,所以我告诉Stranger Loop我们可能需要更长的时间去完成任务。他们都表示理解,并愿意往后推迟发行时间。

事情的一切进展顺利——在8月末,实体线程可以基于每帧33毫秒而有效地运行着,我们已经走到成功的半路了!将游戏保存在适当的位置,优化了加载同时也运行了战利品。不幸的是,游戏线程的速度放慢了下来——Ben开始使用代码和数据去解决问题,并需要处理更多的情况。例如,之前的音频不能运行,现在需要更多的音频,所以显然我们需要花费更多的处理时间。

在9月的第二周,我意识到关于《Vessel》的游戏线程我做出了一个带有缺陷的假设。我发现它必须运行于带有实体线程的步骤中,如此它必须基于每帧16毫秒的速度运行。降低实体的帧率并不是有效的选择,因为它需要固定于每帧16毫秒的速度上——进行不同的尝试将导致流体碰撞遭到严重的破坏。

所以这时候我们让实体运行于每帧30毫秒的速度上,游戏线程则运行于30毫秒以上的速度。虽然非常接近目标,但我们仍有很长的一段路要走—-现在我们必须明确如何优化Lua的游戏线程以达到每帧16毫秒的速度。实际上应该比这个更快—-因为PS3只有2个硬件线程,所以3个软件线程必须与这两个硬件线程进行分享,这意味着游戏线程加上渲染线程必须快于每帧16毫秒。

然后Ben发现了传送关节轴承的理由——一个编译器漏洞触发于一个优化组合,并引起关节轴承在跳跃过程中不能准确地估算它们的轨道,从而导致它们的消失。为了解决该问题我们关闭了功能的优化。这是一个很难找到的漏洞,所以我们真的很着急——直到我们意识到这表示现在视野范围内的关节轴承数量高于之前的数量。有更多关节轴承也意味着更多的处理过程—-更多的实体,更多的掉落物,更多的渲染以及更多的音频。这一修改导致实体线程从24毫秒回到了50毫秒,而渲染线程变成了12毫秒,游戏线程上升到40毫秒。

我非常担心。我们几乎回到了一开始。现在我们需要为3个线程节省总共60毫秒的时间。我不知道到底能不能做到这点。这就像是在恐怖电影中,主角杀死了怪兽,但却导致怪兽变得比之前更大且更厉害。

关于项目的最初优化非常简单。但随着时间的发展,当我们完成所有简单的任务后,剩下的优化便非常难执行。我们真的做好去迎接这些挑战的准备,并且现在的我们也不能再回头了。

在意识到游戏线程和渲染线程都需要基于60帧每秒运行时,我们需要重新规划优化工作。游戏线程所做的是完全不同的事—-Lua脚本语言编写,播放音频,处理AI,创造动画精灵等等。这里有许多需要理解的内容,基于此,我们需要投入更多精力于实体线程和渲染线程中。但性能并非我们唯一关注的内容—-还有TRC规则,游戏漏洞,因为改变而引出的新漏洞以及PC创建的需要等等工作。

Screenshot(from altdevblogaday)

Screenshot(from altdevblogaday)

当我们开始时,PC上的《Vessel》具有非常大的漏洞,我们在Lua和游戏代码中发现了一些漏洞,还有少量来自编译器和着色器的漏洞,但关于这两个平台在数字方面的处理也存在一些细微的差别,这意味着有时候也会发生一些古怪的情况。例如一个关卡上的一扇门被卡住,你的角色将为了打开它而与之相间。这是因为在适当位置上的变量大约是5个小数位,从而引起了PS3的阻塞。此外,还有一些位置上的代码表现如下:

x = MIN( y/z, 1.0);

在PC上这意味着z将趋于0,并将x变成1.0 ( 最小值(无限大, 1.0) = 1.0)。不幸的是在PS3中值/0.0是QNAN(无意义的非数字),而MIN(QNAN,1.0)便是QNAN。所以这将导致我们很难去复制位置(只有在z是0时才可能)。

这意味着我们不能只是优化并添加全新的PS3功能,我们也要调试一个大型的外来代码库。我从未期待着我们需要为《Vessel》做出许多改变—-毕竟这是一款已经发行的游戏,所以我会假设它没有太多的漏洞。

我还假设从很大程度上看来我们不需要对任何资产做出太大的改变。像针对于控制器和按键的平台特定图像便只是在纹理上做出改变,所以这并不是问题所在。文本也是如此。不幸的是,像之前提到的卡住的门这样的问题意味着我们需要深入某些关卡并做出更大的改变。在移植的后半段,我们必须调整流体流动,排放速度,排除情况以及其它基于流体的效果去提高性能。

所有的这些都意味着我们需要打开编辑器并运行,因为编辑器是使用于游戏同样的引擎进行创造,所以我们需要维持游戏的两个架构—-一个针对于PS3,一个针对于PC。我们通过定义大多数的PS3改变并将PC代码有条件地保存着而粗糙地做到了这点。如此不仅放慢了我们的速度,同时还导致我们的代码变得更糟糕,但这也具有一定的副作用,即让我们能够轻松地测试一个代码是基于PC架构还是针对于PS3。

在9月末,我们修正了许多音频问题并努力让LuaJIT(游戏邦注:Lua的优化版本)再次运行起来(虽然这大致可行,但却在某些位置上引起了许多重复的问题)。Ben同样也规划出Lua的内存使用,并调整了它的垃圾回收为更加CPU友好型(有时候会达到13毫秒)。渐渐地,我们找回了在游戏线程所花费的时间。虽然这仍然是一个严重的问题,但却在逐渐好转。

回到实体系统上,我们可以看到越来越多CPU代码已经被移植到SPU上了—-基于各种方法,我创造了许多基础设施去推动这种移植更加便利化。我用于一种系统的许多模式也能够通过转换而为其它系统所利用。我开始减少实体的某部分所使用的内存数,如将WaterDrop结构的规格从144字节减少到128字节,这便意味着在SPU上创造出更大的缓存使用以及更更简单的处理过程。

一些优化非常直接。例如,我改变了我们处理关节轴承结构上的掉落物方法。比起在整个掉落物名单上进行迭代并检查每个掉落物以明确它是否是“N”结构的一部分,我让它能够忽略掉落物名单,而将掉落物分成每个结构的不同掉落物桶。然后我们便可以只是处理特定的结构,只接触那些基于该结构的掉落物。显然这是一种后见之明,在PC中是不需要的,但它却能够在你真正开始理解代码并开始改变它的运行时发挥作用(比起优化一个功能去执行同样的公式,这种方法更快)。

这时候,我们也同样看到渲染线程变得更容易处理。所以我们开始致力于这些内容—-展开循环,预取,缩小数据复制,调整,刺,戳,打破,测试,测量,然后咒骂或欢呼以及登记。

在9月末,我们仍然未看到真正的尽头。我继续与Strange Loop进行沟通,尝试着告诉他们最新的情况,而我们也仍然致力于完成这款游戏。我们已经不再使用最初的报价,合同规定我们本来应该在剩下的时间里减少日工资汇率,并通过销售进行补偿。所以我们不仅承受了巨大的时间压力,同时还需要承担着较大的财政压力。

10月

10月是个高产的月份—-在这个月里我们修正了超过150个漏洞并为下个月的优化做好了准备。随着游戏性能的完善,并变得更加可行,我们的QA James能够开始频繁地玩游戏。这一阶段我们并未真正尝试着进行完整的游戏测试,所以我们主要还是专注于删除所有的漏洞并让游戏达到一个可发行的状态。

在第二周,即与Strange Loop Games讨论后,我们决定将游戏的发行时间延迟到1月。这意味着我们需要在年底将游戏提交给欧洲(SCEE)和美国(SCEA)的索尼。我们不能再延长这一时间了。

Ben仍然致力于优化Lua执行和音频(在这一阶段仍然需要花费每帧4或5毫秒的速度)。我想的是因为《Vessel》使用了FMOD,所以当我打开它时它能够有效运行。但不幸的是,音频在PS3中简直是一团糟。尽管我们详细描绘了游戏线程,但却发现需要花费大量的时间,并破坏帧的流畅于运行。我们花费了几周的时间并与FMOD负责人进行各种讨论才明确了问题所在。最终证明许多针对于PS3结构的声音设置都是错误的—-一些在PC上不可行的效果和层面都被留给了PS3。一些声音只是无限循环着,许多声音还会同时响起。所有的这些都需要时间去发现,调查与纠正。但是我们却没有多少时间了。

更糟的是,《Vessel》的执行并未选择渲染,AI或音频。从根本上来看,整个关卡被递交给一个选择了它,然后忽视了SPU渲染器的可视位元的渲染器。同样地,Lua的脚本得到调用,甚至是当它们所修改的内容不处于视线范围内时。音频也会被离玩家很长一段距离的对象所触发,依赖于距离衰减去减少容量。所有的这些都意味着在不可视与不可听的影响下我们还有许多事要做。

我们还发现了一些古怪的小问题。例如,游戏中的动画纹理没有任何生气。虽然它们都是基于正确的加载,但却从未发生改变。Ben追踪发现一个尾数法交换功能被破坏了(PC和PS3具有不同的字节顺序,所以当从文件中加载同样的数据时必须居于不同平台做出修改)。我们只能在一个位置称它们为尾数法交换功能,并且只能是基于这一特殊的动画材料。

问题的修改通常都很简单。但是找到问题却非常困难。

11月

尽管我们面对着非常紧迫的时间表,但我们同样需要履行对其他客户的承诺,所以在11月我们各自有2周时间不能致力于该款游戏。虽然这能够帮助我们缓解财政上的压力(如果对方能够及时付钱的话),但却也带给我们更多的时间压力。但不管怎样,我们都有信心能够在12月底完成工作。

我们最终在11月进行了一次完整的游戏测试。并只发现最终游戏不能有效运行。所以对此进行了修改,完善并测试。

在11月的最后一周,Ben取得了突破性的发展。他为游戏线程执行了一个基于选择系统的容器。在一个装载关卡中所有对象都被放置于一个矩形的容器中,每个关卡具有任意的对象。这意味着我们可以通过着眼于容器以及其旁边的内容而快速明确该渲染什么或处理什么。结果很简单,但执行要求我们修改编辑器和渲染器,然后基于编辑器访问每个关卡并手动添加新的容器,然后再次输出一切内容。

这一修改为游戏的性能创造了巨大的改变。音频变得更快,并且不再像许多声音那样混乱。当活跃的内容越少时,我们便越不需要进行Lua处理。当我们需要发送给渲染线程的内容越少时,渲染便会越快。所以一切都能够有效地运行。实际上,我们必须限制帧率为30帧每秒,因为在所有线程中一些位置是基于60帧每秒进行渲染。

容器的修改有助于让游戏线程发展成更加可游戏的速度,但这同样也带来了许多小漏洞,并需要我们花费一个月的时间进行修改。

12月

离圣诞假期只有3周了,我们清楚自己需要开始缩减某些关卡中的内容从而才能提升帧率。我们选择了一些最糟糕的内容并仔细检查它们以尝试着明确如何为它们提升速度。我们悟出了以下真谛:

树木非常昂贵。所以我们砍掉了一些。

排水沟非常昂贵,所以我们删除了一些。

带有许多水的区域非常昂贵,所以我们减少了某些区域水的设置,并削减了流或添加排水沟去减少水池中的水量。

我们调整了对象限制代码以确保总是有足够的关节轴承能够完成特定的关卡,然而不会设置太多以避免它运行得过满。掉落物和种子的数量也是如此。

上述的这些真谛非常有帮助,现在的游戏已经能够非常有效地运行了。可能在有些区域的速度还是较慢,而你可以通过迸射出水并生成许多关节轴承而故意放慢某些区域的速度,但是我们却没有时间进行进一步的优化了。我已经创造了13个不同的SPU任务去加速实体,其中的1或2个是用于渲染线程—-我们很难再进一步去加快速度,而在这一阶段做出任何额外的重要改变也具有很大的风险。但这却是我们必须做的事。

现在James注意到了一些特定的游戏漏洞。一些关节轴承不能有效地移动—-跳太高并在某些场景中摧毁了自己或表现糟糕。这在大多数情况下是合理的,但如果是伴随着你需要用于完成关卡的关节轴承的话便不是这样。我们必须修改关卡让它能够再次运行。

除了优化外,我们仍然在执行,关于一些奇怪内容的漏洞修改也进行着,我们还必须确保游戏是符合TRC。James查阅了大量的信件,并突出了我们需要检查的内容—-游戏保存/加载的运行,为何游戏会停止运行,在没有硬盘驱动器空间的时候会发生什么,以及战利品打开的顺序等等。

基于此,因为《Vessel》移植所需要的时间而带来的财政压力,以及用于支付工资的成本的减少,并且在接下来的时间里的工作量也没有那么多了,我决定对Ben放手。

这是我需要做的最困难的一件事之一。我觉得自己好像背叛了他,他是如此优秀的程序员,并且我们还成为了好朋友。但就是在圣诞节之前,我用Skype跟他讲了这件事。真是糟糕透了。

不过在几个小时内他打电话跟我说,自己刚刚找到了一份新工作。真的就只是在2个小时内。可见他有多出色。

让我们回到代码中。在剩下一周多的时间里,我们主要专注于剩下的TRC问题。游戏的运行就如我们想象中的那般顺利,我也因此感到大满足。我们并未基于所有关卡而达到完整的帧率,但它最终会做到这点。TRC的妨碍消失了,它看起来就像我们所想的那样。

在最后一周(周三),我意识到我们正在使用的文件的格式需要复合DRM,如此才能兼容TRC,所以我便投入了17个小时尝试着去修改它。经历了我的第一次通宵工作并想办法在早上5点的时候解决该问题。在接下来的一天时间里我继续修改,调整并测试内容,同时喝下好几杯咖啡以保持清醒。

提交事件

最后的周五到来了。我们进行了一些清理并为最后的提交准备好了代码。我们即将完成任务!在最后一次浏览TRC清单时,我发现我们漏掉了某些内容。如果漏掉任何必须的启动文件的话我们就需要报告错误,但因为引擎的设计方式,我们只有等到那些主要文件加载好后才能呈现任何内容。我们只有几个小时的时间,所以我便快速启用了一个备用渲染器(这是我在遭遇主要失败时能够使用到工具)。我只给自己2个小时的时间去完成这项工作,但最终我在30分钟内便完成了。太棒了!不过因为进一步的测试,加载屏幕拒绝运行。我并不清楚具体原因—-我所添加的代码从未执行,并且仍然影响着加载屏幕。游戏本身的运行是好的,但那些该死的加载屏幕却不能运行!我并不想提交一个自己知道在TRC上遭遇失败的内容,所以再一次地,我们又错过了截止期限。

我们不得不将提交时间延迟到下一年。

M._Arkwright_Sprays_Lava_Fluros_with_Liquid_Gun(from altdevblogaday)

M._Arkwright_Sprays_Lava_Fluros_with_Liquid_Gun(from altdevblogaday)

在圣诞假期后(我被迫又加入了另一次的家庭旅行),我直接投入代码中,希望能够修改最后一个问题,并尽快提交游戏。我们还有时间去提交游戏,如果我是第一次通过索尼的提交过程,我们便仍然可以在1月末将游戏推向PSN商店。我修改了剩下的内容并发送了错误报告,添加了更多错误校验以确保不再漏掉什么问题。我非常担心提交过程,因为我不想再因为失败而延迟游戏提交—-我仔仔细细地检查了所有内容两次,并最终于1月9日点击了“提交”按键,将游戏提交给了Sony America(SCEA)。

与代码提交过程并行的是“元数据”提交过程。这相当于PlayStation Store资产和游戏的呈现。这包含了所有的文本,图像,预告片,价格等等,并且它们也都具有特定的要求(为了能够通过提交)。James在这方面投入了巨大的努力,并与Strange Loop的美术负责人Milenko保持着交流,要求不同的分辨率图像和截图,以及不同版本的游戏文本。我们进行了几次元数据提交才成功,但转变速度真的很快,而持续与索尼的交流也起到很大的帮助。

代码提交过程包含在特别游戏包格式中加载游戏,并加上一些描述战利品和其它组件的额外文件。我们必须一起提交给SCEA和Sony Europe(SCEE),如此我们才能同时在这些区域发行游戏。我们不需要同时面向SCEE和SCEA提交游戏,因为我们仍然需要等待一些发行商登记细节,所以我能做的便是等待SCEA对于我们最初提交的回应,同时致力于其它工作。

在1月18日,我收到了来自索尼的第一次失败通知。他们告知了我三个“必须修改的”漏洞:一个因为我在提交游戏包的时候在某一区域输入了错误数据,另外两个是因为当没有磁盘空间时未能有效保存游戏。他们同样也提到了某些关卡中的怠工情况—-这是我能猜到的,但因为我认为这不是什么主要漏洞,便将其忽视了。游戏保存漏洞有点麻烦—-Ben编写了所有的游戏保存代码,但因为他已经离开了,所以现在我必须按照索尼的要求,Ben的做法学着去解决这一问题。我花了几天的时间去寻找并解决问题,同时我也收到了SCEE的要求细节,所以这一次我同时向SCEE和SCEA提交游戏(1月24日)。

这一次我对游戏充满自信。我完完全全地测试了游戏保存代码,并且它们都能够有效运行。如此还有哪些方面会出错呢?我甚至购买了一篇Chimay Blue想要作为庆功酒了。

然而在2月2日我收到来自SCEE的第一次失败通知。我竟然再一次搞砸了提交数据并且他们还发现了一个漏洞,即当你完成游戏时不能从保存好的游戏中继续游戏。当我在2月4日重新提交了修改后的新内容到SCEE后,我便开始担心之前提交给SCEA的内容—-它本应该与SCEE的结果一起回来的。同时我也担心SCEA会通过我的提交,毕竟它是带有漏洞的。结果是我白担心了,因为SCEA团队找出了之前任何测试未提及的新漏洞。

这一新漏洞报告了SCEA退还了我的游戏提交的原因。他们所报告的崩溃内容听起来就像我之前所修改过的漏洞。其中一个漏洞就像是“悬挂在关卡中的绳子上,然后投掷种子到一个没有水的区域中至少5分钟,70%的情况下都会遭遇崩溃。”在着眼于这一漏洞的同时,我收到了来自SCEE的通知,他们通过了《Vessel》的提交了。在兴奋的同时,我也担心游戏结构中是否还有其它重复的崩溃。

到目前为止我真的有点疲惫了。这就像是主角杀死了邪恶的僵尸,但却发现它能够不断复生一样。难道我的提交过程也会这样永无止尽吗?

我跟踪着崩溃的漏洞最终发现了一个非常愚蠢的错误。这里有一部分的代码是负责限制特定部分中流体,关节轴承和种子的数量。当帧率下降到60帧每秒以下时,代码将开始运行,掉落对于关卡来说最适合的最小值的掉落物/关节轴承或种子。在开发的最后阶段,我们创造了一个FINAL_RELEASE模式,它能够关掉所有不必要的调试代码。不幸的是,这错误地包含了一些会提升有限的代码所使用的时间值的代码,如此流体,关节轴承和种子将永远都不可能减少。这意味着游戏将更慢地运行,并可能在特定的应代码限制出现时遭遇崩溃。我从未对失败感到如此的释怀。

在此我想要避免的某些内容便是伴随着《Vessel》一起提交的性能等级。Ben和我花了许多时间尝试着在我们所使用的平台上呈现更好的游戏性能。大多数情况下我对结果都是很满意的。你可以滥用游戏并让它非常缓慢地运行着—-而如果你尝试着用关节轴承,水和种子填满一个空间,你便有可能看到一些怠工的出现,但在正常游戏过程中,游戏将基于30帧每秒的帧率维持着60帧每秒的模拟(图像是30帧每秒,所以每个可视的帧上会通过2次模拟)。然而某些关卡还要求特定的关节轴承和流体的数量去解决谜题,这意味着帧率将降至20帧每秒。如果还有一个月的时间,我便能够修改这一问题,但考虑到时间和预算的限制,这并不是一个好的选择。所以在兴奋于完善游戏帧率的同时,我仍然对不能实现完整的60帧每秒的模拟表示失望。

当然了,PlayStation 3硬件会减慢游戏最初的移植,但考虑到流体模拟的复杂性,我怀疑是否还会发行一个Xbox 360版本的游戏。SPU让我们能够基于至少80%的实体模拟而执行—-一点都不影响到主处理器。而在X360上我们不可能达到同样的性能等级。

随着FINAL_RELEASE漏洞的修改,我最终将游戏提交给了SCEE和SCEA。元数据经过了排序,现在的我们可以期待着在3月11日于SCEA区域,以及3月12日于SCEE区域发行游戏了。真心希望游戏的表现能够弥补我们的成本。

什么是对的

找到一个QA人员是非常重要的。一个公正且非技术型第三方成员能够诚实地找出需要修复的内容。作为开发者的你将非常了解游戏并总是想着假设自己的修改内容具有正面的差异。但事实并总非如此。

与发行商/客户进行诚恳的交流。我尝试着让Strange Loop Games的John保持对游戏进程的了解。如此他便能够理解持续的延迟,并且尽管传递坏消息是件很困难的事,但这却能够降低坏消息的冲击力。

拥有一位出色且资深的程序员。将Ben带到团队中是一个很棒的选择,尽管最后我不得不与他分开。如果没有他我便不可能完成任何内容。在同一个办公室工作也是非常有帮助的。我曾经于团队成员远程工作超过5年,所以我发现如果是分开工作的话我们传递的信息也会出现延迟。

什么是错的

工作评估。我从根本上低估了所需工作的数量。我应该使用Yossarian的Circular Estimation Conjecture并将其乘以PI—-这将更接近目标。最致命的元素便是误解了线程应该基于60帧每秒而运行。如果这只是我所低估的实体元素,我可能只会耽搁1个月(或者2个月),但因为实体线程和游戏线程都需要基于16.6毫秒而运行,所以我们将需要投入更多的工作。花费在提交游戏的时间也不能被漏掉‘在最初提交后4至8周你就应该在商店中看到自己的游戏了。

我同样也会建议任何在创造主机游戏的人应该在开发过程中尽快遵从TRC。尽快确保加载/保存内容的运行,有效设置战利品,错误报告,退出,控制器连接,文件错误报告等等内容。

结论

对我来说这款游戏的移植真的是一次重要磨练。在9月/10月中有1,2周的时间我真的在怀疑自己是否能够基于可游戏的帧率去传递游戏。最终我对自己所传递的内容感到自豪。从这次的经历中我真的学到了许多,并真正期待着下一次的挑战。

本文为游戏邦/gamerboom.com编译,拒绝任何不保留版权的转载,如需转载请联系:游戏邦

Vessel Post Mortem: Part 1

by Tony Albrech

I started looking at Vessel in January 2013 – initially just in my evenings. In my ‘spare’ time. I was looking at it because John and Martin (the founders of Strange Loop Games and the creators of Vessel) had asked me to consider porting it to PS3.  We knew each other  from our time together at Pandemic Studios in Brisbane where I had worked closely with Martin in the Engine team while John worked as an AI programmer. I was keen to do the port but I wanted to be as sure as possible that I knew how much work was involved in it before committing.

I found the idea of porting such a game daunting – it had a custom rendering engine, a bespoke fluid physics system with major parts of the game code written in Lua – but I figured it was time to step up and take on such a challenge. I have an immense amount of respect for John and Martin – they are both incredibly smart coders and I relished the chance of working with them again. I wanted to do it but I didn’t want to let them down. They were going to be my clients, but they were also friends.

I started by looking at their code – in order to give a quote on the porting process I needed to have the game running so that I could profile it. Vessel had already been partially ported to the PS3 by a third party, but wasn’t compiling when I first got my hands on it. So I spent a few weeks of long evenings fixing the FileIO, the template compilation errors (GCC and MSVC disagree on some of the finer points of templates), and just commenting out anything I didn’t think was relevant in order to get the game up and running. Once it was working, I ran my trusty profiler over the game and tried to guess how much time it would take to get this baby to a shippable state. What I found was a little scary.

Visualisation of the code changes made during the porting of Vessel to PS3.

Vessel runs predominantly on three threads. The game thread which runs all the Lua scripts which control the majority of the game logic and in game objects, the render thread which takes the output of the game thread and converts it to a format that can be fed to the SPUs which then build the graphics command buffer and feed it to the GPU in parallel, plus the physics thread which manages all the physics processing – fluid dynamics, rigid body collisions, everything. John had built the physics system himself and it was pretty impressive (and complicated). It was too much to grok initially, so all I could do was look at the game’s performance on some of the worst levels, profile it there and see how much I needed to speed it up.

I investigated a few of the levels on the PS3 build and picked two that I thought were the worst. What they told me was that the physics thread was the primary cause of slowdown and so I concentrated my analysis on that thread. The game was running at about 60ms per frame in the worst levels I was testing – about 15 to 20 frames per second. From discussions with Strange Loop, I knew that the physics thread had to run at 60fps so this meant that I needed a factor of 4 speed up. From experience, I knew that this was something that was achievable. The game thread was running at around 30ms per frame and the Render thread was clocking in at around 5ms per frame. SPU rendering was pretty fast, and I figured that wouldn’t be much of a problem.

Months of fannying about took place and a contract was finally signed in June. So I did the only appropriate thing one could do in that situation – I did two weeks of work on Vessel and then went on a 6 week family holiday to the UK.

In anticipation of the Vessel contract I’d hired an experienced programmer, Ben Driehuis. Ben came highly recommended and had worked on the Bioshock series of games, plus, most importantly, he was living in the same city as me. Having spent the last 6 years working remotely I knew that my best bet of delivering this game on time was to be in the same physical space as my team. I threw Ben in the deep end. He was hired and then given work on platforms and engines that he’d never had to deal with before. Then when I buggered off to the UK he had to manage by himself, dealing with a performance contract on a big name title and then kicking on with Vessel in my absence. Poor bugger.

I also had a friend who was interested in the QA side of things, so James (an old Uni mate) was responsible for playing Vessel over and over and over again, reporting bugs to us and telling us that the performance improvements we’d just made hadn’t made any difference. James also helped us out on the testing for the TRC (Technical Requirements Checklist – a massive list of tests that need to be checked off in order to get your game through submission to Sony) and the actual submission of the game.

Once I got back from my holiday we moved into the new office and started working in earnest on the Vessel port. Our goal was to finish Vessel by the end of August – we had about four man-months to do the work, and I had carefully calculated how much time each section of the port would take. Surely that was enough time?

Ben and I divided up the work between us: I took on the optimisation of the physics and Ben was to deal with pretty much everything else – audio was broken, some of the in game objects didn’t animate, loading needed to be optimised, memory usage had to be reduced, we needed save games and trophies and a whole swathe of PS3 specific features implemented. Ben has written about some of the fun he had during the port here, here and here.

From my initial inspection of the code, I had noticed a lot of memory allocations taking place, so I was optimistic that I could reduce (or remove) these and scrape back a lot of frame time. I also figured that rearranging the memory usage to be more cache friendly would help the PlayStation 3’s main processor so I could leave the core of the physics code alone. The last thing I wanted to do was to port all of the physics to SPU.

Three weeks later I started porting the physics to SPU. The changes I was making were incrementally improving the performance, but I could see that we’d never hit the speed we needed without dramatic changes. Porting code to SPU isn’t like porting code from one platform to another. You can program SPUs in C++, but they have very limited memory available to them – only 256kb each. So, in order to port any significantly complex task, you need to understand how that task uses memory – where it is, what order it reads it, where it writes it – the things that most programmers of high level languages take for granted. Not only that, but you have to consider how the task will parallelise. Running on one SPU is good, but running on 6 is awesome.  (I’ll go into more detail on the SPU porting process in later articles.)

The first SPU task I built was one that looked for drops that were near a set of objects. My first shot at this shaved almost 2ms off the frame time – the function in question dropped from a peak of 2.4ms to 0.6ms! That meant that in order to hit 16ms per frame I just needed to do this 21 more times!

Still, Ben and I kept hammering away at the code. By the end of the first week in August I’d shaved 15ms off the Physics thread and yet that thread remained the bottleneck. Ben had freed up over 200MB of memory and we were close to running on a retail PS3. But we’d started to notice a few things – some of the levers and triggers in the game weren’t working, fluid shaders weren’t working properly, we were getting spikes of over 200ms in some cases and the game still wasn’t playable due to not just the levers not working but the AI of the fluros seemed to be broken (some would teleport away).

It was evident at this point that we weren’t going to hit the end of August deadline. Other projects had taken some time from both Ben and myself, so we still had billable time available going into September, so I let Strange Loop know that things were going to take a little longer than I had expected. They were very understanding and comfortable with the release date pushing out a little more.

Things were progressing well though – by the end of August the physics thread was running at better than 33ms per frame – we were over halfway there! Save games were in place, loading was optimised, and trophies were functioning. Unfortunately, the game thread was slowing down – as Ben was fixing problems with the code and data, more things were being processed. For example, audio wasn’t working before and now there was a lot of audio being triggered so this obviously took more processing time.

It was the second week in September when I realised that I’d made a horribly flawed assumption about the game thread in Vessel. I discovered that it had to function in lock step with the physics thread and so it too had to run at 16ms per frame. Reducing the framerate of the physics wasn’t an option as it was fixed to 16ms per frame – experimentation with varying that resulted in fluid collisions breaking horribly.

So, at that point we had the physics running at sub-30ms per frame, and the game thread running at 30ms and above. We were close, but we still had a long way to go – we now had to figure out how to optimise the Lua heavy game thread down to 16ms per frame. Actually, it had to be faster than that – as the PS3 has only two hardware threads, the 3 software threads have to share those two hardware threads which meant that the game thread plus the render thread had to be faster than 16ms per frame.

Then Ben discovered the reason for the teleporting fluros – a compiler bug triggered only in optimised builds was causing fluros to incorrectly calculate their trajectory during jumps, making them disappear. Turning off optimisation for that function fixed the problem. That was a tricky bug to find, so we were pretty stoked – until we realised that this meant that the number of fluros in view was now higher that it was before. Much higher. More fluros means more processing – more physics, more drops, more rendering, more audio. This one fix resulted in the Physics thread jumping back up to 50ms from around 24ms, the render thread jumping up to 12ms and the Game thread climbing to 40ms.

I was horrified. We were effectively back to where we had started. We now needed to save a total of 60ms over three threads. I wasn’t sure we could do it. It was like that bit in horror movies where the protagonist has killed the evil monster, only to have it rise up, stronger than it was before.

The initial optimisations on a project are the easiest. As time goes on and all the low hanging fruit has been picked, the remaining optimisations are harder and harder to perform. We really had our work cut out for us – and we’d come too far to go back now.

Vessel Post Mortem: Part 2

In the last episode, our intrepid heroes had discovered that rather than having almost finished the PlayStation 3 port of Vessel, they were barely half way there. Read on to find out what happens next…

(This is part 2 of a 3 part series on the porting of Vessel from PC to PS3. Part 1 can be found here)

With the realisation that both the game and render threads needed to run a 60fps, we had to reassess where we focussed our optimisation efforts. The game thread was doing a lot of different jobs – Lua scripting, playing audio, AI, animating sprites – all sorts of things. There was a lot to understand there, and on top of that, both the physics and render threads still needed more work. But performance wasn’t the only concern – there was TRC compliance, game play bugs, new bugs introduced by our changes and, as we eventually discovered, the need for a PC build (more on that later).

Now, Vessel was ‘mostly’ bug free on PC when we started – we did find a few bugs in the Lua and in game code and the few compiler and shader compiler bugs added to that, but there were subtle differences in the way the two platforms dealt with numbers which meant that weird things sometimes happened. For instance, a door on one of the levels would get stuck and your character would have to bump into it in order for it to open. This was due to a variation in position values about 5 decimal places in causing the jamming on PS3. Additionally, there was some code that was used in a number of places that did something like the following;

x = MIN( y/z, 1.0);

What this did on PC was catch the case where z tended to zero and clamped x to 1.0 ( MIN(infinity, 1.0) = 1.0). Unfortunately, on PS3 value/0.0 was QNAN and MIN(QNAN, 1.0) was QNAN. So the clamping never happened resulting in much weirdness in hard to reproduce places (it only occurred when z was zero), and was therefore a bugger to find.

This meant that we weren’t just optimising and adding in new PS3 functionality, we were also debugging a large foreign codebase. I hadn’t expected that we’d need to change Vessel much – it was a shipped game after all and so I had assumed that it was pretty much bug free.

I had also assumed that for the most part we wouldn’t need to change any assets significantly. Things like platform specific images for controllers and buttons were just texture changes and so weren’t a problem. Text was fine too. Unfortunately, problems like the sticking door mentioned above meant that we need to go into some of the levels and jiggle bits around. Later in the life of the port we had to tweak fluid flows, emission rates, drains and other fluid based effects to help with performance.

All of this meant that we needed to have the editor up and running, and the editor was built using the same engine as the game so we needed to maintain two builds of the game – one for PS3 and one for PC solely for the editor. This was done crudely by #defineing most of our PS3 changes out and leaving the PC code in there inside the #else conditional. This did slow us down a bit and also made our code that much uglier, but it had the side effect of allowing us to easily test if a bug was present in the PC build or was specific to the PS3.

By the end of September we had fixed many audio issues and managed to get LuaJIT (an optimised version of Lua) working again (it was mostly working but was causing some repeatable problems in some places). Ben also profiled the Lua memory usage (article here) and tweaked its garbage collection to be more CPU friendly (it was occasionally peaking at 13ms in a call). So, slowly, we were scraping back time spent on the game thread. It was still a serious problem, but it was getting better.

Back on the physics system, more and more the CPU code was being ported to the SPUs – in many ways, this was getting easier as I was building a lot of infrastructure which made it more convenient to port. Many of the patterns that I used for one system would translate well to others. I started reducing the amount of memory used by parts of the physics, like dropping the size of the WaterDrop struct to 128 bytes from 144 bytes meant better cache usage and simpler processing on SPU (nicely aligned structs meant that I could assume alignment for SIMD processing).

Some optimisations were straightforward. For example, I changed the way that we processed drops on fluros skeletons. Instead of iterating over the entire drop list and checking each drop to see if it was part of skeleton ‘N’ I changed it to pass over the drop list once, partitioning the drops into buckets of drops per skeleton. Then we could just process a given skeleton, touching only those drops on that skeleton. Obvious in hindsight, and unnecessary on the PC build, but it takes time to get to the point where you really start to understand the code and can begin to change the way it works (rather than optimise a function to perform the same algorithm, just faster).

At this time we also saw the transition of the render thread to the status of low hanging fruit. So we started working on that – unrolling loops, prefetching, minimising data copying. Tweaking, poking, prodding and breaking, testing, measuring, then swearing or cheering and checking in.

The end of September rolled around and we still had no real end in sight. I kept talking to Strange Loop, trying to keep them up to date while we furiously worked to finish this game. As we’d already blown the initial quote, our contract specified that we would have reduce our daily rate for the remainder of the project, to be recouped on sales. So not only did we have significant time pressure, we now had financial pressure on top of that.

October

October was very productive – it saw over 150 bug fixes and optimisations checked in for the month. As the game was improving in performance and becoming more playable James, our QA guy, was able to play more and more of it. We hadn’t actually managed to do a complete play through of the game by this stage, so we focussed heavily on removing all blocking bugs and getting the game in a functionally shippable state (assuming that we would continue to make performance improvements).

Second week in, after discussions with Strange Loop Games we decided to postpone the release of the game to January. This meant submitting the game to Sony in Europe (SCEE) and in America (SCEA) by the end of the year, a mere 4 months after our initial estimate. There was no way I was going to let this date slip again.

Ben was still working on optimising the Lua execution and audio (which at this stage was still taking 4 or 5 ms per frame with peaks way higher than that). I’d thought that as Vessel used FMOD that it would all just work when I turned it on. Unfortunately, the audio was a complete mess on the PS3. While profiling the game thread in detail we discovered that it was taking up huge amounts of time, spiking and causing frame stutters as well as just plain behaving badly. It took weeks of intermittent work plus lots of discussions with the FMOD boys to figure out what the problems were. Turns out, many of the sounds were just set up incorrectly for the PS3 build – effects and layers that were disabled on PC were left on for PS3. Some sounds would loop infinitely and multiple sounds would play at the same time. All these things took time to discover, time to investigate a solution and time to fix. Something we had very little of.

To make things worse, Vessel performed no culling for the rendering, AI or audio. At all. The entire level was submitted to the renderer which culled it and then passed off the visible bits to the SPU renderer. Also, Lua scripts (which are slow at the best of times) were being called even when the things they were modifying weren’t in view. Audio was also being triggered by objects that were a long way from the player, relying on distance attenuation to reduce volume. All of this meant that there was a lot of work going on under the hood for no visible or audible impact.

We were still stumbling across weird little issues. For example, the animated textures in the game were never animating. They were loading correctly, but never changing. Ben tracked this down to an endian swap function which was broken (the PC and PS3 have different byte ordering for their numbers and so the same data when loaded from file must be modified depending on platform). This endian swap was only ever called in one place and only by this particular animated material.

The fix is often easy. Finding what to fix is hard.

November

Even though we had a tight timeline, we also had other client obligations which saw us each lose 2 weeks in November. This helped financially (well, it would have if they had paid on time) but put even more time pressure on us. Regardless, we were confident that we’d make the end of December deadline. Plenty of time.

We finally managed to get a complete play though of the game in November. Only to realise that the endgame didn’t work. Fix, patch, hack, test.

It was during the last week of November that Ben had a breakthrough. He bit the bullet and implemented a simple container based culling system for the game thread. All objects in a loaded level were placed within a rectangular container and there were an arbitrary number of these per level. This meant that we could quickly check what should be rendered or processed (Lua or audio) by looking at the container in view and those next to it. Conceptually simple, but implementation required that we modify the editor (which we’d not done before) and renderer, then visit each level in the editor and manually add the new containers then re-export everything.

This fix made a massive difference to performance. Audio was faster as it wasn’t playing as many sounds. There was less Lua processing happening as there were fewer things active. And rendering was faster as there was less being sent to the render thread. So it worked, and it worked well. In fact, we had to limit the frame rate to 30fps as some places were now rendering at 60fps on all threads!

The container fix was instrumental in getting the game thread to a more playable speed, but it also introduced a lot of little bugs that took a month to fix. And still, it was too slow.

December

With only three weeks until the Christmas break we figured we needed to start cutting back on the content in some of the levels in order to get the frame rate up. We picked a few of the worst and closely examined them to try and figure out how we could speed them up. We found the following;

Trees were very costly (as were their roots). So we cut some of them down.

Drains were expensive, so we removed some of them.

Areas with lots of water were obviously expensive, so we reduced the amount of water in some areas, cutting back on the flow or adding drains to reduce the amount of water in pools.

We tweaked the object limiter code to ensure that there were always enough fluros to finish a given level, yet not too many to make it run too slow. Same with the number of drops and seeds.

All of the above helped, and the game was now running pretty well. There were still areas where it would slow down, and you could (and still can) slow down some areas intentionally by spraying lots of water and generating lots of fluros, but there was no time left for further optimisations. I’d already built 13 different SPU tasks to speed the physics up and one or two for the render thread – it was getting very hard to speed things up more, and at this stage it was risky to make any extra significant changes. This would just have to do.

James was now noticing some play specific bugs. Some fluros weren’t moving correctly – jumping too high and destroying themselves on the scenery or just behaving badly. Which is fine in most cases, but this was happening with a key fluro that you needed to use to complete the level. We had to modify the level to make it work again.

In addition to the optimisations that we were still implementing, and the bug fixes for the odd things that were happening, we also had to make sure that the game was TRC compliant. James trawled through that massive missive, highlighting things that we needed to check – the ways that save/load games functioned, how the game shut down, what would happen in the case of no hard drive space left, the order that trophies unlock, so many things. And so little time.

And, on top of that, the financial pressure on the company due to the length of time that Vessel was taking to port, plus the reduced pay rate we were getting for the overage and the fact that there was very little work lined up for the new year meant that I had to notify Ben that we were going to have to let him go.

It was one of the hardest things I’ve ever had to do. I felt like I’d betrayed him – he is an awesome coder and he’d become a good friend. And just before Xmas FFS. To make things worse, the day I had to do it he was working from home and I had to let him know over Skype. It was just awful.

He called me back within a couple of hours to tell me he had just managed to land a new job. In just two hours. I told you he was good.

So, back to the code. With a week and a bit to go we mainly focussed on the remaining TRC issues. The game was running as good as we were going to get it to in the time we had and I was satisfied with that. We weren’t hitting the full frame rate on all the levels or in all cases, but it would just have to do. TRC violations were disappearing and it looked like we might just make it.

In the last week (on the Wednesday) I realised that the file format that we’re using needed to be DRM enabled in order to be TRC compliant, so I spent 17 hours straight trying to fix it. Did my first all nighter in a long time and managed to solve the problem at 5am. I pushed through the next day, fixing, tweaking, hacking, testing while full of sugar and caffeine and then dragged myself home to sleep on the Thursday night.

Submission Time

The final Friday arrived. We performed some clean ups and readied the code for submission. We were going to make it! On one last pass over the TRC checklist I realised that we’d missed something. We needed to be able to report an error if any of the required startup files were missing, but the way that the engine was designed, there was no way to display anything until those critical files were loaded. We had a few hours so I quickly hacked together an emergency renderer that I could use in the case of a critical failure. I gave myself 2 hours to do it and it was working within 30 minutes. Awesome! But, upon further testing, the loading screens refused to work. I still have no idea why – the code I added was never executed and yet still affected the loading screens. The game itself played fine, but those bloody loading screens didn’t bloody work. I wasn’t willing to submit with what I knew would be a TRC failure so, once again, we’d missed our deadline.

We’d have to delay submission until next year.

To be continued…

Vessel Post Mortem: Part 3

After the Christmas break (where I was forced into yet another family vacation) I dove straight back into the code, hoping to get the final issue fixed and the game submitted as soon as possible. There was still time to submit and if I passed Sony’s submission process first time then we could still have the game on the PSN store by the end of January. I fixed the remaining problem with the start up error reporting and added in a little more error checking just to be sure. I was pretty paranoid about the submission process as I didn’t want to delay the game anymore by failing – I carefully checked everything twice, crossed my fingers and hit the ‘submit’ button to Sony America (SCEA) on January 9.

In parallel to the code submission process there is a ‘Metadata’ submission process. This corresponds to the PlayStation Store assets and the game’s presence there. It consists of all the text, images, trailers, prices etc and they all have very specific requirements that must be met in order to pass submission. James (our QA guy) managed most of this and involved communicating with Strange Loop’s art guy Milenko (who was incredibly responsive – I’m not sure he ever sleeps) and consisted of asking for different resolution images and screenshots, as well as sourcing the different language versions of the game text. It took us a few submissions of the meta data to get it all right, but the turnaround was pretty quick and a continuous dialog with the Sony helped a lot.

The code submission process consists of uploading the game in a special package format plus some extra documents that describe the trophies and other bits and pieces. We had to submit to both SCEA and Sony Europe (SCEE) so we could have the game released in both those regions. We hadn’t submitted to SCEE at the same time as SCEA as we were still waiting on some publisher registration details to come through, so all I could do was wait for that and for the response from SCEA on our initial submission while I busied myself with some other work.

On January 18th I received the first fail from Sony. As fails go, it was a pretty good one. There were three “Must Fix” bugs: one was due to my entering the wrong data in a field when submitting the game package, and two were due to saving games failing when there was no disk space left. They’d also mentioned some slowdown in some of the levels – I’d expected that, and as it wasn’t a critical bug, I ignored it. The save game bugs proved troublesome – Ben had written all of the save game code and with him gone I now had to learn how Sony wanted you to do it, how Ben had done it and how I could make it work correctly. It took me a few days to find and fix the problems and by this time the details that SCEE required had come through so I resubmitted to SCEE and SCEA at the same time (January 24)

I was quietly confident that it’d all pass this time. I’d thoroughly tested the save game code now and it all worked. What could go wrong? I seriously considered heading out and buying a bottle of Chimay Blue as a pre-emptive celebratory reward.

The first fail I saw came back from SCEE on February 2nd. I’d messed up the submission data again (I hate filling out forms) plus there was a legitimate bug they’d found where once you’d finished the game you couldn’t continue on from that save game if you wanted to continue playing to complete all the trophies. When I had fixed that I re-submitted the new build to SCEE on February 4th and then began to wonder what had happened to the SCEA build – it should have come back at about the same time as the SCEE one. I also worried that they would pass it with the “continue when finished” bug in it. I needn’t have worried, as the SCEA team came up with a whole slew of new bugs that weren’t mentioned in any of the previous tests.

This new bug report that SCEA had sent back was very concerning. The crashes they were reporting sounded like bugs I’d previously fixed – I was also impressed with their thoroughness. One of the bugs was something like “Hang from the rope in level blah and throw seeds for at least 5 minutes into an area with no water. Crashes 70% of the time”.  While looking at this bug I received notice from SCEE that they had passed Vessel. Which was great, but it was also worrying as I now had numerous repeatable crashes in the build – so I failed the SCEE build myself in order to be able to submit a fixed build which would match the SCEA build.

By now I was getting a little frazzled. It was like one of those movies where the protagonist has slain the evil uber zombie, only to have it rise from the dead over and over again. Would this submission process never end?

I eventually tracked the crash bugs down to a simple stupid error. There was a section of code that was responsible for limiting the amount of fluid, fluros and seeds in a given section. When the frame rate dropped below 60fps, this code would kick in and drop the number of drops/fluros or seeds to the minimum value deemed suitable for that level. During the final stages of development we had created a FINAL_RELEASE mode which turned off all of the unnecessary debugging code. Unfortunately, this erroneously included some code which updated the time values that the limiting code used, so the fluid, fluros and seeds were never being reduced. Ever. This meant that the game would have been running much slower than it should have, and was prone to crashes whenever certain hard coded limits were exceeded. I’ve never been so relieved to fail anything.

Something I’d like to focus on here is the level of performance that Vessel was submitted with; Ben and I spent a huge amount of our time just trying to squeeze as much performance out of this game as possible on the platform we had. For the most part, I’m pretty happy with the results. Yes, you can abuse the game and make it slow down quite easily – if you try to fill a room with fluros and water and seeds then you’ll most likely see some slowdown – but during normal play, the game maintains 60fps simulation at 30fps frame rate (the graphics is 30fps, so there are two simulation passes per visible frame). However there are a few levels where the sheer number of fluoros and fluid required to solve the puzzles means that the frame rate will drop to 20fps. Given another month I reckon I could have fixed that too, but given how far over time and budget we were, that wasn’t an option. So, while I’m happy with how much we improved the frame rate of the game, I’m still a little disappointed that we didn’t hit a full 60fps simulation everywhere.

Sure, the PlayStation 3 hardware did slow down the porting of the game initially, but given the complexity of the fluid simulation required I doubt that there will ever be an Xbox 360 version released. The SPUs allowed us to perform at least 80% of the physics simulation completely in parallel – with no impact on the main processor at all. There is no way you’d get the same level of performance on the X360.

With the FINAL_RELEASE bug fixed and the game resubmitted to SCEE and SCEA (which passed on the 20th and 22nd respectively) I was finally free! The metadata was all sorted and we’re now expecting the release on March 11 in the SCEA territories and March 12 in the SCEE regions. We’re all hoping it does well enough to cover our costs (as all devs with newborn progeny are).

What went right

Having a QA guy in house was invaluable. An impartial, non-technical third party to throw our fixes at kept us honest. As a developer you get very close to the game and like to assume that your fixes have made a positive difference and you’re moving forwards. That’s not always the case.

Honest communication with the Publisher/client. I tried to keep John at Strange Loop Games constantly up to date with the progress (or lack thereof) of the game. He was incredibly understanding with the continuous delays, and while it was hard to deliver bad news, at least it reduced the surprises.

Having good, experienced coders on the job. Bringing Ben onboard was a good choice, even though we had to let him go in the end. There is no way I could have done this without him. Working from the same physical office was also beneficial. I worked remotely for over 5 years leading up to this port, and I think we’d have delivered even later if we’d worked separately.

What went wrong

Work estimation. I fundamentally underestimated the amount of work required. I should have applied Yossarian’s Circular Estimation Conjecture and multiplied it by PI – that would have been closer to the mark. The big killer was the misunderstanding with which threads needed to run at 60fps. If it was just the physics I’d underestimated, we probably would have been a shade over a month late (maybe two), but with both the physics and game threads needing to execute in under 16.6ms, we really had our work cut out for us. The amount of time taken for submission shouldn’t be forgotten either; 4 to 8 weeks after the initial submission should see your game on the store.

I’d also recommend to anyone doing a console game that they get as much TRC compliance done as soon as possible in the development of the game. Get the load/save stuff working, trophies, error reporting, quitting, controller connection, file error reporting and the like in place and functioning sooner rather than later.

In Closing

The porting of this game was a trial for me. There was a week or two around September/October where I was really doubting that we could deliver a game that ran at a playable frame rate. I’m proud of what we delivered in the end, and if you’d taken away the financial and time pressures I would have thoroughly enjoyed the challenge. I’ve learnt a lot from this experience, and I’m looking forwards to the next one.

Just not right now, OK? Maybe a bit later in the year.(source:overbyte)


上一篇:

下一篇: