Why Industry Standard Game Metrics Don’t Matter (Enough)
by William Grosso
The world of free-to-play metrics can be very confusing, with literally hundreds of metrics out there, and literally thousands of introductory articles on the topic. Many developers largely give up tracking metrics at all, or simply stick with the 15 Metrics All Game Developers Should Know by Heart, using these instead of conducting a deeper, more qualitative analysis. Other developers obsess over their analytics, struggling to discern trends on a daily or even hourly basis. Given the natural variance in most free-to-play-games, that’s a lot like cutting open a chicken and reading the entrails in a vain attempt to figure out whether the game is going to prosper.
There’s a very good reason for this all confusion: The game industry has given us a complex metrics language that is almost entirely built out of alerts and benchmarks. It’s what all the game conference presentations are in. It’s what all the analytics packages measure.
And it’s not very useful.
Industry metrics don’t tell you anything about the state of your F2P game’s virtual economy. Consequently, there’s very little specific guidance out there which explains how to use metrics to make your game better, or goes beyond elementary advice such as “promote relevant offers”. Outside the giant publishers which have the resources to build their own in-house analytics system, this lack of clarity is hurting just about every developer, and hurting the industry as a whole.
Let me explain.
Quick Qualifier on Analytics
The previous paragraph might have sounded a bit dismissive. So let’s add a quick but important qualifier: A basic analytics package that can track standard metrics like ARPPU, conversion rates and all the other items I’m about to discuss is a worthwhile investment every developer should get. There’s a lot of analytics providers out there, most of which track these core metrics for free, or close to it.
While these entry level analytics won’t tell you much about the state of your virtual economy, they’re still important: You want to understand your games’ performance relative to other games, for one thing. For another, you can track internal goals for how many players you want and retention levels during a soft launch, and then better decide when it’s time to boost acquisition through through ad campaigns. Perhaps just as key, industry standard data will help your investors decide if your game is good enough to get more resources.
That said, let’s take three canonical metrics and look at how far they fall short:
Let’s Talk about ARPDAU
The Average Revenue Per Daily Active User (ARPDAU), is one of the most commonly discussed metrics in mobile/F2P games. And to be sure, it is a useful metric, allowing you to understand how your game performs on a daily basis, while also a key reference for tracking the before and after results of any user acquisition campaign.
Still, there’s a lot in there that ARPDAU doesn’t tell us: For instance, do the paying players on a given day purchase again? How much of this revenue is generated by advertising versus IAP, and how is this revenue distributed among your entire player base? ARPDAU doesn’t tell us that.
And what do you do when your ARPDAU drops? Well, you fix the game, I guess — but how, and at what point? If the drop wasn’t very big, you wait and see if the game recovers. If the drop was very big, you start looking for causes. And for most games, that’s usually a qualitative exercise, instead of a quantitative one.
How About ARPPU?
Average Revenue Per Paying User (ARPPU) measures only the subset of users who have completed a purchase in a game. For that reason, it’s somewhat more useful than the aforementioned ARPDAU. Once again, however, ARPPU has limited use, because it can vary dramatically based on game genre. Hardcore and midcore games (primarily strategy, RPG, and shooters) tend to have much higher monetization metrics in regards to ARPPU, but they also lack the mass appeal of more casual games.
You can bet, for example, that ARPPU rates for Game of War: Fire Age are much, much higher than Candy Crush, even though both games earned about as much revenue last year.
Okay, But Surely Conversion Rates Are Valuable?
While ARPDAU and ARPPU give you (limited) insights into how much money your monetized players are paying you, the conversion rate measures the total percentage of unique users who have made a purchase out of the total number of users during a given period. (You can also measure the conversion rate of ads served in a free-to-play game.) Getting users to pay real money in a game that they can play for free is a difficult assignment, and if you can earn conversion rates over 5%, you are doing extremely well. So this metric is a good benchmark for how well your F2P game is doing on market.
But once again, conversion rates are not a be-all, end-all metric. As with many other industries, repeat purchasers generate the majority of revenue in free-to-play games. You could have 5% percent of your players making an IAP one month, and declare victory — only to watch, crestfallen, as these same players never again make another purchase.
In the end, all three are largely vanity metrics — the kind that developers humblebrag about after a few drinks during a GDC party. They might tell you whether to acquire users or not, but they don’t point to any fixable issues in your game. Changes in them signal that something is wrong (or right), but they don’t give any hints as to what.
Even worse, while ARPDAU, ARPPU and Conversion are all financial metrics, they don’t even give you a clear picture of how players are engaging with your in-game economy, what they’re buying within it, and whether they’re likely to buy again. That’s the key to increasing your revenue, and knowing your ARPDAU, ARPPU, and conversion rates won’t tell you much about this. The same can be said of funnel metrics, which may be essential for marketing, and measure where users falls off the train, but still don’t give you sufficient diagnostics to help explain why.
Why the Game Metrics We Use Are So Limited
How did we end up with such limited metrics to guide us? That takes a bit more explanation, because we first need to think about metrics in terms of two axes:
Game-specific versus generic: The former relate only to a particular game or clearly defined sub-genre of game, while the latter apply to all games and even to non-game apps.
Alerting versus Benchmarking versus Diagnoses versus Control: The standard metrics like ARPPU, ARPDAU, and Conversion fall into both the Alerting and Benchmarking categories, telling developers how well their game is monetizing in relation to industry standards. Funnel metrics fall into the Diagnostic category, pointing out problem points in the user experience, while Control metrics are the most useful at all, giving you actionable insights on specific interactions. I believe this is where the problem arises.
As this chart above hopefully suggests, it’s very difficult and/or time-consuming for any particular developer to discuss the metrics which matter most in their particular game, and even when they can (say, at an in-depth GDC talk), they worry that they’re giving away too much special sauce recipe to potential competitors. Alternately, they’re often too embarrassed to acknowledge bad metrics. At the same time, the Flurries and App Annies of the world are able to run successful services which focus on the top level alerting and benchmarking stats that work best for public consumption.
The upshot? We wind up talking about the public metrics and cite the companies made to measure them, and we maintain an entire industry around both. And while developers can add custom events in many analytics services, there’s no guidance for what to add, because that falls outside the public conversation. And if you think about how you’re going to talk about your game to your CEO, or to outside parties, you have to speak in terms of generic metrics.
But these are not control metrics, the kind of data developers need most to survive and possibly thrive. And that’s the ultimate problem. We have a complex metrics language that is almost built entirely out of alerts and benchmarks.
Toward Control-Based Game Metrics
What our industry needs is a public conversation around the metrics that matter most to the success of our games, and some consensus around them.
We’re starting to see analytics/monetizations services work on creating these. For instance, see this excellent DeltaDNA case study for Thumbspire, a F2P publisher. They measured Day 1 retention against industry benchmarks, found them lacking, so then went on to do a qualitative analysis (presumably guided by event logs). This showed them that 20% of players dropped out during the tutorial, which pointed them to improving this for first-time users. (Which improved D1 retention by 66%.) Perhaps, therefore, we should make a metric like “Tutorial Abandonment” as standard and widely-cited as conversion rates?
In my next Gamasutra post, I’ll start to propose some other metrics worth standardizing, particularly those related to revenue. As a preview, here’s some reliable KPIs my team and I have added to our own tracking system, which customizable analytics services on the market can also measure:
Percent Of Spend by End of Day One: I.E. the % of users who complete a purchase within one day of downloading the app (out of the total number of users who download the app on that same day). The specific percentage rate varies quite a bit depending on the game and the genre, of course, but a relatively high percentage means the game is connecting with an audience. After pushing an update to your game, substantial changes to this D1 % spend immediately alert you to how well (or not) those changes are working with your userbase.
Percent of Spenders Who Buy a Second Time: You want at least 40% of your paying players going on to make a second IAP — a strong indicator that they enjoyed the first purchase, found it valuable, and wanted more. If this number is significantly less than 20%, you should worry that these first time purchasers might have (for example) felt cheated by the experience, or realized that your game’s free daily rewards and other free giveaways are sufficient for play.
Time Until Second Purchase: Most purchases should occur within two weeks of one another. If the number is substantially higher than 14 days, your game probably provides too little incentive for the users to make subsequent purchases. As with D1 % spend, this two week rule of thumb can vary somewhat by genre, since successful core/mid-core games typically enjoy extended gameplay over many months, while more casual games on average have a shorter shelf life.(source:gamasutra)