360 achievements unlock developer tips, DLC dates
GDC 2010: Researcher Bruce Phillips' detailed study of 18,000 players' habits, multiplayer usage, and achievement acquisition is helping Microsoft Game Studios make better games.
Who Was There: Bruce Phillips (User Researcher, Microsoft).
What They Talked About: Many people take their Xbox 360 achievements seriously. Nobody takes them more seriously than Bruce Phillips--because studying achievements is part of his job. As head of Microsoft's player-behavior research project, the Ph.D. in psychology spends his time studying how players play games on Xbox Live. By looking at how--and when--thousands of gamers rack up achievements, choose multiplayer modes, and slog through single-player campaigns, Microsoft is collecting information it hopes will help developers make better games.
In return, Microsoft gets data--mountains of it. Phillips' team knows every single Panel member's achievement record, which lets them track all manner of player behavior. They know when Panel members stop playing a game, they know how many multiplayer hours they've logged, they know when they're online, and they know when they're offline. The one thing they don't know is how Panel members interact with their friends, since that would violate the privacy of said friends who are not Panel members.
One might think all this data is being monitored via some sinister, supersecret, WOPR-like supcomputer that acquires it through some secret back door in Xbox Live. That would be wrong, according to Phillips. He said the information is collected via HTML parsing of public gamertag profiles on Xbox.com and via an XML feed provided by the Microsoft Community Developer Program. Though it's not open to the public, any developer can apply to get access to the tool to access the XML feed, and Phillips said third-party publishers also track data from there. The applications collect data every 15 minutes, and data of online players every five minutes.
Phillips said his first job after setting up the Panel was to make sure its data was accurate. So he compared its general gameplay statistics to those of Xbox Live, which boasts more than 23 million users. Both the Panel and overall XBL user base played the same games in the same amounts, with popularity bar graphs of the two being nearly identical. Phillips showed off a chart of achievement completion rates, which also showed the two in sync.
SINGLE-PLAYER STATS & GAME COMPLETION
Confident that the Panel data was statistically sound, Phillips and his team began to use it to answer questions about gamers' habits. The first metric was game completion. Using achievements as markers, Microsoft can determine how far members of its Panel get in a single-player campaign before quitting--and what difficulty they played the game at. By finding out how much of a game consumers play until they quit, the company hopes it can isolate the issue that caused players to stop--and keep it out of any future Microsoft Game Studios titles.
As an example, Phillips showed the completion rates of two games--Halo 3 and Battlefield: Bad Company. His team measured the number of players who earned the first-chapter campaign-completion achievement in each game and how many completed the game. Some 93.22 percent of Halo 3 players finished the first chapter on normal difficulty, compared to 71.82 percent who finished the campaign.
By contrast, Battlefield: Bad Company saw only 39 percent of players finish the campaign in its entirety. He showed that each campaign-completion achievement saw a marked drop in continuing players--the first chapter was 70 percent, the second was 58 percent, and the third was 51 percent. "That's a very steep drop-off early on," he said. Phillips said Rainbow Six Vegas 2 had an even higher rate of attrition.
Microsoft can also track players' preferences via the achievement meta-data included with achievements. They can tell if the achievement is awarded due to completion, collection, questing, or gameplay. This also helps their in-house development team figure out what kinds of missions to put into games, since they can see which are the most popular.
Another graph presented by Phillips showed how a game--in this case Call of Duty: World at War--is played for the first 30 days. He presented a graph that showed that virtually all single-player achievements are earned in the first four to five days after a game is first played. After that, Zombie mode picked up speed, leaving co-op campaign a distant fourth after multiplayer.
MULTIPLAYER UNDER A MICROSCOPE
Unfortunately for single-player purists, by far the most popular game modes are multiplayer. Phillips and his team track the Panel's online play with a metaphorical microscope, tracking the popularity of each game's various modes and how they are used over time. Team Deathmatch dwarfed all other multiplayer modes for Call of Duty: World at War, with the others changing places at the bottom as people would try out different modes.
Microsoft also measures which multiplayer maps the Panel plays. Phillips displayed a dizzyingly detailed chart that showed what percentage of the Panel played particular World at War maps, when they played them, and in which modes they played them.
The rate at which achievements are earned is also closely studied. For Modern Warfare 2 multiplayer, people earn about five achievements on average in the first 30 minutes, but then it levels off, with five achievements earned every 1.5 hours. Phillips said Modern Warfare 2 was an example of a game that did a good job rewarding players for continuing to play by continuing to dole out achievements, albeit at a slower rate.
"GAME TENURE" & DATING DLC
One of the most important concepts Phillips and his team have come up with by studying achievement data is "game tenure." This is the duration of time between the day players first fire up a game in their Xbox and the last time they play it.
The vast majority of game tenure loss happens in the first few days, due to people not liking the game or renting the game. Then it flattens out, since people keep coming back to play it over and over again. One of the games with the best tenure was Rock Band 2, and another was Call of Duty: World at War, with a whopping 132 days. The game with the highest game tenure? Geometry Wars: Retro Evolved, which players are still frequently playing nearly four and a half years after its release.
Why is game tenure important? Because according to Phillips' research, DLC is more successful when it is released while players are still actively playing a game. That's why so many DLC packs are being released within weeks, days, or even hours of a game's launch. A good recent example of this (not provided by Phillips) is Dragon Age, which had DLC on day one and a full-fledged expansion, Awakening, just four months after the game hit the market. (A sequel also appears to be scheduled less than a year and a half after the original debuted.)
WHY?
Though Phillips' team definitely knows how to detect patterns of behavior from Xbox Live data, its ultimate job is summed up in a simple question: "Why?" Why do players stop playing when they do? Why do they prefer one multiplayer mode over another? Why will they spend hours pursuing one achievement but leave others untouched?
To answer those questions, the Panel is bombarded with surveys about their behavior. These surveys are sent out automatically and correspond to behavior triggers in a game. Most interestingly, they can be sent out days, weeks, or months after the event occurred, allowing developers at Microsoft Game Studios to pick the brains of players while fashioning a sequel or DLC for a particular title.
Quote: "Now, for the cool stuff."--Phillips, showing a slide of Arthur Fonzarelli from Happy Days.
Takeaway: Many people consider achievements "nerd cred." Microsoft and other publishers consider them clues to be closely studied on how to make a better game.
Got a news tip or want to contact us directly? Email news@gamespot.com
Join the conversation