Against Statistical Design

Posted on June 9, 2011 by rewardingplay

Statistical Design

I suggested that a good way of improving one’s design sense is by staring at Rorschach Tests, and here is a practical example of the importance of practicing pattern-avoidance.

To me it looks like a designer's brain in a vice

Stop seeing patterns!

This image is a heatmap showing where people most often die on Assembly, a Halo multiplayer map. These heatmaps were first used by the Halo design team to analyze maps during testing, but were so interesting looking they became part of the bungie.net statistics pages. This data is so rich — so detailed and specific — it must be useful to a designer in some way, right? The problem-loving brain of the game designer latches on to this as The Solution and immediately starts searching for The Problem. It is tempting, given a powerful tool like statistical analysis, to incorporate it into the design process somehow — especially since design is often stranded in a world of abstraction and uncertainty. Having concrete numbers is a rare treat.

However, what does this data mean? Are red areas bad? Should dark areas be eliminated? Does a well-designed multiplayer map have a symmetrical shape? What percentage of a map should be yellow? Something about high-contrast feels unbalanced, so perhaps the map should be revised so that the gradient from safe to dangerous is more continuous. And areas where nobody dies seem wasteful, maybe they should be removed. And obviously the red areas will be frustrating, so they should be made safer by limiting line-of-sight and adding cover. Pretty soon we have a completely yellow multiplayer map, that we have tricked ourselves into believing is balanced because our data looks pretty. We have fallen victim to statistical design.

Players Aren’t Statistical

Statistics are powerful tools because they aggregate a large number of unique instances into a manageable form so it can be analyzed. It would be impossible to watch every death of every player across thousands of games and have any cohesive understanding of how often players were dying in a given area. Given enough examples, we would develop an emotional feeling of dread or security associated with certain spots, but the brain uses a very unscientific method to determine these attachments. Exciting experiences are weighted much too heavily, which is why the impartiality of statistics is useful in discovering imbalances. Using statistics to find problems is fine; designers go wrong when they use statistics to evaluate solutions.

Players don’t engage with the game statistically — they experience it personally. It doesn’t matter if more players are killed standing in a specific spot than anywhere else on the map, what matters is the unique experience of a player killed in that spot. If they realize that they shouldn’t have crested the hill with no cover that is right below where the Sniper Rifle spawns, vow not to do that again and move on, there is nothing wrong with the map. Even if they do it over and over, growing more and more frustrated at their repeated mistake and creating a bright red dot on the heatmap, the map is not unbalanced. However, if players are forced to expose themselves at a single chokepoint, or get sniped through a hidden line-of-sight in an otherwise safe area, it doesn’t matter if it is a rare experience and there is no red, the map ought to be fixed. Neither of these situations can be found through statistical analysis, and neither of them are fixed by a solution that merely addresses the probability of being killed in a given area.

Avoiding Statistical Design

Some systems can only be balanced statistically. If there are three factions in the game, and one faction wins 43% of the time, the factions are not balanced. If a map is intended to be used for two-flag CTF, but the bases aren’t mirror images of one another, then the two sides had better be perfectly fair. The necessity of reverting to statistical methods is inherent in the design of the system itself. The designer will be forced to make changes that do not change the unique player experience — or may even harm it– in order to fix a statistical imbalance. Worse still, players are skilled at detecting when a system must be balanced statistically, but since they do not have access to hard numbers their personal experience will tell them that it isn’t balanced — even when the data says that it is!

Nerf Paladin?

Well-designed systems do not need to be balanced through data-manipulation. If there are 10 weapons in the game, and one weapon is responsible for 20% of the kills, there is probably not a problem. If the unique player experience isn’t negatively impacted, the statistical difference isn’t a balance issue. So, the easiest way to avoid the trap of statistical design is to avoid systems that must be balanced mathematically in favor of those that can be balanced behaviorally. If a system requires a large amount of instrumentation and is extremely sensitive to tiny value changes, instead of obsessing over statistical patterns, try revisiting the system’s design and making it less brittle.

He's about to have an experience he won't forget!

Put to the Question

Posted on May 25, 2011 by rewardingplay

If you want to learn something, read about it. If you want to understand something, write about it. If you want to master something, teach it.

– Yogi Bhajan

Successful designers are those that have the discipline to edit their work. Generating gameplay ideas is exciting and easy; discerning those with potential and removing the rest takes resolve and vision. But you’ll never get where you are going if you walk down every road — you must cut!

Removing game elements is usually straightforward. They are almost always part of a collection of similar elements, so they can be compared on an apples-to-apples basis. They are often specific instances of a general system — the Assault Rifle is an instance of the Weapon system, the Goblin is an instance of the enemy AI systems — which means that most of the work that goes into them can be applied to other elements and isn’t wasted. An element is associated with a certain amount of work that must be scheduled for the models, textures, effects, sounds and animations, so it is easy to measure its impact on the overall scope. And finally, since an element is by definition discrete, it can usually be removed with virtually no impact on the rest of the game.

Some elements just don't fit in

“Is this mechanic going to be fun?” is a good question, but impossible to answer. Prototyping can help, but it still takes discernment to recognize the potential in a prototype. It also represents an investment of resources, which can bias the team toward keeping a prototyped mechanic that ought to be cut. “Can we implement this mechanic?” is also a good question, but “can” is not “should” and “implemented” is not “tuned”. “Has a mechanic appeared in another game?” is a useful question, but isn’t necessarily relevant to the current game. “Does everyone agree this mechanic is good?” is a seductive, but destructive, question; it replaces a designer’s instinct with general consensus and will lead to “downhill design” where the only possible solutions are the easiest ones. The single best question for determining the potential of a mechanic is “Can the player be taught to enjoy this mechanic?”

Not everything is teachable

Answered honestly, this question has far-reaching implications. If a player never uses a mechanic, it might as well not exist in the first place. If a player fails to realize that a mechanic exists, it is too subtle or the game is overly complicated. If a player refuses to learn a mechanic, it probably doesn’t fulfill a fundamental aspiration for them — they will never be interested. If a player is unable to learn a mechanic, it is probably unintuitive or it clashes with other aspects of the game. If the designer cannot distill a mechanic down to teach it, they likely don’t understand the mechanic themselves. If a mechanic can only be learned through explicit tutorial, it probably has an awkward or obscure control scheme. Any of these problems are fundamental enough to warrant removal.

On the other hand, if a mechanic can be taught effortlessly — or better yet, players discover it on their own — or if mere awareness that an experience is possible is enough to motivate them to learn how to access it, this is a sign that a mechanic is reaching them on a subconscious level. This kind of mechanic can be integrated into their thought process and lead to a fluid flow experience.

GDC 2010: Design in Detail XV

Posted on May 20, 2011 by rewardingplay

Without anyone getting kicked in the face…

You always need to listen when people don’t like something. You are too close to the game; You probably already fixed all the things you didn’t like, so you should value a fresh perspective. Keep in mind that you can always trust someone’s emotional reactions, they are always authentic and valuable, but never just blindly take their advice. The designer’s job is to separate emotional feedback from thoughtful suggestions and treat the appropriately.

Before you can interpret someone’s feedback, you need to understand the source. Feedback means “the game in my head is different” and often times your response to feedback should be to probe about what kind of game they are imagining. You don’t necessarily need to agree on the game you are making to benefit from their feedback; they probably represent some portion of your audience.

You see Development Bias a lot with the public when the development process is very open. Playtesters know the game isn’t finished, they know you expect them to provide constructive criticism, so they become a lot more sensitive and more likely to complain. Once the game is on the shelves, those small problems fade into the background and players rarely notice them.

You also need to understand the source of feedback; If you can categorize someone’s play style, it will help you understand how to react to their feedback. You can weight their comments appropriately.
Here are some examples:
(The names have been changed to protect the guilty)

I used to balance “Easy” by playing with my nose (true story) but Steve still couldn’t beat it. I miss that guy, he was incredibly useful for balancing.

Even more important than categorizing other players, you need to understand your own playstyle. For instance, I’m a “role-player”, so I tend to ignore small balance problems if the results are still dramatic. I have to recruit “pros” that are more sensitive to useless or underpowered elements.

It’s a 7.5

Posted on March 23, 2011 by rewardingplay

Uncharted 2

I recently finished Uncharted 2. I thought the acting and the pacing were excellent. The story followed a well-worn arc, but had enough unexpected set pieces to keep it fresh. The combat was dramatically improved over the first game, but still didn’t come together enough to justify the amount of fighting required. The stealth segments swerved between fantastic and frustrating, often clashing with the combat and story elements. (We can’t shoot the museum guards, but we can throw them off a roof onto paving stones five stories below?) I loved the sections where I was working with another character to solve puzzles or fight through enemy territory, partially because I know how difficult it is to do useful AI allies.

Maybe "ally" is the wrong word

My main criticism of the game is that it is crushingly linear. There is only one path, straying from it usually ends in a fail screen, and I could rarely anticipate what was going to happen. I just followed the breadcrumbs and assumed the level designer knew what they were doing. On several occasions the game goes beyond leading the player by the nose and starts dragging them by the hair. If the story required a pistol with infinite ammo, or a checkpoint to skip me through a tough section, or a hint that told me how to solve a puzzle I hadn’t even seen yet, then gameplay took a backseat.

All in all, a new high-bar for storytelling in games and a fantastic experience. I give it a 7.5! (Which according to IGN means “there are some issues, maybe it lacks ambition or it is repetitive or has too many technical glitches, but I had fun playing it nonetheless and think you will too.”)

Jumper

I also played a game based on the 2008 movie where Anikin Skywalker learns to teleport and Mace Windu tries to kill him… or something. The control scheme is unique. Instead of traditional brawler controls, each of the four face buttons represents a cardinal direction and tapping it will teleport you to that side of the targeted enemy and execute a melee attack. By using alternating buttons, you can “jump” behind an enemy, kick him into the air, and then “jump” to where he will land and hit him again from the other side. The animations are stellar, especially when you start chaining combos and beating the hair-dye out of the chumps trying to capture you.

It's a Tasersword (tm)

They use the main character’s signature ability in lots of clever ways. You can use it to travel through walls and find secret rooms or solve puzzles. There’s a finishing move where you grab an enemy and teleport you both to some remote location like a mile above the Grand Canyon or the middle of Antarctica and leave them there to die. (Of course, this makes it ridiculous when they use the tired trope of locking you in a room until all the enemies are dead. Why can’t I just teleport through the door?) There are even some enemies that are protected from certain directions, requiring you to time your attacks from different directions or risk being blocked.

A mercifully short campaign featuring unusual and well-realized combat mechanics. I give it a 7.5!

The Game Designer Review Scale

Did I just say that some budget movie tie-in schlock is just as good as the highest rated game of 2009? What kind of rating system would result in giving them both a 7.5? What are you doing reviewing games anyway, isn’t this a game design blog?! The answer is simple. In the Game Designer Review Scale every game gets a 7.5. From the latest AAA masterpiece to the quirky indie game to the bug-ridden RPG epic. Because for the game designer the goal is not to compare or judge the games themselves, but to exercise and strengthen your conscious control of your own experience.

Your own games get a 6.5

At the beginning of a project, when you are prototyping a new game mechanic, you are not going to have a polished, tuned experience. It’s going to be noisy and buggy and awkward. You are going to need the ability to spot the glimpses of fun, no matter how obscured or faint, even if they only exist for a few seconds at a time. You need to lower your flow barrier, learn to ignore distractions and technical errors, to focus in on fun gameplay instantly before it slips away. You need to spontaneously create a polished form of the game through imagination and mental tricks like making your own sound effects and storylines. All so you can snatch up those seeds and grow them until everyone on the team can see them.

You can practice this form of autoregulation by playing a deeply flawed game as if it were a 7.5.

On the other hand, at the end of the production cycle, when the game is smooth and playing it has become effortless and comfortable, you are going to need to look at it with fresh, critical eyes. You need to raise your flow barrier and become oversensitive and harsh. Every tiny flaw needs to become a glaring blight that must be fixed. Every inconsistency or imbalance, no matter how trivial, needs to break you out of the experience. You need to be able to put yourself in the shoes of a new player and find every confusing control mapping or unclear mission objective.

And you can practice by playing a truly exceptional game and fixating on the problems until it’s just a 7.5. You aren’t flattening all games to the same level, you are changing yourself and controlling your experience to learn the most possible.

Definition: Affordance

Posted on January 3, 2011 by rewardingplay

Affordance (Also: Usability, Discoverability, Intuitiveness)

The quality of an object or environment that allows a Player to intuitively discern and perform the gameplay action associated with that object or environment.

In his most profound philosophical work Being and Time, Martin Heidegger makes a distinction between two types of attitudes that we can have toward an object. First, an object can be “present-to-hand”, which means that it exists and we can observe it and theorize about it. Heidegger claims that this is an uncomfortable mode for us, that it is inferior to the more natural second attitude where an object has an immediate purpose, which he calls being “ready-to-hand”.

Imagine you are walking through a Home Depot and see a collection of hammers hanging from pegboard in the tool section. Let’s say you are an English major and you have never seen a hammer before. You might assume that you had stumbled into an unusual art gallery and start admiring the variety of colors and shapes the artist had created. Clearly they are a phallic representation of our patriarchal history…

In this example, you would be treating the hammers as if they were “present-at-hand”. In a game, every time the player is forced to stop and think about the possible actions an object affords they are forced out of their flow state. If this happens too often they will never be able to relax and enjoy the game.

Chess - A terrible example of percieved affordance

Now imagine you are in a burning building and the only exit is blocked by a flimsy plywood door. This time when you see the hammer on the wall you do not perceive it as an object, but as a tool with a clear purpose. There is no hesitation because there is no analysis. The hammer is “ready-at-hand” and that door is “ready-to-smash”. There is no break in flow because both the hammer and the exit door have clearly perceived affordances.

This is especially important in games, because the player cannot even begin to play until they understand what actions they can take. The faster they can understand what is possible, the earlier they can get past the theorizing state and into the flow state of proper play.

Rewarding Play

by Jaime Griesemer

Tag Archives: usability