Perceived Simultaneity Threshold: 100 milliseconds
Let me tell you how I got my nickname, Jaime “Three Frames” Griesemer.
Emily "Three Frames" Crazyhair
We were working on the melee animations for Halo 3, trying to get the timing right. There were already fast. Like, blink-and-you’ll-miss-it speed. The animators were having trouble because there wasn’t enough time to add a sense of anticipation and sell the impact properly. And for some unknown reason I wanted the damage keyframe to happen faster. “The melee animations can be as long as they need to be… as long as they’re exactly three frames.” We went back and forth about it, but eventually I traded three frame melee attacks for some extra flourish on the recovery after the hit. Everybody was happy.
Until the next day when I went back to try the new melee animations. I picked up the Assault Rifle. I pressed the B button. Three frames later, the butt of the rifle crunched into the face of a helpless alien invader. I picked up the Plasma Pistol. I pressed the B button. Three frames later, my left hook connected with a random marine that was looking at me funny. I picked up a shotgun. I pressed the B button. And for some unknown reason it didn’t feel right when I cracked open the head of a passing parasite. “Wait. That wasn’t three frames. Yeah, I’m sure, check the file. See… four frames!” (In hindsight, I’m lucky my uncompromising design demands didn’t earn me anything worse than a mocking moniker.)
What was the unknown reason? How could I tell the difference of only a single frame? Why did I insist on the speed at the expense of animation quality? The answer lies in our physiological limitations. What we experience as consciousness — as a stream of consecutive events in a clear and tidy order — is not nearly so neat. It’s a cacophony. A discordant mess. Your senses collect an incredible amount of information; they’re always on, always filling your nervous system with an unrelenting torrent of unfiltered data. It comes in at different speeds and at different times. The visual part is heavily processed, so it comes in late and is usually out-of-sync with the rest of your senses.
Your brain at work
Your brain is used to this. It is, in fact, extremely good at taking all that information and combining it into a coherent experience. So good, it can take distinct events and merge them. Asynchronous events like pressing a button and seeing a melee attack. This feeling of simultaneity greatly increases the player’s connection with the action on-screen. Instead of a second-hand signal that transmits their intentions, pressing the B button becomes the act of dishing out a knuckle sandwich. But that requires your brain to blur over the delay and merge the movement of your thumb with the satisfying crunch.
Below a certain time threshold, the brain takes two events that are functionally separate — one causes the other and could not happen at the same time — and gives the impression that they were simultaneous. It turns out that threshold is around 100 milliseconds, or a little more than three frames. The reason four frames felt wrong was because the events were experienced as distinct, disassociated actions. The extra time allowed my brain to distinguish between them, in a sense inserting the controller as a step in the process and reminding me that I was not actually the one wearing metallic boxing gloves. So, for games where it is important to give the player a feeling of direct control, be careful of crossing the 100ms threshold.
Rewarding Play 
How did you eliminate the screen processing delay? Engines (buffers, vsync, graphics, component syncing) can easily eat 1-4 frames from controller press to feedback. I don’t think the button press is faster then a frame now either, with the wireless controllers and HD tvs, iirc. :(
Great post! Really enjoyed reading the paragraph on consciousness – if you haven’t heard of him already, you might want to look at Thomas Metzinger. He’s a German philosopher who mostly works in philosophy of mind, especially consciousness and self-models, and some of it directly relevant to game design. His book “The Ego Tunnel” is a good introduction to the sort of thing he does, or, if you want to plunge in head-first, author Peter Watts one wrote that Metzinger’s “Being No One” was the hardest thing he (Watts) had ever read. (I haven’t myself, so personally that title goes to my former professor William Wimsatt’s book, “Re-Engineering Philosophy for Limited Beings”, which let me tell you is quite the mindbender. :)
As for the point of the post, I am reminded of that chestnut of game design wisdom that says, in effect, “Don’t design to how it is, design to how it should be.” Such as, when Mario jumps, it’s rather different from how jumping works in reality. But if there were to be the (realistic) process of crouching then pushing off and then finally actually leaving the ground, those playing the game would howl over how unresponsive the controls were, and probably for the exact same 100ms reason.
@Dan Yeah, I fudged the numbers a mite to make a more coherent story. 100ms is an approximate number, not just because of additional sources of lag, but because different players have different thresholds for simultaneity. The nerve impulse from your finger is a little delayed, too.
But even with that, you do have to worry about image processing latency. Which is why game designers hate rendering methods that add delays. (And why Halo 3 didn’t have high-quality anti-aliasing, because it would have cost us a frame for the post-processing.) All you can do is talk to your rendering guys and stress the importance of every ms of delay between input and action on the screen. Several high-profile games have sacrificed controls for certain rendering effects, and when reviewers knock them for being “sluggish” or “having sloppy aiming” that is often what they are running into.
Wireless controllers have gotten pretty good, actually. The tech has come a long way since the Wavebird. It’s usually a negligible delay. TVs are worse, but also out of your control. I’ve seen several games reminding players to put their TVs into “Game Mode” which turns off the smoothing or whatever that is making the TV display a laggy image.
Killzone 2 was a good example of going whole hog into the mire of post-processing at the cost of the perception threshold that you’re talking about. The gains in doing so are immediately obvious, as it was a striking game to look at, but the disconnect hurt the experience significantly.
This was really cool to read about. One more little thing designers think about that we players might never realize when it’s done right, but would bitch about if it were wrong. :P
I’ve heard about the 100ms perception threshold before, in the context of network-induced lag in multiplayer gaming. It’s fascinating stuff. I remember using that info to explain to people why a particular satellite internet connection would never work for most online gaming no matter what kind of bandwidth it had, because this connection had an always-present 700ms latency.
I like having games display a notice about “gaming mode.” Many people don’t realize that video processing is taking place at all, much less that they can disable it. And they’re unfortunately missing out on a better gaming experience. I’ve heard several people insist there’s an input lag in Halo Anniversary’s updated graphics mode (I know it’s not your game directly, but it still sort of is, and it seems relevant) that isn’t present in the “classic” mode, but I can’t detect it for the life of me, and I’ve been playing Halo 1 since it released – if there were anything different about the controls I know I would notice it. It would be like my own arm malfunctioning. The only thing I can figure is that the people experiencing this have some kind of video processing happening on their TV. It seems reasonable to me that there might be more of a delay when running the game in the high-res mode due to the increased detail of the image, especially with noise-reducing or motion-smoothing processing.
Interesting post! It’s crazy how much thought gets put into such a small part of a game.
I’ve been reading through Earnest Adams’ “Designer’s Notebook” columns and I just came to this one.
http://www.designersnotebook.com/Columns/063_Postmodernism/063_postmodernism.htm
It seems to me that “tactical immersion” may be dealing with decisions that can be made within this ~100 ms window. being “in the zone” could be thought of as having one of these “single-action” packages of thought flow seamlessly into the next somehow.
Actually, this further reminds me of a blog post that I recorded like 6 months ago about “Player Information Processing Periods.” I think it might be a slightly incompatible model, but it’s similar nonetheless.
http://hallmonitor.ca/game-logic-episode-5-time
Thank you for posting this, Jaime! By the way, it was great meeting you at the VIP party at HaloFest! I am taking this information into account as I prepare to move forward with my team and I’s CryEngine3-powered Halo Fiction project, Venerance, which was announced on HBO here: http://halo.bungie.org/news.html?item=35464
I take every scrap of advice from any of you guys as serious as a heart attack.
If I may ask, in what ways would the kinds of problems you discuss here in this blog entry be affected if a magic switch was flipped, and everyone in the world had a “True 240Hz” OLED display and every game magically operated at 240fps? Would animators have more room to wiggle when there were 240 frames reaching your eye per second instead of 30 or 60?
It is a personal hope of mine that animations will be the next area of dramatic improvement in interactive entertainment, and I think that the 60Hz, 30fps barrier is a real problem for that to happen. OLED is already here, so maybe it’s just a matter of time?
I game on my PC using a Samsung 120Hz monitor, and I love the difference it makes in games, Skyrim being the most recent example.
Thank you for your time, and I hope to meet and chat with you again someday soon!
@Angelica A higher framerate would effect the number of frames of animation, but not the timing. It would still need to be less than 100ms, but you would get 24 frames instead of 3. The animators were more interested in how much time they had to sell the movement, although more frames would have helped a bit. I’m pretty sure the human eye cannot process complex images past ~120 hertz, so at some point there is no benefit to increasing the fps. I think OLED screens only go to 240Hz so they can do 3D at 120Hz.
@Anon The time for synchronous perception is generally faster than human reaction time, so while the two are related, you generally won’t experience it as a removal of the cause-effect relationship between an event and your reaction. However, since the 100ms gets tacked on to any actions the player takes, it essentially increases their reaction time. That can be quite frustrating in an action game, because it will make a player feel less competent than they actually are.
@Adjunct I generally don’t like to call out other devs on the blog, but yeah, Killzone suffered from ~100ms of latency on _every_ input. That’s on top of whatever in-game latency they added to make animations read, effects work, etc. It certainly hurt the feel of the controls.