What do we really want from our rendering?

As of late on Twitter and elsewhere, I've had/read a few discussions about the goals of the time and energy poured into our pretty graphics, and I realized it might make an interesting blog post. I can still remember sitting in my first graphics programming class and hearing the professor talk about how it is the more often the goal to simulate film and photography than real life. This is a pretty broad generalization that I've found to actually hold true more often than not.

Even after I decided to write about this, I spotted this part of a fitting conversation between Emil Persson, John Carmack, and Stephen Hill:

This is actually a dilemma that is difficult to answer, and bears in mind questions about what other disciplines and groups have in mind. To me, the problem is that both approaches could be perfectly acceptable. In fact, I would go so far to argue that as broad generalization, that game players almost *want* to be put into a movie when they play a video game, often demanding increasingly cinematic experiences. On the other hand, I think a lot of gamers imagine the future as being "like you're actually in the game," which strays more toward the argument that we want to mimic the human eye over the camera lens. Maybe thinking about what players want isn't the right way to go about it, players want to have fun, and they don't always want to understand why. I have a feeling designers would care even less about this issue, caring more about effectively immersing the player in their levels and plots, which is not entirely relevant here because though because this is more heavily dependent on the actual content of the game. We don't care about content so much, we care about what the "eye" is.

The argument here is very much a discussion of whether or not rendering programmers are trying to simulate a film camera or a human eye, with a very significant part of that being the lens. If you're not familiar, some effects used in video games such as depth of field and lens flares are largely due to issues with capturing images by using a camera, instead of directly with a human eye. Consider that these are some of the things that gamers rave about over every time a big title puts out a new trailer or demo, yet if it seems a little crazy that these things we sometimes try very hard to reproduce, are actually artifacts. I believe there are a couple factors at play here, a big one being that filmmakers have employed the classic "it's not a bug, it's a feature!" mentality and have often used these "flaws" to great effect, which ties directly into what Stephen Hill is arguing in that Twitter conversation. Depth of field in particular jumps to mind as an effect useful for storytelling, and even things like the poor film quality simulated in Tarantino and Rodriguez's Grindhouse movies play a role in setting the mood, and I'm sure that such an effect could play a noticeable role in the right game. On the other hand, there's this amusing little comparison between video games and photography that shows just how ridiculous this copying of camera flaws can get:

Yet one has to admit that HDR bloom tends to get a good "wow" factor from players, which leads to programmers running around on the internet sarcastically shouting for "MOAR BLOOM." After several experiences with team members boosting effects way too high (not to say I haven't done this myself, the key is to expect to adjust it back down), I've begun to understand why so many graphics programmers hold the physical basis of what they are creating to be of upmost importance. That being said, is it a physical basis of a camera or a human eye? I've read papers that discuss the physical basis of how an effect occurs in a camera, and I've read papers that discuss how an effect occurs in an eye, so really either could apply. However, that reminds me of a similar discussion...

Part 2: Reality or Art?

A related and perhaps more interesting discussion, is whether what looks "good" or what is actually "correct" is more important. This is something that when I started making games, I wasn't familiar with how technical this issue could get, but I think it stems from more than a few core deviations between artists and programmers. When it comes to visuals, everyone wants to have the "best," but just like the debate of whether or not that means duplicating camera artifacts among rendering programmers, there is the question of whether or not the best result is the one that is most realistic or the one that the artists think is best.

In a simple world the answer is easy when both parties agree. However, as technology capabilities grow, there are many more chances for rendering programmers to implement systems that do a much better job of modeling how these effect happen in real life. As our graphics improve, art content may start to look different in fundamental ways. Shifting our techniques to draw from a more accurate physical basis can allow us to create even more stunning visuals, but when I was reading through the course notes for Naty Hoffman's "Crafting Physically Motivated Shading Models for Game Development," I wondered if suddenly changing the way the specular highlight behaves might have a traumatizing effects on artists. I can easily recall several artists and designers that I've worked with that would take the side that a physical basis is not nearly as important as achieving a "good look." It doesn't help that programmers are not artists, which I know can make a lot of artists doubt their credibility if they suddenly think an "improvement" makes their art look worse. And speaking as a programmer myself, it doesn't help that sometimes we can be pretty stubborn ourselves, especially when the math has been proven out that the change makes for a more accurate simulation of real life.

This issue is something that I've become increasingly aware of from the excellent work of Naughty Dog's John Hable, from which I'd like to draw a couple examples of where this debate can come into play. In Hable's 2010 GDC presentation about some of the rendering techniques used in Uncharted 2, he discussed the importance of doing calculations in linear space. One consequence of this is that the diffuse falloff will be much sharper when calculations are done in the correct space. This is something that artists might initially might be turned off by, because have soft falloffs is supposed to be a good thing, right? But this is really just a case of not knowing what we really want, and Hable points out the lack of soft falloffs in the film Avatar, which as you may recall, was highly praised for its visual quality. In the realm of skin rendering, something that Hable is also involved in, it is often noted that as we account for subsurface scattering, we need to use detailed textures as input. Artists may have had the inclination to blur details in the textures themselves from wanting to account for the subsurface scattering. However, once programmers account for the scattering in code, they need to become aware that they no longer need to soften texture details to achieve the best results. If you're interested in checking out some of Hable's work for yourself, I highly recommend you visit his website www.filmicgames.com.

There will always be a great deal of approximations and fudge work involved in real-time rendering, but where the fudge work happens is a constantly moving target as graphics hardware continues expand the possibilities for what can be properly simulated. As this target moves, it also creates an environment that reminds me of something a graphic designer once told me about people that are truly pro status at using Photoshop: the people who really know what they're doing can make a dozen smaller tweaks that are hard to notice on their own, but when put together the end result is stunning. I think the same can be said about the direction that rendering is going. Just remember that improvements are not always improvements in everyone's eyes, let alone easy to notice, and it doesn't help that we all mistakes and sometimes our "improvements" are actually in the wrong. It's all fun and games until someone gets stabbed over chromatic aberration.