So the first implication of this is the following: organize your rays into some sort of spatial subdivision structure (e.g. an R-tree). Efficiency of construction is probably a concern here since we have a boatload of rays. Then, send your triangles down this structure one at a time finding which rays it intersects, jotting down the distance for each (so you can keep track of the minimum). You may want something more clever like some kind of cone-based tree to gather up clusters of rays facing roughly in the same direction from roughly the same location, but AO rays in particular tend to have a short maximum distance and are therefore pretty “stubby”, so something as simple as an AABB wouldn’t be terribly inefficient, I suspect.

Then of course the next step is to put the scene geometry in a spatial sudivision structure as well. Now you have two trees that sudivides the scene and rays respectively, so you can simply join them spatially to very efficiently prune out areas of your rays that can not touch areas of the scene. You can think of this as amortizing the cost of traversing the scene structure over all rays, since most of the scene traversal will be done for large chunks of your rays at the same time rather than having to repeat it for each ray. Basic CPU-pseudo-code would be:

raySceneJoin( rayNode, sceneNode ){

if ( rayNode and sceneNode are both leafs){

intersectRayTriangle(rayNode,sceneNode)

return

}

// We have at least one non-leaf. Figure out what children we have.

rayChildren = rayNode.isLeaf ? [rayNode] : rayNode.children

sceneChildren = sceneNode.isLeaf ? [sceneNode] : sceneNode.children

foreach rayChild in rayChildren {

foreach nodeChild in sceneChildren{

// Any pairs of nodes that intersect needs to be refined further

if ( intersection( rayChild, nodeChild) ){

raySceneJoin( rayChild, nodeChild)

}

}

}

}

This would give you something like ~O( log m log n ) complexity where m and n is the number of rays and triangles respectively  (as opposed to O(m log n) which is the usual “foreach ray, trace against scene structure” type of AO computation).

This can obvoiusly be done by the GPU in a variety of ways, e.g. by traversing the trees depth first (keep a global queue of ray/scene node pairs, and just keep adding intersecting pairs to it until you hit a child, read from the queue until it’s empty), although there are further things you may want to do to improve performance and dynamic space utilization (e.g. use a short local stack for parts of the traversal).

Inheritance sucks partly because you’re expected to extend a class that you didn’t write (or that you at least ostensibly plan to modify independently), by patching into a few of the methods here and there and mucking around with the “protected” innards. This is hardly ever a great long-term design strategy for all but the most carefully designed frameworks (which, in turn, tend to make heavy use of “sealed” classes – i.e. disallowing inheritance!).

I much prefer to put all the “hierarchy” in interfaces, and let the classes themselves be “flat”. That way only you need to worry about the internals of your class, and you don’t have to care about what anyone else might do to them without your knowledge.

But even when inheritance works it’s often not desirable. Consider a simple C# GUI dialogue box that asks the user for his name. What’s the interface to this class that a client needs to cope with? Well, if you write it in idiomatic C# by inheriting from Form, it’ll have over 300 public members! 300! For asking the user what his name is! How many of those members do you actually need to use the form? About two or three, at most, I’d guess. So the problem with just blindly extending another class is that each level in the hierarchy you can only add more and more cruft, never remove, and you probably don’t need most of it for your specific version of the class in question. Wouldn’t it make more sense if more specific classes had more specific interfaces?

I’d prefer to write a simple dialogue box by sticking a Form as a member, and write two or three public methods to do the things that people need to do for this specific class. That keeps it simple and obvious. I’m still reusing the heavy-weight and general Form class, but the users of my new class don’t have to see all that. This works fine until you need to use the new dialogue box polymorphically with some other Form-based class.

Really what you’d want is each class to implement a ton of tiny interfaces for each aspect of its behaviour. Most of the time you’d just implement all the interfaces from scratch. Occasionally though, you really do want to just override some tiny part, and you would do so through delegation, not inheritance. You’d keep the “base class” as a member, implement all the interfaces you care about (ideally a small subset of the “base class” interfaces), but forward some of them directly to the “base”. Clearly a language could benefit from supporting this kind of delegation strategy directly, to avoid tedious typing.

I’d like to see a modern, statically typed, OOP language try this delegation/traits based approach to see how it would work in practice. I suspect it would encourage much better programming patterns (customizable and generic components that get specialized and wrapped up into a clean interface for specific instances, rather than relying on inheritance to override bits here and there of some non-generic base class).

As an aside, functional programming in the guise of e.g. Haskell shows how composition and delegation (while implementing type hierarchy “interfaces”) can work beautifully. But I’m sure OOP could be made to work if we just get rid of this unhealthy obsession with inheritance (which was never really the point of it in the first place).

At the extreme you could have built in signature verification on the CPU itself so it will simply refuse to load any code that isn’t cryptographically signed for that chip and that chip alone, then distribute executables as DLC, with per-customer signatures.

Will this work? Well, the world’s armed forces somehow manages to communicate securely with themselves and another, with very high incentives to breach each others communications. That should be enough to tell you that if we really wanted to, we could make sure you and only you see what we (and only we) want you to see.

Another simple existence proof is something like OnLive. If you released a game only on a streaming service so that customers never even have any code running on their machine then it’s obviously impossible for them to pirate it (assuming you maintain security of the main servers which do have the code).

Piracy is inevitable with the current business model, but in a future with digital distribution only (at least for the executable), this is totally a solvable problem, and there’s pretty strong incentives to get it done too.

Not to mention the fact that both the PS3 and to a lesser extent the 360 had a pretty good run of being piracy free. The number of security holes is finite, so eventually you’d expect to be able to plug all of them, and get zero piracy even with the current business model.

Quick idea I’ve had for a while, but likely won’t have time to try out, for getting rid of UV sets once and for all by just looking up textures with a 3D position.

Start with a coarse grained 3D hash grid (i.e a hash map of your grid cell coordinates), then each of these cells would represent, say, a 256³ volume texture indexed by your vertex position. For compression (this volume data would be extremely sparse, usually little more than a flat 2D slice through it), it would use 3D Random Access trees (read this if you haven’t heard about primal vs dual subdivision – eye opener for me!), but probably with a higher branching factor to reduce the number of levels required. Probably 4³ child pointers per node, or maybe 3³. The grid cells are best looked as “short cuts” into the RA-tree, rather than viewing each cell as its own tree.

This means we get a properly quadrilinear-filtered 3D lookup into a sparse tree at the cost of 8 fetches per level in the tree (if you’re going “huh, how does he get 8?” here, then read the aforementioned paper regarding the magic of primal subdivision as it pertains to filtering), so in the higher branching case it would be 3*8=24 fetches total. But we can cache a lot of this because of locality. You’d start traversing the tree in the geometry shader for the whole triangle (possibly tessellating it so it fits entirely within one hash grid cell, and obviously making sure that the nodes referred to by each such cell covers the neighbourhood, and not just what’s strictly internal to that cell, since a triangle may stick out a bit). Most triangles are small so hopefully you could do 1-2 levels of traversal on the triangle level, and then only do the final 1-2 levels in the pixel shader (you may also store the leaf nodes as plain old 3D volume textures, or rather a part of a single giant one).

If you want to get clever you could do some sort of tetrahedral tessellation of the volume (so rather than keeping a “cube” that you progressively make smaller to zero in on the desired sample, you keep a tetrahedron). This means you need fewer samples per level, but also makes my head hurt a bit so you’d need to think that through carefully (in particular how to sort out a primal-esque subdivision). This obviously would make the filtering look “different” in some sense, but it would still be smooth.

Clearly, you’d stop early in this hierarchy based on LOD.

For streaming, you’d store the active nodes of the trees in a big pool and just request the unavailable ones on the fly a la standard 2D virtual texturing.

Maybe you’d build a simple 3D volume texture “brick” to substitute for the most detailed node currently loaded to cut down on “manual” filtering.

Maybe you’d store multiple hash grids. For very small triangles (the majority of them, these days) you may be able to “short cut” much deeper into the tree than for big triangles. These grids would be extremely sparse so storage should be okay if you have some very coarse streaming on top of this. For very distant triangles you may need to start at the real root of the RA-tree instead of using the grids as a short cut.

I’m generally unimpressed by specialized game courses at university. Generally I’d be more inclined to trust a traditional CS or CE degree with a strong side-interest in game programming over a specialized game course.

One of the problems is that many game courses either don’t teach any low-level programming at all, or teach it poorly.

Obviously we still all use C++, so it’s crucial that you understand low-level programming, and every so often you actually need to know it for real, and not just because the language is too low level for you (e.g. you definitely need to understand cache misses, maybe TLB misses, definitely know what virtual memory is, etc.). The problem is that many game courses teach C by forcing you to use it for game programming, but most game programming just isn’t that low level – say 99% of it is high level application code. Learning low-level concepts by writing games is thus an exceptionally poor strategy. You’ll eventually pick it up, but it’ll take a lot longer than if you learnt it by writing something that was 100% low-level instead.

So courses that teach low level programming concepts by having their students write all their games in C/C++ are probably not giving them the kind of experience they need with low level concepts. They’re probably doing a better job than courses that omit C entirely and just use Java or C# or something, but the difference between the two is probably a lot less significant then they’d like to think.

The other downside to teach C by programming games in it is that C is not a very productive language, so your students will take much longer to get anything done, which means they’ll get less practical experience with actual game concepts.

The solution is to admit that low level programming and game programming are two skills that are only occasionally overlapping and the best way to teach both is to give up on the idea that games courses should just do things that are strictly related to game development. Have them spend two months implementing a simple OS (or parts thereof) and they’ll learn more about low-level programming then many years of just game programming will teach them. Then the rest of the time they can use C# for the actual game stuff, which means they’ll get more stuff done, and learn more game development topics in the same time.

It’s been a couple of months since GDC, and I’ve been meaning to do a blog post about the talk I gave there (and if you have membership to the GDC vault you can even see a video).

An R-tree is a data structure for spatial indexing that’s optimized for out-of-core scenarios (basically databases). Spatial indexing is used for storing “stuff” in space. For example, a game might store all of the game objects in a spatial index so that we can efficiently query for things like “give me all objects that are within the view frustum”, or “give me all pairs of monsters and exploding barrels within 5m of each other”.

So, spatial indices are useful for lots of things, especially in games. The problem is that most spatial index structures use tons of small nodes and pointers, and for architectures like the Xbox 360 or PS3 where the L2 cache is small, memory access is slow, and the CPU is too dumb to reorder instructions on the fly, the cost of following a pointer can be massive. An L2 cache miss on the 360 is around 600 cycles. And that’s when you have immediate access to the memory system, if something else is putting pressure on things you may have to wait even longer than that. In comparison, an L1 cache miss and L2 cache hit is about 40 cycles. As Allan Murphy said on one of the other GDC talks: “How many things do you know of that can routinely give you a 15x speed boost?“. In fact, it’s even higher than that since L2-friendly algorithms tend to be reasonably L1 friendly too, so you generally don’t get an L1 miss on each memory access if you’ve made sure the L2 cache is hot.

So, the main idea of my talk is that we need to use different algorithms on these “pointer challenged” architectures in order to minimize L2 cache misses. Basically we need to start treating memory like we treat disk. People have been writing disk-based data structures for decades in out-of-core scenarios (e.g. databases), and indeed R-trees are an example of one such data structure. So what I’ve done is basically just taken that data structure and shifted it up the memory hierarchy one level so that we can use it to minimize memory access instead of using it to minimize disk access.

That’s the rough overview. I won’t repeat my whole talk here, so I’ll just link the slides and ask that you flip through them if you didn’t see the talk, before continuing on to the next section. Here they are.

Hindsight

Looking back on this there are a number of things I would’ve wanted to talk about but simply didn’t have time for in the 25 minutes I had available. This is essentially a brain dump of various things I wanted to flesh out, not necessarily a thorough treatment of each of the issues.

First, it’s worth discussing the breadth-first versus depth-first traversal issue in more detail. Breadth-first does mean we can pre-fetch nodes because unless the “traversal queue” is empty we know ahead of time what the next node will be. What I glossed over in the talk is that breadth-first traversal tends to need more storage for the traversal queue itself, and may therefore cause extra cache misses due to that. Switching between BFS and DFS is a simple matter of switching between a queue and a stack though, so it’s easy enough to benchmark for your particular scenario to see if you happen to hit a case where DFS doesn’t lead to a speedup anymore. In general R-trees have pretty few child nodes for most of the tree (since it has such a high branching factor), so the queue doesn’t really get that full.

Then there were a couple of optimizations that I briefly mentioned, but didn’t go into detail on. For example unrolling the AABB-tests to avoid VMX latency. It’s probably worth noting that you need two different AABB/Frustum tests here. For internal nodes you need to test both the “near” and “far” points on the bounding box because it matters if an AABB is entirely outside/inside or intersecting. If it’s entirely inside the frustum you can skip any further tests and just add all of the children directly (see further discussion on this later), but if it’s intersecting you need to add the children to the traversal queue/stack. However, for leaf nodes you do not care about whether or not the frustum is intersecting an AABB, you only care if the AABB is entirely outside the frustum or not, which means you can test only one point instead of two. This simplified test also means you have to unroll more nodes in order to hide all the VMX latency (because we’re doing less work per AABB). Because of the way the hierarchy looks, you’ll typically spend the vast majority of your time testing leaf nodes so optimizing for that case by using a simpler test gives a major speed boost.

Once we have determined that an internal node is entirely within the frustum we need to add all of the leaf nodes underneath it to the output without further testing. Currently I do this by just putting it on a separate queue and traversing it at the end without tests. Unfortunately because there are no tests we don’t really do any work to hide any latency here. One optimization would be to link together all of the leaf nodes in a linked list (in pre-order traversal order), and then in each internal node store pointers to the first and last leaves in that sub tree. The closer to the root you get, the larger this range would be. This way we can save the traversal required to actually find the leaf nodes – when an internal node is determined to be entirely within the frustum, we simply add its two “range pointers” to the queue, and then do a linked-list traversal without having to process any more internal nodes.

Another interesting thing that struck me while working on this is how neglected spatial joins usually are in games. We have tree-like structures everywhere, but if we want to find the closest cover for each enemy, or the N closest lights for each object, we generally do an O(n log n)-style search where we loop through each of the objects and do an O( log n ) search in the tree for each one. Reading about R-trees, which is a database algorithm, you’re bound to come across joins as they are the bread and butter of database operations, and that’s how I realised that we’ve been neglecting this really cool tool. We should start adding spatial joins to our spatial indices (to get average time O(log² n) performance), as they’re pretty useful. Basically the implementation just steps down two trees in lock step, and at each level scans through all the possible pairs of sub-trees and determines if those two sub-trees are close enough to need further processing, and if so adds them to the stack/queue. The reason this is faster is because we can say, effectively, “this big part of the level is too far away from that big part of the level that no objects in each could interact with each other”.

One aspects of AABBs that I didn’t really touch on is the actual “AABB” part of them. It’s tempting to think that you should save space and ALU by using a bounding sphere instead of an AABB, but this is almost certainly a mistake. The reason for this is that AABBs are pretty good at containing other AABBs with minimal space wastage, whereas spheres absolutely suck at containing other spheres, and of course in a hierarchical setting this  is a key property. You might go even further in this direction and use an 8-DOP which is even better at packing a bunch of 8-DOP children (and only requires two more floats per node), but I do think that AABBs are probably the simplest you can go without seriously sacrificing query performance due to wasteful bounding volumes. That said, spheres can be very good at packing the leaf objects themselves, so you could potentially switch to spheres in the leaves, but then use AABBs for the internal bounding volumes. I haven’t tried this yet, but it seems like it could save some space, and improve efficiency of the leaf-node tests (which constitute the bulk of the tests).

Also, I’d obviously like to implement this on the GPU. It seems very suited for that kind of thing (since each node is very wide, you could even process them in a SIMD-fashion). In particular I’d like to try it for ray tracing (tracing a single ray using all N lanes of the ALU cores to process each node, one lane per child AABB, instead of resorting to things like packet tracing), and maybe even a FEM-based global illumination renderer (“find me all pairs of hierarchical surface elements that are close enough to either transmit radiance or occlude each other”).

Other benefits

There are two major benefits that I feel like I didn’t really emphasize enough in the talk. The first is that R-trees are just plain easy to implement. I mean, there are tons of variations in all the different parts, and some of these variations are quite tricky to get right, but the final configuration I ended up with is very simple. So even if you’re not interested in cache performance, but need a spatial index, I’d still recommend R-trees over just about any other system. Wide trees are usually considered more complicated than binary ones, but in practice it turns out to be the other way around. Once you have enough children they end up giving you a good estimate of what the entire sub-tree at a given node looks like. That means you can do node splitting by looking only at the immediate children. With a binary tree, however, the immediate children tell you almost nothing, so if you want to split a node you’d have to do some serious analysis of the whole sub-tree. In the end, this leads to R-trees having a very simple, dynamic, and incremental construction strategy based on node-splitting, that still produces very good hierarchies.

The other benefit that I didn’t focus enough on isn’t really unique to R-trees, but AABB-trees in general, and it’s that they’re pretty bullet-proof. Other strategies have all sorts of corner cases that are painful to deal with (KD-trees need splitting, uniform grids need some way of dealing with nodes that cover multiple cells and large empty areas etc.), but AABB-trees just kind of adapt flexibly to the input and works just as well for pretty much anything you throw at it. That’s a very nice property to have for your main scene organisation structure.

Performance

Of course, the biggest omission in my talk was rigorous performance comparisons. One reason for this was simply lack of time in the talk itself. I only really had time for one slide, so I tried to pick one that would illustrate the general “shape” of the curve for various parameters and trade-offs for R-trees themselves. The problem with that slide is that it only compares R-trees against other R-trees. I would’ve liked to compare it to other strategies, in particular binary AABB-trees, or BIH-trees (since they have roughly the same strengths as R-trees), implemented with all the usual cache-tricks to improve performance. I suspect that R-trees are a lot faster, at least on the Xbox 360 due to the high cost of cache misses, simply because the other techniques try to improve cache performance very indirectly (by compressing nodes, rearranging them in memory etc., to try to improve the probability of a cache hit, and without really attempting to maximize memory bandwidth), whereas R-trees improves cache performance in a more direct way by explicitly fetching “blocks” of memory into the cache at near-maximum memory bandwidth, just when needed, and not fetching (much) data that’s not actually used.

So, does this mean that I have no clue how fast R-trees are? No, of course not, I just don’t have a rigorous and publishable clue. The next paragraph is therefore very tentative and should be taken with a few grains of salt.

I know they’re faster than the old method of visibility tests in  Red (the engine used for all Rare games these days, based on the Banjo engine) because I can do rough comparisons of roughly the same workloads. That system is essentially just a uniform grid. The problem is that it’s a uniform grid that hasn’t really been optimized with nearly the same effort as the R-trees have (in particular, it makes no effort to pre-fetch memory, relying entirely on its relatively benign data layout instead), so I wouldn’t feel comfortable concluding that R-trees are better than uniform grids in general. For our own engine, though, I feel fairly comfortable in saying that R-trees perform better than the old system (as well as scaling better, using less memory, and generally having no real annoying corner cases that have to be worked around, unlike uniform grids). Prior to the uniform grid (which was implemented for Banjo) we had a fairly standard AABB-system, so because we switched away from that for performance reasons, I’d transitively expect R-trees to be faster in a real benchmark than that system too. But again, I couldn’t claim that the old AABB-implementation was in any way optimized, so I’m not prepared to make any definite statements about them in general.

This may seem like a shocking lack of academic rigour on my part, and maybe it is, but at the end of the day I’m not a researcher, and I don’t always have time to implement every conceivable data structure for performance evaluation, so I have to make an educated guess at where the best performance will be found, and then go for it. If someone is looking for an interesting MSc thesis to do, though, I’d be very interested in seeing up-to-date benchmarks on recent consoles (against the data structures mentioned above but also against others like hash grids, or regular uniform grids, even though they aren’t necessarily comparable w.r.t. flexibility and features). It would also be interesting to see how these cache-oriented optimizations fare on a modern PC which aren’t nearly as “pointer-challenged” – perhaps you could even identify a specific threshold point (in “cycles per cache miss”) where you’re better off using cache-oriented data structures.

Jonathan Blow is working on stable cascaded shadow maps, and trying to reduce wasted texels due to the way the shadow frustum is set up. Well, as it happens I have my own minor tweak to Valient’s technique that also reduces wastage, so I’ figured I’d describe it here with pretty pictures. I’m going to assume that you’re roughly familiar with the general idea, if not, read e.g. Blow’s first blog post on the topic, or ShaderX 6.

The root of the problem is that with realistic camera FOVs, you often get a situation like below.

The first slice of the view frustum (the trapezoidal shape in the middle) is inscribed in a sphere (for rotational invariance – don’t want those shadows flickering!), and then that sphere inscribed in a box (the shadow map’s orthogonal view frustum). Now, you could take the position that all that red area is wasted, since only the green area is inside the frustum and the first splitting plane. Blow performs a few very clever tricks to overlap the shadow map slices to avoid all those red parts taking up too much, for a total of 25% memory savings.

However, who says that only shadow texels within the first slice are useful? Anything to the right of the splitting plane is also within the view frustum, and can easily be used for shadowing. Why reject these high-res shadow texels and skip to the next slice just because we’re on the other side of the splitting plane?

So rather than trying so hard to not store these “wasted” shadow map texels, we should instead make sure we’re making use of those texels whenever we can so that they’ aren’t wasted anymore! As we sample the shadow map we just test the current point against the first shadow frustum – not the bounding sphere, and not the splitting plane, the frustum. That changes the picture to something like this:

This is all pretty standard so far. My own little tweak comes from the realization that if we’re not going to bother using the splitting plane as our boundary for sampling then why bother with it at all? The implication is that when we compute our bounding sphere we shouldn’t worry about how it fits with some arbitrarily positioned plane, we should only worry about how it fits with the frustum itself (and previous slices). In other words, we position our bounding sphere (of the same size) to minimize waste by just shifting it to the right. We have to stop before we pass the near plane of the view frustum, of course. Like so:

As you see we’ve reduced texel waste for the first slice by quite a lot here. We’re using approximately 5/6 of that texture space, instead of 1/6 like the first scenario. However, notice what happened at the corners where the splitting plane intersects the frustum, they’re outside the sphere! That means that we’ve effectively moved the splitting plane back compared to the previous image. If you were in the business of worrying about the location of the splitting plane this would be a problem  – luckily we aren’t, so it’s not. Freeing ourselves from the constraint of the splitting plane means we can find a bounding sphere with a better overall utilization of the shadow map.

If the FOV increases, or the sphere gets smaller, the “corners” (determined by the original splitting plane) will eventually drop outside not just the sphere, but the shadow map too, and for those pixels we do need to drop to the second slice when sampling, which of course means that we have to be careful that the second slice is far enough back that it doesn’t leave any gaps. This also means that there are minor areas near the edges of the screen where we may drop to a lower resolution shadow map sooner than we would with the original scheme, but in return for that we get a big area in the centre of the screen that can now use a higher res shadow map due to the better utilization.

Now, most of the first slice will actually overlap the second slice – we will still always use the first one when we can to maximize the amount of pixels that get to use highest shadow resolution, but there is some overlap where parts of the second slice will be “overruled” by the previous slice. In practice, however, this isn’t such a big deal because the second bounding sphere is usually several times larger than the first, so as a proportion of the overall area the overlap is actually quite minor.

For example, take a look at this illustration of the second slice:

We’ve zoomed out a bit. You can see the first slice as the rectangular red area to the left and centre (and note how the bounding sphere has been carefully positioned to just barely cover the intersection of the previous slice with the frustum, leaving no gaps).

So while there is still some waste for things that are outside the frustum, and there’s also wastage for texels that are redundant because they’re already covered by the previous slice, by and large we’re getting very good texel utilization.

As you may have heard, Direct X 11 brings tessellation support, and all the hardware vendors and benchmarks are going crazy with super-finely tessellated meshes, promising us automatic level of detail and unprecedented visual fidelity. What you may not be aware of is that the Xbox 360 has very similar tessellation hardware too, which means many game developers have had the opportunity to use these same techniques for about five years now. So why aren’t all console games using tessellation pervasively? That’s what I will tell you in this blog post, as well as demonstrate a solution to the problem.

The clue is present in just about any of the dozens of tessellation demos that are now available. In case you haven’t seen one, here’s a capture I just took of one of the samples in DirectX 11 (ignore the frame rate, the capturing software interferes).

This looks good and all, but notice the mesh density. Even when not doing any tessellation at all this is a 50×50 grid! That’s five thousand triangles at the lowest level of detail for roughly one square metre of cobblestone. That’s plainly a ridiculous poly-count for any practical purposes in games. There are two main problems with such a high vertex density: it wastes time processing more vertices than needed, and leads to the small triangle problem.

It’s wasteful because even at a modest distance you’re not going to need 5000 triangles to represent a square metre of ground, so transforming that many vertices is just throwing precious cycles away.

The small triangle problem refers to the efficiency loss when rendering small triangles. Current GPUs rasterize pixels in small groups of at least 2×2 pixels. Whenever a triangle covers the entire “quad” of 2×2 pixels all is well, if however some of those pixels are not covered then the GPU resources associated with the unused pixels are just squandered. For example, if each triangle only covers a single pixel, then every one of those quads will be 3/4 unused, leading to just 25% pixel shading efficiency.

The problem

So why are all these demos using control meshes that are already finely tessellated? It’s no accident. There are two main reasons for this. The first is that you can only specify tessellation factors on a per-edge level, so if you want to adaptively tessellate some areas more than others, you’re going to have to have a dense distribution of edges to support that variation across the surface. The bigger reason, though, is that smoothly varying tessellation on a displaced surface looks like rubbish at lower tessellation levels. See this example, which is just the previous demo with a more reasonable base mesh resolution (4×4, instead of 50×50).

Notice the shimmering, bucking, artefact as the tessellation slider is moved up and down. This is the dirty little secret that Xbox 360 developers have known for years: at reasonable mesh densities, continuous tessellation just looks awful and is unusable as a general strategy. We used it in Banjo Kazooie : Nuts & Bolts for water, because the shimmering artefact actually looked pretty decent on a water surface that was supposed to shimmer, but while we tried to use it on some other things we could never live with the artefacts (the solution below is something that came to me after BKNB shipped, and could in principle be implemented in a future title).

A solution

So what’s going on here? Well the problem is that as vertices get added and removed they smoothly travel to their final location by sliding across the surface, this lead to vertices rapidly bobbing up and down as they move over the surface and sample different displacements from the displacement map. This problem is not a new artefact, it’s actually just standard minification aliasing. When you do normal straight texture mapping, the texels are sampled at pixel locations. If the ratio of texel density to pixel density gets too high you get minification aliasing because each pixel “doesn’t know” which of the many texels it covers to sample from, so small changes in pixel position will lead to entirely different texels being used. For regular texture mapping we solve this by using MIP-mapping. Effectively just choosing a lower resolution version of the texture when the sample locations are too sparsely distributed to accurately reconstruct the full resolution texture.

Displacement mapping is no different. Instead of sampling at pixels we have vertices, and instead of colours the values sampled are geometric offsets so the aliasing artefacts manifest in a different way. The basic problem, though, is that we’re reading a high frequency texture (the displacement map) at too low of a sampling frequency. The solution to this problem is the same as for regular texture mapping – use MIP-mapping to choose a lower resolution texture when the sampling frequency (i.e. tessellation factor) is lower.

So how do we determine which LOD to use? Well, one simple way is to look at the length of each edge in the control mesh in texture space, and choose a MIP level for each edge so that the distance in texels for each subdivision will be no more than 0.5 texels (this is to get us under the Nyquist limit which says that the sampling frequency must be twice that of the signal frequency). In other words, if the length of the edge in pixels is L, and the edge’s tessellation factor is T, then we will get L/T pixels per subdivision. We want that to be 0.5 by choosing a MIP level, so we have to choose a MIP level M so that (L/T)*2^(-M) = 0.5 (the linear distance in texture space decreases by a factor of two for each MIP level). Solve for M and we get: M = 2*log2( L/T ). In practice, linear interpolation isn’t a perfect reconstruction filter, so we may need a small fudge factor to boost the MIP level up slightly further. Note that although we use MIP mapping, the MIP level used doesn’t depend on viewing distance or angle, just on the tessellation factor (which in turn may depend on those factors, of course).

So, now that we have a MIP level per edge, we can simply interpolate between them in the Domain Shader to pick a suitable MIP level for each verex. It’s important that the interpolation you use has the property that when the point is on an edge, the weights for the two other edges are zero, so that edge-vertices use the same MIP level regardless of which patch they belong to. Here’s how this looks:

We can see the basic idea working here. Rather than shimmering artefacts when the tessellation level is too low to adequately represent the displacement, we just get a flatter surface instead (due to choosing a lower-res MIP level of the displacement map). However, it should be obvious that there’s a problem here if you look closely at the “spikes” visible at each vertex in the base mesh. Each one of those lies on two edges per patch, so won’t know from which to retrieve a MIP level. So what do we do? Just pick one of the two candidate edges at random? Take the average? No, neither will work work because neighbouring patches can have entirely different edges using the vertex. In fact, there can be an arbitrary number of patches associated with a vertex, and for each of them that vertex must pick the exact same MIP level in order to avoid cracks.

So what do we do for the control vertices themselves? Ideally, we’d find the average MIP level for all the edges used by a vertex, but that would be expensive since tessellation factors (and therefore MIP levels) can change each frame. The simplest solution I can think of is to store a “preferred edge” index for each control vertex. This would simply be a randomly chosen edge that uses that vertex. When you detect that you’re on a control point (by checking the barycentric coordinates), you simply check the vertex’s preferred edge index, and fetch the MIP level associated with that edge. Note that the preferred edge is not necessarily in the same patch as the current patch, but it is consistent in that every patch using that vertex will use the same preferred edge, and therefore MIP level, which eliminates cracks.

Here’s how this looks:

This is much better. We’ve got rid of most of the aliasing, and the corners fall in line with a MIP level chosen from its immediate neighbourhood. Notice that there’s still some ever-so-subtle shimmering going on in the creases here. That’s because we’re approximating the MIP level based on the coarse control mesh, whereas in reality the actual triangles in the tessellated mesh vary in size and shape. The main downside to this strategy is that we need to compute edge tessellation and MIP levels outside the main draw calls to produce a buffer of per-edge tessellation factors and MIP levels, which not only adds a draw call and associated bandwidth increases, but also requires us to transform all of our control vertices several times (assuming the tessellation factor depends on viewing direction, skinning etc.).

It would be ideal if the domain shader would give you some information about adjacent patches, so that we could easily compute an average LOD for all the edges connected to a vertex.

Other solutions

There are a couple of other potential solutions to this problem, so I’m not trying to say that tessellation and displacement mapping is doomed or anything, I’m just trying to temper some of the wild enthusiasm by pointing out that the claims of “automatic LOD” etc. are vastly overstated.

For example, this could clearly be combined with a standard LOD system, where only the high-res mesh uses tessellation. Although at that point it’s unclear whether it’s even worth the effort. Just let the high res model be really high-res instead.

Another option is to use fixed-step tessellation factors, instead of the continuous ones. This causes harsher transitions between levels of details, but at least you don’t have vertices sliding across the surface causing the disturbing shimmering demonstrated above. But then again, if you’re going to use discrete tessellation levels then simply using a normal LOD system is likely to be faster, and will definitely be a lot simpler.

Using power-of-two tessellation is promising. In this mode vertices are added in a “power of two” fashion by subdividing each edge. This is attractive because the tessellation pattern is simple enough that you could do geo-morphing by computing, in the domain shader, the previous height for the current point and the actual displaced height and then interpolating between them. The downside with this is that in order to compute the previous height you have to somehow figure out where the current point lies on the surface of the mesh tessellated at a lower level. This means you’re effectively duplicating the effort of the fixed function tessellator in the domain shader, which doesn’t sit right with me.

Conclusion

Generating geometry by sampling a texture at varying frequencies is not a trivial problem, and requires careful consideration in order to avoid the same aliasing problems that we’ve already dealt with for regular texture mapping. Unlike regular texture mapping, however, we don’t have any real help from the hardware in figuring out the appropriate MIP level for a displacement map, so we have to figure out approximations ourselves, and while there are workable solutions, I haven’t been able to figure anything out that isn’t ugly or inefficient in some way. I do hope that future talks about and demos of tessellation and displacement mapping will at least acknowledge this problem, instead of just ramping up the base mesh’s polycount, sweeping the issue under the carpet.

Update

A simple variation of this idea which has the benefit of being very cheap is to compute a per-object representative MIP level on the CPU (in some approximate way), and then in the domain shader simply interpolate between the MIP level you got from the edges, and this per-object MIP level, based on how close you are to a hull vertex (e.g. use the maximum value of your barycentric coordinates). You’d probably want to ensure that only the vertices that are very close to the hull vertices get influenced by the per-object MIP level, but this would at least ensure that all hull vertices use exactly the same MIP level (the per-object one!) while keeping the surface looking smooth.

I just watched a panel from PDC called Microsoft Perspectives on the Future of Programming with loads of “big name” programming gurus. At the end of the session Herb Sutter points out that eventually Moore’s law will end, and then optimization will become extremely sexy again. This is a very good point, but I think there’s another law that will cause optimization to become sexy again far sooner than that: Almdahl’s law.

Almdahl’s law basically states that once your performance increases come from parallelism rather than increasing clock speeds the bottle neck will rapidly become the serial bits of your program. The bits that can’t be parallelized. Once that happens (if it hasn’t already!) it will become extremely important to write those serial bits in a way that is extremely efficient.

Note that this scenario is exactly the opposite of the situation capitalized on by lots of current dynamically typed languages like Python or Ruby. These languages work because they assume that most of the time will be spent on “bulk” code that can be written in C libraries, and that the “glue code” which plugs all these bulky computations together will account for only a tiny fraction of the runtime and therefore doesn’t need to be that fast. This is true today, but in the future the wall-clock time will be dominated by that glue code because it’s full of sequential dependencies and can’t be parallelized.

In short: If your “glue code” is written in a scripting language that’s 20x slower than an efficient compiled language, then Almdahl’s law states that the whole application will be 20x slower (in the limit). It’s true that the majority of clock cycles will still be spent in the “bulky” computations (which will hopefully be parallelized to hell), but the critical path of the program will be dominated by the glue. We need faster glue!

So what’s the solution? Go back to C? I sure hope not! Productivity and safety will still be extremely important, but I do think that merely sweeping performance concerns under the rug, as many high level languages do today, will very rapidly become untenable as we rush towards the performance wall of Almdahl’s law. I think a high level language can be designed that can be compiled to efficient code. Such a language would need to take conscious steps to ensure that abstractions don’t cost too much performance-wise (occasionally making sacrifices favouring performance over productivity), but at the end of the day I see no reason why it couldn’t be type safe, null safe, side-effect safe and most of the other goodness we expect from modern languages like Haskell, while still producing native code on par with good C compilers.

Signed Distance Functions are a pretty simple concept. Basically for each point in the world, you return a distance to the nearest surface, negative distances are inside geometry. One neat application of them to represent scene geometry, and ray trace into the SDF. Why would anyone do such a thing? Well, it turns out that when marching along a ray looking for intersections (which is obviously not the only way to trace rays), it’s jolly useful to know a minimum bound on when you might expect to encounter a surface. This way you can keep taking large jumps along the ray, using the SDF to figure out how large, until you finally reach some threshold distance. Here’s the end result of the code I’m presenting in this blog post.

Note the cool twisted pillar, and the nice soft ambient occlusion around contact points. Both of these features are simple to do when tracing SDFs, but much harder with other methods.

So basically we represent our scene as a SDF, but in order to have varying materials we need this function to not only return the distance to the closest surface, but the material of that surface too, so we end up with the following Haskell type for our scenes:

type Scene = Vec3 -> (Double, Material)

In order to define a simple scene we just need to write a function which returns the distance to the surfaces, for example here’s a unit sphere:

sphere pt = (mag pt – 1.0, defaultMaterial )

In other words, we take the magnitude of the position (i.e. distance from the origin) and subtract 1.0 since this is our radius. Simple! To build larger scenes from simple scenes such as this, we need a way to combine two scenes. This is done by just taking the minimum of the distances, and picking the corresponding material.

mergeScenes :: Scene -> Scene -> Scene
mergeScenes scene1 scene2 pt 
    | d1 < d2 = res1
    | otherwise = res2
    where   res1@(d1,_) = scene1 pt
            res2@(d2,_) = scene2 pt

Since our scenes are just functions from position to distance, it’s very easy to manipulate the geometry by warping the inputs to the SDF – for example, in order to build the twisted column shown in the screenshot above we first build a regular column, and then twist it by rotating the input by varying amounts (related to the y component):

column pt@(Vec3 x y z) = (mag pt – mag closestPt, defaultMaterial )
    where   clamp x = max (-0.5) (min 0.5 x)
            closestPt = Vec3 (clamp x) y (clamp z)              
twistedColumn pt@(Vec3 _ y _) = column (rotateY y pt)

There are many other possibilities for these kinds of modifications, for example we could add some random noise to the distance function to get bumpy surfaces etc.

Once we have a scene to ray trace against, we simply march through the SDF until we reach a distance small enough to consider “on” the surface:

type Ray = (Vec3, Vec3) — origin, direction

rayTrace :: Scene -> Double -> Ray -> Maybe Color
rayTrace s end (pos,dir)
    | end <= 0 = Nothing
    | dist < 0.0001 = Just $ shade s material dir n pos defaultLight `scale` getAO s pos n
    | otherwise = rayTrace s (end-dist) (pos + dir `scale` dist, dir)
    where   (dist, material) = s pos                           
            n = getNormal dist s pos 

The first equation of rayTrace simply checks if we’ve reached the far plane, in which case we return Nothing, since there was no hit. The second guard checks if we’ve reached a surface, and if so shades the point using our light and the normal is computed by looking at the distances in the neighbourhood of the point. The whole thing is modulated by our ambient occlusion factor (which may not be strictly correct, I know).

This ambient occlusion factor is computed by looking at a small number of samples “above” our surface point and comparing the distance to these samples with the distances in the SDF for those samples. If the sampled distances are smaller than the distance to the surface, then that means there’s some amount of “occlusion” for that sample (since there’s evidently some object other than the surface nearby). We weight these occlusion estimates and sum them to get the final ambient occlusion. This is pretty hacky with several magic numbers to tweak, but it works really well, giving smooth 3D ambient occlusion with very little cost.

Here’s the full code listing. I’ve implemented a small vector library with the stuff we need in order to keep it reasonably self-contained. You will need some way of actually displaying/saving the final image, though – the code below uses the ppm package (”cabal install ppm” should do it). This code is not intended to be an example of “nice” Haskell code – it’s pretty much just a copy of the code I ended up with after experimenting with SDFs, so there’s lots of hard coded values and other limitations (e.g. just one light source) which are, erm, ”left as an exercise to the reader”.

{-# LANGUAGE  ParallelListComp #-}
import Codec.Image.PPM hiding ( Color )
import System.IO
import Data.List
import Control.Parallel.Strategies

type Material = (Vec3, Double, Double) — Color, Specular power, gloss
type Scene = Vec3 -> (Double, Material)
type Light = (Vec3, Vec3, Double)           
type Color = Vec3
type Position = Vec3
type Direction = Vec3
type Ray = (Vec3, Vec3)

main = do
    let rays = getRays 256 (pi/2)
    let colors = parMap rnf (map (maybe (0,0,0) toRGB24 . rayTrace scene 100)) rays
    writePPM "output.ppm" colors

colorize :: Color -> Scene -> Scene
colorize c s pt = let (d, (_,p,g)) = s pt in (d, ( c,p,g ))

red, green :: Scene -> Scene
red = colorize (Vec3 1 0 0)
green = colorize (Vec3 0 1 0)

True `xor` a = not a
False `xor` a = a         

checker x y = checker1D x `xor` checker1D y
    where checker1D x = floor x `mod` 2 == 0

– the scenes
sphere pt = (mag pt – 1.0, (Vec3 1 1 1, 20, 0.5) )
plane (Vec3 x y z) = (y + 2, if checker x z
                                then (Vec3 1 1 0.5, 10,0.4)
                                else (Vec3 0.3 0.3 0.1, 10,1) )
plane2 (Vec3 _ y _) = (1.5-y, (Vec3 1 1 1, 10, 1))

column pt@(Vec3 x y z) = (mag pt – mag closestPt, (Vec3 0.4 0.7 1, 2, 0.5) )
    where   clamp x = max (-0.5) (min 0.5 x)
            closestPt = Vec3 (clamp x) y (clamp z)              
twistedColumn pt@(Vec3 _ y _) = column (rotateY y pt)

mergeScenes :: Scene -> Scene -> Scene
mergeScenes scene1 scene2 pt 
    | d1 < d2 = res1
    | otherwise = res2
    where   res1@(d1,_) = scene1 pt
            res2@(d2,_) = scene2 pt
scene :: Scene
scene = finalScene . (+ Vec3 0 0 5)
    where finalScene = foldl1′ mergeScenes [
            green sphere,
            plane2,
            plane,
            twistedColumn . (+ Vec3 2.2 0 0),
            red sphere . (+Vec3 (-1.8) 1 0) ]

getNormal :: Double -> Scene -> Position -> Direction
getNormal d s pt = normalize (Vec3 x y z)
    where   eps = 0.0001
            x = fst ( s (pt + Vec3 eps 0 0 )) – d
            y = fst ( s (pt + Vec3 0 eps 0 )) – d
            z = fst ( s (pt + Vec3 0 0 eps )) – d

shade :: Scene -> Material -> Direction -> Direction -> Position -> Light -> Color
shade s (materialColor, specPower, gloss) eye n pt (color, pos, att)
    = combinedColor `scale` (attenuation*lambert*shadow)
    where   attenuation = 1/((att * mag lightDir )^2)
            lightDir = pos-pt
            lightDirN = normalize lightDir
            lambert = max 0 (n `dot` lightDirN )
            combinedColor = color * materialColor + spec
            r = eye – n `scale` (n `dot` eye)*2
            spec = color `scale` ( gloss * (max 0 (r `dot` lightDirN))**specPower)
            shadow  | lambert == 0 = 0
                    | otherwise = case rayTrace s (mag lightDir) (pt + n `scale` 0.0001, lightDirN) of
                                    Nothing -> 1.0
                                    Just _ -> 0.4                   
defaultLight = (Vec3 15 15 14, Vec3 2 1 0, 0.5)

getAO :: Scene -> Position -> Direction -> Double
getAO s pt n = clamp $ 1.0-k*sum (take count [(x-y)*w | w <- iterate (*falloff) 1 | x <- dists | y <- sampledDists])
    where   dists = [0, delta ..] :: [Double]
            sampledDists = map (d -> fst $ s ( pt + n `scale` d ) ) dists :: [Double]
            clamp x = max 0 (min 1 x)           
            k = 2.75
            count = 5
            delta = 0.1
            falloff = 0.57            

rayTrace :: Scene -> Double -> Ray -> Maybe Color
rayTrace s end (pos,dir)
    | end <= 0 = Nothing
    | dist < 0.0001 = Just $ shade s material dir n pos defaultLight `scale` getAO s pos n
    | otherwise = rayTrace s (end-dist) (pos + dir `scale` dist, dir)
    where   (dist, material) = s pos                           
            n = getNormal dist s pos 

toRGB24 (Vec3 r g b) = (exposure r, exposure g, exposure b)                           
    where exposure f = max 0 (min 255 (round ( 255 * (1-exp (-f)) )))

writePPM fname img = do
    let imgData = ppm_p6 img
    withBinaryFile fname WriteMode (h -> hPutStr h imgData )

getRays sz horizFOV = map computeRow pixelCoords1D
    where   pixelCoords1D = take sz [-1, -1 + 2/fromIntegral sz .. ]
            z = -(1/atan (horizFOV / 2))
            computeRow y = [ (Vec3 0 0 0, normalize (Vec3 x (-y) z) ) | x <- pixelCoords1D ]  

– quick and dirty vector maths
data Vec3 = Vec3 !Double !Double !Double deriving (Show,Eq)

instance Num Vec3 where
    (Vec3 x y z) + (Vec3 x’ y’ z’) = Vec3 (x+x’) (y+y’) (z+z’)
    negate (Vec3 x y z) = Vec3 (-x) (-y) (-z)
    x – y = x + negate y
    abs (Vec3 x y z) = Vec3 (abs x) (abs y) (abs z)
    (Vec3 x y z) * (Vec3 x’ y’ z’) = Vec3 (x*x’) (y*y’) (z*z’)
    fromInteger x = let x’ = fromInteger x in Vec3 x’ x’ x’
    signum = error "signum on Vec3 is not implemented"

(Vec3 x y z) `dot` (Vec3 x’ y’ z’) = x*x’ + y*y’ + z*z’
(Vec3 x y z) `scale` a = Vec3 (a*x) (a*y) (a*z)
mag v = sqrt (v `dot` v)
normalize v = v `scale` (1/mag v)
rotateY theta (Vec3 x y z) = Vec3 (x*costheta – z*sintheta) y (z*costheta + x*sintheta)
    where   costheta = cos theta
            sintheta = sin theta

Side note: the outer loop for our ray tracer uses “parMap” for almost perfect linear scaling (I get 1.9x on my dual core machine, and I suspect a lot of the overhead is just the serial business of actually writing the image to disk). It’s interesting to note that this was added pretty much as an afterthought; while playing with the code and adding more and more stuff the runtime got slower and slower, so as a quick’n’dirty speedup I changed a “map” to “parMap rnf” and it ran twice as fast. This wasn’t a carefully considered optimization pass, it was more like “This is slow, I’ll just hack in some parallelization real quick”. I dare say I’ll never utter that sentence when writing C – even if it had a “parMap” equivalent, I’d be very careful about using it since there’s no way of knowing that there won’t be any hidden side effects somewhere.