I've been playing around with different AA (anti-aliasing) techniques that I've never heard of anywhere (I invented them all myself) and I don't think the general gaming community will be very receptive to, but I think this is something that all games should use. It really should be the standard everywhere. I want to share it here because I think the audience will be receptive.
I just put the last piece of the puzzle together minutes ago. Probably the single largest problem for realtime 3D is how to get straight lines. No staircase patterns. I'm not shallow about visuals, I think this is a fundamental problem. The trends have all been to solve this problem by brute force. I wanted to do otherwise. I always thought computers would be much more affordable by this decade, they haven't been, so I've been in a kind of rebellion for some time, only using very inexpensive computers.
Two or three months ago I wanted to play someone's game, since they made it with the tools I maintain for this purpose. But it pushed my workstation beyond its limits. So the first thing I did was turn off MSAA to gain back some "FPS". But I was only using the first level of MSAA (2x I think it's called) so it was a compromise that didn't always work, failing especially to address lines that were high contrast and had very long runs of pixels, like 5 to 10 pixels plus in a straight line, and then 1 pixel along the other bias...
I didn't like turning the MSAA off, even though I knew it was a brute force technique that I probably should not have been using, even in my little testing environments. Still the image was a world of pain without it. Jagged rents everywhere. So I thought about what I could do instead; I generally prefer a dithered image, so I was already used to dithering, which is a little like pointillism. Not stylized, just enough to take the horizontal/vertical bias out of the image, so the eye/brain can be leveraged instead. I don't know of any other way to do that. Computers can't compete with the eye/brain.
So to get to the level of 2x MSAA, what I decided should work was to sample the final image (there is almost always a separate scene buffer and effects buffer in modern graphics) in a checkerboard pattern. The easiest way to think of this is to take two of the same images, and think of them as checkboards, and on one cut out the white squares, and on the other cut out the black squares, and then shift the images until the holes disappear. This is a single mathematical calculation in a "shader" just changing where the images are sampled, no difference at all really.
I should add that another simple technique that precedes this is to sample the images not in the middle of their pixels, but in the corners of the pixels, so that the colors are blurred together to smooth things out. To do that you just need to make sure that interpolation is enabled. I'm guessing that comes for free on modern systems. The checkerboard technique might be too stark without this smoothing, I don't know. It's up to the eye/brain to smooth the checkerboard, so there are two levels of smoothing/eye/brain confusion going on at this point.
The next step, which is probably not optional, is to periodically flip the squares of the checkerboard. I find the optimal rate at 60fps to be a flip every 8 frames. This causes the squares to blend together in a confusing way, and adds a grain to the image, which is important later, but also probably has applications for LOD since pop-in is something that absolutely plagues of the games of today (I'd rather play all games in their lowest LOD than have to look at any amount of pop-in myself.)
Now comes the final step. Up to this point the high-contrast long-run lines are still a problem. Last night I got tired of them and had an aborted idea to try to selectively lower the contrast in a no-cost way somehow, but this doesn't really help because it tends to just accentuate the ends the of the runs, making them fatter, and is generally a dead in. Wracking my brain I had one last idea, that I didn't know if it would even work or not. But I tried it, and it seemed like it would be technically feasible. So after a night's rest I sat down just now to program it and was surprised that the first attempt came out exactly the way I wanted it...
How to deal with these lines/edges is hard to describe. But consider that the staircase pattern is basically arbitrary. You can see it shift as you move through the scene. The points where the lines break and move up/down/left/right by one pixel is arbitrary, really it's a rounding error. So the idea is inside the "vertex shader" where the points are computed, when the points are in homogeneous coordinates, you want to modulate that rounding error every frame so the staircase pattern is always moving, and so what the eye actually sees is the superimposition of all of the staircase patterns before it (you can do blend frames like motion blur as well, which you want to do anyway, to further smooth these lines)
The last technique works without any kind of dither like techniques, but I don't know if it would be desirable without all of the other confusion happening or not. Plus to the degree these edges are always moving, the grain like jitter from flipping the checkerboard squares plays into this for an overall solid image. The checkerboard in effect creates a stipple across the entire image, which I find removes all of the origami like qualities of polygons and makes the image appear solid, substantial.
The most important points are 2. 1) these techniques don't cost a dime. They are ready for GPU-based integrated graphics, whatever you've got. 2) MSAA is falling out of fashion, and isn't applicable to everything in a scene, since it only works on edges. Instead image space AA techniques have become popular. But these are very expensive. Even when they are described as "cheap" the underlying assumption is that there is a multi hundred dollar piece of hardware involved. And screen space AA generally amounts to very expensive ways to smudge pixels around as an after effect. This is not a 3D technique, and who knows how it will distort the image unevenly...
The technique(s) described here (they really need to be used in conjunction) are also all technically screen space, but they are not smudging pixels, they are treating the image identically at every pixel. They do present problems for screenshots, since these effects are not even apparent in screenshots for the most part, except for the stipple and dither, which tends to make screenshots thumbnail poorly. The same can be said for videos. I think this may be the single reason these techniques are not common, because they don't look right in an advertisement, or cannot even be communicated without seeing the game itself. This is using the eye/brain to their utmost. I'm not sure the technique can be improved beyond this point. I believe games need to reach a common ground, and I have a feeling this is the end of the story for AA.
Update: The gravity of the final step described here finally hit me by a day or so after I originally posted this. Later the technique was perfected to the point that while the way it was when I was describing it was certainly better than jagged edges, I now find it underwhelming, as much so as I found it astonishing then, because later the technique was perfected, so the end result is NOW on a level that doesn't even seem real. It will probably take me a year or two to get used to it. I've spent hours on end just gawking at it for days on end.
I posted here (http://notgames.org/forum/index.php?topic=849.msg7239#msg7239) about a Gamasutra blog post I later made for lack of a better idea. It hasn't attracted any comments. I did approach Michael first to try to arrange for something monumental to happen; why not? But just ended up making a public statement for the record, where I reckoned the video game complex/industry should be able to come to find it.
