Thanks for the OS X build - certainly an interesting effect! How many individual panoramas are there in this version?
You'd likely end up with way too much data (as well as way too long rendering times) if you tried to do something like this with hi-res, full color graphics though. And then there's the additional problem of adding in animation - you would need to render the same movie again for every single one of these panoramas, so you'd end up with a huge amount of data just to get away with that.
These guys have some interesting stuff going on -
www.areograph.com. Their technology also makes it possible to turn a photographed or prerendered scene into a live 3d world. They seem to have the same problems with animation and moving lights etc though...