Maybe what we need is a way to record 3D reality. So we don't have to start with this empty "space-less" world.
In fact, Manovich mentions such a project, "Aspen Movie Map" (1978) and remarks:
The idea of constructing a large-scale virtual space from photographs or a video of a real space was never systematically attempted again, despite the fact that it opens up unique aesthetic possibilities not available with 3-D computer graphics.