Virtual Cinematography Has Arrived With 'The Matrix Reloaded'
After inventing the special effect technique known as Bullet Time for the 1999 sci-fi thriller The Matrix, special effects supervisor John Gaeta has been continually asked how he would top that in the sequel The Matrix Reloaded, which opens in theatres May 15. "People get really preoccupied with, 'Are you going to top yourselves this time? Are you really gonna come up with a zinger?'" Gaeta tells Wired in a new feature article. "My job has nothing to do with making zingers. The point is not to knock you over with a visual trick. The point is to be able to construct events that are so complex, in terms of what human bodies need to do, that the total 'effect' is impossible choreography. 'My God! It looks real, but it just can't be.'"
Like many in the film industry, Gaeta has long talked of the promise of virtual cinematography, a confluence of technologies that will allow directors to digitally sculpt actors' performances with ease. To create Reloaded's fight scene between Neo and a hundred Agent Smiths (nicknamed the 'Burly Brawl'), Bullet Time would have required tangles of crisscrossing still-camera rigs and years of compositing. What Gaeta needed was a virtual camera that could fly through the 3-D scene with absolute freedom. "The concept of Bullet Time had to graduate to the true technology it suggested," he says. "For Reloaded, we had to finish the job so that we could get relentless, uninterrupted, and editable chunks of Neo in the zone." Essentially, Gaeta needed to make virtual cinematography a reality.
When Neo and Agent Smith walk into the courtyard, we see the real actors Keanu Reeves and Hugo Weaving. But once the fighting starts, everyone and everything on the screen is computer-generated--including the perspective of the camera itself, steering through arcs at speeds that would tear any physical camera apart. Perhaps the most impressive aspect of the Burly Brawl is that it doesn't look virtual at all. The digital faces of Reeves and Weaving are flawless, the buildings look dreary and lived-in.
The standard way of creating a computer generated world is to build it from the inside out, building objects out of polygons and applying textures and lighting. For Reloaded, the special effects team took a radically different approach known as image-based rendering, loading as much of the real world as possible into the computer first, essentially building from the outside in. While the topography of the human face is difficult to simulate digitally, it turns out to be easy to map photogrammetrically, having few shadows and occlusions. To replace the faces of the stuntmen with that of Agent Smith, Gaeta and his team built a system for sampling the real at a higher resolution than had ever before been attempted, dubbing this process "universal capture". Once a scene is captured, filmmakers can now fly the virtual camera through thousands of "takes" of the original performance, from any angle, zooming in and out, or launching into the sky. Virtual cinematography.
But the ability to create photorealistic virtual human beings, and to cut-and-paste them into any landscape, raises unsettling questions. Questions that troubled Gaeta so much that, several years ago, he wrote a letter alerting President Clinton to the fact that such technology could be used for purposes of mass deception. The letter was never answered. It turns out that DARPA (Defense Advanced Research Projects Agency) is very interested in using image-based rendering and lighting for use in immersive battle simulations. In 1999, the US Army launched the Institute for Creative Technologies at the University of Southern California.
The paradox is not lossed on Gaeta. "You have these paranoid films about the Matrix depicting how people are put in a mental prison by misusing this technology, and you have the military constructing something like the actual Matrix. Or maybe our technology will become the actual Matrix, and we have inadvertently spilled the vial of green shit out onto the planet."