Jul 4, 2017

Inside peter Jackson's new augmented reality studio

At Apple's recent Worldwide Developers Conference (WWDC) in San Jose, one of the stand-out demos was from Wingnut AR, the augmented reality studio started by director Peter Jackson and his partner Fran Walsh.

On stage, Wingnut AR's creative director Alasdair Coull demonstrated a tabletop ar experience made using Apple's upcoming augmented reality developer kit called ARKit and Epic Games' Unreal Engine 4. The experience blended a real world environment – the tabletop – with digital objects, in this case a sci-fi location complete with attacking spaceships, while being viewed live, on an iPad.

Cartoon Brew asked Coull more about Wingnut AR, how the demo was made, his thoughts on opportunities for vfx and animation artists in ar, and the future of ar from the company.

Based in Wellington, New Zealand, Wingnut AR was formed just over a year ago by Jackson and Walsh (Wingnut Films is the name of their film production company and the vehicle behind most of Jackson's movies). Coull told Cartoon Brew that the pair created Wingnut AR “to explore this exciting new medium and make content for the various ar platforms that are here or on the way soon.”

Wellington might seem like a long way from the main location of ar and vr startups in California, but the city is of course the home base for hundreds of visual effects, animation, practical effects, sound designers, and music artists thanks to the rise of Weta Digital, Weta Workshop, and Park Road Post (Coull was himself formerly the head of R&D at Weta Digital). The team at Wingnut AR also includes games professionals, crucial for realizing real-time content in ar.

“For the last year we've been in something of an R&D phase,” said Coull when asked what Wingnut AR has been up to. “We've been figuring out what is fun, and how we tell stories and entertain people in ar. There are a lot of challenges, both technical and creative, when creating exciting ar experiences and this past year has been about building the tools and doing a lot of interesting exploration.”

Apple's ARKit was one of the technologies Wingnut AR had been able to try out during this R&D phase. The ar company's idea was to create something that would push the available hardware and software. “As ar is such a new field, demos are often reasonably simple in their scope and we really wanted to make something more cinematic,” said Coull. “We created this world to tell a short little story whilst also really pushing the hardware and software to see what it could do.”

The idea behind ARKit is essentially a framework for building ar experiences. It lets creators lock their digital content onto some kind of object such as a tabletop, and then track the iPad or iPhone camera as someone moves the device around. On the device screen, it appears as if the action is taking place on the table or other object even during movement.

To make the the action in the town and the attack of the fighters, Wingnut AR worked in the game development tool Unreal Engine 4 (UE4). But when the company started on the project, there was actually no existing ARKit integration into the game engine.

“So our developers Dan Smith and Andy Styles built a plugin to hook up ARKit and UE4 that let us use the power of ARKit quite easily within UE4,” explained Coull. “During the project, Epic Games got very involved and they've now taken ownership of the plugin, tidied it up, and have released it to the world, so anyone can use UE4 [with ARKit].”

The WWDC demo ran on a new iPad Pro at 60 frames per second, which impressed Coull. “As with any real-time project, we had to be careful about how we created assets, and in particular how the fx were constructed to keep that frame rate up. We built some custom tools to help us optimize effects after they are simulated and that helped a lot at keeping the performance up in key points, like when the building explodes at the end.”

That explosion obviously made use of existing skills and toolsets in gaming and vfx to pull off, within the confines of what could be produced on a portable device. It was crafted by Gray Horsfield. “He just made something super cool with little restrictions,” said Coull. “We then figured out the technology we needed to create to take that awesome explosion and make it run at 60 fps on an iPad.”

Indeed, artists with skills in vfx and animation are already extremely well-suited to creating ar content, especially where cg modeling and rendering, stitching imagery, tracking, and compositing are concerned. Coull points out there are new challenges that ar brings, too, such as no longer having control over the camera. In traditional narrative and even in some vr content, a content creator tends to make careful decisions about camera movement and editing. But in ar, the camera can move anywhere.

“Even in a reasonably contained environment, with a trained camera operator like this stage demo, the camera movement is different each time,” said Coull. “With the viewer being at a huge scale over the town, moving a foot left or right, translates to nearly 20 feet in the scene scale. That can make a huge difference in whether a ship flies past the camera in a cool sweeping motion, or you miss it completely.”

These camera issues are also faced by vr creators, but ar has the added complexity of not necessarily having control of the environment (i.e. users will have different rooms and tabletops). That has led Wingnut AR and Coull to question how to set the right kind of tone or feelings of immersion during ar experiences; testing in this regard is on-going.

And immediately following the WWDC, many wondered whether the Wingnut AR demo might end up becoming some kind of playable game. But Coull says that's currently not on the agenda. “At this point we don't have solid plans to release a game around it,” he said. “Ar is a very new platform and we're really just coming out of this R&D phase and starting to focus on consumer content, and we want to ensure we get it right.”