Aug 15, 2016

Who does what with CGI characters: Many manufacturers of Elliott in "Pete's Dragon"

Just about any film is a huge collaborative undertaking, and that's especially the case with Hollywood films featuring extensive visual effects. But we don't always get the chance to talk to the individual artists at a visual effects studio responsible for bringing a character to life or crafting a complicated effects shot.

So, with the release of director David Lowery's Pete's Dragon, we decided to ask a number of artists at Weta Digital working in different disciplines – from paint and roto to creature design, modeling, effects and compositing – how they contributed to the making of the central character Elliot.

Eric Saindon, visual effects supervisor: As the visual effects supervisor it is my responsibility to interpret the director's vision and make it happen on screen. I work closely with a great team of artists that all help to bring the thinking, feeling character of Elliot to life. My main goal is to make sure that everything we produce is of the highest quality and integrates seamlessly into the main body of the film without the audience being aware of the work that goes into it.

A great example of this is the scene with Pete coaxing Elliot out of his cave to be introduced to Grace, Meacham, and Natalie. We get to see him really close up in a tricky lighting situation. The dappled light through the trees had to be matched by our lighting artists while getting all of that hair to render was a challenge. I made sure that the level of detail was there in those scenes including a lot of subtle animation in the face and the movements in his whiskers and facial hair. Tiny highlights in the eyes were carefully art directed and the shaders team had to get just the right levels of translucence on the skin, shininess on his nose as well as the right balance of elements to make his teeth feel just right. All in all, it is the attention to details that makes Elliot not only hold up that close but truly come alive.

Marten Coombe, visual effects producer: As the visual effects producer on Pete's Dragon, it is my job to make sure that both the client and Weta have an aligned vision throughout the production. Elliot's creation was very much the product of our fantastic vfx team led by Eric Saindon. From the Art and Models team to the work of the Creatures team and Elliot's 20 million hairs, everyone threw themselves into the project and it's why he looks so good. The animation team led by Mike Cozens really shone on this project. You can see it in every frame of the shot of Elliot and Pete playing in the river. There is a lot of careful observation and subtle moments that show the skill and artistry involved in bringing the dragon to life.

It was really important for the audience to relate to Elliot, and to feel for him and his relationship with Pete. The animators spent long hours reviewing reference footage of all kind of animals to capture the essence of who Elliot is without relying on human characteristics. It allows the viewer to really believe in him and to never be pulled out of the story.

Gios Johnston, head of creatures: Elliot is a unique creature – he is over seven-meters tall with emerald-green fur, and has a playful, intelligent personality that had to be expressed through his behavior and look, rather than dialogue. As the Head of Creatures, I oversaw the team from the Creatures department who collaborated to build and support the various rigs and simulation elements for Elliot: Aaron Holly, Carlos Lin, Tim Teramoto and Chris McConnachie. Producing a fictional character that believably performed alongside his human co-star was achieved through a team effort.

We've created large dragons at Weta Digital before, most notably Smaug for The Hobbit trilogy, but we'd never created a big, green, furry dragon. It was the size of Elliot, the shear amount of fur required to cover his body (20 million strands) and Pete's interaction with Elliot's fur that were the hardest things to get right. We had several very close-up shots where Pete was running his hands through the fur. Making that look good and believable was definitely a challenge.

Because we needed to simulate all that fur, the turnaround time from animation publishing to shots receiving something to light took a long time. That meant we needed to be careful to make sure that the animation we were receiving was nice and clean, before we started our simulations. There are lots of flying shots in the movie and dialing in the right amount of wind for a dragon flying at very fast speeds also took some doing.

Jedrzej Wojtowicz, senior head of department: look-development: I oversee the Shader and Textures Departments and the overall development of look-development technologies and process. Textures and Shaders crew were involved from the early stages of Elliot's development; in the beginning a lot of the work was more compositional in nature, working with the vfx supervisor as well as the other Departments, helping to explore the overall look of the character.

The most challenging aspects of our Elliot work were conceptual in nature: Elliot is not a real thing – he is a mixture of animals, real and fantastical; he is inspired by illustration but needs to exist as a natural, photoreal character. Elliot is also both endearing and imposing. We had to pay careful attention to the details of his huge scale and the nature of his green fur. Does a gigantic hairy dragon have hairs that are a centimeter thick or does he have a thousand times more hairs, or something in between- What different color variations of green fur look natural-

While our mission was to create a believable realistic creature, we couldn't disconnect him from his illustrative and playful past. There was exploration in a variety of areas, referencing a number of real-world animals for ideas on patterning and colouration right down to the details such as the structure and look of his irises, the feel and coloration of his skin, including the color and depth of light scattering through his wings.

At each step of developing his appearance we'd find ourselves bouncing between the constraints of preserving the soft and approachable quality, maintaining his grand scale and gravitas-photorealism vs. character, structure vs. softness. While there was one Elliot, we eventually had a number of manifestations we would utilize for the different camera vantage points. We also had alternate states of the fur geometry and appearance we developed and could invoke when appropriate. For example, we spent time working towards a “close up” look, which pushed the visual complexity of the fur, and adding minute imperfections to the fur and tiny pieces of debris helping the audience believe that he was a large beast.

Andreja Vuckovic, senior modeller: My role was to build a digital model of Elliot based on a digital maquette that we received from Disney. I also made displacement maps for the details all over the body, adding spikes on the back and the neck and details on the paws and claws, or fine bones structures for the wings and facial details for the lips, nose, or his broken tooth. I was involved in the character development process. There were many changes to improve the model, technically and artistically, from the concept design to the final model. Requests for those changes were coming from different departments: models, textures, creatures, art department, animation, shaders or from the vfx supervisor, and the client.

The most challenging technical and aesthetic aspect was integrating Elliot's wings with his body to look functional and anatomically correct. Adding shapes without a good reason had the potential to look lumpy and odd in the final shots or may have caused an issue in simulation. I had to be sure that the wings were a good size and shape, and had good proportions and look natural when folded. The accuracy of our physically-based render model leaves little room for error and the overall look had to be astonishing.

Nick Gaul, senior modeler: I contributed by grooming and maintaining all the various grooms for Elliot through out the show. This would include anything from adding debris into the groom such as leaves and dirt, and groom variations such as a 'wet' groom.

The super close-up shots presented a whole other challenge with the hair on Elliot. For those shots we needed to add millions more hairs to simulate all the fine interaction between the actors and Elliot's hair. We couldn't possibly have that level of density on the entire groom so we would make custom grooms that accounted for higher density of hairs in the areas we'd see up-close in specific areas. For this we ended up creating 28 custom grooms.

Pete Godden, lead layout technical director: In the layout department at Weta Digital, our task is to create immersive environments and backdrops in which to frame digital characters such as Elliot. The lush forest playground that Eliot has made his home is an integral part of what makes up his rich personality. The incredibly beautiful forest and bush lands that are found only in New Zealand naturally seem magical and breathtaking, providing the perfect backdrop for our magical Dragon. We took frequent visits to nearby forest parks and referred to a vast library of photos in order to enhance location shots and, in some cases, build entirely digital sections of the environments. The director considered it of utmost importance that Eliot would be framed within a beautiful, lush natural environment, as this is integral to his character.

Ant Webb, senior paint artist; Ryan Hutchings, senior paint artist; Alice Collins, senior roto artist; Chris Templeman, senior roto artist; Hanna Stewart, senior roto artist; Gareth Thomas, senior roto artist

Our job as paint and roto artists on Pete's Dragon was to remove the large green proxy, which is a stand-in for Elliot that the actors could perform with. The paint work was extremely difficult due to the Elliot proxy moving in and out of trees, fine branches and bush.

The forest was constantly moving in the wind, which made it difficult to replace parts of the background. To complete the paint work, each individual tree had to be tracked in and the movement of the branches animated. Without the great cameras and decent surveys that the Camera Department did of the trees and bush, it would have been a near impossible task.

One of the many roto tasks was to make Elliot the Dragon really live in his environment. By rotoing individual trees and shrubs, it meant the Animation Department could really immerse Elliot amongst the forest.

Mike Cozens, animation supervisor: Elliot's character development began with posing and moved quickly into motion study work. We started with initial pose tests of Elliot matching a variety of animals to see how he looked in different postures. We also did posing for facial expressions in order to see how far we could push the facial expressions and still have a believable character. Elliot needed to be a believable animal but he also needed to communicate with Pete (and the audience) without being anthropomorphic. We needed his performance to be based on recognizable animal behaviors and characteristics.

Our motion study work gave us to reference to build his performance through a variety of recognizable animal characteristics so that when he needed to communicate he did so as an animal would. We were able to build a library of motion we could share amongst artists to keep his performance “on model.” The motion studies also allowed us to play with his huge size and understand how a creature as big as Elliot would move in a variety of circumstances-from delicate emotional moments to broad action beats. For example when we needed him jumping and making a big splash we used reference of an arctic fox, but changed the mechanics to move with the weight of a big creature. Alternately, when Elliot first meets Natalie we used subtle head ticks and face behaviors that mimicked a dog or a cat. We did this over and over again and carefully narrowed down the things that worked to create a specific set of characteristics that were uniquely “Elliot.”

Elliot's flight style took a lot of development. He is a big dragon and initially we tried to make him very aerodynamic by tucking his limbs close to his body so it was believable that he could stay aloft. Unfortunately the posing wasn't working. An animator (Graham Binding) found a solution in two pieces of reference: one was the Pegasus flight from Disney's Fantasia and the other was a video of polar bears swimming under water. In both pieces of reference the animals use their limbs to swim through the water and air. When we translated that onto Elliot's flight we immediately knew we had a solution. Seeing his gangly limbs swim through the air became a part of his character and allowed us to play with his awkward flight in a fun and comical style.

One of the technical challenges was keeping Elliot's behavior animalistic and not becoming too human with his performance. There is a key scene at the end of the film where Elliot needs to communicate something important to Pete but we had to do this using his animal physicality and body performance. It was an interesting challenge finding behavior that communicated Elliot's intention in a clear way but kept him in character as an animal.

Yann Moalic, effects TD: As an FX artist, my job is to recreate the physical phenomena that ground us in reality. On Pete's Dragon I had the opportunity to immerse Elliot into the forest environment in a physically plausible way. We created plant simulations to blend Elliot into the scene and recreated a section of the forest with CG elements. This gave us the freedom to animate every single plant from the trunk to the tips of pine tree needles.

An interesting shot from the trailer I worked on is when Pete jumps from a cliff and Elliot takes him flying above the trees. Our plant dynamics are done with an in-house tool called Onodrim. This is a physical solver, so we specify the material properties of each tree (wood density; rigidity; shear; young's modulus; internal viscosity; plasticity) using physical values. Then we use Elliot's animation to generate a velocity field to control the forces applied to the plants. We get the result of the simulation almost in real time so that gives us great control.

Karl Sisson, senior matte painter: Pete discovers the magic and wonder of flight with the help of Elliot when he leaps off a cliff in free-fall and is scooped up by Elliot to soar through the clouds. Part of the excitement of flight is danger and we needed the right camera moves to show this – but having a helicopter do a barrel roll thought a narrow ravine was not an option.

VFX Supervisor Eric Saindon tasked me with creating a photorealistic virtual ravine environment, which would allow us to have a digital version of Pete and Elliot barrel roll ridiculously close to the river. The key challenge on this shot was having 3D matte painting start and end with filmed helicopter footage. Not an easy task. Trees and river walls moving in parallax along with flowing water would be required. In the end this was achieved using a mix of optical flow, photogrammetry, and straight up matte painting projection. To help bring Pete and the audience along for a ride with Elliot was a team effort and was a great joy to work on.

Nick Epstein, CG Supervisor: My team was responsible for lighting Elliot in all of the forest sequences, roughly half the film. Our work also included partially and fully digital environments such as the cave interior, tree house, river, and various areas of forest.

The biggest general challenge was the realistic integration of a huge, stylized character into very naturally lit plates. On top of our usual onset data acquisition techniques, we used naturalistic elements to light Elliot's eyes-the sun breaking behind a tree canopy, firelight, and specular kicks from water or glass would all be used to emphasize Elliot's emotions while still keeping him grounded in the scene.

Elliot's action was mostly shot with only a model of his head as a stand-in, meaning interaction with the forest always had to be digitally created and given his size, it often meant large areas of foliage needed to be completely reconstructed. Trickier than that was the integration of Elliot with the actors, most of whom had hands, arms, and most often hair tracked, simulated, lit and digitally replaced at different points. A particular challenge was the shots of Elliot and Pete playing in the river. Most of the live-action water had to first be matched to maintain Pete and his practical interaction, then simulated against Elliot so he could be properly integrated. Finally, subtleties such as caustics and even the reflection of Pete in Elliot's eyes were added. In the film, the shots go by pretty quickly but a lot of love went into them.

Ben Roberts, compositing supervisor: Our job was to integrate a giant, green fury dragon into the environment. We also had loads of set extensions to integrate into the plate photography and a ton of green screens to extract and combine with CG environments. Elliot was not the only 3D asset we had to work with though; we had to comp a large variety of plants, trees, foliage, fire, smoke, vehicles, roads, bridges, and a few animals. Matte paintings were recreated with plate photography, CG, and photographs, and used extensively in a few of the key sequences.

For Elliot we had to be careful to keep true to this performance, making sure we could still read his emotion, whilst sitting him into a green forest. You had to read him, but he also had to blend in. That in itself took a little bit of balancing to get the look right, especially given his color and contrast.

Scenes integrating Pete (Oakes Fegley), Natalie (Oona Laurence) and Elliot together took some nutting out to get the interaction and scale right, along with the lighting. Of course Oakes wasn't interacting with a dragon so tweaking the photography and CG just took some time and thought to blend it together. Some shots were so tricky that in the end, it made more sense to just replace parts of Pete and Natalie with CG, to get the interaction with Elliot just right. See if you can spot them!

We worked closely with the lighting department and our deep compositing engineer to replicate the volumetric light rays seen on set, often filled with smoke with directional lights. Elliot was lit and rendered using our regular tools, but we augmented this by using these propriety tools to create deep volumes that could be shadowed by geometry. This technique, called shadow sling, helped replicate the way the thick volumes react with light and shadow and provided a much more natural feel to our shots. We could easily deep composite these volume rays with the dragon, our forest assets, and the plate photography. We also used in-house tools to enhance and sometimes create some of the dappled lighting on Elliot, often seen under tree canopies.

Elliot's invisibility effect was also a challenge. Originally, director David Lowery wanted to go with a camouflage approach where Elliot would take on the colors of his environment. This was done using modified plate projections and highly detailed c-ref passes of Elliot, but would only work if he wasn't moving too much. In the end we added to this effect by introducing a more transparent effect later in the film. Like all creative ideas, this evolved as the film progressed and Elliot's performance was honed. We generated hundreds of iterations throughout this process and eventually settled on whatever worked for the shot, especially since reading Elliot's performance was sometimes just as important as hiding him.