May 18, 2016

How 'The Simpsons' Used Adobe Character Animator to Create Live Episodes

When the Simpsons Live Corner was announced a few weeks ago, many speculated about how it would be realized. Would it use motion capture or markerless facial animation...

Ultimately, the three-minute segment in which Homer (voiced by Dan Castellaneta) answers live questions from fans was realized using Adobe Character Animator, still in development, with a mix of lip-sync and keyboard trigger animation. Cartoon Brew is proud to announce that David Silverman, producer and director of "The Simpsons," and Adobe's Senior Principal Scientist for Character Animator and co-creator of After Effects David Simons gave us the technical details.

But first, a live segment:

According to Silverman, the idea for live animation had been around for several years and was intended to take advantage of Castellaneta's improv skills. He says, "We all know that Dan is a great improviser. He's from Chicago's Second City and has had comedians like Bill Murray and John Belushi on the show." But it wasn't so clear what technology could be used to produce a live broadcast. Until the Simpsons team observed how Fox Sports' on-air graphics department was implementing live manipulation of the robot mascot, Cleatus. That led to an investigation of Adobe Character Animator.

Still a relatively new feature in After Effects CC, Character Animator is designed to animate layered 2D characters created in Photoshop CC or Illustrator CC, and to transfer real human movements are transferred into animated form. While this can be done with keystrokes, the real attraction of this tool is the conversion of the user's facial expressions into 2D characters via webcam and the user's dialog that drives the lip-sync.

In the live broadcast of The Simpsons, facial animation was not used, but lip-sync was used directly from Castellaneta's performance. The lip-sync portion works by analyzing the speech input and converting this into a series of phonemes." Taking the word "map" as an example, each letter of the word becomes an individual phoneme. The final step is to display what we call "visemes," which are the phonemes of the word "map. In the "map" example, the phonemes "m" and "p" can be represented by the same viseme. We support 11 phonemes in total, but recognize many more (over 60). This means that if you create a mouth shape in Photoshop or Illustrator and tag it appropriately in Character Animator, you can animate the mouth by simply speaking into a microphone."

Interestingly, when The Simpsons team was about to adopt Character Animator for live segments, the tool was then and still is in preview release version (now in preview 4). However, The Simpsons team was able to work with Fox Sports to produce a prototype of the Homer puppet with this software, which convinced everyone that a live Simpsons segment was possible. "To make sure the Simpsons team was using a very stable product, we created a new branch of Preview 4 called "Springfield." We knew that good lip-sync was a priority, so we put a lot of effort into tweaking the lip-sync algorithm so that the end result was broadcast quality"

.

In two live segments recorded for the program's West Coast and East Coast audiences, Castellaneta was in an isolated sound booth at the Fox Sports facility, listening to and answering callers, while Silverman was in a printed Adobe also implemented a way to send the output of Character Animator directly as a video signal via SDI to enable live broadcasts. Adobe also implemented a way to send the output of Character Animator directly as a video signal via SDI, allowing for live broadcasts.

So why was Silverman given the task of pushing buttons? The director, who has worked on the program almost since day one, admits, "They wanted me to handle the animation because I was familiar with it." I'm the one who invented many of the rules for Homer, and I've always been looked to as an expert on Homer. So they thought it would be a good idea to appoint someone who knows how this character sounds and works."

Of course, before the show could air, the animatable pieces had to be assembled. This was done in Photoshop by The Simpsons' animation team and converted to character animators. Says Silverman, "One of our animation directors, Eric Koenig, set up the animation stems that would be used." We prepared the animation for Homer to speak, all the dialogue mouths, the layout of the room, Homer raising his arms, turning from side to side, and blinking his eyes. Eric Kurland then worked with Adobe to do all the buttons and rigging for the character and set up the programming for it.

Various animations were developed but were not necessarily used in the final live performance.

"But Dan was improvising, so it didn't seem likely that he would ever make those catch phrases. Homer would say, "Just kidding, The Simpsons will never die," because it was a pre-written part of the script"

.

"Then there was a special animation where you could push a button and Lisa would come in, for example," Silverman added.

"Originally, I had all of these people come in at the push of a button, but in the end I had them all come in at very specific points. Also, initially the cut from wide shots to close-ups was done by another director, but one of the producers suggested that it happen automatically with the push of a button. That way I could concentrate all my attention on Dan's performance as Homer."

Silverman rehearsed with Dan several times before the broadcast, setting up the keypad and the half-second delay. He said, "People asked me what the buttons were that were hidden in the pictures of the keypad I showed them. I just hid the buttons I don't use. There were a bunch of buttons for the characters to walk around, but I didn't need those because I was doing that automatically. There were a couple of other buttons that we didn't think we would use."

Asked if he was nervous during the live broadcast, Silverman said, "I wasn't worried about anything. Everyone else was more worried than I was. Maybe it's because I'm a part time musician and have no worries about being on stage. Playing bass, especially on tuba, means a steady beat. Dan and I have known each other for decades now, and I knew exactly how he would approach it"

. [It is clear that there are many tools available today for live animation, from game engines to setups such as those used in Character Animator that allow for real-time, markerless facial animation.

Adobe's Simons noted that more software is being developed in this area. Originally, Character Animator only had the ability to use the camera to control the position, rotation, and scale of the head. Then we added the ability to look left, right, up, and down, assuming the artwork was drawn to match. There is a lot of room for innovation here. We may be able to do some clever things with parallax, and who knows what user requests we may receive. We also get a lot of questions about full body capture, depth cameras, and other input devices."

Adobe continues to develop other features of Character Animator. The current Preview 4 has improved rigging features that previously had to be set up in Illustrator or Photoshop. A feature called "Tags" has been added that allows you to select a layer in Character Animator and tag it with an existing tag name. There is also a new behavior called "Motion Trigger. This behavior triggers animation based on the character's movements. We still have some fundamentals to deliver, such as end-to-end workflows and further integration with After Effects and Adobe Media Encoder. We also want to improve the interoperability of our products for those who want to create recorded animation."

Silverman, who participated in a live segment of "The Simpsons," was pleased with the results." There were a few scenes where Homer smiled and nodded his head, something he doesn't usually do, but overall he thought it was great. He said, "If I had practiced more, I might have been a little more, you know, animated.

amzn_assoc_placement = "adunit0"; amzn_assoc_enable_interest_ads = "true"; amzn_assoc_tracking_id = "animationblast08"; amzn_assoc_ad_mode = "auto"; amzn_assoc_ad_type = "smart"; amzn_assoc_marketplace = "amazon"; amzn_assoc_region = "US"; amzn_assoc_linkid = "29d275a1274544d48da95696e660840c"; amzn_assoc_emphasize_categories = "229534"; amzn_assoc_fallback_mode = {"type": "search", "value": "Adobe"} amzn_assoc_default_category = "All";

.