AI in Autodesk Post

Autodesk is using Artificial Intelligence to teach its Flame 2020 software how to help creative artists do their job more efficiently when working on integrated VFX, color correction, look development and final mastering.

Artificial Intelligence (AI) was a leitmotif flowing through many of the introductions at April’s NAB Show 2019 and as usually happens, the most intriguing systems impacted by it were in post-production.

But what AI actually is sometime lies in the inner eye of the beholder, since AI can mean different things to different people.

So the series of articles of which this installment is the opening salvo is going to recognize that when delving into the implementation of AI in post, it’s useful to try to lasso this untamed topic with a definition.

Since we are beginning with some of the software released by Autodesk, I was very grateful when Will Harris, the Flame family product manager at Autodesk, avowed that to him, AI means, “Utilizing established methods for machine learning to produce an algorithm that can predictably produce results.”

In other words, you create a system that can learn, then teach it stuff, and let it autonomously do tasks for you based on what it has learned.

Facial recognition can determine not only identity, but also mood.

Facial recognition can determine not only identity, but also mood.

Harris started with the example of facial recognition, in which a variant of AI can identify different people by tagging their facemasks, but said that Autodesk has progressed the art far beyond that.

“In the Media and Entertainment space at Autodesk, we are taking the intelligence of VFX into the world of post-production,” he began. “For example, if you render something in a c.g. rendering engine you can produce a beauty image, but you can also produce render passes or something that is called AOV’s or Arbitrary Output Variables.

“They are the intelligence of the renderer spat out in different forms so you can post-process and adjust those renderings,” he continued. “A great example would be if in addition to that beauty image you rendered out a diffuse, non-lit version, and also a specular highlight and shadow pass. If you combine those all together, you could create a unique look to that face which did not necessarily exist before.”

Thinking in these terms lead Harris’s Flame 2020 development team at Autodesk toward the first way they decided to implement AI. Manually rotoscoping human images can be a very time consuming operation, but maybe AI could help alleviate this.

So now, with the release of Flame 2020 at the NAB Show last April, this result of AI learning is baked into the software.

Now, if you feed the system a source image and a reference Z-depth, it can produce a new image with the intelligence to identify the desired separation.

Flame lets you see the Z-depth created, and manipulate it.

Flame lets you see the Z-depth created, and manipulate it.

“The system also knows that sky is usually at the top of the image, and ground is conventionally at the bottom,” Harris said. “So we gave the software what is sometimes known as a ‘suck it and see’ tool to give it a help figuring that out.”

You might like to know that The Cambridge University English Dictionary defines “suck it and see” as UK informal for “to try something to find out if it will be successful”.

“What you end up with is sort of an automatic tool in Flame 2020 that you can try and see if you like the results,” Harris said. “If the outcome is acceptable, you keep it. If not, you try something else.”

A second implementation of AI in Flame 2020 has to do with what c.g. renderers call a “normals pass” in which you are trying to re-light a face or any object to create a different effect.

Suppose the original shot was too flat, and you need a more dramatic look. Now you can create a “normal map” in 3D using another AI feature of Flame 2020.

“Again, through the training of scanning 100,00’s of face images, we’ve taught the software how to identify various ethnicities, facial variations, skin tones and expressions,” Harris said. “We actually used many 3D renders and then taught the software how to apply what it had learned to real human faces.”

With this kind of image manipulation control...

With this kind of image manipulation control...

...it is amazing how you can alter the impact of facial expression.

...it is amazing how you can alter the impact of facial expression.

This is all you need to dramatically enhance the lighting on a character’s face.

“Or, you can invoke many different kinds of keyers by using AI as a reference,” Harris said, “using any shape or shot that you choose. It’s all a matter of training the system which shape you want to pattern the key after.”

“Even if it doesn’t work perfectly, it gets you 80% of the way there,” Harris finished up. “Then you can take it from there. This is only the beginning of the ways that Autodesk is planning on incorporating AI in our Flame 2020 and other softwares.”

Let us know what you think…

Log-in or Register for free to post comments…

You might also like...

The State of HDR

Over the century or so we’ve been making moving images, a lot of improvements have been dreamed up. Some of them, like stereo 3D and high frame rate, have repeatedly suffered a lukewarm reception. Other things, like HD, and e…

Color and Colorimetry – Part 2

At one time the only repeatable source of light on Earth was the sun. Later it was found that if bodies were made hot enough, they would radiate light. Any treatment of illumination has to start with the radiation from…

AI – ML – DL – The Differences

Artificial Intelligence is more than just one element. In this article, we look at and describe the many parts AI encompasses.

How Live Production Benefits From Software-Defined Switching

As the myriad of live competition television shows continue to attract new and ever larger audiences for TV networks, producing them live has become so complicated that a second technical director (TD), often called a “screens TD,” is now often bei…

TV’s ‘Back to the Future’ Moment?

Philo T. Farnsworth was the original TV pioneer. When he transmitted the first picture from a camera to a receiver in another room in 1927, he exclaimed to technicians helping him, “There you are – electronic television!” What’s never been quoted but lik…