Hybrid workflow for asset builds.
Is it possible to take a 3d model and let AI handle grooming & lookdev?
It is. But why would you when you can prompt whatever in minutes? You do this because you want to be able to direct performances.
Based off of our carefully crafted 3d model we let AI handle grooming & lookdev, two tasks that takes quite some time to do properly with standard VFX tools. With AI it still takes a bit of time getting things right but it’s definitely a faster process. It’s not possible to tweak and finesse as you would with a regular VFX asset and the final image is not a 1:1 match, but close enough.
Hybrid workflow for shot production.
Is it possible to animate the hybrid asset, light it and comp it into an AI generated plate?
It is. For the first few tasks we did everything as usual:
- Rig the asset.
- Animate the asset as usual.
We then use the 3d animation to drive the performance of the hybrid asset in our AI tools. As you can see here it starts in the classical “bind-pose” since that’s how the image of the asset was generated, this results in a wee bit of an offset in the actual animation but once it’s walking it reflects the animation dailie pretty well. Again, not a 1:1 match but AI smooths out minor kinks and also does the CFX pass (muscles, skin and fur simulations) at the same time. Another step that takes quite some time to setup and run on a per shot basis, so this is quite a time saver.
All in all doing things like this is a bit of a trade-off, you get what you get but it’s a lot faster than doing it with traditional tools.
Can we integrate it in a live action plate?
Yes we can!
- We undistort and track the plate as usual so we can animate within that 3d space. Standard VFX practice.
- We animate in the live action plate. And this is where the standard VFX practice ends.
- To get a start frame corresponding closer to the actual 3d output we generate a new start frame based off of the 3d.
- Based off of the plate and carefully crafted prompts lighting, CFX and comp are all handled by the AI.
- AI adds extra fidelity to the animation, a lot of little details that takes time to finesse traditionally. More often than not AI also creates all kinds of crazyness but with this approach we have a much higher level of control than you would with just prompting.
- Again, not a 1:1 match on the plate or performance but close enough. You win some and you loose some.
Can we the use the plate camera and generate an AI plate?
We sure can. Here we have a live action camera, a hand animated creature and AI generated backplate, fur, lookdev, lighting and comp. To do this we need to let the AI dream more freely hence some of the camera work is lost in translation.
Here’s one with the original background used as a reference for the prompt. Next one is something completely different while still keeping the bits and pieces we want to keep.
Wait, I changed my mind- a Bear in the northern woods is so cliché. I want it in an urban environment.
No problem. For this one we kept the asset, the animation and the overall camera movement but generated a completely different background.
And how about expensive FX work like water interaction?
Oh yes, we can do that too. Again we kept the asset, the animation and the overall camera movement but generated a new background that we interact with.