You only need to provide one or two reference images, and the system will color the input sketches based on the color and style of the reference images.
ToonCrafter can adapt to different animation styles, whether it is simple line animation or complex cartoon scenes, it can be easily handled.
The following content is from the original translation
We introduced ToonCrafter, a new method that goes beyond traditional communication-based cartoon video interpolation and paves the way for generating interpolation. Traditional methods implicitly assume linear motion and have no complex phenomena like de-occlusion. They often struggle with exaggerated nonlinearities and large motions with occlusion common in cartoons, resulting in unbelievable or even failure of interpolation results.
To overcome these limitations, we explored the potential of adjusting real-life video priors within the generation framework to better accommodate cartoon interpolation. ToonCrafter effectively solves the challenge of applying real-life video motion priors to generate cartoon interpolation.
First, we designed a cartoon rectification learning strategy that seamlessly adapts real-life videos to the cartoon field a priori, solving the problem of domain gaps and content leakage. Next, we introduced a dual-reference-based 3D decoder to compensate for lost details due to the highly compressed potential prior space, ensuring that fine details are preserved in the interpolation results. Finally, we designed a flexible sketch encoder that allows users to interactively control the interpolation results. Experimental results show that the proposed method can not only produce visually convincing and more natural dynamic effects, but also effectively process de-occlusion. Comparative evaluations show that our method is significantly better than existing competitors.
If you want to learn more, you can click on the link below the video.
Thank you for watching this video. If you like it, please subscribe and like it. thank
GitHub：https://github.com/ToonCrafter/ToonCrafter
Original text:https://doubiiu.github.io/projects/ToonCrafter
Arxiv：https://arxiv.org/abs/2405.17933v1
Oil tubing: