ZeST (Zero-Shot Material Transfer) is a zero-sample-based method
Introducing ZeST, a zero-sample, training-free method for
(a)Image to image material transfer.
(b)ZeST can be easily expanded to perform multiple material edits in a single image, and
(c)Performs implicit light-aware editing on the rendering of the texture mesh.
abstract
ZeST, a method to transfer zero sample material to objects in an input image given a material sample image, was proposed. ZeST utilizes existing diffusion adapters to extract implicit material representations from example images. This representation is used to transfer materials using a pre-trained repair diffusion model on objects in the input image, depth estimation as geometric cues and grayscale object coloring as lighting cues. This method is suitable for real images without any training, thus realizing a zero-sample method. Qualitative and quantitative results from both real and synthetic datasets show that ZeST can output realistic images with transferred material. % We evaluate our method for qualitative and quantitative evaluation of a set of real-life images. We also showed ZeST’s application to perform multiple edits and powerful material assignments under different lighting.
overview of the methods of
The method consists of 3 branches. Given a material sample M and an input image I, we first encode the material sample using an image encoder (e.g., IP-Adaptor). At the same time, we convert the input image into a depth map and a foreground grayscale image to input geometric and potential illumination guide branches respectively. By combining two guiding sources with potential features of material coding, ZeST can transfer material attributes to objects in the input image while retaining all other attributes.
Results Gallery
If you want to learn more, you can click on the link below the video.
Thank you for watching this video. If you like it, please subscribe and like it. thank
Original text:https://ttchengab.github.io/zest/
Paper：https://arxiv.org/abs/2404.06425
Video: