Open source tool to turn sketches into stunning 3D drawings

By /

Vibe Draw, through a simple drawing interface, allows users to draw freely on a 2D canvas, then uses the “Improve Drawing” function to refine the sketch, and then converts it into a 3D model through the “Make 3D” button.

This GitHub project is an open source AI tool that aims to quickly transform users ‘hand-drawn sketches into interactive 3D models, lower the technical threshold of 3D modeling, and is suitable for many fields such as education and creative design.

Project introduction

Vibe Draw converts from sketches to 3D models through the following steps:

  1. sketching: Users draw freely on the 2D canvas.
  2. optimization sketch: Click the “Improve Drawing” button, and the AI will optimize the sketch into a finer image.
  3. generating a 3D model: Click the “Make 3D” button and the system converts the optimized image into a 3D model.
  4. Building a 3D world: In the “3D World” tab, users can combine multiple 3D models to build a complete scene.
  5. iterative optimization: Users can further optimize the model by drawing again or entering text prompts such as “make the castle teller”.
  6. export models: One-click export is standard .glTF Format for easy editing in tools such as Unity and Blender.

🧠 Technical architecture

  • front end: Build using Next.js and React, combine TLDRaw to achieve 2D drawing, Three.js to achieve 3D rendering, and Zustand to manage state.
  • rear end: Based on the FastAPI, Celery is used to process asynchronous tasks, Redis implements real-time communication, and SSE (Server-Sent Events) is used for real-time progress updates.
  • AI support: Integrate multiple AI models such as Claude, Gemini, Cerebras and PiAPI for image recognition and natural language processing.

Focus on project highlights

  • education application: In children’s education, Vibe Draw can transform children’s simple paintings into a vivid 3D world, stimulating creativity and spatial thinking skills.
  • open source sharing: The project adopts the AGPL-3.0 license, has more than 1.8k stars and 263 forks, and the community is active.
  • rapid deployment: The front end uses Node.js 18+, and the back end uses Python 3.10+. Docker deployment is supported, making it easy to get started quickly.

Project author

The project was developed by Martin Sit, a computer science student at the University of Waterloo in Canada, who interned at Shopify and has experience in multiple fast-going AI projects.

Github：https://github.com/martin226/vibe-draw

Oil tubing:

Scroll to Top