NanoChat: Build a mini chat model from scratch

By /

The project, open-sourced by Andrej Karpathy and designed to build a ChatGPT-like conversational system at a relatively low cost and in a short time, has garnered a lot of attention.

In an era where large language models such as ChatGPT, Claude, and Gemini have taken the world by storm, we are used to “calling” AI but rarely truly understand its underlying principles. Andrej Karpathy – former OpenAI researcher and head of Tesla AI – has launched an exciting open-source project called NanoChat, which allows us to build a chatting miniature language model from scratch.

1. Project introduction

NanoChat is a minimalist chat language model implementation project. Its goal is not to create a practical chatbot, but to help you fully understand the working mechanisms behind models like ChatGPT.

This project contains only a few hundred lines of Python code, but covers all the core parts of a language model—from data preprocessing to transformer architecture, from training flows to interactive conversations.

2. Project structure

NanoChat’s code organization is very clear, with almost every file corresponding to a learning phase:

fileFunction description:
train.pyThe main script for training the model
model.pyDefine the Transformer model structure
chat.pyThe entry point of the chat interface
data/Store training samples and corpora
config.pyModel and training parameter settings

3. Core principles

The soul of NanoChat lies in its simplified Transformer architecture.
It implements the following key mechanisms in handwriting:

  • Embedding: Convert text into points in vector space.
  • Self-Attention: Let the model “focus” on different locations in the input.
  • Position Encoding: Introduce sequential information into the sequence.
  • Causal Masking: Guarantees that the model only predicts future tokens.
  • Sampling: Generating natural language step by step based on probability.

By reading this short but highly enlightening code, you can visualize how an LLM “thinks” and “speaks.”

4. Project operation

Installation and operation are very simple:

git clone https://github.com/karpathy/nanochat
cd nanochat
pip install -r requirements.txt
python train.py
python chat.py

Once the training is complete, you can chat with your “small model” in the terminal.
While the answers may not be “smart” enough, this is a chat AI that you have trained with your own hands.

5. Why it is worth learning

Karpathy’s “nano” series of projects (such as nanoGPT, nanoLLM, nanoChat) have always been known for their “high readability”.
Instead of industrial-grade frameworks, they’re teaching-grade “microscopes” – allowing you to truly understand the underlying logic of AI models.

With NanoChat, you can:

  • Understand the inner workings of transformers;
  • master the complete process of training language models;
  • Experience the construction philosophy of LLMs like ChatGPT.

6. Summary

“NanoChat is not just a project, it’s an enlightenment lesson in intelligence.”

In the age of AI, understanding principles is more powerful than blind use.
Starting with NanoChat, dismantling the “black box” of large language models,
You will find that true wisdom is hidden in every line of code.

Github：https://github.com/karpathy/nanochat

Tubing:

Scroll to Top