Elegance - complete GPT from training to inference in 200 lines

[Knajjd]

200 lines of code, no libs except for standard libs.

From the author:
"This file contains the full algorithmic content of what is needed: dataset of documents, tokenizer, autograd engine, a GPT-2-like neural network architecture, the Adam optimizer, training loop, and inference loop. Everything else is just efficiency. I cannot simplify this any further."

Source:

microgpt

Musings of a Computer Scientist.

karpathy.github.io

Code + Timeline:

The Art of GPT

And here is the complete code:

 

[Maikowski]

scam

 

[Knajjd]

scam

What's a scam?

 

[ZX888]

200 lines of code, no libs except for standard libs.

From the author:
"This file contains the full algorithmic content of what is needed: dataset of documents, tokenizer, autograd engine, a GPT-2-like neural network architecture, the Adam optimizer, training loop, and inference loop. Everything else is just efficiency. I cannot simplify this any further."

Source:

microgpt

Musings of a Computer Scientist.

karpathy.github.io

Code + Timeline:

The Art of GPT

And here is the complete code:

View attachment 1683755

What's even the purpose of this

 

[Knajjd]

What's even the purpose of this

It is considered remarkable for structural and conceptual reasons,
not performance.

1) It compresses a modern Transformer language model into a single,
dependency-free Python file. Most real systems rely on large
frameworks (e.g., PyTorch), GPU kernels, tensor libraries, and
distributed infrastructure. microgpt removes all of that and
still reproduces the essential algorithmic structure of GPT.

2) It implements its own automatic differentiation engine.
Backpropagation via the chain rule is central to deep learning.
Building a working autograd system from scratch — and making it
function correctly through a Transformer — is technically
non-trivial.

3) It reveals how compact the intellectual core of GPT models is.
Attention, embeddings, normalization, nonlinear layers,
cross-entropy loss, and Adam optimization are sufficient to
create a functioning generative model. The mathematics itself
is not enormous.

4) It is pedagogically powerful.
Every step — raw text → tokenization → embeddings →
attention → loss → gradient → parameter update — is visible
and inspectable. Modern frameworks often hide this structure.

5) It clarifies the role of scale.
The difference between microgpt and large production models
is not new mathematics. It is scale: more parameters,
more data, more compute, and heavy engineering optimization.

In short, it is remarkable because it reduces a GPT-style
language model to a compact, readable artifact while preserving
the complete training and inference loop.

 

[IggyCope]

Very interesting. Thanks.

I want to start studyng LLMs in deep. I feel curiosity by small LLM as llama2.c and others, and to analyze their code.

 

[Knajjd]

Very interesting. Thanks.

I want to start studyng LLMs in deep. I feel curiosity by small LLM as llama2.c and others, and to analyze their code.

I found the link here which carries a lot of AI articles that the mainstream sites don't carry.

Hacker News

news.ycombinator.com

I posted this a few months back which you might also find interesting. It's about the story of how two failed techs, CUDA and neural nets, that when combined created AI.

Incel Invented AI

Photo of Hinton, Sutskever and Krizhevsky. In 2012 Krizhevsky purchased two Nvidia GeForce GTX 580s. He combined the CUDA tech on the GPU's with neural network tech to develop the first artificial neural network. Prior to this both Nvidia's CUDA and neural networks were considered failures...

incels.is

 

[svgmn1]

Are you the real knajjd?

 

[Chud Norris72]

Are you the real knajjd?

no he's not

 

[svgmn1]

no he's not

he is the fake knajjd

 

[Chud Norris72]

he is the fake knajjd

I wonder how that troon is doing now.