microGPT for Music

Situation

Large language models look like black boxes because of their size and the libraries around them. Stripped down, the core algorithm is small. @karpathy's microgpt shows this by training a GPT on names in a single dependency-free Python file. I wanted to see whether the same tiny model could learn the patterns of music instead of names.

Task

Build a GPT that composes melodies, with:

• Zero dependencies: pure Python, no NumPy and no PyTorch.
• One file that holds the whole model, training loop and output.
• Output you can actually listen to.

Action

Building on @kaizenman's micromusic, I put everything into a single train.py.

1. Melodies as text

Each melody is one line of text. A note is two tokens, pitch then duration, and BAR marks a measure boundary:

C3 q C3 q D3 q E3 q BAR G3 h

Durations are w whole, h half, q quarter, e eighth and s sixteenth.

2. The model, from scratch

• A scalar autograd engine (Value) that tracks the computation graph and backpropagates gradients.
• A 1-layer transformer with 4 attention heads and 16-dim embeddings, about 6k parameters.
• The Adam optimizer, written by hand.

3. Training and sampling

• On first run the script downloads a dataset of about 11.5k melodies.
• It trains for 1000 steps (around 10 minutes on a single CPU core), then samples 20 new melodies.

4. MIDI output

• A small MIDI writer turns the samples into melody_01.mid … melody_20.mid (with --save), which open in any MIDI player.
• test.py checks the autograd engine's forward and backward passes on a small expression.

Result

After training, the model writes melodies that respect bar structure and stay in a key:

sample  5: G3 e B3 e B3 e C4 e BAR D4 e D4 e D4 e D4 e
sample 11: D4 q F#4 q G4 q G4 q BAR B4 q C5 q A#4 q

• Fully transparent: every gradient, attention head and optimizer step can be read in one file.
• Runs anywhere: Python 3.8+ is the only requirement.
• The domain can be swapped: changing the tokenization is enough to point the same algorithm at a new kind of data.