← avinash / experiments Source ↗

RESEARCH NOTE / 001 · SMALL MODEL, REAL MEASUREMENTS

The vocabulary
doesn’t have to live on the GPU.

Per-Layer Embeddings give each transformer layer a different view of a token. PLE-Lab is a small-scale experiment inspired by Gemma 3n: it trains a small GPT on TinyStories, benchmarks reading the PLE table from GPU, CPU and memory-mapped files, and lets you explore the learned token vectors.

Inspired by the per-layer embedding idea in Gemma 3n · independent small-scale reimplementation without Gemma 3n's AltUp streams; not a Gemma 3n benchmark.

PLE-LAB / FIELD NOTESSCROLL TO FOLLOW THE TOKEN ↓RTX 4050 LAPTOP · TINYSTORIES
01 / THE MECHANISMMODEL / OFFLOAD / INSPECTION

THE DESIGN QUESTION

Large table.
Small transfer.

A normal embedding gives a token one vector. Here a token has six separately indexed, jointly trained 64-dimensional slices—one for each transformer block. The token IDs of a forward pass are known before that pass runs, so their rows can be fetched from CPU RAM or a memory-mapped file instead of keeping the entire table in GPU memory.

For ple-small, one fp16 row is 768 bytes; the whole fp16 table is 38.6 MB. Offloading is a memory strategy, not free acceleration: the latency measurements below show the cost.

THE PATH THROUGH ONE BLOCK

Trace a slice

Pick a layer to inspect its measured single-layer ablation.

INPUTtoken IDknown in advance
OFF-DEVICE STOREE[token]6 × 64 values
SELECT THE SLICE ↓
64 values / layer
AFTER ATTENTION + MLPgate × projectionadd to residual stream
SELECTED L0

Removing only layer 0’s token input raises validation loss by +0.090 nats per prediction.

At each block, its token slice is combined with a projected initial-embedding context vector before the gated projection adds to the residual stream. The context term remains active in each single-layer ablation. Layers can compensate: their individual losses do not add up to the all-layer ablation. Model contract ↗ · Ablation definition ↗

02 / MEASURED, NOT ASSUMEDSELECT A VIEW BELOW

INTERACTIVE DATA PLATE

What changed
when we measured?

Four views of the recorded runs. Change the view or select a mark to read its source and context; every number comes from the linked project results.

FIGURE 01 / 04

A slight loss gap. Not a win.

Same dense configuration and 24.6M sampled training-token positions, one seed each. The PLE model has more parameters and took longer to train. A 0.009 nats/token gap from one seed cannot establish an improvement.

Select a plotted point to inspect it.

VIEW SOURCE DATA ↗
Training validation lossInteractive chart showing the measured baseline and PLE training validation loss.

TAP, CLICK OR TAB TO A MARK FOR ITS EXACT VALUE

+0.621 ± 0.008 SE

nats/token when the PLE token-table part is zeroed in all layers (context part active) · 25,500 predictions · token-level standard error, not run-to-run

0.872

Spearman correlation: PLE row norm vs. log frequency, for seen tokens

95.5%

prefill token hits with a 5% static-top-k cache on synthetic-big

Measurements: TinyStories, RTX 4050 Laptop GPU (6 GB), Windows/WDDM; one training seed. Synthetic-big is a generated table benchmark, not a trained language model. GPU throughput is noisy on this shared laptop (about ±30% between runs); offload results use a warm OS page cache. Benchmark methods ↗

03 / OPEN THE INSTRUMENTLOCAL APPLICATION / STATIC PREVIEW

THE PLE EXPLORER

Numbers have
neighborhoods.

Try the public token map ↗

Search a published subset of trained tokens and explore their vectors in your browser. The screenshots below show the full local app, which also runs model-backed text analysis and ablations. GitHub Pages does not run PyTorch inference.

04 / REPRODUCE THE EXPERIMENTNOT A HOSTED INFERENCE SERVICE

TAKE IT OFF THE PAGE

Run the full
Explorer locally.

The public browser explorer shows a subset of token vectors without a backend. Full-vocabulary search, live text analysis and ablations require the FastAPI + PyTorch app on your own machine.

Clone PLE-Lab ↗
WINDOWS POWERSHELLLOCAL ONLY
# Clone the source:
git clone https://github.com/Avinashistoocool/PLE-Lab.git
cd PLE-Lab
pip install -r requirements.txt
# Train a run first, or create a random demo run:
python -m explorer.make_demo_run
python -m explorer.server --runs runs --port 8765
# Open http://127.0.0.1:8765

The public repository does not include trained checkpoints: model weights and table binaries under runs/ are git-ignored. For the measured ple-small model instead of the random demo, prepare TinyStories and train both runs using README reproduction steps ↗.