THE TOKEN ATLAS
One token.
Six places.
Move through the six learned embedding spaces. Search a token, rotate its neighborhood, and read the actual 384 published vector values.
Trained small GPT on TinyStories; not Gemma 3n.
Vocabulary in space
This 3D PCA projection explains only a small fraction of the embedding variance. Distances on this map do not replace cosine similarity in the full-dimensional vectors.
Look up a row
Search matches the published top tokens only. Press Enter to jump to the first match; use ↑ / ↓ to choose another.
Select a token
Six slices × 64 values
QUANTIZED INT8 × ROW SCALEHover or focus a cell for its reconstructed value.
Per-layer vector norms
Nearest neighbors
Exact cosine on reconstructed vectors, restricted to the published top 2000 tokens (or fewer if the export contains fewer). Not nearest neighbors across the full model vocabulary.
A small table,
not a model in your browser.
These are pre-exported rows from a trained small GPT on TinyStories, not Gemma 3n. Each token has six 64-dimensional learned PLE slices. The map stores three PCA coordinates per slice; the heatmap and cosine search reconstruct vector values from signed 8-bit integers multiplied by each row’s published float32 scale. Quantization means these approximate the original float vectors; similarity is exact for this published, quantized subset.
This page cannot generate text, run PyTorch, or perform live ablations. To try the full model-backed explorer locally, clone PLE-Lab ↗, install pip install -r requirements.txt, prepare a run (or python -m explorer.make_demo_run for a random demonstration), then launch python -m explorer.server --runs runs --port 8765 and open http://127.0.0.1:8765. For trained results, follow the training instructions ↗; the random demo is not the measured model.