← avinash / llm-nas
EXPERIMENT 01 · CIFAR-10 · 12 NOV 2026

LLM-guided architecture search
compared with random search

A local Qwen2.5-Coder 3B proposed PyTorch architectures; random sampling was the baseline. With 30 proxy-training attempts per strategy, the best random finalist scored 92.72% on the test set versus 89.34% for the best LLM finalist.

One run per strategy · CIFAR-10 · 20-epoch finalist training · not evidence that random search always wins.

01 / THE EVIDENCE

Search progress and top candidates

Left: best proxy validation accuracy after each of 30 proxy-training attempts per strategy. The random run has 2 timed-out attempts, shown as flat steps. Right: the ten report-listed top-five candidates, plotted against parameter count. Neither plot shows test accuracy; finalist test scores appear below.

Two data plots. Best proxy validation accuracy across 30 proxy-training attempts per strategy ends at 69.08% for random and 66.68% for LLM. A scatter of ten reported top-five candidates shows random models generally larger; the highest random proxy score is 69.08%.
Figure 01 · Derived from runs/llm_qwen3b/archive.jsonl, runs/random/archive.jsonl and analysis/RESULTS.md. Hand-designed LLM seeds are included in the trajectory.Download 2400 × 1350 PNG ↗
RANDOM · #592.72%test accuracy · 801,326 parameters
LLM · #4889.34%test accuracy · 298,026 parameters

+3.38 percentage points for the random finalist. Search used validation accuracy. Test accuracy was evaluated once after training for each of the four finalists. These are different-sized models; the result does not isolate search strategy from model size.

02 / OPEN THE NOTEBOOK

Which architectures made the cut?

All ten top-five candidates named in the original report. Sort the table to compare proxy validation accuracy and size; only four finalists have reported test results. Proxy and test accuracy are different evaluations.

Top five reported candidates per strategy; proxy validation accuracy, parameter count and finalist test accuracy
Origin / idea

The left plot shows all 60 proxy-training attempts, but the table and right-hand plot show only the ten published top-five entries. Source: analysis/RESULTS.md ↗. Original lower-resolution report plot ↗

03 / HOW IT WORKED

Method

The search changes only model architecture, not the training recipe. The LLM runs on CPU while the laptop GPU trains candidates.

01 / PROPOSE

Proposal

Qwen2.5-Coder 3B mutates or crosses over selected PyTorch CNNs. Random search samples a hand-designed architecture space.

02 / VALIDATE

Validation

Static checks and a CPU dry run enforce shape and compute limits (≤1M parameters and ≤600 MFLOPs). The AST checks are not a security boundary.

03 / MEASURE

Training

A 5-epoch proxy trains on 10k images and scores on held-out validation data. The finalists retrain for 20 epochs on 45k images before the test evaluation.

SEARCH SUMMARY

Proxy search summary

The LLM's median non-seed proxy candidate reached 59.78% against 56.75% for random; the best proxy candidates reached 66.68% and 69.08%, respectively. Each strategy had 30 proxy-training attempts: 30 completed for the LLM and 28 for random search. Proposal counts exclude the 3 LLM seeds (46 LLM, 33 random). The two proposal spaces differ, and wall-clock time differed.

WHAT THIS CANNOT SAY

Limitations

The LLM repeatedly proposed depthwise-separable convolutions close to its seeds; its run logged 9 duplicates and 8 errors. Random explored larger inverted-residual networks. A noisy proxy, unequal model sizes and no repeated seeds limit any broader conclusion. Larger LLMs and ablations remain future work, not reported experiments.

READ THE EXPERIMENT

Source and report

See the source code, training recipe, published figures and report in the public repository. The run archives and checkpoints are not public. This page is a static account, not a live training demo.

Browse the source ↗Setup and limitations ↗