Below the agent’s read budget, file and unit length is not a cost driver

What was tested

Sokrates' AI ease-of-change score and its AI Cost Estimator assumed that long files and long units make every change more expensive for an AI agent, because the agent has to read more to make the same change. The same claim was made with numbers in July 2026 in The Economic Benefit of Refactoring on martinfowler.com: after splitting a 17,155-line file, a change cost 83% fewer input tokens. This experiment measured the claim with repeats, controls and acceptance tests on a mature human-written codebase: Sokrates itself, as it was a year before the experiment.

Two variants of one codebase

Variant A is Sokrates at commit 5e6aa97a of 2025-09-20. Variant B is A with every main file brought under 500 lines of code and every unit under 50 lines and McCabe 25, by extract method and extract class only: 52 package-private helper classes in the original packages, every original class keeping its name and public API. The split was done in five parallel batches by coding agents under a strict brief, with the module tests and a golden-output check after every file: the reports both variants generate for a fixture are byte-identical across 850 normalized files.

The result: no difference

For every measure, the ratio of what variant B cost to what variant A cost, pooled over the ten target tasks (whose change lands in a split file) and separately over the two control tasks (whose files are byte-identical in A and B, so they should sit at 1.0). The pooled ratio is the geometric mean over tasks of the ratio of means, with a 95% bootstrap interval over runs. A value of 1.0 means the split changed nothing.

Per task

Input tokens per run, variant A and variant B, every run shown; the bar is the median. Tasks are ordered as in the task list; the two control tasks are last.

How the two agents read

The tool log of every run records each file read, with its line count, and each shell command with its output. The two models behave differently, and that is what decides whether length can matter.

The tasks

Real changes made to this code in the year after the snapshot, mined from the git history and re-derived against the snapshot. Each task is a behaviour description (what should happen, not where) and a JUnit acceptance test verified to fail on each untouched variant and to pass with a reference patch, on A and on B.

Compared with the martinfowler.com experiment

The two experiments agree on the mechanism and disagree on the result. His explanation for the 83% drop is that the agent read less code, not that there was less code; our logs say the same from the other side: what the agent reads is decided by how it searches, not by how long the file is. The difference in starting points is a factor of ten: his largest file was 17,155 lines, beyond what one read returns, so the agent had to page through it or read blind; ours all fit in one read, and a grep landed on one or two hits. If the cost of length is a step around the agent's read budget rather than a slope from zero, both results are consistent. That is the next experiment.

Conclusions

• Below the read budget, length is not a cost driver. Splitting files and units that already fit in one read did not change input tokens, output tokens, cost, turns or time, for either model. The target tasks sit at 1.0 with intervals of a few percent; the controls scatter more.
• The bill is the re-sent context. A run is 13 to 16 turns, and each turn re-reads the whole conversation as cache reads. The number of turns is set by how many greps, edits and test runs the task needs, and the split neither adds nor removes those on average.
• A split can cost reading. The weaker model, which opens files whole, looked at about a quarter more code in B, because the same feature spans more files there. The extra reading did not reach the token bill.
• Unit length did not matter either, for local edits. Seven target tasks land in units of 55 to 128 lines in A, split to under 45 in B, with no separation from the three tasks in short units. The agent greps for the line and reads twenty lines either side; it never needs the rest of the unit.
• For Sokrates: before the experiment its AI ease-of-change score rose from 6.6 to 7.7 for the split and the predicted context per change halved, while the measured cost did not move. The scoring was changed on 2026-10-09 in response: unit size and the 500-LOC file size no longer count for agents, a new measure counts only files beyond one read (2,000 lines), and context per change is capped at a 200-line window per touched file. Re-scored, the two variants sit at 7.2 and 7.3.

Limits

• One agent harness (Claude Code), two models, one Java codebase, tasks phrased with greppable terms, a mechanical split.
• The original layout is public and plausibly in the models' training data, which would favour A; with ratios at 1.0 it is not hiding an effect.
• Every task is a local edit. Changes that depend on state far away in a long unit, or files above the read budget, are where length could show, and this experiment did not create that situation.

Next

• The same harness on a codebase with a file far above the read budget, split to under it, to find the step.
• Tasks whose change is non-local inside a long unit.
• Sokrates' thresholds for agents moved to where the step is, not where the human thresholds are.

Method

• Runs. Claude Code headless (claude -p … --output-format json), the same prompt, tools, permissions and turn limit for every run, in a fresh one-commit copy of the variant with a neutral folder name. One worker per variant with its own Maven repository. Token counts are the CLI's own, with input split by cache state.
• Per run: prebuild, agent, a PostToolUse hook logging every tool call (file reads with line counts, shell commands and output size, edits, test runs), the diff, the acceptance test with the upstream modules built from the working tree, one row in the results file and a full record with the tool log.
• Analysis. Per task: medians and quartiles over runs. Pooled: geometric mean over tasks of the ratio of means, 95% bootstrap interval (2,000 resamples of runs within cells). Control tasks on identical code bound the noise.
• Reproduce. The repository holds the variants, the analyses, the tasks with acceptance tests and reference patches, the harness and every run's record:

harness/run.py --tasks <tasks> --variants a --repeats 10 --m2 work/m2-a            # one worker per variant
harness/run.py --tasks <tasks> --variants b --repeats 10 --m2 work/m2-b
harness/compare.py --csv results/runs.csv                                           # the pooled tables
scripts/check_tasks.py                                                              # every task fails before, passes after, on both variants
scripts/golden.sh b                                                                 # B generates the same reports as A
scripts/build_site.py                                                               # this page

Longer write-up: docs/posts/analysis.md. Turn-by-turn traces of example runs (what each turn re-sent, read and edited): trace.html.