I wanted to sanity-check a common coding-agent claim: if a plugin compresses one piece of context by 90%, does the complete task actually become close to 90% cheaper?
The matched task was a repository rewrite: reproduce the Rust eza CLI as a behavior-compatible Python implementation and face 52 harness assertions. All six runs used GPT-5.6-sol High and Codex CLI 0.144.1, with two runs each for no plugin, Ponytail (full hook + skill), and isolated RTK.
The averages were:
- No plugin: 78.85% harness score, 6.660M tokens, $5.282 modeled cost, 62.5 rounds
- Ponytail: 80.77% score, 7.56% fewer tokens and 8.87% lower cost, but 13.51% longer duration
- RTK: 76.92% score, 13.20% more tokens, 7.18% higher cost, 44% more rounds
The larger warning is variance. Cost range divided by mean within each two-run arm was 43.25% without a plugin, 51.69% for Ponytail, and 30.78% for RTK. With n=2, these are not confidence intervals and do not identify a plugin effect. The small mean changes sit inside much larger run-to-run swings.
A separate set of 140 Codex runs contained 901.6M tokens and $680.34 of modeled API cost. Cached input was 96.46% of tokens and 63.91% of cost; model output was only 0.38% of tokens. That denominator mismatch seems important: a large reduction in a prompt fragment or one class of shell output may barely move the complete trajectory.
My current view is that token-saving tools should be compared on cost per verified completed task, with repeated-run variance, rounds, latency, and task quality reported together. What minimum number of repetitions and which success metric would you consider credible for this kind of agent benchmark?
Full methodology, per-run data, and limitations:
Project repository:
Disclosure: I maintain Tura and wrote this analysis. The plugin comparison contains only two runs per arm and is not presented as a causal estimate.