Which model and effort level should the CI rule check use?

For engineers who run an LLM as a judge in CI. Published 2026-11-10. This is a measurement on one repository's rules and diffs, with a small sample. It shows a pattern, not a precise figure, and the caveats below apply to every number on the page.

See the results ↓

TL;DR

  • The CI rule check (strspc check) sends the rules in the configured states and the diff to Claude, one request per owning entity. It fails the check when a rule in a fail_on state is judged violated, and other violations are warnings. In strspc-manager's config, Implemented rules block and the others warn (check reference).
  • On 18 hand-written diffs with known answers, run 3 times each, Haiku 5.5 at medium effort got 54 of 54 right. Sonnet 5.5 at medium got 51 of 54, and all 3 of its wrong answers came from one ambiguous case (insight 3).
  • One CI run cost $0.047 on Haiku 5.5 and $0.855 on Sonnet 5.5, about 1/18, averaged over 8 real diffs, each evaluated in 25 entity groups, with prices as set in the benchmark source (Caveats).
  • Higher effort on Haiku did not improve the labelled result, which was already perfect. On 8 real commits, one run each, it flagged more violations, including 2 that would have blocked a merge: 1 real catch and 1 false positive (insight 2).
  • Limits: 18 cases, 8 commits, one repository, strspc-manager's provider on its develop branch rather than the released CLI.

// 1. The question

Sonnet 5.5 made the check expensive. The proposal was to switch to Haiku 5.5 and raise the effort level to make up for the smaller model. We measured both ideas before changing anything.

// 2. Method

The benchmark is src/ruleeval/claude/bench_test.go in strspc-manager, behind a bench build tag. It calls the provider as strspc check does, except that it does not model inheritance between entities: one request group per entity, holding that entity's Draft, Published and Implemented rules, with a detail request for each violated verdict. Two case sets answer two different questions.

Labelled cases (accuracy). 18 hand-written diffs. For each of six rules whose answer can be checked mechanically, one diff breaks the rule, one keeps it and one does not touch it. The six rules:

  • GOCONV-004: wrap errors with %w.
  • GOCONV-005: context.Context first for network work.
  • GOCONV-007: exported identifiers need a doc comment.
  • MOD-001: no logging or printing inside a module.
  • MOD-005: no imports of strspc-CLI.
  • MOD-006: every diagnostic code is described in the spec.

Each case ran 3 times per config, so an unreliable verdict can be told from a wrong one. “Violated” is the positive class, because it is what blocks a merge.

Real commits (cost and realism). The diffs of the last 8 commits on strspc-manager's develop, from #327 to #349, each against all 247 rules CI sends (the core realm the repository pins, plus its own rules), in 25 entity groups. These have no labels, so each config is compared with Sonnet, and every disagreement on a violation was judged by hand. Results on this set are agreement and manual review, not accuracy.

Four configs: Sonnet 5.5 at medium effort (the CI check at the time) and Haiku 5.5 at medium, high and xhigh. 248 benchmark jobs (216 labelled-case runs and 32 commit runs), about 20 minutes with 4 workers, about $7.80 in all. The scoring was checked first: a perfect answer set scores 100%, and one that never says “violated” catches nothing.

// 3. Caveats

  • The sample is small (18 cases × 3 runs, 8 commits × 1). The results show a pattern, not a precise measurement.
  • Only this repository's rules and diffs were used. Results may differ for other rules and realms.
  • The benchmark does not model inheritance between entities.
  • It used strspc-manager's provider on develop, not the released CLI.
  • Prices are Anthropic's first-party rates for prompts up to 100K tokens: Sonnet 5.5 $2 input and $10 output, and Haiku 5.5 $0.10 input and $0.50 output, per million tokens. Cache writes are priced at 1.25× and cache reads at 0.1× the input price. These rates are the ones set in the benchmark source. Check Anthropic's current price list before you reuse the cost figures.
  • The costs and times are per CI run in this scenario: averaged over 8 real diffs, each evaluated in 25 request groups (one per entity), caches written by the run itself (see insight 4).

// 4. Results

“Wrong blocks” counts the 54 labelled runs only, where a config said “violated” and the label said otherwise. The cost for Haiku at high and xhigh is approximate (~) and priced as a real run would be, with cache writes counted for every config. Same sample and caveats as above.

Config Labelled cases right Wrong blocks Violations flagged on 8 real commits Cost per CI run Time per CI run
Sonnet 5.5, medium51/5400$0.85553 s
Haiku 5.5, medium54/5403, all on Draft rules (warnings, not blocks)$0.04770 s
Haiku 5.5, high54/5408, 2 on Implemented rules (blocks)~$0.05195 s
Haiku 5.5, xhigh54/5404, and 4 of 8 runs cut off at the output limit~$0.059157 s

// 5. Eight insights

1. Higher effort did not make up for a smaller model; here it flagged more

On the labelled cases Haiku was already perfect at medium. On the 8 real diffs, more effort meant more violations flagged:

  • medium flagged 3, all on Draft rules, so warnings only;
  • high flagged 8, including 2 that would have blocked a merge;
  • xhigh ran past the 8,192-token output limit in 4 of 8 runs.

Our reading is that more thinking turned into more reasons to object. (8 commits, one run each: a pattern, not a rate. The source records no review of the 3 flags at medium, so we do not say whether they were correct.)

2. One of high effort's blocks was a real catch

It flagged GOCONV-007 on #349: an exported method, roundTripper.RoundTrip in a test file, had no doc comment. The rule says exported identifiers need one, and Sonnet missed it. The other block was a false positive: GOCONV-005 on #339 flagged a test helper that drives a replay transport. In this run, higher effort was more thorough and more pedantic at the same time, and for a merge gate the one false block outweighed the one catch. That is two flags, one run each.

3. The bigger model's “misses” came from one ambiguous case

Sonnet's 3 wrong answers were all mod006-violated. The diff adds diagnostic code RL099 without touching the spec, but nothing in a diff proves the code is not already documented there, so Sonnet's “not applicable” is defensible. Writing labels for an LLM judge is itself hard: a case has to be decidable from the diff alone.

4. Prompt caching skews naive cost comparisons

The raw numbers showed Haiku at high effort cheaper than at medium. The reason: the configs share Anthropic's prompt cache within one model, so later configs read a cache an earlier one had paid to write. A real CI run writes its own cache, so the fair figure prices those writes for every config. Each commit's diff is written once and then read by the other 24 entity groups, and that is most of what keeps either model cheap.

5. Cost tracks cache writes and output, not input size

About 3 million prompt tokens per config came down to $0.05 to $0.86 per run. Haiku at xhigh produced roughly 3× the output tokens of Haiku at medium for no gain in accuracy.

6. The cheaper model is slower

In the scenario above, Haiku took 70 s per CI run against Sonnet's 53 s, one run each. In a CI job, 17 s more was acceptable for us. For interactive use it might not be.

7. What a merge gate rewards

Our view: a gate pays for few false blocks and stable verdicts more than for raw thoroughness. Haiku at medium gave every labelled case the same answer in all three runs on the labelled set.

8. Method notes worth sharing

  • Take labels from mechanical rules, not from a model's output.
  • Repeat each case, to separate unreliable answers from wrong ones.
  • Make “violated” the positive class, since it is what blocks.
  • Measure real diffs by agreement plus manual review, and never present that as accuracy.
  • Check the scoring with a perfect answer set and an all-wrong one before any paid call.

// 6. Outcome

Since the commit linked here, the default for the CI rule check is claude-haiku-5-5 at medium effort. strspc-manager made the change (commit for pull request #354, feat(ruleeval): default the claude provider to claude-haiku-5-5; the benchmark above is in the same tagged release), and the strspc-CLI documents the same default in its configuration reference.

These are defaults, not limits. You choose the model and the effort in the evaluator block of .strspc/config.yaml, with evaluator.model and evaluator.effort (configuration spec). This article measured the defaults on one repository. It does not say which model is right for your rules.

Where next: knowledge base, SteerSpec 101, concepts and the core realm.