Benchmarks
Opheleon plans complex software, so we measure the planning judgment of the models we build on. We publish two open-source benchmarks with fully deterministic grading: no LLM judge scores any answer, every result is reproducible from a public grader, and every gold answer is entailed by the scenario's stated constraints.
SysDesign-Bench measures system design from requirements. ProductJudge-Bench measures policy-conditioned product decisions. Both are open source under the opheleon/SysDesign-Bench and opheleon/ProductJudge-Bench repositories.
SysDesign-Bench: where models struggle in system design
SysDesign-Bench gives a model a requirements document and asks for a structured design: components, patterns, per-flow decisions, and a coverage map, all chosen from fixed catalogs and graded by set matching. Scenarios are de-labeled and carry constraint inversions, so reciting a famous architecture actively fails. The July 2026 baseline covers five models at high reasoning effort across 38 public scenarios.
View as table
| Model | Overall |
|---|---|
| Claude Fable 5 | 92% |
| Grok 4.5 | 88% |
| GPT-5.6 Sol | 81% |
| Claude Opus 4.8 | 75% |
| GLM-5.2 | 72% |
The headline number hides the real story. Architectural correctness is saturated: every model solves 96 to 100 percent of the mode-defining decisions (database choices, consistency models, migration sequencing, infeasibility detection). The spread lives in two supporting judgments:
- Over-engineering. Parsimony, scored as staying within the pattern budget the requirements actually entail, spans 24 to 50 percent. No model shows restraint; the strongest designer on the board is also a chronic over-builder.
- Knowing when to ask. When a load-bearing fact is missing, clarification-seeking is binary and thinking-invariant: two models always ask, two models never ask at any thinking budget we tested, and one splits. A model that designs confidently on missing facts is guessing with conviction.
View as table
| Model | Solution | Parsimony | Clarification |
|---|---|---|---|
| Claude Fable 5 | 100% | 50% | 100% |
| Grok 4.5 | 100% | 24% | 100% |
| GPT-5.6 Sol | 98% | 47% | 50% |
| Claude Opus 4.8 | 98% | 32% | 0% |
| GLM-5.2 | 96% | 47% | 0% |
What this means for planning: current frontier models produce architecturally sound designs, but they over-build and they do not reliably stop to ask when requirements are incomplete. Those are exactly the failure modes a planning system has to check for, which is why Opheleon keeps generated plans behind human review gates and records unresolved questions as structured unknowns instead of letting a model guess.
ProductJudge-Bench: the decision layer is nearly solved at high effort
ProductJudge-Bench measures policy-conditioned product decisions: prioritizing a backlog under a real capacity constraint, separating genuine trends from loud noise, choosing build versus configure versus decline, reading an experiment correctly, deciding when to go back to the customer, and attributing outcomes across sequential attempts with confounds. Every scenario carries a machine-executable decision policy, and an oracle derives the provably correct answer, so grading is exact.
The scenarios are adversarial. They include an aggregation reversal where the blended metric and the per-segment truth disagree, a 12-candidate prioritization whose optimum no greedy heuristic finds, a salient unknown that is worthless to ask about because a policy gate already determines the outcome, matched counterfactual pairs where one changed fact must flip the decision, and a messy-intake family where the facts arrive as raw call transcripts, Slack threads, and email chains full of contradictions that stated source-precedence rules resolve.
View as table
| Model | High thinking effort | No extended thinking |
|---|---|---|
| Claude Opus 4.8 | 100% | 100% |
| Claude Fable 5 | 97% | 97% |
| Grok 4.5 | 97% | 97% |
| GPT-5.6 Sol | 97% | 100% |
| GLM-5.2 | 97% | 75% |
At high thinking effort, the enumerated-options traps all get defused: every model finds the non-greedy optimum, holds on the metric reversal, declines without stalling on the gate-pinned unknown, and refuses to roll out a change whose history was confounded. The one crack is the interesting part. In a messy-intake scenario where a champion claims a large expansion, procurement's file says zero, and no precedence rule ranks the two sources, four of five models resolve the conflict by silently siding with the conservative source and deferring, when the entailed move is to get the number in writing from the economic buyer. That failure survives full thinking effort: models would rather guess than ask. Claude Opus 4.8 is the only model that asks, and the only perfect score.
What this means for product teams: once the options, constraints, and evidence are on the table, frontier models already make the call correctly at high thinking effort. The scarce human contribution is shifting upstream, to creativity and ideation: framing the opportunity, inventing the options worth comparing, and deciding what the policy should be. The residual model weakness is epistemic rather than analytical: recognizing when the record cannot answer the question and someone has to go ask.
Methodology
Both benchmarks grade with pure set, enum, and arithmetic matching in open-source Python; no LLM judges any answer. Gold answers must be entailed by stated constraints, with worked math published per scenario. ProductJudge-Bench additionally derives gold from a machine-executable policy oracle that accepts every optimal answer and reports normalized regret on near-misses, and its validator rejects any scenario whose hand-written answer disagrees with its own policy.
Protocol: pass@1, single run, July 2026. SysDesign-Bench v0.3: 38 public scenarios, five models, reasoning effort high. ProductJudge-Bench: 24 public scenarios across eight families, five models at two thinking settings; still pilot scale, so treat results as directional until the multi-run baseline lands. Deterministic trivial baselines (always-ask, always-build, greedy ranking) floor at 38 to 55 percent, and the grading harness self-tests by requiring its own oracle to score 100 percent.