Ora Performance reports what happened when the same analytical work was run six ways and scored for quality and cost. This page describes the procedure that produced those numbers, in enough detail that a third party could run a comparable evaluation of their own harness and check the findings — including months from now, once the specific models involved have moved on.

The procedure has two halves, each published as a self-contained, downloadable framework that uses ordinary API calls only:

Both are written as procedures for any comparable harness, not as a literal transcript of one particular tool. Read them as recipes, not as source code.

A note on model names — read this first

Every configuration in this procedure is defined by role and policy, never by a pinned model name. “The flagship model,” “the selected ~9B model,” “the cost-aware picker’s choice for the fast role” — these are roles that resolve to whatever the best current model is at the time you run it. The frameworks deliberately instruct you to run your model picker first, then freeze the chosen model IDs, providers, and prices as a dated snapshot for that run.

This matters because model quality moves quickly. Any specific model named in the original run — or in the results paper — should be read as a snapshot of what filled that role in mid-2026, not a fixed ingredient. If you rerun the campaign, re-resolve every role to its current equivalent; do not copy the old model IDs forward. A comparison that pins yesterday’s models measures yesterday, not the harness.

Why the runner itself isn’t published

The original comparison was produced by an internal script. That script is not published, and could not usefully be — it is bound to the author’s own installation in ways a third party can’t reproduce:

  • It drives a private, running Ora server over that server’s own internal request contract, rather than a portable engine.
  • It depends on Ora’s own model picker and model registry to build each configuration; without them there are no configurations to bake.
  • It reads credentials from the author’s machine keychain, and its top two configurations ride the author’s personal subscription through a locally installed command-line tool — neither of which is transferable.
  • It audits each capture against Ora’s private execution traces to enforce its fidelity checks.
  • It renders visual outputs through Ora’s own compiler and headless rendering toolchain, with a machine-specific asset layout.

A third party can’t run that script without the entire Ora installation, the author’s keychain, and the author’s personal subscription. That is exactly why the reproducible version is the two API-only frameworks below: they reproduce the same six-configuration campaign from a self-contained prompt library, using nothing but API access — no subscription account, no private vault, no local install required.

The six configurations

The comparison runs each prompt through six configurations, spanning the range from “no harness at all” to “maximum-capability harness.” Each is defined by role:

  1. Flagship Model — one bare call to the flagship model, with no harness, no tools, no review, no memory. The single-model baseline at the top of the market.
  2. Small Model, Raw — one bare call to a small (~9B-class) model, again with no harness. The cheapest possible baseline, and the control twin of the next configuration.
  3. Small Local Only — that same small model, but placed in every harness role: the large-model roles, the fast roles, the utility, consolidation, and verification roles. Tests what the harness alone adds to a model small enough to run on consumer hardware.
  4. Optimum — the harness run under a cost-aware policy: a model picker fills each role (two large-model roles, two fast roles, a small role, a utility role, a consolidation role, and a verification role) with the best value at each price/capability point. This is the default configuration.
  5. Optimum + Premium — the frozen Optimum configuration with only its final consolidation role upgraded to the flagship model. Tests the “spend the money where it lands last” hypothesis.
  6. Premium — the harness run under a maximum-capability policy, using premium/frontier-tier models throughout.

The same prompt goes through all six. The point of the comparison is what changes across them — how much of the final quality comes from the model, and how much from the harness wrapped around it.

Capturing the outputs

The capture framework takes the one featured prompt for each technique and runs it through all six configurations. Every capture is recorded in full — the exact prompt, the complete output, token counts, cost, and wall-clock time — and each one must pass a seven-point acceptance gate before it counts: the frozen model IDs actually served their roles, with no silent fallback or substitution; the prompt was unchanged; the technique was the one intended; any visual artifact was preserved; no private context leaked in; and cost and time were recorded from the run itself.

The output is a capture bundle, not a judgment — the raw material the scoring pass then grades. The completed comparison covers all 198 kind-qualified entries in the corpus, with all six configurations accepted for each: 198 × 6 = 1,188 captures. Shared public names never share identity or storage: mode:causal-dag remains distinct from visual:causal-dag, and visual:fishbone-diagram remains distinct from lens:fishbone-diagram. Four recovered subscription-inclusive cells in the reference bundle use explicitly marked API-equivalent cost estimates because their raw usage records did not survive; a new reproduction must still record cost from the run itself.

Scoring the outputs

The judging framework grades the capture bundle, and can run any time after capture. It is also API-only. The current panel uses the frozen Optimum configuration’s Big 1 (qwen/qwen3.7-plus) and Big 2 (minimax/minimax-m3) through independent direct API routes.

Each judge independently scores every capture from 0 to 10 on five criteria, weighted:

  • Fidelity (0.35) — did the output actually execute the technique it was asked for?
  • Soundness (0.25) — is the reasoning valid?
  • Completeness (0.18) — did it cover what the technique requires?
  • Relevance (0.12) — did it stay on the actual request?
  • Calibration (0.10) — is its confidence matched to its evidence?

The two judges are averaged per criterion, then combined into a weighted score — with one hard rule: overall = min(weighted, fidelity). A fluent essay that doesn’t actually perform the method cannot outscore its own fidelity. A capture is counted as a model-quality failure when its overall score falls below 4.0. The finalized 27 affected cells retain current raw panel records; the other 1,161 legacy cells retain their existing two-judge aggregate ratings, without a claim that their raw judge payloads survive.

The two frameworks

For the input side of the procedure — every prompt, for every technique — see the Trigger Prompt Corpus. For what the campaign found, see Ora Performance.