Ora Performance reports what happened when the same analytical work was run six ways and scored for quality and cost. This page describes the procedure that produced those numbers, in enough detail that a third party could run a comparable evaluation of their own harness and check the findings — including months from now, once the specific models involved have moved on.

The procedure has two halves, each published as a self-contained, downloadable framework that uses ordinary API calls only:

Both are written as procedures for any comparable harness, not as a literal transcript of one particular tool. Read them as recipes, not as source code.

A note on model names — read this first

Every configuration in this procedure is defined by role and policy, never by a pinned model name. “The flagship model,” “the selected ~9B model,” “the cost-aware picker’s choice for the fast role” — these are roles that resolve to whatever the best current model is at the time you run it. The frameworks deliberately instruct you to run your model picker first, then freeze the chosen model IDs, providers, and prices as a dated snapshot for that run.

This matters because model quality moves quickly. Any specific model named in the original run — or in the results paper — should be read as a snapshot of what filled that role in mid-2026, not a fixed ingredient. If you rerun the campaign, re-resolve every role to its current equivalent; do not copy the old model IDs forward. A comparison that pins yesterday’s models measures yesterday, not the harness.

Why the runner itself isn’t published

The original comparison was produced by an internal script. That script is not published, and could not usefully be — it is bound to the author’s own installation in ways a third party can’t reproduce:

  • It drives a private, running Ora server over that server’s own internal request contract, rather than a portable engine.
  • It depends on Ora’s own model picker and model registry to build each configuration; without them there are no configurations to bake.
  • It reads credentials from the author’s machine keychain, and its top two configurations ride the author’s personal subscription through a locally installed command-line tool — neither of which is transferable.
  • It audits each capture against Ora’s private execution traces to enforce its fidelity checks.
  • It renders visual outputs through Ora’s own compiler and headless rendering toolchain, with a machine-specific asset layout.

A third party can’t run that script without the entire Ora installation, the author’s keychain, and the author’s personal subscription. That is exactly why the reproducible version is the two API-only frameworks below: they reproduce the same six-configuration campaign from a self-contained prompt library, using nothing but API access — no subscription account, no private vault, no local install required.

The six configurations

The comparison runs each prompt through six configurations, spanning the range from “no harness at all” to “maximum-capability harness.” Each is defined by role:

  1. Flagship Model — one bare call to the flagship model, with no harness, no tools, no review, no memory. The single-model baseline at the top of the market.
  2. Small Model, Raw — one bare call to a small (~9B-class) model, again with no harness. The cheapest possible baseline, and the control twin of the next configuration.
  3. Small Local Only — that same small model, but placed in every harness role: the large-model roles, the fast roles, the utility, consolidation, and verification roles. Tests what the harness alone adds to a model small enough to run on consumer hardware.
  4. Optimum — the harness run under a cost-aware policy: a model picker fills each role (two large-model roles, two fast roles, a small role, a utility role, a consolidation role, and a verification role) with the best value at each price/capability point. This is the default configuration.
  5. Optimum + Premium — the frozen Optimum configuration with only its final consolidation role upgraded to the flagship model. Tests the “spend the money where it lands last” hypothesis.
  6. Premium — the harness run under a maximum-capability policy, using premium/frontier-tier models throughout.

The same prompt goes through all six. The point of the comparison is what changes across them — how much of the final quality comes from the model, and how much from the harness wrapped around it.

Capturing the outputs

The capture framework takes the one featured prompt for each technique and runs it through all six configurations. Every capture is recorded in full — the exact prompt, the complete output, token counts, cost, and wall-clock time — and each one must pass a seven-point acceptance gate before it counts: the frozen model IDs actually served their roles, with no silent fallback or substitution; the prompt was unchanged; the technique was the one intended; any visual artifact was preserved; no private context leaked in; and cost and time were recorded from the run itself.

The output is a capture bundle, not a judgment — the raw material the scoring pass then grades. The comparison covers 196 of the 198 entries in the corpus: two IDs (causal-dag and fishbone-diagram) each exist under two kinds, and the run scored one entry apiece, so two kind-specific entries were not separately scored (see the corpus page for the detail). At six configurations each, that is 196 × 6 = 1,176 possible captures; the reference run holds 1,173 accepted — two modes (passion-exploration and structured-output) captured on fewer than all six lanes, three lanes short of the full grid.

Scoring the outputs

The judging framework grades the capture bundle, and can run any time after capture. It is also API-only, and it picks its two judges automatically from the frozen Optimum configuration’s two large-model roles — so the scoring follows the public configuration rather than a private preference.

Each judge independently scores every capture from 0 to 10 on five criteria, weighted:

  • Fidelity (0.35) — did the output actually execute the technique it was asked for?
  • Soundness (0.25) — is the reasoning valid?
  • Completeness (0.18) — did it cover what the technique requires?
  • Relevance (0.12) — did it stay on the actual request?
  • Calibration (0.10) — is its confidence matched to its evidence?

The two judges are averaged per criterion, then combined into a weighted score — with one hard rule: the overall score can never exceed the Fidelity score. A fluent essay that doesn’t actually perform the method cannot outscore its own fidelity. A capture is counted as a model-quality failure when its overall score falls below 4.0.

The two frameworks

For the input side of the procedure — every prompt, for every technique — see the Trigger Prompt Corpus. For what the campaign found, see Ora Performance.