--- nexus: - ora type: framework status: active tags: - campaign - reproducibility - judging - scoring - framework/instruction date created: 2026-06-20 date modified: 2026-06-20 related: - "[[Paper — Ora Performance]]" - "[[Framework — Campaign Self-Verification]]" - "[[Working — Campaign Run Plan]]" --- # Framework — Ora Performance Judging and Scoring ## Display Name Ora Performance Judging and Scoring ## Display Description Scores a completed Ora Performance capture bundle with two API-based judges selected automatically from the Optimum configuration's two large-model slots. Produces criterion scores, capped overall scores, failure rates, and summary tables matching the paper. ## Purpose This framework reproduces the scoring side of the Ora Performance comparison. It assumes the capture bundle already exists. The capture framework may take days; this judging framework can be run later against the completed bundle. Judging is API-only. Do not use subscription accounts, web-chat accounts, hidden quota, or manual scoring by a human judge. The two judge models are selected from the frozen Optimum configuration's **Big 1** and **Big 2** roles so the judging pass follows the public configuration rather than a private preference. ## Setup Questions ### Capture bundle Required. Provide the capture bundle produced by `Framework — Campaign Self-Verification` or an equivalent bundle with accepted captures, prompts, model snapshots, costs, timings, and visual artifacts. ### Optimum configuration snapshot Required. Provide the frozen Optimum role map from the capture run. It must include Big 1 and Big 2 model IDs, providers, and API routes. ### Technique specifications Required. Provide access to the public Ora technique specifications for the scored entries: mode specs for modes, visual-output specs for visuals, and lens/host-mode specs for lenses. The judge needs the method being tested, not only the prompt and answer. ### Judge API credentials Required. Provide metered API access to the two judge models selected from Optimum Big 1 and Big 2. ### Output location Required. Choose where the framework writes judge packets, per-judge scores, averaged scores, capped overall scores, failure classifications, and summary tables. ## Intake Protocol 1. Confirm the capture bundle's scored set contains 196 entries: all 60 analysis modes (including `passion-exploration` and `structured-output`), 21 visual outputs, and 115 lenses, with each shared-ID collision (`causal-dag`, `fishbone-diagram`) resolved to one entry. 2. Confirm every capture being scored passed the capture framework's acceptance gate. Do not score invalid captures as if they were low-quality model answers. 3. Read the frozen Optimum snapshot and select two judge models automatically: Judge A = Big 1, Judge B = Big 2. 4. Confirm both judge models are reachable by API. If either cannot be reached, stop. Do not substitute a different judge unless the operator explicitly starts a new scoring run and records the substitution. 5. Confirm each scored entry has an available technique specification. If a specification is missing, mark that entry unscorable rather than guessing the method. 6. State the exact scoring formula before judging: weighted average across five criteria, then cap Overall at Fidelity. ## Execution ### Phase 1 — Build Judge Packets For each accepted capture, create a judge packet containing: - qualified entry key - family: mode, visual, or lens - technique name and public specification source - intended mode, visual route, or lens host - exact prime prompt - configuration name - final answer - visual artifact or explicit text-only note - configuration model snapshot - capture acceptance notes Do not include another judge's scores or commentary in a packet. Judges must score independently. ### Phase 2 — Score Independently Send each judge the same packet and the same rubric. The judge returns numeric scores from 0 to 10 for five criteria plus concise notes for each criterion. The five criteria are: 1. **Fidelity (0.35)** — Did the answer actually execute the technique's method and structure, or merely describe it, name-drop it, or write a generic essay? This is the test of whether the analysis was performed at all, which is why Fidelity sets the hard cap. 2. **Soundness (0.25)** — Are the claims and inferences correct and logically valid, with evidence kept distinct from assumption? This judges reasoning inside the answer, not external fact-checking beyond the provided/retrieved material. 3. **Completeness (0.18)** — Does the answer surface what this technique should surface for this prompt, or does it leave major gaps? 4. **Relevance (0.12)** — Does the answer engage this specific prompt and material, or drift into a generic treatment? 5. **Calibration (0.10)** — Is the answer honest about uncertainty, explicit about assumptions, and free of overclaiming? Scoring anchors: - **9-10** — exemplary execution of the actual technique. - **7-8** — strong and usable with minor omissions or calibration issues. - **5-6** — competent but flawed; the technique is recognizable but partial. - **3-4** — mostly gestures at the technique; substantial non-execution. - **0-2** — wrong task, empty answer, unusable output, or nearly no method execution. ### Phase 3 — Average the Judges For each capture, average Judge A and Judge B per criterion: ```text fidelity = mean(judge_a.fidelity, judge_b.fidelity) soundness = mean(judge_a.soundness, judge_b.soundness) completeness = mean(judge_a.completeness, judge_b.completeness) relevance = mean(judge_a.relevance, judge_b.relevance) calibration = mean(judge_a.calibration, judge_b.calibration) ``` Preserve both judges' notes. The averaged criterion scores are the public scores. ### Phase 4 — Apply the Hard Cap Compute the weighted score, then cap it at Fidelity: ```text weighted = 0.35*fidelity + 0.25*soundness + 0.18*completeness + 0.12*relevance + 0.10*calibration overall = min(weighted, fidelity) ``` This is the operational form of the hard cap. Fidelity measures whether the answer performed the assigned analysis. A fluent answer that does not execute the method cannot score higher than its execution. ### Phase 5 — Classify Failures A scored capture is a model-quality failure when `overall < 4.0`. This threshold marks answers closer to "talks about the technique" than to "competent execution of the technique." Do not count invalid captures from the capture framework as failures. Invalid captures are operational failures and belong in the capture audit, not in the quality failure rate. ### Phase 6 — Aggregate Tables For each configuration, compute: - mean Overall - mean Fidelity - mean Soundness - mean Completeness - mean Relevance - mean Calibration - failure rate: scored captures with Overall < 4.0 divided by scored captures - mean API cost per 100 queries from the capture bundle - mean wall-clock minutes per run from the capture bundle - judged count and unscorable count Round public tables to one decimal for ratings, whole percentages for failure rates, whole dollars where the paper presents rounded cost, and one decimal minute where the paper presents run time. ### Phase 7 — Produce the Scoring Report Produce three outputs: 1. **Per-capture score ledger** — one row per capture with both judges, averaged criteria, weighted score, capped Overall, and failure flag. 2. **Configuration summary** — one row per configuration with the aggregate statistics above. 3. **Methodology note** — judge models used, scoring date, scored-entry count, exclusions, unscorable entries, and any deviations. ## Output Contract This framework produces reproducible scoring records from an already accepted capture bundle. The final score for each capture is the average of two API judges' criterion scores, with Overall capped at Fidelity. The full campaign grid is 196 entries x 6 configurations = 1,176 possible scored captures; the reference run holds 1,173 (the modes `passion-exploration` and `structured-output` captured on fewer than all six lanes), minus any captures explicitly marked unscorable for missing specifications or invalid capture provenance. ## Failure Modes ### Judge Substitution Drift The operator swaps in different judge models without restarting and documenting a new scoring run. Stop and either restore Optimum Big 1/Big 2 or label the run as a different judging panel. ### Judge Contamination One judge sees the other's scores before scoring. Discard the contaminated score and rerun independently. ### Specification Blindness Judges score from prompt and answer only, without the technique specification. Mark the affected entries unscorable or rerun with the specification included. ### Invalid Capture Scored as Failure A capture with fallback, wrong model, private context leakage, or prompt drift is scored as if it were a bad model answer. Remove it from quality scoring and return it to capture audit. ### Missing Hard Cap The weighted score is published directly without `overall = min(weighted, fidelity)`. Recompute the ledger; the result is invalid until the cap is applied. ### Collision-ID double count Both entries of a shared-ID collision enter the scoring set — e.g. `causal-dag` scored as both a mode and a visual output, or `fishbone-diagram` as both a lens and a visual. Keep one entry per shared ID (the mode for `causal-dag`, the visual for `fishbone-diagram`) and keep the denominator at 196. ## Appendix — Judge Response Schema ```yaml entry_key: mode:argument-audit configuration: optimum judge_model: provider/model-id scores: fidelity: 0.0 soundness: 0.0 completeness: 0.0 relevance: 0.0 calibration: 0.0 notes: fidelity: "" soundness: "" completeness: "" relevance: "" calibration: "" impression: "" ``` **END OF ORA PERFORMANCE JUDGING AND SCORING FRAMEWORK v1.0**