Scripture response evaluation

This comparison uses the frozen cases and acceptance criteria in the approved development plan. Candidate A uses current guidance and provider-default reasoning. B uses revised guidance with provider-default reasoning. C uses revised guidance with low reasoning. All use gpt-5-nano and the same 8,192-token completion allowance.

The runner copies application code and a minimal canonical repository into separate private directories for each candidate. Each trial creates an isolated thread, selects scripture analysis in Chromium, runs the ordinary task-queue worker, and observes the canonical reply card. A starts the public CLI worker immediately and reloads after it exits; B/C use automatic dispatch and browser status polling. A therefore has no imposed cron delay. Trials run serially in rotated candidate order, with a fresh browser per trial and persistent isolated application state per candidate. The first trial per candidate is cold. The existing isolated fixture disables automatic background fast-scoring enqueue, consistently across candidates; this comparison does not measure production background-scoring load. Supporting moderation calls remain subject to the application's ordinary scripture-task policy; the runner does not call moderation or generation services directly.

tests/Support/scripture_live_evaluation.py NEW_PRIVATE_DIRECTORY starts one comparison. A local HTTP proxy forwards the application's unchanged provider payloads to OpenAI and reserves worst-case spend before every call. It enforces the frozen model, completion allowance, call counts, and $5 budget; uncertain charges retain their reservation. No calls are retried automatically. The explicit resume DIRECTORY mode verifies frozen artifacts and the original ledger, skips completed samples, and permits repeating only a source-creation ASCII rejection with proof that no provider call was dispatched. Explicitly accounted interrupted calls keep their full reservations and are skipped permanently. Failed or uncertain generation is refused. A failed or interrupted run must be reviewed with its existing ledger before any new paid run; creating a new directory does not authorize a new budget.

The existing API key stays in the proxy process's memory. Copied applications use a dummy local-proxy key and an absent secrets path. Canonical records, databases, reply-agent keys, screenshots, worker logs, exact payloads/results, phase markers, artifact hashes and the spend ledger remain under state/private/scripture-quality-evaluation-20261015/. No trial posts are written to production threads.

tests/Support/scripture_trial_review.py DIRECTORY exports randomized answer IDs and review packets without candidate or performance labels. Quality review uses the frozen rubric, followed by source checks against the cited passages and then candidate identification. Quotations may be recognizable excerpts; punctuation, capitalization and orthographic variants are distinguished from invented words or translation substitution. Length alone is not a failure: repetition, needless catalogues, lost context, missing requested depth, and unsupported conversation offers are assessed against the actual concern.

Prices were checked against official OpenAI documentation: $0.05/M input tokens and $0.40/M output tokens. Reported costs conservatively charge all input at the uncached rate and include reasoning output tokens. Supporting-call and generation reservations share the same dollar ceiling.

Paid expansion trials require a qualifying initial candidate. Release requires all initial and expansion gates; a failed experiment does not authorize promotion, prompt retuning against the same samples, or additional calls.

Protocol notes: The first baseline answer appeared with metadata in an early diagnostic read before anonymous review; its rating is therefore not fully blinded. Initial attempt 22 was rejected during source creation because the fixture bypassed normal composer punctuation conversion. It made no provider call. After a stub check, the harness used the application's existing compose normalizer and resumed the same frozen applications and ledger. Original failed-attempt files and all completed samples were retained. These limitations must accompany any interpretation of the comparison.

The background process also stopped during trial 42. Its generation reservation remains unsettled and fully charged against the budget; no response was retained and the call was not repeated. The remaining six scheduled trials completed against the same ledger. Interrupted outcomes must be reported separately and cannot be counted as successful samples. Early trials recorded monotonic submission-to-visible durations and private phase timestamps; browser discovery is therefore bounded within combined request/discovery overhead rather than claimed as an exact separate duration. Later trials also retain browser wall-clock endpoints.