Pressure-testing Ota on Bedrock: query identity as replay evidence

Bedrock is not a normal application test repo.

It is a natural-language-to-SQL stability harness. It records generated SQL across repeated runs, replays the committed fixture against a frozen SQLite store, compares the results with a defended answer key, and blocks a candidate when reliability regresses.

That made it a strong pressure target for a more specific Ota question:

when a verification lane replays prior model behavior, how should Ota distinguish the inputs that defined the replay from observations that explain what happened?

The answer cannot be one undifferentiated receipt field.

Bedrock already had the useful artifacts in its repository:

• data/fixture.jsonl: one recorded SQL query per question and run
• data/store.db: the frozen SQLite state
• data/baseline.json: the defended regression baseline
• docs/scorecard.md: the rendered stability result

Those artifacts answer different questions.

A committed fixture, store, and baseline are replay inputs. They define the deterministic lane. A repeated SQL trace is observed behavior. It can show that one question produced different query shapes, but it is not an input Ota used to decide whether the current command could start.

That distinction became the pressure boundary.

The deterministic gate now declares the replay inputs explicitly:

gate:  replay_inputs:    - id: recorded_sql      kind: static_file      path: data/fixture.jsonl    - id: frozen_store      kind: static_file      path: data/store.db    - id: defended_baseline      kind: static_file      path: data/baseline.json

Those files are captured as declared, static replay truth.

The recorded query trace is declared separately:

gate:  witnessed_observations:    query_traces:      - id: recorded_sql        path: data/fixture.jsonl

That is the important Ota boundary.

• replay_inputs[] says what the deterministic gate consumed
• witnessed_observations.query_traces[] says what behavior the fixture contains

The second must not be promoted into the first. Otherwise a receipt can make historical model output look like a pre-execution decision input, which weakens replay and evidence semantics.

Bedrock's offline path is deliberately narrow and useful:

• replay the committed SQL fixture
• execute it against the committed SQLite store
• compare the result set with the defended answer key
• compare the candidate score with the committed baseline

The contract makes that boundary visible:

gate:  command:    exe: python    args:      - -B      - main.py      - gate  safe_for_agent: true

That proves the declared fixture, store, and baseline path. It does not prove that a live model will generate the same SQL tomorrow, or that a live database has not changed.

Those claims remain separate in the contract's record:live lane:

record:live:  effects:    writes:      - data/fixture.jsonl    network: true    network_kind: integration_test    external_state:      - anthropic_api

That lane reaches a live model and rewrites the recorded fixture. It is intentionally outside the deterministic, agent-safe replay workflow.

Bedrock helped close an Ota evidence-model gap.

The initial temptation was to place per-record query identity under receipt evaluated_inputs[]. That would have been wrong. A query trace is execution evidence, not a current-run input.

Ota now keeps the split explicit:

• declared static files stay under receipt evaluated_inputs[]
• historical query identities and divergence summaries live under receipt

witnessed_observations.query_traces[]

That gives later comparison and correlation a cleaner foundation:

• inputs can remain unchanged
• observed query behavior can still diverge

Neither fact is allowed to impersonate the other.

Bedrock also made one remaining boundary clear: its live recording path depends on a generic, unpinned pip requirements lane. Ota does not yet own that as typed dependency hydration. The contract names it as outside the deterministic proof instead of hiding it in setup shell.

The pressure matrix is intentionally offline and cross-platform:

• ota validate
• ota doctor
• ota tasks --use
• ota tasks --safe --use
• agent-mode task dry-run for the stability gate
• agent-mode workflow dry-run and preparation for verify
• deterministic test execution
• scorecard replay against the committed SQL fixture
• baseline gate execution
• archived receipt for the declared workflow
• native coverage on Ubuntu, macOS, and Windows
• the same replay workflow in Ota's pinned Python container context

This is not a claim that Bedrock's live model lane is hermetic. It is proof that the deterministic replay lane is explicit about its inputs, output evidence, and boundary.

The released v1.6.24 pressure matrix passed across native Ubuntu, macOS, Windows, and the container replay lane: run #29572073749.

Bedrock clarified a product rule that applies beyond AI evaluation:

do not collapse observed behavior into declared execution inputs.

For this repository, a clean replay input can narrow or acquit the input class it actually names. A divergent query trace can explain a flap. Neither artifact should overclaim the other role.

That is the right foundation for Ota's replay and receipt model: declared truth, execution truth, and witnessed evidence remain linked, but separate.

• Contract: Bedrock ota.yaml
• Pressure workflow: ota-pressure.yml
• Released-version pressure matrix: #29572073749