Capability studies

Run controlled environment comparisons and prospective sample planning.

These studies check whether worldseq can distinguish specific planning capabilities through verified business consequences. They use the ordinary engine, public observations, action execution and independent verifier. They do not estimate how well a language model plans or establish real-world effect sizes.

The checked-in capability-controls.json registers six studies: 13 conditions, 28 policy branches and two shared scenario commitments, giving 56 episodes. Sixteen exact capability assertions and one primitive-reconciliation assertion are frozen before execution. Every policy has the same decision, runtime and reported-token ceilings. Scripted policies consume no model tokens.

Study	Positive control	Negative control
Strategic diversity	Service-first wins during an outage; cash-first wins under cash pressure, using the same objective definition.	Each strategy loses in the opposing business condition.
Information value	A paid public recovery certificate improves the response to a nondefault cause.	Withholding the same paid diagnostic has no effect when the default is correct or the information is irrelevant.
Memory	Retaining an earlier published certificate improves a later decision after the incremental evidence page has moved on.	Forgetting does not hurt when the default recovery is correct.
Revised evidence	A later explicit correction improves the decision relative to retaining the first certificate.	A confirming update leaves both policies equivalent.
Planning horizon	A dependency-aware policy pays for preparation when contractual savings justify it over the remaining horizon.	A short horizon makes the larger investment unattractive to every policy.
Replanning	An adaptive policy cancels an unsafe queued deployment after a published disruption.	Stable conditions produce equal outcomes; the registered interaction isolates the disruption response.

Policies receive only a public snapshot, newly available evidence and their remaining decision allowance. Guaranteed action terms are public contractual inputs; hidden effects, cause identifiers and response probabilities are never controller inputs. Information ablations alter only the registered diagnostic channel. Both branches execute and pay for the inspection, so a free-information advantage cannot explain the contrast. Memory ablations remove certificate retention, not public commitments or the task objective.

Fixture definitions are public source. Private scenarios and runtime observation boundaries do not make these fixtures confidential holdouts; the built-in controlled policies deliberately operate only on their registered public inputs.

Each trajectory is independently verified, then its recorded decisions are replayed through the existing mechanism-fixture runner. The report retains all assignments, including infrastructure failures and cases not launched because of budgets. Business constraint failure is a declared binary objectiveSuccess: 0; infrastructure failure remains nonnumeric. Raw service-loss and accounting contrasts describe controlled mechanisms and do not bypass ranked-task eligibility.

Running a registered study

From the worldseq source package directory:

bun src/cli.ts capability validate --file examples/capability-controls.json
bun src/cli.ts capability register --file examples/capability-controls.json --out .runtime/capability-study --control .runtime/capability-study.trusted.json
bun src/cli.ts capability check .runtime/capability-study --control .runtime/capability-study.trusted.json
bun src/cli.ts capability run .runtime/capability-study --control .runtime/capability-study.trusted.json

Use new output paths. Keep the owner-only control file outside the artifact directory. Registration freezes code, evidence, complete conditions, policies, assertions and private scenarios. Public registration exposes commitments only. check verifies that registration and the current execution identity; run produces independently checked outcomes. Source changes require a new registration.

cases/case-N.json retains each result, command sequence and hashes of the actual public inputs. outcomes.json retains the complete schedule. report.json binds evidence to those bytes and reports per-scenario assertion values. A source/control mismatch or final storage failure revokes admission. Storage is a launch threshold: a terminal receipt may exceed it, which also prevents admission. Studies execute sequentially and do not start external agents.

Pilot-based sample planning

The separate sample-planning workflow sizes a future registered campaign. It requires the pilot’s single predeclared primary improvement comparison, the unchanged fixed task set, and every retained attempt from every agent. It independently replays the pilot; it cannot use only successful episodes, choose a favorable secondary comparison or combine parameter variants.

The configuration declares these inputs before simulation:

Minimum detectable effect in the comparison’s oriented units, greater than the unchanged practical tolerance.
Desired power, optional maximum interval width, and maximum tolerated null rejection rate.
An increasing, finite list of candidate scenario counts; repeats stay unchanged.
Deterministic analysis seed, Monte Carlo trials, bootstrap samples and a maximum count of bootstrap pair draws.
Future campaign episode, runtime and storage caps. Per-episode agent budgets must stay identical to the pilot.

Continuous metrics use scenario-block resampling of pilot paired differences, centered separately within each task. The authored detectable effect is then added; an observed favorable pilot effect never determines that effect. The binary objectiveSuccess model instead requires an explicitly authored discordance probability. It generates valid paired zero/one outcomes and assumes perfect dependence between repeated attempts within one scenario. The report shows empirical pilot discordance alongside that assumption. Neither model claims to recover unknown real-world variance.

For every candidate, the planner runs both the authored alternative and a control at the registered improvement-margin boundary through the existing hierarchical bootstrap. A zero margin uses a zero-effect control. It reports Wilson 95% intervals for Monte Carlo power and null rejection, plus the 90th percentile of confidence-interval width. A candidate qualifies only if the lower power bound, upper null-rejection bound and optional width criterion all pass. Every candidate remains in the artifact; the first qualifying count is selected. No qualifying candidate produces an inconclusive plan and no confirmatory registration.

The frozen artifact includes the full paired pilot inventory, configuration, all candidate results, original campaign plan and hashes of registration, reports, retained result manifests and verified pairs. A separate registration command checks the independent planning control and independently rechecks the pilot before generating fresh scenario secrets. Hypotheses, agents, tasks, repeats and parameter bindings are preserved. Registration never starts a provider or runs the confirmation automatically.

The supported estimand is the fixed registered set of companies and tasks. Increasing scenarios does not support claims about unseen companies. Confirmation remains subject to the ordinary campaign budgets, full failure inventory and independent replay. The planner’s simulation is a prospective design calculation, not confirmatory evidence.

sample-planning.json is a complete example for the primary comparison in baseline-pilot.json. Its 10-percentage-point detectable effect and 0.5 discordance are authored demonstration assumptions, not estimates from a measured pilot. Its future caps allow up to 2,304 episodes, one execution day and 128 GiB of retained artifacts; they are ceilings, not cost forecasts. Review and freeze those assumptions and measured capacity before using the example. It never runs a confirmation automatically.

Run the stages separately, supplying your declared assumptions and budgets in sample-parameters.json:

bun src/cli.ts sample plan --file sample-parameters.json --pilot .runtime/pilot --pilot-control .runtime/pilot.trusted.json --out .runtime/sample-plan --control .runtime/sample-plan.trusted.json
bun src/cli.ts sample check .runtime/sample-plan --control .runtime/sample-plan.trusted.json
bun src/cli.ts sample register .runtime/sample-plan --planning-control .runtime/sample-plan.trusted.json --pilot .runtime/pilot --pilot-control .runtime/pilot.trusted.json --out .runtime/confirmation --control .runtime/confirmation.trusted.json

Both sample plan and sample register independently replay the complete pilot; allow for that work beyond the campaign’s episode-execution budget. A separate validation report is unnecessary after a completed validation run, which already writes a verified report. A complete pilot may support sample planning even when its primary comparison fails; missing, failed or unverified attempts prevent planning. Run sample register only when the analysis is recommended.

src/validation/capability-sample-schema.ts defines the strict configuration contract. The underlying APIs are freezeSamplePlan, readSamplePlan and registerConfirmatoryCampaign in src/validation/capability-sample-plan.ts. See validation campaigns for the normal campaign execution and reporting workflow.