Validation campaigns

A campaign freezes its company/task inventory, agents, scenario commitments, parameter member, comparisons and budgets before any episode runs. Its report replays retained results and derives assertion admission from matching evidence. Passing one registered assertion establishes that assertion, not overall business realism or general CEO competence.

Run commands from the worldseq source package directory. The starter plan runs two identical idle policies on two paired scenarios: a small reproducibility control, not a planning quality study.

bun run cli validation check --file examples/validation-campaign.json
bun run cli validation register --file examples/validation-campaign.json --out /tmp/worldseq-control-study --control /tmp/worldseq-control-study.trusted.json
bun run cli validation check /tmp/worldseq-control-study --control /tmp/worldseq-control-study.trusted.json
bun run cli validation run /tmp/worldseq-control-study --control /tmp/worldseq-control-study.trusted.json --workers 2 --no-dashboard

validation check --file validates a plan before registration. Checking an existing campaign instead requires its directory and independent --control file; these selectors cannot be combined. A successful check establishes configuration and registration integrity, not executed admission.

validation run and validation report exit with status 0 only when execution is complete and the registered admission requirements pass. Missing execution, budget exhaustion, failed assertions and inconclusive admission return status 1 while retaining the report and complete inventory. Inspect execution, admission and the per-comparison status in report.json; run also prints progress lines, so its stdout is not a single JSON document. A report before execution therefore exits unsuccessfully with every unlaunched case still visible.

validation run already performs independent replay and writes the final report. Use validation report <directory> --control <file> to rebuild it later; this replays the full inventory again. Campaign execution is single-use and cannot resume after interruption. The graceful pause and --resume options for catalog admission do not apply to campaigns. Preserve an interrupted campaign and register a new experiment instead of replacing its attempts.

Use a new output directory and control filename for each registration. Keep the control file outside the artifact tree, under the evaluator’s authority. Both it and private-campaign.json are created with mode 0600; the campaign directory uses 0700. The private registration contains scenario seeds. Share only public-registration.json, report.json and comparison-*.json; these publish commitments and allowed results without private seeds or hidden company specs. Per-agent result directories remain private because their legacy replay format contains full registrations and seals.

With a configured dashboard database, omit --no-dashboard to see each agent batch and its cases in the existing run history. progress.json remains available locally. Dashboard progress does not establish score validity. A later agent batch appears there when launched; the campaign registration and report retain the complete inventory, including batches that were never launched.

Registering a comparison

Edit the example JSON before registration. taskIds must resolve in the active catalog; all agents are complete manifests. Available scripted policies are idle, conservative, growth, adaptive, cash-preserving and retention-first. The focused policies reuse the adaptive policy’s public signals and resource accounting while limiting new spending to collections, or to collections/support/reliability. They are transparent authored heuristics, not an optimal strategy. Remote/container agents require explicit allowExternalAgents: true; container execution retains its isolated contract.

For operating worlds, the policies consume public business reports and disclosed action roles. Adaptive recovery reacts to observed service/collection failures, preserves reported liabilities and operating-cost reserves, checks department capacity and prerequisites, and cancels remaining expansion when conditions deteriorate. Static policies make an opening choice only. Experimental rollout uses published arm counts and a conservative non-overlapping Wilson-interval heuristic; uncertain samples preserve the baseline. Hidden effects, schedules, question answer keys and scenario seeds never enter policy input.

seeds selects distinct private scenarios per task. repeats repeats each scenario without changing its world draws. Both accept 1–10,000; their product with tasks and agents must fit maxEpisodes (at most 100,000) before inventory expansion. Seeds are shared across compared agents. Retries cannot replace an unsuccessful registered attempt. The older eval CLI retains its existing 1–10 schedule limits.

Each comparison declares its baseline/treatment, metric, direction, practical tolerance, population and expectation (improvement, equivalence or non_inferiority). Metrics initially include primaryValue, netCashCents, activePayingAccounts, mrrCents, supportBacklog and objectiveSuccess. The last is a declared binary outcome: a completed, independently verified attempt meeting all constraints scores 1; a completed constraint violation scores 0. A primaryValue comparison rejects tasks whose primary metrics have different units. Choose tolerances from the question being tested before viewing outcomes; a wide interval is not evidence of equivalence.

At most one comparison can reference a required assertion. Other comparisons are exploratory, with no simultaneous coverage claim. Required assertions without a matching implemented check stay not_run; editing a flag cannot admit them. Every assertion has a supported quality dimension and an explicit applicability declaration. not_applicable needs a registered reason and matching evidence.

Executable sensitivity members

Use parameterSetId: "catalog-default" for unchanged catalog specs, or an exact <ensemble-id>.<member-id> from data/evidence/parameter-ensembles.json. These are unweighted authored alternatives, not samples from an empirical parameter distribution. Run each member as a separate campaign and interpret differences as sensitivity analyses. The selected source member and applied specs enter frozen hashes; workers and replay apply the same binding without changing global state.

The task must declare the corresponding mechanism and contain the exact executable roles below. Unsupported or inapplicable members fail before launch.

Representative-canary allocation, post-payment onboarding duration and hour-based stale-plan delay remain unsupported. The daily engine does not silently reinterpret hours as days or trial length as post-payment adoption.

Interpreting a report

Paired analysis reports direction-adjusted effects, wins/ties/losses and 95% percentile bootstrap intervals, using 10,000 deterministic resamples. Business families are fixed strata with equal weight. Companies, tasks, scenarios and attempts remain explicit nested units:

• registered_tasks keeps registered companies/tasks fixed and resamples scenarios and attempts. At least two scenarios per task are needed for its intervals.
• company_population additionally resamples companies and their tasks. It needs at least two independent company packs per family. Generated siblings may share assumptions; that dependence limits any broader population claim.

The reported estimand weights families equally, companies equally within family, tasks equally within company, scenarios equally within task, and attempts equally within scenario. More repeated attempts do not give one company extra weight.

Agent failures, infrastructure failures, violated business constraints and missing attempts remain in the paired inventory. Execution failures and missing attempts have no invented numeric reward; the metric comparison becomes inconclusive. Constraint violations receive 0 only for the declared objectiveSuccess metric; other metrics remain inconclusive without selecting an eligible-only subset. A missing/unverified batch cannot disappear from the denominator. Unsealed worker failure receipts establish inventory accounting only (inventory-verified), never trajectory replay. Replayed outcomes do not attest remote model identity.

Budgets are explicit operational bounds:

• maxRuntimeSeconds covers episode execution. Cancellation allows up to five seconds for worker cleanup, then up to ten seconds to remove its exact owned container if needed. Independent replay and statistics run afterward and make no additional agent/provider calls.
• maxStoredBytes counts retained artifacts, private/public registration, control file and final report. It stops later launches after retained writes; concurrent writes, failure receipts and final reports can exceed the threshold. An overrun revokes admission while retaining the inventory. This is not a filesystem quota.
• Artifact reads separately bound compressed bytes to 128 MiB and expanded episode JSON to 1 GiB. Episode decision/token/runtime ceilings remain in force.

Execution hashes are checked before every agent batch and reporting. Changing engine, source data, runtime inventory or the private/public registration requires a new campaign. Re-running the same registered directory is refused. Preserve all outcomes, choose subsequent experiments from declared questions, and use new held-out company/scenario inventories for confirmatory claims after development.

Reporting checks the same initial registration again after replay and persistence. An observed source, control or registration change revokes admission and retains an inconclusive report with the failure reason. Persistence failures also revoke any previous passing report; if it cannot be overwritten, the derived report is removed. The command exits unsuccessfully and preserves the registered attempts.

Bounded public pilot

examples/baseline-pilot.json declares 24 task templates, six policies and three scenarios (432 episodes), with a 20-minute execution stop threshold and 4 GiB retained-storage threshold. Run it with two workers on a shared development host. Idle and adaptive batches come first so their registered comparison is attempted before exploratory policies; the deadline can still leave a partial pilot, which cannot pass campaign admission.

bun run cli validation register --file examples/baseline-pilot.json --out /tmp/worldseq-public-pilot --control /tmp/worldseq-public-pilot.trusted.json
bun run cli validation run /tmp/worldseq-public-pilot --control /tmp/worldseq-public-pilot.trusted.json --workers 2 --no-dashboard

The checked-in selection is frozen before execution. Its generator, scripts/prepare-validation-pilot.ts, chooses one task per template while balancing company/family counts without consulting outcomes. Some templates exist only in the public validation/test partitions; this pilot includes those public templates and makes no confidential-holdout claim. Do not tune tasks from its results and reuse the same results as confirmation. The primary assertion compares adaptive against idle on objectiveSuccess; other eligibility and raw-cash comparisons are exploratory. Reports retain success, constraint-failure, execution-failure, unverified and not-run counts for every batch.

Recording reproducible feedback

Use bun run cli feedback record --file feedback-input.json --out /tmp/worldseq-feedback.json to create a local feedback artifact. The output filename must be new and its parent directory must exist. No network request is made and no score, assertion or catalog entry changes. Each record is explicitly unreviewed with affectsScores: false.

The input follows src/validation/feedback-schema.ts: provide schemaVersion: 1, companyId, taskId, dimension, observedBehavior, expectedBehavior, at least one reproductionSteps entry, and at least one publicEvidenceRefs entry. Choose the dimension from the same quality registry used by validation assertions. parameterSetId defaults to catalog-default; executable members use the same bindings as campaigns. Set run to null for a general configuration report, or to { "kind": "episode", "id": "run-id", "caseId": null, "scenarioCommitment": null }. Run kinds also include evaluation and campaign; use the actual case ID and public commitment when available. Do not include private scenario seeds.

Public references have one of three shapes: { "kind": "catalog_source", "sourceId": "source-id" }, { "kind": "observation", "observationId": "notice-id" }, or { "kind": "url", "url": "https://example.com/public-report" }. Catalog source IDs must exist; observation references require a run reference. URLs and run references remain declared evidence, without an automatic truth or replay claim.

The recorder resolves the actual company/task and named parameter member, then stores hashes of that configuration, source evidence and complete execution inventory. It stores no hidden world specification. An optional expectedBindings object copied from a previous feedback record rejects recording against changed source or case configuration. These bindings support reproduction with the matching release and retained run artifacts; the feedback file alone does not reconstruct a private trajectory. Files use mode 0600 and refuse replacement. After review, turn an accepted report into a regression case and a new registered assertion; recording feedback itself never supplies passing validation evidence.