Evaluations and submissions

An agent manifest pins its execution mode, model declaration, sampling settings, configuration digest, supported capabilities, source/license and required secret names. Secret values never belong in a manifest. Check it before a run:

bun run worldseq agent check agent.json
bun run worldseq eval --agent agent.json --suite public-dev --seeds 1 --repeats 1 --out runs/dev
bun run worldseq results verify runs/dev
bun run worldseq submission pack --agent agent.json --results runs/dev --out submissions/my-agent
bun run worldseq submission check submissions/my-agent
bun run worldseq submission import submissions/my-agent

The CLI never uploads, sends a message, opens a PR or publishes results. Supply the checked directory as a maintainer intake artifact with access to the immutable agent image or connector source. A maintainer can establish an artifact-drop or metadata-PR convention without introducing another protocol.

Read scoring and interpreting results to distinguish eligibility, raw business outcomes, individual assessment scores and Harbor rewards. The ordinary evaluation summary does not calculate one combined benchmark score.

Follow an evaluation

From the worldseq package directory, start bun run dev, then run the evaluation command in another terminal. It appears automatically in the same Run history as human-play sessions. Open it for live task progress, current worker activity and results. The sidebar’s Run status section shows up to 15 runs, with active work first and recent history filling the remaining places. The runner and visible dashboard refresh every five seconds. After 30 seconds without a heartbeat, the dashboard shows “No recent activity” and resumes when updates return; it does not claim a stale worker is running.

Additional trusted runner processes use the same WORLDSEQ_DATABASE_URL as the dashboard. Keep that credential in the runner, never the evaluated agent. The default local database matches the development launcher, including WORLDSEQ_DATABASE_PORT. --no-dashboard disables database reporting. If the database is unavailable at startup, execution continues with a warning and a live progress.json in the results directory. A later database outage cannot change a score: the runner keeps executing and retries progress writes on subsequent updates.

Progress records describe execution; independent replay and business verification still determine result validity. Final progress is written before artifact checksums, and the dashboard reports completion after checksums are saved. Submission packing and checking remain artifact workflows; they do not enqueue a hosted evaluation. submission import independently checks and replays a bundle before inserting its complete result inventory into the local dashboard. The import has a stable identity: repeating it does not duplicate history. Imports are terminal, labeled replay verified and explicitly retain self-reported model identity. Keep the bundle at its displayed path; the database stores the verified public projection, not another copy of private trajectories. A mismatched checkout or corrupt artifact is rejected before insertion.

Execution tracks

• Scripted: the bundled transparent control policies; no model inference.
• Remote declared: your HTTP endpoint owns inference and memory. The runner controls simulator access and counts declared tokens, but cannot attest hidden remote compute. WORLDSEQ_AGENT_TOKEN is the only supported injected credential, sent as the endpoint’s bearer token. Redirects are refused.
• Container isolated: a digest-pinned image with a newline JSON protocol. Docker receives no host mounts, Docker socket, credentials or network; root filesystem is read-only, with bounded CPU, memory, PIDs and temporary storage. Use this for self-contained agents. External model inference requires the remote track.

Pull a container’s exact digest-pinned image before evaluation with docker pull <image>@sha256:<digest>. The runner inspects the local image and rejects missing images and declared writable volumes before starting a participant.

Container isolation is not a model-provider token meter. Never compare these tracks as if their compute assurance were identical. Model identity and token declarations remain declarations without an independently controlled inference gateway.

The standard per-episode budget is 128 accepted decisions, one million reported model tokens (including delegated calls), and two real hours. A 1,024-tool-call ceiling also prevents indefinite non-progress. The step runner’s automatic capabilities, snapshot and evidence reads consume that budget alongside requested actions; see the agent protocol. A model step is not one tool call. A model response arriving after the deadline cannot mutate the world. Three consecutive rejected operations close an agent attempt as failed. The complete accepted decision/assessment/wait journal is retained.

Bundle

submission.json
agent.json
checks/conformance.json
results/
  registration.json
  manifest.json
  summary.json
  progress.json
  episodes/case-1.json.gz
  ...
  checksums.sha256
checksums.sha256

Current result verification requires the independently reconstructed execution identity and budget, as well as the agent, catalog, case schedule, seeds and repeats. The execution identity binds simulator inputs, runtime dependency manifests/lock, public contracts, runtime versions, OS/architecture and pinned OCI image references. The legacy sourceHash is an additional narrower check, not a replacement for this identity. Omitting execution or finalization metadata cannot downgrade a current submission to historical grading. Old raw records may still be inspected locally; that does not make them eligible for current import or release credit.

Release gate receipts additionally bind the shipped SDKs, dashboard, packaging code, tests, documentation and build configuration through a separate release-input fingerprint. Changing those inputs invalidates release receipts without changing simulator certificate identity; rebuild and rerun the release workflow. Handoff and smoke receipts bind the same fingerprint, so an old SDK or dashboard archive cannot substitute for the newly tested source. Generated build output, dependency caches and private runtime files are excluded from this source fingerprint; archive inventories separately bind the delivered bytes. Every registered case must appear exactly once. Partial output after a process failure stays partial; verification does not silently accept it as a complete suite. The CLI refuses to overwrite output directories; missing parents are created automatically. Use --workers N (1–4, default 1) to run independent cases concurrently. The parent records every completion and preserves registration order in the manifest.

Episode trajectories use gzip to keep year-long ledgers manageable. Verification checks the compressed bytes against their registered hash before bounded decompression: at most 128 MiB compressed and 1 GiB expanded per episode. Metadata and checksums use separate bounded reads; oversized artifacts are rejected.

Checks reject missing/extra files, unsafe paths, symlinks, invalid JSON, changed checksums, identity mismatches, modified summaries, missing attempts and failed deterministic replay. Per-episode business verification independently reconstructs financial/eligibility facts from the primitive ledger. Worker, initialization, cleanup and sealing failures are infrastructure failures. Exceptions during an established agent session become agent failures; the runner cannot always separate a remote provider outage from an agent defect. Both remain retained attempts.

Local data includes replay inputs and trajectories. Do not publish private holdout bundles through this development submission format. An official runner must retain private episodes internally and release a signed aggregate/report under its own disclosed policy.

Assurance and maintainer admission

self-reported means a local runner produced a result. replay-verified means its registered decisions reproduce the sealed business trajectory and its ledger reconciles; it does not prove the named model produced those decisions. Neither label means an official or empirically calibrated benchmark result.

The signing primitive uses Ed25519 and requires a separately pinned trusted public key. keys init generates private/public material for a maintainer; it does not make a participant official. Do not trust a public key supplied inside a result.

For official admission, maintainers must provision a separate private world package and seed schedule, pin immutable agent/code/source inputs, enforce the published inference/egress policy, preserve all attempts and retry links, run the full schedule, fence the engine, collect trusted evidence, independently verify it and sign the release summary. This repository provides the development runner, replay verifier, isolation/Harbor path and signing primitives; it does not deploy an official hosted leaderboard or a provider-metered inference gateway.

Choose the admitted task inventory and measure execution costs in a registered pilot before sizing a study. Declare the detectable effect, desired power or precision, dependence assumptions and resource limits before inspecting confirmatory outcomes. The sample-planning workflow retains all pilot attempts, evaluates declared scenario counts and can freeze a new confirmation with fresh scenarios. It supports inference for the fixed task set; increasing scenarios cannot establish generalization to new companies. Public source code and public-test labels do not create a holdout.

Registered validation campaigns

Use validation check, validation register, validation run and validation report for a frozen multi-policy comparison. Unlike ordinary public development runs, campaign registration includes the complete execution identity, private unpredictable seeds and a separate owner-only control file outside the artifact tree. Reports independently replay all retained attempts and bind their statistical claims to the registered inputs. Unsupported sensitivity settings fail before launch; supported settings are applied to both execution and replay.

See validation campaigns for a complete plan and commands. Never upload private-campaign.json, the independent control file, or private per-agent result folders. The public registration and report are separate outputs. Campaign progress uses the same dashboard evaluation history and local progress files.