Running agent experiments

The benchmarks/ package runs Codex, OpenCode, Claude Code, and Pi through their native SDKs. Each attempt gets a fresh workspace and a company connection through worldseq’s player SDK. The runner retains the agent session and closes the company through the existing Harbor collector. A separate container checks the retained evidence using the existing verifier and scoring rules.

Experiments run on demand. Until an experiment is exported, the homepage and Benchmark page show clearly labeled sample charts for the configured companies and systems. These illustrative values are separate from experiment records and reports. The first verified export replaces both previews automatically when the site is rebuilt.

Configure accounts and models

Use Bun 1.3.14, Docker with Compose, and uv. The player image includes a pinned Node 24 runtime for the native agent SDKs; Bun runs the player’s worldseq helper and operator tools. Install the root dependencies and build the TypeScript SDK with bun run sdk:build, then install the separate package with cd benchmarks && bun install --frozen-lockfile.

benchmarks/experiment.json names complete systems. The current pilot selects Codex / GPT-6 Astra at ultra; OpenCode / GLM-5.3, DeepSeek V4.1 Flash, and Kimi K3 at max; and Pi / Muse Spark 1.3 at xhigh. Claude Code is omitted. Five systems across three prompts, six tasks, two paired scenarios, and two repeats produce 360 planned attempts. These are explicit account selections; the runner never chooses a replacement model or billing method.

Muse Wonder was disabled in the account catalog at configuration time. The authorized replacement is Spark through the standard Console (Zen) gateway. Spark’s highest listed effort is xhigh; GLM-5.3 supports max. This selection is fixed before registration, rather than applied as a fallback during an attempt.

For DeepSeek, the pilot uses deepseek-v4.1-flash: DeepSeek reports stronger performance than V4 Pro, despite the older model’s Pro name. For Kimi, it uses kimi-k3, which Moonshot describes as its most capable model. These provider reports inform model selection, not conclusions about worldseq performance. Both models passed live access checks through the isolated OpenCode adapter with the existing Zen organization and max reasoning. Their model labels carry the DeepSeek and Kimi icons; the OpenCode name keeps the OpenCode harness icon.

Codex supports ChatGPT sign-in. OpenCode Go uses a subscription-linked key. The Claude Agent SDK supports Claude plans. Availability and account limits depend on the selected plan. Pi resolves models from its installed registry; unknown models fail instead of falling back.

Authentication references belong in the experiment; credentials do not. Set the environment variable named by keyEnv, or point directoryEnv at a dedicated directory containing only the permitted login file:

The runner copies only that file into the container. It never mounts your home directory. Refreshed login credentials are copied back after the session. Use your provider’s supported login flow to create them. A macOS keychain entry alone is not a portable container login file.

For the current pilot, set BENCH_CODEX_AUTH_DIR and BENCH_OPENCODE_AUTH_DIR in an ignored benchmarks/.env.local. Set EXA_API_KEY and FIRECRAWL_API_KEY there for research. Run the CLI from benchmarks/ so Bun loads that file. OpenCode and Pi use the same directory reference and account name, so they share one concurrency allowance.

The auth.source: "opencode-console" setting accepts an auth.json containing the opencode-console OAuth entry saved by Pi’s Console login: access token, refresh token, expiry, and organization metadata. It refreshes that login when needed and passes its token and organization header to the pinned SDK’s selected OpenCode provider. It does not copy an OpenCode database or inherit host plugins and settings. A refresh that changes organization, or a token that cannot last through the attempt’s runtime limit, stops execution. This mode requires matching provider and billing settings: opencode with api for Console/Zen, or opencode-go with subscription for Go. It never switches between these routes on a failed request.

allowedHosts lists the exact HTTPS hosts needed by that account, including any required token-refresh endpoint. No wildcard hosts or private addresses are allowed. Inference remains available to all profiles. Ordinary internet access is blocked; the web profile receives research through the separate proxy. Native browser/search tools are disabled so research access is comparable across harnesses.

Use the same auth.account name for every system sharing one account. The scheduler defaults to one attempt per account. limits.accountConcurrency can explicitly allow more attempts on an account using API keys or a Console login. The earlier six-family pilot sets limits.concurrency: 5 and limits.accountConcurrency: { "opencode-personal": 4 }; the local Convene configuration lowers both to two. Slots are spread across named systems when possible. Native OAuth sessions on one ChatGPT or Claude account remain serialized to protect refresh-token rotation. A quota error retains the active result and pauses remaining work. No automatic model, provider, or billing fallback is introduced. SDK transport retries may still occur within a request; the runner never replaces an attempt with a successful retry.

For a parallel Console account, the operator serializes login refreshes before copying independent credential snapshots. Those containers cannot refresh the shared login or copy stale credentials back. An account lease prevents a second experiment or access check from using the same account concurrently. Limits are frozen at registration; changing parallelism requires a new registration and never rewrites active or completed attempts.

Define the experiment

The versioned schema is benchmarks/schemas/experiment.v1.json. The source schema rejects unknown settings, including credential values in system configuration.

One-company selection: Convene

The short pilot uses benchmarks/experiment-convene.json: one task, five systems, three prompts, one shared scenario, and one attempt per combination, producing 15 attempts. At most two run concurrently. The additional ChatGPT harness comparison in experiment-v.json is deferred so that this run uses the five previously tested systems.

Convene is a collaboration company moving upmarket into larger deployments. The selected task, bridgeboard.commercial-governance, spans 180 simulated business days. Sales opportunities need commercial prerequisites and onboarding assistance while payment disruption reduces available cash. Agents must allocate capacity, preserve customer continuity, and review delayed evidence. This combination motivates the task choice; it is not an empirically established difficulty ranking. The pilot does not cover Convene’s other 39 public tasks.

A scenario defines the starting situation shared across systems and prompts. A repeat is a fresh attempt of the same task and scenario. Here scenariosPerTask: 1 and repeats: 1 mean each system/profile combination runs once. There are no automatic retries and no repeated paid attempts from earlier experiments. Runtime and reported-token caps remain null; worldseq’s existing call and decision budgets still apply.

Publish this pilot’s verified outcomes, usage, trajectories, and failures as observations of this case. One scenario cannot estimate general model performance or establish a prompt effect. The existing statistical analysis returns inconclusive comparisons without uncertainty intervals when there are too few independent scenarios. All three prompt profiles remain visible, and unsuccessful attempts remain in the dataset.

The earlier Riff evidence remains preserved separately. Separate future registrations use separate scenario commitments and must not be treated as paired with this pilot merely because task or case IDs match.

Two-company remote run

The earlier remote proposal selected Stratum and Convene, with every public task and all three prompts. It was not launched: the RunPod instance denied the nested Docker environments required by the runner. The configurations are retained separately from the current Convene pilot. Both companies include enterprise and operational challenges and year-long mandates. The choice reflected task structure and variety; it did not establish that these are the empirically hardest companies.

Stratum contributes 42 tasks and Convene contributes 40. The first configuration retains the systems from the five-company run. The second retains OpenCode and Pi using GPT-6 Astra through ChatGPT from experiment-v.json. Their model, provider, reasoning, prompt, and repeat settings are preserved. Runtime and token caps remain null. Any future launch needs a Docker-capable host and live authentication checks; parallelism must be chosen before registration. The native ChatGPT profiles share an account with Codex and cannot currently run concurrently under separate runners.

The paused five-company run and its frozen source, images, private scenario commitments, and evidence remain intact. Its existing attempts all concern Riff, so none can count as a Stratum or Convene result. Reuse is determined by task, actual scenario commitment, prompt, model settings, and execution compatibility, never by whether the outcome was successful. Case IDs alone do not establish identical scenarios across registrations. Keep both successful and unsuccessful compatible attempts when reusing a study; do not relabel old attempts as new executions.

The two configurations have separate scenario seeds when independently registered. Their within-configuration comparisons are paired; cross-configuration comparisons must not assume the same underlying scenarios.

Five-company challenge selection

benchmarks/experiment-challenge5.json selects every public task for five companies. Riff, Lucent, Stratum, and Convene have the four broadest public task catalogs. Pollen adds multi-product developer tools and dependencies between products. This selection emphasizes task variety and interacting constraints; it is not a measured difficulty ranking. Publishing and memberships are not represented.

The explicit inventory contains 173 task IDs. Every task runs with all five configured systems and the three distinct prompts (ceo, ceo-modeling, and ceo-web), on two shared scenarios with one repeat: 5,190 attempts, or 1,730 per prompt. The focused and three-task subsets are not used. The earlier 360-attempt configuration remains separate.

Tasks cover collections, contract evidence, recovery, growth experiments, hiring, onboarding, and longer operating plans. Payroll, service, customer retention, and cash constraints continue alongside the task-specific decisions. Exact tasks, prompts, scenarios, and settings are frozen at registration. This breadth does not establish that the authored economics match real startups.

The schedule permits five concurrent attempts, including up to four on the shared OpenCode account and one on Codex. The scheduler spreads slots across systems, then across prompt profiles; it does not finish the entire CEO profile before starting the others. Runtime and reported-token caps remain null; existing worldseq call and decision budgets still apply. Seven-day smoke timings do not reliably estimate the longer tasks. Provider limits pause further scheduling while retaining the active attempts.

Modeling and research comparisons are defined separately within each system, using task completion as the shared outcome. Raw business outcomes retain their units. Two scenarios and one repeat give limited evidence about variability; this remains an exploratory pilot.

Use preview experiment-challenge5.json and check experiment-challenge5.json before register experiment-challenge5.json. The registered run directory is .runtime/benchmarks/ceo-challenge5-v2. Source changes after registration require a new experiment; a frozen checkout keeps the running experiment independent of later development.

Later experiments

Keep later configurations in separate files, such as experiment-v.json, with a new id and outputDirectory. Pass that filename to preview, check, and register; renaming the active configuration is unnecessary. Each exported experiment gets its own page, and the benchmark index lists all published experiments. The homepage and main benchmark view use the same featured experiment; other exports retain their own pages. Results from different experiments are not silently merged into a paired comparison.

The supplied benchmarks/experiment-v.json prepares OpenCode and Pi through the same ChatGPT subscription as Codex. OpenCode uses provider openai; Pi uses openai-codex. Both request gpt-6-astra at max, the highest level exposed for this model by the installed Pi registry. This differs from the first pilot’s Codex ultra configuration. Authentication and live model checks are deferred; the file is a proposed schedule, not a verified run.

Set BENCH_OPENCODE_CHATGPT_AUTH_DIR and BENCH_PI_CHATGPT_AUTH_DIR later, using each harness’s supported ChatGPT login flow. OpenCode supports ChatGPT sign-in through its OpenAI provider. Each directory must contain only auth.json, with only that provider’s native OAuth entry. These are different formats from Codex’s auth.json. The preflight rejects API-key credentials for these subscription systems. Both account references are codex-personal, so their attempts cannot overlap with the existing Codex account. OpenCode’s built-in OAuth plugin is enabled for these runs; host plugins and configuration are not copied.

The later matrix has 144 attempts: two harnesses, three prompts, six tasks, two scenarios, and two repeats. It registers modeling and research comparisons within each harness, plus one OpenCode/Pi comparison for each prompt. Cross-harness comparisons use configured-system because the native provider routes differ; sharing a model name does not establish identical serving behavior. Its ID is ceo-pilot-v, its private directory is .runtime/benchmarks/ceo-pilot-v, and its public export can go to site/public/benchmarks/ceo-pilot-v. Separate registrations generate separate private scenario seeds, so the first experiment’s Codex results are not paired controls for this later run.

• Systems name harness/provider/model combinations, auth references, reasoning, endpoints, and auxiliary models. Provider and model are never independently multiplied.
• Profiles are ceo, ceo-modeling, and ceo-web.
• Task sets contain explicit task IDs. Matrix scenarios are shared between systems and profiles. Repeats revisit the same scenario with a fresh agent session.
• Include/exclude entries select a system, profile, and task. Exclusions retain their reasons. Duplicate, unknown, empty, and oversized combinations fail validation.
• Limits include runtime, reported model tokens, research calls, concurrency, and total attempts. Set maxRuntimeSeconds and maxReportedTokens to null to measure elapsed time and usage without capping them; all supplied experiment configurations do this. Positive numeric limits remain available for explicitly budgeted experiments. Infrastructure startup and verification have separate timeouts, and providers can still enforce their own limits. worldseq’s existing Harbor budget is 128 accepted decisions and 1,024 player calls; version 1 records these fixed limits and rejects different values. Reads and rejected operations consume calls too.
• Comparisons identify baseline and treatment configurations, the question, outcome, and equivalence tolerance. Same-model comparisons require matching provider, model, endpoint, and reasoning settings.
• Output specifies the experiment ID, pilot/confirmation stage, and private artifact directory.

The example contains one company from each of the six startup families. Its full matrix is a proposed pilot, not an executed result or a recommended final sample size. Start with a smaller explicitly configured access/integration check. Use the six-family pilot’s measured runtime, usage, failures, and variability to choose a separate confirmation experiment. Freeze prompts and configuration before registration.

Agent labels and logos

Each system can supply optional presentation settings in experiment.json:

"display": {
  "label": "OpenCode · DeepSeek",
  "modelLogo": "/images/agents/deepseek.ico"
}

Harness names always use their harness icon. Model names use the model maker’s icon, with defaults for GPT, Claude, GLM, Muse, DeepSeek, and Kimi. Set modelLogo to override that model icon. To add an asset, place a PNG, SVG, WebP, or ICO file in site/public/images/agents/ and reference its /images/agents/<filename> path. Remote URLs and credentials are not accepted. The older logo field remains a fallback for unknown model families; it never replaces the harness icon. Chart labels show both harness and model, and the serving provider remains visible in the system details.

These settings are retained in the registration and exported dataset. Set them before registration; a frozen experiment cannot be edited in place. The site uses them for legends and tables and checks the local assets during build verification. The included icon sources are recorded in site/design/agent-logos.md.

Prompt profiles

Every run receives a CEO mandate covering objectives, resources, constraints, evidence, decisions, and follow-through. ceo-modeling adds explicit encouragement to calculate, retain state, build causal models, update beliefs, and run experiments. ceo-web adds external research.

The basic and web mandates do not suggest coding, notes, persistent state, or mathematical modeling. Those capabilities remain available if the agent chooses them. SDK syntax is supplied separately in OPERATING-SYSTEM.md, not as business strategy. Exact prompt components and the complete initial message are retained.

The adapters use native agent loops and compaction. Claude retains its native system preset and available subagents; OpenCode captures available child sessions. Pi uses its built-in coding tools and compaction without loading host extensions. Codex’s SDK may omit child-session activity. Native user plugins, local configuration, and personal skills are not imported from the host. Effective settings are determined by the registered adapter source and image, with explicit model settings retained in the dataset.

Check, register, and run

From benchmarks/:

bun run bench preview experiment.json
bun run bench check experiment.json
bun run bench check experiment.json --live
bun run bench register experiment.json
bun run bench run ../.runtime/benchmarks/ceo-pilot
bun run bench resume ../.runtime/benchmarks/ceo-pilot

Preview prints every attempt and exclusion without making model calls. Check validates tasks, auth files, environment references, and local dependencies. --live additionally makes one small inference request per system in the same isolated image and egress policy. It consumes account usage. Registration builds an immutable player image and saves the schedule, exact prompts, source identity, SDK versions, and scenario commitments. Private scenario seeds remain in private storage.

Run and resume use the registered configuration. Source changes require a new registration. A started attempt is never silently rerun, even if the host crashed before it wrote its completion record. It remains interrupted. For an explicit retry of a finished attempt, use bun run bench retry <private directory> <original attempt ID>. The retry gets a new ID and a link to the original, uses the same private scenario, and appears in the export. It never replaces the original in registered comparisons. Interrupted attempts require a new experiment; the runner does not guess whether external work is still active.

If business-ledger collection or copying fails, the runner stops the attempt’s containers and retains their files and database. Its private recovery.json identifies the Compose file and failure; finished.json records containersRetained: true. Recover that evidence before pruning containers or volumes. A verification failure after the ledger has been copied does not need the original containers. Neither failure automatically reruns the agent.

Ctrl-C stops scheduling, cancels the active player request, stops the container, and collects available evidence, including when no agent runtime cap is configured. Collection and verification have their own timeouts. After a hard process or host crash, inspect the retained run.lock and clean up that experiment’s wsbench-<attemptId> Compose projects before removing the lock. Resume starts only untouched attempts.

OpenCode submits the mandate through its asynchronous API and waits for the root session’s completion or error over the event stream. The subscription is connected before submission so a short response cannot finish before capture begins. This avoids the reported five-minute Node HTTP timeout when a synchronous prompt request stays open for the entire turn. An accepted request is not treated as a completed attempt.

Research and capture

Set the configured Exa and Firecrawl environment variables for web runs. The proxy exposes Exa search and Firecrawl scraping as MCP tools, applies a shared research-call budget, excludes configured worldseq source/result domains, and records returned sources. Research credentials and original research documents stay in private storage. Review the exclusion list for your own domain and repositories. Exclusions cannot eliminate all contamination, and externally retrieved text is not trusted company evidence.

The agent workspace cannot read simulator source, private scenarios, database credentials, or verification artifacts. The proxy is the only container with outbound internet access. Allowed inference hosts remain a provider trust boundary; host filtering cannot inspect encrypted provider requests or prevent provider-side routing changes.

Native provider events remain private for auditing conversion. Public trajectories use ATIF-v1.7 and pass Harbor 0.24.0’s trajectory validator after redaction. They preserve tool-call references and available subagent relationships. They omit hidden reasoning and do not fabricate missing tool results. An attempt that never starts a session can have no trajectory.

Export and publish

bun run bench report ../.runtime/benchmarks/ceo-pilot ../site/public/benchmarks/ceo-pilot

The output directory must be new and outside private experiment storage. Export reruns the independent verifier, sanitizes records, validates ATIF, and writes:

Workspace files stay private by default. To publish selected text artifacts, supply a JSON array of { "attemptId": "...", "path": "analysis.md" } as the final report argument. Only small supported text files inside that attempt’s workspace are accepted. Symlinks, hidden files, runtime inputs, and dependency directories are rejected. Review selected material before publishing.

The website reads these files directly and checks their inventory and hashes at build time. Measured result values come only from the export. The sample preview is defined in site/src/components/benchmarks/preview-data.ts and is shown on the benchmark article until the featured study has an export. It is labeled synthetic and does not describe the configured systems’ measured performance. Publishing the site is a separate deployment step. Large trajectories load only when opened.

Read the results

A completed session can still fail the company mandate. Eligibility means all existing task constraints passed; the raw primary outcome keeps its original unit. No new score is introduced. See Scoring and interpreting results for task eligibility and the Harbor reward transform.

Attempt records distinguish unfinished work, agent failures, provider limits, infrastructure failures, and verification failures. A verified business failure is a completed attempt with unmet constraints. Missing outcomes stay missing. Subscription charges per attempt are unknown; reported token counts and any API-price estimate are separate fields. Usage coverage differs by SDK, and token limits are enforced when usage is reported, not before an already-running model request finishes.

Normalized output usage includes reported reasoning tokens. OpenCode supplies visible output and reasoning as separate counters, which are added together. For SDKs whose output counter already includes reasoning, that count is used directly. Token counts do not expose reasoning content; raw provider events remain private.

Comparisons pair the same task, private scenario, and repeat. The existing worldseq statistical routines group related observations and calculate paired differences and 95% bootstrap intervals. They require at least two scenarios per task. Failed or unfinished attempts make a comparison inconclusive; they are not dropped or filled with successful outcomes. Different primary metric types are not pooled. Weeks from one run are never treated as separate samples. A same-model label does not establish identical serving conditions across providers.

Daily records include day zero. Weeks cover days 1–7, 8–14, and so on, including partial final weeks. Cash, MRR, accounts, and backlog are closing stocks; changes in cash and recognized revenue are period flows. Timeline records describe actual business activity, which can differ from information available to the CEO at the time. Decision markers help inspect what happened next, but a selected trace alone does not establish causation.

Findings cite computed comparisons. Use trajectories to investigate repeated failures or decisions, and retain contrary cases. Pilot findings remain exploratory. Internal verification establishes consistency with worldseq’s rules; representativeness of real startup economics requires external calibration.

Charts and company comparisons

The Prompt profile buttons keeps CEO, CEO + modeling, and CEO + web results separate. Agent colors stay consistent across charts, and the legend can hide individual systems without changing their recorded results. A system retains its harness, provider, and model identity.

Average performance across companies is a descriptive task-success percentage. Each company has equal weight; tasks, scenarios, and repeated attempts receive equal weight within their parent. Success means verified eligibility, not optimality. Every system must have a completed outcome for the same selected case inventory. Missing, failed, or unstarted attempts prevent a numeric average instead of being discarded or counted as business failures. Retries are excluded. This company-equal overview differs from the existing family-weighted registered comparisons, which remain the source of paired effect sizes and bootstrap uncertainty.

The company table shows the same success calculation for each company. Choose a company and task to compare cash, MRR, or paying accounts over business time. Daily means require the same full case inventory for each system; a missing day breaks that system’s line. The chart does not continue an unfinished run or average only surviving runs. Cash from different companies is never pooled into a purported universal business score.

Tokens consumed by agent can show input, output, cached input, or input plus output; choose a mean per reported run or the total reported consumption. The scope can cover all companies or the selected task. Each field has its own report count. Missing usage stays unavailable, true zero counts remain zero, and partial capture is disclosed. Reported consumption includes failed attempts when usage is available. Cached input is already included in input and is never added a second time. Provider-specific tokenizers and subagent capture differ, so token counts are a resource measurement, not actual subscription charges.

The outcome-versus-tokens scatter plot uses individual verified runs from one task and profile. A point requires a ledger balance at the task horizon and reported input and output counts. Eligibility and capture coverage are available in the accompanying table. The plot shows association; it does not demonstrate that using more tokens caused a better outcome.

The experiment article and evidence archive

The landing page shows one financial-outcome chart and links to the results article. For the Convene pilot, the value is the verified change in net assets, converted from cents to dollars. Each prompt has equal weight. Verified runs that missed a requirement remain in the average, with their task status reported separately. Financial averages require matching metrics and horizons; mixed outcomes fall back to a task-success summary. Missing outcomes in any profile leave a system incomplete, and retries stay excluded.

The article presents measured findings, a financial comparison, business timelines, reported tokens, and observed prompt differences. Financial outcomes are ordered highest first; token usage is ordered smallest first. The prompt table compares the same task, scenario, and repeat within each system. It reports observed changes without inventing uncertainty intervals for a single scenario. Exact setup, daily tables, capture coverage, and trajectories are on linked methods and experiment pages. Sample charts remain labeled until a verified export replaces them.

The model-cost chart uses retained SDK estimates first, with a versioned provider/model price snapshot for missing estimates. It offers dollars per successful task, cost per run, total spend, and successful tasks per dollar for the selected prompt or all prompts. Aggregate ratios divide total spending by total successful original attempts; failed attempts still consume money. Unknown costs remain missing, and partial capture produces a minimum cost per task (≥) and a maximum tasks-per-dollar ratio (≤). These are labeled limits, not complete cost estimates. These API estimates exclude subscriptions, research, compute, and fees. The methods page links the exact rates and sources; publication never rewrites the retained usage or actual billing fields.

The benchmark page presents methods, prompt links, results, and evidence in one article. Its featured study facts come from benchmarks/experiment-convene.json; counts are calculated from the expanded schedule. The three prompt pages read the same source texts as the runtime. Historical experiment views use their own registered prompt copies, not the latest source files.

Every chart has an italic reading note. Tables stay visible, and model/provider names appear in hover details alongside the active series logo. Evidence filters and paginated attempt tables let readers inspect an individual task, its brief, business timeline, trajectory, and selected artifacts. A trajectory is fetched only when the reader requests it.

Keep the private experiment directory and the source checkout used for registration together. It contains registration and scenario material, attempt receipts, native events, workspace files, trusted ledgers, and verification outputs. Keep the pinned player and verifier images needed for replay. A public export is a reviewed derivative, not a replacement for this private evidence. Do not upload the private directory to a public repository.

Export to a new directory, check the manifest and trajectories, and then publish that complete directory under site/public/benchmarks/<experiment-id>/. The public manifest links each trajectory and selected artifact to its attempt; the dataset links that attempt to its system, profile, task, and scenario commitment. Publishing a partial experiment is supported, but missing attempts remain visible and do not become successful outcomes.