How we ran the pilot

The experiment

A company can grow and still run out of cash. Winning a contract creates work for the delivery team. Hiring takes time before it adds capacity. In worldseq, agents have to manage these consequences through the end of a company mandate.

This exploratory pilot follows one company through one commercial-readiness task. Each of the5 systems starts from the same scenario under 3 prompts. We look for differences in the decisions and outcomes, and use them to develop questions for a larger study.

1 task × 5 systems × 3 prompts ×1 scenario × 1 attempt per combination =15 planned attempts

The company and the task

Convene sells collaboration software and is moving from single teams to larger deployments. Bigger contracts require commercial approvals and onboarding work while payment disruption puts pressure on cash. We chose this task because these demands compete for capacity over six simulated months. This is one case from Convene's catalog, not a test of every task or an established ranking of difficulty.

These are authored SaaS simulations. Their rules let us test decisions consistently; they do not establish how well the same economics describe real startups. Read the modeling assumptions.

The systems in this run

Exact model IDs and requested reasoning settings

Harness	Model / provider	Reasoning	Billing
Codex	gpt-6-astraServed by openai	ultra	Subscription
OpenCode	glm-5.3Served by opencode	max	API credits
Pi	muse-spark-1.3Served by opencode	xhigh	API credits
OpenCode	deepseek-v4.1-flashServed by opencode	max	API credits
OpenCode	kimi-k3Served by opencode	max	API credits

Up to 2 attempts run concurrently. There is no agent runtime capand no reported-token cap. Each attempt has at most 1,024 worldseq calls and 128 decisions. The web profile can make30 research calls. Task-specific business constraints still apply. Token counts measure reported usage; they do not measure the actual cost of a subscription.

Three prompts, the same responsibility

All three profiles start with the same CEO mandate and the task's company brief. The modeling profile adds explicit modeling instructions; the web profile adds research instructions and access to research tools. The links below show the exact text. SDK instructions are supplied in a separate reference.

Research is available only in the web profile. Known worldseq source and results domains are blocked, and retrieved sources are recorded. This reduces exposure to solutions but cannot eliminate contamination. Prompts and configurations are frozen in each registration.

What counts as success

After each run, worldseq seals the retained evidence, replays the actions, and checks the task's constraints. An agent's claim of success is not enough. The simulator and verifier run outside the agent's workspace, and the result uses worldseq's existing scoring rules.

The financial comparison uses the task's primary outcome: closing net assets minus opening net assets, converted from cents to dollars. It includes receivables, liabilities, and remaining service obligations. Each prompt has equal weight in the homepage average. A financial gain does not excuse a failed task constraint; verified failures remain in the comparison.

Prompt differences compare the same task, scenario, and system. They describe the observed runs. One scenario cannot support an uncertainty interval, and weeks from one run are not independent trials. A retry stays separate from the original results. Read the scoring rules.

Model cost and task efficiency

We use the SDK's retained cost estimate when available. Otherwise, we apply the exact provider's published rates to the recorded input, cached input, and output tokens. Cached input is already part of input, so we subtract it before pricing uncached tokens. These are API estimates in US dollars, not allocated subscription charges or the company's simulated money. Research, container compute, taxes, and payment fees are excluded.

Successful tasks per dollar is the number of original runs that met every requirement divided by their total estimated model spend, including unsuccessful runs. Cost per run is total estimated spend divided by the number of original runs. Dollars per successful task uses total spend divided by successful runs; with no successes that ratio is unavailable. The “All prompts” view pools the three profiles before calculating ratios. Retries are separate. Unknown outcomes and missing costs prevent a tasks-per-dollar calculation; they are never treated as free successes. Partial capture gives a minimum cost per task and a maximum number of tasks per dollar, both marked on the charts. With one run per system and prompt, this is an observed ratio, not an estimate of future reliability.

The Codex run uses a subscription and has partial token capture. Its displayed amount is a minimum for recorded tokens at Standard API rates. The aggregate record does not distinguish cache writes or requests above 272,000 input tokens; those can cost more. We show that minimum with “≥” for costs and “≤” for successful tasks per dollar. These limits are not complete cost estimates. Cost minima round down and efficiency maxima round up at the displayed precision. Its subscription's actual per-run charge remains unknown.

OpenCode and Pi use the paid Console/Zen model variants. Their retained SDK estimates take precedence over this price snapshot, so a later price change cannot silently rewrite the measured comparison. Download the versioned rates and source links.