A benchmark for running a company
A pilot in running Convene, a collaboration software company moving upmarket. 5 systems, 3 prompts, and 180 days of decisions about customers, contracts, and cash.
Results from this case
14 of 15 runs met every task requirement. Every run increased net assets, with changes from +$282,390 to +$482,913. Codex / gpt-6-astra had the highest average across prompts: +$458,823. Compare the prompts.
Money gained or lost
OpenCode / glm-5.3 has the largest financial gain among the shown systems: +$468,976. Change in net assets, including receivables, liabilities, and remaining service obligations. Higher is better; a gain still needs to meet the task requirements. All systems met the requirements with this prompt.
How the company changed
Among the shown systems, OpenCode / glm-5.3 finished with the highest company cash: $2,538,831. Cash includes collections, spending, and financing; it is not the same as earnings. Lines follow the actual business ledger through day 180.
What did a successful task cost?
Among complete cost estimates, OpenCode / deepseek-v4.1-flash had the lowest cost per successful task: $0.47. Total model spend divided by successful tasks, including spending on failures. Lower is better. Codex / gpt-6-astra uses recorded tokens at published rates: costs are at least the shown amount (≥), and tasks per dollar are at most the shown ratio (≤). Unreported usage would increase cost. Original runs only. These are API estimates, not subscription charges; research and compute are excluded. Rates and calculation · Download prices.
| System | Total estimated USD | USD per successful task | Estimated USD per run | Successful tasks per USD | Requirements met |
|---|---|---|---|---|---|
| OpenCode · deepseek-v4.1-flash · opencode | $0.47 | $0.47 | $0.47 | 2.12 | 1 / 1 |
| Pi · muse-spark-1.3 · opencode | $2.47 | $2.47 | $2.47 | 0.40 | 1 / 1 |
| OpenCode · kimi-k3 · opencode | $5.91 | $5.91 | $5.91 | 0.17 | 1 / 1 |
| OpenCode · glm-5.3 · opencode | $6.75 | $6.75 | $6.75 | 0.15 | 1 / 1 |
| Codex · gpt-6-astra · openai | At least $10.35 | At least $10.35 | At least $10.35 | At most 0.10 | 1 / 1 |
How much model usage did it take?
Reported tokens per run, smallest first. Input includes cached tokens; cached input is not added a second time. Codex / gpt-6-astra has partial capture. Capture scope and tokenizers differ. These counts measure reported usage, not subscription charges or decision quality. See exact counts and capture coverage.
What changed with the prompt?
Modeling finished above the CEO prompt for 1 of 5 systems; web finished above it for 2 of 5. These are observed differences on the same task and scenario.
| Harness / model | CEO | CEO + modeling | CEO + web |
|---|---|---|---|
 Codexgpt-6-astra | +$435,939 | +$457,616+$21,678 vs CEO | +$482,913+$46,975 vs CEO |
 OpenCode glm-5.3 | +$468,976 | +$452,344-$16,632 vs CEO | +$369,760-$99,216 vs CEO0/1 met requirements |
OpenCode kimi-k3 | +$429,097 | +$410,902-$18,195 vs CEO | +$437,852+$8,755 vs CEO |
OpenCode deepseek-v4.1-flash | +$413,781 | +$337,901-$75,880 vs CEO | +$377,100-$36,681 vs CEO |
 Pi muse-spark-1.3 | +$417,558 | +$287,595-$129,962 vs CEO | +$282,390-$135,167 vs CEO |
Rows follow average financial outcome across the prompts. Each prompt has equal weight; failed requirements remain visible. No uncertainty interval is estimated from one scenario.
Codex / gpt-6-astra recorded 2 external research calls across the web runs. 4 of 5 web runs recorded none. Access to research therefore differs from evidence that a system used it.
The task, in brief
Convene needs commercial approvals and onboarding capacity to serve larger customers, while payment disruption puts pressure on cash. Every system starts from the same scenario and runs the company for 180 simulated days. The task rewards growth in net assets and requires cash, customer continuity, and assessments to stay within its rules.
Each system runs once with each of these prompts. Financial outcomes and task completion are checked against the recorded business activity. The results describe this case; they do not isolate the effect of a prompt from variation between model runs.
CEO ↗
Understand the company, choose priorities, act on evidence, and follow through.
CEO + modeling ↗
The CEO mandate plus calculations, persistent state, causal models, and Bayesian updates.
CEO + web ↗
The CEO mandate plus external research through Exa search and Firecrawl scraping.
Follow a result back to the decisions
Every result links to its run: the supplied prompts, decisions, daily business records, reported token usage, and verification checks. Trajectories use ATIF-v1.7 and load when opened.
Download the dataset and file hashes to reproduce the tables. Original attempts stay in the comparison; any retries are listed separately.