Scoring and interpreting results

worldseq reports several different quantities rather than one universal benchmark score. First check whether the run was successfully verified and eligible. Then interpret its business outcome and individual assessment scores. Harbor adds a specific ranking transform; registered campaigns compare a declared metric.

Plans, hypotheses, milestones and rationale text earn no points by themselves. Typed decisions change the simulated business, and typed assessment answers are graded against the registered question. No language model judges the prose.

Read the result in this order

In a shared-finalizer response, the business fields are under verification and the finalization fields under finalization. Harbor’s result.json puts the verification fields at the top level alongside finalization and reward. Ordinary evaluation summaries retain per-case verification; full attempt artifacts also retain finalization. These formats describe the same underlying checks.

Eligibility is a gate, not an average

An eligible result must satisfy every applicable constraint. These include the spending budget, cash and price bounds, reaching the task horizon, declared business objectives, agreement and growth requirements, mandatory assessment conclusions, and gameplay limits. A large profit cannot compensate for a missed required answer or a breached cash floor.

The task cash floor is checked across the recorded trajectory, including intermediate cash movements. Business objectives specify inclusive day windows: every required day must be present and meet its threshold. A final-day recovery does not erase an earlier breach inside the required window. Check constraints and business.objectives to identify the failed predicate, observed days and worst value.

The standard gameplay budget is 128 accepted decisions, 1,024 player calls and 7,200 real seconds. Reads, rejected operations and idempotent retries consume calls. The agent evaluation runner separately enforces its registered reported-token budget. Virtual business days and elapsed runtime are different clocks. See evaluations and submissions for adapter metering and failure handling.

The primary business outcome

The task declares one primaryMetric. The verifier selects primaryValue as follows:

For recovery, a cumulative shortfall of 200 produces primaryValue: -200. Reducing it to 50 improves the value to -50; zero means no measured shortfall. This is an absolute cumulative measure, not an availability percentage or a per-customer rate. The core and addon measures are the simulator’s authored service units, so compare like tasks and horizons.

Cash, earned revenue and net assets are different. Financing can improve cash while leaving a repayment liability. Annual prepayment creates cash and remaining service obligations. Net assets account for recorded receivables, obligations and deferred service; they are not a valuation of the company. The primary net_assets value is the change from day zero, whereas business.metrics.netAssetsCents is the closing balance. See business mechanics for the accounting definitions.

Raw monetary outputs use cents: 100000 means USD 1,000. A negative value can still be eligible if all declared constraints hold. Eligibility does not establish that the outcome is optimal, profitable, or better than a baseline. A run can also expose a raw outcome while being ineligible; report that distinction.

Growth contribution, retention ratios and other business measurements are supporting metrics or explicit task requirements. They are not automatically added to primaryValue. For example, cohort contribution excludes costs that still affect the company’s financial outcome. See growth experiments and usage billing.

Assessment scores

The latest accepted answer submitted strictly before dueOnDay is graded. Earlier answers are not searched for the best score. A question due on day 7 must be answered on day 6 or earlier. A missing, pending or invalid answer has no numeric score; null is not a perfect zero-error prediction.

Categorical diagnosis

The player supplies probabilities for every candidate. The grader sums probabilities within each registered equivalence group. If p_g is that sum and y_g is 1 for the group containing the truth and 0 otherwise:

The loss ranges from 0 to 1, and higher score is better: 1 is perfect, 0 is the worst forecast. Suppose A and B are indistinguishable and form the correct group, while C is distinguishable. Probabilities A = 0.4, B = 0.4, C = 0.2 give grouped probabilities 0.8 and 0.2, loss 0.04 and score 0.96. Moving probability between A and B does not change the score. The score is not a percentage of questions correct.

A question with requirement.kind: "supported_conclusion" also gates eligibility. Its conclusion must equal the accepted candidate group, and unresolved must be true exactly when that group has multiple candidates. This hard requirement is separate from the probability score: there is no confidence threshold. An unsupported conclusion can fail eligibility despite a high probability score. Current catalog tasks do use mandatory conclusions; consult each public question contract. See assessment requirements.

Numerical forecasts and counterfactuals

For prediction p, verified outcome a, and the question’s declared positive scale s:

Zero is the best score. An error of one scale unit gives score -1; an error of two scale units gives -4. These scores are not clipped to a 0–1 range. Use the question’s metric units for the prediction and scale, including cents for monetary quantities.

A factual forecast uses the independently verified metric on outcomeOnDay. A counterfactual forecast uses intervention minus baseline on that day, from two independently checked branches sharing the registered factual prefix and exogenous realization. A positive difference means the intervention increases the named metric; whether that is desirable depends on the metric. Increasing lost work, for example, is harmful. See counterfactual assessments.

Required counterfactual conclusions concern a disclosed contract component or supported effect set. They do not require an exact prediction of the uncertain total effect. Those conclusions gate eligibility independently of the numerical forecast. An optional forecast’s numerical accuracy does not alter primary business utility or Harbor reward. Assessment scores remain separate rather than being summed into an overall reasoning score.

The dashboard’s assessment Error column displays groupBrier or normalizedSquaredError, where lower is better. The stored score has the opposite direction. A displayed error of zero is perfect for either question type, even though the stored perfect scores are 1 for categorical questions and 0 for forecasts.

Harbor reward

Harbor uses the versioned transform worldseq-harbor-utility/1. For a successfully finalized, eligible run with raw primary outcome x:

The scale s is 1,000,000 cents for net_cash, net_assets and operating_margin. It is 100 for active_paying_accounts, retained_accounts and recovery, in their respective raw units. These are fixed transform constants, not task-specific baseline performance or forecast scales.

Eligible rewards lie between 0.5 and 1.5 and increase with the primary value. Every eligible outcome therefore outranks an ineligible one, including when the eligible business loses money. Rewards above 1 are expected; they are not percentages. For recovery tasks, the best raw outcome is zero, so the best possible reward is 1.

reward.json also contains eligible as 0 or 1, verified: 1, the raw primary_value, and reward_transform_version: 1. verified: 1 does not imply eligibility. result.json retains the detailed constraints, assessments and finalization. See Harbor integration for collection and the trusted verifier boundary.

Comparing agents or combining tasks

Ordinary eval output retains cases, completion/failure counts and resource usage; it does not calculate a universal suite score or automatically average Harbor rewards. Do not average cents, account counts and service shortfall. Even outcomes with the same units can differ in company size, horizon and difficulty. The Harbor transform preserves ordering within a task but does not calibrate difficulty across tasks. A cross-task ranking needs a declared task inventory and weighting policy.

Registered validation campaigns compare a chosen metric on paired cases:

• objectiveSuccess is 1 for a completed, verified eligible attempt and 0 for a completed, verified ineligible attempt. It measures satisfying the required predicates, not optimality or forecast calibration.
• Other business metrics contribute numbers only for completed, verified eligible attempts. Constraint failures remain in the inventory and make the comparison inconclusive, rather than selecting an eligible-only subset.
• Agent failures, infrastructure failures and missing attempts are nonnumeric for these comparisons, including objectiveSuccess. They are retained and prevent a complete numerical comparison; they are not silently omitted or assigned zero.

The effect is treatment minus baseline for a higher-is-better metric, and baseline minus treatment for a lower-is-better metric. Families receive equal weight; companies, tasks, scenarios and repeated attempts receive equal weight within each parent. This differs from a simple mean over all episodes when counts are unequal. Reports include 95% bootstrap intervals and wins, ties and losses under the predeclared practical tolerance. An interval spanning no effect does not establish equivalence. The comparison’s registered expectation and tolerance determine its pass/fail/inconclusive result. See validation campaigns.

Counterfactual failure handling

Required counterfactual grading distinguishes verified player-caused failures from unavailable evaluator evidence. Reserving a required one-shot action at the intervention checkpoint, consuming it earlier, or ending the factual episode before the checkpoint fails the execution precondition. Trusted finalization can complete with eligible: false even when no numerical counterfactual score is available. Cancelled reservations do not consume the option. Missing or corrupt branch evidence for a reached checkpoint remains inconclusive; it is not converted into a player failure merely because the branch could not be graded.

The catalog admit behavior probe uses the shared trusted finalizer for required counterfactual predicates before declaring a witness feasible. Verified branches alone do not establish eligibility: the submitted conclusion, execution preconditions, business constraints and gameplay budget must also pass. Protected instance certification retains its own bound proof artifacts. A feasibility certificate establishes an eligible path for one registered instance; it is not an agent score or evidence of generalization to unseen companies.

Implementation references

These source files define the current rules; the documentation does not introduce another scoring implementation:

Passing release evidence remains bound to its recorded implementation and inputs. These scores describe authored simulated outcomes; they do not establish real-world business performance or attest which model produced an externally submitted trace.