Budget, cost and stuck runs

This page covers a run that stopped, a run that will not stop, and a run whose bill is a surprise: the hard ceiling that fails a run outright, the soft gate that cancels one, the stages whose failure is survivable, and the difference between a run that is waiting and a run that is genuinely wedged. For the cost of routing decisions themselves, read Model routing.

What these failures have in common

A run has one budget, one status and one optionality rule, and almost every symptom here is one of those three. The budget is a limit and a running spend, with two separate controls over it: a hard ceiling checked before every stage, and a soft threshold that asks a human once. The status is the run's own state, and it distinguishes a run that was cancelled from one that failed from one that is waiting on a person. And optionality is per stage, so the same failure that halts one pipeline is recorded and stepped over in another.

Two things to do before anything else on this page. Read the run's own record — its status, its error, its budget limit and spend, and the stage it stopped in — because the three ways a run can end leave different marks there. And read pendingApprovals and activeRuns at the health endpoint, which together separate "waiting for you" from "still working" from "finished and you missed it".

One framing fact is worth stating up front, because it explains several symptoms below. There is no stage-level degraded status. A stage is pending, running, awaiting an approval, done, failed or skipped, and nothing else. What the office does instead of degrading is four other things: relax the router's capability requirements, record a turn-level partial failure, record a tool-level status, or step over an optional stage. If you are looking for a status that means "worked, but worse", it does not exist, and the honest signal is in a reason string or an error field. See How a run flows.

Run budget of $X was exhausted before stage "Y"

What it means

The ceiling is checked before each stage with a greater-or-equal comparison, and crossing it fails the whole run rather than the stage. The stage named in the message is the one the engine refused to start, so the work already done is complete and the work after it never began. This is the strict path, and it is strict on purpose: it is the only thing standing between a runaway pipeline and an invoice.

Where the limit comes from is worth knowing precisely, because the obvious answer is the wrong one. A run's limit is the amount given with the submission if one was given, and otherwise the floor's own default per-run amount, which ships at five dollars. It is not the environment variable that looks like it should set it; that one is documented below.

What to check

1. Read the stage name in the message. It is the first stage that did not run, and the stages before it are complete with their own summaries and artifacts.
2. Compare the run's spend with its limit. Both are on the run record, and the spend usually crossed the limit inside the previous stage rather than at its start.
3. Check which limit applies. A per-submission amount overrides the floor's default; the floor's default is what governs otherwise.
4. Check what made the previous stage expensive. A wide parallel stage, a review loop with several iterations, or an escalation to a higher tier are the usual reasons one stage costs more than expected.
5. Decide between raising the limit and narrowing the work. Raising the floor's default is one field; submitting a smaller brief, or a pipeline with fewer stages, is often the better answer.

DEV3D_RUN_BUDGET_USD is set and nothing changes

What it means

The variable is read into the configuration object and then referenced nowhere else in the source. Its only two occurrences are the field's declaration and the line that reads the environment, so setting it changes a number that nothing consults. The documentation says this plainly rather than removing the variable, because that is its present state, and the project's own example file describes it as a hard ceiling that the engine enforces — which the code does not do.

The real per-run limit is the floor's budget: an explicit amount on the submission if one was given, otherwise the floor's default per-run amount. See Environment for the variable's entry and Known gaps for the audit note.

What to check

1. Stop looking at the environment for the run ceiling. Change the floor's budget instead, or pass an amount with the submission.
2. Check the soft-spend threshold, which is a different control. That one is an installation setting and defaults to a dollar fifty.
3. If you want an installation-wide default, set the floor's default per-run amount, which is what every run inherits when nothing overrides it.

Spend $X of the $Y budget and continue?

What it means

This is the soft gate, and it is easy to mistake for a stall because it does not change the run's status. It fires at most once per run, only between stages, and only once the spend has already crossed the configured threshold. The approval has no turn attached to it — it belongs to the run rather than to an employee — and it is attributed to the owner of the first stage so that somebody is named in the console.

The two answers have different consequences and only one of them is gentle. Approving lets the run continue; refusing does not pause it, it cancels it, with the run's error reading The operator declined to continue spending on this run. Setting the threshold to zero disables the gate entirely.

What to check

1. Confirm it is a spend approval and not a tool approval. The spend approval does not set the run to awaiting-approval, so a run that looks running with a pending approval is this case.
2. Read the detail. It names the amount spent, what remains and which stage is next, which is everything needed to answer.
3. Check the run's spend against the limit before answering. Approving continues towards the ceiling; if the remaining amount is smaller than a typical stage, raising the limit is the more honest change.
4. Check whether it has already fired on this run. It fires once; a second crossing is not asked about again, and the ceiling is what stops the run after that.
5. Check the threshold is not zero. Zero is the documented way to disable the gate, and a gate that never asks is either disabled or never crossed.

A run finished well past its budget

What it means

The ceiling is checked before stages, not inside them. A stage is handed an abort reason it can consult between turns, and only two of the four stage modes ever consult it: the debate mode and the review loop. The single and parallel modes run their turns to completion, so a stage that has already spent past the ceiling will keep taking turns and can finish well beyond the limit.

The overshoot is bounded by one stage rather than by one turn, and a parallel stage is the widest case because several employees run at the configured concurrency. The abort reason's own text is distinct from the halt message you would see if the ceiling had been checked at a stage boundary.

What to check

1. Identify the stage that crossed the ceiling. Its mode tells you whether the mode was capable of consulting the abort reason at all.
2. Check how many turns it took and how many ran at once. A parallel stage with a high concurrency is the widest possible overshoot.
3. Lower the concurrency if the overshoot matters. It is an installation setting, clamped between one and a wide maximum, and it only affects parallel work.
4. Prefer debate and review-loop stages when the budget is tight, because those are the modes that stop between turns.
5. Treat the overshoot as a known limit rather than a bug. It is documented as one, and it is the reason a run's limit is described as a boundary between stages rather than a hard cap on a run.

The stage produced no turns

What it means

A stage's outcome is two separate facts, and both are recorded. A stage failed if it carries an error; a stage is empty if it produced no turns. The empty case is reported with the text above, which is why a stage can fail with a message that names no underlying problem — there was no problem, there was nobody to do the work.

Participants are resolved from the stage's role list against the floor's org chart. A stage whose roles cannot be resolved logs that message and produces nothing; a stage naming a role that does not exist logs a warning and steps over that one role. The stage executor itself never throws, so a failure here is always a recorded result rather than an exception.

What to check

1. Check the floor can staff the pipeline. A pipeline whose stages name roles the floor does not have will fail at the first stage that needs one.
2. Check whether a role was removed or renamed. The org chart is editable, and a pipeline referring to a role that was fired fails quietly at stage time.
3. Check the stage's role list, not the pipeline's description. The participant list on the stage record is what the engine actually resolved.
4. Check whether the stage was skipped rather than empty. A tag-gated stage and a cancelled run both produce a skipped stage, which is a different status with a different meaning.
5. Check the run's continuation. Whether the run stopped tells you whether the stage was optional; see the next symptom.

A stage failed and the run carried on

What it means

Only a non-optional stage halts the run. An optional stage's failure is recorded as failed and the run continues, which is the whole point of marking a stage optional: the research stage in the shipped product pipeline is the one stage in the three shipped pipelines that carries the flag, because external evidence is genuinely useful and genuinely not always available. Exceeding the budget always halts, whatever the flag says.

The consequence worth knowing is that a run can end as done with a failed stage inside it, and the final summary is the last stage's summary rather than an aggregation. A reader who only looks at the run's status will not learn that a stage failed.

What to check

1. Read every stage's status and error, not just the run's. A done run can contain a failed stage, and the run-level error will be empty.
2. Check the stage's optional flag. It is a property of the stage in the pipeline, so it is visible before you run anything.
3. Check whether the failure was expected. The optional research stage failing usually means the office could not reach the web, which is a network question rather than an orchestration one.
4. Check the stage summary, not only its error. A review stage's summary is the chair's verdict, and while the reviewers keep raising objections the run's file producers revise their work; an objection that was never resolved therefore reads as text in the summary rather than as a failed stage.

A run looks stuck

What it means

Three different situations look identical from the console, and separating them is most of the work. A run can be waiting on a person, which is an approval nobody has answered. It can be waiting on a provider, which is a model call that has not returned. Or it can be genuinely busy on a long stage with several employees and a review loop.

The uncomfortable fact is the second one: model calls carry the run's abort signal and no timeout of their own, so a vendor that accepts a connection and never answers will hold the turn open indefinitely. Nothing in the office will time it out. Cancelling the run is what aborts the call, and that is the only thing that does.

What to check

1. Ask the health endpoint. activeRuns and pendingApprovals separate "still alive" from "waiting for you" from "actually finished".
2. Look for a pending approval and answer it. An approval waiting on a human is the most common cause and the easiest to fix.
3. Look for a blocked employee. The employee status for a pending approval is distinct from working and thinking, and its activity line names the approval.
4. Check the last event on the wire. A stream of text deltas means the model is answering; nothing at all for minutes on a stage that should be busy points at the provider.
5. Check the provider directly. A request to the provider with the same key tells you whether the vendor is answering at all.
6. Cancel the run if you have waited long enough. Cancellation is a socket command, not an HTTP route, and it aborts the in-flight turn.

Cancel did nothing to the stage

What it means

Cancellation is cooperative in the same way the budget is: the run stops advancing, the stages that were never reached are marked skipped, and the in-flight turn is aborted through the signal it was given. A turn that was mid-call ends with its own cancellation error, which is a turn-level fact rather than a run-level one. A turn that had already finished keeps its result, because it happened.

What cancellation does not do is rewrite history: completed stages stay done, their artifacts stay, and their spend stays spent. The run ends in the cancelled state with whatever it managed to produce.

What to check

1. Confirm the command carried the right run id. Cancellation names a run; a wrong or stale id cancels nothing and reports nothing.
2. Read each stage's status. Skipped stages were never reached; done stages are real work you still have.
3. Check the in-flight turn's error. A turn aborted by cancellation says so, which distinguishes it from a failure.
4. Check the run's status is cancelled and not failed. The two are different outcomes with different error text, and only one of them means the budget was the cause.

A turn reports done and the work is incomplete

What it means

Both limits produce an error string on the turn without changing its status, and neither emits an error event, because only a turn that genuinely failed does that. The employee returns to idle. So the run's status, the stage's status, the employee's status and the absence of an error event all agree that everything is fine, while the one field that carries the message says otherwise.

The retry path that would help does not exist. The turn prompt accepts a correction note and renders it as a block telling the employee that their previous attempt failed and what went wrong, but no caller ever passes one, so a partial turn is not retried with an explanation. See Known gaps.

What to check

1. Read each turn's error field whenever you automate on run status. This is the single most useful habit for anyone driving the office from a script.
2. Check the role's per-turn output cap. Policies carry one, and it is not uniform across the shipped roles.
3. Check the tool round-trip count. A turn that stopped at the limit was doing work that may belong in a longer pipeline rather than in one turn.
4. Check the work product rather than the status. The turn's own text is what is incomplete; the files it did write are on disk and are real.

The paused status never appears

What it means

The run status union declares a paused value and no code path in the server ever sets it. The one control that sounds like it should — the soft-spend gate — cancels the run instead, which is a deliberate choice: a run holding a half-finished pipeline open across an operator's lunch break would have to keep its workspace, its approvals and its spend consistent, and cancelling is the honest simplification. The stage status for waiting on a person is real and is set by tool approvals, which is a different thing from pausing a run.

What to check

1. Check what you actually want. Stopping before more money is spent is the soft gate, and its answer is to continue or to stop for good.
2. Check whether the run is waiting on a tool approval. That state is real, it is visible on the run and the employee, and it resumes when answered.
3. Treat a cancelled run as finished. Its completed stages remain; there is no resume.

The cost does not match the vendor bill

What it means

Costs are computed from real token counts against the prices the catalog holds, and those prices are routing estimates rather than billing truth. They exist so that routing and reporting can be ordered plausibly, and the settings screen is where an installation corrects them. A model that was discovered rather than curated starts unrated, which affects both the estimate and how much the router lets price influence the decision.

Two smaller effects are worth knowing. The per-turn output cap shapes how much a turn can cost, because it bounds one half of the token count. And a turn that was served by a fallback is billed for what actually ran, which is why the record keeps the served model separate from the routed one.

What to check

1. Check whether the model is rated at all. An unrated model's figures are the population priors rather than anything about that model.
2. Correct the prices in the settings screen if you care about the reported number: rates are editable per model.
3. Check servedBy against route on expensive turns. A fallback that answered is the model whose price was charged.
4. Check the output cap for the roles involved. It is the per-turn half of the cost equation and it is set per policy.
5. Check the floor's spend attribution. Spend accumulates per run and per employee, so a run that looks expensive may be several employees on one stage.

When it is not in this list

Budget and run problems are answered by the run record and the event log, so include both:

• The run's status, error, budget limit and spend, and the stage it stopped in.
• Every stage's status, error and optional flag, because a run's outcome is not the same as its stages' outcomes.
• For a stuck run, the health response with activeRuns and pendingApprovals, plus the last event you saw with its timestamp.
• For a cost question, the per-turn usage with the routed model and the model that actually answered.
• Which per-run limit applies: an amount passed with the submission, or the floor's default.

Send it through contact. The mechanism behind the limits is in How a run flows, the knobs and their real defaults are in Environment, and the list of controls that look live and are not is in Known gaps.