Sortie: a control plane for rented GPUs that leaves nothing behind
A scheduler I was not watching stranded four volumes on a GPU cloud and billed $80 a month for storage I never used. The fix was not a reminder. It was a console where every GPU job is ephemeral by construction: pick a lane, launch with a deadline, work, download, and everything is deleted and verified.
Written October 2026. Sortie has been in daily use since 2026-08-03. The incident that caused it ran from 2026-07-06 to 2026-07-25. Numbers are single live runs from the commit log, not benchmarks.
The incident
In early July 2026 I was running fine-tuning and batch inference on Verda spot GPUs through a third-party scheduler. Spot boxes get evicted without warning. That is the deal, and the scheduler handled it. What it did not handle was the storage.
Two upstream bugs in the scheduler’s Verda backend. First, it created each instance with an OS volume but never set the on-eviction policy, so Verda applied its default and kept the volume detached after every eviction. Second, its terminate call passed the permanent-delete flag without the volume IDs, which the SDK treats as a no-op. The result: four 100 GB volumes stranded over three weeks, $20 a month each, and Verda’s trash still bills. An audit on July 25 found $80 a month of storage for nothing. A reaper script with a daily cron killed it the same day.
Reading the API that week produced a short list of facts that are not obvious from the console. A shutdown bills. Trash bills. Only a delete with the permanent flag stops the charge. A zero balance deletes your volumes. API keys expire silently. There is no native auto-stop on this provider. If I wanted an upper bound on what a forgotten job could cost, I had to build it.
The brief
On August 3 I wrote down what I actually wanted, and it was not a better script. I wanted a web console. Pick a provider and a preset, a GPU comes up, do the work, download the results, and the box is gone. Three lanes from the start: Kaggle because it is free, Modal because it is serverless, Verda because spot is cheap. Two accounts on some providers. Live balances. Notes in the UI on when to use which lane. And two hard rules. No standing storage cost, because $20 a month for an idle volume is exactly the thing I had just paid for. And a five to ten minute cold start is fine, because I would rather wait than pay for a warm disk.
Those two rules shaped everything below. Ephemeral-first means every launch renders its own startup script, pulls model weights inside the datacenter at datacenter speed, and keeps nothing between runs. It also means the application’s own deadline is load-bearing. The provider will not stop the box. Sortie has to.
Nothing is allowed to linger
Every job carries more than one way to end without me.
A deadline is mandatory at launch. There is no launch form without a TTL. The form prints the UTC deadline and the worst-case cost if the TTL runs full, before the button is live. When the deadline hits, the auto-end runs through the same verified delete as a manual kill and books a ledger row marked as fired by TTL.
Idle ends a training lab. Thirty minutes with no busy kernel, GPU utilisation under 5%, and no interactive shell. Idle is measured from the kernels API, never the status endpoint, because the server’s own polls would refresh it. A long training cell is busy and is never killed.
Unreachable ends the pod. A workspace that never answers within 45 minutes of launch is ended. A rented container with no public port and no eviction would otherwise bill until the TTL.
Storage is reaped. Detached volumes get 30 minutes of grace, then are deleted. Trash is emptied. The reaper reports the monthly cost it removed.
A delete is not done until a re-read proves it. Mutations run deleting, verifying, gone. Success means the second read came back empty. A delete that fails verification stays visible in red, raises an alert, and offers a one-click retry. The failure’s grace clock runs from the first record, so a retry loop cannot keep resetting it.
The lamp is reserved for leaks. A MASTER CAUTION lamp lights only when spend has no work behind it: a detached volume, a job past the reminder threshold, trash still billing, a balance under $15. Healthy work reads as billing, amber, not an alarm. The rules are pure functions over the provider snapshot, called by both the poller that pushes to my phone and the API that draws the lamp. They cannot disagree.
Blind outranks billing. Every expected account is declared in config. The calm state can only be shown when all of them answered within 90 seconds and nothing bills. A stale provider is a monitoring-blind state that outranks billing, because unseen spend is worse than seen spend. A credential that drops out of the environment turns health not-ok and nags on every live resource it can no longer watch.
A failed read never proves absence. Three days after the first release, an independent review found a transient read error on one account returned an empty instance list, the deadline enforcer read that as everything ended, and marked live jobs finished. One minute of API outage disarmed the TTL on a billing VM. The fix: an end derives the owning account from the job record, proves the instance is present in that wallet or journals a retry, and only then destroys. A resource is trivially absent from a wallet that never held it.
Three lanes, three billing models
The lanes exist because the three ways to pay for a GPU do not share a mental model, and the console is where the differences are made visible.
Free quota. Kaggle gives about 30 T4-hours a week. The number floats, so the console reads it live. The default accelerator is a P100, and current CUDA 12 PyTorch wheels do not ship sm_60 kernels, so CUDA reports available and the first op dies. The lane pins the T4 by exact enum. Two T4s are 16 GB each, not a 32 GB pool. Three prompts, about three minutes, at no cost.
Per-second serverless. Modal bills from container start to scale-to-zero. Every launch of a 6B image model hit out-of-memory on a 24 GB card until I found that the released diffusers silently ignored the dtype argument and loaded fp32. Sharding the 21.6 GB pipeline across two L4s then measured 14.1 s warm against 28 s, at $0.0075 per image against $0.0087. The larger rental was cheaper because the weight shuffle stopped. A later research pass showed the shard was pipeline-parallel and one L40S would have been three times the throughput at 40% of the cost, but that tier was gated behind a payment method on the account. On August 5 the lane moved to a three-model ladder, each on the GPU that holds it without sharding. Cold start 76 s, warm 28 s, about a cent an image.
Rented VM by the hour. Verda spot is a flat 50% off and the adapter deletes with the permanent flag on every call. Regular-price launches happen only from an explicit, price-labeled click. RunPod’s current API has no spot, so the tier is always explicit, and it bills from pod creation including the image pull. The first live run booked 406 seconds of uptime; the provider billed 690. So every rented pod is booked as an estimate from the job’s lifetime, then a reconciler replaces it with the provider’s own billing total. The history view shows a tilde until that arrives. The first reconciled row went from $0.02 to $0.0436.
Headline price is the wrong number
A marketplace board ranks Vast.ai and RunPod offers. Every Vast listing I sampled billed bandwidth. Once a 25 GB model pull is priced in, the ranking reorders completely. An RTX 5070 at $0.012 an hour really costs $1.36 for the job, 111 times the headline. Every row is scored on what the job will actually cost:
headline × hours + GB_in × $/GB_down + GB_out × $/GB_up + disk × $/GB-month × hours / 730If you want to set this up yourself
Sortie is a FastAPI backend and a Vite plus React frontend, run as one container behind a private reverse proxy. Everything below is a credential the backend reads from its environment at start. I render them from a secrets vault at deploy time; nothing lives in a file on disk. Each provider has one gotcha that cost me time, so each one gets its own note.
Verda. Create an API client in the console and set VERDA_CLIENT_ID and VERDA_CLIENT_SECRET per account. Sortie supports several accounts, and the list of accounts that must be present goes in SORTIE_EXPECTED_VERDA_ACCOUNTS. If a listed account has no credentials, health turns not-ok instead of the account silently vanishing from monitoring. Gotcha: the SDK authenticates eagerly, so a bad or expired key raises at construction. Keys do expire silently. Watch for the auth-failed alert, because an account you cannot read is an account you cannot reap.
RunPod. One API key per account, RUNPOD_API_KEY_<NAME>, with the expected names in SORTIE_EXPECTED_RUNPOD_ACCOUNTS. Use the v2 REST API only. Gotcha: v2 has no spot or interruptible field, so every launch is on-demand and the tier picker is always explicit. RunPod bills from pod creation including the image pull, so the ledger books from launch and reconciles against the provider’s billing endpoint afterward.
Modal. A token pair, MODAL_TOKEN_ID and MODAL_TOKEN_SECRET, plus MODAL_MONTHLY_CREDITS so the allowance tile has a denominator. Each model is its own deployed Modal app with weights baked at image build and scale-to-zero at 30 seconds idle. Gotcha: warm containers keep serving old code after a redeploy. Expose a version marker from the app and check it, or you will debug a bug you already fixed. Also read credits from the provider’s credits adjustment, not from gross usage, because free storage offsets do not consume the compute allowance.
Kaggle. Set KAGGLE_API_TOKEN to a bearer token from the account settings page or from the CLI’s login flow. Gotcha: the legacy username and key pair returns 401 on the quota endpoint while it still works for datasets. Only the modern bearer token works with the current CLI, and only it can read the weekly GPU meter. Pin the accelerator to the T4 by exact enum, because the P100 default fails on the first CUDA op with current PyTorch wheels.
Tailscale. Cloud workspaces join a private network as ephemeral, pre-authorized, force-tagged nodes, minted per launch from an OAuth client with a single tag scope. Gotcha: this is a hard dependency by design. If the key mint fails, the launch returns 503. There is no public-IP fallback for any lane that serves a UI, because a bearer header over plain HTTP to a public IP is not a door I want to exist.
Container images. Workspace images are prebuilt and digest-pinned in the provider’s own registry so a fresh VM pulls instead of reinstalling. Point SORTIE_RUNPOD_BASE_IMAGE and the preset templates at your own registry. Gotcha: a cross-cloud pull is about five minutes cold and it is billed, so bake, do not install.
The safety knobs. SORTIE_TTL_HOURS is the reminder threshold for anything still running, default 6. SORTIE_LOW_BALANCE_USD is the balance alert, default 15, and on Verda a zero balance deletes your volumes, so do not set it low. NTFY_URL is where the caution poller pushes. Leave it empty and the poller is a true no-op with no alert rows written, so the backlog fires cleanly once you wire it.
What it is not
A single-operator tool. I route every job by hand. There is no auto-failover between providers by design: a capacity error on one lane stays an error and I switch the lane myself. The numbers above are single live runs. What the project demonstrates is narrower than a platform and, I think, more useful: when the cheapest compute is spot capacity on several providers with incompatible billing rules, the bill is a system invariant, and the control plane has to enforce it the way it enforces anything else.