Infiano came to us with a hard, fixed deadline. Their AI hackathon had produced 240 submissions from 57 teams, and every one of them had to be evaluated by a large language model within a single week. There would be no second window, no "we'll rerun it next sprint." The judging model had to be up, correct, and fast for six straight days, and then it could vanish.
The reflexive answer to "I need a big model for a week" is a cloud API. Point the eval harness at a hosted endpoint, pay per token, tear it down when you're done. For a lot of workloads that's the right call. For this one it was the wrong one on two counts that matter to anyone running high-volume inference: cost and control. The evaluation rubric was custom, the model was a custom 20B-parameter open-weight checkpoint tuned for Infiano's grading criteria, and the submission data was not something anyone wanted flowing through a third-party API's logs. So we did what LLMDeploy exists to do: we stood up self-hosted, open-weight inference on rented hardware, ran it hard for six days, and shut it off.
The headline: 40 GPUs, built in 48 hours, ran for 6 days for about $3,000 — versus roughly $6,750 for a GPT-4o-equivalent — with zero infrastructure-caused restarts and zero HTTP 5xx errors. Ephemeral self-hosting didn't just match a cloud API for a short burst job. It beat it.
The build: 40 rented GPUs in two days
We rented 10 bare-metal servers on vast.ai, each carrying 4× NVIDIA RTX 5090 (32GB) — 40 GPUs in total. In front of them sat a single high-memory orchestration server with 503GB of RAM to hold the queue, the router state, and the eval pipeline itself. That was the whole fleet: ten rented GPU boxes and one CPU brain.
Spot-priced, marketplace GPUs are the part that makes people nervous, and understandably. They are cheaper than the hyperscalers — meaningfully cheaper here — but you are renting other people's machines, and they can disappear. The engineering that follows is entirely about making that class of hardware behave like something you'd trust with a client's deadline.
The model was the custom 20B open-weight checkpoint, served identically across all 40 GPUs. One weight file, forty copies, no per-node special cases. Homogeneity is what lets you treat any GPU as interchangeable and reroute around a dead one without thinking.
Making rented GPUs reliable
This is the real work. Renting compute is easy; making rented compute dependable is the job.
Serving layer
Each GPU server ran four independent vLLM processes, one per card. Independent matters: a crash or a CUDA fault on one process takes down one GPU's worth of capacity, not the whole box. nginx load-balanced requests across the fleet, and a Caddy layer in front handled authentication so no raw vLLM port was ever exposed.
All of that surfaced as 31 API endpoints, and — this is the operational trick that kept the whole thing sane — they were managed from a single providers.json. Add a server, pull a server, shift weight away from a flaky card: one file, one change. When a machine vanishes at 3am you do not want to be editing config in five places under pressure.
Monitoring and failover
You cannot trust hardware you rent, so you watch it relentlessly. The stack ran on three layers:
- Watchdog v3, on both the GPU side and the CPU side, catching stuck processes and hung requests.
- A dedicated GPU health monitor tracking each card's temperature, memory, and error state.
- Zabbix collecting 15 metrics per GPU — 600 time series across the fleet — for the full six-day picture.
On top of monitoring sat automatic failover and recovery: an endpoint that stopped answering health checks was pulled from the nginx rotation automatically, and traffic redistributed across the survivors. We also hardened the eval pipeline itself while it ran, fixing skip-task handling so a single un-gradable submission couldn't wedge the queue, and adding HTTP 400 spam protection so a malformed request couldn't hammer a node.
Three incidents, zero downtime
The design earned its keep on live incidents. None of them caused an outage.
The server that vanished overnight. One spot instance, SRV9, simply disappeared — reclaimed by the marketplace, exactly the failure mode people fear about rented GPUs. Because every endpoint lived in one providers.json and nginx health-checked continuously, its four GPUs were dropped from rotation and traffic rerouted in under a minute. The eval queue never stalled.
The GPU that went bad. A single card on SRV7 threw a hardware error. Because each GPU ran its own vLLM process, we pulled just that one card out while the other three on the box — and the other 36 across the fleet — kept serving. One degraded GPU, not one degraded cluster.
The cache collision. Two submissions running in parallel collided on a shared cache key, a classic concurrency bug that only shows up under real load. We caught it in the monitoring, isolated it, and fixed the keying without pausing the run. This is the kind of thing you never see in a demo and always see in production.
The numbers
Over six days the cluster scored 236 submissions across 55 teams. Here is how self-hosted stacked up against the obvious cloud alternative:
| Metric | Self-hosted (40× RTX 5090) | GPT-4o-equivalent API |
|---|---|---|
| Total infrastructure cost | ~$3,000 | ~$6,750 |
| Submissions scored | 236 across 55 teams | — |
| Time-to-first-token | 1.1–1.4s across all 40 GPUs | — |
| Infrastructure-caused restarts | 0 | — |
| HTTP 5xx errors | 0 | — |
| Data control | On our infrastructure, our model | Third-party endpoint |
Roughly 55% of the cloud cost, a time-to-first-token that stayed in a tight 1.1–1.4s band across the entire fleet for six days, and — through a vanished server, a dead GPU, and a cache collision — not one infrastructure-caused restart and not one 5xx returned to the eval pipeline. And the model and the submissions never left infrastructure we controlled.
The lesson: self-hosting isn't only for permanent deployments
The usual framing says cloud APIs win on short, bursty jobs and self-hosting only pays off for steady, long-running load. Infiano is the counterexample. A six-day, 240-submission spike — about as bursty and temporary as a workload gets — came in at less than half the cloud cost, with lower latency variance and full custody of the model and the data.
What made it work wasn't the GPUs; anyone can rent GPUs. It was the operational engineering: one file describing the whole fleet, per-GPU process isolation, three layers of monitoring, and automatic failover that assumed rented hardware would fail and rerouted around it before a human noticed. That's the discipline that turns a marketplace of spot instances into infrastructure you can put a client deadline on.
This is exactly the work LLMDeploy does: standing up open-weight models on infrastructure you control — for a permanent production service or a one-week evaluation sprint — so you keep the cost advantage and the data control without giving up reliability. If you have a bursty, high-volume inference job and the cloud API quote is making you wince, that's precisely the case where self-hosting tends to win.