Infiano hackathon
Event inference billCloud vs on-prem
Cluster stood up in48 hours
Submissions received240
Time to first token1.1–1.4 s
Infrastructure-caused restarts0
Difference, event≈$3,750
A 40-GPU inference cluster in 48 hours, at under half the cloud price.
Infiano’s hackathon produced 240 submissions that a language model had to judge within a single week. We built self-hosted inference on rented GPUs, ran it for six days and shut it off.
≈$3,000against ≈$6,750 on a cloud API
Measured by us, not a customer testimonialThe run figures come from our own records of the event. The cloud price is our estimate for a GPT-4o-equivalent API. Infiano has not given a quote for this page.
Illustrative order and timing, not an event log: the cluster was built in 48 hours from 10 rented servers of 4 GPUs each.
A hard deadline and no second window.
The hackathon produced 240 submissions from 57 teams. Every one had to be evaluated by a language model inside a single week. The judging model had to be up, correct and fast for six straight days, and then it could vanish.
A hosted API was the obvious answer, and the wrong one on two counts: cost and control. The rubric was custom. The model was a custom 20B-parameter open-weight checkpoint tuned for Infiano’s grading criteria. And nobody wanted the submission data in a third party’s API logs.
Ten rented servers and one orchestration host.
We rented 10 bare-metal servers on vast.ai, each with 4 NVIDIA RTX 5090 cards of 32 GB, 40 GPUs in total. One high-memory server with 503 GB of RAM held the queue, the router state and the evaluation pipeline.
The same 20B checkpoint ran identically on all 40 GPUs. With one weight file and no per-node special cases, any GPU could be treated as interchangeable and rerouted around when it died.
Serving
Four independent vLLM processes per server, one per card, so a fault on one process costs one GPU of capacity. nginx balanced load across the fleet and a Caddy layer handled authentication, so no raw vLLM port was exposed.
One file for the fleet
All 31 API endpoints were managed from a single providers.json. Adding a server, pulling one or shifting weight away from a flaky card was one change in one file.
Monitoring
Watchdog v3 on both the GPU and CPU side, a dedicated GPU health monitor, and Zabbix collecting 15 metrics per GPU, 600 time series across the fleet.
Failover
An endpoint that stopped answering health checks was pulled from the nginx rotation automatically. The pipeline was also hardened while live, with skip-task handling and protection against HTTP 400 floods.
Three incidents, no downtime.
- Server
SRV9 vanished overnight
The marketplace reclaimed the spot instance. Its four GPUs left the rotation and traffic rerouted in under a minute. The queue never stalled.
- GPU
A card on SRV7 threw a hardware error
Each GPU ran its own vLLM process, so we pulled that one card. The other three on the box and the other 36 across the fleet kept serving.
- Software
A cache-key collision
Two parallel submissions collided on a shared cache key, a concurrency bug that only shows under real load. We found it in monitoring and fixed the keying without pausing the run.
What the run showed.
≈$3,000Total infrastructure cost, against about $6,750 for a GPT-4o-equivalent API.
1.1–1.4 sTime to first token, held across all 40 GPUs for six days.
0Infrastructure-caused restarts. Zero HTTP 5xx errors as well.
236Submissions scored across 55 teams, from the 240 received.
| Metric | Self-hosted, 40× RTX 5090 | GPT-4o-equivalent API |
|---|---|---|
| Infrastructure cost | ≈$3,000 | ≈$6,750 |
| Time to first token | 1.1–1.4 s across 40 GPUs | Not measured |
| Infrastructure-caused restarts | 0 | Not measured |
| Data control | Our infrastructure, our model | Third-party endpoint |
Self-hosting is not only for permanent loads.
The usual framing says cloud APIs win short bursty jobs and self-hosting pays off only on steady load. A six-day spike of 240 submissions came in at less than half the cloud cost, with a tight latency band and full custody of the model and the data.
The GPUs were not the point, since anyone can rent them. The work was one file describing the fleet, per-GPU process isolation, three layers of monitoring and failover that assumed rented hardware would fail.
Want the same on your data?
The pilot puts one open-weight model live on your own data in 72 hours, at a fixed $4,900. See every step priced before you decide.