API vs self-hosted GPUs: the break-even math

Every growing AI product eventually asks whether it would be cheaper to stop paying per token and run a model on its own GPUs. The answer depends less on GPU prices than on two numbers most teams have never written down: how many requests they serve, and how busy they could keep the hardware. This guide sets out the break-even math step by step, with GPU and API prices from our dataset, and then lists the costs the math leaves out.

Prices come from each provider's published price list and the throughput figure from a published benchmark (sources on our methodology page). Request sizes and traffic levels are illustrative.

Two different cost shapes

An API bill is variable. You pay for each token, the bill scales with traffic and falls to zero when traffic stops. A GPU bill is fixed. You pay for every hour a GPU is rented, whether it is generating tokens or sitting idle.

API cost per month = requests per day × cost per request × 365 ÷ 12 GPU cost per month = GPUs × price per GPU-hour × hours rented per month

Self-hosting wins when the fixed GPU bill, spread over all the requests the GPUs serve, comes out below the API price per request. That makes the break-even a question of volume and of idle time.

Step 1: Work out what the GPUs can produce

Throughput is the number that is hardest to pin down, because it depends on the model, its precision, the serving software, batch sizes and how long prompts and answers are. The only defensible starting point is a published, reproducible benchmark, followed by your own measurement.

Our dataset has one verified benchmark figure: 11,436 tokens per second per GPU for the NVIDIA B200 180GB. MLPerf Inference v6.1, Offline scenario, Closed Division, gpt-oss 120B: 91,487 tokens/sec on CoreWeave B200 SXM (8x B200-SXM-180GB). Divided by 8 GPUs. A maximum-throughput benchmark result for one model, not a guarantee for other models or traffic patterns. We treat these as generated tokens. At that rate, 8 GPUs running around the clock produce about 240,427,836,000 tokens a month.

Treat that as a ceiling. An offline benchmark measures maximum throughput with no limit on how long each request waits. A live service with latency targets runs smaller batches and produces less. Long prompts also take GPU time to process before the first output token appears. Before you decide anything, benchmark the model you would actually run, with your own prompt lengths, on the hardware you would actually rent.

Step 2: Price the hardware

Of the 4 providers in our data with an on-demand price for the NVIDIA B200 180GB, the cheapest for 8 GPUs is Runpod Community Cloud, at $47.84 an hour for all 8. Running them 24 hours a day costs $34,923 a month. That figure does not change if traffic falls overnight, at weekends or after a launch spike fades.

Step 3: Cost per token depends on how busy the GPUs are

Divide the monthly GPU bill by the tokens actually produced, and you have a cost per token you can compare with an API. The key variable is the busy share: the fraction of rented hours the GPUs spend generating at full throughput. Using a request of 1,000 input and 300 output tokens:

Every time the busy share halves, the cost per token doubles. This is the core of the decision. GPUs that are busy all the time look very cheap per token; the same GPUs sitting mostly idle can cost more per token than a premium API. Real services rarely run near 100%, because traffic follows the day, and you need headroom for peaks.

Step 4: Find the break-even volume

The cleanest way to compare is to ask how many requests a day it takes for the API bill to match the GPU bill:

break-even requests per day = GPU cost per month ÷ API cost per request ÷ (365 ÷ 12)

Then compare that with what the GPUs can serve. At full benchmark throughput, with every request generating 300 tokens, 8 GPUs could handle about 26,348,256 requests a day. The busy share needed is the break-even volume divided by that capacity.

Against the cheapest API in our data, Claude Haiku 5.5, the GPUs break even only at about 4,592,640 requests a day, every day. For comparison, the product in our pre-launch estimating guide, at 20,000 requests a day, would pay $152.08 a month for Claude Haiku 5.5 in this request shape, against $34,923 for the GPUs.

The table also seems to say that self-hosting beats the most expensive APIs, such as GPT-6 Astra, at a small fraction of capacity. Read that row with care. The benchmark measures one open-weight model, and the expensive APIs are priced for capabilities that model may not match. The comparison only holds if a model you can run yourself meets your quality bar on your own evaluation set. If it does not, the right comparison is with an API model of similar quality, which is usually one of the cheaper rows.

Step 5: Decide whether to commit

Reserved capacity lowers the hourly rate in exchange for paying for every hour of the term. Take Microsoft Azure's Standard_ND96isr_H100_v5 (NVIDIA H100 80GB (SXM)). On demand, it costs $98.32 an hour. The published 1-year reservation works out to $62.92 an hour, spread over every hour of the year: $45,935 a month, against $71,774 for running on demand around the clock.

The reservation pays off only if you would otherwise run on demand for more than about 15.4 hours a day, on average, for the whole year. Below that, on-demand hours you can switch off are cheaper. Reserve the base load you are confident about, and keep peaks on demand or on an API.

Costs the formula leaves out

The GPU line is the visible part of self-hosting. Before you compare it with an API invoice, add:

• Engineering time. Someone has to choose and tune the serving stack, deploy model updates, patch drivers and handle incidents. For a small team, this often costs more than the GPUs.
• Redundancy. One node is a single point of failure. A production service usually needs at least two, which halves the busy share of each.
• Peak capacity. You must provision for your busiest hour, not your average one, so the average busy share is always lower than the peak.
• Storage, networking and data transfer, which GPU hourly prices do not include.
• Evaluation and model upgrades. When a better open model appears, you rerun your evaluations and redeploy. With an API, the provider does most of that.

The API side has hidden costs too: rate limits that cap growth, price changes outside your control, and data handling terms that may not suit every customer. Weigh these honestly, but they rarely change the arithmetic as much as idle GPU hours do.

When each option wins

APIs usually win before product-market fit, for spiky or modest traffic, when you need the most capable models, and when the team is small. You pay only for what you use and can switch models in an afternoon.

Self-hosting starts to win when traffic is high and steady, when an open-weight model passes your evaluations, and when you have the people to run it. Data control and residency requirements can also tip the decision, independently of cost.

Many teams end up hybrid: steady, high-volume, well-understood work on their own GPUs, and everything else, including peaks and the hardest tasks, on an API. The break-even formula tells you where to draw that line, one workload at a time.