Are Open Models Cheaper Than Closed-Weight APIs?

Summary

The cost breakdown exists, but it doesn't compare open models to closed ones as categories. It compares a model customized for a specific job against a general-purpose closed-weight model doing that same job without any tuning. NVIDIA's Nemotron 3.5 Lightning work put numbers on this: a fine-tuned version of the model ran at roughly one-third the cost of Opus 4.8 at comparable accuracy (NVIDIA, 2026). The savings come from being able to retrain the weights for a narrow task, something a closed-weight API never lets you do.

Direct Answer

Cost savings from open models don't come only from choosing smaller models and avoiding the overhead of a closed-weight API. A base open model and a base closed-weight model of similar sizes doing the same general-purpose task often land in a similar cost range per token. The gap opens up once you fine-tune, distil, or quantize the open model for one specific job. You can do that because the weights are released and yours to modify. With a closed-weight API, you're paying general-purpose rates on every call, no matter how repetitive or narrow your actual workload is.

LangChain's deployment against the Nemotron 3.5 Lightning and NeMo Switchyard setup shows what this looks like. By routing most requests to a fine-tuned model and sending only the hardest 7% of traffic to a frontier model, the team cut overall cost by 74% (NVIDIA, 2026). The frontier model still handles what genuinely needs it. Everything else runs on a model tuned for the job, at a fraction of the price.

Thoughtworks measured a related but separate lever: throughput. Testing the fine-tuned model's native speculative decoding, which predicts multiple tokens per step instead of one, the team saw throughput improve by 1.46 to 1.96 times over the baseline (Thoughtworks, 2026). That gain pushed cost per million output tokens from $0.477 down to $0.250, nearly half. None of this required switching vendors or waiting for a price cut. It came from tuning and serving the same open weights more efficiently.

Takeaway

The cost advantage from open models goes beyond raw price per token. It comes from the effort to fine-tune those weights for a specific task and route traffic accordingly. NVIDIA's work on the Nemotron open model family gives teams a documented starting point for that process, from the tuning itself to the serving techniques that squeeze more throughput out of the result, so the savings are something you can reproduce rather than take on faith.

Sources: NVIDIA · Thoughtworks