How MartLLM (formerly UberLLM) routing picks the cheapest healthy provider
For each request, MartLLM looks up every seller offering the model you named, ranks them by live price, and sends the request to the cheapest healthy one that is fast enough for your prompt size. If that seller fails before any output has streamed, the request is retried on the next cheapest healthy seller — so you get the lowest available price without losing reliability, and you are never billed for a failed attempt.
The one-sentence version
Cheapest healthy seller, decided per request. Both words carry weight: cheapest, so you pay the lowest live price; healthy, so you do not get routed to a seller that is erroring or slow.
Step by step
- Resolve the model. The
modelid you send maps to one logical model (old ids keep resolving to their current replacement, so links and code do not rot). - Gather the sellers. MartLLM collects every active offer for that model — the gateway’s own curated route plus any seller-pool offers listed on the marketplace.
- Rank by live price. Offers are ordered by their current price. The price you are quoted has no MartLLM fee added, so ranking and billing use the same number.
- Filter to healthy. Sellers that are failing, rate-limiting, or probing unhealthy are skipped. Health is tracked continuously from real traffic and background probes.
- Send to the cheapest one that is fast enough. Of the healthy sellers, those within 2x of the fastest measured time to first token at your prompt size are kept, and the cheapest of them gets the request. Long prompts also go to a second fast seller if the first is slow to start. The details are in smart routing.
- Fail over before output. If that seller errors before any tokens have streamed to you, the request is retried on the next cheapest healthy seller. A request that ultimately returns no result costs $0.
Why "healthy" is not optional
A pure cheapest-wins router sends you to whichever seller underprices everyone — including a seller that is cheap because it is broken. MartLLM’s health filter is what makes "cheapest" safe: the live board surfaces each provider’s reliability, and you can inspect it yourself on the uptime page. Routing uses the same signal.
What you can see
- The live price per model, no fee added — pricing board.
- Which models get the most traffic — rankings.
- Provider reliability over time — uptime.
For sellers
The same mechanism is the seller side of the marketplace: list spare capacity for a model, stay healthy and competitively priced, and requests route to you. See how selling works and the sellers doc.