Streaming
Both the OpenAI (/v1/chat/completions) and Anthropic (/v1/messages) surfaces
support streaming over Server-Sent Events. The Responses API (/v1/responses)
streams its own response.* events; see Responses API. Streaming is the recommended default
for anything with a long prompt, long output, or a high token cap — it avoids
request timeouts and lets you render tokens as they arrive.
OpenAI-style stream
curl -N https://api.martllm.com/v1/chat/completions \
-H "Authorization: Bearer $UBERLLM_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "deepseek/deepseek-v4-flash",
"stream": true,
"messages": [{"role": "user", "content": "Count to five, one per line."}]
}'
The response is a sequence of data: lines, each a chat completion chunk, ending
with data: [DONE].
from openai import OpenAI
client = OpenAI(base_url="https://api.martllm.com/v1", api_key="ull_...")
with client.chat.completions.create(
model="deepseek/deepseek-v4-flash",
stream=True,
messages=[{"role": "user", "content": "Write a short poem."}],
) as stream:
for chunk in stream:
delta = chunk.choices[0].delta.content
if delta:
print(delta, end="", flush=True)
Usage accounting
MartLLM injects stream_options: {"include_usage": true} for upstreams that
support it, so the final chunk carries a usage object with prompt and
completion token counts, and cost_usd: what this request is billed, in US
dollars (what the provider charged for it). When the
upstream sends no usage chunk, MartLLM adds one before data: [DONE]. On
/v1/messages the cost is in the usage of the message_delta event.
Your request is settled at the cost the marketplace reports for that call; when
usage is missing, the gateway falls back to a tokenizer estimate and flags the
request as estimated in your API Usage. A stream's
headers are sent before the cost is known, so a stream has no
X-UberLLM-Cost-USD header: read usage.cost_usd.
Aborts
If you disconnect mid-stream, the gateway cancels the upstream request and settles for the tokens that were actually streamed. You are never charged for a full response you did not receive, and no reserved balance is left stuck.
Keep-alives
The gateway sends an immediate : ping comment on connect and periodic
keep-alives, so long-running streams survive proxy idle timeouts.