Serverless Overview ↗

Documentation Index#

Fetch the complete documentation index at: https://docs.fireworks.ai/llms.txt Use this file to discover all available pages before exploring further.

How Serverless inference works on Fireworks: serving paths, billing, request/response headers, prompt caching, model lifecycle, and when to choose Serverless over On-demand

What is Serverless#

Serverless is multi-tenant inference for popular open models running on Fireworks-managed infrastructure. You point your client at api.fireworks.ai, send tokens, and pay only for what you use — no GPUs to size, no autoscaler to tune, no cold starts to wait through. Models eligible for Serverless carry the Serverless tag in the model library. To make your first call, see the Serverless quickstart.

Serverless products at a glance#

Three serving paths run on the same Serverless framework. They share the same rate-limit policy, but route and price differently:

• Standard — the default serving path. No service_tier parameter needed.
• Priority — higher reliability during peak periods. Opt in by setting service_tier: "priority" on chat completions. Priced at a premium.
• Fast — high-speed deployments for latency-sensitive workloads. Selected by switching the model ID to a Fast variant (for example, accounts/fireworks/routers/kimi-k2p6-fast).

For usage examples and the full list of supported models, see Serverless Serving Paths. For pricing by serving path, see Serverless pricing.

Billing#

Serverless is priced per token. Three dimensions are billed:

• Input tokens — what you send to the model.
• Cached input tokens — input tokens served from prompt cache, discounted (default 50% of input on text and vision models, unless a model lists a different cached rate).
• Generated tokens — what the model produces.

Other things to know:

• The usage object in each response is the source of truth for what was billed (prompt_tokens, completion_tokens, total_tokens).
• Batch inference is billed at 50% of standard Serverless rates on both input and output. See Batch inference.
• Your spend tier influences Serverless capacity caps in addition to your monthly budget — higher spend tiers unlock higher TPM upper bounds. See Account quotas and Serverless rate limits.

Request and response headers#

Headers a Serverless caller will set or read.

Request headers#

Response headers#

Fireworks sets the following on Serverless inference responses:

Streaming responses don’t carry per-request perf headers. To get the same metrics in the streaming response body, set the perf_metrics_in_response parameter on the request. See Querying text models.

Prompt caching#

Prompt caching is on by default for every Serverless model. Cached input tokens are billed at the discounted rate (default 50% of input). Caching is replica-local, so to maximize hit rate you should route repeated prompts to the same replica — pass a stable identifier in x-session-affinity (or in the OpenAI user field) for each user or session whose prompts share a prefix.

For the full guide, including how to structure prompts for cache hits and how to read cache metrics, see Prompt caching.

Serverless model lifecycle#

Serverless models are managed by the Fireworks team and may be updated or deprecated as new models are released. We provide at least 2 weeks advance notice before removing any model, with longer notice periods for popular models based on usage.

For production workloads requiring long-term model stability, we recommend on-demand deployments, which give you full control over model versions and updates.

Serverless vs On-demand#

For dedicated infrastructure, see On-demand deployments.

Next steps#

Make your first Serverless API call.

Higher-reliability and higher-speed serving paths.

Per-token rates for text, vision, embeddings, and Priority.

Adaptive TPM bounds and how the limit ramps with usage.

How caching works and how to maximize hit rate.

Dedicated GPUs for predictable throughput and custom models.