12 Techniques to Cut
AI Inference Costs

The single highest-impact structural decision in any AI system is which model handles which query. Frontier models like GPT-4o and Claude Opus are priced at $5–15 per million tokens — 10–50× more expensive than capable mid-tier models like GPT-4o mini or Claude Haiku at $0.15–1 per million tokens. Routing every query to the frontier is the most expensive default in AI engineering.

Intelligent routing inserts a task classifier before the model selection step. Simple queries — intent classification, summarisation, extraction — route to a cheap fast model. Only queries requiring deep reasoning, multi-step planning, or domain synthesis route to the frontier model. Stanford’s FrugalGPT research demonstrated up to 98% cost reduction using LLM cascade routing with no quality degradation, making this the highest-ceiling single technique in the stack.

Implementation options range from simple (a hardcoded rule: queries under 200 tokens route to the small model) to sophisticated (a trained routing classifier that scores queries on complexity, domain specificity, and required reasoning depth). Tools like PortKey AI, LiteLLM, and OpenRouter provide model routing infrastructure out of the box.

Prompt caching is the quickest, highest-return optimisation for the majority of production workloads. Both Anthropic and OpenAI provide automatic prefix caching: when a prompt shares a long prefix with a previously processed prompt — a system prompt, a set of instructions, a document — the provider reuses the cached KV computation instead of reprocessing the prefix from scratch. Cache hits are priced at 90% discount (Anthropic) or treated as free (OpenAI).

Implementation for managed APIs requires only proper prompt structure: place the stable, shared content (system prompt, document context, few-shot examples) at the top of the prompt where caching operates, and the dynamic, query-specific content at the bottom. For self-hosted models, prefix caching in vLLM achieves 30–50% compute reduction at high request volumes with shared prefixes, and KV-cache-aware routing (as in the llm-d project) directs requests to the node holding the relevant cache to avoid redundant computation.

Google’s TurboQuant (March 2026) compresses the KV cache itself to 3 bits per value with zero measured accuracy loss, achieving 6× memory reduction — enabling dramatically more aggressive caching on the same hardware.

LLMs default to generating as many tokens as they deem necessary. Without explicit constraints, models over-generate on nearly every task — adding preambles (“Great question!”), unnecessary caveats, redundant explanations, and verbose summaries when concise answers are needed. Output tokens are typically priced 3–5× higher than input tokens, so unconstrained generation directly inflates the per-request cost.

The max_tokens parameter is the first lever. Set it as low as the task can tolerate: a classification task needs 5–10 tokens, not 500. A structured JSON output needs a schema-defined ceiling. A summary needs a word count limit enforced in the system prompt (“respond in under 150 words”). Add explicit prompt instructions: “Be concise. Answer directly without preamble.”

For structured outputs, using JSON schema or function calling constrains the output to exactly the fields requested, preventing the model from appending explanatory text after the structured response. Monitor output token distributions per endpoint — a p95 that is 5× the p50 is a reliable signal of inconsistent generation that a tighter max_tokens policy and clearer prompt instructions can resolve.

Both Anthropic’s Batch API and OpenAI’s Batch API offer a guaranteed 50% discount for asynchronous batch processing — the easiest and most guaranteed cost reduction in the managed API space. Batching aggregates multiple requests and processes them together during off-peak periods, trading latency (24-hour completion window) for a flat cost halving. No quality change, no model change — just a different API endpoint.

On self-hosted infrastructure, continuous batching in vLLM and TensorRT-LLM dramatically improves GPU utilisation by dynamically grouping requests rather than processing them one-at-a-time. High-QPS background jobs achieve 3–5× cost reduction versus one-at-a-time; interactive chat with continuous batching achieves 1.5–2.5× at acceptable latency; strict real-time sub-100ms requests see a 1.2–1.5× ceiling before latency breaks.

Identify which workloads in your system are genuinely latency-tolerant: nightly report generation, document classification pipelines, embedding generation for vector indexing, offline fine-tuning data preparation, and batch evaluation runs. These can all move to batch processing with zero user-facing impact.

Synchronous inference requires the server to maintain a connection and hold capacity for the full duration of generation — driving peak capacity costs even during low-demand periods. Async inference decouples request submission from result retrieval: a task is submitted to a queue, a worker picks it up when capacity is available, and the result is delivered via webhook or polling. This dramatically smooths demand peaks and eliminates the over-provisioning needed to handle burst traffic synchronously.

The practical impact: fewer timeouts, fewer retries (each retry is a full duplicate cost), and the ability to right-size infrastructure for average load rather than peak load. Retry storms — where a wave of timeouts triggers a cascade of retries that amplifies load further — are one of the most common sources of unexpected cost spikes in production LLM systems. Async queuing eliminates this failure mode entirely for tolerant workloads.

Implement with a message queue (Redis, SQS, RabbitMQ) between your API layer and the inference workers. Include idempotency keys to prevent duplicate processing of retried submissions, and set appropriate queue depth limits to prevent the queue from growing unboundedly during outages.

Quantization converts model weights from high-precision formats (FP32, FP16) to lower-precision formats (INT8, INT4, FP8), shrinking model size, reducing VRAM requirements, and enabling the same model to run on fewer — or smaller — GPUs. A 70B parameter model in FP16 requires ~140 GB VRAM. In FP8 it needs ~70 GB, fitting on a single H100 (80 GB) instead of requiring two GPUs. In INT4 it drops to ~35 GB. On GMI Cloud, serving a 70B model on 1× H100 at ~$2.10/GPU-hour versus 2× H100 at ~$4.20/GPU-hour cuts your GPU bill in half.

Quantizing from FP16 to INT8 or INT4 reduces memory by 2–4× and cuts inference cost by roughly 50% while maintaining 95–99% of original accuracy. FP8 quantization on H100/H200 hardware with TensorRT-LLM delivers 1.5–2× throughput improvement over FP16 with minimal quality degradation for most LLM tasks. The golden rule: always test output quality at your target precision before deploying to production, particularly for tasks requiring precise arithmetic or low-frequency vocabulary.

For managed API users, quantization is handled by your provider — but model selection effectively encodes a quantization decision. Providers like Together AI and Fireworks serve quantized variants of open models at dramatically lower prices. The optimal compression pipeline for self-hosted models is P-KD-Q: prune first, distill second, quantize last — each step compounds the savings.

Few-shot prompting with 10–20 examples is a common technique for steering model behaviour without fine-tuning. But each example adds hundreds to thousands of tokens per request — and at scale, those tokens compound. Fine-tuning bakes those examples into model weights, allowing you to serve the same behaviour with a much shorter prompt (often under 100 tokens), or to use a significantly smaller model that matches the larger model’s quality on your specific task.

The break-even analysis is straightforward: compare (fine-tuning cost + smaller model inference cost) against (larger model inference cost × request volume). For high-volume workloads — tens of thousands of daily requests — fine-tuning typically breaks even within 2–4 weeks and delivers 60–80% ongoing cost reduction. For low-volume workloads, prompting remains more economical.

The 2026 standard approach is PEFT (Parameter-Efficient Fine-Tuning) via LoRA or QLoRA adapters — fine-tuning only a small fraction of model parameters at a fraction of the compute cost of full fine-tuning. A LoRA-adapted Llama 3 8B can match GPT-4-level accuracy on a narrow, well-defined enterprise task, served at 1/50th of the API cost.

Chat applications that naively pass the full conversation history on every turn are one of the most common sources of runaway token spend. A 20-turn conversation with 500 tokens per turn accumulates 10,000 input tokens — most of which are not relevant to answering the current question. Auditing and compressing context is a direct, high-impact input token reduction strategy.

The practical toolkit: sliding window (keep only the last N turns); summarisation (compress older turns into a rolling summary using a cheap model); selective retrieval (retrieve only the conversation turns semantically relevant to the current query); and compaction (Anthropic’s built-in context compaction, introduced in 2025, which automatically summarises the oldest portion of the context when approaching the window limit). Combine summarisation with retrieval for maximum compression without quality loss.

For RAG pipelines, context auditing means sending only the top-K most relevant retrieved chunks — not the full retrieved set — and keeping chunk sizes tight enough to be useful without padding the context with surrounding irrelevant text. A retriever that returns 20 chunks when 3 would suffice is adding 17 chunks of unnecessary input tokens to every request.

Streaming doesn’t reduce the token count for a given request — it changes when tokens arrive. By delivering tokens to the user as they are generated (server-sent events), the perceived latency drops dramatically even though total generation time stays the same. The cost impact comes from the downstream effect: users who see immediate token delivery are far less likely to refresh, retry, or abandon and resubmit a request.

In high-traffic production systems, retry-driven duplicate requests can account for 15–25% of total token spend. A user whose synchronous request times out at 15 seconds and retries generates 2× the token cost for 1 result. The same user receiving a streaming response that shows the first token within 200ms (TTFT — time to first token) will almost never retry. Streaming is therefore a cost optimisation as much as a UX improvement.

Additionally, streaming enables early termination: if a user finds the partial response sufficient before generation completes, you can stop generation and avoid billing for the remaining tokens. This is particularly effective for open-ended generation tasks where users often want just enough — not everything the model was going to produce.

In virtually every multi-user AI deployment, a small percentage of users accounts for a disproportionate share of token spend. The top 5% of users by consumption commonly generate 30–50% of total token cost. Without rate limits or per-user budgets, a single heavy user — or an automated process running out of control — can trigger cost spikes that account for a month’s expected spend in a single day.

The solution is a token budget enforcement layer — an AI gateway or middleware component that tracks token consumption per user, team, or feature, and applies throttling or hard limits when budgets are approached or exceeded. This is distinct from the provider-level rate limits (which protect the provider’s infrastructure) — this is your application-level spending control.

Implement with soft limits (warning at 80% of daily budget, suggest more concise prompts) and hard limits (throttle at 100%, return a graceful “quota reached” message). For enterprise deployments, attribute spend by team, product feature, and use case — this gives engineering and finance teams the visibility to make smart allocation decisions rather than cutting costs blindly. The FinOps Foundation State of FinOps 2026 Report found that mature FinOps practices achieve 20–30% cloud cost reductions without performance degradation, but only 42% of teams implement these practices consistently.

You cannot optimise what you cannot measure. Monitoring and cost attribution are not themselves a cost reduction technique — they are the prerequisite for every other technique on this list. Teams that deploy AI without per-feature, per-user, per-endpoint cost attribution routinely over-invest in the wrong optimisations and miss the highest-impact ones entirely.

The correct unit of measurement is cost per successful outcome — not cost per token, not cost per request. A request that generates 5,000 tokens to answer a simple question that needed 100 is a measurement failure as much as a prompt engineering failure. Track: input tokens by endpoint (p50/p95), output tokens by endpoint, cost per user action, retry rates, cache hit rates, and batch utilisation.

# Minimum viable inference cost tracking
metrics = {
  "input_tokens":  response.usage.input_tokens,
  "output_tokens": response.usage.output_tokens,
  "cache_read":    response.usage.cache_read_input_tokens,
  "model":         response.model,
  "feature":       context.feature_name,
  "user_id":       context.user_id,
  "cost_usd":      calculate_cost(response.usage, response.model)
}

Attribute costs by product feature, team, and user segment. This visibility alone typically reveals 2–3 features responsible for 60%+ of spend, a small set of users generating disproportionate cost, and optimisation opportunities that were invisible without the data.

Cloud frontier model APIs are priced for capability. For queries that don’t require that capability, you’re paying a premium for nothing. Hybrid routing sends simple, high-volume, low-sensitivity queries to locally-hosted open-weight models (Llama 3, Mistral, Qwen, Phi-4) — which have effectively zero marginal cost once the GPU is provisioned — and reserves cloud API calls for complex queries that genuinely require frontier capabilities.

The economics work at meaningful scale: at 10,000 daily requests where 70% are classifiable as simple, routing those 7,000 to a local model at near-zero marginal cost versus a frontier API at $0.50/1M tokens ($3.50/day saved) vs $5/1M tokens ($35/day saved) creates substantial sustained savings. The break-even on GPU provisioning depends on utilisation rates and the cost of on-prem or reserved cloud GPU capacity.

Tools like LiteLLM, PortKey, and OpenRouter implement hybrid routing natively — routing rules can be configured without custom code. Privacy and data sovereignty are additional motivations beyond cost: sensitive data that cannot leave the enterprise perimeter must route to on-prem models regardless of query complexity, making hybrid routing a compliance architecture as much as a cost architecture.