What Is LLM Inference? Latency, Throughput, Batching, and Cost Explained

Table of Contents

Every response from an LLM-powered chatbot, coding assistant, or document summarization tool depends on LLM inference. The model must process the prompt, manage its working context, and generate output tokens while sharing compute resources with other requests. Prompt length, output length, GPU memory, request concurrency, batching, and serving architecture all affect the result. Understanding these factors helps teams choose models, interpret benchmarks, control costs, and design more reliable production applications.

What Is LLM Inference?

LLM inference is the process of running a trained large language model on new input. The model uses its existing parameters to calculate token probabilities and generate an output. Its weights remain unchanged during this process.

Training creates the model's general capabilities by updating its parameters across large datasets. Fine-tuning continues that training on a narrower dataset or task. Inference uses the completed model to answer questions, generate code, summarize documents, or produce other outputs.

AI inference also covers image classification, speech recognition, computer vision, and recommendation systems. Large language model inference specifically concerns models that process and generate tokens.

In production, inference usually runs behind a serving layer. An LLM inference server loads model weights, accepts API requests, schedules concurrent workloads, manages GPU memory, and returns generated output to the application.

What Happens When an LLM Processes a Request?

An inference request moves through several stages:

  1. The serving system receives and validates the API request.

  2. The request may enter a queue while compute resources are occupied.

  3. A tokenizer converts the prompt into token IDs.

  4. The model processes the input and builds a key-value cache.

  5. A sampling method selects the first output token from the model's probability distribution.

  6. The model generates additional tokens one step at a time.

  7. The tokenizer converts the output tokens back into text.

  8. The server streams tokens to the application or returns the completed response.

Network transfer, queueing, tokenization, model execution, scheduling, and streaming all contribute to user-visible latency. A benchmark that measures GPU execution alone may therefore look faster than the complete API experience.

Our OpenAI-compatible API gives developers access to optimized language and multimodal models without requiring them to build and maintain the serving layer independently.

Prefill vs. Decode: The Two Main Stages of LLM Inference

Generative LLM inference has two main computational stages.

During prefill, the model processes all input tokens in the prompt. These calculations can run in parallel because the full input sequence is already available. The model also creates the initial KV cache. Longer prompts require more prefill work and usually increase time to first token.

During decode, the model generates output tokens autoregressively. Each new token depends on the input and all previously generated tokens. The process continues until the model reaches a stopping condition or output limit.

Prefill and decode on an inference timeline: prefill runs in parallel over the full prompt, then decode emits one token per step

Prefill is often compute-intensive, especially with long inputs. Decode repeatedly reads model weights and cached attention states, so memory bandwidth often becomes a major constraint.

Workload shape determines which stage dominates. A summarization request with a long document and short answer places more pressure on prefill. A short prompt requesting a long report spends more time in decode.

TTFT, TPOT, TPS, Latency, and Throughput Explained

A single response-time figure cannot describe the complete inference experience.

TTFT can include network time, queueing, tokenization, and prefill. TPOT, also called inter-token latency, focuses on token delivery after generation begins.

TPS needs a clearly stated scope. Higher concurrency may increase system TPS while reducing per-user TPS. Different tokenizers also represent text differently, so tokens per second cannot always be compared directly across models.

Metric

What it measures

Where it shows up

TTFT

Network time, queueing, tokenization, and prefill before the first token

Interactive assistants where the first token drives perceived speed

TPOT (inter-token latency)

Token delivery after generation begins

Streaming output that stalls mid-answer

TPS

Token output rate, per user or for the whole system

Throughput-oriented serving configurations

End-to-end latency

Total time from request to completed response

Long outputs and queueing under load

Goodput

Throughput that still meets the required latency target

High raw throughput with missed latency targets

Where each LLM inference metric applies: time to first token, TPOT, tokens per second, and end-to-end latency

These metrics can help locate performance problems:

  • High TTFT with normal TPOT can indicate long prompts, prefill pressure, queueing, or network delay.

  • Normal TTFT with high TPOT points more strongly toward decode speed, memory bandwidth, or heavy concurrency.

  • High system TPS with slow per-user TPS indicates a throughput-oriented serving configuration.

  • High raw throughput with low goodput means many requests are missing the required latency target.

Production tests should report model version, input and output lengths, concurrency, precision, streaming mode, and hardware. Percentiles such as p50 and p95 reveal performance variation that an average can hide.

Why GPU Memory Capacity and Bandwidth Matter for LLM Inference

GPU memory capacity determines which models can run and how much concurrent traffic the system can support. The main memory consumers include model weights, KV cache, intermediate activations, and runtime workspaces.

Model-weight memory grows with parameter count and numerical precision. As a rough estimate, a 7-billion-parameter model stored in 16-bit precision requires about 14 GB for weights before other memory allocations.

KV cache usage grows with the number of active requests and the total sequence length of each request. Long prompts, long outputs, and high concurrency can consume substantial memory. Limited capacity may restrict batch size, context length, or the number of simultaneous requests.

Memory bandwidth controls how quickly weights and cached states move between GPU memory and compute units. It has a particularly strong effect during decode.

Quantization can reduce weight or cache memory, while efficient attention kernels and KV cache management can improve resource use. Model parallelism allows larger models to run across multiple GPUs, although communication between devices can add latency. Effective LLM inference optimization depends on balancing these techniques against model quality and workload requirements.

How Batching Improves Throughput and Affects Latency

Batching groups multiple requests into shared execution steps. The GPU performs more useful work each time it reads model weights, which can raise total LLM inference throughput and reduce infrastructure cost per token.

Static batching starts a group of requests together. Requests often produce different output lengths, leaving shorter sequences waiting while longer ones continue.

Continuous or in-flight batching allows completed requests to leave and new requests to enter as capacity becomes available. This scheduling approach helps maintain GPU utilization under changing traffic.

Static versus continuous batching: short sequences idle, while continuous batching admits new requests as slots free

Larger batches also consume more KV cache memory. They can increase queueing time, TTFT, or TPOT when the system prioritizes total throughput. Interactive assistants usually need tight first-token and streaming latency targets. Offline extraction, evaluation, and document-processing jobs can often accept more latency for higher throughput.

What Affects the Cost of LLM Inference?

For API users, inference cost begins with model choice and token consumption. Text generation on SiliconFlow uses separate input and output pricing:

Total cost = input tokens × input unit price + output tokens × output unit price

Current prices vary by model and can be checked in the Model Plaza.

Several application choices affect total spending:

  • Larger or more compute-intensive models may carry higher token prices.

  • Long system prompts and conversation histories increase input usage.

  • Verbose responses and reasoning content increase output usage.

  • Repeated context adds tokens when prompt caching is unavailable.

  • Uncontrolled retries can duplicate token consumption after timeouts.

  • Routing simple tasks to the largest available model can waste budget.

Input and output tokens also create different infrastructure demands. Prefill processes input tokens with substantial parallelism. Output generation requires repeated decode steps. Providers may price these workloads differently, so teams should use the published rates for each model.

At the infrastructure level, hardware type, model precision, GPU utilization, batch efficiency, replica count, and idle capacity all influence cost. The most useful comparison is often cost per successful task under a defined quality and latency target.

Serverless vs. Dedicated Inference

Serverless inference works well for development, prototypes, traffic spikes, and applications with variable demand. The provider manages the serving infrastructure and scaling, while usage-based billing reduces the cost of idle capacity.

SiliconFlow serverless inference provides immediate access to optimized language and multimodal models. Developers can test different models through one integration and allow capacity to scale with changing request volume.

Dedicated inference assigns isolated compute resources to a workload. It can provide more predictable capacity for stable, high-volume production traffic. Its economics depend on sustained utilization because reserved resources continue to generate cost during quiet periods.

SiliconFlow currently provides serverless inference and reserved GPU resources for workloads that need consistent compute capacity. Dedicated endpoints are marked as coming soon, so their availability should be verified before deployment planning.

Dimension

Serverless inference

Dedicated inference

Capacity

Provider manages the serving infrastructure and scaling

Isolated compute resources assigned to one workload

Billing

Usage-based, so idle capacity does not add cost

Reserved resources keep generating cost during quiet periods

Utilization

Follows actual request volume

Economics depend on sustained utilization

Best fit

Development, prototypes, traffic spikes, variable demand

Stable, high-volume production traffic

Teams choosing a deployment model should compare:

  • Baseline and peak request volume

  • Input and output length distributions

  • Required p95 TTFT and TPOT

  • Expected GPU utilization

  • Isolation requirements

  • Cost per successful request

  • Traffic growth and capacity planning needs

How to Evaluate an Inference Provider

Benchmark providers with traffic that resembles the real application. A coding assistant, customer-support chatbot, summarization pipeline, and offline extraction system have different input lengths, output lengths, concurrency, and latency targets.

Use the same model version, generation parameters, prompt distribution, and output limits during each test. Measure p50 and p95 TTFT, TPOT, end-to-end latency, system TPS, RPS, goodput, and error rates. Continue the test long enough to expose queueing, throttling, and performance changes under sustained load.

Price comparisons should include the full workload. A low token rate may provide limited value if the model needs longer prompts, produces excessive output, or fails frequently. Calculate cost per successful task at the required quality level.

Also review:

  • Model and modality coverage

  • Rate limits and capacity-increase options

  • Streaming and structured-output support

  • API compatibility

  • Model-version stability

  • Monitoring and error visibility

  • Data handling and deployment requirements

On SiliconFlow, teams can test models in the Model Plaza, review current pricing and usage limits, and access models through an OpenAI-compatible API. Serverless inference supports flexible demand, while reserved GPU resources provide an option for sustained workloads. This allows teams to evaluate model quality, latency, throughput, and cost before committing to a production configuration.

LLM Inference FAQs

Is LLM Inference the Same as Model Training?

No. Training updates model parameters using large datasets. Inference keeps the trained parameters fixed and uses them to process new requests. Fine-tuning adapts an existing model through additional training on a narrower dataset.

What Is a Good LLM Inference Latency?

The target depends on the application. Interactive assistants usually need a short TTFT and steady token delivery. Batch systems often prioritize total throughput and cost. Define a latency target first, then test it with realistic prompt lengths and concurrency.

Does a Longer Context Increase Inference Time?

Usually. More input tokens increase prefill work and KV cache usage. The larger cache remains active during decode, adding memory pressure. The exact effect depends on model architecture, attention implementation, hardware, and caching behavior.

Does Batching Make Every Request Faster?

No. Batching mainly improves system throughput and GPU utilization. Individual requests may wait longer or receive tokens more slowly as concurrency increases. Scheduling settings determine how the system balances throughput, TTFT, and TPOT.

Can Quantization Reduce LLM Inference Cost?

Quantization reduces the numerical precision used for model weights or cached states. It can lower memory requirements and allow larger batches. Speed and output quality vary by model, quantization method, hardware, and task, so both performance and accuracy should be tested.

What Does an LLM Inference Server Do?

An inference server loads model weights, exposes an API, schedules requests, manages batching and memory, and streams generated tokens. Production serving systems also handle concurrent traffic, rate limits, monitoring, scaling, and failures.

Ready to accelerate your AI development?

Ready to accelerate your AI development?

Ready to accelerate your AI development?

© 2026 SiliconFlow

© 2026 SiliconFlow

© 2026 SiliconFlow