Do You Need a Reranker for RAG? Embeddings, Reranking, and Two-Stage Retrieval Explained

Not every RAG application needs a reranker. It becomes useful when the retrieval system regularly finds relevant evidence but does not rank the best passages high enough. Embedding models support efficient candidate retrieval across large document collections, while reranking models compare the query with those candidates in greater detail. SiliconFlow provides both Embedding and Rerank APIs, enabling developers to build, test, and optimize a two-stage retrieval pipeline through one model API platform.

Why Does Vector Search Sometimes Miss the Best RAG Context?

Vector search is a practical foundation for retrieval-augmented generation because it can compare a query with a large collection of precomputed document vectors. However, the passages with the highest similarity scores are not always the passages most useful for answering the user's question.

Semantic Similarity Does Not Always Equal Answer Relevance

An embedding model may identify two passages as semantically similar because they discuss the same topic. That does not mean both passages satisfy the user's exact intent.

For example, a query about increasing an API rate limit might retrieve:

• A general definition of rate limits

• A pricing page that mentions usage

• A troubleshooting guide for HTTP 429 errors

• The specific procedure for requesting a higher limit

All four passages are related to the query, but only one may contain the required action. A reranker can compare the complete query with each retrieved candidate and place the most useful passage closer to the top.

Large Collections Contain More Hard Negatives

A large knowledge base does not automatically require reranking. However, larger collections often contain more hard negatives: documents that appear highly relevant but do not provide the evidence needed to answer the question.

This is common in collections containing:

• Multiple versions of API documentation

• Similar product specifications

• Repeated support articles

• Regional variations of the same policy

• Archived and current documents

• Pages that reuse similar technical terminology

These passages may appear close together in an embedding space. A reranker adds a second relevance signal that can distinguish between broadly similar content and content that directly satisfies the query.

Duplicate and Versioned Content Can Compete

Near-duplicate documents may occupy several positions in the initial retrieval results. This can reduce source diversity and allow outdated information to compete with a newer version.

Reranking may improve their order, but it does not replace document management. Version metadata, publication dates, access permissions, duplicate removal, and archive rules should still be handled during ingestion or retrieval.

Ambiguous Queries Can Match Multiple Meanings

Short queries, abbreviations, product names, and technical terms may have several interpretations. Embedding search can retrieve documents associated with more than one meaning, especially when the query lacks context.

A reranker may improve the order when the intended meaning is clear from the full query or conversation. When essential information is missing, query rewriting, metadata filtering, conversation history, or user clarification may still be necessary.

Poor Chunking Cannot Be Solved by Ranking Alone

Some retrieval problems begin before vector search. Source documents may be divided into chunks that are too short, too long, or disconnected from their headings and surrounding context.

For example:

• A table may be separated from its column labels.

• An API limit may be separated from the model it applies to.

• A procedure may be split from an important warning.

• A heading may be stored separately from the paragraph it describes.

A reranker can only reorder the candidates it receives. It cannot reconstruct missing context or recover a relevant passage that the first-stage retriever failed to return.

Embedding Model vs. Reranking Model

Embedding and reranking models perform complementary roles in a RAG pipeline. They are not interchangeable alternatives.

An embedding model converts document chunks into vectors that can be stored in a vector database. At query time, the same type of model converts the query into a vector, and the database retrieves nearby candidates.

This architecture scales because document embeddings can be generated before users submit their queries. The system does not need to compare every query directly with every source document.

A reranking model operates after candidate retrieval. It receives the original query and a limited set of candidate documents, then produces a relevance score or revised order for each result.

The Qwen3 Embedding series includes separate embedding and reranking models in 0.6B, 4B, and 8B sizes. These options are designed to support different efficiency and effectiveness requirements.

The two stages answer different questions:

• Embedding model: Which documents might be relevant?

• Reranking model: Which of those candidates best satisfy this query?

A reranker therefore does not normally replace embeddings in a large RAG system. Embeddings first reduce the search space, while the reranker applies more detailed relevance evaluation to a manageable candidate set.

When Do You Need a Reranker for RAG?

The decision should be based on retrieval evaluation rather than the size of the knowledge base alone.

The Correct Passage Is Retrieved but Ranked Too Low

One of the clearest signals is that the correct passage frequently appears in the initial candidate set but does not reach the final context.

For example, the relevant chunk may regularly appear in the top 20 results but rarely enter the top three. The generation model then receives less useful passages even though the retrieval system technically found the required information.

This is primarily a ranking problem, making it a strong use case for a reranker.

Queries Include Several Conditions

Broad semantic similarity may be insufficient when a query contains several constraints.

Examples include:

• Comparing two products under a specific budget

• Finding an API setting for one model and region

• Identifying a policy that applies after a particular date

• Troubleshooting an error under a specific configuration

• Retrieving evidence that must satisfy several technical conditions

A reranker can evaluate the full query against every candidate rather than relying only on vector proximity.

Retrieved Documents Are Highly Similar

Product catalogs, API documentation, internal knowledge bases, legal policies, and research collections often contain passages with overlapping language.

Reranking can help when the initial results are topically correct but differ in details such as:

• Product or model version

• Publication date

• Region

• Software environment

• User type

• Configuration

• Eligibility conditions

Only a Small Amount of Context Can Be Sent to the LLM

A retrieval system may return many potentially relevant passages, but the generation model should not always receive all of them.

Too much context can:

• Increase input token usage

• Add duplicate information

• Introduce conflicting versions

• Distract the model from the strongest evidence

• Increase generation latency

A reranker can help select a smaller, higher-signal context set from a broader group of candidates.

Retrieval Accuracy Has High Business Value

Technical support, enterprise search, compliance workflows, financial research, and other accuracy-sensitive applications may benefit from more precise evidence selection.

The additional inference step may be justified when selecting the wrong passage can produce a misleading answer, trigger repeated corrections, or reduce user trust.

First-Stage Retrieval Already Works Well

A reranker may provide little measurable value when:

• The knowledge base is small and clearly structured.

• The correct answer consistently appears near the top.

• Metadata filters already separate similar documents.

• Queries closely match stored FAQ questions.

• Exact matching or hybrid search handles the workload.

• The application has an extremely strict latency budget.

• Offline evaluation shows no meaningful improvement.

Adding a reranker without measurable gains increases system complexity without necessarily improving the user experience.

What Can a Reranker Not Fix?

Reranking is one part of RAG retrieval optimization. It cannot directly fix:

• Missing or unindexed documents

• Incorrect document permissions

• Outdated source material

• Poor OCR or text extraction

• Weak chunk boundaries

• Incorrect metadata filters

• An incomplete first-stage candidate set

• Queries that omit essential details

• Answers that do not exist in the knowledge base

Before adding a reranker, inspect failed queries and determine whether the correct evidence was retrieved at all.

When the correct passage is missing from a reasonably broad candidate set, focus first on the embedding model, chunking strategy, metadata, query transformation, hybrid retrieval, or index configuration.

When the correct passage is present but ranked below weaker results, reranking is more likely to help.

How Two-Stage Retrieval Works with SiliconFlow APIs

A two-stage retrieval architecture separates broad candidate discovery from detailed relevance ranking.

Stage one casts a wide net; stage two decides what the model actually reads.

Stage 1: Generate and Store Document Embeddings

During ingestion, divide source documents into meaningful chunks and send the text to the SiliconFlow Embedding API:

POST https://api.siliconflow.com/v1/embeddings

POST https://api.siliconflow.com/v1/embeddings

POST https://api.siliconflow.com/v1/embeddings

The API converts input text into embedding vectors. The current international API documentation includes Qwen/Qwen3-Embedding-8B, alongside the 0.6B and 4B variants. The Qwen3 models also support configurable output dimensions, which must remain consistent with the target vector index.

A simplified Python request looks like this:

import os
from typing import Any

import requests

API_KEY = os.environ["SILICONFLOW_API_KEY"]
BASE_URL = "https://api.siliconflow.com/v1"


def create_embedding(text: str) -> list[float]:
    if not text.strip():
        raise ValueError("text must not be empty")

    response = requests.post(
        f"{BASE_URL}/embeddings",
        headers={
            "Authorization": f"Bearer {API_KEY}",
            "Content-Type": "application/json",
        },
        json={
            "model": "Qwen/Qwen3-Embedding-8B",
            "input": text,
        },
        timeout=60,
    )
    response.raise_for_status()

    result: dict[str, Any] = response.json()
    return result["data"][0]["embedding"]

import os
from typing import Any

import requests

API_KEY = os.environ["SILICONFLOW_API_KEY"]
BASE_URL = "https://api.siliconflow.com/v1"


def create_embedding(text: str) -> list[float]:
    if not text.strip():
        raise ValueError("text must not be empty")

    response = requests.post(
        f"{BASE_URL}/embeddings",
        headers={
            "Authorization": f"Bearer {API_KEY}",
            "Content-Type": "application/json",
        },
        json={
            "model": "Qwen/Qwen3-Embedding-8B",
            "input": text,
        },
        timeout=60,
    )
    response.raise_for_status()

    result: dict[str, Any] = response.json()
    return result["data"][0]["embedding"]

import os
from typing import Any

import requests

API_KEY = os.environ["SILICONFLOW_API_KEY"]
BASE_URL = "https://api.siliconflow.com/v1"


def create_embedding(text: str) -> list[float]:
    if not text.strip():
        raise ValueError("text must not be empty")

    response = requests.post(
        f"{BASE_URL}/embeddings",
        headers={
            "Authorization": f"Bearer {API_KEY}",
            "Content-Type": "application/json",
        },
        json={
            "model": "Qwen/Qwen3-Embedding-8B",
            "input": text,
        },
        timeout=60,
    )
    response.raise_for_status()

    result: dict[str, Any] = response.json()
    return result["data"][0]["embedding"]

Store each vector together with the original text, document identifier, source URL, version, access information, and other useful metadata.

Stage 2: Retrieve a Candidate Set

When a user submits a query:

1. Generate an embedding for the query.

2. Search the vector database.

3. Return a candidate set for reranking.

The first stage should prioritize recall. Its job is to make sure relevant evidence enters the candidate set, not necessarily to produce the final order.

The right candidate depth depends on the dataset and latency budget. Retrieving too few passages limits what the reranker can select. Retrieving too many increases reranking input size and processing time.

There is no universal candidate count that works for every RAG system.

Stage 3: Rerank the Candidates

Send the original query and the retrieved document strings to the SiliconFlow Rerank API:

POST https://api.siliconflow.com/v1/rerank

POST https://api.siliconflow.com/v1/rerank

POST https://api.siliconflow.com/v1/rerank

The current international API accepts a query, a string array of documents, and optional fields such as top_n and return_documents. Results include the original candidate index and a relevance score, allowing the application to reconnect each result with its stored source data.

from collections.abc import Sequence
from typing import Any


def rerank_documents(
    query: str,
    documents: Sequence[str],
    top_n: int = 5,
) -> list[dict[str, Any]]:
    if not query.strip():
        raise ValueError("query must not be empty")

    if not documents:
        return []

    if top_n < 1:
        raise ValueError("top_n must be at least 1")

    response = requests.post(
        f"{BASE_URL}/rerank",
        headers={
            "Authorization": f"Bearer {API_KEY}",
            "Content-Type": "application/json",
        },
        json={
            "model": "Qwen/Qwen3-Reranker-8B",
            "query": query,
            "documents": list(documents),
            "top_n": min(top_n, len(documents)),
            "return_documents": True,
        },
        timeout=60,
    )
    response.raise_for_status()

    result: dict[str, Any] = response.json()
    return result["results"]

from collections.abc import Sequence
from typing import Any


def rerank_documents(
    query: str,
    documents: Sequence[str],
    top_n: int = 5,
) -> list[dict[str, Any]]:
    if not query.strip():
        raise ValueError("query must not be empty")

    if not documents:
        return []

    if top_n < 1:
        raise ValueError("top_n must be at least 1")

    response = requests.post(
        f"{BASE_URL}/rerank",
        headers={
            "Authorization": f"Bearer {API_KEY}",
            "Content-Type": "application/json",
        },
        json={
            "model": "Qwen/Qwen3-Reranker-8B",
            "query": query,
            "documents": list(documents),
            "top_n": min(top_n, len(documents)),
            "return_documents": True,
        },
        timeout=60,
    )
    response.raise_for_status()

    result: dict[str, Any] = response.json()
    return result["results"]

from collections.abc import Sequence
from typing import Any


def rerank_documents(
    query: str,
    documents: Sequence[str],
    top_n: int = 5,
) -> list[dict[str, Any]]:
    if not query.strip():
        raise ValueError("query must not be empty")

    if not documents:
        return []

    if top_n < 1:
        raise ValueError("top_n must be at least 1")

    response = requests.post(
        f"{BASE_URL}/rerank",
        headers={
            "Authorization": f"Bearer {API_KEY}",
            "Content-Type": "application/json",
        },
        json={
            "model": "Qwen/Qwen3-Reranker-8B",
            "query": query,
            "documents": list(documents),
            "top_n": min(top_n, len(documents)),
            "return_documents": True,
        },
        timeout=60,
    )
    response.raise_for_status()

    result: dict[str, Any] = response.json()
    return result["results"]

Model availability can change over time. Keep model IDs in configuration rather than hard-coding them throughout the application, and verify current availability before production deployment.

Stage 4: Build the Final Context

After reranking, select the highest-ranked passages that fit the application's context budget.

The final context-building step may also apply:

• Duplicate removal

• Source diversity rules

• Token limits

• Version or date checks

• Access-control validation

• Citation formatting

• Document-level grouping

Avoid treating one relevance-score threshold as universally valid. Score distributions can vary by model, dataset, and query type. Thresholds should be calibrated using representative queries.

The final context can then be sent to a generation model through SiliconFlow's Chat Completions API, creating a complete retrieval, reranking, and generation workflow through the same AI infrastructure platform.

How to Measure Whether Reranking Improves Your RAG System

Compare an embedding-only baseline with a two-stage pipeline using the same evaluation queries. Do not judge the reranker from a few hand-selected examples.

Measure First-Stage Recall

First confirm that the initial retriever supplies useful candidates.

Common metrics include:

• Recall@k: How much of the relevant evidence appears within the first k candidates

• Hit Rate@k: How often at least one relevant passage appears within the first k candidates

If first-stage recall is weak, reranking cannot compensate for missing evidence.

Measure Ranking Quality

Ranking metrics show whether the reranker moves useful evidence closer to the top.

Useful metrics include:

• Precision@k: The proportion of the first k results that are relevant

• Mean Reciprocal Rank: How high the first relevant result appears

• nDCG@k: Whether more relevant documents are ranked above less relevant ones

• Recall@k after reranking: Whether the smaller final result set retains the required evidence

Choose metrics that match the application. A support assistant may need one correct passage near the top, while a research workflow may require several complementary sources.

Evaluate the Final Answer

An improved ranking matters only when it improves the complete RAG application.

Measure:

• Answer correctness

• Answer relevance

• Faithfulness to the retrieved context

• Unsupported claim rate

• Citation accuracy

• Task completion rate

• Frequency of user corrections

The evaluation set should include common queries, difficult queries, ambiguous wording, version-sensitive questions, and high-risk cases.

Measure Context Efficiency

Track whether reranking helps the application use its context window more effectively:

• Number of initial candidates

• Number of reranked results retained

• Final context tokens

• Duplicate context rate

• Relevant evidence per context window

Reranking does not automatically reduce context length. Token savings occur only when the application uses the new ranking to remove low-value passages.

Measure Latency and Cost

Reranking adds another inference step, so track:

• Reranker latency

• End-to-end latency

• p50 and p95 latency

• Reranker input tokens

• Generation input tokens

• Cost per completed query

• Error and timeout rates

A reranker may increase retrieval-stage latency while reducing the amount of context processed during generation. Evaluate the full request path rather than the reranker call alone.

Test several combinations of candidate depth and top_n. The best configuration is the one that meets the application's retrieval and answer-quality targets within its latency and cost limits.

FAQs

Is a Reranker Necessary for Every RAG Application?

No. Small or clearly structured knowledge bases may work well with embedding search, metadata filtering, keyword matching, or hybrid retrieval alone. Add a reranker when testing shows that useful candidates are retrieved but are not consistently ranked high enough.

Does a Reranker Replace Embeddings?

No. Embedding models retrieve candidates efficiently from a large collection. Reranking models refine the order of a smaller candidate set. The two model types perform complementary roles in a typical two-stage retrieval architecture.

Does Reranking Increase RAG Latency?

Usually. Reranking introduces another inference step. However, better ranking may allow the system to send fewer low-value chunks to the generation model. Measure total end-to-end latency rather than evaluating the reranker call in isolation.

How Many Documents Should I Send to a Reranker?

There is no universal number. The candidate set should be large enough to maintain first-stage recall but small enough to meet latency and cost requirements. Test several candidate depths on representative queries and compare retrieval quality, answer quality, token usage, and response time.

Can SiliconFlow Provide Both Embedding and Reranking APIs?

Yes. SiliconFlow provides an Embedding API for generating vectors and a Rerank API for scoring query-document relevance. Developers can combine them in a two-stage retrieval pipeline and send the selected context to a generation model through the same platform.

Build Reranking Around Measured Retrieval Problems

A reranker is most valuable when first-stage retrieval finds relevant evidence but does not rank it reliably. Before adding one, confirm that the correct chunks enter the candidate set and compare an embedding-only baseline with a two-stage pipeline. With SiliconFlow's Embedding and Rerank APIs, developers can test both approaches, adjust candidate depth and final context size, and choose a configuration that meets their application's accuracy, latency, and cost requirements.