RAG and SEO

In brief

RAG (Retrieval-Augmented Generation), described in the original 2020 research paper, is an architecture that allows a model to consult external sources before generating its response. Without RAG, the model responds only from what it memorised during training. With RAG, it searches, retrieves and cites passages from relevant documents in real time. Search products may combine retrieval, ranking and generation, but providers do not expose identical pipelines. Optimising a private RAG index is different from making a public web page accessible to a search product.

1. How RAG works

A RAG system follows a four-step pipeline:

1. Indexation. Documents (web pages, PDFs, databases) may be cut into chunks and transformed into numerical vectors (embeddings) that represent their meaning. These vectors are stored in a vector database.
2. Retrieval. When a user asks a question, the query is transformed into a vector. The system looks for chunks whose vector is closest ("semantic neighbours"). The number of retrieved items is a system setting, not a universal range.
3. Augmentation. The selected chunks are injected into the LLM context ("prompt"), with the user question. The model therefore receives: [question] + [source passages].
4. Generation. The LLM produces a response that draws on the injected passages. In engines that display citations (Perplexity, ChatGPT Search), each claim is associated with its source.

Retrieval is one prerequisite in a RAG pipeline. Generation, citation rendering and product policies can still alter the final answer, so retrieval alone does not guarantee attribution.

2. Two types of RAG to know

2.1 Web RAG (real-time)

Used by Perplexity, ChatGPT Search, Bing Copilot and surfaces with active retrieval. The system performs a real web search on each query, crawls pages in the top results, chunks and semantically evaluates passages in a few seconds. Freshness and indexability (robots.txt, HTML rendering) are critical here.

2.2 Model memory and private retrieval are different

Parametric memory is information represented in model weights after training. It is not RAG. A private retrieval index is an external data store queried at answer time. Do not infer that a public page entered training data from a bot visit, a model mention or site popularity.

Practical implication: measure the surface you can observe. For web search, verify access, displayed sources and referral traffic. For a private RAG system, evaluate retrieval against a labelled test set.

3. What retrieval evaluates in your content

3.1 Semantic proximity

Vector retrieval measures the cosine distance between the query vector and the vector of each chunk. The closer a chunk is semantically to the question, the more chance it has of being selected.

Practical consequence: cover the exact vocabulary of the query in your content. Not keyword stuffing, but content that uses the same terms and synonyms that users employ when searching. An article on "generative engine optimisation" that never uses the word "GEO" or "AEO" will be sub-optimal for queries including those terms.

3.2 Informational density of the chunk

A focused passage is easier to retrieve and evaluate in many systems, but the result depends on the embedding model, index and query. Test this in the system you control rather than presenting it as a universal preference.

3.3 Self-containment

A chunk injected into an LLM prompt is read without its original context. If the passage says "this technique can increase citation rate" without naming the technique, the LLM cannot use this information. Each passage must be understandable on its own.

3.4 Factual precision

Verifiable claims are useful because an auditor can match the answer to its evidence: figures, dates, proper nouns, URLs, references to standards or norms. A vague claim ("many sites") is less exploitable than a precise claim ("the vendor documentation updated on 12 September 2026 lists this crawler").

4. Chunking: understanding how your page is cut

RAG systems automatically cut pages into chunks. Two chunking strategies coexist:

• Fixed chunking (by tokens). The page is cut at a configured token interval, sometimes with overlap. The values depend on the implementation.
• Structured chunking (by HTML tags / sections). The page is cut according to its HTML tags (h2, h3, p, li). Each section becomes a chunk. This can preserve editorial boundaries when the parser uses HTML structure.

Implication for the writer: structuring content into clear HTML sections can preserve context for parsers that use headings. Do not assume that every external product maps one H2 to one chunk or treats tables and code blocks identically.

5. What web RAG does not read (or reads poorly)

The following elements can create access or extraction problems. Behaviour varies by crawler and parser, so verify the returned HTML and extracted text:

• Content rendered only in JavaScript: an SPA without SSR returns empty HTML to the crawler. The chunk will be empty or unusable.
• Images without alt text: an infographic with key data but no alternative text is invisible to retrieval.
• Embedded videos without a transcript: important claims may be unavailable as text.
• PDFs without extractable text: PDFs that are scans or image PDFs cannot be parsed.
• Navigation menus: RAG systems attempt to filter navigation content to keep only editorial content, but navigational content integrated into the body can be incorrectly captured.

6. Concrete levers for RAG-ready content

6.1 Structure in self-contained sections

Each h2/h3 section of your page must be able to function as an independent chunk. Practical rule: if you could copy this paragraph into a tweet without context, is it still understandable? If not, add an introductory sentence that explicitly names the subject.

6.2 Favour definitions at the start of sections

Explicit subjects reduce ambiguity when a passage is read out of context. A useful head of the chunk. Preferred format: "[Term] is [complete definition]. It works by [mechanism]. The main use cases are [list]." Keep the definition natural and include only the context required by the reader.

6.3 Include sourced quantitative data

Precise figures need named sources, dates, populations and methods. This makes them auditable; it does not guarantee a better retrieval position. A useful format is:

"According to [source], [precise fact with figure], measured in [year]."

6.4 Add Article schema with dateModified

Article markup can describe publication and modification dates when those dates match the visible page. External answer engines do not publish a universal preference for schema.org dates, so use markup for accurate description rather than as a freshness trick.

6.5 Enable server rendering (SSR)

If your site is in React, Next.js or Vue, make sure the editorial content is server-rendered and present in the initial HTML. A curl https://your-site.com/page/ must return the text of the page, not an empty DOM. That is the simplest test to verify the RAG eligibility of your page.

6.6 Cover terminological variants of your subject

Semantic retrieval does not rely only on exact words, but covering synonyms and acronyms improves the overall embedding. On an article about RAG, including "retrieval augmented generation", "RAG", "search augmented generation", "vector database", "embeddings" in the same document improves semantic proximity for all these queries.

7. RAG and glossary: a powerful combination

A glossary page can serve a clear user need when it provides a sourced definition, scope and example. It is not automatically selected by RAG, and thin definitions copied from other pages add little value.

Create a dedicated page only when the term deserves a distinct explanation. A useful entry contains: formal definition, synonyms, difference from adjacent terms, and a concrete example. This format is directly compatible with structured chunking.

8. Measuring RAG eligibility of your pages

There is no native "RAG score" tool. The most useful proxies:

• Manual Perplexity test. Ask a very specific query that your page should cover. Do you appear in the sources? If not, which sources are cited and why are they better structured?
• Self-containment audit. Take 5 random paragraphs from your key pages. Read each one out of context. Count how many make sense without the rest. Target: 4/5 minimum.
• HTML analysis. curl -s https://your-page.com/ | grep -c "<p>" gives the number of p tags in the initial HTML. If result = 0, the page is not server-rendered.
• Third-party tools. Profound, Otterly and Scrunch offer citation reports that indirectly reveal if your pages pass the retrieval filtering of Perplexity and ChatGPT.

RAG-ready checklist

• Content server-rendered (SSR), verifiable via curl
• Each h2/h3 section is self-contained (understandable without context)
• No orphan pronouns at the start of sections ("it", "this method" without a named referent)
• At least 3 sourced quantitative data points in the article
• Key terms and synonyms present in the body text
• Article schema with datePublished + dateModified up to date
• Tables and lists with explicit headings (not "see below")
• Strategic images with descriptive and factual alt text

FAQ