AI-Powered Semantic Search for Your Website: Hybrid Search With PostgreSQL

How to replace keyword-only site search with hybrid semantic search in PostgreSQL, combine rankings with reciprocal rank fusion, and use visitors' searches to find content gaps.

Ashruba Pankhade Published 4 min read

AI-powered semantic search lets visitors search your website in their own words and still find the right page, even when they use different terms from your content. It works by combining keyword matching with meaning-based matching through embeddings, a setup called hybrid search. For most sites it can be built on PostgreSQL with its built-in full-text search and the pgvector extension. As a bonus, the searches visitors run show you exactly which questions your content fails to answer.

Why keyword site search disappoints

Traditional site search matches words. A visitor who types “how long until I see results” will not find a page titled “GEO timeline” unless the words overlap. Meaning-based search compares the intent of the query with the meaning of each page or passage, so it finds the right page even without shared words. Exact terms, such as product codes and names, still work best with keyword matching, which is why hybrid search combines both.

Architecture

  1. IndexSplit pages into passages; store text and embeddings.
  2. QueryEmbed the visitor’s search text.
  3. Search twiceKeyword search and vector search.
  4. CombineMerge the two rankings.
  5. ShowPages with the matching passage highlighted.

Indexing works the same way as for a RAG chatbot; see building a RAG chatbot on your website content.

The hybrid query in PostgreSQL

-- passages(id, url, title, content, tsv tsvector, embedding vector(1536))
-- $1 = search text, $2 = embedding of the search text
WITH keyword AS (
  SELECT id, ROW_NUMBER() OVER (ORDER BY ts_rank(tsv, q) DESC) AS rank
  FROM passages, websearch_to_tsquery('english', $1) q
  WHERE tsv @@ q
  ORDER BY rank
  LIMIT 50
),
semantic AS (
  SELECT id, ROW_NUMBER() OVER (ORDER BY embedding <=> $2) AS rank
  FROM passages
  ORDER BY embedding <=> $2
  LIMIT 50
)
SELECT p.url, p.title, p.content,
       COALESCE(1.0 / (60 + k.rank), 0) + COALESCE(1.0 / (60 + s.rank), 0) AS score
FROM passages p
LEFT JOIN keyword k ON k.id = p.id
LEFT JOIN semantic s ON s.id = p.id
WHERE k.id IS NOT NULL OR s.id IS NOT NULL
ORDER BY score DESC
LIMIT 10;

This combines the two rankings with reciprocal rank fusion: each result scores by its position in each list, so a passage ranked well by both methods rises to the top. The constant 60 is the value commonly used for the method. Store a tsvector column updated from the content, with a GIN index, alongside the vector column and its HNSW index.

Show results well

  • Group results by page so one page does not fill the list with its passages.
  • Show the matching passage as the snippet, so visitors see why the result matched.
  • Offer a clear “no results” state with suggestions and a contact option.

Keep search results out of search engines

Internal search result pages should not be crawled or indexed. Google’s guidelines advise against letting search result pages be crawled, as they create endless low-value URLs. Disallow the search path in robots.txt and add noindex to result pages.

Use the searches as research

Track searches in GA4 with enhanced measurement’s site search event, or log them yourself. Queries with no good result are questions your site does not answer. They are some of the best content ideas you will find, because real visitors asked them in their own words. Add them to your topic map; see topical authority.

Frequently asked questions

Do we need a separate search service?

Not for most sites. PostgreSQL handles hybrid search for tens of thousands of passages. Dedicated search engines help at larger scale or when advanced features are needed.

Should search answer questions directly, like a chatbot?

You can add a short generated answer above the results, grounded in the top passages with links. Keep the list of results as well, since many visitors prefer to choose a page.

Does semantic search help SEO?

Not directly. It improves the experience for visitors already on your site, and its logs improve your content planning.

Sources

  1. PostgreSQL documentation, full-text search
  2. pgvector documentation
  3. Cormack, Clarke and Buettcher, “Reciprocal Rank Fusion outperforms Condorcet and individual rank learning methods”, 2009
  4. Google Search Central, guidance on crawling internal search results

Related reading