You can find missing internal links automatically by representing every page as a vector, comparing each page with every other, and listing the most similar pages it does not already link to. A person then reviews each suggestion and adds the link with suitable anchor text where the topic comes up. The script below works offline with TF-IDF word matching, and can switch to an embedding model for meaning-based matching, which finds related pages even when they use different words.
Why automate it
On a site with hundreds of articles, nobody remembers every related page. Older articles rarely link to newer ones, and pages on the same subject written months apart often do not connect at all. Internal links help search engines understand structure and help readers, and on this site we link every new article to related guides and posts. A suggestion tool makes that practical at scale. See hub-and-spoke internal linking.
The script
Input is a CSV with each page’s URL, title, main text and the URLs it already links to, which you can export from a crawler or your CMS.
#!/usr/bin/env python3
"""Suggest internal links: for each page, the most similar pages it does not link to yet.
Input: pages.csv with columns url,title,text,links (links = space-separated URLs already linked).
Usage: python3 link_suggest.py pages.csv"""
import csv, sys
import numpy as np
from sklearn.feature_extraction.text import TfidfVectorizer
def embed(texts):
# Replace with an embedding model for meaning-based matching; TF-IDF works offline.
m = TfidfVectorizer(stop_words="english", sublinear_tf=True).fit_transform(texts)
return m.toarray()
pages = list(csv.DictReader(open(sys.argv[1], encoding="utf-8")))
vecs = embed([p["title"] + " " + p["text"] for p in pages])
norms = np.linalg.norm(vecs, axis=1, keepdims=True); norms[norms == 0] = 1
sim = (vecs / norms) @ (vecs / norms).T
for i, p in enumerate(pages):
linked = set(p["links"].split())
ranked = [j for j in np.argsort(-sim[i]) if j != i and pages[j]["url"] not in linked]
picks = [(pages[j]["url"], round(float(sim[i][j]), 2)) for j in ranked[:3] if sim[i][j] > 0.1]
print(p["url"], "->", picks)
Install the libraries with pip install numpy scikit-learn.
What our test showed
We ran it on five short sample pages: three about getting cited by AI assistants, one about Core Web Vitals on WordPress and one about fixing INP on React. It correctly suggested links between the three AI pages, skipping one link that already existed. It suggested nothing between the Core Web Vitals page and the INP page, even though they are closely related, because they share few exact words.
That is the limit of word matching. An embedding model represents meaning, so it would recognise that both pages are about page speed and responsiveness. Replace the embed function with calls to an embedding model to get those connections. See embeddings and cosine similarity.
Reviewing suggestions
- Add a link only where the target helps the reader at that point in the text.
- Use anchor text that describes the target page, not “click here”.
- Prefer linking from older, well-crawled pages to newer ones that need discovery.
- Limit links per page to those that genuinely help; the tool shows the top three, not a quota.
Scaling it
- For thousands of pages, store vectors in a database such as PostgreSQL with pgvector and query nearest neighbours instead of comparing all pairs in memory.
- Run it after each batch of new content, and focus on pages with the fewest internal links pointing to them.
- Use the same vectors to choose related articles shown at the end of each post.
Frequently asked questions
Should links be added automatically?
No. Automatic insertion produces awkward anchors and irrelevant links. Use the tool to suggest; let a person decide.
Which text should be compared?
The title and main content, without navigation and footers. Including boilerplate makes every page look similar.
Does this help AI visibility?
Indirectly. Well-connected pages are found and understood more easily, and readers of one answer can reach the next question.
Sources
- scikit-learn documentation, TfidfVectorizer
- Google Search Central, “Link best practices for Google”