404s and Soft 404s at Scale: Rules, Fixes and a Checker Script

A 404 tells search engines a page does not exist; a soft 404 is a page that says it does not exist, or has almost nothing on it, while returning status 200. Real 404s are normal and harmless. Soft 404s waste crawling, confuse search engines and appear as errors in Search Console. At scale, the fix is to set rules by page type: return 404 or 410 for pages that are truly gone, redirect only to genuinely equivalent pages, keep temporarily unavailable products live with useful information, and make empty search and filter pages return 404 or stay out of crawling.

404, 410 and soft 404

Google treats mass redirects of removed pages to the home page, or to unrelated pages, as soft 404s too.

Where soft 404s come from

• Custom “not found” pages that return 200 instead of 404
• Single-page apps that answer every URL with the same shell and 200
• Out-of-stock or discontinued product pages emptied of content
• Search and filter pages with no results
• Expired listings, events or jobs left with a “no longer available” notice
• Pages that fail to load their data from an API during rendering

Find them

1. Search Console’s Page indexing report lists “Soft 404” and “Not found (404)” URLs.
2. Crawl the site and flag 200 pages with very little text or error wording.
3. Check server logs for Googlebot requests to removed URL patterns.

This script flags likely soft 404s in a list of URLs: pages that return 200 but contain error phrases or very few words. Adjust the phrases and the word threshold to your site.

#!/usr/bin/env python3
"""Flag likely soft 404s: URLs that return 200 but look like error or empty pages.
Usage: python3 soft404_check.py urls.txt"""
import re, sys, urllib.request, urllib.error

PHRASES = ["page not found", "not found", "no longer available", "no products match",
           "no results", "0 results", "does not exist", "has been removed"]
MIN_WORDS = 80

def fetch(url):
    try:
        with urllib.request.urlopen(urllib.request.Request(url, headers={"User-Agent": "soft404-check/1.0"}), timeout=20) as r:
            return r.status, r.read().decode("utf-8", "replace")
    except urllib.error.HTTPError as e:
        return e.code, ""

for url in [u.strip() for u in open(sys.argv[1]) if u.strip()]:
    status, html = fetch(url)
    if status != 200:
        print(f"{status}  {url}"); continue
    text = re.sub(r"<script.*?</script>|<style.*?</style>|<[^>]+>", " ", html, flags=re.S | re.I)
    words = len(text.split())
    hits = [p for p in PHRASES if p in text.lower()]
    flag = "SOFT-404?" if hits or words < MIN_WORDS else "ok"
    print(f"200 {flag:10} {url}  words={words} {'phrases=' + ','.join(hits) if hits else ''}")

We tested it against a small local site with a “product no longer available” page, an empty filter page, a normal guide and a missing URL. It flagged the first two as likely soft 404s, passed the guide, and reported the missing URL’s 404 correctly. Treat flags as candidates to review, not verdicts; short pages can be legitimate.

Rules by page type

Keep a helpful 404 page

A good 404 page, returned with status 404, offers search, links to main sections and popular pages. It helps visitors without misleading search engines.

Frequently asked questions

Do many 404s hurt rankings?

Google says 404s for pages that should not exist are normal and do not harm the rest of the site. Fix 404s only for pages that should exist or that have valuable links.

Should we redirect all 404s to the home page?

No. Google treats that as soft 404s, and visitors lose context. Redirect only to genuinely equivalent pages.

404 or 410?

Both work. 410 states the removal is intentional and may be processed slightly faster; the difference is small.

Sources

1. Google Search Central, “HTTP status codes and network errors” and soft 404 guidance
2. Search Console Help, “Page indexing report”