Bulk Index Checks With the Search Console URL Inspection API

How to check the index status, coverage, canonical and last crawl of hundreds of URLs with the Search Console URL Inspection API, with a Python script and guidance on reading the results.

Ashruba Pankhade Published 3 min read

Search Console’s URL Inspection API returns what Google knows about a URL: whether it is indexed, its coverage state, the canonical Google chose, when it was last crawled, and whether robots.txt or fetching blocked it. It lets you check hundreds of important pages automatically instead of one at a time in the interface, within a quota of 2,000 inspections per day per property. It cannot request indexing; that remains a manual action in Search Console.

What the API tells you

Field Meaning
verdict Overall result, such as PASS or NEUTRAL
coverageState For example “Submitted and indexed” or “Crawled – currently not indexed”
robotsTxtState Whether robots.txt allows crawling
indexingState Whether indexing is allowed, or blocked by noindex
pageFetchState Whether Google could fetch the page
lastCrawlTime When Google last crawled it
googleCanonical and userCanonical The canonical Google chose and the one you declared

Setup

The setup is the same as for the performance export: a Google Cloud project with the Search Console API enabled, a service account added as a user in Search Console, and the client library installed. See exporting Search Console data with Python.

The script

#!/usr/bin/env python3
"""Inspect a list of URLs with the Search Console URL Inspection API and save results to CSV.
Usage: python3 inspect_urls.py urls.txt"""
import csv, sys, time
from google.oauth2 import service_account
from googleapiclient.discovery import build

SITE = "sc-domain:example.com"          # your property, exactly as in Search Console
creds = service_account.Credentials.from_service_account_file(
    "service-account.json", scopes=["https://www.googleapis.com/auth/webmasters.readonly"])
service = build("searchconsole", "v1", credentials=creds)

urls = [u.strip() for u in open(sys.argv[1]) if u.strip()]
with open("inspection.csv", "w", newline="", encoding="utf-8") as f:
    w = csv.writer(f)
    w.writerow(["url", "verdict", "coverage", "robots", "indexing", "fetch",
                "last_crawl", "google_canonical", "user_canonical", "canonical_mismatch"])
    for url in urls:
        r = service.urlInspection().index().inspect(
            body={"inspectionUrl": url, "siteUrl": SITE}).execute()
        s = r.get("inspectionResult", {}).get("indexStatusResult", {})
        gc, uc = s.get("googleCanonical", ""), s.get("userCanonical", "")
        w.writerow([url, s.get("verdict"), s.get("coverageState"), s.get("robotsTxtState"),
                    s.get("indexingState"), s.get("pageFetchState"), s.get("lastCrawlTime"),
                    gc, uc, bool(gc and uc and gc != uc)])
        time.sleep(0.2)                    # stay well under the per-minute limit

Useful checks to run

  • Your most important pages: service, product and top-traffic pages, weekly
  • New pages a week after publishing, to confirm indexing
  • Pages after a migration or template change
  • Pages where Google chose a different canonical from yours, which often reveals duplicates

Reading the results

Finding Next step
Not indexed, crawled Check content quality and rendering; see “Crawled, currently not indexed”
Blocked by robots.txt or noindex Fix the rule if the block is unintended
Canonical mismatch Consolidate duplicates or fix canonical signals
Last crawl long ago Improve internal links to the page

Frequently asked questions

Can the API request indexing?

No. Requesting indexing is only available in the Search Console interface.

Why do API results differ from a live test?

The API reports the indexed version, as Google last processed it. The live test in the interface fetches the page now.

Is 2,000 inspections a day enough?

For monitoring important pages, usually yes. For full-site coverage on large sites, combine it with the Page indexing report and server logs.

Sources

  1. Google Search Console API documentation, urlInspection.index.inspect and usage limits
  2. Search Console Help, “URL Inspection tool”

Related reading