AI crawler access checker (Brilad, September 2026)
https://www.brilad.com/

Checks one page the way search and AI crawlers meet it: robots.txt rules,
the HTTP response for each crawler's user agent, firewall challenges,
noindex signals, text present before JavaScript, and /llms.txt.

Requirements: Python 3.8 or newer. No other packages.

Run:
  python3 ai_crawler_check.py https://www.yoursite.com/your-page/
  python3 ai_crawler_check.py https://www.yoursite.com/ --csv results.csv

On Windows, use "py" instead of "python3" if needed.

Limits: requests come from your computer with each crawler's user agent.
Firewalls that verify crawlers by IP address may treat the real crawlers
differently, so confirm important findings in your server logs.
