About our website checker

What it is

ProspectorBot is the website checker behind Site Review Scout. When one of our customers — usually a web designer, or the business owner themselves — asks for a review of a local business’s website, ProspectorBot visits that site the way a browser would and checks things like whether it works on phones, loads securely, and shows the business’s contact details.

What it does on your site

• Reads robots.txt first and follows it.
• Fetches the home page, sitemap.xml and any redirects, and may take one screenshot of the home page as a phone would see it.
• Never logs in, fills in forms, or crawls your whole site.
• Sends at most about one request per second to any one website, however many people are checking it.
• Identifies itself in the User-Agent header: Mozilla/5.0 (compatible; ProspectorBot/1.0; +https://sitereviewscout.com/bot)

Page-speed scores come from Google PageSpeed Insights, so you may also see a request from Google’s own systems at the same time.

How to opt out

Add this to your site’s robots.txt and ProspectorBot won’t check any page on it:

User-agent: ProspectorBot
Disallow: /

Or contact us and choose “Opt-out request”, telling us your website address. Once we add it to our opt-out list, nobody can run a check on it through Site Review Scout.