WebAnswerlyBot

1. Why it visits

A Web Answerly customer added your website as a knowledge source for their agent. Customers must confirm that they own the site or are authorised to have it crawled. What we fetch is used only to answer questions put to that customer's agent, with links back to your pages. It is not used to train AI models.

2. How to recognise it

Ordinary page requests identify themselves with this user agent:

WebAnswerlyBot/1.0 (+https://webanswerly.com/bot)

For pages that need JavaScript to show their content, the crawler uses a headless browser. Its user agent is a standard Chrome string ending in WebAnswerlyBot/1.0.

3. How it behaves

• Reads robots.txt before crawling and obeys its Allow and Disallow rules and Crawl-delay for WebAnswerlyBot.
• Prefers your sitemap over following links deep into the site.
• Paces itself, pausing between requests, or longer if your Crawl-delay asks.
• Fetches web pages, plus documents they link to (PDF, Word, CSV, Markdown and plain text, up to 10 MB each). It skips images, video and other media.
• Does not log in, and does not try to get past paywalls, CAPTCHAs or bot protection.
• Re-visits on the schedule the customer's plan allows, using conditional requests, so unchanged pages are not downloaded again.

4. Opting out

To block it completely, add this to your robots.txt:

User-agent: WebAnswerlyBot
Disallow: /

The crawler checks robots.txt at the start of every crawl, so a block applies from the next one.

5. Removal requests

If you own a site and did not authorise its crawling, email support@webanswerly.com with "Crawler" in the subject and the affected domain. Once we have confirmed you control the domain, we will remove its indexed content from any workspace that is not authorised to use it. See also our Acceptable Use Policy.