About ScalixSearchBot

ScalixSearchBot is the web crawler for Scalix Search, the retrieval layer of Scalix Cloud. It fetches publicly available pages to build a search index used by the platform and its agents. This page explains how to identify it, what it respects, and how to control or block it.

Identifying ScalixSearchBot

Requests from ScalixSearchBot carry this User-Agent:

ScalixSearchBot currently crawls from the IP address 103.153.105.120. Traffic claiming this User-Agent from any other address is not ours. This list will be kept current as our infrastructure grows.

What it respects

• robots.txt allow and disallow rules. A group addressed to ScalixSearchBot takes precedence; otherwise the * group applies. Rules are re-checked regularly.
• Crawl-delay. We honour the robots.txt Crawl-delay directive, and we never fetch more than one page at a time per host regardless of it.
• Page-level meta robots and X-Robots-Tag. noindex keeps a page out of the index, and removes it if you add the directive after we have already crawled it. nofollow stops us following or recording its links. A tag addressed to ScalixSearchBot takes precedence over a generic one.
• Conditional requests. We store your ETag and Last-Modified and send them back on every recrawl, so an unchanged page costs you a 304 instead of a full response body. We request compressed responses and accept them.
• Back-off signals. A 429 or 503 with a Retry-After is obeyed as written. If your robots.txt is unreachable or returning errors, we treat your whole site as disallowed until it comes back.
• Redirect and size limits. Bounded redirects, bounded response sizes, HTML and text only, no cookies, no JavaScript execution.

Blocking or limiting the crawler

To stop ScalixSearchBot from crawling your site, add this to your robots.txt:

To slow it down instead:

Page-level control works too, if you would rather not edit robots.txt: <meta name="ScalixSearchBot" content="noindex"> or the equivalent X-Robots-Tag response header.

Directives we do not support yet

We read noindex, nofollow and none. Other directives, including nosnippet and unavailable_after, parse cleanly but are not acted on yet, and we say so here rather than imply otherwise. Sitemap discovery is not wired up either: we find pages by following links from the docs sites we seed.

Contact

Questions, complaints, or unexpected traffic from our crawler: hello@scalix.world with the subject line “ScalixSearchBot”. Include the affected host and a timestamp and we will investigate promptly.