The crawler

What it sends

gameindustrytrends-bot/0.1 (+https://gameindustrytrends.com/bot)

One User-Agent, never a browser’s. Pretending to be Chrome would work and would take away your ability to see us, rate-limit us, or write to us, which is not a trade worth making for a site that depends on your continued goodwill.

How fast

At most 2 requests a second, from a single process making one request at a time. Two runs never crawl the same host at once — that is enforced by a lock, not by a schedule, so an overrunning job delays or skips the next one rather than doubling the rate you see.

Those counts include the crawler’s own robots.txt requests, and they are the size of a normal day rather than a cap: they move with how many tags and games exist, and the Roblox run in particular spends two more requests for every fifty experiences it has ever seen. The rate is the number that is fixed.

What it asks for

Nothing else. No search, no checkout, no download, no upload, no form — it issues GET requests and nothing else, and it does not log in.

What it keeps

Counters and metadata. The numbers a page publishes about a game — ratings, comments, topic views, visits, favourites, votes, concurrent players — plus the title, the author, the tags or genre, the thumbnail URL and the page’s own URL.

It stores no page text, no descriptions, no comments, no reviews and no images. We never republish page content; every game on this site links out to the page the numbers came from, which is the only place to read it.

How it behaves when you push back

robots.txt, on every run

Fetched once per host per run and applied before any other request to that host, so a rule you add is read on the next run rather than at some cache expiry. A disallowed URL is not requested. If robots.txt cannot be read at all — a 5xx, a timeout — the crawler treats the host as closed and does not crawl it.

Crawl-delay only slows it

A declared Crawl-delay is taken as a minimum gap between requests whenever it is wider than our own. It is never used to go faster: a host asking for a tenth of a second still gets 2 requests a second at most.

Retry-After is honoured

On a 429, 500, 502, 503 or 504 carrying Retry-After, the crawler waits the longer of that value and its own backoff. Ask for longer than 15 minutes and it issues no further request to that host for the rest of that run — it does not trim the wait and keep knocking, and it does not carry on to the next page either. A wait that long outlasts the run that made it, so the run is the thing that gives way. Both the seconds form and the HTTP-date form are understood. Those five statuses are also the only ones it retries at all.

404 is a stop signal

Not retried. Listing pagination is walked until a 404 says the pages have run out, which is how the crawler knows to stop rather than by guessing a page count.

How to block it

Put this in your robots.txt. It is read before anything else on the next run, so it takes effect within a day and needs no reply from us.

User-agent: gameindustrytrends-bot
Disallow: /

To slow it down instead, add a Crawl-delay under the same User-agent line. To block one section, list the paths under Disallow as usual.

Or just write to us

bot@gameindustrytrends.com reaches the person who runs it. A rate that is wrong for your server, a path that should not be fetched, or a request to stop entirely — all of them are fine, and none of them needs a reason.