The Hobbydex crawler

Hobbydex is an open parts database for RC cars. Its crawler fetches public pages and manuals from the makers and shops that publish them, so that each record can be checked against the source it was read from. It does not log in, does not buy, does not fill in forms and does not fetch anything a site has asked it to leave alone.

How it identifies itself

Every request carries this User-Agent:

hobbydex-crawler/0.1 (+https://hobbydex.com)

Requests to sites that ask for it are also signed with Web Bot Auth (HTTP Message Signatures, RFC 9421). The public key that verifies the signature is published at https://hobbydex.com/.well-known/http-message-signatures-directory. A signed request names hobbydex.com as its signature agent, so the site can check that it comes from this project and not from someone copying the name.

How it behaves

• robots.txt first. Every site's robots.txt is read before its pages are requested, and its rules are kept. A path a site disallows is not fetched.
• Slow and sequential. One site gets one request at a time, with a pause of 15 to 30 seconds between requests by default, and longer for some sites. Servers are never hit in parallel.
• Retry-After is obeyed. When a site answers 429 or 503 with a Retry-After header, the crawler waits exactly that long, no longer. A 429 without one pauses the site for a day.
• Cached. Each page is fetched once and kept, so repeat visits do not load the site again.
• Only public pages. No account, no cart, no checkout, no search form and no page behind a login.

Block it or ask to be left out

A site owner can keep the crawler out with robots.txt, for example:

User-agent: hobbydex-crawler
Disallow: /

Or ask us directly: open an issue on GitHub, naming the site and the paths to leave alone. We stop fetching that site once the request is received.

Who runs it

The crawler and hobbydex.com are run by Somatik BV, Belgium. Company details and contact: About and privacy.

What we keep

The part numbers, names, sizes and fitment that a source states, with a link to the maker page that lists them. The data is open under CC BY 4.0 (see the FAQ). Nothing from a site is published as a copy of its pages.