Most of my website's visitors are bots. Is that bad?

Not bad in itself. On motdang.net, bots sent most of the requests (a request is one ask for a page or file) in one week of September 2026, and requests that look like a person's were 0.5% of them. A site can let bots read it, charge them or block them, and each choice has a different result.

Bots sent most of the requests

On 23 September 2026 I asked for the visits to all my projects to be compared "with demarcation between likely humans and likely bots." The public page from that work is What Bots Craaave ¹. A crawler is a program that reads pages automatically. The page counts motdang.net for 20 to 26 September 2026 from three places: Cloudflare's edge analytics (the count made by Cloudflare's servers in front of the site), Cloudflare's browser beacon (a snippet that counts a page load only when a browser ran the page), and the site's own counter ¹.

The shares are calculated from the page's figures. The answer to "how many visitors are bots" depends on the counting method: an edge count includes every request a bot makes, and a browser count includes only the pages a browser ran ¹.

Bots want new facts they can check

On 28 September 2026 I said what bots want is "new information and groundtruth from citable sources." Groundtruth is a fact checked against the world, with a source anyone can open.

The page relies on a 2024 paper in the journal Nature ¹². It reports that training AI on AI-written text without choosing it causes defects in which "tails of the original content distribution disappear" ². Its abstract adds that data collected about human interactions with systems becomes more valuable as AI-written text fills the web that crawlers collect ².

Jeremy Howard, an AI researcher, proposed llms.txt on 3 September 2024: a plain-text file, usually at the top level of a site, with a short guide and links for AI agents (programs that act on their own). His page says thousands of sites publish one ³.

Let bots read, charge them, or block them

Cloudflare's AI Crawl Control, available on all plans, shows which AI services access a site's content, lets you set allow or block rules for individual crawlers, and tracks which crawlers follow a site's robots.txt (the file where a site lists pages crawlers may skip) ⁴. Cloudflare's own documentation separates crawlers that index pages for search engines from crawlers that copy content without permission ⁵.

Cloudflare's pay per crawl page, checked on 24 October 2026, calls the feature a closed beta (a test open only to some users) ⁵.

Hongdam installs charging and llms.txt

Hongdam's "Charge AI crawlers" shows which bots read your site and turns on Cloudflare's pay per crawl. Its page notes the closed beta and that crawlers that do not pay receive an HTTP 402 with the price ⁶.

"Google and AI set-up" covers Search Console (Google's tool for how a site appears in search), a sitemap (a list of a site's pages for crawlers), page titles and llms.txt ⁷.