llms.txt

I run digital marketing for an industrial chemical manufacturer and for Peplio, and one question keeps coming up in both jobs: should we let AI crawlers read our website or block them? Peplio’s guide to llms.txt already gets search impressions from people asking the same thing. Global studies have counted AI crawlers on the world’s biggest websites, but I could not find anyone who had checked Indian websites. So on November 1, 2026, I checked 300 of them myself. This page shares every number, the full method and the raw dataset, free to use with credit.

7 Key Findings on AI Crawlers and Indian Websites

1. Blocking AI crawlers is still the minority choice in India. 39 of 230 readable robots.txt files (17%) fully block at least one of the 14 AI crawlers we tracked. The other 191 do not fully block any of them.
2. News publishers are the exception. 24 of 44 news and media sites (55%) fully block at least one AI crawler. No other sector goes above 14%.
3. The most blocked AI crawlers are not OpenAI’s. ByteDance’s Bytespider and Common Crawl’s CCBot are blocked by 22 sites each, Meta’s crawler by 21, and OpenAI’s GPTBot and Anthropic’s ClaudeBot by 20 each.
4. 11 sites also block ChatGPT search. They block OAI-SearchBot, which OpenAI says keeps a site out of ChatGPT search answers.
5. More sites invite AI than block it. 84 of 300 sites publish an llms.txt file, four times the 20 that block GPTBot.
6. Nobody has adopted the new shared opt-out rule yet. None of the 230 files used the __usage_ai-training token that Google, Microsoft, OpenAI and Anthropic backed on October 30, 2026.
7. 1 in 12 sites turned our check away. 25 of the 300 sites refused a normal request for their public robots.txt file, which shows how aggressive bot protection has become.

Which AI Crawlers Do Indian Websites Block?

We counted a site as blocking an AI crawler when its robots.txt names that bot and tells it Disallow: /, which means “do not read any page”. Here is how often each of the main AI crawlers is fully blocked across the 230 readable files.

Download chart

Use this chart on your site

<a href=”https://peplio.com/indian-websites-block-ai-crawlers-study/”><img src=”https://peplio.com/wp-content/uploads/2026/11/indian-websites-ai-crawlers-block-rate-by-bot-chart.png” alt=”Which AI crawlers Indian websites block, Peplio study 2026″ width=”1200″ height=”900″ style=”max-width:100%;height:auto”></a><p>Source: <a href=”https://peplio.com/indian-websites-block-ai-crawlers-study/”>Peplio AI Crawler Study India 2026</a></p>Copy embed code

Out of 230 readable robots.txt files. “Partly blocked” means the bot is named with some Disallow rules but not the whole site. Bot purposes from OpenAI, Anthropic, Perplexity, Common Crawl, Apple and Meta.

One pattern stands out: Indian sites that block AI crawlers usually block the less known training bots first. Bytespider, CCBot and Meta’s crawler send no visitors back, so blocking them costs nothing. OpenAI’s and Anthropic’s bots come next. Search-style bots such as PerplexityBot are named and allowed more often (23 sites) than they are blocked (11).

Which Indian Sectors Block AI Crawlers Most?

The difference between sectors is the biggest finding of this study. News publishers want to protect their journalism from AI training. Banks, fintech apps and tech companies want AI tools to recommend them, so they do the opposite.

Blocking percentages use readable robots.txt files; llms.txt percentages use all sites checked. Source: Peplio AI Crawler Study India 2026.

Government sites stand apart in a different way: only 25 of 50 had a readable robots.txt at all. Seven returned a web page instead of a robots.txt file, six had no file, four refused our request and eight did not respond or returned an error. None of the 25 readable government files names a single AI crawler.

Who Blocks Which AI Crawlers: 12 Big Indian Websites

These are the exact rules we found in each site’s public robots.txt on November 1, 2026. Sites change these files often, so check the live file before quoting a single site.

Source: each site’s public robots.txt, read on November 1, 2026. Full rules for all 300 sites are in the free dataset below.

Look Up Any of the 300 Sites

Type a domain from the study to see which AI crawlers it blocks, whether it shuts out ChatGPT search, and whether it publishes llms.txt.

The ChatGPT Search Trap: Blocking the Wrong AI Crawler

OpenAI runs three different bots, and they do different jobs. GPTBot collects training data. OAI-SearchBot decides which sites appear in ChatGPT search. ChatGPT-User opens a page when a person asks ChatGPT to. OpenAI’s documentation is clear about the second one: “Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers.”In our study, 11 sites block OAI-SearchBot as well as GPTBot: The Hindu, News18, Firstpost, Deccan Herald, Dainik Bhaskar, Mathrubhumi, Eenadu, Amazon.in, Zomato, Internshala and Naukri. That may be a deliberate choice, but it also means ChatGPT search cannot cite them.9 sites split it the smart way: Moneycontrol, Manorama Online, Dinamalar, Oneindia, Inc42, YourStory, MediaNama, Decathlon India and Country Delight block GPTBot from training but leave OAI-SearchBot open, so they can still appear and be cited in ChatGPT search. If you want to protect your content from AI training without disappearing from AI answers, this is the pattern to copy. Learn how citations work in our guide on how to get cited by Perplexity.

The llms.txt Surprise: Indian Websites Are Inviting AI Crawlers In

An llms.txt file is a plain text guide that tells AI systems what a website is about and which pages matter. Our llms.txt guide explains how it works. We expected few Indian sites to have one. Instead, 84 of 300 sites (28%) publish an llms.txt file, four times the number that block GPTBot.

Download chart

Use this chart on your site

<a href=”https://peplio.com/indian-websites-block-ai-crawlers-study/”><img src=”https://peplio.com/wp-content/uploads/2026/11/indian-websites-block-ai-vs-llms-txt-by-sector-chart.png” alt=”Indian sectors: block AI crawlers or publish llms.txt, Peplio study 2026″ width=”1200″ height=”900″ style=”max-width:100%;height:auto”></a><p>Source: <a href=”https://peplio.com/indian-websites-block-ai-crawlers-study/”>Peplio AI Crawler Study India 2026</a></p>Copy embed code

Tech and consumer apps lead with 23 llms.txt files, followed by banks and fintech with 20. Names include Zerodha, Razorpay, Paytm, Policybazaar, Airtel, Jio, Zoho, Postman, Upgrad and Goibibo. Several files openly welcome AI: Federal Bank’s and Airtel’s files both begin by saying they welcome AI systems such as Google AI and ChatGPT.Six sites send mixed signals. Zomato, Naukri, Internshala, Decathlon India, Country Delight and Dinamalar block GPTBot in robots.txt but publish an llms.txt file for AI systems. Google says its AI features need no llms.txt file, but other AI tools may read it, so having both a block and an invitation is worth reviewing. See what actually helps in our generative engine optimization guide.

Zero Indian Sites Use the New Shared Opt-Out for AI Crawlers

On October 30, 2026, Google, Microsoft, OpenAI and Anthropic backed one shared robots.txt token, __usage_ai-training, that tells all of them at once not to train on a site. Microsoft will honour it from February 1, 2027 and Google from early 2027, according to PPC Land. Two days later, none of the 230 Indian robots.txt files we read used it. That gives a clean starting point: we will run this AI crawlers study again and report how fast Indian sites adopt it.

What This AI Crawlers Study Means for You

Build your own AI crawler rules

Tick the AI crawlers you want to block. Use a preset to start. The rules update instantly; copy them to the top of your robots.txt file.

Never block Googlebot or Bingbot to stop AI training. That removes you from search. Google-Extended is the right control for Gemini training, according to Google’s crawler documentation.

Free Downloads: AI Crawlers Dataset, Policy Pack and Press Kit

Everything from this study is free for bloggers, journalists, PR agencies, brands and researchers. No signup. Please credit Peplio and link to this page when you use it.

How to cite: Peplio AI Crawler Study India 2026, peplio.com/indian-websites-block-ai-crawlers-study/

How We Ran This AI Crawlers Study (Method and Limits)

• Sites: 300 well-known Indian websites, 50 in each of 6 sectors, chosen by Peplio from leading Indian publishers, listed companies, banks, unicorn startups, universities and government portals. The full list is in the dataset.
• Date and method: on November 1, 2026, a script downloaded each site’s public /robots.txt and /llms.txt once, from a home internet connection in India, using a standard browser user agent. If the plain domain failed, it tried the www version.
• Blocking rule: a bot counts as fully blocked when a robots.txt group that names it contains Disallow: /. Wildcard rules (User-agent: *) were recorded separately; only zomato.com and data.gov.in block all unnamed bots this way.
• llms.txt rule: counted only when the file returned plain text, not a web page or robots rules.
• Not readable: 70 sites. 25 refused our request (403), 20 did not respond, 14 returned a web page instead of robots.txt, 8 had no file, and 3 returned other errors.
• Limits: robots.txt shows a site’s stated wishes, not which bots actually visit or obey. 300 sites is a focused sample of well-known Indian websites, not every website in India. Files change often, so we will repeat the study and date every update.

FAQ About AI Crawlers and Indian Websites

Sources for this AI crawlers study: OpenAI crawler documentation, Anthropic crawler help article, Google common crawlers, Perplexity bots, Common Crawl CCBot, Apple Applebot, Meta web crawlers, Amazonbot, PPC Land on the shared opt-out, AIrefs crawler reference.