AI Crawlers: What Gulf Websites Should Allow in robots.txt

AI crawlers are the automated bots that AI companies send to read websites. Each one now decides something different: whether your pages can train a model, whether they can be cited in an AI answer, or whether a user’s assistant can open them on request.

For a Gulf business with an Arabic and an English site, the robots.txt rules you set for these AI crawlers shape where your content appears in ChatGPT, Gemini, Claude and Perplexity. This guide shows you how to tell the bots apart, decide what to allow, and write rules that do what you intend.

How we researched this: on 17 October 2026 we read the official documentation for each of the AI crawlers below, published by Google, OpenAI, Anthropic, Perplexity and Common Crawl, and the internet standard that defines robots.txt. Every claim about a named bot below links to that operator’s own page. The decisions and examples come from the robots.txt and log reviews we run on bilingual Gulf sites.

What are AI crawlers, and why do they matter for Gulf websites?

AI crawlers are programs that fetch web pages for an AI company rather than for a classic search index. Some collect pages to train language models. Some build an index that an AI search product cites. Others fetch a single page only because a person asked an assistant to read it. Treating all AI crawlers as one group is the most common mistake we see.

The distinction matters more in the Gulf than in most markets. Many sites here publish the same offer twice, in Arabic and English. The Arabic version often has fewer competing pages in AI answers. If your rules turn away the wrong AI crawlers, you can disappear from AI search answers in Arabic while gaining nothing in protection. If they allow everything by default, regulated sectors such as banking, healthcare and government services may be sharing more than their legal teams expect.

There is also a practical shift in how people search. Our guide to AI Overviews and Arabic search in the GCC shows how answer-style results now sit above the classic links for many questions, and Google AI Mode in the Gulf takes that further. Your robots.txt is the first layer that decides whether any of these systems can see you at all: it is where you tell AI crawlers what they may read.

Timing matters too. AI crawlers need time to pick up changes: OpenAI and Perplexity both say robots.txt updates take about a day to reach their systems. Make changes well before seasonal campaigns such as Ramadan, White Friday or National Day sales, not on launch day.

The AI crawlers that visit Gulf sites, compared

Each operator now runs several AI crawlers with different jobs, and each one reads its own line in robots.txt. The table below lists the ones we see most often in Gulf server logs, with the effect of blocking each, as documented by the operator on the date we checked.

Sources: Google’s list of common crawlers, OpenAI’s crawler overview, Anthropic’s crawler help page, Perplexity’s bot guide and Common Crawl’s CCBot page, all checked on 17 October 2026.

Three jobs: training, search and user requests

Read the list of AI crawlers again by job rather than by company and the pattern is clear. Training bots (GPTBot, ClaudeBot, and the Google-Extended control) decide whether your words can shape future models. Search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot, and Googlebot itself) decide whether you can be cited and linked. User fetchers (ChatGPT-User, Claude-User, Perplexity-User) act for one person at one moment.

This split between AI crawlers gives you a choice most site owners did not have two years ago. You can refuse model training and still be eligible for citation in AI search answers, which is the position many Gulf publishers and brands now take.

Does Google-Extended affect your rankings or AI Overviews?

No. Google states that Google-Extended “does not impact a site’s inclusion in Google Search nor is it used as a ranking signal in Google Search”. It is a control token that governs whether content Google already crawls may be used to train future Gemini models and to ground answers in Gemini Apps and Vertex AI. Blocking it leaves your Google rankings untouched. In short, Google-Extended sits with the training AI crawlers in your policy, while Googlebot sits with search.

AI Overviews and AI Mode are part of Google Search, so they follow Googlebot, not Google-Extended. Google’s page on AI features and your website (last updated 10 December 2025) says there are no additional requirements to appear in them, and that the controls for limiting what is shown are the same preview controls used for ordinary results: nosnippet, data-nosnippet, max-snippet and noindex.

What this means in practice

• To stay out of Gemini training and grounding, add a Google-Extended group to robots.txt. Your Google traffic does not change.
• To limit how much of a page AI Overviews can quote, use max-snippet or data-nosnippet on the passages you want to protect. The page stays indexed.
• Never block Googlebot to avoid AI Overviews. It removes the page from Google Search entirely, including the classic blue links.
• Treat Google-Extended like the other training AI crawlers in your policy: decide it once, write it down, and review it with the rest.
• After any change, check the tag is visible to Googlebot with Search Console’s URL Inspection tool. Google notes that recrawling can take from several days to several months.

Should you block GPTBot? A decision framework for Gulf businesses

Whether to block GPTBot depends on what your content is for. If your pages exist to win customers, being read by AI crawlers usually helps more than it hurts. If your content is your product, such as paid research, a news archive or a proprietary dataset, training use may compete with you directly. Decide per site, not per trend.

OpenAI is explicit that the two jobs are separate: disallowing GPTBot signals that content should not be used for training foundation models, while sites “opted out of OAI-SearchBot will not be shown in ChatGPT search answers”. So you can block GPTBot and still allow OAI-SearchBot, and keep your ChatGPT search visibility. That separation is what lets you block GPTBot for legal or commercial reasons without losing citations in ChatGPT answers.

The framework is ours, built from the documented behaviour above. It is not legal advice: in the UAE and Saudi Arabia, data protection and sector rules may shape the training decision for some organisations, so involve your legal team before you block GPTBot or allow it on regulated content.

How robots.txt rules for AI crawlers actually work

robots.txt is defined by RFC 9309, the Robots Exclusion Protocol. Three rules from that standard explain almost every surprise we find in audits of AI bots in robots.txt files.

1. A bot follows only the group that names it. Crawlers match their own token case-insensitively. If a group names GPTBot, GPTBot follows that group and ignores the rules under User-agent: *.
2. The star group is a fallback. A bot with no named group follows the * group. If you write Disallow: / under the star group to stop scrapers, you also stop every one of the AI crawlers you did not name, search bots included.
3. Several groups for one bot are merged. If two groups name the same token, their rules combine, which can produce a block nobody intended.

The same standard is blunt about one more point: robots.txt rules “are not a form of access authorization”. Well-behaved AI crawlers honour them, but they protect nothing that is truly private. Anything sensitive needs a login, not a Disallow line.

A starting template for a bilingual Gulf site

The example below is a hypothetical robots.txt for a site that wants citation in AI search answers but no training use. It keeps classic search open and names each of the AI crawlers explicitly, so nothing depends on the star group.

User-agent: GPTBot

Disallow: /

User-agent: ClaudeBot

Disallow: /

User-agent: Google-Extended

Disallow: /

User-agent: CCBot

Disallow: /

User-agent: OAI-SearchBot

User-agent: Claude-SearchBot

User-agent: PerplexityBot

Allow: /

User-agent: *

Disallow: /search

Disallow: /checkout

Note what this does not cover. User fetchers such as ChatGPT-User and Perplexity-User may ignore robots.txt because a person asked for the page. If a page must never reach an assistant, it must not be public.

Worked example: a bilingual Dubai property portal

This is a hypothetical example built from patterns we see. A property portal in Dubai runs English pages at the root and Arabic pages under /ar/. Its developer copied robots.txt from a staging server, where User-agent: * carried Disallow: / for months. The staging file reached the live site during a redesign.

What the symptoms looked like

• Google traffic was stable, because a separate Googlebot group allowed everything.
• The brand had no ChatGPT search visibility for Dubai rentals, in Arabic or English, and never appeared in Perplexity either, while smaller rivals were cited.
• Server logs showed OAI-SearchBot and PerplexityBot requesting robots.txt and nothing else.

What the fix involved

The team named all the AI crawlers explicitly, allowed the three search bots, blocked the two training bots on paid market reports, and reduced the star group to two private paths. OpenAI and Perplexity both say robots.txt changes take about 24 hours to reach their systems, so the team waited a day before checking logs again. Requests for listing pages from both search bots appeared within that window.

The lesson travels well: in AI search, the star group is where hidden blocks live. Name the bots you care about.

How to audit AI crawlers on your site in seven steps

Run this audit of AI crawlers once now and again whenever the site is redesigned, migrated or moved behind a new CDN. It takes about an hour on a typical bilingual site.

1. Read the live robots.txt at your root domain and every subdomain. Each host has its own file: a rule on www does not cover shop.
2. List every group of AI bots in robots.txt and write down which AI crawlers each one names. Flag any bot covered only by the star group.
3. Check for merged or conflicting groups for the same token, the merge behaviour RFC 9309 describes.
4. Test the firewall and CDN. Many security settings block AI bots before robots.txt is even read. Look for rules labelled as bot management or AI blocking.
5. Pull 30 days of server logs and count requests from each of the AI crawlers, split by /ar/ and English paths. A search bot that fetches only robots.txt is being turned away.
6. Confirm the bot is genuine where the operator publishes IP addresses. User agent strings are easy to fake, and fake bots inflate the counts.
7. Record the decision for each bot in a one-page policy with an owner and a review date, so the next redesign does not undo it.

The AI search readiness checklist we are preparing turns these steps, and the page-level checks that follow, into a shareable sheet.

What we see in audits of AI bots in robots.txt

Across the Gulf sites we review, the same five problems account for most of the visibility lost to misconfigured AI crawlers. None is exotic, and each one can be found in an afternoon.

• Star group blocks. Disallow: / under User-agent: * left over from staging, catching every AI search bot that is not named.
• Firewall rules nobody owns. A one-click setting to block AI bots, switched on during an incident and never revisited, which also blocks OAI-SearchBot and PerplexityBot.
• Blocking Googlebot to escape AI Overviews. This removes the page from Google Search entirely. Use max-snippet or data-nosnippet instead.
• Arabic paths forgotten. Rules written for /en/ paths only, so the Arabic site is treated differently without anyone deciding it.
• robots.txt errors. A file that times out or returns a server error during traffic peaks, often in Ramadan campaigns, which leaves crawlers guessing.

Beyond robots.txt: making pages worth citing

Allowing AI crawlers in is only the door. Whether an AI answer quotes you depends on the page itself, and Google’s own guidance for AI features points back to the basics: content in text, clear internal links, and structured data that matches what is visible on the page.

Opening the door to AI crawlers without fixing the page means they read content that is hard to quote. In our reviews, the Arabic version of a page is more often the weak one: thinner copy, missing headings and key facts left inside images. Fixing that is usually worth more than any further robots.txt change.

Page-level signals that help citation

• Answer the question in the first two sentences under a heading phrased the way people ask it, in both Arabic and English.
• Keep key facts in HTML text, not inside images or scripts that a crawler may not render.
• Add structured data that matches visible content; our guide to schema markup for Arabic content covers what works for Arabic pages.
• Keep pages fast on mobile, since slow responses reduce how much any crawler fetches; see Core Web Vitals for mobile-first GCC audiences.
• For conversational Arabic questions, our guide to Arabic voice search and conversational queries shows how to phrase answers people actually ask aloud.

How to measure whether your AI crawler settings work

Measure the result, not the rule. A correct robots.txt is only useful if the right AI crawlers visit and your pages begin to appear in answers. Track three layers, monthly, with Arabic and English reported separately. Citations and referrals are also the clearest view you have of your ChatGPT search visibility.

Our article on measuring AI search visibility sets out how to build that citation check into a repeatable report. Track changes over the same weeks each year, because Ramadan and Eid shift demand sharply in every GCC market.

Your next step

Open your live robots.txt today and answer one question for each of the AI crawlers in the comparison table: is it named, and does the rule match what your business wants? If any AI search bot is covered only by the star group, fix that first, because it is the cheapest visibility you can recover.

Then write the one-page policy, agree it with legal where your sector requires it, and set a review date for the next quarter. For the wider picture of how AI answers are changing search in the region, start at our AI search hub, follow the AI Overviews coverage, and see how these changes play out in the United Arab Emirates.