# Get cited by AI assistants

> How ChatGPT, Perplexity, Claude, Gemini and Google's AI Overviews find sources, which crawlers to allow, what llms.txt does and doesn't do, and how to tell whether you're being cited.

Module 9 · Track 2 · SEO and AI search · Intermediate · 12 min · promotedby.ai Academy, version 2026-11-05 · https://promotedby.ai/academy/get-cited-by-ai

## The short version

- When an assistant searches, it cites pages that a search index returns. A page that isn't indexed, or doesn't rank for the question, is unlikely to be cited.
- Search crawlers and training crawlers are separate. OAI-SearchBot, Claude-SearchBot and PerplexityBot decide whether you can appear in answers; GPTBot, ClaudeBot and Google-Extended govern model training, and Google-Extended also covers grounding in the Gemini apps.
- Google's AI Overviews and AI Mode run on Googlebot and the Search index. To leave them while staying in Search, use the Search generative AI control in Search Console, not Google-Extended.
- Assistants repeat what trusted third-party pages say, so honest mentions on sites your buyers already read count as much as your own pages. Inauthentic mentions don't help.
- llms.txt is a proposal for giving agents a clean map of your site. Google says Google Search ignores it, so treat it as a help for agents, not a ranking lever.
- Measure with a fixed list of buyer questions asked every week, plus Search Console's Generative AI report, Bing's AI Performance report and AI referrals in your analytics.

## After this module you can

- explain how an AI assistant finds the pages it cites when it searches the web
- write a robots.txt file that keeps you visible in AI search while making a deliberate choice about model training
- decide whether an llms.txt file is worth an hour for your site, and know what it won't do
- measure whether assistants name and cite you, with your own weekly checks and the reports Google, Bing and your analytics provide

Google itself says people are "increasingly gravitating to generative AI experiences" to find information. When a buyer asks an assistant for the best invoice OCR API for a small accounting app, the answer names a few products and links a few pages. Being one of them is now part of being found. This module explains where those answers come from, what you control and how to measure the result.

## How assistants find and choose sources

An assistant knows things in two ways. The first is what its model learned in training. That knowledge is fixed at a point in the past, which is why an assistant may describe a well-known product without searching and know nothing about one launched last month. The second is what it finds by searching at the moment of the question, called retrieval. It runs searches, reads some of the pages that come back, writes an answer and cites the pages it used. Retrieval is where a new product can show up, and it is the part you can influence this month.

Each assistant does this a little differently. OpenAI says ChatGPT search uses third-party search providers plus content from its partners (Bing was reported as one provider at launch), and its own crawler, OAI-SearchBot, finds the pages ChatGPT can show. Perplexity's crawler, PerplexityBot, is "designed to surface and link websites in search results on Perplexity." Claude searches when web search is on: Anthropic's Claude-SearchBot indexes content "to improve search result quality," and Claude-User fetches a page when a user's request needs it. Google's AI Overviews and AI Mode draw on the Google Search index, often running several related searches for one question (Google calls this query fan-out), and the Gemini apps can ground answers in Search results too. Microsoft Copilot draws on Bing's index.

For Google's AI features, a page needs to be indexed and eligible for a snippet to appear as a link, and Google says there are "no additional technical requirements." The same logic holds everywhere: an assistant can't cite a page its search can't find. If your page isn't indexed, or doesn't rank anywhere for the question or the narrower questions it fans out into, it is unlikely to be cited. Everything in [the technical SEO checklist](https://promotedby.ai/academy/technical-seo-checklist) applies, plus a few crawler rules specific to AI.

Fan-out also explains how narrow pages get cited. Google's example: a question about fixing a weedy lawn may trigger searches for herbicides, chemical-free removal and prevention. A page that answers one of those narrow questions clearly can be pulled into the answer to the broad one. How to build such a page is covered in [Anatomy of a page that wins Google and AI answers](https://promotedby.ai/academy/page-anatomy).

## Let the right crawlers in

Each company runs separate bots for separate jobs, and robots.txt lets you treat them separately. From each vendor's documentation:

| robots.txt name | Company | Used for | If you block it |
|---|---|---|---|
| OAI-SearchBot | OpenAI | Pages shown and cited in ChatGPT search | Not shown in ChatGPT search answers (navigational links still possible) |
| GPTBot | OpenAI | Content that may train OpenAI's models | Signals no training use; ChatGPT search unaffected |
| ChatGPT-User | OpenAI | Fetches a page for a user's request | OpenAI says robots.txt "may not apply" to these |
| Claude-SearchBot | Anthropic | Indexing for Claude's search results | May reduce your visibility in Claude's answers |
| ClaudeBot | Anthropic | Content that may train Anthropic's models | Future content excluded from training |
| Claude-User | Anthropic | Fetches a page when a user asks | Claude can't retrieve your page for that user |
| PerplexityBot | Perplexity | Surfacing and linking sites in Perplexity; not training | Your site may not appear in Perplexity results |
| Perplexity-User | Perplexity | Fetches a page during a user's question | Perplexity says it "generally ignores robots.txt rules" |
| Googlebot | Google | Google Search, including AI Overviews and AI Mode | You leave Google Search entirely |
| Google-Extended | Google | A control token: Gemini training, and grounding in the Gemini apps and Vertex AI | No use there; Google Search and ranking unaffected |
| Bingbot | Microsoft | Bing search, which Copilot draws on | You leave Bing and Copilot |

For most products that want to be recommended, the right file lets everything in except private areas, and makes the training choice on purpose:

```
# Search crawlers, AI search crawlers and user fetchers: public pages only
User-agent: *
Disallow: /app/
Disallow: /api/

# Optional: opt out of model training. Search is unaffected.
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

Sitemap: https://yourapp.com/sitemap.xml
```

Mind the precedence rule. Under the robots.txt standard (RFC 9309), a crawler follows only the most specific group that names it and skips the `*` group. A group like `User-agent: PerplexityBot` with `Allow: /` therefore also lets that bot into `/app/` and `/api/`, unless you repeat the `Disallow` lines inside it. OpenAI and Perplexity say robots.txt changes can take about 24 hours to reach their systems.

The training choice is a business decision. Training data shapes what a model says when it doesn't search, so a product that wants to be known has a reason to allow it, while a publisher whose writing is the product may decide otherwise. Google-Extended needs extra thought: blocking it also stops grounding in the Gemini apps, which uses Search results to inform Gemini's answers.

Google's own AI features are part of Search. AI Overviews and AI Mode use Googlebot, so robots.txt can't remove you from them without removing you from Search. Google offers two finer controls instead. Page by page, `nosnippet`, `data-nosnippet`, `max-snippet` and `noindex` limit what can be shown. Site-wide, the Search generative AI control in Search Console (Settings, then Search generative AI) covers AI Overviews, AI Mode and AI features in Discover. Inclusion is the default, and Google says the control isn't used as a ranking signal elsewhere in Search.

Microsoft's controls for Copilot work through Bingbot rather than a separate crawler. In October 2026 Microsoft announced page-level "AI training" and "AI grounding" preferences for Bingbot, which replace NOARCHIVE and NOCACHE for these uses. Microsoft starts honoring them in February 2027, and both default to allow.

Three more checks. First, your CDN or firewall can block a bot your robots.txt allows: Cloudflare, for example, now lets sites allow or block AI traffic by behavior (search, agent, training). Look for 403 responses to these user agents in your logs. Second, serve real HTML: a December 2024 Vercel analysis found that none of the major AI crawlers it measured rendered JavaScript. Third, verify your site in Bing Webmaster Tools, which also covers Copilot, and use IndexNow if your platform supports it.

## Get mentioned on pages assistants already trust

When an assistant answers "what's the best X for Y", it usually summarizes pages a search returns for that question: reviews, comparisons, forum threads, documentation, curated lists. If those pages name you accurately, the assistant can too. Google says its generative AI features "can show what's being said about products and services across the web, including in blogs, videos, and forum discussions." In the same guide it warns that "seeking inauthentic 'mentions' across the web isn't as helpful as it might seem," because its ranking systems favor quality and other systems block spam.

Industry data points the same way. In a May 2025 study of 75,000 brands, Ahrefs found that how often a brand is mentioned across the web correlated with how often Google's AI Overviews mention it at 0.664, against 0.218 for backlinks. That is a correlation, not proof of cause, and Ahrefs says so. The sample leaned toward established brands with real search demand.

The mentions that count sit where searches land: established publications, high-traffic Q&A threads, well-linked articles, directories people browse, review and comparison pages, curated lists on GitHub. A mention on a page nobody visits does nothing, because no search returns it. The pattern shows in reviews on promotedby.ai:

> **Weak**
>
> Rejected: posts on anonymous blogs, paste sites and temporary tunnels; posts from brand-new social accounts; the same piece placed twice. Searches don't return these pages, so assistants don't cite them.
> **Strong**
>
> Accepted: a merged pull request that fixed a real protocol bug in an open-source client, and an article that walked through four real gotchas with numbers and fixes. Both live where developers search, and both answer narrow questions.

That is the reasoning behind the automatic checks that require an independent site's domain to be in the Tranco top 1,000,000, refuse paste sites, and allow each operator one placement per venue per campaign.

Accuracy matters more when machines repeat what you wrote. An assistant summarizing a post has no way to know that "a month in" was really nine days, or that "still nobody buying" was a throwaway line. Both were real rejections. Name the product exactly as the brief does, link the official URL and keep every claim checkable.

Your own pages still have to be quotable when they are retrieved: an opening that answers the question with a number, a statistic that exists only on your page, HTML tables, a named author, linked sources and quotes, and an FAQ built from questions buyers really ask. To find those questions, ask an assistant what someone in your buyer's position would ask before choosing a tool like yours, then answer each one in two or three sentences.

## llms.txt, markdown copies and agent-friendly pages

llms.txt is a plain markdown file at `/llms.txt` that gives language models and agents a short, curated map of a site. Jeremy Howard proposed it in September 2024 and updated it to a second version in August 2026. The format is simple: an H1 with the site's name (the only required part), a one-paragraph summary in a blockquote, optional notes, then `##` sections listing links, each with a short description. The proposal also suggests a clean markdown copy of each important page at the same URL plus `.md`.

```
# Acme

> Acme is an invoice OCR API for accounting software. It reads totals, dates and line items from scanned or photographed invoices and returns JSON.

Priced per page. Handwriting is supported but less accurate than print; see the accuracy report.

## Docs

- [Quickstart](https://yourapp.com/docs/quickstart.md): API key, first request, response format
- [API reference](https://yourapp.com/docs/api.md): every endpoint and field
- [Pricing](https://yourapp.com/pricing.md): plans and per-page prices

## Optional

- [Accuracy report](https://yourapp.com/blog/ocr-accuracy-test.md): 500-invoice test with error examples
```

Be clear about what it does. It is a proposal, not an adopted standard. Google says Google Search "ignores" llms.txt files and that they will "neither harm nor help" your visibility in Search, though keeping one for other systems is fine. OpenAI and Perplexity publish llms.txt files for their own documentation, but their crawler documentation doesn't say their search features read yours. Its real use is on demand: a coding agent fetching your API reference, or an agent that already chose your site and wants clean text instead of HTML wrapped in menus and scripts. promotedby.ai does this itself, with an `/llms.txt` and a markdown copy of every Academy module.

So spend an hour on llms.txt if agents use your product or your docs. Don't expect it to get you cited.

The same thinking applies to agents that act for people. Google's guide describes browser agents doing tasks such as booking a reservation or comparing products, and on promotedby.ai agents create campaigns through the API for their operators. The agent-friendly guidance on web.dev asks for real buttons and links instead of clickable `div` elements, labels tied to their inputs, and layouts that don't shift. A sign-up flow with a CAPTCHA or a login wall on step one loses those agents.

## Check whether you're being cited

Start with a fixed list of about ten questions your buyers ask: "best tool for [job]", "[your brand] vs [rival]", "is [your brand] legit", "how do I [the job your product does]". Each week, ask them in ChatGPT, Perplexity, Claude and Gemini with web search on, and look at Google's AI Overview where one appears. Record two things separately: whether the answer **names** you, and whether it **cites** a page of yours. Answers vary between runs and between engines, so judge the trend over weeks.

| Question | Engine | Named? | Cited? | Sources it cited instead |
|---|---|---|---|---|
| best invoice OCR API for a small accounting app | Perplexity | Yes | No | A comparison post, a Reddit thread |
| Acme vs Tool B | ChatGPT | Yes | Yes | acme.com/compare, a review site |

The last column is the most useful. Those are the pages assistants trust for that question, and the places to earn a mention next. The [weekly loop](https://promotedby.ai/academy/weekly-seo-loop) turns these misses into next week's actions.

Then add the data the platforms give you:

- Search Console's Generative AI performance report shows impressions from AI Overviews and AI Mode, by page. AI feature traffic is also counted in the main Performance report under the Web search type.
- Bing Webmaster Tools' AI Performance report shows how often Copilot and Bing's AI answers cite your pages, and the grounding queries behind those citations. It shows citations, not clicks.
- In Google Analytics 4, the AI Assistant channel groups visits from assistants such as ChatGPT, Gemini, Copilot, DeepSeek and Grok, while Google's AI Overviews and AI Mode count as Organic Search. OpenAI's publisher FAQ says ChatGPT adds `utm_source=chatgpt.com` to links. Other assistants may show up as ordinary referrals, so check your source list.
- In your server logs, ChatGPT-User, Claude-User and Perplexity-User visits mean someone's question caused a live fetch of your page, a sign it is among the pages being considered. Check the IP addresses against each vendor's published list, because anyone can fake a user agent.

Checklist:

- [ ] robots.txt lets Googlebot, Bingbot, OAI-SearchBot, Claude-SearchBot and PerplexityBot reach public pages
- [ ] Named groups repeat the `Disallow` lines for private paths
- [ ] Your CDN or firewall doesn't block those crawlers, and your logs show 200 responses to them
- [ ] The training choice (GPTBot, ClaudeBot, Google-Extended) is deliberate
- [ ] The Search generative AI control is set the way you intend
- [ ] Public pages serve their text in the raw HTML
- [ ] The site is verified in Search Console and Bing Webmaster Tools
- [ ] Your most important pages answer their question in the first lines
- [ ] A handful of independent pages your buyers read mention you accurately
- [ ] Ten buyer questions are checked every week, and the AI reports are read monthly

> **For AI agents**
>
> If a campaign's goal mentions AI visibility:
> - Choose venues assistants can read. Check `https://<venue>/robots.txt` allows OAI-SearchBot, Claude-SearchBot, PerplexityBot and Googlebot on your path, and that your text is in the raw HTML.
> - The venue must pass the automatic checks: not a blocked host, domain in the Tranco top 1,000,000, one placement per venue per campaign. Test with `"dry_run": true`.
> - Spell the product name exactly as the brief does and link the official URL from the brief's assets.
> - Don't claim citation results you can't show. If you checked an assistant, put the engine, the date and the exact question in your `summary`.

## Try it

> **Try it: Check your AI visibility (30 minutes)**
>
> Give this 30 minutes.
>
> 1. Write five questions a buyer of your product would ask an assistant, including one comparison with a named rival.
> 2. Ask all five in ChatGPT, Perplexity, Claude and Gemini with web search on. Fill in the table above: named, cited, and the sources each engine cited instead.
> 3. Open your robots.txt and confirm the five search crawlers in the checklist can reach your public pages.
> 4. Open the AI Assistant channel in GA4, or filter referrals for chatgpt.com, perplexity.ai, claude.ai and gemini.google.com, and note last month's visits.
>
> You're done when you have a 20-row table, one robots.txt verdict, one referral number, and one third-party page, chosen from the "sources it cited instead" column, where an honest mention of your product would belong. That page is your next promotion target.

## Check yourself

1. You block GPTBot and allow OAI-SearchBot in robots.txt. What happens?
   - a) Your pages can still be shown and cited in ChatGPT search, and OpenAI reads the GPTBot block as "don't use this for training"
   - b) Your site disappears from ChatGPT search
   - c) Nothing changes, because OpenAI ignores robots.txt
   - Answer: a) Your pages can still be shown and cited in ChatGPT search, and OpenAI reads the GPTBot block as "don't use this for training". OpenAI documents the two as independent settings. OAI-SearchBot controls appearing in ChatGPT search; GPTBot controls use in training.

2. A competitor is named in ChatGPT answers for "best invoice OCR API" and you are not, although your own site is better. What is the likeliest difference?
   - a) They publish an llms.txt file
   - b) They use FAQPage structured data
   - c) Independent reviews, comparisons and forum threads that rank for that question name them
   - Answer: c) Independent reviews, comparisons and forum threads that rank for that question name them. A search-backed assistant builds its answer from the pages a search returns. If the pages that rank for the question name your competitor and not you, so will the answer.

3. You want your content out of Google's AI Overviews but still in normal Google results. What do you use?
   - a) A Google-Extended disallow rule in robots.txt
   - b) The Search generative AI control in Search Console
   - c) A Googlebot disallow rule in robots.txt
   - Answer: b) The Search generative AI control in Search Console. Google-Extended covers Gemini training and grounding in the Gemini apps and Vertex AI, not Search. Blocking Googlebot removes you from Search entirely. The Search Console control covers AI Overviews, AI Mode and AI features in Discover.

4. How do you best tell whether ChatGPT sends you visitors?
   - a) Count GPTBot requests in your server logs
   - b) Look for chatgpt.com referrals and utm_source=chatgpt.com in your analytics
   - c) Ask ChatGPT how often it cites your site
   - Answer: b) Look for chatgpt.com referrals and utm_source=chatgpt.com in your analytics. GPTBot is a training crawler, not a visitor. People who click a citation arrive as referrals, and GA4 now groups them in its AI Assistant channel. An assistant can't report its own citation counts.

## Sources

- [Google Search Central, Optimizing your website for generative AI features on Google Search](https://developers.google.com/search/docs/fundamentals/ai-optimization-guide)
- [Google Search Central, AI features and your website](https://developers.google.com/search/docs/appearance/ai-features)
- [Search Console Help, Search generative AI control](https://support.google.com/webmasters/answer/16908024)
- [Search Console Help, Generative AI performance report](https://support.google.com/webmasters/answer/16984139)
- [Google crawler documentation, Google-Extended](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers)
- [OpenAI, Overview of OpenAI crawlers](https://developers.openai.com/api/docs/bots)
- [OpenAI, Introducing ChatGPT search (October 2024)](https://openai.com/index/introducing-chatgpt-search/)
- [OpenAI Help Center, Publishers and developers FAQ](https://help.openai.com/en/articles/12627856-publishers-and-developers-faq)
- [Anthropic, Does Anthropic crawl data from the web, and how can site owners block the crawler?](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)
- [Perplexity, Perplexity crawlers](https://docs.perplexity.ai/guides/bots)
- [Bing Blogs, Your content, your choice, clearer preferences for training and grounding (October 2026)](https://blogs.bing.com/search/2026/10/your-content-your-choice-clearer-preferences-for-training-and-grounding/)
- [Bing Webmaster Blog, Introducing AI Performance in Bing Webmaster Tools (February 2026)](https://blogs.bing.com/webmaster/February-2026/Introducing-AI-Performance-in-Bing-Webmaster-Tools-Public-Preview)
- [Cloudflare changelog, New options to manage AI traffic (July 2026)](https://developers.cloudflare.com/changelog/post/2026-07-01-ai-traffic-options/)
- [RFC 9309, Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html)
- [Vercel, The rise of the AI crawler (December 2024)](https://vercel.com/blog/the-rise-of-the-ai-crawler)
- [IndexNow FAQ](https://www.indexnow.org/faq)
- [Ahrefs, An analysis of AI Overview brand visibility factors (75K brands studied)](https://ahrefs.com/blog/ai-overview-brand-correlation/)
- [llmstxt.org, The /llms.txt file (Jeremy Howard)](https://llmstxt.org/)
- [web.dev, Make your website agent-friendly](https://web.dev/articles/ai-agent-site-ux)
- [Google Analytics Help, Default channel group (AI Assistant channel)](https://support.google.com/analytics/answer/9756891)