Get cited by AI assistants

Google itself says people are "increasingly gravitating to generative AI experiences" to find information. When a buyer asks an assistant for the best invoice OCR API for a small accounting app, the answer names a few products and links a few pages. Being one of them is now part of being found. This module explains where those answers come from, what you control and how to measure the result.

#How assistants find and choose sources

An assistant knows things in two ways. The first is what its model learned in training. That knowledge is fixed at a point in the past, which is why an assistant may describe a well-known product without searching and know nothing about one launched last month. The second is what it finds by searching at the moment of the question, called retrieval. It runs searches, reads some of the pages that come back, writes an answer and cites the pages it used. Retrieval is where a new product can show up, and it is the part you can influence this month.

Each assistant does this a little differently. OpenAI says ChatGPT search uses third-party search providers plus content from its partners (Bing was reported as one provider at launch), and its own crawler, OAI-SearchBot, finds the pages ChatGPT can show. Perplexity's crawler, PerplexityBot, is "designed to surface and link websites in search results on Perplexity." Claude searches when web search is on: Anthropic's Claude-SearchBot indexes content "to improve search result quality," and Claude-User fetches a page when a user's request needs it. Google's AI Overviews and AI Mode draw on the Google Search index, often running several related searches for one question (Google calls this query fan-out), and the Gemini apps can ground answers in Search results too. Microsoft Copilot draws on Bing's index.

For Google's AI features, a page needs to be indexed and eligible for a snippet to appear as a link, and Google says there are "no additional technical requirements." The same logic holds everywhere: an assistant can't cite a page its search can't find. If your page isn't indexed, or doesn't rank anywhere for the question or the narrower questions it fans out into, it is unlikely to be cited. Everything in the technical SEO checklist applies, plus a few crawler rules specific to AI.

Fan-out also explains how narrow pages get cited. Google's example: a question about fixing a weedy lawn may trigger searches for herbicides, chemical-free removal and prevention. A page that answers one of those narrow questions clearly can be pulled into the answer to the broad one. How to build such a page is covered in Anatomy of a page that wins Google and AI answers.

#Let the right crawlers in

Each company runs separate bots for separate jobs, and robots.txt lets you treat them separately. From each vendor's documentation:

For most products that want to be recommended, the right file lets everything in except private areas, and makes the training choice on purpose:

# Search crawlers, AI search crawlers and user fetchers: public pages only
User-agent: *
Disallow: /app/
Disallow: /api/

# Optional: opt out of model training. Search is unaffected.
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

Sitemap: https://yourapp.com/sitemap.xml

Mind the precedence rule. Under the robots.txt standard (RFC 9309), a crawler follows only the most specific group that names it and skips the * group. A group like User-agent: PerplexityBot with Allow: / therefore also lets that bot into /app/ and /api/, unless you repeat the Disallow lines inside it. OpenAI and Perplexity say robots.txt changes can take about 24 hours to reach their systems.

The training choice is a business decision. Training data shapes what a model says when it doesn't search, so a product that wants to be known has a reason to allow it, while a publisher whose writing is the product may decide otherwise. Google-Extended needs extra thought: blocking it also stops grounding in the Gemini apps, which uses Search results to inform Gemini's answers.

Google's own AI features are part of Search. AI Overviews and AI Mode use Googlebot, so robots.txt can't remove you from them without removing you from Search. Google offers two finer controls instead. Page by page, nosnippet, data-nosnippet, max-snippet and noindex limit what can be shown. Site-wide, the Search generative AI control in Search Console (Settings, then Search generative AI) covers AI Overviews, AI Mode and AI features in Discover. Inclusion is the default, and Google says the control isn't used as a ranking signal elsewhere in Search.

Microsoft's controls for Copilot work through Bingbot rather than a separate crawler. In October 2026 Microsoft announced page-level "AI training" and "AI grounding" preferences for Bingbot, which replace NOARCHIVE and NOCACHE for these uses. Microsoft starts honoring them in February 2027, and both default to allow.

Three more checks. First, your CDN or firewall can block a bot your robots.txt allows: Cloudflare, for example, now lets sites allow or block AI traffic by behavior (search, agent, training). Look for 403 responses to these user agents in your logs. Second, serve real HTML: a December 2024 Vercel analysis found that none of the major AI crawlers it measured rendered JavaScript. Third, verify your site in Bing Webmaster Tools, which also covers Copilot, and use IndexNow if your platform supports it.

#Get mentioned on pages assistants already trust

When an assistant answers "what's the best X for Y", it usually summarizes pages a search returns for that question: reviews, comparisons, forum threads, documentation, curated lists. If those pages name you accurately, the assistant can too. Google says its generative AI features "can show what's being said about products and services across the web, including in blogs, videos, and forum discussions." In the same guide it warns that "seeking inauthentic 'mentions' across the web isn't as helpful as it might seem," because its ranking systems favor quality and other systems block spam.

Industry data points the same way. In a May 2025 study of 75,000 brands, Ahrefs found that how often a brand is mentioned across the web correlated with how often Google's AI Overviews mention it at 0.664, against 0.218 for backlinks. That is a correlation, not proof of cause, and Ahrefs says so. The sample leaned toward established brands with real search demand.

The mentions that count sit where searches land: established publications, high-traffic Q&A threads, well-linked articles, directories people browse, review and comparison pages, curated lists on GitHub. A mention on a page nobody visits does nothing, because no search returns it. The pattern shows in reviews on promotedby.ai:

That is the reasoning behind the automatic checks that require an independent site's domain to be in the Tranco top 1,000,000, refuse paste sites, and allow each operator one placement per venue per campaign.

Accuracy matters more when machines repeat what you wrote. An assistant summarizing a post has no way to know that "a month in" was really nine days, or that "still nobody buying" was a throwaway line. Both were real rejections. Name the product exactly as the brief does, link the official URL and keep every claim checkable.

Your own pages still have to be quotable when they are retrieved: an opening that answers the question with a number, a statistic that exists only on your page, HTML tables, a named author, linked sources and quotes, and an FAQ built from questions buyers really ask. To find those questions, ask an assistant what someone in your buyer's position would ask before choosing a tool like yours, then answer each one in two or three sentences.

#llms.txt, markdown copies and agent-friendly pages

llms.txt is a plain markdown file at /llms.txt that gives language models and agents a short, curated map of a site. Jeremy Howard proposed it in September 2024 and updated it to a second version in August 2026. The format is simple: an H1 with the site's name (the only required part), a one-paragraph summary in a blockquote, optional notes, then ## sections listing links, each with a short description. The proposal also suggests a clean markdown copy of each important page at the same URL plus .md.

# Acme

> Acme is an invoice OCR API for accounting software. It reads totals, dates and line items from scanned or photographed invoices and returns JSON.

Priced per page. Handwriting is supported but less accurate than print; see the accuracy report.

## Docs

- [Quickstart](https://yourapp.com/docs/quickstart.md): API key, first request, response format
- [API reference](https://yourapp.com/docs/api.md): every endpoint and field
- [Pricing](https://yourapp.com/pricing.md): plans and per-page prices

## Optional

- [Accuracy report](https://yourapp.com/blog/ocr-accuracy-test.md): 500-invoice test with error examples

Be clear about what it does. It is a proposal, not an adopted standard. Google says Google Search "ignores" llms.txt files and that they will "neither harm nor help" your visibility in Search, though keeping one for other systems is fine. OpenAI and Perplexity publish llms.txt files for their own documentation, but their crawler documentation doesn't say their search features read yours. Its real use is on demand: a coding agent fetching your API reference, or an agent that already chose your site and wants clean text instead of HTML wrapped in menus and scripts. promotedby.ai does this itself, with an /llms.txt and a markdown copy of every Academy module.

So spend an hour on llms.txt if agents use your product or your docs. Don't expect it to get you cited.

The same thinking applies to agents that act for people. Google's guide describes browser agents doing tasks such as booking a reservation or comparing products, and on promotedby.ai agents create campaigns through the API for their operators. The agent-friendly guidance on web.dev asks for real buttons and links instead of clickable div elements, labels tied to their inputs, and layouts that don't shift. A sign-up flow with a CAPTCHA or a login wall on step one loses those agents.

#Check whether you're being cited

Start with a fixed list of about ten questions your buyers ask: "best tool for [job]", "[your brand] vs [rival]", "is [your brand] legit", "how do I [the job your product does]". Each week, ask them in ChatGPT, Perplexity, Claude and Gemini with web search on, and look at Google's AI Overview where one appears. Record two things separately: whether the answer names you, and whether it cites a page of yours. Answers vary between runs and between engines, so judge the trend over weeks.

The last column is the most useful. Those are the pages assistants trust for that question, and the places to earn a mention next. The weekly loop turns these misses into next week's actions.

Then add the data the platforms give you:

• Search Console's Generative AI performance report shows impressions from AI Overviews and AI Mode, by page. AI feature traffic is also counted in the main Performance report under the Web search type.
• Bing Webmaster Tools' AI Performance report shows how often Copilot and Bing's AI answers cite your pages, and the grounding queries behind those citations. It shows citations, not clicks.
• In Google Analytics 4, the AI Assistant channel groups visits from assistants such as ChatGPT, Gemini, Copilot, DeepSeek and Grok, while Google's AI Overviews and AI Mode count as Organic Search. OpenAI's publisher FAQ says ChatGPT adds utm_source=chatgpt.com to links. Other assistants may show up as ordinary referrals, so check your source list.
• In your server logs, ChatGPT-User, Claude-User and Perplexity-User visits mean someone's question caused a live fetch of your page, a sign it is among the pages being considered. Check the IP addresses against each vendor's published list, because anyone can fake a user agent.

Checklist:

#Try it