# The technical SEO checklist
> The checks that decide whether Google can find, render and index your site, explained for founders who built it with an AI coding tool.
Module 7 · Track 2 · SEO and AI search · Foundation · 13 min · promotedby.ai Academy, version 2026-11-05 · https://promotedby.ai/academy/technical-seo-checklist
## The short version
- If a page's text and links are not in the HTML your server sends, some crawlers never see them. Google renders JavaScript in a later step, and the major AI crawlers Vercel measured did not render it at all.
- robots.txt controls crawling, not indexing. To keep a page out of Google, use noindex and leave the page crawlable so Google can read the tag.
- Every public page needs its own title, a self-referencing canonical URL and a correct status code. A "not found" screen served with status 200 is a soft 404.
- Google's Core Web Vitals targets are LCP within 2.5 seconds, INP of 200 milliseconds or less and CLS of 0.1 or less, measured at the 75th percentile of real visits.
- Structured data must describe what is visible on the page. Google no longer shows HowTo or FAQ rich results, so add markup for accuracy, not decoration.
- Most of this list is a day's work. Content quality, a real identity and links from sites people read take ongoing effort.
## After this module you can
- check whether Google can crawl, render and index any page on your site, and read the answer in Search Console
- write a robots.txt file, a sitemap and canonical tags that do what you intend
- spot the problems AI-built sites usually ship with, such as client-only rendering, soft 404s and duplicate preview domains, and fix them
- read your Core Web Vitals against Google's published thresholds and judge how much they matter
Before anyone can find your page, a search engine has to **crawl** it (download it), **render** it (run it like a browser would, if it needs JavaScript) and **index** it (store it so it can be shown for a search). Ranking only starts after that. If any step fails, the best writing on your site earns nothing, because nobody is shown it.
This module covers those basics in the order you should check them, for a site built with an AI coding tool such as Lovable, Bolt, v0, Cursor or Replit. Those tools produce good-looking sites fast, but they can skip the parts no visitor sees, like the HTML that crawlers read. Everything below follows Google Search Central documentation, listed in the sources.
## How Google reads a page, and where AI-built sites go wrong
Googlebot first fetches the HTML your server sends. If the page is a JavaScript app, Google queues it, runs the JavaScript in a headless Chromium browser later, and indexes what appears. Google's own advice is that server-side rendering or pre-rendering "is still a great idea because it makes your website faster for users and crawlers, and not all bots can run JavaScript."
Other crawlers are less patient. In a December 2024 analysis of its network traffic, Vercel found that none of the major AI crawlers it measured rendered JavaScript, including OpenAI's, Anthropic's ClaudeBot and PerplexityBot. If your pricing table only exists after JavaScript runs, they see an empty page.
Many AI-built sites are single-page apps of exactly this kind: one `index.html` containing `` and a JavaScript bundle. Every URL returns the same HTML, the same title and no text. You can check yours in ten seconds:
```
curl -s https://yourapp.com/pricing | grep -i -E "|<h1"
```
If every page shows the same title and no `<h1>`, crawlers that don't run JavaScript see nothing. "View page source" in your browser shows the same raw HTML; the Inspect panel shows the page after JavaScript ran, so it can fool you.
The fix is to send real HTML. Frameworks such as Next.js, Astro, Nuxt and SvelteKit can render pages on the server or at build time. Or pre-render just the public pages (home, pricing, docs, blog) and keep client-only rendering for the logged-in app. Two smaller rules from Google's JavaScript guide: use real paths through the History API instead of `#/pricing` fragments, and make links real `<a href="...">` elements, because Google only follows `<a>` tags with an `href`.
## Set up Search Console before anything else
Google Search Console is Google's free dashboard for site owners. It tells you, page by page, whether Google indexed your site and why not. Set it up on launch day.
1. Add your site as a Domain property (it covers http, https, www and every subdomain) and verify it with the DNS record Search Console gives you.
2. Submit your sitemap in the Sitemaps report (see below).
3. Paste a URL into URL Inspection. It shows whether the page is on Google and which canonical Google chose. "Test live URL" shows the HTML Google rendered, and "Request indexing" asks Google to crawl the page.
4. A few days later, open the Pages report. Every URL Google knows about is listed as indexed or not indexed, with a reason.
Each reason points to a fix. `URL marked 'noindex'` and `URL blocked by robots.txt` are instructions you gave, on purpose or not. `Soft 404` means the page returned status 200 but looks like an error or an empty page. `Duplicate without user-selected canonical` means Google found copies and you never said which is the original. `Crawled - currently not indexed` means Google fetched the page and chose not to index it for now. Google calls that "a common message for many sites"; a better page with more internal links to it is often the fix.
Then add the site to Bing Webmaster Tools, which can import and verify it straight from Search Console. Bing's index also serves Microsoft Copilot, and Bing supports IndexNow, a protocol for telling search engines a URL changed. Google does not take part in IndexNow. For a rough check at any time, search Google for `site:yourapp.com`: if results appear, at least part of the site is indexed.
## Tell crawlers what to fetch and which URL is the real one
### robots.txt
robots.txt is a plain text file at the root of your domain (`https://yourapp.com/robots.txt`) that tells crawlers which URLs they may fetch. Google is explicit that it "is not a mechanism for keeping a web page out of Google": a blocked URL can still be indexed, without its content, if other pages link to it. Use it to keep crawlers out of areas that don't belong in search, like the logged-in app, the API and checkout.
```
# https://yourapp.com/robots.txt
User-agent: *
Disallow: /app/
Disallow: /api/
Disallow: /checkout/
Sitemap: https://yourapp.com/sitemap.xml
```
Check for two mistakes. One is a staging rule shipped to production: `Disallow: /` under `User-agent: *` blocks the whole site. The other involves named groups. Under the robots.txt standard (RFC 9309), which Google follows, a crawler obeys only the most specific group that names it and ignores the `*` group. If you add a group for one bot, repeat the `Disallow` lines it should still respect. Which AI crawlers to allow, and what blocking each one does, is covered in [Get cited by AI assistants](https://promotedby.ai/academy/get-cited-by-ai).
### XML sitemap
A sitemap lists the URLs you want indexed. Google finds most pages through links, so a sitemap guarantees nothing, but it helps a new site get discovered and says when each page last changed. Google's rules: use full absolute URLs, list only the canonical version of each page, stay under 50,000 URLs or 50 MB per file, and keep `<lastmod>` honest, because Google only uses it when it is "consistently and verifiably" accurate. Google ignores `<priority>` and `<changefreq>`, so leave them out.
```xml
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url>
<loc>https://yourapp.com/</loc>
<lastmod>2026-10-28</lastmod>
</url>
<url>
<loc>https://yourapp.com/pricing</loc>
<lastmod>2026-10-14</lastmod>
</url>
</urlset>
```
Most frameworks have a plugin that builds it. Reference it in robots.txt and submit it in both webmaster tools.
### Canonical URLs
The same page often answers at several addresses: with and without `www`, with a trailing slash, with `?utm_source=` tracking, or on your host's preview domain as well as your own. A **canonical URL** is the one address you want search engines to show. Declare it with a tag in the `<head>` of every page, pointing at the page itself:
```html
<link rel="canonical" href="https://yourapp.com/pricing">
```
Use absolute URLs. Google treats both a canonical tag and a permanent redirect as strong signals, so the best fix for a preview domain is to redirect it to your own. Google can still pick a different canonical when your signals disagree, for example when the sitemap lists `www` URLs and the tags don't. Don't use robots.txt or noindex to choose between duplicates.
### Status codes and redirects
Google reads the status code first. 200 means "here is the page." 301 and 308 are permanent redirects, which tell Google to show the new URL; 302 and 307 are temporary, so the old URL can stay in results. Google doesn't index URLs that return 404 or 410. Server errors (5xx) slow Google's crawling, and pages that keep failing are eventually dropped.
The classic single-page-app bug is the soft 404: a mistyped URL like `/pricng` shows a friendly "not found" screen with status 200, so Google may index piles of empty pages. Return a real 404 from the server, or add `<meta name="robots" content="noindex">` to the not-found view.
Redirect straight to the final URL. Googlebot follows up to 10 hops, but Google advises "redirecting to the final destination directly", and every hop adds delay for people. A 404 for a page that is really gone is correct. Redirect it only when links or visitors still arrive and a close replacement exists.
### noindex
To keep a page out of Google (a thank-you page, internal search results), add `<meta name="robots" content="noindex">` or send an `X-Robots-Tag: noindex` header. The page must stay crawlable, because if robots.txt blocks it Google never sees the tag. Before launch, search your codebase for `noindex`: a leftover from staging is a common reason a new site never appears.
## Help Google understand each page
### Titles and meta descriptions
Every page needs its own `<title>`. Google uses it for the clickable headline in results, the title link. Its guidance: descriptive and concise, unique to the page, no keyword stuffing or boilerplate, the brand added briefly at the start or end. There is no hard character limit; Google truncates titles to fit the screen.
> **Weak**
>
> Every page: `<title>Home | Acme`
> **Strong**
>
> Home: `Invoice OCR API for accounting apps | Acme`. Pricing: `Acme pricing: from $0 to $99 a month`
The meta description is the summary under the title. Google says snippets are "primarily created from the page content itself" and uses your description when it fits the page better. Write one per page anyway, one or two sentences on what the page offers and to whom. Long ones get cut.
### Headings
Give each page one `