How to Verify Whether AI Crawlers Can Access Your Website

Before you invest in content for AI search, confirm that AI crawlers can access your website. If GPTBot, OAI-SearchBot, ClaudeBot or PerplexityBot cannot fetch your pages, no amount of optimization will make those pages appear in answers. Access problems are surprisingly common, and they rarely come from a deliberate decision. They come from security plugins, CDN bot protection, rate limits and old robots.txt rules. This guide focuses only on testing access, separate from the wider topic of how AI systems index or rank content.

Know Which Crawlers You Are Testing

AI companies operate different user agents for different purposes. Treat each one separately, because you may want to allow some and block others.

Company	User agent	Main purpose
OpenAI	GPTBot	Collecting data for model training
OpenAI	OAI-SearchBot	Indexing for ChatGPT search results
OpenAI	ChatGPT-User	Fetching pages when a user asks
Anthropic	ClaudeBot	Collecting data for model training
Anthropic	Claude-User, Claude-SearchBot	User fetches and search indexing
Perplexity	PerplexityBot	Indexing for Perplexity answers
Perplexity	Perplexity-User	Fetching pages for a user’s question
Google	Google-Extended	A robots.txt token controlling Gemini training use, not a separate crawler
Apple	Applebot-Extended	A token controlling AI training use of Applebot data

User agent names and purposes change from time to time, so confirm them against each company’s official documentation before finalising rules.

Step 1: Check Robots.txt Rules

Open yoursite.com/robots.txt and read it line by line. Look for:

A blanket Disallow: / under User-agent: * left over from a staging site
Specific blocks for GPTBot, ClaudeBot or PerplexityBot added by a plugin
Disallow rules covering folders that hold your most important content

Watch for Plugin-Generated Rules

SEO and security plugins sometimes write their own robots.txt or serve a virtual file that differs from the one on disk. Always test the live URL in a browser rather than reading the file in your hosting panel. If you run several subdomains, remember each one needs its own robots.txt.

Remember that a crawler follows the most specific group that matches its name. If you have a User-agent: GPTBot group, GPTBot ignores your general * rules entirely. Decide deliberately which crawlers you allow. Blocking training crawlers while allowing search and user fetchers is a common, reasonable policy.

Step 2: Check Firewall, CDN and Security Settings

Robots.txt is a request, not a gate. The real gate is your infrastructure. Review:

CDN bot management: Cloudflare and similar services offer settings that block AI crawlers by default or challenge them with JavaScript. A crawler that cannot solve a challenge sees an error or an empty page.
Web application firewall rules: custom rules that block unknown bots or specific countries.
Security plugins: WordPress plugins that rate-limit or block non-browser user agents.
Hosting limits: shared hosts sometimes throttle aggressive crawling with 429 or 503 responses.

These settings often override everything in robots.txt, so check them even if robots.txt looks perfect. If you rely on a managed host or an agency for infrastructure, ask them for the exact bot rules in writing. Access problems frequently start when a well-meaning security update is applied across many sites at once, and they can quietly undo months of AI search optimization work.

Step 3: Test Responses Directly

Request a few important URLs while identifying as each crawler and inspect the response:

curl -I -A "Mozilla/5.0 (compatible; GPTBot/1.2; +https://openai.com/gptbot)" https://www.example.com/services/

You are looking for a 200 status code, the correct content type and no redirect to a challenge page. Then fetch the full HTML and confirm the main content is present. A 403, 429 or a page containing “verify you are human” means the crawler is blocked.

Understand the Limits of User-Agent Testing

Some firewalls verify crawlers by IP address, not by user agent. A spoofed request from your laptop may be blocked even though the real crawler is allowed, or allowed even though the real crawler is blocked. Treat this test as a quick screen and confirm with server logs.

Step 4: Prove AI Crawlers Can Access Pages Using Server Logs

Logs are the strongest evidence. Filter access logs for AI user agents and check:

Are the crawlers visiting at all?
Which URLs do they request most?
What status codes do they receive?
Do requests come from IP ranges the company publishes?

OpenAI and Perplexity publish IP lists for their crawlers, and other companies provide verification guidance. Matching IPs protects you from fake bots using AI user agents. Also compare crawler activity before and after any infrastructure change. A sudden drop to zero is the clearest sign of a new block, and catching it quickly can prevent the slow decline described in our AI traffic drop recovery guide. If you see no AI crawler visits for weeks, access is likely blocked somewhere upstream. Our article on how to optimize for AI crawlers and LLMs explains what to do once access is confirmed.

Step 5: Check That Content Is Readable

Access is not only about status codes. Many AI crawlers do not run JavaScript reliably. If your page content loads only after client-side rendering, the crawler may receive an almost empty shell. View the raw page source and confirm that headings, body copy and key facts are present without JavaScript. A fast, clean page also helps, so review website speed optimization if response times are slow.

A Quick Access Checklist

Robots.txt allows the crawlers you want, with no leftover blanket blocks
CDN and firewall settings do not challenge or block those crawlers
Important URLs return 200 to crawler requests
Main content is present in raw HTML
Logs show real crawler visits from published IP ranges
Changes are documented with dates

Frequently Asked Questions

Does Allowing GPTBot Mean ChatGPT Will Cite Me?

No. GPTBot relates to training data. Citations in ChatGPT search depend more on OAI-SearchBot access and the quality of your pages.

Should I Block AI Training Crawlers?

It is a business decision. Many sites block training crawlers while allowing search and user-triggered fetchers so they stay eligible for citations.

How Often Should I Re-Test Access?

After every CDN, firewall, hosting or plugin change, and at least monthly as part of routine monitoring.

Confirmed access is the foundation of every AI visibility strategy. If you need help, explore our fix AI crawler indexing service, read about building an AI-friendly website, book a technical SEO audit, see why your website is not ranking in AI search, learn about AI training data optimization, explore our LLM SEO services, or check our website maintenance services.

How to Test Robots.txt Rules Without Accidentally Blocking Search Engines

A single misplaced character in robots.txt can remove an entire website from search. That is why you should test robots.txt rules before they reach production, not after traffic falls. The risk has grown as businesses add rules for AI crawlers such as GPTBot, ClaudeBot and PerplexityBot, because each new group creates another chance to block Googlebot or Bingbot by mistake. This guide explains how directives are matched, how to test crawler-specific rules on staging, how to validate changes and how to roll them out safely.

How Robots.txt Rules Are Matched

Understanding the matching logic prevents most mistakes. The standard, formalised as RFC 9309, works like this:

Groups are chosen by user agent. A crawler looks for the group with the most specific matching User-agent line. If it finds one, it follows only that group and ignores the * group completely.
Within a group, the longest matching path wins. Allow: /blog/ beats Disallow: / for URLs under /blog/ because it is a longer, more specific match.
Wildcards are supported by major crawlers. * matches any sequence of characters and $ marks the end of a URL.
Paths are case-sensitive. /Blog/ and /blog/ are different.
Robots.txt controls crawling, not indexing. A blocked URL can still appear in results if other pages link to it. Use noindex for pages that must stay out of the index, and make sure those pages are crawlable so the noindex can be seen.

The Classic Mistake With AI Crawler Groups

Teams often add this to block a training crawler:

User-agent: GPTBot
Disallow: /

That is fine. Problems start when someone tries to apply site rules to a list of bots and accidentally includes Googlebot in a blocking group, or when shared rules such as Disallow: /checkout/ are written only in the * group. Because GPTBot now has its own group, it ignores the * rules. Every crawler with a named group must repeat the rules you want it to follow.

Step 1: Write the Intended Policy in Plain Language

Before editing the file, write a short table describing what each crawler should be able to do.

Crawler	Policy
Googlebot, Bingbot	Crawl everything except cart, account and internal search
OAI-SearchBot, PerplexityBot	Same as search engines
GPTBot, ClaudeBot	Block entirely, training opt-out
Google-Extended	Block, training opt-out only
All others	Same as search engines

This table becomes the specification your tests check against. It also makes business decisions visible, which matters if you want to stay eligible for citations in AI search while opting out of training. Our guide to optimizing for AI crawlers and LLMs explains the trade-offs.

Step 2: Test Robots.txt Rules on Staging First

Upload the new file to a staging environment that mirrors production URLs. Then test a list of representative URLs against every crawler in your policy table:

Homepage and top service or category pages
Blog posts and resource pages
Pages that must be blocked, such as cart and account
Parameter URLs and internal search results
Assets such as CSS, JavaScript and images, which Google needs to render pages

Use a robots.txt parser to check each combination. Google publishes its open-source robots.txt parser, and many SEO crawlers include a testing mode that applies a custom robots.txt to a crawl. The aim is a simple matrix showing allowed or blocked for every URL and crawler, compared against your policy table.

Keep Staging Itself Out of Search

Staging should be protected with authentication, not only robots.txt. If staging uses Disallow: /, make sure your deployment process never copies that file to production. This exact mistake causes many sudden traffic losses.

Step 3: Validate Before and After Deployment

Before release, run a final diff between the current and new files and confirm every changed line is intentional. After release:

Load the live file in a browser and confirm it matches what you tested.
Check the robots.txt report in Google Search Console for fetch errors and the version Google has cached.
Use URL Inspection on a few key pages to confirm they are not reported as blocked.
Watch server logs for 24 to 72 hours to confirm Googlebot and Bingbot keep crawling normally.
Check AI crawler activity matches the new policy.

Note that Google caches robots.txt, usually for up to a day, so effects are not always immediate.

Common Robots.txt Errors

Blocking CSS or JavaScript folders, which prevents proper rendering
Using Disallow: with an empty value, which actually allows everything
Forgetting the trailing slash difference between /blog and /blog/
Returning a 5xx error for robots.txt, which can make Google pause crawling
Serving different robots.txt content to different user agents by accident through a CDN
Assuming noindex inside robots.txt works, which Google stopped supporting

Errors like these are among the first things checked in a technical SEO audit, because the impact is large and the fix is quick.

A Safe Rollout Process

Treat robots.txt as code. Store it in version control, require review from an SEO specialist for every change, and keep a short changelog explaining why each rule exists. Add an automated check to your deployment pipeline that fails the build if production robots.txt contains Disallow: / under User-agent: * or under Googlebot. Teams that follow a proper SEO deployment process rarely suffer robots.txt accidents.

Frequently Asked Questions

How Long Does a Robots.txt Change Take Effect?

Google usually refreshes its cached copy within a day. AI crawlers vary, so allow several days before judging the effect in logs.

Can I Block AI Crawlers Without Hurting Google Rankings?

Yes, if you block them by name in their own groups and leave Googlebot rules unchanged. Google-Extended affects Gemini training use and does not affect Google Search rankings.

Should Every Subdomain Have Its Own File?

Yes. Robots.txt applies only to the host it is served from, so blog, shop and app subdomains each need their own.

Careful testing makes robots.txt a precise tool instead of a risk. For expert help, explore our fix AI crawler indexing service, strengthen your core search engine optimization, learn about AI training data optimization, build an AI-friendly website, see our AI indexing case study, review our SEO services, or contact our team.

Why JavaScript-Rendered Content May Be Missing From Third-Party Crawlers

JavaScript-rendered content is one of the most common reasons a page looks perfect to visitors yet appears almost empty to crawlers. Googlebot renders JavaScript, although with a delay. Many other crawlers, including most AI crawlers, SEO tools, social media preview bots and smaller search engines, either do not render JavaScript at all or render it inconsistently. If your main content only appears after scripts run in the browser, those systems may never see it. This guide explains why that happens, how to test it and what to change.

How Pages Get Their Content

There are three broad ways a page can deliver content:

Approach	What the crawler receives first	Risk for crawlers
Server-side rendering (SSR)	Complete HTML with content	Low
Static generation	Complete HTML built in advance	Low
Client-side rendering (CSR)	A near-empty shell plus scripts	High

With client-side rendering, the server sends a basic HTML file containing a root element and links to JavaScript bundles. The browser downloads and runs the scripts, fetches data from APIs and then builds the page. A crawler that reads only the initial HTML sees the shell, not the content.

Where Hydration Fits In

Modern frameworks such as Next.js, Nuxt and Angular can render HTML on the server and then “hydrate” it in the browser, attaching interactivity to the existing markup. That is usually crawler-friendly. Problems appear when parts of the page are deliberately skipped on the server, for example product details, reviews, pricing or tabs that load only on the client, or when hydration errors replace server content with loading states.

Why Third-Party Crawlers Miss JavaScript-Rendered Content

Rendering JavaScript at web scale is expensive. It requires a headless browser, time for scripts and API calls to finish, and handling of errors and timeouts. Google invested heavily in this. Most other crawlers prioritise speed and cost, so they fetch raw HTML and move on. AI crawlers collecting pages for search or training typically behave this way, and user-triggered fetchers working in real time have even less time to wait. Understanding how AI search works makes it clear why raw HTML matters so much.

Content can also go missing because:

Data loads only after a user interaction, such as clicking a tab or scrolling
API calls require cookies, tokens or a logged-in session
Content is injected after a long timer or animation
Bot protection blocks the API requests even though the HTML loads
Errors in a script stop rendering for non-browser user agents

How to Test What Crawlers See

Test 1: View the Raw HTML

Right-click and choose “View page source”, or fetch the page with curl. Search for a sentence from the main content. If it is not there, any crawler that does not render JavaScript will miss it.

Test 2: Compare Raw HTML With the Rendered DOM

Open developer tools and inspect the rendered DOM, or use a headless browser to save the rendered HTML. Compare word counts, headings and links between the two versions. Large differences show exactly what depends on JavaScript.

Test 3: Disable JavaScript

Turn JavaScript off in your browser and reload. This simulates a non-rendering crawler quickly and shows navigation, content and internal links that disappear.

Test 4: Use Google’s Tools for Google Only

URL Inspection in Search Console shows the rendered HTML Googlebot produced. It is useful, but it does not tell you what AI crawlers see. Pair it with raw HTML tests.

Test 5: Check Server Logs

Look at AI crawler requests in logs. If crawlers request your pages but never request the API endpoints that supply content, they are reading only the shell.

How to Fix Missing Content

Render primary content on the server. Use SSR or static generation for headings, body copy, product details, prices, FAQs and internal links.
Keep interactive enhancements on the client. Filters, carousels and calculators can remain client-side if the key information also exists in HTML.
Avoid content hidden behind interactions. Tabs and accordions are fine if their content is in the HTML on load.
Use real links. Navigation must use <a href> elements, not click handlers, so crawlers can discover pages.
Put structured data in the initial HTML. Schema injected late may be missed by non-rendering crawlers. Our structured data optimization work covers this.
Monitor hydration errors. Errors that blank sections of the page affect crawlers and users alike.

If rebuilding the front end is not possible immediately, prerendering services can serve a static HTML snapshot to crawlers. Treat this as a temporary bridge and make sure the snapshot matches what users see, so it is not considered cloaking.

A Simple Example

Imagine an online store whose product pages are built with client-side rendering. The raw HTML contains the site header, an empty product container and a script tag. Price, description, specifications and reviews arrive from an API after the page loads. Googlebot eventually renders the page and indexes the product. An AI shopping assistant, however, fetches the raw HTML, finds no product name or price, and cites a marketplace listing instead. Moving the product details into server-rendered HTML fixes both problems at once, and usually improves load speed as well.

Prioritise by Business Impact

Start with templates that drive revenue: service pages, product pages, location pages and key guides. A single template fix can expose thousands of pages. Platforms with heavy JavaScript, such as some headless commerce builds, benefit most. Teams working with custom web development can plan SSR from the start rather than retrofitting it.

Frequently Asked Questions

Does Google Still Have Trouble With JavaScript?

Google renders most JavaScript, but rendering can be delayed and errors still cause missing content. Server-rendered HTML remains the safer choice.

Do AI Crawlers Execute JavaScript?

Most do not render JavaScript reliably. Assume they read raw HTML unless you have evidence otherwise from your own logs.

Is a Single-Page Application Always Bad for SEO?

No. A single-page application with server rendering or static generation can perform very well.

Making sure crawlers see your real content is often the fastest technical win for AI visibility. For help, explore our fix AI crawler indexing service, read why your content is not showing in AI search, learn why a blog is not showing in AI results, book a technical SEO audit, improve Core Web Vitals optimization, review our web design and development services, or build an AI-friendly website.

How to Diagnose Canonical and Redirect Conflicts After a Website Migration

Canonical and redirect conflicts are among the most damaging problems that appear after a website migration, and among the least visible. Pages load normally for visitors, so nobody notices. Meanwhile, search engines receive mixed signals: a redirect says “this page moved here”, while the canonical on the new page says “the real version is somewhere else”. The result can be dropped rankings, the wrong URLs in search results and lost citations in AI answers. This is a migration-focused diagnostic, not a general SEO audit. It covers how to find these conflicts, what each pattern means and how to fix it.

Why Migrations Create Conflicts

During a migration, several systems change at once: URL structure, CMS, templates, server configuration, CDN rules and sometimes domains or protocols. Redirects are often managed by developers in server or CDN rules, while canonicals come from CMS templates or SEO plugins. When these two systems are configured separately, they can disagree. Common triggers include:

Canonical tags still pointing to old URLs copied from the previous CMS
Templates generating canonicals from a hard-coded staging or old domain
Redirect rules sending users to a page whose canonical points elsewhere
Trailing slash, uppercase or www rules applied in redirects but not in canonicals
HTTP to HTTPS redirects missing on some paths

Common Canonical and Redirect Conflicts

Pattern	What happens	Typical result
Canonical to a redirecting URL	New page canonicalises to an old URL that redirects back	Signals loop, Google may choose its own canonical
Redirect chain	Old URL → interim URL → final URL	Slower crawling and diluted signals
Redirect loop	A → B → A	Page unreachable
Canonical to a 404	Canonical points to a URL that no longer exists	Canonical ignored, possible deindexing confusion
Canonical to a non-equivalent page	Product page canonicalises to category	Product page dropped from index
Mixed protocol or host	Canonical uses http or non-www, redirect uses https and www	Duplicate versions indexed

Step 1: Gather the URL Lists

You need three lists:

Old URLs: from the pre-migration crawl, sitemap, analytics landing pages, Search Console top pages and backlink exports.
Redirect map: the intended old-to-new mapping agreed before launch.
New URLs: from a fresh crawl of the live site and the new XML sitemaps.

If you did not crawl before migrating, rebuild the old list from Search Console, analytics and archived sitemaps.

Step 2: Crawl Old URLs and Follow Redirects

Crawl the old URL list in list mode with redirect following enabled. For each URL, record the full redirect path, final status code and final URL. Flag:

Chains with more than one hop
Loops
Final status codes other than 200
Redirects to the homepage or a generic category instead of an equivalent page
Temporary redirects (302 or 307) where permanent ones (301 or 308) were intended

Compare final URLs with your redirect map. Any mismatch is either a configuration error or an undocumented decision that needs review.

Step 3: Audit Canonicals on the Destination Pages

Crawl the new site and extract the canonical tag and any canonical HTTP header for every page. Then check:

Does each page’s canonical point to itself, or to a deliberate equivalent?
Does any canonical point to an old URL, a redirecting URL or a non-200 URL?
Do canonicals use the same protocol, host, case and trailing slash as the live URL?
Is the canonical in the raw HTML, not only added by JavaScript?
Are there multiple canonical tags on one page?

Join this data with the redirect crawl. The dangerous cases are where the redirect target and the canonical disagree.

Step 4: Check How Search Engines Interpreted the Signals

In Google Search Console, use URL Inspection on a sample of affected pages and compare “User-declared canonical” with “Google-selected canonical”. When they differ, Google has decided your signals are unreliable. Review the Page indexing report for “Duplicate, Google chose different canonical than user” and “Page with redirect”. Rising counts after launch point directly at canonical and redirect conflicts.

Watch for Signals in Other Places

Canonical intent is also expressed through hreflang annotations, pagination links, Open Graph URLs and sitemaps. After a migration, any of these may still reference old URLs. Search engines combine all of these signals, so a perfect canonical tag can still be overruled if hreflang and sitemaps disagree with it. Extract them in the same crawl and check that every reference points to a final, indexable URL.

Step 5: Fix in the Right Order

Remove redirect loops first, because they make pages unreachable.
Collapse redirect chains so every old URL points directly to its final destination.
Update canonicals to self-reference the final live URL, using the same format as your redirects.
Update internal links to point straight to final URLs, so crawlers do not need redirects at all.
Update XML sitemaps to list only final, canonical, 200-status URLs.
Request recrawling of key pages and monitor the indexing reports.

Fixing internal links is often overlooked, yet it is one of the strongest consistency signals you can send. If the migration also changed navigation or templates, review whether important pages lost links entirely, and use a data-driven SEO strategy to decide which pages deserve the strongest internal support.

Document Every Decision

Keep the redirect map, the canonical rules and the list of fixes in one shared document. Future developers will otherwise repeat the same mistakes during the next redesign. Strong on-page SEO optimization keeps these signals aligned.

Why This Matters for AI Search

AI search systems that rely on search indexes inherit these problems. If Google or Bing selects the wrong canonical, AI answers may cite an outdated URL, a redirecting page or a competitor’s duplicate content. Pages that previously earned citations can disappear from answers after migration. Our AI traffic drop recovery service often begins by fixing exactly these conflicts.

Frequently Asked Questions

How Long Should Redirects Stay in Place After a Migration?

Keep them for at least a year, and permanently for URLs with backlinks or ongoing traffic.

Can a Canonical Replace a Redirect?

No. A canonical is a hint for duplicate content. When a page has moved, use a permanent redirect.

How Soon Will Rankings Recover After Fixes?

Small fixes can show results within weeks. Large sites may take several months for search engines to reprocess all URLs.

Diagnosing canonical and redirect conflicts early protects the value you built before migration. If you are planning or recovering from a migration, book a technical SEO audit, explore our international SEO services for multi-domain moves, read our AI indexing case study, review website maintenance services, see our SEO services, learn how to fix AI search visibility, or understand why your website is not ranking in AI search.

How to Test Whether Important Content Is Available in Raw HTML

Checking whether your important content is in raw HTML is one of the simplest technical tests in SEO, and one of the most revealing. Raw HTML is what the server sends before any JavaScript runs. It is what most AI crawlers, link preview bots and many SEO tools read. The rendered DOM is what a browser builds after scripts execute, and it is what users see. When the two differ, crawlers may be missing your headings, product details, prices, FAQs or internal links. This guide shows how to compare them methodically and decide what needs fixing.

Raw HTML vs Rendered DOM vs Crawlable Content

It helps to define the three things you are comparing:

Raw HTML (page source): the response body from the server. View it with “View page source” or a command such as curl https://example.com/page/.
Rendered DOM: the live document after JavaScript has run, cookies have loaded and APIs have returned data. Inspect it in developer tools or save it using a headless browser.
Crawlable primary content: the part of the page that defines what it is about, such as the main heading, body copy, key facts, structured data and links to related pages.

Your goal is simple: crawlable primary content should be present in raw HTML.

Step 1: Define What “Important Content” Means for Each Template

Different templates have different essentials. Write them down before testing so the check is objective.

Template	Content that must be in raw HTML
Service page	H1, introduction, service details, pricing approach, FAQs, contact path
Product page	Product name, price, availability, description, specifications, reviews summary
Blog post	Title, author, date, full article text, images with alt text
Location page	Business name, address, phone, opening hours, area served
Category page	Heading, intro copy, links to products or subpages

Include internal links and structured data in every list. Links help crawlers discover pages, and schema helps machines understand them. See our structured data optimization approach for what to include.

Step 2: Test Content in Raw HTML

For each template, choose three to five representative URLs. Then:

Fetch the raw HTML with curl, or view source in a browser.
Search for a distinctive sentence from each important content element.
Record whether it is present, partly present or missing.
Check that headings use proper heading tags and that links are real <a href> elements.
Confirm the canonical tag, meta robots and structured data are in the initial response.

If you prefer a visual check, disable JavaScript in your browser and reload the page. Anything that disappears is not in raw HTML.

Watch for Placeholder Text

Some pages show “Loading…”, skeleton blocks or empty containers in raw HTML. These are clear signs that content depends on client-side requests.

Step 3: Compare Against the Rendered DOM

Save the rendered DOM for the same URLs using a headless browser such as Chrome with Puppeteer or Playwright, or use a crawler with JavaScript rendering switched on. Compare:

Word count of main content
Number and text of headings
Number of internal links
Presence of structured data blocks
Key facts such as prices or phone numbers

Many SEO crawlers can crawl twice, once with rendering off and once with it on, and report differences automatically. Large gaps highlight the templates that need work first.

Step 4: Check What Real Crawlers Receive

Your own browser test is not the whole story. Real crawlers may be served different responses due to CDN rules, geolocation, A/B testing tools or bot protection. Check:

Search Console URL Inspection for Google’s rendered HTML and screenshot.
Server logs to confirm AI crawlers received a 200 response and a normal response size. Very small responses suggest a shell or a block page.
User-agent testing with curl, using crawler user agents, while remembering that IP-based rules may treat your test differently from the real crawler.

Problems here often connect with access issues covered in our guide to fixing AI crawler indexing.

Common Reasons Content Is Missing From Raw HTML

Client-side rendered frameworks without server rendering
Reviews, prices or stock loaded from third-party widgets
Tabs and accordions that fetch their content only when clicked
Infinite scroll without paginated URLs
Personalisation scripts that replace default content
Consent banners that block scripts that build the page
Lazy-loaded text, not just lazy-loaded images

How to Fix the Gaps

Move primary content into the server response through server-side rendering, static generation or prerendering of the key elements. Keep interactive features client-side, but include their essential information in HTML. For third-party widgets, ask the vendor for a server-side integration or output a static summary, such as average rating and review count, in the page itself. Our custom web development and web design and development teams usually start with the highest-traffic templates.

Why This Matters for AI Search

AI systems quote, summarise and cite text they can extract. If your best explanation sits in a tab that loads on click, a competitor’s simpler page will be chosen instead. Raw HTML availability is therefore not just a technical tidy-up. It decides whether your content can be selected at all, which is why it is a core part of AI content optimization and of our approach to why AI is not picking your content.

Make It a Routine Check

Add raw HTML tests to your release checklist. Whenever a template, framework, CMS plugin or third-party script changes, re-run the comparison on representative URLs. A five-minute test can prevent months of lost visibility.

Frequently Asked Questions

Is It a Problem if Images Load Lazily?

Not usually, as long as image elements and alt text are present in HTML and native lazy loading is used. Text should not be lazy-loaded.

Do I Need Server-Side Rendering for Every Page?

Not necessarily. Static generation or partial server rendering of the primary content is often enough.

Will Google Still Index Client-Rendered Content?

Usually, but with delays and occasional failures. Other crawlers may not index it at all.

Testing raw HTML is quick and gives a clear list of fixes. For expert help, book a technical SEO audit, read why your content is not showing in AI search, learn how to build an AI-friendly website, review our AI indexing case study, or explore how to optimize for AI crawlers and LLMs.

How to Validate Structured Data When Search Features Do Not Appear

You added schema markup, the testing tool shows green ticks, yet no stars, FAQs or product details appear in search. This is one of the most common frustrations in technical SEO. To validate structured data properly, you need to separate three different questions: is the markup technically valid, is the page eligible for the feature, and does Google choose to show it? Valid markup is only the first step. This guide sets out a practical validation workflow and explains the limits of what structured data can achieve.

Understand What Structured Data Does and Does Not Do

Structured data describes your content in a machine-readable format, usually JSON-LD using the Schema.org vocabulary. It helps search engines understand entities, such as products, organisations, events and articles, and can make pages eligible for rich results. It does not guarantee a rich result, it does not directly improve rankings, and it cannot make low-quality content appear better than it is. Google decides whether to display a rich result based on eligibility rules, page quality, query context and its own presentation choices.

Step 1: Validate Structured Data for Syntax and Errors

Start with two tools:

Rich Results Test: checks whether markup is valid for Google-supported rich result types and shows errors and warnings.
Schema Markup Validator: checks markup against the full Schema.org vocabulary, including types Google does not use for rich results.

Fix all errors first. Errors usually mean required properties are missing or values are in the wrong format, such as a price written as text with a currency symbol. Warnings are optional properties. They do not block eligibility, but adding them can improve how complete your data is.

Test the Live URL, Not Just Code

Always test the live URL. Markup that works in a code snippet can break on the live page because of plugins, caching, duplicate scripts or JavaScript that injects schema too late.

Look for Duplicate and Conflicting Markup

WordPress sites often output schema from the theme, an SEO plugin and a reviews plugin at the same time. The result can be two Product entities with different prices, or an Organization described three different ways. Choose one source for each entity type and switch the others off, so search engines receive one consistent description.

Step 2: Check Eligibility Rules

Each rich result type has its own guidelines. Common reasons valid markup is not eligible:

Feature	Common eligibility problem
Review snippets	Self-serving reviews of your own business on your own site
FAQ rich results	Google now shows them only for a limited set of well-known authoritative sites
HowTo rich results	Retired by Google, so markup no longer produces the feature
Product snippets	Missing price, availability or review data
Breadcrumbs	Markup does not match visible navigation
Local business	Information conflicts with the Google Business Profile

Google changes these rules from time to time, so check the current Search Central documentation for each type before deciding something is broken.

Step 3: Confirm the Markup Matches Visible Content

Structured data must describe content users can see on the page. Ratings that do not appear on the page, prices that differ from the displayed price, or FAQs hidden from users can lead Google to ignore the markup or apply a manual action. Compare each property in the JSON-LD with the visible page. This is also important for AI systems, which may read both the visible text and the markup and lose trust when they disagree.

Step 4: Check Search Console Reports

Search Console’s Enhancements reports list valid items, items with warnings and invalid items for each detected rich result type. Use them to:

Confirm Google has actually detected the markup across your site
Find templates where errors affect many pages
Track whether fixes are accepted after you click “Validate fix”

Also check the Performance report with the “Search appearance” filter. It shows whether rich results are being displayed and how they perform.

Step 5: Check Indexing and Page Quality

If a page is not indexed, or is indexed under a different canonical, its structured data will not be used. Use URL Inspection to confirm the page is indexed and that Google selected your canonical. Then consider page quality honestly. Thin pages, duplicate content or pages with little evidence of expertise are less likely to receive enhanced presentation, even with perfect markup. Improving the content through on-page SEO optimization often does more than adjusting schema.

Step 6: Allow Time and Monitor

After fixes, Google needs to recrawl and reprocess pages. That can take days for important pages and weeks for large sites. Keep a simple log of changes and dates, and monitor search appearance data over at least a month before drawing conclusions.

A Repeatable Validation Workflow

Test live URLs in the Rich Results Test and Schema Markup Validator
Fix errors, then add useful optional properties
Check eligibility rules for each feature
Match markup with visible content
Review Search Console enhancement and performance reports
Confirm indexing and canonical selection
Improve page quality where needed
Monitor for four to six weeks

Structured Data and AI Search

Even when rich results do not appear, structured data still helps. Clear organisation, product, author and article markup supports entity understanding, which helps both search engines and AI systems connect your content to the right brand and topic. Our entity SEO optimization and knowledge graph optimization work builds on this foundation.

Frequently Asked Questions

Why Does the Rich Results Test Pass but Nothing Shows?

Validity only confirms the markup can be read. Display depends on eligibility, quality, query context and Google’s choice.

Should I Remove FAQ Schema Now That It Rarely Shows?

It does no harm when it matches visible content, and it can help machines understand your page. Do not expect a rich result from it.

Does Structured Data Improve Rankings?

Not directly. It improves understanding and can improve click-through when rich results appear.

Validating structured data properly saves wasted effort and focuses work where it counts. For help, explore our rich results optimization service, review structured data optimization, read about featured snippet optimization, learn about People Also Ask optimization, see our FAQ optimization services, book a technical SEO audit, or explore zero-click search optimization.

How to Monitor AI Crawler Requests in Server Logs

To monitor AI crawler requests accurately, you need to look where the evidence lives: your server logs. Analytics tools such as GA4 rely on JavaScript, which crawlers usually do not run, so bot activity never appears there. Logs record every request your server receives, including which crawler asked for which URL, when, and what response it got. This guide explains how to filter logs for AI crawlers, verify that requests are genuine, measure frequency and interpret access patterns.

Why Server Logs Matter for AI Visibility

Logs answer questions no other tool can:

Are AI crawlers visiting the site at all?
Which pages do they request most, and which do they never touch?
Do they receive 200 responses, or errors and blocks?
Did a CDN or firewall change suddenly stop their visits?
Are user-triggered fetchers, which retrieve pages to answer live questions, requesting your content?

If AI crawlers never request a page, that page has little chance of appearing in AI answers that depend on fresh retrieval. Logs give you early warning, long before visibility reports show a decline.

Step 1: Get Access to the Right Logs

Depending on your setup, logs may sit in several places:

Web server logs: Apache or Nginx access logs on your server
CDN logs: Cloudflare, Fastly, Akamai or CloudFront, which may see requests that never reach your server
Hosting dashboards: some managed hosts offer raw log downloads
Load balancer logs: for larger infrastructures

If you use a CDN, CDN logs are essential, because blocked bots may be stopped before they reach your origin server. Keep at least 30 to 90 days of history so you can compare periods.

Step 2: Filter AI Crawler User Agents

Create a filter list of known AI user agents. A practical starting list:

Company	User agents to match
OpenAI	GPTBot, OAI-SearchBot, ChatGPT-User
Anthropic	ClaudeBot, Claude-User, Claude-SearchBot
Perplexity	PerplexityBot, Perplexity-User
Google	Googlebot (Google-Extended is a robots.txt token, not a separate agent)
Microsoft	bingbot (also supports Copilot)
Apple	Applebot
Others	CCBot, Bytespider, Amazonbot, meta-externalagent

Separate them into three groups: training crawlers, search indexing crawlers and user-triggered fetchers. Their behaviour and business value differ, and mixing them hides useful patterns.

A simple command-line example for an Nginx log:

grep -Ei "gptbot|oai-searchbot|chatgpt-user|claudebot|perplexitybot" access.log

For regular monitoring, load logs into a spreadsheet, a log analyser or a data warehouse such as BigQuery, and build saved queries.

Step 3: Verify Crawler Identity

User agents are easy to fake. Scrapers often pretend to be GPTBot or Googlebot to get past filters. Verify identity before trusting the numbers:

Published IP ranges: OpenAI and Perplexity publish JSON lists of IP ranges for their crawlers. Match request IPs against these lists.
Reverse DNS: Google and Bing support reverse DNS verification. Look up the hostname for the IP, confirm it belongs to the company’s domain, then run a forward lookup to confirm it resolves back to the same IP.
Company documentation: check each provider’s crawler documentation for current verification methods.

Label each request as verified, unverified or spoofed. Report only verified requests as genuine crawler activity, and consider blocking spoofed traffic.

Step 4: Monitor AI Crawler Requests by Frequency and Pattern

Once filtered and verified, track:

Requests per day per crawler
Unique URLs requested per crawler
Status code distribution, such as 200, 301, 304, 403, 404, 429 and 5xx
Average response time and response size
Most requested and never requested sections of the site

Read the Patterns

Sudden drop to zero: usually a new block in the CDN, firewall or robots.txt.
Many 403 or 429 responses: rate limiting or bot protection is interfering.
Requests only to the homepage: crawlers may struggle to discover deeper pages, which points to weak internal linking or missing sitemaps.
Rising user-triggered fetches on specific URLs: those pages are probably being used in live AI answers.
Heavy crawling of low-value URLs: parameters or internal search pages may be wasting crawl capacity.

Step 5: Connect Log Data With Other Signals

Logs become far more useful when combined with other data. Compare frequently fetched URLs with pages that receive AI referral sessions in analytics and pages cited in your prompt testing. If a page is crawled often but never cited, the content may not answer questions clearly. If a page is cited but rarely crawled, monitor it closely after any update. Our guide to how AI models select content sources explains how retrieval and selection relate.

Step 6: Set Up Alerts

Automate simple alerts:

AI crawler requests fall by more than 50 percent week on week
Error responses to verified crawlers exceed a set threshold
Robots.txt returns anything other than 200
A new, unknown AI user agent appears in volume

Alerts turn log monitoring from an occasional project into routine protection.

A Monthly Log Report Template

Keep the monthly summary short so busy teams actually read it:

Verified requests per crawler group, compared with the previous month
Top 20 URLs requested by user-triggered fetchers
Important sections with no AI crawler requests
Error rate for verified crawlers, with causes
Infrastructure or robots.txt changes made during the month

Share it with developers as well as marketers. Many fixes, such as rate limits and firewall rules, sit with engineering teams. When those teams see the impact on visibility, fixes happen faster. For businesses with large catalogues, combine this report with a programmatic SEO review, because thousands of template pages can quietly absorb crawler attention. If crawlers ignore whole sections, the keyword cluster SEO structure and internal linking of those sections deserve a closer look.

Frequently Asked Questions

Can I See AI Crawlers in Google Analytics?

No. Crawlers generally do not execute analytics scripts. Use server or CDN logs.

How Often Should I Review Logs?

Weekly is enough for most sites, with automated alerts for sudden changes.

Should I Block Unverified AI User Agents?

Blocking spoofed requests is reasonable. Make sure verification works first, so you do not block genuine crawlers.

Log monitoring is the most reliable way to know whether AI systems can actually reach your content. For help setting it up, explore our fix AI crawler indexing service, learn how to optimize for AI crawlers and LLMs, review SEO automation services, book a technical SEO audit, see our AI indexing case study, understand AI training data optimization, read about website speed optimization, or explore LLM optimization solutions.

What to Check When a Website Migration Changes AI Referral Traffic

When a website migration changes AI referral traffic, the first instinct is to assume ChatGPT, Perplexity or Gemini stopped recommending you. Sometimes that is true. Often, however, the drop is caused by something more mundane: broken redirects from URLs that AI answers still cite, analytics tags that changed during the rebuild, or new security settings that block AI crawlers. This migration-specific checklist helps you separate real visibility loss from measurement errors and technical breaks, in the order that finds problems fastest.

Step 1: Confirm the Change Is Real

Before investigating causes, make sure the numbers can be trusted.

Compare the same date ranges before and after launch, adjusting for weekday patterns and seasonality.
Check whether your AI channel grouping still exists in the new analytics property or container.
Confirm the analytics tag fires on every template of the new site, including blog, product and location pages.
Check whether consent banner settings changed, which can reduce tracked sessions across all channels.

If organic, direct and AI referral traffic all fell by a similar proportion on launch day, the cause is probably tracking, not AI visibility. Our guide to AI search impact on traffic explains how to tell channel-wide shifts from AI-specific ones.

Step 2: Check Whether Referrer Data Survived the Migration

Migrations can change how referral information is captured:

New redirects that strip parameters: AI platforms often add utm_source values to cited links. A redirect that drops query strings loses that tagging.
Domain or subdomain changes: cross-domain tracking that was not configured can turn referrals into direct traffic or self-referrals.
Referrer-Policy headers: a stricter policy on the new site does not affect incoming referrers, but misconfigured intermediate redirects through other domains can.
Single-page app routing: if the new site uses client-side routing, page views may not be recorded correctly after the first page.

Test by clicking a cited link from a real AI answer and checking the session in GA4 DebugView or real-time reports. Repeat the test on mobile, because some AI apps open links in in-app browsers that behave differently from desktop browsers. If you need a refresher on channel setup, our AI search optimization team documents the referral sources worth tracking.

Step 3: Test the URLs AI Platforms Actually Cite

This is the most important migration-specific check. AI answers keep citing old URLs for weeks or months after a migration, because their indexes and training data update slowly. If those old URLs return 404 errors, users clicking from AI answers land on error pages, and the referral session may not register at all.

Build a list of previously cited URLs from:

Pre-migration landing pages for AI referral sessions in analytics
Your prompt testing records of cited URLs
User-triggered AI fetcher requests in historical server logs

Then crawl the list and confirm every URL redirects with a single permanent redirect to the closest equivalent new page.

Old URL status	Impact on AI referral traffic	Fix
404	Clicks lost, citation likely dropped later	Add permanent redirect
Redirect chain	Slower, sometimes timed out by fetchers	Point directly to final URL
Redirect to homepage	Poor user experience, low conversion	Map to equivalent page
200 on new equivalent	Healthy	Monitor

Step 4: Verify AI Crawlers Can Still Access the New Site

New hosting, CDN or security settings often arrive with a migration. Check that robots.txt on the new site matches your intended crawler policy, that firewall and bot management rules do not challenge AI crawlers, and that server logs show verified requests from OAI-SearchBot, ChatGPT-User, PerplexityBot and similar agents after launch. If AI crawler activity dropped sharply on launch day, investigate access before anything else. Our fix AI crawler indexing service starts with these checks.

Step 5: Compare Content Between Old and New Pages

Rebuilds often change content without anyone intending it to:

Long explanations shortened into marketing copy
FAQs removed or moved into tabs loaded by JavaScript
Tables and specifications converted into images
Author information, dates and sources dropped from templates
Internal links reduced by new navigation designs

These changes can reduce how often a page is selected as a source, even when redirects are perfect. Compare archived versions of top cited pages with their new equivalents and restore the elements that made them citable. AI content optimization can then build on that restored base.

Step 6: Check Canonicals, Sitemaps and Indexing

AI systems that rely on Google or Bing indexes inherit any indexing problems created by the migration. Confirm that canonicals on new pages self-reference, that XML sitemaps list only final URLs, and that Search Console shows new pages being indexed at the expected rate. A slow transfer of indexing signals delays recovery in AI answers too.

Step 7: Re-Test Visibility With Your Prompt Set

Once technical and tracking issues are resolved, run your standard prompt set and compare mention and citation rates with pre-migration results. If mentions remain stable but citations now point to third-party sites, your new URLs are not yet trusted or indexed. If mentions have dropped too, focus on content and entity signals.

Plan the Next Migration Better

Most of these problems are preventable. Before any future migration, export every URL that receives AI referral sessions, add them to the redirect map with high priority, and include AI crawler access in your launch checklist. Record a prompt-testing baseline in the month before launch, so you can compare like with like afterwards. If the new site changes templates, involve your content team early so that tables, FAQs and author details survive the redesign. Teams that plan this way usually see AI referrals recover within weeks rather than months, and some see no dip at all. Treat the AI-cited URL list as seriously as your top organic landing pages, because for many businesses those pages now drive the most qualified enquiries. Include Core Web Vitals optimization checks too, since slow new templates frustrate the visitors AI platforms send.

Attribution Notes for Your AI Referral Traffic Report

When reporting on AI referral traffic after a migration, separate the findings clearly: measurement changes, broken URLs, access issues and genuine visibility changes. Leaders need to know which problems are already fixed and which require longer-term work.

Frequently Asked Questions

How Long Do AI Platforms Keep Citing Old URLs?

It varies. Live retrieval platforms update faster, while answers influenced by training data may reference old URLs for months. Keep redirects permanent.

Should I Ask AI Platforms to Update My URLs?

There is usually no direct submission process. Make sure search engines index the new URLs and that redirects are clean.

Why Did AI Traffic Drop While Organic Traffic Held Steady?

Usually because AI platforms cite specific deep URLs that lost their redirects, or because AI crawlers are now blocked.

A careful post-migration check protects one of your most valuable traffic sources. For hands-on support, see our AI traffic drop recovery service, read our AI traffic recovery case study, book a technical SEO audit, learn how to fix AI search visibility, explore website maintenance services, or review ChatGPT visibility ranking.