Technical GEO Agency | AI Crawler Access & Schema

What a model can read,
and what it can’t

AI crawlers don’t run your site the way a browser does. GPTBot,
ClaudeBot, Google-Extended and PerplexityBot fetch raw HTML. Most of
them do not execute JavaScript. If your product details, pricing or key
facts are injected client-side, a common pattern in React and Vue
builds, a browser sees them fine and these bots see an empty shell.
Google’s own crawler for Search still renders JavaScript for ranking
purposes; Google-Extended, the separate crawler that feeds AI Overviews
and Gemini training, follows the same fetch behaviour as the others and
is not guaranteed the same rendering budget. The safest assumption is
that nothing client-side-only gets read by any AI system. We check
what’s actually in the response HTML, not what’s on screen.

What’s included

• Robots policy audit. We check what your robots.txt
  allows and blocks for GPTBot, ClaudeBot, Google-Extended, PerplexityBot,
  Applebot-Extended and the others, against what you actually intend, sites block AI crawlers by accident more often than on purpose.
• Rendering audit. We fetch your key pages the way a
  bot does (raw HTML, no JavaScript) and compare that to what a browser
  shows, so you know exactly what’s invisible to retrieval.
• Structured data that’s real. Organization, Article
  and BreadcrumbList markup, validated against schema.org and Google’s own
  guidance, kept accurate as content changes. We do not sell FAQPage,
  HowTo, Product, Offer, Review or AggregateRating schema. Google has
  restricted or removed rich-result eligibility for most of these, and
  stacking unused schema types adds risk without benefit.
• Content chunking and heading structure. Passages
  retrieval systems can lift on their own, one idea per section, a
  heading that states the question, a self-contained answer underneath,
  rather than a single wall of text a model has to guess where to
  cut.
• Entity consistency. The same name, description and
  key facts stated the same way across your site, your Organization
  schema, your review-platform profiles and any third-party listings, so
  nothing contradicts what a model has already learned about you
  elsewhere.
• Speed and canonical hygiene. Slow pages get fetched
  less often and less fully; duplicate or conflicting canonical tags split
  whatever authority a page has. Both are unglamorous, both matter to
  whether a model bothers to fully retrieve a page.

The JSON-LD we actually ship

Every page we build for GEO carries three schema types, no more:
Organization (or ProfessionalService) for
entity identity, Article for editorial content, and
BreadcrumbList for site structure. Each validates clean
against schema.org’s own vocabulary and is generated from the page’s
real content, not hand-typed once and left to drift. If a client asks
for FAQPage or HowTo markup, we explain why we don’t add it: it doesn’t
influence AI citation, and Google has already pulled back on its
search-result eligibility for most sites, so it’s a compliance surface
with no upside.

Query fan-out

A single question to ChatGPT, Gemini or AI Overviews often triggers
several retrieval passes behind the scenes, the system breaks “best X
for Y” into narrower sub-queries and merges what it finds. So a page
needs to answer more than its headline question. We structure content so
each section can independently satisfy one sub-query (pricing on its
own, scope on its own, comparison on its own) rather than betting
everything on one exact phrase.

Does llms.txt do anything?

Short answer: not for Google. In June 2026 Google confirmed it does
not use llms.txt as a ranking or retrieval input, its systems read your
actual page content, not a separate file summarising it. Other AI
systems have shown no consistent public commitment to reading it either,
and it is not a W3C or schema.org standard, just a convention some sites
have adopted. We don’t add llms.txt to client sites and we don’t sell it
as a service. If a page isn’t being retrieved, the fix is almost always
in the page itself (crawler access, rendering, or content structure) not a text file sitting next to it.

Proof

Official London
Theatre reached 85% AI search visibility and the #1 recommended
source position in its category. That’s a whole-account GEO result, not
a technical-layer-only claim. Entity work, content and technical access
all fed it together, and we won’t pretend otherwise here. What we can
say for the technical layer specifically: our schema and crawler-access
audit follows the same process on every account, GEO or SEO. The fastest
way to see where your own site stands is the free check below, not a
client number that isn’t your site.

Free schema and crawler
check

Run our schema and crawler-access
checker against your own domain. It checks your robots.txt against
the major AI crawlers, pulls your live JSON-LD and flags what’s missing
or invalid, and shows you the raw HTML a bot actually receives on your
key pages, free, no call required.

What it costs

Technical GEO runs on a GEO retainer, from £1,250 a month ex VAT. There is no minimum term and either side can give 60 days’ notice at any point. The work scales with what is already broken. A site with clean schema and open crawler access needs a light audit. A JavaScript heavy build with blocked bots and no structured data needs real engineering time. The GEO pricing page sets out the tiers. Book a call and we will scope it against your site.

Related services

(template-rendered, not written here; see Related links below for
the three to configure)