How to improve your AI Score

If our audit shows "Your site appears empty to AI crawlers" (Content quality = 0), it means our crawler couldn't extract any meaningful text from your homepage. Even if humans see a beautiful site, AI bots see nothing - so they can't recommend your business. Common reasons:

Bot protection (most common)

Cloudflare's "Bot Fight Mode", AWS WAF, or similar tools block requests that don't look like a normal browser. AI crawlers (GPTBot, ClaudeBot, PerplexityBot) get rejected before they can read your content.

Fix on Cloudflare:

1. Log in to your Cloudflare dashboard
2. Go to Security → Bots
3. Make sure "Verified bots" are allowed
4. Or add an explicit allow rule for these user-agents: GPTBot, ClaudeBot, Google-Extended, PerplexityBot, Anthropic-AI, CCBot

For AWS WAF, ask your webmaster to whitelist these AI bot user-agents.

JavaScript-only website (Single Page App)

If your site is built with React, Vue, Angular, or similar without server-side rendering (SSR), AI crawlers see an empty HTML shell. The actual content only appears after JavaScript runs - and most AI crawlers don't run JavaScript.

Quick test: right-click your homepage → "View Page Source" (or type view-source:https://yoursite.com in your browser address bar). If you see mostly empty body and just <script> tags, AI bots see exactly that - nothing useful.

Fix: implement Server-Side Rendering (SSR), Static Site Generation (SSG), or pre-rendering for crawlers. Frameworks like Next.js, Nuxt, or SvelteKit support this out of the box.

Login/paywall

If your content is hidden behind login or paywall, AI crawlers can't access it.

Fix: consider making at least your homepage and key landing pages publicly accessible. AI tools won't recommend content they can't read.

Server returning errors

Your site may be returning 403, 503, or other error codes to non-browser requests.

Fix: check server logs for failed requests with AI bot user-agents. Adjust firewall/CDN rules to allow them.

Why this matters: if AI bots can't read your content, your business won't appear in ChatGPT, Claude, Gemini, or Perplexity answers. As of 2026, this is becoming as important as Google search visibility.

Your robots.txt file tells crawlers what they can and cannot access. AI bots (ChatGPT, Claude, Gemini) check it before reading your site. If our audit gives you a low score for "AI bots access", here are the typical scenarios and fixes:

Your site asks AI not to read it

Your robots.txt contains explicit blocks for AI crawlers like GPTBot or ClaudeBot. This is intentional opt-out - AI tools won't index or cite your content.

Fix: open robots.txt, find lines like User-agent: GPTBot followed by Disallow: /, and remove them (or change to Allow: /). Ask your webmaster.

No robots.txt found (404)

Your site doesn't have a robots.txt file at all. AI bots may be cautious without explicit guidance.

Fix: create a simple text file at https://yoursite.com/robots.txt with this content:

User-agent: *
Allow: /

Sitemap: https://yoursite.com/sitemap.xml

Server is blocking automated access (403)

Your server returns "Forbidden" when crawlers try to read robots.txt. This usually means a firewall, bot-protection rule, or CDN (Cloudflare, AWS WAF) is blocking non-browser requests. AI bots can't verify what's allowed, so they often skip your site.

Fix:

• Cloudflare: go to your dashboard → Security → Bots → make sure "Bot Fight Mode" allows verified bots, or add an exception rule for /robots.txt and /sitemap.xml.
• AWS / other: ask your webmaster to allow public access to /robots.txt in the WAF rules.
• Quick check: open https://yoursite.com/robots.txt in a private browser window. If you see "403 Forbidden" or a CAPTCHA, that's the issue.

robots.txt returns HTML

Your robots.txt URL returns a webpage (HTML) instead of plain text robots rules - usually a 404 error page or an app redirect.

Fix: ask your webmaster to add a real robots.txt file in the site's root directory. The content should be plain text (see "No robots.txt found" above).

Your website needs a short description that tells search engines and AI bots what your page is about. This appears in Google search results, when sharing your link on Facebook/LinkedIn/X, and in AI answers. Without it, you waste a critical first impression.

Meta description

A 1-2 sentence summary of your page (max 155 characters). Should mention what you do, where you are (for local businesses), and what makes you stand out.

Example for a barber shop:

<meta name="description" content="Men's barber shop in Warsaw centre. Cuts from 100 PLN,
beard trim 70 PLN, combo 140 PLN. Online booking via Booksy.">

Open Graph tags (for social sharing)

When someone shares your link on Facebook, LinkedIn, WhatsApp, or in AI chats, these tags control how the preview looks (title, description, image).

<meta property="og:title" content="Marina Safronova - Men's Barber Warsaw">
<meta property="og:description" content="Cuts, beard care, premium men's grooming in Warsaw centre.">
<meta property="og:image" content="https://yoursite.com/images/preview.jpg">
<meta property="og:type" content="website">

How to add them

These tags go inside the <head> section of your HTML. If you use:

• WordPress: install Yoast SEO or Rank Math plugin - both add meta + OG tags via simple form
• Squarespace / Wix / Webflow: built-in SEO settings under "Page settings" or similar
• Custom HTML: ask your webmaster to add tags above (or we can generate them for you)

Quick test

Paste your URL into opengraph.xyz or metatags.io - they preview how your link looks when shared.

Why this matters: meta description directly affects click-through rate from Google (good description = more clicks). For AI bots, it's often the first signal of what your site offers - they may rely on it when summarizing your business.