behlabs · engineering notes · september 2026
A marketing site can have flawless meta tags and still be invisible — because a CDN toggle is quietly telling every crawler to go away. Here's the full pipeline I built, the four traps I hit, and the playbook I'll reuse on every project from now on.
SoriGamis is a mobile app that turns recorded conversations into speaker-attributed transcripts, summaries, and decision records. Its marketing site is a small static-first React app — five routes, prerendered to plain HTML at build time, deployed on Vercel behind Cloudflare. The goal wasn't just classic SEO: I wanted the site legible to AI search too — ChatGPT (whose web search leans on Bing), Claude, and Perplexity all crawl the web with their own bots, and they can only cite what they're allowed to read.
What follows is what I actually did, in order, including everything that went wrong. The reusable step-by-step playbook is at the end.
The site already had the basics — per-route titles, descriptions, canonicals, Open Graph tags, and JSON-LD structured data baked into prerendered HTML. The audit still found real rot:
lastmod from weeks ago, and adding
a new page required remembering to edit it. I replaced it with
build-time generation: the prerender script now emits
sitemap.xml from the same route map it renders pages
from, with lastmod derived from the git history of each
page's source files. One source of truth; a forgotten route now fails
the build loudly.
llms.txt — a
plain-text site summary that LLM crawlers can ingest directly.
Deriving lastmod from git log works great
locally and silently produces wrong dates in CI: shallow
clones (Vercel builds, actions/checkout with default
depth) report the clone-boundary commit's date, stamping every page as
freshly modified on every deploy — worse than the stale dates I was
replacing. The fix: check
git rev-parse --is-shallow-repository and omit
lastmod entirely in shallow clones. A missing date is
honest; a wrong one erodes crawler trust.
On a React site, client-side components that rewrite
<head> tags and JSON-LD will
overwrite your server-rendered metadata after page load. I
fixed the server copies, shipped them — and a code review caught that
two client components still carried the old text and replaced the
fresh JSON-LD on every hydration. Any JavaScript-executing crawler
(Googlebot included) was indexing the stale version. If metadata lives
in more than one place, sync all of them or the newest copy loses.
With the site itself in order, I checked the live robots.txt — and found a block of directives I never wrote, sitting above mine:
# BEGIN Cloudflare Managed content
User-agent: *
Content-Signal: search=yes,ai-train=no,use=reference
Allow: /
User-agent: ClaudeBot
Disallow: /
User-agent: GPTBot
Disallow: /
# ... CCBot, Google-Extended, Amazonbot, Bytespider, meta-externalagent ...Cloudflare was rewriting my robots.txt at the edge to tell the exact crawlers I'd just welcomed to go away — and separately 403-blocking them at the network level, so even a bot that ignored robots.txt couldn't get through. My repo's file was irrelevant; the deployed origin had its own policy.
It turned out to be two independent toggles in the Cloudflare dashboard, both under AI Crawl Control:
Disabling one without the other isn't enough — I know because I disabled one, re-checked the live file, and the block was still there. After turning both off, the served robots.txt finally matched the repo byte for byte.
Within 24 hours of unblocking, Cloudflare's crawler analytics showed Claude-SearchBot had already made 11 successful requests. The crawlers were waiting at the door the whole time.
Both consoles offer convenient shortcuts that trade account access for saved clicks. I skipped both:
sitemap.xml.
<code>.yourdomain.com → verify.bing.com), then submitted the same sitemap.
On Cloudflare, new CNAME records default to Proxied, which hides the record's real target behind Cloudflare's edge — and breaks DNS-based verification. Set verification CNAMEs to DNS only. (TXT records are safe; they can't be proxied.) And afterward: never delete these records. Each one silently anchors its console's ownership verification.
Both submissions confirmed instantly — Google reported "Sitemap submitted successfully" with 5 pages discovered on the spot; Bing showed the sitemap as processing, flipping to Success, 5 URLs discovered by the next morning.
Submitting a sitemap once is a snapshot. IndexNow is the standing arrangement: an open protocol where you ping an API whenever content changes, and Bing, Naver, Yandex, and Seznam re-crawl within minutes instead of on their own schedule. (Naver in that list is a real bonus for a Korean/English product.) The setup is refreshingly simple:
openssl rand -hex 16) and serve it
as <key>.txt at your site root — that public file
is the authentication.
api.indexnow.org with the key. My script fetches the
site's own live sitemap for the URL list, so the sitemap generator
stays the single source of routes.
const sitemap = await fetch(`${siteUrl}/sitemap.xml`);
const urlList = [...(await sitemap.text())
.matchAll(/<loc>([^<]+)<\/loc>/g)].map(m => m[1]);
await fetch("https://api.indexnow.org/indexnow", {
method: "POST",
headers: { "Content-Type": "application/json; charset=utf-8" },
body: JSON.stringify({ host, key, keyLocation, urlList }),
});
Two failures on the way in, both legitimate: the repo's secret scanner
flagged the hardcoded IndexNow key as a leaked API key — the right fix
wasn't an allowlist entry but removing the literal, deriving the key
from the checked-in public key file instead (and amending the commit
so the literal never entered history). Then the workflow's hardened
permissions: {} stripped even
contents: read, so checkout couldn't see the private
repo. Minimal isn't zero: permissions: contents: read.
First automated ping: IndexNow accepted 5 URLs (HTTP 202) —
accepted, key validation pending. By the next day, a direct submission
to Bing's endpoint returned HTTP 200: key validated,
submissions accepted at face value. From now on, every release re-pings
automatically.
| When | Event |
|---|---|
| Day 0, ~11:00 | Sitemap submitted to Google Search Console |
| Day 0, 11:11 | Googlebot crawls the homepage — eleven minutes later |
| Day 0 | Cloudflare unblocked; Claude-SearchBot logs 11 successful requests within 24h |
| Day 0 | IndexNow ships; first automated ping accepted (202) |
| Day 1 | URL Inspection: homepage "URL is on Google", canonical agreed, all signals green |
| Day 1 | Bing sitemap status: Success, 5 URLs discovered; IndexNow key validated (200) |
| Day 1 | /faq "Discovered — currently not indexed" → priority indexing requested |
Aggregate dashboards lag behind all of this — Search Console's Pages report and Bing's IndexNow insights both take days to start rendering for a new property. The per-URL inspection tools are the ground truth in week one; don't panic-debug a dashboard that's still saying "processing data".
The order matters: on-site correctness first (so crawlers see something worth indexing), unblocking second (so they can reach it), registration third (so you can see what they saw), automation last.
lastmod from git
history of page sources — and omit it in shallow clones rather
than emit wrong dates. Fail the build if a route is
missing.
The whole thing — audit to automated pings — fit in a day, and most of that day was spent on the two things no meta tag can fix: the CDN silently overriding site policy, and five copies of "the same" metadata quietly disagreeing. Check the live wire, not the repo.