We Audited Our Own Site and Found It Returns 200 for Everything — Longtail Analytics
It's easy to write about site audits in the abstract. It's more useful to run one on yourself and publish what you find, including the parts that are mildly embarrassing. While validating the delivery mechanism behind this section, we requested a handful of paths on longtailanalytics.io that we knew didn't exist. Every one of them came back 200 OK with the homepage. That includes /robots.txt.
What "soft-404-everything" means
A soft 404 is a page that behaves like an error to a human — wrong content, nothing they were looking for — but returns a success status to a crawler. Ours doesn't soft-404 a few stray URLs; it soft-404s every unknown path, uniformly, by serving the homepage with a 200. There's no dedicated not-found page and no distinct status code signaling "this doesn't exist." Functionally, the site has no concept of a broken link.
The part that's a real cost
Two things follow directly from this, and neither is speculative. First, /robots.txt returning homepage HTML instead of a robots ruleset parses, per the documented spec behavior, as an empty ruleset — which is functionally allow-all. That's not dangerous by itself, but it means we have no working Sitemap: directive for crawlers to discover, because there's nowhere valid to put one. Second, it's a crawl-efficiency cost: a crawler that requests a mistyped or stale URL and gets a confident 200 has no signal to deprioritize that path, which spends crawl budget on pages that were never real.
The part we're not going to claim
It's tempting to extend this into "and that's why Google doesn't trust new URLs on this site." We looked for evidence of that specific causal claim and didn't find it documented anywhere credible — it's a plausible-sounding inference, not a demonstrated mechanism. We're flagging it as a hygiene defect and a crawl-budget cost, not dressing it up as a bigger villain than the evidence supports. Overclaiming the damage of our own bug wouldn't make the audit more honest; it would make it less.
Why we're publishing this instead of quietly fixing it
Because the fix is worth documenting on its own terms. The plan is to serve a real robots.txt (allow-all, with a working Sitemap: directive) and a real sitemap.xml through the same signed edge delivery path covered in our zero-deploy article — reflecting existing site pages plus any new /insights/ pages as they go live. That's a design decision at the time of writing, not a shipped one; we'd rather say exactly where it stands than round up.
The takeaway if you're checking your own site
Request a URL you know doesn't exist. Check the status code, not just what renders in a browser. Then check /robots.txt the same way. If both come back 200 with your homepage, you have the same defect we found on our own site — and the free site audit is a fast way to check the related metadata fundamentals while you're in there.