Gary Illyes from Google explained two reasons why soft 404 errors are bad. Soft 404s are when a page returns a 200 status okay, but Google thinks that page should return a 404 page not found error. According to Gary, soft 404s are bad because they (1) limit crawl budget and (2) the pages won’t likely show up in Google Search.
Gary wrote on LinkedIn wrote, “Crawlers have lots of resources, they can afford to waste some, your site likely doesn’t. Soft errors are bad because:”
- The limited “crawl budget” spent on them could’ve been spent on real pages.
- The pages will unlikely to show up in search because during indexing they’re filtered out, basically no ROI on the resources you’ve spent on serving them.
Gary called soft 404s and other soft/crypto errors (“Crypto here means “hidden”, not what the bros are trying to convince you to invest in,” Gary explained.), “the banes of my existence and all other robots’.”
He wrote:
You go to your favorite coffee shop after consulting their online menu and you order your favorite corn spice latte with yak milk. They’re all out even though the menu claimed they had it. You order a half espresso. They’re all out. Fine, you order a matcha latte with water chestnut milk. They’re all out. Frustrating. Is this a coffee shop or Wendy’s?!
While for users it might not matter much that your error page came back with a HTTP 200 (OK) status code, crawlers use the status codes to interpret whether a fetch was successful, even if the contents of the page is basically just an error message. They might happily go back to the same page again and again wasting your resources, and if there are many such pages, exponentially more resources. All while they could spend the time and resources on fetching real pages, with actual helpful content.
Forum discussion at LinkedIn.