If you’ve done some digging into Google Webmaster Tools, you may have seen something called a soft 404 crawl error show up in your reports. If you don’t know what a soft 404 is, and don’t know if you should worry about it, our search engine optimization team at Big Red SEO is here to explain!
Soft 404s are often a normal part of how your website functions, but they can cause frustration for users coming to your site from search engines. That’s why our local Search Engine Optimization company recommends keeping soft 404 pages out of search engine indexes.
If you’re seeing soft 404 errors for pages on your website in Google Webmaster Tools, it’s time for you to learn how to fix them!
What is a Soft 404?
When a user searches with Google, two things happen–Google will deliver the page that the user has requested, and that webserver will also send an HTTP response code to the search engine to report the status of that page. This is where terms like “200 OK,” “301 redirect,” and “404 Not Found” come from.
A 200 OK response code tells the search engine that the page was found where it was supposed to be, while a 301 Moved Permanently code redirects the search engine to a new URL. A 404 Not Found code tells search engines that the requested URL could not be found, either because it was moved without a redirect or it never existed on the website in the first place.
A soft 404 occurs when a visitor uses your website’s search function to look for content that you don’t have. The page that shows up in this instance will say something like this:
No search results found, but here are similar articles that might interest you!
While this alerts a user that the content they searched for doesn’t exist on your website, which is what a 404 Not Found response code is supposed to do, it actually creates a unique URL that sends a 200 OK code to search engines, giving them the okay to potentially index those pages unless you tell them otherwise.
Why You Should Not Index Your Site’s Search Result Pages?
Think of this from a user’s point of view–when you search on Google, do you want your search results to be other pages of search results (many of which will be “No search results found” pages pointing to other content that likely doesn’t interest you)? Or would you rather have those results point to pages that have the specific information you want? Our SEO team thinks that most everyone prefers the latter, and so does Google.
In fact, Google’s Webmaster Guidelines state that you definitely should “prevent crawling of search results pages or other auto-generated pages that don’t add much value to users coming from search engines.” With that clear answer straight from Google, it’s important to know how you can prevent these soft 404 pages from being indexed.
How to Stop Your Internal Search Results Pages from Being Indexed
This is a pretty simple process, and one that we have implemented on our site at Big Red SEO, too. We’ll stress here that this is how we solve the problem on our website, and isn’t necessarily a solution that will work for your situation. If you’re not sure how to block crawlers from indexing search results pages on your website, it’s best to rely on the help of a professional. With that being said, here’s the piece of code that we add to our website:
<meta name=”robots” content=”noindex, nofollow”>
We add this code between the <head></head> section of our internal Search Results Page, telling search engines not to crawl, index, or follow any of the links that appear on that page. While a soft 404 page can be useful to visitors who are already on your website searching for content, they’re rarely useful for those coming from search engines, and Google doesn’t want them showing up in their search results.
Contact Big Red SEO Today at (402) 522-6468!
While there are many 404 Not Found pages (and even soft 404s) that won’t harm your search engine rankings in any way, it’s still a good idea to block search engine crawlers from indexing these pages. It’s not good for user experience when these pages show up in Google’s search results, so it’s in your best interest as a business to make sure that they don’t.
When you need help making sure that every aspect of your website is up to Google’s Webmaster Guidelines, give our professional SEO team a call at (402) 522-6468, or fill out our online contact form to request a free basic website audit!