Test Crawling to Find SEO Errors

Mistakes happen; we’re only human – and we’re working with computers, which doubles the risk of errors.  Even just one line of code that was entered incorrectly, one that was added erroneously, or one that was neglected can be counteracting your best SEO efforts.  And you may not even be aware of it.  Errors happen, but they don’t have to keep happening. Test crawling your site will help you spot errors and increase the efficacy of your optimization techniques.

Say that you operate an e-commerce site and have a shiny, sleek new smartphone that comes in silver, black, red, and blue.  When users click through to look at the product information, they can opt to view the phone in each color if they want.  To most people, this would simply be the same website but with a different picture.  To the people who have to create the website and the search engines that crawl it, it is really four different URLs.  You do not want to be penalized for having so much duplicate content, so you tell Google which URL you would like to be indexed.  This is your canonical URL.  It tells Google that the content of the other pages should be attributed to this one page.

But let’s say you misuse the canonical URL by telling the search engine that all the pages on your site should be attributed to one single page.  Or that the URL you use for the canonical URL tag is broken.  This is a big problem because it can compromise your search rankings and keep paying customers from finding you.

This is just one of the reasons why a test crawl is so critical to any site.  You can also uncover server errors, 404s, bad outbound links that could put your site in the penalty box, and other errors, but it can also provide you with invaluable data for optimising your site more effectively.   A test crawl looks over your site as a bot would; it sees your site through Google’s eyes to help you find crawlability problems and other issues that could be impeding traffic flow.

There are a variety of free tools and paid downloads available to help you conduct test crawls. Some good ones include:

• Spidertest.  You simply enter in your website URL to test your technical SEO.  The site combs and tests your page and generates suggestions to help you improve.
• TotheWeb. This is a search engine emulator that will show you how your site appears to search engines like Google.
• SEOMoz.  They offer a free tool, as well as a more comprehensive paid tester that allows you to test as many as 30,000 pages a day and more sites than the free version, which allows you to test up to 250 pages.
• Xenu’s Link Sleuth. This is designed to find broken links.  Xenu is highly regarded by industry experts and is available for free.

Running a test crawl can help you correct and avoid problems and create a more effective and user-friendly website.  Make it a regular part of your SEO efforts.