TL;DR
- Internal redirects may not seem like much but are a waste of time for crawlers.
- Donatos Pizza only shows 7% of their crawl are redirects, but thats over 160,000 links to crawl.
- Crawl error cleanup is tedious, but sets a great base for SEO.
One of the first things I do, whenever I start optimizing a website, is looking at the response codes from a Screaming Frog crawl. I have had a ton of success improving rankings whenever I clean up any unnecessary internal crawl errors. These errors can be obvious 4xx or 5xx errors, but I like to take it a step further and eliminate any unnecessary redirect hops. This isn’t removing redirects, it’s just updating any links we have on our site and pointing it to the final destination.
Many large sites can have insane amounts of internal redirects from over the years so it becomes difficult to start this process. I have put together a simple pivot table that can make this process much easier. It breaks down all redirects on the site sorted by the most amount of inlinks (usually navigation or footer) and shows exactly where they are and what link to add to clear up the error.
I utilized Donatos Pizza which is one of my favorite places, unfortunately, it is not in Chicago. In the crawl, it only shows 1,584 redirected URLs which is only 7% of the crawl. Very minor in terms of the Response Code chart, but when you export all redirects there are over 160,000 due to the number of times those redirected URLs are referenced on the site. That is a ton of unnecessary work search engine crawlers need to do just to navigate the website. Cleaning up all these redirected links could go a long way in improving the visibility of all your content.
Step 1: Screaming Frog Crawl
With any sort of optimization, you should most likely be starting with a Screaming Frog crawl to see what is going on under the hood. I like to update the Configuration to include the XML, follow all redirects, etc to make sure that I can access everything out there. When the crawl is complete, check out the Response Codes tab to see the breakdown.
In my experience, the greener this pie chart is, the more success your site will have. This is not to say this will fix your SEO, but it will eliminate many issues search engines will have when navigating your site.
Step 2: Export All Redirects
Under Reports –> Redirects –> All Redirects
This will allow you to get every redirected URL on the site (internal and external). It will also show the chains with each step, the anchor text, source, etc. Essentially everything you need.
Step 3: Create Pivot Table
Open the file you just exported in Step 2. It is going to be a massive ugly list that you won’t really be able to make sense of without a lot of cleanup.
Under Insert –> Create Pivot Table in a new sheet (title it whatever you want).
You will now have a blank pivot table in a new sheet and we can start moving the different fields to make this report useful.
- Under Rows (In Order)
- Address
- Source
- Anchor Text
- Final Address
- Link Position
- Under Values
- Address
Step 4: Sort Pivot Table For Most Errors
In the first column in the top right of the cell, you will see a drop-down arrow. Click on that and select More Sort Options.
Then choose Descending (Z to A) by: Count Of Address.
This will sort all errors from the most to least allowing you to see which links are causing the most issues. This is usually the navigation or footer, sitewide issues that can easily be cleaned up to remove a lot of crawl errors.
Step 5: Start Cleaning Crawl Errors
Now the table is set up to show you exactly what URLs are causing issues and where they are. You can start going down the list to get these cleaned up. Some may be easier URLs that can be fixed in one place if it is a template issue. Others may be one-by-one which will take some time. Even though it is tedious, and not a sexy optimization, it can really go a long way to improve the visibility of all your content!
Shoot for your Screaming Frog crawl to have the Response Code chart as green as possible!
