Configuration
Use this configuration file to get the required results: SEO Spider Config.seospiderconfig.
Manual configuration:
🚧 If you are using the configuration file above, manual configuration is not required.
After downloading Screaming frog:
Navigate to Configurations > Robots.txt > Settings > Ignore and select the settings below.
Navigate to Configurations > Spider > Extraction > Structured Data and select the settings below:
Navigate to Configuration > Spider > Rendering, select JavaScript and enter 10 second timeout.
Tests to Perform before Launch
Images
All images should be hosted by f.shgcdn
Check for wrong CDNs
Scrape subdomain
Click on Images Tab
Type the following regex into the Search bar:
/(?!.*(quality|afterpay|airtable|dropinblog|facebook|getelevar|ggpht|googleapis|googletagmanager|instagram|klarna|klaviyo|pinterest|stamped|tiktok|yotpo|ytimg))/
Export results
Fix as needed
Images missing height/width attributes
Scrape subdomain
Click on Custom Extraction Tab
Filter by Images missing height/width attributes
Export results
Fix it =)
Structured Data
Scrape subdomain
Click Structured Data Tab
Filter by Missing
If any URLs appear, make note of them here.
Missing Structured Data: These are URLs that do not contain any structured data.
Validation Errors: These are URLs that contain validation errors. The errors can be either Schema.org, Google rich result features, or both depending on your configuration. Schema.org issues will always be classed as errors, rather than warnings. Google rich result feature validation will show errors for missing required properties or problems with the implementation of required and recommended properties. Google’s required properties must be included and be valid for content to be eligible for display as a rich result.
Validation Warnings: These are URLs that contain validation warnings for Google rich result features. These will always be for recommended properties rather than required properties. Recommended properties can be included to add more information about content, which could provide a better user experience, but they do not disqualify you from being eligible for rich snippets. There are no warnings for Schema.org validation issues; however, there is a warning for using the older data-vocabulary.org schema.
Parse Errors: These are URLs containing structured data that failed to parse correctly. This is often due to incorrect mark-up. If you’re using Google’s preferred format, JSON-LD, then the JSON-LD Playground is an excellent tool to help debug parsing errors.
Microdata URLs: These are URLs that contain structured data in microdata format.
JSON-LD URLs: These are URLs that contain structured data in JSON-LD format.
Identify 4xx and 5xx Links
Scrape subdomain
Click Bulk Export (top)
Response Codes
4xx Inlinks
5xx Inlinks
Check that all pages have canonical links
Scrape subdomain
Click Canonicals Tab
Click All filter and select Missing
If any URLs appear, make note of them here.
Check for Headings (H1 and H2)
This check can be done for h1 and h2 tags, but each page must have one h1 tag.
Scrape subdomain
Click H1/H2 Tab
Filter by Missing/Duplicate
If any URLs appear, make note of them.
Check for Title and Meta Description on all pages
Scrape subdomain
Click on the Page Titles tab then click on the All filter and select Missing
Click on the Meta Description tab then click on the All filter and select Missing
Report any URLS with missing titles and descriptions.
Also note the length of the title and Meta Descriptions.
Actions:
If title or description is missing from pages that are not tied to a Template Page, fix them on a Page-Level under Page Settings in the XM
If title or description is missing from pages that ARE tied to Template pages, ensure the Page Settings are configured correctly on the Template Page
Page Title Length
Over 60 characters – Any pages which have page titles over 60 characters in length. Characters over this limit might be truncated in Google’s search results and carry less weight in scoring.
Below 30 characters – Any pages which have page titles under 30 characters in length. This isn’t necessarily an issue, but you have more room to target additional keywords or communicate your USPs.
Meta Description Length
Over 155 characters – Any pages which have meta descriptions over 155 characters in length. Characters over this limit might be truncated in Google’s search results.
Below 70 characters – Any pages which have meta descriptions below 70 characters in length. This isn’t strictly an issue, but an opportunity. There is additional room to communicate benefits, USPs or call to actions.
Check the CLS score
Why it's important? → https://web.dev/cls/
Scrape subdomain
Click PageSpeed Tab
Report any URLs with Cumulative Layout Shift value grater than 0.1