Small desktop program you can install on your Mac to spiders websites' links, images, CSS & scripts from an SEO perspective.
Screaming Frog SEO Spider is a small desktop program you can install on your Mac to spiders websites' links, images, CSS & scripts from an SEO perspective.
Hash value / MD5checksums lookup for duplicate pages Page Title. Missing, duplicate, over 70 characters, same as h1, multiple Meta Description. Missing, duplicate, over 156 characters, multiple Meta Keyword. Mainly for reference as it's only (barely) used by Yahoo. Missing, duplicate, multiple H1. Missing, duplicate, over 70 characters, multiple H2. Missing, duplicate, over 70 characters, multiple Meta Robots. Index, noindex, follow, nofollow, noarchive, nosnippet, noodp, noydir etc Meta Refresh. Including target page and time delay Canonical link element File Size Page depth level Inlinks. All pages linking to a URI Outlinks. All pages a URI links out to Anchor Text. All link text. Alt text from images with links Follow & Nofollow. At link level (true/false) Images. All URIs with the image link & all images from a given page. Images over 100kb, missing alt text, alt text over 100 characters. User-Agent Switcher. Crawl as Googlebot, Bingbot, or Yahoo! Slurp Custom Source Code Search. The spider allows you to find anything you want in the source code of a website. Whether that's analytics code, specific text, or code etc. XML Sitemap Generator. You can create a basic XML sitemap using the SEO spider.
What's new in this version:
Google Search Analytics Integration: You can now connect to the Google Search Analytics API and pull in impression, click, CTR and average position data from your Search Console profile. Alongside Google Analytics integration, this should be valuable for Panda and content audits respectively. View & Audit URLs Blocked By Robots.txt: You can now view URLs disallowed by robots.txt protocol during a crawl. GA & GSC Not Matched Report: The 'GA Not Matched' report ha...