Screaming Frog SEO Spider
A desktop crawler that walks a site the way a search engine bot would, cataloguing every URL, response, and on-page element it finds.
Analogy Mental model
A scanning machine in a hospital. Feed the site in and it reports which vessels are blocked, which organs are duplicated, and which nerves are miswired, all in a table you can sort.
In practice What you actually do
This is the working tool of technical SEO. Render JavaScript when auditing a client-side app, use custom extraction with XPath or regex to pull publication dates or author names in bulk, and connect the Search Console and analytics APIs so traffic and speed sit beside each URL.
Where people get burned Pitfalls
Crawling a large site at default settings can knock your own server over. Raise memory, enable database storage, and throttle to a few requests per second before pointing it at anything with six figures of URLs.
This entry was written in English from the Chinese original, not machine-translated. Read the Chinese version →