Screaming Frog SEO Spider in-depth SEO and log analysis!

Hosting Mautic QuickStart -50% with support in Spanish
24 hours and free training

Send up to 1,000,000 emails/year. Mautic support in Spanish 24 hours, 365 days

In recent times, many development companies have released tools, most of them commercial, to analyze websites in a serious and professional way, allowing them to be audited, and other factors that affect the positioning and performance of the site.

Screaming Frog SEO Spider is one of those ‘desktop’ tools oriented to SEO analysis, which pretending to be a Google spider (crawler), crawls a website collecting useful information such as image meta tags, page titles and meta descriptions, canonical links, response, problems and other errors.

From the information obtained with Screaming Frog SEO Spider It is possible to carry out strategies aimed at improving the optimization of the SEO/SEO on-page of websites, be they WordPress, Joomla, PrestaShop or another CMS.

It is a powerful tool developed by Dan Sharp founder and director of Screaming Frog Ltddesigned and oriented to analyze and audit websites from the SEO point of view, designed for small sites and mainly oriented to large sites, where manually checking each page would become a heavy task that would consume a lot of human resources.

What can you do with Screaming Frog SEO Spider?

Find broken links and errors.
Review of robots and directives.
Analyze page titles and metadata.
Discover duplicate content.
Generate XML sitemaps.
Log analysis.
Bot tracking, limits and settings.
Extract data with XPath.
Parse images, CSS, javascript, SWF files.
Redirect Audit.

The “free” version of Screaming Frog SEO Spider allows to analyze or trace 500 URIs; for very large sites you will have to consider the commercial license of the tool.

Screaming Frog SEO Spider Highlight Features:

Errors: Client errors such as broken links, server errors (No response, 4XX, 5XX).
Redirects: permanent or temporary redirects (3XX responses).
Blocked URLs: View and audit URLs not allowed by the robots.txt protocol.
External Links – All external links and their status codes.
Protocol: Whether the URLs are secure (HTTPS) or insecure (HTTP).
URI issues: non-ASCII characters, hyphens, uppercase characters, parameters, or long URLs.
Duplicate Pages: Hash value/ MD5 checksum for pages with duplicate content.
Page Titles: Titles that are missing, duplicated, longer than 65 characters, short, truncation pixel width, same as H1, or multiple.
Meta Description: Nonexistent, Duplicate, Longer than 156 characters, Short, Truncation pixel width, or Multiple.
Meta Keywords: Mainly for referrals as they are not used by Google, Bing or Yahoo.
File size: size of URLs and images.
Response time.
Last modification of the header.
Page depth level.
Word count.
H1: non-existent, duplicate, with more than 70 characters, multiple.
H2: non-existent, duplicate, with more than 70 characters, multiple.
Meta Robots: index, noindex, follow, nofollow, noarchive, no snippet, NOODP, NOYDIR, etc.
Meta Refresh: including landing page and delay time.
Element with canonical link and canonical HTTP headers.
X-Robots-Tag.
rel=“next” and rel=“prev”.
AJAX – The SEO spider obeys Google’s crawling AJAX scheme.
Inlinks: all pages with links to a certain URL.
Outlinks – All outbound links to a specific URL.
Anchor text: all the text of the link. The alternative text (alt) of images with links.
Follow and Nofollow: on the page and at the link level (true/false).
Images: All URLs with the image link and all images on a given page. Images of more than 100kb, without the alternative text (alt), or with text of more than 100 characters.
User-Agent Switcher: track like Google Bot, Bing Bot, Yahoo! Slurp, mobile user agents or with your custom UA code.
Configurable Accept-Language Header: Provide HTTP Accept-Language header to track locale content.
Redirect Chains: Discover redirected chains and loops.
Custom Source Code Search: SEO Spider allows you to find anything you want in the source code of a web page. Whether it’s Google Analytics code, a specific text, or code, etc.
Custom extraction: You can collect any data from the HTML code of a URL using XPath, CSS Path selectors or regular expressions.
Integration with Google Analytics: you can connect to the Google Analytics API and pull user and direct conversion data during a crawl.
Integration with Google Search Console: You can connect to the Google Search Analytics API and collect impressions, clicks, and average URL position data.
XML Sitemap Generator – You can create an XML sitemap and imagemap using SEO Spider.

Download Screaming Frog SEO Spider

it is written to be used on different operating systems, Linux, Windows and OSX. You can download it from the official website of the project.

Screaming Frog SEO Spider Response Codes

When you make use of the tool the results can return a series of response codes. Knowing and understanding these codes will help you make more efficient use of it.

The response codes tab of the tool includes internal and external URI response information.

These common HTTP response codes are:

Unanswered: When we do not receive a response to our request. Typically a malformed URI or a connection timeout.
Correct (2XX): The requested URI was successfully received, understood, accepted, and processed.
Redirection (3XX): An address change has been detected.
Client error (4xx): Indicates that a problem has occurred with the request.
Server error (5XX): The server was unable to fulfill an apparently valid request (locks, etc).

Some examples:

200: OKAY.
301: Permanent Redirect.
302: Temporary Redirect.
404: Resources not found.
500: Server error
503: Resource not available.

In it you can find the complete description of each response code (in English).

If when executing in Screaming Frog SEO Spider a query, the answers you receive are 502 Bad Gateway errorscheck with your Support Department, as it is possible that the Firewall is blocking the access of the tool to your website.

Log analysis with Screaming Frog SEO Spider

Recently the development team of Screaming Frog SEO Spider has released Log File Analyzer, a separate tool from SEO Spider designed to analyze log files (logs) of activity, etc. Basically this tool allows you to see exactly what the search engines have experienced by crawling the website, over a period of time.

The Log File Parser is lightweight, yet very powerful, capable of processing, storing, and parsing millions of lines of log file event data into an intelligent database.

Currently Log File Analyzer it is commercial, but you can use the free licensed version to analyze 1 website or 1000 lines of logs, enough to determine if this tool is useful for your web projects and see if you jump to the commercial version.

This tool gathers key data from the log file to enable informed decisions. Some of the common uses include:

Identify URL trackers.
Locate broken links and errors.
Analyze most and least tracked urls.
Detect uncrawled and orphan pages.
Analyze the frequency of crawlers.
Audit redirects.
Identify large pages and performance issues.
Combine and compare data (GA and logs).

Thanks to the “drag & drop” technology you can drag one or several large log files directly into the interface of Log File Analyzerand the tool will automatically detect the log file format and compile the data into a local database, capable of storing millions of log events.

Multiple log files or folders can be dragged at the same time, which can be gzipped and from any server, be it Apache, Nginx or ISS.

At the moment the data collection is focused on Google robots, Bing Bot, Yandex and Baidu. In later versions this functionality will be configurable. You can change the user agent to see only specific robots, via the dropdown list in the top right next to the date range.

Conclusions

There is no doubt that much Screaming Frog SEO Spider What Log File Analyzer These are two very important tools in the work arsenal of those professionals focused on SEO services, since the information obtained with these tools not only allows us to draw up strategies to improve the loading and optimization of the sites, but also provides valuable information on the behavior of the users. crawling bots from the usual search engines.

Screaming Frog SEO Spider is one of several website data analysis tools that you should always have on hand for your projects or those of your clients.

Other tools recommended by :

Vikinguard to parse PrestaShop Stores ().
Vikinguard to parse Shop with WooCommerce in WordPress ().
Metricool to analyze websites and social networks ().
wpDoctor to analyze websites with WordPress ().

Help us improve our content by sharing your opinion

Member of the technical support team.
Coordinator of contents in the Blog and in Youtube.
Technical support in CyberProtector. Teacher at University