📖 Quick Summary
Tool: Screaming Frog SEO Spider
Difficulty: Beginner
Reading Time: 6 minutes
Business Impact: ⭐⭐⭐⭐☆
Directives Explained
Robots Directives tell search engines how they should crawl, index and interact with your website. They act as instructions for search engine crawlers, helping control which pages should appear in search results and which should remain hidden.
When configured correctly, Robots Directives improve crawl efficiency, protect low-value content from being indexed and help search engines focus on the pages that matter most. Incorrect configuration, however, can accidentally prevent important pages from appearing in Google, resulting in lost organic traffic and missed business opportunities.
Robots.txt
What this means
The robots.txt file provides search engines with instructions about which areas of your website they are allowed or not allowed to crawl.
Why this matters to the business
A properly configured robots.txt file helps search engines focus on valuable content while preventing unnecessary crawling of low-value pages, improving crawl efficiency and protecting sensitive areas of the website.
How Growth Architect AI identifies this
By analysing your website’s robots.txt file and comparing its directives against the URLs discovered during the Screaming Frog SEO Spider crawl.
What to do about it
Review your robots.txt file regularly to ensure important pages remain crawlable while only blocking content that genuinely should not be accessed by search engines.
Related terms
Meta Robots • Noindex • X-Robots-Tag
Meta Robots
What this means
A Meta Robots tag is placed within a webpage’s HTML and tells search engines how that individual page should be crawled and indexed.
Why this matters to the business
Meta Robots directives provide page-level control, allowing you to prevent specific pages from being indexed while keeping the rest of your website visible in search engines.
How Growth Architect AI identifies this
By analysing the Meta Robots directives present within every page discovered during the Screaming Frog crawl.
What to do about it
Ensure important pages allow indexing while only applying restrictive directives to pages that should remain out of search results.
Related terms
Noindex • Nofollow • Robots.txt
Noindex
What this means
The Noindex directive instructs search engines not to include a page within their search index.
Why this matters to the business
Pages marked as Noindex cannot generate organic search traffic regardless of their content quality. Accidentally applying Noindex to important pages can significantly reduce visibility and revenue opportunities.
How Growth Architect AI identifies this
By detecting Noindex directives contained within Meta Robots tags or HTTP response headers.
What to do about it
Only use Noindex on pages that genuinely should not appear in search results, such as thank-you pages, internal search results or duplicate content.
Related terms
Meta Robots • X-Robots-Tag • Indexability
Nofollow
What this means
The Nofollow directive tells search engines not to follow the links found on a webpage.
Why this matters to the business
Using Nofollow incorrectly may prevent search engines from discovering important pages, reducing crawl efficiency and limiting the flow of authority throughout your website.
How Growth Architect AI identifies this
By analysing Meta Robots directives assigned to every crawled page.
What to do about it
Use Nofollow only where appropriate, such as sponsored or user-generated content. Avoid applying it to pages that should help search engines discover your website.
Related terms
Internal Links • External Links • Meta Robots
X-Robots-Tag
What this means
The X-Robots-Tag is an HTTP response header that provides search engine directives without requiring changes to the page’s HTML.
Why this matters to the business
X-Robots-Tag allows indexing rules to be applied to non-HTML resources such as PDFs, images and other downloadable files.
How Growth Architect AI identifies this
By analysing HTTP response headers returned during the Screaming Frog crawl.
What to do about it
Ensure X-Robots-Tag directives support your indexing strategy and do not unintentionally block valuable content from appearing in search results.
Related terms
Meta Robots • Noindex • Robots.txt
Why Growth Architect AI analyses Robots Directives
Growth Architect AI doesn’t simply identify blocked pages or Noindex directives. It combines Screaming Frog crawl data with Google Search Console indexing information to determine whether important business pages are being unintentionally excluded from search results. This allows Technical SEO issues to be prioritised based on their potential impact on organic visibility, leads and revenue.