What is Meta-ExternalAgent and should you block it?
meta-externalagent · Meta · Training crawler
Meta-ExternalAgent (user agent token meta-externalagent) is Meta's web crawler. Meta says it is used for training foundation AI models or improving products by indexing content directly. Blocking it does not affect Google or the major AI answer engines.
What it does
meta-externalagent collects page content to be used in model training. Collects content for Meta AI model training.
It does not execute JavaScript. Whatever your server returns in the initial HTML response is the entire page as far as meta-externalagent is concerned — if your content is rendered client-side, this crawler reads an empty container.
Meta-ExternalAgent vs Meta's other fetchers
Meta runs separate agents. facebookexternalhit fetches a link when it is shared on Facebook, Instagram or WhatsApp, to build the preview card; block it and your shared links lose their title, description and image. Meta-ExternalFetcher fetches individual links when a user asks for them, and Meta says it may bypass robots.txt. Meta-ExternalAgent is the one that crawls for AI training and indexing.
So a rule for meta-externalagent leaves your link previews intact. Meta says robots.txt changes can take up to 24 hours to apply.
meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)Should you block meta-externalagent?
It is a licensing decision, not a visibility one. meta-externalagent collects content that may be used to train AI models; blocking it does not remove you from Google Search or from AI answer engines.
Your content is not collected for Meta AI model training. Meta's assistant surfaces that rely on live retrieval are governed separately, so this is a training-data decision.
To allow it
User-agent: meta-externalagent
Allow: /To block it
User-agent: meta-externalagent
Disallow: /Put the rule in your robots.txt at the site root. A group that names meta-externalagent explicitly takes precedence over your User-agent: * group, so naming it is how you make your intent survive later edits to the wildcard rules.
Verifying a real meta-externalagent request
Meta publishes IP ranges for its crawling infrastructure. Verify against those; the user agent string alone tells you nothing you should act on.
The mistake people make
Blocking only the older Meta crawler token and assuming coverage. Meta has used more than one name for this function over time, and an outdated robots.txt entry names a bot that no longer visits.
Frequently asked questions
- What is meta-externalagent?
- Meta-ExternalAgent (user agent token meta-externalagent) is Meta's web crawler. Meta says it is used for training foundation AI models or improving products by indexing content directly. Blocking it does not affect Google or the major AI answer engines.
- Should I block meta-externalagent?
- It is a licensing decision, not a visibility one. meta-externalagent collects content that may be used to train AI models; blocking it does not remove you from Google Search or from AI answer engines.
- How do I block meta-externalagent in robots.txt?
- Add a group to the robots.txt at your site root: "User-agent: meta-externalagent" followed by "Disallow: /". Use "Allow: /" instead to allow it explicitly.
- Will blocking meta-externalagent break my Facebook link previews?
- No. Link previews are fetched by facebookexternalhit, a separate agent. A rule for meta-externalagent does not apply to it.
Further reading
- The Complete Guide to robots.txt for AI Crawlers: Which Bots to Allow
GPTBot vs OAI-SearchBot, ClaudeBot vs Claude-User, Googlebot vs Google-Extended: which AI crawlers to allow, which to block, and copy-ready robots.txt templates.
- How to Create and Optimize an llms.txt File for Your Website
The llms.txt format, where the file goes, how to keep it token-efficient, working examples for Next.js and static sites, and an honest account of who uses it.
- What is Generative Engine Optimization (GEO)? The Complete 2026 Guide
How ChatGPT Search, Perplexity and Claude find, read and cite pages, why that differs from Googlebot, and what the research says actually improves AI visibility.
Test your site against 15 AI crawlers now
The free scan parses your robots.txt, evaluates every AI crawler against it, and tells you how much of a page meta-externalagent would actually come away with.
Run the free scan