Read the identity
A request identifies itself as a supported crawler. The name selects the appropriate verification method.
TRABOTPRO / BOT MANAGEMENT
A bot name is a claim.
Verify it.
Recognise legitimate crawlers, catch impersonation, and make access a deliberate policy decision.
01 / IDENTITY BEFORE ACCESS
A familiar name.
Evidence to back it up.
Anyone can send a Googlebot User-Agent. TraBotPro checks supported crawler claims against the verification method registered for that bot, then applies your policy.
Check the origin
Use the bot’s supported address lists, forward-confirmed reverse DNS or network ownership checks.
Apply your policy
Verified identity informs access. It does not automatically give every bot unlimited access to your store.
Identity confirmed
Apply the configured bot policy, subject to higher-priority rules. Decide whether this bot should be allowed, scored, limited or blocked.
Impersonation detected
A supported identity check rejects the claim. TraBotPro blocks in enforcement mode and records “would block” in monitor mode.
Uncertainty stays uncertainty
A check still running, unavailable data or a DNS timeout does not by itself prove impersonation. Ordinary risk evaluation continues.
Verification methods vary by bot. Unknown automation is not automatically a known crawler, and a User-Agent is never sufficient proof.
02 / DIFFERENT JOBS. DIFFERENT POLICIES.
Choose what your store
makes room for.
Search discovery, AI training and SEO analysis serve different purposes. Give them deliberate policies rather than treating every crawler alike.
Keep discovery working
Verify legitimate search crawlers and decide how they can access your catalogue.
Googlebot · Bingbot · ApplebotSupport AI discovery
Control crawlers and user-triggered fetchers separately from bots categorised for training.
OAI-SearchBot · ChatGPT-User · PerplexityBotMake training access a choice
Set access for recognised training crawlers according to your store’s policy.
GPTBot · ClaudeBot · CCBotManage crawl intensity
Let useful analysis run at a suitable pace rather than granting unlimited application access.
AhrefsBotKeep shared links useful
Recognise supported services fetching previews while retaining control over access.
facebookexternalhit · PinterestbotProtect essential checks
Use supported bot identity or a scoped exception for your own uptime monitors and integrations.
Known fetchers · configured monitor exceptionsExamples from the documented bot registry. Coverage and verification depend on the installed version and available verification data.
03 / YOUR ACCESS POLICY
Verified does not mean
unrestricted.
Choose an action per category or per bot, with shared fleet policy and more specific site settings.
Allow
Allow the verified bot without a browser challenge or risk score, subject to higher-priority controls.
Score
Evaluate the verified bot through the ordinary risk engine. Identity becomes context rather than an automatic pass.
Rate limit
Allow requests within the configured bot budget. Above it, return 429 with Retry-After when supported and enforcing.
Block
Refuse the verified bot with a 403 response in enforcement mode. Monitor mode records the proposed action.
Follow robots.txt
Allow the verified bot where the host’s available robots.txt rules permit it. Refuse disallowed paths when enforcing.
ILLUSTRATIVE CONFIGURATIONOne store.
One store.
Several sensible decisions.
These are example choices, not a promise of your current configuration.
- Verified search engine
- Allow
- Recognised AI training crawler
- Block
- Verified SEO crawler
- Rate limit
- Failed Googlebot identity check
- Block impersonation
- Verification unavailable
- Ordinary risk evaluation
04 / BUILT FOR REAL OPERATIONS
Make crawler policy
safe to change.
Fleet policy, local control
Start with shared category rules. Override a category or an individual bot for a specific store, and see which policy source determines the outcome.
Monitor before blocking
Review proposed blocks and limits before enforcement. Confirm that legitimate search engines, previews and integrations are receiving the intended response.
Investigate the reason
Inspect crawler claims, verification outcomes and policy decisions. Distinguish an identity failure from a configured block on a genuine crawler.
Respect the integration limits
Bot rate-limit enforcement requires the supported Lua integration. With auth_request, a rate-limit action is recorded rather than returned as a 429.
Check the real client address
Correct CDN or load-balancer configuration matters. Verification must assess the intended client, not accidentally treat the proxy address as the crawler.
Account for other controls
A TraBotPro allow does not bypass application authentication or independent hosting limits. A 429 response can originate outside TraBotPro.
COMMON QUESTIONS
A few useful distinctions.
Does a DNS timeout mean a bot is fake?
No. Pending and unknown outcomes continue through ordinary risk evaluation. A failed identity claim and an unavailable verification service are different situations.
Does robots.txt block unwanted bots by itself?
No. It normally communicates crawling preferences. TraBotPro’s robots.txt-aware policy can enforce available rules for supported, verified bots. If no valid rules are available, that policy does not invent a denial.
Can I block all AI access with one crawler rule?
A rule applies to the recognised bot or category. It does not prove that every unrecognised client with an AI-related purpose will be identified. Unknown traffic still requires the other protection layers.
Can my own monitor receive an exception?
Yes. Use a scoped exception matching the intended monitor or integration. A User-Agent alone should not grant trust, and the application must still validate any credentials it requires.
SEE THE DECISION IN CONTEXT
Follow a crawler request.
Compare a verified Googlebot, an impersonator and a rate-limited SEO crawler.
MAKE ROOM FOR REAL CUSTOMERS
Your traffic deserves
a smarter gatekeeper.
Start with your setup. Build a rollout brief for your TraBotPro contact.