AGI Agent
AI2Bot
AI2Bot-DeepResearchEval
AIWebIndex
AddSearchBot
AgentDataBot
AgentTimes
Ai2Bot-Dolma
AmazonBuyForMe
Amazonbot
Amzn-SearchBot
Amzn-User
Andibot
Anomura
ApifyBot
ApifyWebsiteContentCrawler
Applebot
Applebot-Extended
Aranet-SearchBot
Awario
AzureAI-SearchBot
BixelBot
Bravebot
Brightbot
Brightbot 1.0
BuddyBot
Bytespider
CCBot
CartAI Agentic Commerce
Channel3Bot
ChatGLM-Spider
ChatGPT Agent
ChatGPT-User
Claude-Agent
Claude-Code
Claude-SearchBot
Claude-User
Claude-Web
ClaudeBot
CledaraBot
CloudVertexBot
Cloudflare-AutoRAG
CloudflareBrowserRenderingCrawler
Code
Codex
Cotoyogi
CragCrawler
Crawl4AI
Crawlspace
Cursor
Datenbank Crawler
DeepSeekBot
Devin
Diffbot
Diffbot-User
Dot
DoubaoBot
DuckAssistBot
ERNIEBot
Echobot Bot
EchoboxBot
ElevenlabsBot
ElevenlabsCrawler
ExaBot
ExaSearchBot
ExaUser
FacebookBot
Factset_spyderbot
FirecrawlAgent
FriendlyCrawler
GPTBot
GeistHaus-PageFetcher
Gemini-Deep-Research
Google-Agent
Google-CloudVertexBot
Google-Extended
Google-Firebase
Google-Gemini-CLI
Google-GeminiNotebook
Google-NotebookLM
GoogleAgent-Mariner
GoogleAgent-URLContext
GoogleOther
GoogleOther-Image
GoogleOther-Video
Grok Bot
GrokAppAndroid
GrokBot
Haiku-SearchBot
HarkBot
HatchFeed
HenkBot
Hunyuan
ICC-Crawler
ISSCyberRiskCrawler
ImagesiftBot
Instinct
Kangaroo Bot
KeenableBot
Kimi-Agent
Kimi-SearchBot
Kimi-User
KimiBot
KlaviyoAIBot
KunatoCrawler
LAIONDownloader
LCC
Lightpanda
Liner-User
LinerBot
Linguee Bot
LinkupBot
Manus-User
Meta-ExternalAgent
Meta-ExternalFetcher
MiniMaxAgent
MistralAI-Index
MistralAI-Training
MistralAI-User
MistralAI-User/1.0
MoonshotBot
Mozilla-Tabstack
Muse
MyCentralAIScraperBot
NagetBot
NotebookLM
NovaAct
OAI-AdsBot
OAI-SearchBot
OpenAI
Operator
PanguBot
Panscient
Payhawk Invoice Fetching Agent
Perplexity-User
PerplexityBot
PetalBot
PhindBot
Poggio-Citations
Poseidon Research Crawler
QualifiedBot
Querit-SearchBot
QueritBot
QuillBot
QwenBot
Reflectionbot
SBIntuitionsBot
Scrapy
ScryBot
SemrushBot-OCOB
SemrushBot-SWA
Shap-User
ShapBot
Sidetrade indexer bot
Spider
TavilyBot
Terra Cotta
TerraCotta
Thinkbot
TikTokSpider
Timpibot
TongyiBot
Trae
TwinAgent
UseAI
VelenPublicWebCrawler
WARDBot
WRTNBot
Webzio-Extended
YaK
YandexAdditional
YandexAdditionalBot
YiBot
YiyanBot
YouBot
ZanistaBot
__usage_ai-training
aiHitBot
amazon-QBusiness
amazon-kendra
anthropic-ai
atlassian-bot
bedrockbot
bigsur.ai
cohere-ai
cohere-training-data-crawler
facebookexternalhit
iAskBot
iaskspider
iaskspider/2.0
imageSpider
img2dataset
kagi-fetcher
laion-huggingface-processor
meta-externalagent
meta-externalfetcher
meta-webindexer
nbot
netEstate Imprint Crawler
newsai
omgili
omgilibot
opencode
panscient.com
qodercli
quillbot.com
webzio-extended
wpbot
xAI-Grok
