Let CloudFlare Manage robots.txt (AI Audit)
Cloudflare has one-click feature that allows to manage the robots.txt for you.
And once it is enabled, you will find robots.txt generated by CloudFlare:
# NOTICE: The collection of content and other data on this # site through automated means, including any device, tool, # or process designed to data mine or scrape content, is # prohibited except (1) for the purpose of search engine indexing or # artificial intelligence retrieval augmented generation or (2) with express # written permission from this site’s operator.To request permission to license our intellectual
property and/or other materials, please contact this
site’s operator directly.
BEGIN Cloudflare Managed content
User-agent: Amazonbot
Disallow: /User-agent: Applebot-Extended
Disallow: /User-agent: Bytespider
Disallow: /User-agent: CCBot
Disallow: /User-agent: ClaudeBot
Disallow: /User-agent: Google-Extended
Disallow: /User-agent: GPTBot
Disallow: /User-agent: meta-externalagent
Disallow: /END Cloudflare Managed Content
Steem to the Moon🚀!
- You can rent Steem Power via rentsp!
- You can swap the TRON:TRX/USDT/USDD to STEEM via tron2steem!
- You can swap the STEEM/SBD to SUI via steem2sui!
- You can swap the STEEM/SBD to SOL Solana via steem2sol!
- You can swap the STEEM/SBD to ETH Ethereum via steem2eth!
- You can swap the STEEM/SBD to Tether USDT (TRC-20) via steem2usdt!
- You can swap the STEEM/SBD to TRX (TRON) via steem2trx!
- You can swap the STEEM/SBD to BTS (BitShares) via steem2bts!
- Register a free STEEM account at SteemYY!
- Steem Block Explorer
- ChatGPT/Steem Integration: You can type !ask command to invoke ChatGPT
- Steem Witness Table and API
- Other Steem Tools