Cloudflare has one-click feature that allows to manage the robots.txt for you.

And once it is enabled, you will find robots.txt generated by CloudFlare:

# NOTICE: The collection of content and other data on this # site through automated means, including any device, tool, # or process designed to data mine or scrape content, is # prohibited except (1) for the purpose of search engine indexing or # artificial intelligence retrieval augmented generation or (2) with express # written permission from this site’s operator.

To request permission to license our intellectual

property and/or other materials, please contact this

site’s operator directly.

BEGIN Cloudflare Managed content

User-agent: Amazonbot
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: GPTBot
Disallow: /

User-agent: meta-externalagent
Disallow: /

END Cloudflare Managed Content

Steem to the Moon🚀!

• You can rent Steem Power via rentsp!
• You can swap the TRON:TRX/USDT/USDD to STEEM via tron2steem!
• You can swap the STEEM/SBD to SUI via steem2sui!
• You can swap the STEEM/SBD to SOL Solana via steem2sol!
• You can swap the STEEM/SBD to ETH Ethereum via steem2eth!
• You can swap the STEEM/SBD to Tether USDT (TRC-20) via steem2usdt!
• You can swap the STEEM/SBD to TRX (TRON) via steem2trx!
• You can swap the STEEM/SBD to BTS (BitShares) via steem2bts!
• Register a free STEEM account at SteemYY!
• Steem Block Explorer
• ChatGPT/Steem Integration: You can type !ask command to invoke ChatGPT
• Steem Witness Table and API
• Other Steem Tools