I wondered about the robots.txt

I can see the case for it, I could also see the case for allowing at least Google to index the site.

Has there been some discussion about this previously?

  • Sam_uk@slrpnk.netOPEnglish
    1·
    2 months ago

    I think it would just be

    User-agent: *
Disallow: /
User-agent: Googlebot
Allow: /
    • poVoq@slrpnk.netMEnglish
      1·
      2 months ago

      Ok I tried to allow-list some search engine spiders in the robot.txt, however they will probably still just run into the AI scraper block if they act too shady.

      But honestly, I highly doubt we will get much traffic from Google search. It’s completely gone to shit these days.