Pro@programming.dev to Technology@lemmy.worldEnglish · 4 days agoCodeberg: army of AI crawlers are extremely slowing us; AI crawlers learned how to solve the Anubis challenges.i.imgur.comimagemessage-square234linkfedilinkarrow-up11.15Karrow-down112file-text
arrow-up11.14Karrow-down1imageCodeberg: army of AI crawlers are extremely slowing us; AI crawlers learned how to solve the Anubis challenges.i.imgur.comPro@programming.dev to Technology@lemmy.worldEnglish · 4 days agomessage-square234linkfedilinkfile-text
minus-squareT156@lemmy.worldlinkfedilinkEnglisharrow-up8·4 days agoThe website would also have to display to users at the end of the day. It’s a similar problem as trying to solve media piracy. Worst comes to it, the crawlers could read the page like a person would.
minus-squareacockworkorange@mander.xyzlinkfedilinkEnglisharrow-up1·8 hours agoYou could have a server for open code access with very limited bandwidth and another for authenticated users with higher bandwidth.
The website would also have to display to users at the end of the day. It’s a similar problem as trying to solve media piracy. Worst comes to it, the crawlers could read the page like a person would.
You could have a server for open code access with very limited bandwidth and another for authenticated users with higher bandwidth.