dusty.ninja

@bryan https://github.com/lwthiker/curl-impersonate but it seems to have gone stale...

@bryan @bagder Any victory (on either side) is temporary here. It‘s a war of exhaustion. If it‘s not the cipher list, then it‘ll be other properties. I‘ve heard from people measuring timings as well.

If someone does not want to serve non-browser requests, is it valid to circumvent that? Used to happen mainly for sites wanting to serve ads, but AI crawlers are the new destructive forces here.

Which is all a sad state of things. I doubt it can be solved with technology.

@bryan @bagder This is more of a privacy problem for human users than it is an overblocking problem for bots. If a site wants to block your bot, it should be able to do so.

That being said, the list of cipher suites and other TLS handshake details are probably more specific to OS and crypto library defaults than to curl. Compile curl against GnuTLS and it would probably pass through this filter. On Fedora, switch your active crypto-policy and it will definitely pass.

@bryan @bagder The privacy problem is something to be solved by the crypto libraries, although I'm not quite sure how. Maybe include an undefined code point randomly, or randomly include an old algorithm that the library would refuse to speak if the server chose it?

None of those are easy, though, because most TLS libraries don't handle establishing the connection, so if the handshake fails because of anti-fingerprinting, the lib can't transparently retry without application support.