TLS stack–based fingerprinting and blocking

[borhanDZ]

Description

When using httpx v1.7.1, some websites respond with 403 Forbidden getting us false results, but using curl with the same URL returns a normal response 200 OK with full HTML content.
##for example:
httpx -u https://hola.example.com -no-decode -debug

While with curl:
curl -k -L https://hola.example.com -i
You get:
HTTP/1.1 200 OK
... (headers)
<html> ... full page content ... </html>

Actual behavior

httpx consistently gets 403 (Access Denied) on this site and several similar Akamai-protected domains

Additional context

• Happens even with -H "User-Agent: ...", -random-agent, or -H "Accept-Encoding: identity".

• May be related to subtle differences in TLS fingerprinting / HTTP2 vs HTTP1 negotiation that Akamai WAF detects.

• Sites behind Akamai often do bot detection using TLS fingerprints. curl has a different fingerprint than httpx.

[ehsandeep]

@borhanDZ don't think we can do much here but we've added an option to tlsi randomizer that you can try to use.

   -tlsi, -tls-impersonate  enable experimental client hello (ja3) tls randomization

[borhanDZ]

hi bro i know, i tried before but result same behavior look the screen below 403 always present while curl dump the content you can try it yourself

this is happen on many websites behind t-mobileassets i've been tested, if you want to provide you my lists just dropped here.

[ehsandeep]

@borhanDZ thanks!
I don’t think we can do much here since the WAF is doing TLS stack–based fingerprinting and blocking. curl uses a different TLS stack than Go (crypto/tls), which explains the different results.

[cdev95]

Hello all. I just now ran into this problem as well. In my particular case, I've found that the server is blocking the Accept-Charset: header. I've tried just sending the header with an empty value, but that doesn't work. I'm wondering if there is a way to edit the config.yaml file to avoid sending this header. If anyone has any suggestions, please let me know. Thanks!

Edit: I couldn't find a way to drop the header using config or options, so I just made an mitmproxy script called remove_accept_charset.py, and that works.

[Bundy01]

Hi @ehsandeep,

Can you remove Accept-Charset: utf-8 from the Httpx code? Most browsers no longer use it, and it creates a unique identifier.

[dogancanbakir]

We compared several Akamai-fronted sites with and without Accept-Charset and got the same status either way. If you have a domain where only that header makes the difference, please share it and we'll look at dropping it.