I implemented those recommendations in Caddy to enable a "trusted proxies" system which informs the proxy, logging, request matching etc to safely use the client IP taken from proxy headers.
For example it reminds that standards & intermediaries you trust might still allow multiple unconcatenated XFF headers though and the XFF unpeel by right to left heuristic fails if you only look at the e.g. the first such header.
This reminded me that there always seems to be a past or future pitfall case lurking, such as using old code/library honoring a X-HTTP-Method-Override header (e.g. historically used to ~convert POST to PUT bypassing client CORS restrictions).
Many holes I've noticed over the years seem to spring from legacy preservation of well intended endruns of restrictions, for example I can presently in Safari exploit a bug to get a crypto digest calculated for me within an insecure context ... I find this quite usefully locally (e.g. lan http: , data: , ~bookmarklets that I control) but nonetheless I reported the bug and am trying not to rely on it.
This was the only article that opened my eyes; it's a lot better than the Wikipedia article on the topic. It covers the XFF header, also related headers, clearly from both defense and offense perspectives.
Two things it failed to advise for defense are:
(1) I can simply just reject requests that provide multiple keys of this header. In Go, I will use `http.Header.Values(headerName)` to check the count. There is no good reason for having multiple keys of it. Any misconfiguration in setting them is the client's problem.
(2) I can and I must reject large requests that have too many header bytes. In Go, when initializing `http.Server`, I can give it the `MaxHeaderBytes` argument. Sending megabytes of headers stops here.
If I understood correctly, when wanting the rightmost-ish XFF value, I can use the rightmost value that is not in a list of trusted subnets, assuming there is at least one remaining value left of it.
XFF handling is the bug that keeps on giving. I'd estimate I've seen incorrect parsing of it in at least half of the web applications I've audited professionally.
The funniest is when the app renders user IP addresses somewhere and you can get XSS through it.
I have a rate limiting library, and for a long time, some of the most frequent issues related to misconfiguration around X-Forwarded-For headers: either ignoring them when it shouldn't and limiting the load balancer's IP instead of the end user, or blindly trusting any XFF header and allowing limits to be trivially bypassed.
Eventually I added some runtime checks that log a warning and linked to documentation for both of those issues and a few other common ones.
My support burden has decreased dramatically since then.
Brings back memories of changing the header to watch South Park episodes for free (the official site was very vulnerable to just changing the header to a NA IP).
A properly configured load balancer is going to drop this header if the client sends it, and then set it itself, with the request connection's ip being first, then the proxy ip being second. Every proxy after that should append its own ip to that header, then finally when the request reaches your app server, you should filter out your known proxy ips to be left hopefully with just the ip address of the connection the request was forwarded for, which was not set via any client header, and not able to be spoofed.
I'm sure plenty of lbs/reverse proxies and app servers don't set things, establish trust, or filter the header properly though, because, people, but it is easy to lock down.
Yeah we got dinged by our pentesters a few years ago because the LB didn't clear X-Forwarded-For headers. So you could just set some trusted IP into the X-Forwarded-For header and various ip whitelists went "Well, it came from there, so we gonna let it though".
Oops :)
It is one of these trust-based headers that need to be cleared at the edge of your network / trust zone.
I do not agree that the XFF header must be dropped and re-set. Doing so can in fact be harmful. There is a reason for preserving the chain of IPs, which is that it allows the app to use the rightmost-ish IP after skipping the known proxy IPs.
You do need to make sure only trusted parties get to add headers. Otherwise, you are letting the client freely define their own IP as seen by your server.
Dropping and re-setting the header at your trust boundary is just the simplest implementation of that. You could also count instances, or sign the header, but I don’t think there’s a standard for that, and it would mean quite some overhead.
I see it as a tradeoff. By dropping the header, one maintains trust, but one loses the ability to geolocate.
Instead of dropping, I maintain a list of trusted proxies, and I remove them from the list instead at the application level. The rightmost or final value is then the client.
If I remember correctly, it's sometimes used by nginx (used as a reverse proxy) to inject values into the request of the real ip address. e.g. nginx in front of several node processes.
I've had problems with this though. I want to get the ip4 address from cloudflare, instead of ip6 and it's next to impossible AFAICT. (for the free plan anyway)
A request doesn't come in with both an IPv4 and an IPv6. If the user connected over IPv6, the connection only has an IPv6 address. You can't get the IPv4 address, there is none.
> I've had problems with this though. I want to get the ip4 address from cloudflare, instead of ip6 and it's next to impossible AFAICT. (for the free plan anyway)
I have multiple websites on Cloudflare and can receive IPv4 addresses just fine, though for Cloudflare fronted websites it's usually better to use the CF-Connecting-IP as people can send any value in the X-Forwarded-For header.
Maybe some intermediate layer turns the IPv4 into a IPv6-mapped IPv4 address?
The XFF header is set a lot more commonly, and this gives the app the freedom to be implicitly compatible with a lot more reverse proxy servers than the Forwarded header without needing special configuration.
Moreover, the Forwarded header has all the security pitfalls of the XFF header.
1. Have (and maintain!) a list of addresses you trust to not lie (e.g. your own proxy layers, cloudflare's proxy IP list, akamai, GCP LB, AWS LB, etc…)
2. If the connecting party (real TCP connection remote end) is in the trusted list, then take the rightmost address in XFF and set as remote end.
3. Repeat 2 until you get an address not in the trusted list.
4. That is now the real client IP. Discard anything to the left of it in XFF. (though maybe log it, if you want)
The article seems to forget the step of checking the real TCP connection remote address (from my skimming), which means that if the web server can be accessed directly, and not just through a load balancer that always sets the header, then the article is a security hole.
> The article seems to forget the step of checking the real TCP connection remote address (from my skimming)
as this alerted me when reading the article to see their very important, but not highlighted, caveat emptor that covers this dangerous case :
Note that this logic assumes that your server is not directly accessible.
If it is, you need to check the actual request source IP address is one of yours first -
effectively treating that as an extra right-most address.
But when you make an assumption, you make an ass of u and mption. To have your webserver simply assume that anybody who manages to connect is trusted not a great plan.
There's a right way to do it (see my previous comment), so seems to me that one shouldn't do the wrong thing and hope that it's not a problem.
> the actual request source IP address is one of yours first
I guess this also confused me skimming. "One of yours". No, you check if it's coming from where it's supposed to be coming, I'd say. Or from the trusted list, as I'd call it.
> Dropping all external values like this is the safest approach when you're not sure how secure and reliable the rest of your call chain is going to be. If other proxies and backend apps are likely to blindly trust the incoming information, or generally make insecure choices (which we'll get into more later) then it's probably safest to completely replace the X-Forwarded-For header at that outside-world facing reverse proxy, and ditch any untrustworthy data in the process.
> Just logging the X-Forwarded-For field is not always enough as the last proxy IP address in a chain is not contained within the X-Forwarded-For field, it is in the actual IP header. A web server should log both the request's source IP address and the X-Forwarded-For field information for completeness
I implemented those recommendations in Caddy to enable a "trusted proxies" system which informs the proxy, logging, request matching etc to safely use the client IP taken from proxy headers.
For example it reminds that standards & intermediaries you trust might still allow multiple unconcatenated XFF headers though and the XFF unpeel by right to left heuristic fails if you only look at the e.g. the first such header.
This reminded me that there always seems to be a past or future pitfall case lurking, such as using old code/library honoring a X-HTTP-Method-Override header (e.g. historically used to ~convert POST to PUT bypassing client CORS restrictions).
Many holes I've noticed over the years seem to spring from legacy preservation of well intended endruns of restrictions, for example I can presently in Safari exploit a bug to get a crypto digest calculated for me within an insecure context ... I find this quite usefully locally (e.g. lan http: , data: , ~bookmarklets that I control) but nonetheless I reported the bug and am trying not to rely on it.
Two things it failed to advise for defense are:
(1) I can simply just reject requests that provide multiple keys of this header. In Go, I will use `http.Header.Values(headerName)` to check the count. There is no good reason for having multiple keys of it. Any misconfiguration in setting them is the client's problem.
(2) I can and I must reject large requests that have too many header bytes. In Go, when initializing `http.Server`, I can give it the `MaxHeaderBytes` argument. Sending megabytes of headers stops here.
If I understood correctly, when wanting the rightmost-ish XFF value, I can use the rightmost value that is not in a list of trusted subnets, assuming there is at least one remaining value left of it.
The funniest is when the app renders user IP addresses somewhere and you can get XSS through it.
Eventually I added some runtime checks that log a warning and linked to documentation for both of those issues and a few other common ones.
My support burden has decreased dramatically since then.
I'm sure plenty of lbs/reverse proxies and app servers don't set things, establish trust, or filter the header properly though, because, people, but it is easy to lock down.
Oops :)
It is one of these trust-based headers that need to be cleared at the edge of your network / trust zone.
Dropping and re-setting the header at your trust boundary is just the simplest implementation of that. You could also count instances, or sign the header, but I don’t think there’s a standard for that, and it would mean quite some overhead.
Instead of dropping, I maintain a list of trusted proxies, and I remove them from the list instead at the application level. The rightmost or final value is then the client.
[1] https://werkzeug.palletsprojects.com/en/stable/middleware/pr...
I've had problems with this though. I want to get the ip4 address from cloudflare, instead of ip6 and it's next to impossible AFAICT. (for the free plan anyway)
I have multiple websites on Cloudflare and can receive IPv4 addresses just fine, though for Cloudflare fronted websites it's usually better to use the CF-Connecting-IP as people can send any value in the X-Forwarded-For header.
Maybe some intermediate layer turns the IPv4 into a IPv6-mapped IPv4 address?
In fact if the user has IPv6, IP blocks/Rate limits wont affect the other users on the CGNAT legacy address.
Moreover, the Forwarded header has all the security pitfalls of the XFF header.
2. If the connecting party (real TCP connection remote end) is in the trusted list, then take the rightmost address in XFF and set as remote end.
3. Repeat 2 until you get an address not in the trusted list.
4. That is now the real client IP. Discard anything to the left of it in XFF. (though maybe log it, if you want)
The article seems to forget the step of checking the real TCP connection remote address (from my skimming), which means that if the web server can be accessed directly, and not just through a load balancer that always sets the header, then the article is a security hole.
> The article seems to forget the step of checking the real TCP connection remote address (from my skimming)
as this alerted me when reading the article to see their very important, but not highlighted, caveat emptor that covers this dangerous case :
But when you make an assumption, you make an ass of u and mption. To have your webserver simply assume that anybody who manages to connect is trusted not a great plan.
There's a right way to do it (see my previous comment), so seems to me that one shouldn't do the wrong thing and hope that it's not a problem.
> the actual request source IP address is one of yours first
I guess this also confused me skimming. "One of yours". No, you check if it's coming from where it's supposed to be coming, I'd say. Or from the trusted list, as I'd call it.
> Dropping all external values like this is the safest approach when you're not sure how secure and reliable the rest of your call chain is going to be. If other proxies and backend apps are likely to blindly trust the incoming information, or generally make insecure choices (which we'll get into more later) then it's probably safest to completely replace the X-Forwarded-For header at that outside-world facing reverse proxy, and ditch any untrustworthy data in the process.
X-Forwarded-For: https://en.wikipedia.org/wiki/X-Forwarded-For :
> Just logging the X-Forwarded-For field is not always enough as the last proxy IP address in a chain is not contained within the X-Forwarded-For field, it is in the actual IP header. A web server should log both the request's source IP address and the X-Forwarded-For field information for completeness
HTTP header injection: https://en.wikipedia.org/wiki/HTTP_header_injection
This OWASP page has a list of X-Forwarded-For and X-FORWARDED-foR and similar headers; "Headers for IP Spoofing" https://owasp.org/www-community/pages/attacks/ip_spoofing_vi...
A sufficient WAF should detect all such attempts.
The X-Forwarded-For Wikipedia article mentions that RFC 7239 actually standardizes the header and parsing:
RFC 7239: "Forwarded HTTP Extension" (2014): https://www.rfc-editor.org/rfc/rfc7239