You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Best I can tell, it's not possible to bind node_exporter to a unix socket, is this something that we can consider adding? (I couldn't find any issues about this after a few moments of searching, apologies of this is a duplicate issue).
@SuperQ Having some potentially-sensitive metrics in a shared environment, I'd like to reverse proxy the daemon behind nginx adding basic auth support there. It's a nice to have, but not a complete deal breaker I guess.
@mdlayher The problem is I don't trust every user on localhost, and there's metrics I want to feed through node_exporter I don't necessarily want exposed.
If you have sensitive metrics, I recommend not sending them through the node exporter, and adding them directly to your application, or writing your own exporter.
Also note that basic auth is not secure. It is sent in plain text in the http headers.
Reacted by Dominik Schulz
Reacted by Elizabeth, KSoute, Paul Querna, msr, Paul Dejean, Vinicius Fortuna, Andrew Dunham, Andrii Chubatiuk, Andreas Karlsson, Christian Koop and 10 more
"Basic auth is not secure" is also a bit of a cop out here, it's a mechanism (username and password over headers) that can be verified in a number of ways, and can be rolled easily per-platform just like API keys are.
I think the problem is that this is a pretty unusual request when the rest of the prometheus ecosystem uses and expects HTTP. I would also agree that you're probably best off doing a custom exporter as I suspect the data you're concerned about would be sourced from a text file script anyway.
@mdlayher Just because it's unusual, doesn't make it irrelevant. The alternative here would be going crazy with iptables, I suppose, but that seems bizarre when it's relatively easy to just patch node_exporter to allow unix sockets. If the Prometheus organization doesn't want to do it in repo, I guess I get to maintain my own builds instead.
Reacted by Andreas Karlsson, Sivizius and Alexander Dimitrov
Instead, a feature request from someone really wanting to use Prometheus in our infrastructure just got shut down as meaningless in under 24 hours, which rather saddens me.
The node_exporter is only one exporter of hundreds. We only scrape over TCP, so adding UNIX sockets to one exporter is now a pattern we would need to follow in all exporters. This is not something we can support.
I'm sorry if you feel this was shot down quickly, but we've had this discussion before. It's not our intention to shoot down ideas for no reason.
Our real end goal is to develop our security guides such that we can eliminate the need for reverse proxies altogether. We think the story will be TLS + client certs. But we're not security experts, so we need time to develop a proper security story.
I've been told that the node_exporter main file is basically copy and pasted between the exporters, so issuing patches would be relatively simple (this was my understanding, not something they said). The fear here is that they don't want to have to potentially support something like this on everything, when they feel like TLS auth is the way to go in the future. It has been requested that instead of this issue, I go email the prometheus-developers mailing list (which I intend to do when I get the time). Just wanted to plop this down for anybody else that stumbles upon this issue.
What's to support? Unix sockets don't behave any differently in the normal case where you don't care about the peer's address and aren't sending any fancy messages across. As far as net/http is concerned, it's the same thing as a TCP socket.
Reacted by Vinicius Fortuna, Caleb P. Burns, andrewchambers, Roman, purevg, Biling Uliar, sa, Rit, Zavier Divelbiss, Tom Foster and 6 more
Looking from a security perspective I would love to see this decision being reconsidered. Using Unix sockets is a much cleaner approach regarding locking down access and preventing information leakage. It can be locked down with user permissions and additionally with SELinux. Reading from TCP ports is usually open though, and anyone could quite easily (eg with PHP) read from it.
I think putting exporters behind a reverse-proxy is a very sensible idea. It prevents from opening a port for every exporter (about 8 in my case), that would double the amount of open ports on my system (and double the amount of possible attack vectors...). Additionally, it simplifies setting up Basic Auth and HTTPS for all exporters in one place, which seem to be re-implemented in about every exporter.
Also, it would skip the whole TCP overhead locally, while allowing me to use keep alive or connection pools towards the reverse proxy.
Additional ports cannot be opened. Port resources are controlled in certain environments. These problems can be solved by reverse proxy, so unix socket is necessary.
my 5 cents - unix sockets definitely are not the same thing as tcp, access can be restricted using file system permissions, no consumption of port numbers. it is very often used for reverse proxy configuration.
Activity
SuperQ commentedon Dec 11, 2017
Prometheus only supports polling targets over http/tcp. Adding unix socket support to the node_exporter wouldn't be very useful.
What is the use case you are trying to solve?
Omeryl commentedon Dec 12, 2017
@SuperQ Having some potentially-sensitive metrics in a shared environment, I'd like to reverse proxy the daemon behind nginx adding basic auth support there. It's a nice to have, but not a complete deal breaker I guess.
mdlayher commentedon Dec 12, 2017
Bind to localhost only and then go through nginx?
Omeryl commentedon Dec 12, 2017
@mdlayher The problem is I don't trust every user on localhost, and there's metrics I want to feed through node_exporter I don't necessarily want exposed.
mdlayher commentedon Dec 12, 2017
And nobody else on localhost can use sudo/root and view your metrics?
Omeryl commentedon Dec 12, 2017
@mdlayher correct.
SuperQ commentedon Dec 12, 2017
If you have sensitive metrics, I recommend not sending them through the node exporter, and adding them directly to your application, or writing your own exporter.
Also note that basic auth is not secure. It is sent in plain text in the http headers.
Omeryl commentedon Dec 12, 2017
@SuperQ Why .. this is such a simple thing to add in the long run (only a few lines of change to the node_exporter).
Omeryl commentedon Dec 12, 2017
"Basic auth is not secure" is also a bit of a cop out here, it's a mechanism (username and password over headers) that can be verified in a number of ways, and can be rolled easily per-platform just like API keys are.
mdlayher commentedon Dec 12, 2017
I think the problem is that this is a pretty unusual request when the rest of the prometheus ecosystem uses and expects HTTP. I would also agree that you're probably best off doing a custom exporter as I suspect the data you're concerned about would be sourced from a text file script anyway.
Omeryl commentedon Dec 12, 2017
@mdlayher Just because it's unusual, doesn't make it irrelevant. The alternative here would be going crazy with iptables, I suppose, but that seems bizarre when it's relatively easy to just patch node_exporter to allow unix sockets. If the Prometheus organization doesn't want to do it in repo, I guess I get to maintain my own builds instead.
Omeryl commentedon Dec 12, 2017
Instead, a feature request from someone really wanting to use Prometheus in our infrastructure just got shut down as meaningless in under 24 hours, which rather saddens me.
SuperQ commentedon Dec 12, 2017
The node_exporter is only one exporter of hundreds. We only scrape over TCP, so adding UNIX sockets to one exporter is now a pattern we would need to follow in all exporters. This is not something we can support.
I'm sorry if you feel this was shot down quickly, but we've had this discussion before. It's not our intention to shoot down ideas for no reason.
Our real end goal is to develop our security guides such that we can eliminate the need for reverse proxies altogether. We think the story will be TLS + client certs. But we're not security experts, so we need time to develop a proper security story.
Omeryl commentedon Dec 12, 2017
Some thoughts after discussion in IRC:
I've been told that the node_exporter main file is basically copy and pasted between the exporters, so issuing patches would be relatively simple (this was my understanding, not something they said). The fear here is that they don't want to have to potentially support something like this on everything, when they feel like TLS auth is the way to go in the future. It has been requested that instead of this issue, I go email the prometheus-developers mailing list (which I intend to do when I get the time). Just wanted to plop this down for anybody else that stumbles upon this issue.
dwfreed commentedon Dec 12, 2017
What's to support? Unix sockets don't behave any differently in the normal case where you don't care about the peer's address and aren't sending any fancy messages across. As far as net/http is concerned, it's the same thing as a TCP socket.
tdewolff commentedon Jan 6, 2024
Looking from a security perspective I would love to see this decision being reconsidered. Using Unix sockets is a much cleaner approach regarding locking down access and preventing information leakage. It can be locked down with user permissions and additionally with SELinux. Reading from TCP ports is usually open though, and anyone could quite easily (eg with PHP) read from it.
I think putting exporters behind a reverse-proxy is a very sensible idea. It prevents from opening a port for every exporter (about 8 in my case), that would double the amount of open ports on my system (and double the amount of possible attack vectors...). Additionally, it simplifies setting up Basic Auth and HTTPS for all exporters in one place, which seem to be re-implemented in about every exporter.
Also, it would skip the whole TCP overhead locally, while allowing me to use keep alive or connection pools towards the reverse proxy.
EDIT: just to show how important protecting the information scraped by Prometheus is: https://sysdig.com/blog/exposed-prometheus-exploit-kubernetes-kubeconeu/
ws-weisb commentedon Nov 13, 2024
Additional ports cannot be opened. Port resources are controlled in certain environments. These problems can be solved by reverse proxy, so unix socket is necessary.
Shaeto commentedon Nov 20, 2024
my 5 cents - unix sockets definitely are not the same thing as tcp, access can be restricted using file system permissions, no consumption of port numbers. it is very often used for reverse proxy configuration.
Add Unix domain socket listener for node_exporter prometheus#3526 pro…
Add Unix domain socket listener for node_exporter prometheus#3526 pro…
brandon1024 commentedon Oct 16, 2026
It might be worth revisiting this, given that prometheus now supports unix domain socket scrape targets: prometheus/prometheus#18091