Repository navigation
Conversation
The startup script selected the listen address with `hostname -I`, which the net-tools hostname on Azure Linux 3 does not support. NODE_IP was empty, so node-exporter listened on every address, including loopback, the node's IPv6 address on dual-stack clusters and any secondary NIC. Use the IPv4 address from kubelet's --node-ip (the Node InternalIP that scrapers use), falling back to the primary NIC's source address. Fail instead of listening on all interfaces when neither is available, and assert the listener in E2E.
Contributor
Windows Unit Test Results 3 files 18 suites 1m 6s ⏱️ Results for commit 76c2f34. ♻️ This comment has been updated with latest results. |
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
The startup script can expose the unauthenticated exporter on all IPv4 interfaces when the node IP is unspecified, and can select an unrelated address from a node label, making metrics unavailable.
2 open issues
What changed in this PR
This PR updates AgentBaker’s VHD-delivered node-exporter startup script to bind to the node’s IPv4 address, addressing unintended all-interface listening on Azure Linux 3.
Changes:
- Selects IPv4 from kubelet’s
--node-ip, with a primary-NIC route fallback. - Stops startup when address discovery fails.
- Adds ShellSpec coverage and an E2E listener check that skips older startup scripts.
| File | Description |
|---|---|
spec/parts/linux/cloud-init/artifacts/node-exporter-startup_spec.sh |
Tests address selection, fallback, and startup failure. |
parts/linux/cloud-init/artifacts/node-exporter/node-exporter-startup.sh |
Replaces hostname-based address discovery with kubelet and route lookup. |
e2e/scenario/validators.go |
Checks that port 19100 listens only on the node IP. |
Review effort: Balanced
Contributor
|
Contributor
|
Contributor
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it:
Makes node-exporter listen only on the node IP on Azure Linux 3, as it already does on Ubuntu.
node-exporter-startup.shchose the listen address withhostname -I | awk '{print $1}'. On Azure Linux 3,hostnamecomes fromnet-tools, which does not support-I, soNODE_IPwas empty and the exporter started with--web.listen-address=:19100. That means it listened on every address. The AKS VM extension's copy of the script has the same bug, so extension-managed Azure Linux 3 nodes are affected too.Listening on every address exposes the unauthenticated endpoint (TLS is off by default) beyond the node IP that scrapers use:
Change
getNodeExporterListenIPreturns the IPv4 address kubelet registers as the Node InternalIP: the IPv4 entry of--node-ipin/etc/default/kubelet. CSE sets that flag from IMDS, and it may list IPv4 and IPv6 in either order. OnlyKUBELET_FLAGStokens are read, so a node label containing--node-ip=cannot match, and unspecified or loopback addresses (0.0.0.0,127.*) are never used.--node-ipis absent (it is set only for Kubernetes ≥ 1.29, and only when IMDS succeeds), the function falls back to the primary NIC's source address for the WireServer route (ip -4 route get 168.63.129.16). That is the address cloud-node-manager assigns as InternalIP.Restart=on-failure,RestartSec=10).hostname -Ipath and the Flatcareth0path. Ubuntu's selected address is unchanged.Consumers checked
NODE_IP:19100, whereNODE_IPisstatus.hostIP, the node's IPv4 InternalIP./api/v1/nodes/<node>:19100/proxy/metrics), which connects to the node InternalIP.check_node_exporter.sh): reads the actual listen address and probes it. It already handles both wildcard and specific addresses.PrivateIP:19100.Tests
--node-ipIPv4;--node-ipin either order;--node-ip, a missing flag, or a missing file → route fallback;ValidateNodeExporternow asserts that the only:19100listener is<node IP>:19100. VHDs built before this change (used by standalone PR E2E) skip the check, because the startup script is VHD-delivered.Validation
Dual-stack Azure CNI overlay cluster on the current production images,
AKSAzureLinux-V3gen2-202610.23.0andAKSUbuntu-2404gen2containerd-202610.23.0:--web.listen-address:1910010.224.0.4:1910010.224.0.5:19100*:1910010.224.0.4:1910010.224.0.5:19100:19100:19100nodetarget (http://<node IP>:19100/metrics)upupcheck_node_exporter.sh10.224.0.4:19100. The route fallback also returned10.224.0.4on the node.node-exporter-startup_spec.shplusspec/vhdbuilder, 618 examples), shellcheck, e2ego vet.<node IP>:19100, including theSecondaryNICandSecondaryNIC_DualStackscenarios. The review fixes only change--node-ipparsing and are covered by ShellSpec and a re-check on the dual-stack Azure Linux 3 node.Follow-up
The AKS VM extension's
node-exporter-startup.shhas the samehostname -Ilogic for extension-managed nodes (nodes withoutskip_vhd_node_exporter) and needs the matching fix.Which issue(s) this PR fixes:
N/A