Skip to content

Latest commit

 

History

History
152 lines (104 loc) · 27 KB

KUBERNETES.md

File metadata and controls

152 lines (104 loc) · 27 KB

Kubernetes Ingress controller

Documentation · 한국어

Ingress usage, scenarios and live qualification.

Hangang can continuously reconcile networking.k8s.io/v1 Ingress and Service objects into the live proxy configuration. Each proxy replica reads the API independently, so route availability does not depend on a leader. The older --import-ingress command remains available for offline conversion.

Generated Ingress routes preserve the original request Host header, including its port, for both host/path rules and the default backend. The Service DNS name selects the upstream connection; it does not replace the application's public host.

Deploy

Build and publish this repository's image, replace the placeholder image in deploy/kubernetes/deployment.yaml, create a private administrator token file, and apply the kustomization. The root Dockerfile needs make static first and packages only the gateway binary:

umask 077
openssl rand -hex 32 > hangang-admin-token.private
kubectl create secret generic hangang-admin \
  --from-file=token=hangang-admin-token.private
kubectl apply -k deploy/kubernetes

Keep hangang-admin-token.private out of the repository and remove the local copy after storing it in your approved secret manager. The example Secret manifest is not part of the kustomization and must not be applied with its placeholder value. The supplied emptyDir is disposable: it loses administrator accounts, sessions and local audit on Pod replacement. Replace it with an appropriately private per-Pod persistent volume before using those features; never share one SQLite account database between replicas. The controller Deployment takes routes from the Kubernetes API; it needs no route ConfigMap.

The supplied Deployment watches only its own namespace and selects class hangang. It uses a namespace-scoped ServiceAccount, Role, and RoleBinding. The Role includes Secret reads for the optional TLS mode and ingresses/status writes for optional address publication; remove those rules if those features stay disabled. A cluster-wide controller must omit --kubernetes-namespace and replace the Role/RoleBinding with an intentionally reviewed ClusterRole/ClusterRoleBinding. The Deployment's readiness probe is an HTTP check of --health-path /-/ready on the public listener, which reflects whether this replica has published a snapshot and still holds fresh, complete API caches (see Freshness and readiness); the liveness probe stays a socket check so a replica cut off from the API is drained from the Service, not restarted.

The controller uses the projected service-account CA and token paths by default. It verifies the API server against that CA, disables redirects and public trust roots, and rereads both bounded files before every new API request. Valid CA rotation atomically replaces the HTTP client; invalid or partially projected credentials cause a retry and never replace the active routing snapshot. Kubernetes documents the projected, rotating token and CA volume in Managing Service Accounts.

Useful flags are:

  • --kubernetes-controller: make Kubernetes the exclusive live configuration source.
  • --kubernetes-namespace NAME: scope all lists, watches, Secrets, and status writes to one namespace. Omit it only with cluster-wide RBAC.
  • --ingress-class NAME: select spec.ingressClassName, with the legacy annotation as a fallback. The default is hangang.
  • --kubernetes-api URL, --kubernetes-ca PATH, and --kubernetes-token PATH: override in-cluster discovery and credentials.
  • --kubernetes-tls: watch claimed TLS Secrets and publish one verified SNI certificate map with the route snapshot.
  • --kubernetes-publish-address ADDRESS: publish one IP address or hostname to the status of every admitted Ingress, merged with entries written by other replicas or controllers. Use the address clients actually connect to (the Service or load balancer in front of the fleet), shared by every replica; see Running more than one replica for why a per-replica address is not maintained.
  • --kubernetes-stale-seconds N: how long a watched resource type may go without a list, watch event, bookmark, or watch closure before this replica reports itself not ready (default 60; must exceed the 30-second watch timeout). The replica keeps serving its last snapshot either way.
  • --health-path PATH: unauthenticated readiness endpoint on the public listener, used by the supplied readiness probe (/-/ready). It answers 503 before the first snapshot, while the controller is not authoritative, and while draining.

To enable Secret-backed TLS with the supplied manifests:

kubectl patch deployment hangang --type strategic \
  --patch-file deploy/kubernetes/tls-patch.yaml

Every selected Ingress TLS entry must name a kubernetes.io/tls Secret and list at least one host. Hangang base64-decodes tls.crt and tls.key, verifies the PEM certificate/key pair, checks that every claimed host is covered, and builds the complete exact/wildcard SNI resolver before publication. A wildcard claim such as *.example.test requires a certificate that actually carries that wildcard subject alternative name; a certificate that merely covers an internal probe name is rejected. Certificate issuance and renewal remain the responsibility of cert-manager or another issuer. A deleted or malformed claimed Secret withdraws each affected Ingress together with its certificate while every other Ingress keeps serving; restoring the Secret restores the route. Only Secrets of type kubernetes.io/tls are listed and watched (the API field selector type=kubernetes.io/tls), so unrelated Secrets in the namespace are never transferred.

Hostname ownership

A shared controller must not let one namespace serve traffic for another namespace's hostname, with or without a certificate. Every selected Ingress claims a set of host patterns: each spec.rules[].host, each spec.tls[].hosts[] entry, and the catch-all pattern * for a hostless HTTP rule or a defaultBackend. Hosts must be exact DNS names or one-label wildcards (*.example.test), as the Kubernetes API already requires. Ownership is decided per pattern and per namespace, automatically and without annotations, from the objects that currently exist and nothing else:

  • A pattern belongs to the namespace of the oldest Ingress that claims it: ordered by metadata.creationTimestamp, then metadata.uid for objects created in the same second. Both are facts recorded by the API server, so every replica of the controller reaches the same decision from the same set of objects, regardless of the order in which it happened to list or watch them, and a restarted replica reaches the decision its peers already hold. Every other namespace's claim for the pattern is rejected: the claiming Ingress is withdrawn as a whole with a warning that names the contested hosts, and the older owner is never evicted or re-ordered behind it.
  • The owner keeps a pattern for as long as any of its selected Ingresses still claims it, even while that Ingress is temporarily withdrawn for another reason such as a missing Secret. When the owning namespace stops claiming the pattern, the next reconciliation grants it to the oldest remaining claimant.
  • An exact host and the wildcard covering it (api.example.test and *.example.test) cannot be owned by different namespaces. If any exact host under a wildcard is owned by another namespace and is older than the wildcard's oldest claimant, the wildcard is withdrawn; otherwise the younger exact hosts of other namespaces are withdrawn. A namespace may hold both.
  • Deleting and recreating an Ingress renews its creation timestamp. If another namespace claimed the same host in between, the recreated object is now the newcomer and stays withdrawn until the squatter is removed. Clusters in which namespaces are administered by different teams should therefore enforce host uniqueness at admission (a validating admission policy over spec.rules[].host and spec.tls[].hosts[]) rather than rely on ownership alone; the controller's rule is a fail-safe, not an access-control system.

Ownership decisions and changes are logged: a withdrawn Ingress is reported with its namespace, name, and the hosts it lost, and every transfer or release of a pattern is reported with the namespaces involved. Within a namespace several Ingresses may share a host (different paths, or one certificate reused by several objects). TLS claims follow the same rule: a namespace can only claim certificates for hosts it owns, and a claim that conflicts with another Ingress in the same namespace is withdrawn only for the later object by name.

Reconciliation behavior

Readiness also requires successful activation of the desired snapshot. A failed preparation or sink apply retains the last-good routes but withdraws readiness until reconciliation applies or confirms the desired state; fresh watch events alone cannot clear that failure. During a supervised handoff, a write freeze still tracks both loss and recovery of controller authority. Resuming a failed replacement preserves that authority state, and an endpoint explicitly withdrawn for shutdown cannot be revived by a late controller confirmation.

Initial startup lists Services and Ingresses, plus Secrets when TLS is enabled, before publishing anything. Lists use consistent paginated snapshots and preserve the server-provided continue token and collection resourceVersion. The default bounds are 500 objects per page (50 for Secrets), 10,000 objects total per resource type, 8 MiB per response, 4 MiB per watch event, and 1 MiB per decoded certificate or key. Secret lists are served from storage rather than from the API server's watch cache (which ignores limit), and a Secret page that still exceeds the response bound is requested again with half the page size, down to one object, so a few large Secrets cannot block startup or a relist.

Cached objects are stripped to the fields the controller reads: Secrets keep only their identity, type, tls.crt, and tls.key; Services keep their type and ports; Ingresses keep their spec, class annotation, and load-balancer status. Managed fields and other annotations are discarded. In addition to the object count, retained bytes are bounded at 64 MiB per resource type. A list that exceeds either bound fails and keeps the last good configuration. A watch ADDED or grown MODIFIED object that exceeds a bound is rejected, the cache is marked incomplete, and a relist is scheduled: immediately after the next deletion frees capacity, at most once every 5 seconds, or from a 60-second timer that only fires once the cache can admit more objects. An incomplete cache keeps being served but is never treated as authoritative until that relist succeeds.

Independent watches start from each list's collection resourceVersion. Added, modified, and deleted objects update the complete cache; bookmarks advance the version without changing objects. A 410 Gone event discards the cache and performs complete lists again, as required by Kubernetes API resource-version semantics. Full relists, whatever their trigger and whatever the outcome of the previous one, start at most once every 5 seconds; a replica that keeps receiving 410 therefore lists at most once per window instead of looping. Other API failures reconnect with bounded exponential backoff from 200 ms to 10 seconds; a watch that the server closes within a second of opening reconnects with the same backoff up to 5 seconds instead of immediately. Every relist spacing, reconnect, backoff, and status retry delay is jittered by ±20 % with a per-process random factor, so replicas that observe the same expiry or outage do not hit the API server in lockstep. Watch requests use a 30-second server timeout and a 40-second client timeout so token and CA rotation are observed without waiting indefinitely. Cancellation interrupts list, watch, retry, and status waits, and every watcher send is cancellation-aware, so shutdown completes even while reconciliation is slow and the internal event queue is full.

Freshness and readiness

The controller records, per watched resource type, the last moment the API server confirmed its cache: a completed list, a watch event, a bookmark, or a watch the server closed normally. A list confirms each type at the moment its snapshot resourceVersion was taken (the first page's response), not when the whole relist finished: later pages of a paginated list are served from that same snapshot, and a type listed early in a slow relist is exactly as old as that snapshot. An accepted watch request is not a confirmation by itself: a 200 only proves that the start version is still within the server's history, not that the events after it have been delivered, so freshness comes from the events, bookmarks, and normal closures that follow. A replica is authoritative while every required type (Service, Ingress, and Secret when --kubernetes-tls is set) was confirmed within --kubernetes-stale-seconds (default 60 seconds), no type's cache is incomplete, and a snapshot has been published. When a type stays silent for longer than that, or an object was rejected at capacity and the recovering relist has not succeeded yet, the replica reports itself not ready with the reason in the log, and the public --health-path endpoint starts answering 503 so the load balancer or Service stops sending it new connections. The deadline is checked while a list or a snapshot build is in progress too, so a relist that stalls on the API server withdraws readiness at the threshold rather than when the request finally returns or times out. It does not withdraw the routes it serves: a replica partitioned from the API keeps serving its last good snapshot to connections that still reach it. Readiness returns as soon as every type is fresh and complete again, whether or not the objects changed meanwhile. The threshold must exceed the watch timeout, because an idle watch only proves liveness when the server closes it.

Hangang validates a whole candidate before calling the runtime configuration sink. API failures, malformed list/watch payloads, and runtime validation failures retain the last successfully applied configuration; no list page or incomplete relist is published. Individual selected Ingresses are isolated: an unsupported field, a missing Service, an unowned hostname, or a missing or invalid claimed TLS Secret withdraws that Ingress with a warning while the others are published, so a deleted certificate cannot leave its route serving and one tenant's mistake cannot freeze another's updates.

The generated configuration is limited to 1,024 routes and 1 MiB of serialized JSON. These budgets are charged while Ingresses are admitted in namespace/name order: an object that would push the total over either budget is withdrawn on its own with a warning, and admission continues with the remaining objects. Aggregate capacity therefore never blocks the withdrawal of a deleted Secret or an unrelated update; an oversized tenant object simply stays excluded until it fits.

Ingress status is updated only for Ingresses that made it into the accepted snapshot; an Ingress withdrawn for any reason (unowned hostname, missing or invalid Secret, budget, validation) does not receive the address, and if its status still carries this controller's address from before, the address is removed. The reason for a withdrawal appears in the log. The controller merges its address into status.loadBalancer.ingress: it reads the current list, adds its own entry if absent, and never removes an entry it did not write, so replicas with different --kubernetes-publish-address values and other controllers sharing the class converge on a list that names all of them. Each patch carries the observed metadata.resourceVersion as a conflict precondition. A 409 Conflict is handled per object: the controller reads that Ingress again and, provided the re-read object is still the one the decision was made for (the same metadata.uid and metadata.generation, and still selected by the class), merges into the fresh list, skips it if the address is already present, and otherwise retries with the fresh resourceVersion up to three times with jittered backoff. If the name was meanwhile recreated as a different object, its spec changed, or it moved to another class, the queued update is dropped instead of being applied to the object the re-read returned; the next reconciliation decides again from the watch's view of the new object, so an address is never published to an Ingress that was not admitted. Status work runs in a background queue with at most four patches in flight, coalesced per object so the latest desired state wins; publication of a new snapshot never waits for the status endpoint, and a stalled or failing status API delays status only. The queue is reconciled against the cache on every publication: work for an Ingress that was deleted, left the class, or already carries the right list is dropped, so the queue holds at most one entry per live selected Ingress (the cache's own copy of the object, not a duplicate) and cannot grow under create/delete churn while the status endpoint stalls. Whatever was dropped is recomputed from the cache by the next reconciliation that needs it. Results are folded back into the cache so unrelated events do not repeat a patch. The status permission is unnecessary when the flag is absent. Kubernetes RBAC remains deny-by-default as described in the authorization documentation.

Ingress Exact, Prefix, and ImplementationSpecific paths map to Hangang exact, segment_prefix, and raw prefix. Named and numeric TCP Service ports are supported, and upstreams use SERVICE.NAMESPACE.svc:PORT, leaving endpoint selection to Kubernetes networking. Rules are sorted deterministically the way Kubernetes selects hosts: routes for an exact host come first, then one-label wildcard hosts, then hostless rules, and among equally specific hosts the longest path, then an exact path match, then the generated route ID decide. A wildcard or hostless rule can therefore never capture a request for an exact host on a longer path. Kubernetes wildcard hosts match one leading DNS label.

Running more than one replica

Each replica runs its own controller against the API server and publishes into its own proxy; there is no leader election and no shared state between replicas. The design keeps replicas interchangeable:

  • Hostname ownership is a pure function of the current objects (oldest claimant wins), so replicas that observed different event histories, or that restarted at different times, publish identical routing for identical objects.

  • Readiness is per replica. A replica whose watches go stale, or whose cache is incomplete, is taken out of rotation by the readiness probe while it keeps serving its last snapshot; the others continue. Set replicas in deploy/kubernetes/deployment.yaml and keep the HTTP readiness probe on --health-path; a plain socket probe would keep advertising a stale replica.

  • Status addresses are merged, so replicas may publish the same address or different ones; nothing is overwritten. Each replica adds and removes only the address it is configured with, and nothing else: there is no liveness record behind the list, and no replica ever removes another replica's entry. An address published by a replica that was then scaled away, crashed, or restarted with a different --kubernetes-publish-address therefore stays in status.loadBalancer.ingress of every Ingress it was admitted for, indefinitely, and anything that consumes that status (an external DNS controller, service discovery, dashboards) keeps treating it as a live endpoint. Graceful shutdown deliberately does not remove the entry either: a rolling update or a supervised generation replacement restarts a replica with the same address, and removing and re-adding it would flap every consumer of the status for no reason.

  • Because of that, run replicas with one shared address: the IP address or DNS name of the Service or load balancer in front of the fleet, which is what clients actually connect to and which stays valid across restarts, rescheduling, and scaling. The list then names one endpoint that no single replica owns, and it is only ever wrong when the load balancer itself changes. Per-replica addresses (a node IP or pod IP per replica) are appropriate only when something else keeps the list current; Hangang does not. A per-Ingress heartbeat annotation was considered and rejected for that job: it needs write access to Ingress metadata rather than the status subresource alone, costs one write per replica per Ingress per period plus a watch event for every watcher of Ingresses in the cluster, depends on clocks, and cannot tell a dead replica's entry from an entry written by another controller that legitimately publishes the same address.

  • To retire an address that no live replica publishes any more, remove it from the status lists once; the replicas that remain leave the rest of the list intact:

    RETIRED=192.0.2.7
    kubectl get ingress -A -o json \
      | jq -r --arg a "$RETIRED" '.items[] | select(.status.loadBalancer.ingress[]? | .ip == $a or .hostname == $a) | "\(.metadata.namespace) \(.metadata.name)"' \
      | while read -r ns name; do
          kubectl patch ingress -n "$ns" "$name" --subresource=status --type=merge -p "$(
            kubectl get ingress -n "$ns" "$name" -o json \
              | jq -c --arg a "$RETIRED" '{status:{loadBalancer:{ingress:[.status.loadBalancer.ingress[] | select(.ip != $a and .hostname != $a)]}}}'
          )"
        done
  • Relists are spaced and every delay is jittered per process, so a fleet that receives the same 410 Gone or the same outage does not relist in lockstep.

Replicas still share the API server's rate limits: N replicas perform N lists at startup and hold N watches per resource type, which is the same load as N independent clients.

Try the Ingress adapter without a cluster

The example Kubernetes List contains one Service and one Ingress. It exercises an exact /health path, a path-element prefix / path, class selection, and a named Service port. From the repository root:

cargo run --quiet --bin hangang -- --import-ingress examples/kubernetes/ingress-list.json --ingress-class hangang \
  | python3 -c 'import json,sys; d=json.load(sys.stdin); assert [r["path_match"] for r in d["http"]]==["exact","segment_prefix"]; assert all(r["backends"]==["http://api.demo.svc:8080"] for r in d["http"]); print("Ingress import: 2 ordered routes")'

This verifies conversion only; the sample Service has no backing Pods. The disposable kind qualification below tests the live controller and a real backend. The supplied deploy/kubernetes manifests likewise provide the proxy controller and its public Service, not an application backend or Ingress object. Replace the placeholder image and create the admin Secret as shown above, then supply a real application Service and Ingress. The controller currently selects the class name string without reading an IngressClass object; review class registration separately for your cluster. Gateway API resources are not currently reconciled.

Verification

The mock HTTPS API test covers CA verification and rotation, token rotation, pagination, Secret SNI material, deletion, watch reconnect, 410 Gone, relist outage retention, status gating, and cancellation, plus cross-namespace hostname claims against an older owner, two replicas with different event histories publishing identical snapshots, readiness expiring while watches are stale and returning when the API resumes, an incomplete cache reporting not ready until its relist succeeds, two replicas merging different status addresses, a withdrawn Ingress losing and never receiving the address, publication proceeding while the status endpoint stalls, relist spacing under repeated 410, oversized Secret pages, recovery of an object rejected at capacity, cancellation while watchers block on a full queue, per-object status conflict retries, the status queue staying bounded under create/delete churn while every patch stalls, a conflict re-read that returns a recreated object of another class being dropped rather than patched, readiness expiring while a relist is stalled on the API server, a completed list not refreshing a type whose snapshot is already older than the threshold, and an accepted-but-silent watch not restoring readiness. Unit tests cover route ordering, per-Ingress budgets, wildcard certificate claims, cache stripping and byte bounds, oldest-wins ownership (timestamp, uid tiebreak, undated objects, history independence), exact/wildcard ownership overlap, status list merging, status queue coalescing, the queue following the live cache (deleted, unselected, and already-correct objects are pruned; the object limit caps it; released objects are not retained), the uid/generation/class binding of conflict re-reads, and jitter bounds. The disposable kind qualification creates a unique cluster, Docker network, image, and kubeconfig; it verifies live TLS routing, status publication, Secret rotation/deletion, Ingress deletion/recreation, and a supervised SIGHUP generation replacement, then removes only those owned resources:

python3 tests/kubernetes_cluster.py

Set HANGANG_BINARY=/absolute/path/to/hangang to qualify an already built executable. The script skips only when its repository-local target/test-tools/kind binary is absent; it never uses or changes the user's current Kubernetes context.

Pumgye source diagnostics in the console

The integrated Pumgye page shows the source phase, authority availability, last applied policy revision and reported source lease remaining at the last refresh. Refresh retrieves a new observation; the displayed countdown is not a permission lease. Source failure clears prior status. Unknown source ownership leaves native policy editing disabled. Kubernetes-owned policies remain inspectable but must be changed through their owning source; Kubernetes configuration editing and full four-component deployment remain implementation gates.

Native Service discovery

Native Hopae, Pumgye and Sumunjang endpoints accept validated Service DNS names with fixed ports. DNS names require mTLS; localhost is not treated as authenticated loopback. DNS is resolved when opening a pooled connection, within the existing request deadline. Connection reuse and retirement stay unchanged. This enables ClusterIP and headless Service addressing, not a completed membership protocol. For Pumgye's HTTPS management endpoint, a DNS endpoint must match the configured TLS server name; literal IP endpoints retain the existing fixed-name resolution override. Use the Service DNS in the server certificate SAN. Hopae console SSO uses the same native address contract. No change activates Kubernetes policy authority, weakens native TLS or completes the four-component package by itself.