pdettori · Oct 28, 2026 · Oct 26, 2026 · ingpaolodettori-dev · Oct 27, 2026 · ingpaolodettori-dev
@@ -273,6 +273,7 @@ through the environment of the one `jq` that writes each Secret, and are applied
 | `moca-relay`          | `moca`         | `SH_RELAY_TOKEN`, `MOCA_RELAY_EXEC_TOKEN`                                                                | relay (both, env); supervisor (`MOCA_RELAY_EXEC_TOKEN` only, env)                                                                           |
 | `moca-redis`          | `moca`         | `REDIS_PASSWORD`, `REDIS_URL`, `redis.conf`                                                              | Redis (`redis.conf` as a file; `REDIS_PASSWORD` as `REDISCLI_AUTH` for probes); supervisor, relay, control plane, provisioner (`REDIS_URL`) |
 | `moca-mu1`            | `moca`         | `SH_SESSION_TOKEN_PRIVATE_KEY`, `SH_SESSION_TOKEN_PUBLIC_KEYS`, `SH_CREDENTIAL_KEK`, `SH_EXCHANGE_TOKEN` | control plane (private key, KEK, exchange token as files); supervisor (exchange token as a file, public keyset as env)                      |
+| `moca-test-users`     | `moca`         | `SH_TEST_USER_SECRET` (only while `SH_TEST_USERS` > 0; deleted when off)                                 | control plane (as a file, from an `optional` projected source; ADR-0045, §13)                                                               |
 | `moca-relay-attach`   | `moca-sandbox` | `SH_RELAY_TOKEN` (copied from `moca-relay`)                                                              | sandboxes                                                                                                                                   |
 | `moca-supervisor-tls` | `moca` (OCP)   | `tls.crt`, `tls.key`                                                                                     | ghostunnel sidecar                                                                                                                          |
 
@@ -1116,3 +1117,106 @@ The §12.3 tenant verified all of it: the operator's Let's Encrypt wildcard for
 `*.moca-ns.vpc-int.res.ibm.com` (served by `--tls-secret`, and also covering the edge Route as the
 controller's default certificate), a namespace-scoped ingress controller admitted by the
 `policy-group.network.openshift.io/ingress` selector, and mocactl logins and turns over the Routes.
+
+## 13. Test users, for unattended multi-user testing (`SH_TEST_USERS`)
+
+> **Preview.** Test-user logins are implemented but not yet declared Stable; interfaces and limits
+> may change. Known limits: dev and test only, container tier only, and not available on the
+> sandbox profile (see below and [ADR-0045](../../docs/adrs/0045-test-user-logins.md)).
+
+Every login is GitHub's device flow, which needs a person and a browser. `SH_TEST_USERS=N` adds N
+synthetic subjects `test:1` … `test:N` that log in with a shared secret instead, so a script can
+exercise the multi-user claims — each subject spending only its own credential, and one user being
+refused another's session. A test user is configured exactly like a GitHub user: the control plane
+issues the same refresh family and the same token lifetimes, so rotation, roles
+(`SH_ADMIN_SUBJECTS`), the ownership index and the audit trail behave identically.
+
+**It is off unless asked for,** and gated by a secret, because the endpoint issues a real login for
+an arbitrary subject: without one, anyone who can reach the control plane is every test user.
+`POST /v1/auth/test-token` answers 404 while `SH_TEST_USERS` is unset, the route is not mounted on
+the sandbox profile, and the control plane refuses to boot if the two are combined.
+
+### 13.1 Bring it up
+
+`SH_TEST_USERS` is sticky like every other setting (section 2); `SH_TEST_USERS=` turns it off and
+deletes the secret with it. On `--target ocp` -- the shared-cluster target -- it additionally needs
+`SH_TEST_USERS_ACK=i-know-this-is-not-for-production`, so a stale export cannot quietly leave a
+password-gated login path on a cluster other people use.
+
+```bash
+export LOG_DIR=/tmp/kagenti/tdd/moca; mkdir -p "$LOG_DIR"
+SH_TEST_USERS=10 SH_GITHUB_CLIENT_ID=<client id> SH_SINGLE_NAMESPACE=<ns> \
+  deploy/k8s/setup.sh --target ocp-single --skip-build \
+  --image <registry>/moca:<tag> --sandbox-image <registry>/moca-remote-worker:<tag> \
+  >"$LOG_DIR/t-users-setup.log" 2>&1; echo "EXIT:$?"
+```
+
+**Look for:** `WARNING: SH_TEST_USERS=10: test:1..test:10 can log in with the shared secret`, and
+either `generated SH_TEST_USER_SECRET` or nothing (a re-run keeps the stored one and never rotates
+it). The secret lives in a Secret of its own, `moca-test-users`, and reaches the control plane as a
+credential **file** under `CREDENTIALS_DIRECTORY`, never an environment line. Its own Secret
+because the credential loader treats an empty file as an error and only an _absent_ one as "unset":
+`setup.sh` therefore writes it only while test users are on and **deletes** it when they go off, and
+`control-plane.yaml` projects it from an `optional` source so a stack without it still boots. Give your own instead of a generated one
+with `SH_TEST_USER_SECRET=<at least 16 characters>`.
+
+```bash
+kubectl -n <ns> get secret moca-test-users -o jsonpath='{.data.SH_TEST_USER_SECRET}' | base64 -d
+```
+
+### 13.2 Check the gate first
+
+Before trusting the feature, prove it is invisible without it. On a stack where `SH_TEST_USERS` is
+unset this must answer `404`:
+
+```bash
+curl -s -o /dev/null -w '%{http_code}\n' -X POST -H 'Content-Type: application/json' \
+  -d '{"index":1}' http://127.0.0.1:8090/v1/auth/test-token
+```
+
+### 13.3 One user, by hand
+
+The secret travels in `MOCA_TEST_USER_SECRET`, never a flag — a flag would land in the shell history
+and in every `ps` listing on the machine.
+
+```bash
+export MOCA_TEST_USER_SECRET=<the secret>
+mocactl --control-plane-url http://127.0.0.1:8090 login --test-user 3
+# logged in as Test User 3 (test:3)
+```
+
+### 13.4 N users, and the cross-user case
+
+`deploy/k8s/test-users.py` drives N users (default 10, `--users N`) through real `mocactl`
+commands, then asserts the cross-user claims. `--api` runs the same claims straight against the
+HTTP API, which separates a client bug from a server one.
+
+```bash
+export MOCA_TEST_USER_SECRET=<the secret>
+./deploy/k8s/test-users.py --users 10 --control-plane-url http://127.0.0.1:8090
+./deploy/k8s/test-users.py --users 10 --api --json
+```
+
+**Look for:** `users 10/10 fully ok; cross-user 4/4` and `RESULT: PASS`.
+
+Each user gets its own `XDG_CONFIG_HOME`, because `mocactl` keeps one login in one `auth.json`
+(`packages/mocactl/src/config.ts`): a second `login` in the same directory overwrites the first, so
+one process in one directory cannot hold ten logins. That is also what makes `--jobs` safe.
+
+The operator's check, that each turn spent its own user's credential — the audit records credential
+**names**, never values:
+
+```bash
+kubectl -n <ns> exec redis-0 -- sh -c \
+  'redis-cli --no-auth-warning -a "$REDISCLI_AUTH" XREVRANGE sh:cp:audit + - COUNT 300' |
+  grep -A1 '^subject$' | grep '^test:' | sort -u    # ten distinct subjects
+```
+
+**Stop any `port-forward` on the harness port (`8080`) and on `18090` before running `smoke.sh`**:
+it binds `18090` itself and reaches the harness at the URL the control plane advertises, so a
+leftover forward makes every `/turn` answer 404. Section 9.3 gives the same warning for its
+connection count.
+
+Tested on OpenShift 4.x (IBM Cloud ROKS, `--target ocp-single`, edit-only namespace rights),
+2026-10-26: N=10 passed in both CLI and `--api` modes, with ten distinct `test:N` subjects and ten
+credential names in the audit stream.
@@ -61,6 +61,13 @@ spec:
               valueFrom: { configMapKeyRef: { name: moca-settings, key: SH_GITHUB_CLIENT_ID } }
             - name: SH_ADMIN_SUBJECTS
               valueFrom: { configMapKeyRef: { name: moca-settings, key: SH_ADMIN_SUBJECTS } }
+            # ADR-0045: the COUNT is a setting; the secret beside it is a file (see volumes).
+            # `optional`, like the tier keys below: a moca-settings written by an older setup.sh has
+            # no SH_TEST_USERS, and a required ref would stop the control plane from starting at all
+            # on an upgraded stack. Absent means unset, which configFromEnv reads as 0 (off).
+            - name: SH_TEST_USERS
+              valueFrom:
+                { configMapKeyRef: { name: moca-settings, key: SH_TEST_USERS, optional: true } }
             # P6.3 sandbox tiers: written to moca-settings by setup.sh ONLY on a stack with both
             # tiers; optional, so a stack without them renders exactly as before (untiered).
             # Must match the supervisor's: one list validates a session, the other places its turns.
@@ -118,14 +125,29 @@ spec:
             - { name: credentials, mountPath: /run/credentials, readOnly: true }
       volumes:
         - { name: tmp, emptyDir: {} }
+        # A PROJECTED volume, not a plain `secret` one, so the test-user secret can live in a
+        # Secret of its own marked `optional`. ADR-0045: credentialValue (systemd-credentials.ts)
+        # returns "unset" only for ENOENT and THROWS on an empty file, so an empty
+        # SH_TEST_USER_SECRET key in moca-mu1 crash-loops the control plane before identity.ts
+        # runs. A `secret` volume's `optional` is whole-source, never per item, so the key cannot
+        # be made optional inside moca-mu1; a second, optional source can. setup.sh writes
+        # moca-test-users only when SH_TEST_USERS > 0 and deletes it when they go off, so with test
+        # users off the file is absent (ENOENT -> unset), which is what the code already handles.
         - name: credentials
-          secret:
-            secretName: moca-mu1
-            items:
-              - { key: SH_SESSION_TOKEN_PRIVATE_KEY, path: SH_SESSION_TOKEN_PRIVATE_KEY }
-              - { key: SH_CREDENTIAL_KEK, path: SH_CREDENTIAL_KEK }
-              - { key: SH_EXCHANGE_TOKEN, path: SH_EXCHANGE_TOKEN }
+          projected:
             defaultMode: 256 # 0400
+            sources:
+              - secret:
+                  name: moca-mu1
+                  items:
+                    - { key: SH_SESSION_TOKEN_PRIVATE_KEY, path: SH_SESSION_TOKEN_PRIVATE_KEY }
+                    - { key: SH_CREDENTIAL_KEK, path: SH_CREDENTIAL_KEK }
+                    - { key: SH_EXCHANGE_TOKEN, path: SH_EXCHANGE_TOKEN }
+              - secret:
+                  name: moca-test-users
+                  optional: true
+                  items:
+                    - { key: SH_TEST_USER_SECRET, path: SH_TEST_USER_SECRET }
 ---
 apiVersion: v1
 kind: Service

@@ -48,6 +48,12 @@ spec:
               valueFrom: { configMapKeyRef: { name: sbx-settings, key: SH_SANDBOX_MAX_TOTAL } }
       volumes:
         - name: credentials
+          # The base (ADR-0045) makes this a PROJECTED volume so the test-user secret can come from
+          # an `optional` source of its own. Volumes merge by name, so without this the strategic
+          # merge would keep the base's `projected:` ALONGSIDE the `secret:` below -- a volume with
+          # two types, which the API server refuses, and which would also mount the KEK and
+          # exchange token that SBX0 §4.1 deliberately removes here.
+          projected: null
           secret:
             secretName: moca-mu1
             items:

@@ -74,6 +74,9 @@ P4_IDS=''
 P4_IDS_GIVEN=''
 # SH_SANDBOX_DEFAULT_TIER was set (even to empty) -- set in parse_args, resolved in load_setup_inputs.
 DEFAULT_TIER_GIVEN=''
+# ADR-0045: how many test:N logins to serve. Resolved sticky in load_setup_inputs, which runs
+# before ensure_secrets (the secret) and write_settings (the count).
+TEST_USERS=0
 # ocp-single's namespace (README §12), validated in parse_args.
 SINGLE_NS=''
 # ocp-single's optional Routes (README §12.5): the public DNS domain when SH_ROUTE_DOMAIN opted in,
@@ -486,8 +489,27 @@ derive_tiers() {
 # loaded dev.local tags, and --image there only picks what to pull, so only the sandbox count is
 # stored for it (and kind runs no P4 hosts, so it is never tiered and keeps no default tier).
 load_setup_inputs() {
-  local json stored
+  local json stored settings
   json="$(configmap_json moca-setup)"
+  # The count lives in moca-settings beside the other control-plane settings, so a re-run keeps it.
+  settings="$(configmap_json moca-settings)"
+  if [[ -n "${SH_TEST_USERS+x}" ]]; then TEST_USERS="${SH_TEST_USERS:-0}"; else TEST_USERS="$(cm_value "$settings" SH_TEST_USERS)"; fi
+  TEST_USERS="${TEST_USERS:-0}"
+  [[ "$TEST_USERS" =~ ^[0-9]+$ ]] || die "SH_TEST_USERS must be a whole number of users (got '$TEST_USERS')"
+  # Base 10 explicitly, and stored normalised: `08` is not valid octal, so (( )) below would die
+  # with a misleading "at most 1000", and `010` would be checked as 8 yet stored as `010`.
+  TEST_USERS="$((10#$TEST_USERS))"
+  ((TEST_USERS <= 1000)) || die "SH_TEST_USERS must be at most 1000 (got $TEST_USERS)"
+  if ((TEST_USERS > 0)); then
+    # ADR-0045: dev/test only. `--target ocp` is the shared-cluster target, where a stale export or
+    # a sticky ConfigMap value would quietly keep a password-gated login path alive, so it needs an
+    # explicit acknowledgement rather than a log line nobody reads. kind and ocp-single are
+    # dev/test targets by definition and need none.
+    if [[ "$TARGET" == ocp && "${SH_TEST_USERS_ACK-}" != 'i-know-this-is-not-for-production' ]]; then
+      die "SH_TEST_USERS=$TEST_USERS on --target ocp: test users are a dev/test feature and this is the shared-cluster target. Re-run with SH_TEST_USERS_ACK=i-know-this-is-not-for-production to confirm, or SH_TEST_USERS= to turn them off"
+    fi
+    log "WARNING: SH_TEST_USERS=$TEST_USERS: test:1..test:$TEST_USERS can log in with the shared secret and no browser (dev/test only, ADR-0045)"
+  fi
   if ! is_kind; then
     [[ -n "$IMAGE" ]] || IMAGE="$(cm_value "$json" IMAGE)"
     [[ -n "$SANDBOX_IMAGE" ]] || SANDBOX_IMAGE="$(cm_value "$json" SANDBOX_IMAGE)"
@@ -752,6 +774,43 @@ ensure_mu1_secret() {
   )
 }
 
+# ADR-0045: the test-user secret, in a Secret OF ITS OWN so control-plane.yaml can mark that source
+# `optional`. It must EXIST only while test users are on: credentialValue returns "unset" only for
+# ENOENT and throws on an empty file, so an empty value anywhere on this path crash-loops the
+# control plane before identity.ts runs. Hence write when on, DELETE when off -- which is also what
+# makes SH_TEST_USERS= (on -> off) leave no secret behind for testUserSecretFromEnv to refuse.
+ensure_test_user_secret() {
+  local json stored tsecret
+  if ((TEST_USERS == 0)); then
+    # Deleted, not emptied: absent is the only state the credential loader reads as unset.
+    kc delete secret moca-test-users -n "$NS" --ignore-not-found >/dev/null
+    TEST_USER_SECRET_DIGEST=''
+    return 0
+  fi
+  json="$(secret_json moca-test-users "$NS")"
+  stored="$(json_value "$json" SH_TEST_USER_SECRET)"
+  json=''
+  # Given wins, else keep what is stored (never rotated on a re-run), else generate.
+  tsecret="${SH_TEST_USER_SECRET-$stored}"
+  if [[ -z "$tsecret" ]]; then
+    # 64 hex characters, well past the 16-character minimum identity.ts enforces.
+    tsecret="$(openssl rand -hex 32)"
+    [[ "$tsecret" =~ ^[0-9a-f]{64}$ ]] || die 'could not generate SH_TEST_USER_SECRET'
+    log 'generated SH_TEST_USER_SECRET; read it with:'
+    log "  kubectl -n $NS get secret moca-test-users -o jsonpath='{.data.SH_TEST_USER_SECRET}' | base64 -d"
+  fi
+  # A DIGEST, never the value: write_settings folds it into the pods' settings-hash so that
+  # changing SH_TEST_USER_SECRET with the SAME user count still rolls the control plane. Without
+  # it the Secret would be rewritten while the running pod kept the old value until something else
+  # restarted it, and `--test-user` logins would fail against a secret the operator had just set.
+  TEST_USER_SECRET_DIGEST="$(printf '%s' "$tsecret" | sha256)"
+  [[ "$tsecret" == "$stored" ]] && return 0
+  (
+    export S_0="$tsecret"
+    apply_secret moca-test-users "$NS" SH_TEST_USER_SECRET
+  )
+}
+
 # moca-provisioner's Secret (spec §3.1, §4.1), only with SH_WORKLOAD_SANDBOXES=1: its Ed25519 attach
 # key (MOCA_ATTACH_PRIVATE_KEY) and the public half the relay verifies with (MOCA_ATTACH_PUBLIC_KEYS),
 # from the same key generator as moca-mu1's signing key -- an attach key is a different key of the
@@ -902,6 +961,7 @@ ensure_secrets() {
   ensure_p4_tokens
   ensure_redis_secret
   ensure_mu1_secret
+  ensure_test_user_secret
   ensure_provisioner_secret
 }
 
@@ -910,6 +970,9 @@ SUP_HOST=''
 CP_HOST=''
 RELAY_HOST=''
 SETTINGS_HASH=''
+# ADR-0045: sha256 of the test-user secret ('' when test users are off), set by
+# ensure_test_user_secret and folded into SETTINGS_HASH so a changed secret rolls the pods.
+TEST_USER_SECRET_DIGEST=''
 # The sandbox tiers (derive_tiers): both '' on a single-tier stack. STORED_DEFAULT_TIER is the sticky
 # SH_SANDBOX_DEFAULT_TIER input as resolved by load_setup_inputs (possibly ''); DEFAULT_TIER is what
 # moca-settings gets.
@@ -997,7 +1060,7 @@ hand_patched_catalog_hosts() {
 # the Route host every run (load_setup_inputs refuses a given value once ROUTE_DOMAIN resolves on).
 # SH_SANDBOX_TIERS and SH_SANDBOX_DEFAULT_TIER are derived too, by derive_tiers from the stack
 # (its sticky input lives in moca-setup). They are written even when '' so every stack hashes the
-# same nine keys; an empty token lifetime is the control plane's default, and an empty catalog
+# same ten keys; an empty token lifetime is the control plane's default, and an empty catalog
 # allow-list its own.
 write_settings() {
   local before after admins fb api_ttl session_ttl catalog_hosts catalog_src url
@@ -1044,15 +1107,16 @@ write_settings() {
     log "WARNING: SH_SESSION_TOKEN_TTL_SECONDS=$session_ttl is over 7 days: every session token stays valid that long"
   after="$(jq -ncS --arg id "$CLIENT_ID" --arg admins "$admins" --arg url "$url" \
     --arg fb "$fb" --arg tiers "$TIERS" --arg dtier "$DEFAULT_TIER" --arg api_ttl "$api_ttl" --arg session_ttl "$session_ttl" \
-    --arg catalog_hosts "$catalog_hosts" \
+    --arg catalog_hosts "$catalog_hosts" --arg test_users "$TEST_USERS" \
     '{SH_GITHUB_CLIENT_ID: $id, SH_ADMIN_SUBJECTS: $admins, SH_PUBLIC_HARNESS_URL: $url, SH_ALLOW_OPERATOR_FALLBACK: $fb,
       SH_SANDBOX_TIERS: $tiers, SH_SANDBOX_DEFAULT_TIER: $dtier,
       SH_API_TOKEN_TTL_SECONDS: $api_ttl, SH_SESSION_TOKEN_TTL_SECONDS: $session_ttl,
-      SH_MODEL_CATALOG_HOSTS: $catalog_hosts}')"
+      SH_MODEL_CATALOG_HOSTS: $catalog_hosts, SH_TEST_USERS: $test_users}')"
   jq -n --arg ns "$NS" --argjson data "$after" \
     '{apiVersion: "v1", kind: "ConfigMap", metadata: {name: "moca-settings", namespace: $ns}, data: $data}' |
     kc apply --server-side --force-conflicts --field-manager=moca-setup -f - >/dev/null
-  SETTINGS_HASH="$(printf '%s' "$after" | sha256)"
+  # The digest joins the HASH INPUT only -- never the ConfigMap, which holds nothing secret.
+  SETTINGS_HASH="$(printf '%s\n%s' "$after" "$TEST_USER_SECRET_DIGEST" | sha256)"
 }
 
 # kind-ci's mock model, from the file itself: no kustomization reads outside its root.