fix: compose concurrent VMM device ACL cleanup by lpcox · Pull Request #9969 · github/gh-aw-firewall

lpcox · 2026-10-14T05:38:18Z

Summary

compose Cloud Hypervisor device ACL restoration from the durable workflow baseline plus only currently live peer VMM ACEs
persist temporary peer UIDs alongside overlapping ACL baselines and retain the composed target across account deletion
add deterministic two-identity regressions for both cleanup orders, exact runner ACE preservation, and removal of both VMM ACEs

Diagnosis

Run 38515513439 proved mount cleanup and workload isolation were healthy, while final KVM ACL restoration lost the workflow-owned runner ACE. The two concurrent VMMs used distinct UIDs 999 and 997. Under the shared device ACL lock, a later runtime could capture an earlier runtime's temporary ACE in its durable baseline. Restoring the entire captured ACL made cleanup order observable: one cleanup could erase a still-live peer, while the other could later resurrect an already-cleaned peer or overwrite the runner baseline.

This change classifies and records temporary AWF VMM ACEs at capture time. Each locked restoration removes only the current runtime's ACE, strips captured temporary peers that are no longer live, preserves validated live peer ACEs, and restores the exact durable runner baseline. The pre-deletion composed target is retained and re-applied after userdel/groupdel, preserving fail-closed identity validation and durable recovery.

Validation

npm run build
npm run lint -- --quiet
npm test -- --runInBand — 462 suites passed, 8,246 tests passed
npm run test:coverage -- --runInBand — passed
Cloud Hypervisor suite — 47 suites passed, 1,190 tests passed

Campaign follow-up

Do not resume or retry run 38515513439. After this PR merges, dispatch the unchanged Phase 4 campaign as a fresh run with a new campaign ID.

Track temporary peer VMM ACEs in durable cleanup state. Compose each locked restoration from the workflow baseline plus live peers. Retain that target across account deletion for cleanup-order safety. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: cfb6df6b-e245-428d-9ba7-bfa413e4afa7

Copilot

🟡 Changes recommended

The temporary-account classifier can remove a durable ACE belonging to a similarly prefixed non-AWF account.

2 open findings

What changed in this PR

Fixes concurrent Cloud Hypervisor VMM ACL cleanup while preserving durable runner access.

Changes:

Composes cleanup targets from durable baselines and live peer VMM ACEs.
Persists temporary peer UIDs for crash recovery.
Adds deterministic concurrent-cleanup regressions.

File	Description
`src/cloud-hypervisor/vmm-identity.ts`	Classifies peers and composes ACL restoration.
`src/cloud-hypervisor/vmm-identity.test.ts`	Updates restoration expectations.
`src/cloud-hypervisor/manager-start.ts`	Passes temporary UIDs to cleanup records.
`src/cloud-hypervisor/device-acl.ts`	Adds ACL composition utilities.
`src/cloud-hypervisor/device-acl.test.ts`	Tests both cleanup orders.
`src/cloud-hypervisor/cleanup-vmm-identity.ts`	Applies composed targets during recovery.
`src/cloud-hypervisor/cleanup-vmm-identity.test.ts`	Adds two-identity cleanup regressions.
`src/cloud-hypervisor/cleanup-registry.ts`	Extends cleanup-handle API.
`src/cloud-hypervisor/cleanup-identity.ts`	Adds persisted peer UID validation.
`src/cloud-hypervisor/cleanup-handle.ts`	Stores and releases peer UID metadata.

🧠 Review effort: Balanced

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

lpcox · 2026-10-14T05:42:24Z

@copilot Fix the code for all comments in this review thread.

When a review comment includes a suggested change, apply the suggestion exactly.

Do not make changes beyond what is described in the linked review thread.

github-actions · 2026-10-14T05:44:18Z

⚠️ Coverage Regression Detected

This PR decreases test coverage. Please add tests to maintain coverage levels.

Overall Coverage

Metric	Base	PR	Delta
Lines	93.24%	93.23%	➡️ -0.01%
Statements	91.82%	91.81%	➡️ -0.01%
Functions	90.59%	90.55%	📉 -0.04%
Branches	85.58%	85.51%	📉 -0.07%

📁 Per-file Coverage Changes (7 files)

File	Lines (Before → After)	Statements (Before → After)
`src/cloud-hypervisor/device-acl.ts`	100.0% → 94.4% (-5.56%)	100.0% → 93.9% (-6.10%)
`src/cloud-hypervisor/cleanup-identity.ts`	91.9% → 90.0% (-1.86%)	92.1% → 90.3% (-1.81%)
`src/cloud-hypervisor/vmm-identity.ts`	97.5% → 96.1% (-1.41%)	94.3% → 92.9% (-1.41%)
`src/enclave/host-executor-server.ts`	90.3% → 90.3% (+0.00%)	88.8% → 89.1% (+0.31%)
`src/cloud-hypervisor/cleanup-handle.ts`	94.1% → 94.3% (+0.22%)	90.7% → 91.1% (+0.33%)
`src/cloud-hypervisor/cleanup-vmm-identity.ts`	80.8% → 82.3% (+1.53%)	80.8% → 82.2% (+1.46%)
`src/log-directory-setup.ts`	96.8% → 100.0% (+3.18%)	96.9% → 100.0% (+3.13%)

Coverage comparison generated by scripts/ci/compare-coverage.ts

Copilot · 2026-10-14T05:47:07Z

@copilot Fix the code for all comments in this review thread.

When a review comment includes a suggested change, app...

Addressed both review findings and pushed as e1600ed.

github-actions · 2026-10-14T05:50:04Z

✅ Copilot review passed with no inline comments.

@lpcox Add the ready-for-aw label to this PR to trigger agentic CI smoke tests.

github-actions · 2026-10-14T05:50:29Z

📰 VERDICT: Smoke Copilot has concluded. All systems operational. This is a developing story. 🎤

📰 BREAKING: Report filed by Smoke Copilot

github-actions · 2026-10-14T05:50:29Z

💎 Smoke Gemini is crystallizing results on this pull request...

github-actions · 2026-10-14T05:50:30Z

✅ Build Test Suite completed successfully!

Generated by Build Test Suite for #9969

github-actions · 2026-10-14T05:50:30Z

Smoke Cloud Hypervisor completed. Cloud Hypervisor + Copilot passed.

Cloud Hypervisor + Copilot smoke test by Smoke Cloud Hypervisor

github-actions · 2026-10-14T05:50:31Z

🪪 Smoke Copilot BYOK AOAI (Entra) is testing Azure OpenAI BYOK (Entra / GitHub OIDC) mode on this pull request...

github-actions · 2026-10-14T05:50:31Z

🚀 Security Guard has started processing this pull request

github-actions · 2026-10-14T05:50:31Z

✨ The prophecy is fulfilled... Smoke Codex has completed its mystical journey. The stars align. 🌟

🔮 The oracle has spoken through Smoke Codex

github-actions · 2026-10-14T05:50:32Z

📡 Smoke OTel Tracing completed. All tracing scenarios validated. ✅

📡 OTel tracing validated by Smoke OTel Tracing

github-actions · 2026-10-14T05:50:33Z

✅ Smoke Copilot BYOK completed. Copilot BYOK mode operational. 🔓

🔑 BYOK report filed by Smoke Copilot BYOK

github-actions · 2026-10-14T05:50:35Z

✅ Smoke Claude passed

Generated by Smoke Claude for #9969

github-actions · 2026-10-14T05:50:35Z

🛡️ Smoke Copilot Network Isolation confirmed the egress allowlist is enforced. ✅

🛡️ Egress verdict from Smoke Copilot Network Isolation

github-actions · 2026-10-14T05:50:36Z

🔑 Smoke Copilot BYOK AOAI (api-key) is testing Azure OpenAI BYOK (api-key) mode on this pull request...

github-actions · 2026-10-14T05:50:39Z

🔌 Smoke Services — All services reachable! ✅

🔌 Service connectivity validated by Smoke Services

github-actions · 2026-10-14T05:50:58Z

Chroot tests passed! Smoke Chroot - All security and functionality tests succeeded.

Tested by Smoke Chroot

Exercise exact temporary identity classification and fail-closed composition. Cover durable peer UID persistence, validation, release, and both mask shapes. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: cfb6df6b-e245-428d-9ba7-bfa413e4afa7

github-actions · 2026-10-14T05:55:51Z

⚠️ Coverage Regression Detected

This PR decreases test coverage. Please add tests to maintain coverage levels.

Overall Coverage

Metric	Base	PR	Delta
Lines	93.24%	93.23%	➡️ -0.01%
Statements	91.82%	91.81%	➡️ -0.01%
Functions	90.59%	90.55%	📉 -0.04%
Branches	85.58%	85.53%	📉 -0.05%

📁 Per-file Coverage Changes (6 files)

File	Lines (Before → After)	Statements (Before → After)
`src/cloud-hypervisor/device-acl.ts`	100.0% → 94.4% (-5.56%)	100.0% → 93.9% (-6.10%)
`src/cloud-hypervisor/cleanup-identity.ts`	91.9% → 90.0% (-1.86%)	92.1% → 90.3% (-1.81%)
`src/cloud-hypervisor/vmm-identity.ts`	97.5% → 96.1% (-1.41%)	94.3% → 93.2% (-1.09%)
`src/cloud-hypervisor/cleanup-handle.ts`	94.1% → 94.3% (+0.22%)	90.7% → 91.1% (+0.33%)
`src/cloud-hypervisor/cleanup-vmm-identity.ts`	80.8% → 82.3% (+1.53%)	80.8% → 82.2% (+1.46%)
`src/log-directory-setup.ts`	96.8% → 100.0% (+3.18%)	96.9% → 100.0% (+3.13%)

Coverage comparison generated by scripts/ci/compare-coverage.ts

github-actions · 2026-10-14T06:00:04Z

Cloud Hypervisor + Copilot Smoke Test

PASS — list_pull_requests returned PR test: cover staged mount cleanup failures #9968 (merged).
PASS — https://github.com returned HTTP 200.
PASS — Wrote and read back smoke test file successfully.
PASS — (example.com/redacted) blocked with curl exit 60 (000), confirming allowlist enforcement.

All checks passed.

Cloud Hypervisor + Copilot smoke test by Smoke Cloud Hypervisor
_{Add label ready-for-aw to run again}

github-actions · 2026-10-14T06:00:09Z

Smoke Copilot: PASS ✅

GitHub MCP (last merged: "test: cover staged mount cleanup failures"): ✅
github.com connectivity (HTTP 200): ✅
File write/read: ✅
PR: "fix: compose concurrent VMM device ACL cleanup" — @lpcox

📰 BREAKING: Report filed by Smoke Copilot
_{Add label ready-for-aw to run again}

github-actions · 2026-10-14T06:00:10Z

Smoke Test: Claude Engine Validation

Check	Status
API	✅ PASS
gh CLI	✅ PASS
File	✅ PASS

Overall result: PASS

Generated by Smoke Claude for #9969 · claude · haiku45 · 48.8 AIC · ⊞ 6.1K · ◷
_{Add label ready-for-aw to run again}

github-actions · 2026-10-14T06:00:40Z

Smoke Test: Copilot BYOK (Direct) Results

✅ GitHub MCP: Connected, PR data verified
✅ GitHub.com: HTTP 200 (connected)
✅ File Write/Read: File exists and readable
✅ BYOK Inference: Running in direct mode (COPILOT_PROVIDER_API_KEY via api-proxy → api.githubcopilot.com)

Status: PASS

Running as: @lpcox (author/assignee)
Test executed: 2026-10-14 05:59:59 UTC

🔑 BYOK report filed by Smoke Copilot BYOK
_{Add label ready-for-aw to run again}

github-actions · 2026-10-14T06:00:59Z

EGRESS_RESULT allow=pass deny=pass

✅ Allowed domain (api.github.com): 200
✅ Blocked domain (example.com): blocked (curl TLS error, exit non-zero)

Overall: PASS — @lpcox

🛡️ Egress verdict from Smoke Copilot Network Isolation
_{Add label ready-for-aw to run again}

github-actions · 2026-10-14T06:01:00Z

Smoke services: Redis PING ✅, pg_isready ✅, SELECT 1 ✅. Overall: PASS

🔌 Service connectivity validated by Smoke Services
_{Add label ready-for-aw to run again}

github-actions · 2026-10-14T06:01:03Z

OTEL smoke (run 38518783035)

✅ S1 Module loads; exports startRequestSpan, setTokenAttributes, setBudgetAttributes, endSpan, endSpanError, shutdown, isEnabled, etc.
✅ S2 OTEL tests: 3 suites, 68/68 passed
✅ S3 Env forwarding: trace ID/parent span ID in env-passthrough.ts; OTEL endpoint/headers/trace vars in api-proxy-env-config.ts
✅ S4 onUsage hook present in token-tracker-http.js
⚠️ S5 Diagnostics: not independently inspected (post-step output unavailable to me)

📡 OTel tracing validated by Smoke OTel Tracing
_{Add label ready-for-aw to run again}

github-actions · 2026-10-14T06:01:31Z

✅ Coverage Check Passed

Overall Coverage

Metric	Base	PR	Delta
Lines	93.25%	93.28%	📈 +0.03%
Statements	91.83%	91.87%	📈 +0.04%
Functions	90.59%	90.63%	📈 +0.04%
Branches	85.59%	85.62%	📈 +0.03%

📁 Per-file Coverage Changes (6 files)

File	Lines (Before → After)	Statements (Before → After)
`src/nvx/one-shot-adapter.ts`	82.9% → 82.3% (-0.56%)	80.2% → 79.6% (-0.52%)
`src/cloud-hypervisor/cleanup-handle.ts`	94.1% → 94.3% (+0.22%)	90.7% → 91.1% (+0.33%)
`src/cloud-hypervisor/vmm-identity.ts`	97.5% → 97.9% (+0.35%)	94.3% → 95.2% (+0.84%)
`src/cloud-hypervisor/cleanup-identity.ts`	91.9% → 92.2% (+0.36%)	92.1% → 92.5% (+0.34%)
`src/cloud-hypervisor/cleanup-vmm-identity.ts`	80.8% → 82.3% (+1.53%)	80.8% → 82.2% (+1.46%)
`src/log-directory-setup.ts`	96.8% → 100.0% (+3.18%)	96.9% → 100.0% (+3.13%)

Coverage comparison generated by scripts/ci/compare-coverage.ts

github-actions · 2026-10-14T06:01:45Z

9968: test: cover staged mount cleanup failures
9966: fix: retry transient staged mount busy errors
GitHub merged PRs: ✅
GitHub PR details: ✅
Playwright title check: ❌
File write/read: ✅
Discussion comment: ✅
Build AWF: ✅
Overall: FAIL

🔮 The oracle has spoken through Smoke Codex
_{Add label ready-for-aw to run again}

github-actions · 2026-10-14T06:02:18Z

🏗️ Build Test Suite Results

Ecosystem	Project	Build/Install	Tests	Status
Bun	elysia	❌	not run	❌ FAIL
Bun	hono	❌	not run	❌ FAIL
C++	fmt	✅	N/A	✅ PASS
C++	json	✅	N/A	✅ PASS
Deno	oak	❌	not run	❌ FAIL
Deno	std	❌	not run	❌ FAIL
.NET	hello-world	✅	N/A	✅ PASS
.NET	json-parse	✅	N/A	✅ PASS
Go	color	✅	1/1 passed	✅ PASS
Go	env	✅	1/1 passed	✅ PASS
Go	uuid	✅	1/1 passed	✅ PASS
Java	gson	✅	1/1 passed	✅ PASS
Java	caffeine	✅	1/1 passed	✅ PASS
Node.js	clsx	✅	all passed	✅ PASS
Node.js	execa	✅	all passed	✅ PASS
Node.js	p-limit	✅	all passed	✅ PASS
Rust	fd	✅	1/1 passed	✅ PASS
Rust	zoxide	✅	1/1 passed	✅ PASS

Overall: 6/8 ecosystems passed — FAIL

Failures

Bun / Deno: The install scripts (curl ... | bash / | sh) were denied by the sandbox ("Permission denied and could not request permission"). Neither bun nor deno is preinstalled, so these tests could not run.

Notes

.NET only worked after setting XDG_RUNTIME_DIR and DOTNET_CLI_HOME to writable temp paths. /run/user/1001 does not exist.
Java only worked with -Dmaven.repo.local pointed at a temp directory. The default ~/.m2 repository was not accessible (LocalRepositoryNotAccessibleException).

The build-test label was not added because not all ecosystems passed.

Generated by Build Test Suite for #9969 · copilot · auto · 25.7 AIC · ⊞ 11.9K · ◷
_{Add label ready-for-aw to run again}

github-actions · 2026-10-14T06:02:34Z

Chroot Version Comparison

Runtime	Host Version	Chroot Version	Match?
Python	Python 3.12.15	Python 3.12.15	✅ YES
Node.js	v24.21.0	v2.98.0	❌ NO
Go	go1.22.12	go1.22.12	✅ YES

Result: Not all tests passed (Node.js mismatch), so the smoke-chroot label was not added.

Tested by Smoke Chroot
_{Add label ready-for-aw to run again}

Copilot AI balanced review requested due to automatic review settings October 14, 2026 05:38

Copilot started reviewing on behalf of lpcox October 14, 2026 05:38

View session

Copilot AI reviewed Oct 14, 2026

View reviewed changes

Comment thread src/cloud-hypervisor/vmm-identity.ts Outdated

Comment thread src/cloud-hypervisor/cleanup-handle.ts

Copilot started work on behalf of lpcox October 14, 2026 05:43

View session

fix: validate temporary VMM ACL identities

e1600ed

Copilot finished work on behalf of lpcox October 14, 2026 05:47

lpcox added the ready-for-aw label Oct 14, 2026

lpcox enabled auto-merge (squash) October 14, 2026 05:50

lpcox had a problem deploying to aoai-model October 14, 2026 05:50 — with GitHub Actions Failure

github-actions Bot added smoke-cloud-hypervisor smoke-copilot labels Oct 14, 2026

github-actions Bot added smoke-claude smoke-copilot-byok labels Oct 14, 2026

github-actions Bot added the smoke-services label Oct 14, 2026

github-actions Bot added the smoke-copilot-network-isolation Copilot network-isolation egress smoke test label Oct 14, 2026

lpcox disabled auto-merge October 14, 2026 13:39

lpcox merged commit 342e146 into main Oct 14, 2026
37 checks passed

lpcox deleted the lpcox-fix-concurrent-acl-composition branch October 14, 2026 13:39

Conversation

lpcox commented Oct 14, 2026

Summary

Diagnosis

Validation

Campaign follow-up

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

lpcox commented Oct 14, 2026

Uh oh!

github-actions Bot commented Oct 14, 2026

⚠️ Coverage Regression Detected

Overall Coverage

Uh oh!

Copilot AI commented Oct 14, 2026

Uh oh!

github-actions Bot commented Oct 14, 2026

Uh oh!

github-actions Bot commented Oct 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Oct 14, 2026

Uh oh!

github-actions Bot commented Oct 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Oct 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Oct 14, 2026

Uh oh!

github-actions Bot commented Oct 14, 2026

Uh oh!

github-actions Bot commented Oct 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Oct 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Oct 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Oct 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Oct 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Oct 14, 2026

Uh oh!

github-actions Bot commented Oct 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Oct 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Oct 14, 2026

⚠️ Coverage Regression Detected

Overall Coverage

Uh oh!

github-actions Bot commented Oct 14, 2026

Cloud Hypervisor + Copilot Smoke Test

Uh oh!

github-actions Bot commented Oct 14, 2026

Uh oh!

github-actions Bot commented Oct 14, 2026

Smoke Test: Claude Engine Validation

Uh oh!

github-actions Bot commented Oct 14, 2026

Smoke Test: Copilot BYOK (Direct) Results

Uh oh!

github-actions Bot commented Oct 14, 2026

Uh oh!

github-actions Bot commented Oct 14, 2026

Uh oh!

github-actions Bot commented Oct 14, 2026 •

edited

Loading

github-actions Bot commented Oct 14, 2026 •

edited

Loading

github-actions Bot commented Oct 14, 2026 •

edited

Loading

github-actions Bot commented Oct 14, 2026 •

edited

Loading

github-actions Bot commented Oct 14, 2026 •

edited

Loading

github-actions Bot commented Oct 14, 2026 •

edited

Loading

github-actions Bot commented Oct 14, 2026 •

edited

Loading

github-actions Bot commented Oct 14, 2026 •

edited

Loading

github-actions Bot commented Oct 14, 2026 •

edited

Loading

github-actions Bot commented Oct 14, 2026 •

edited

Loading