Skip to content

feat(training): add GR00T N1.7 Azure ML smoke workflow - #1712

Open
Sean Ma (maye-msft) wants to merge 15 commits into
microsoft:mainfrom
maye-msft:smoke-test
Open

Sean Ma (maye-msft) wants to merge 15 commits into
microsoft:mainfrom
maye-msft:smoke-test

Conversation

@maye-msft

@maye-msft Sean Ma (maye-msft) commented Sep 27, 2026 •

Copy link
Copy Markdown

Pull Request

Description

Add an Azure ML path for NVIDIA GR00T N1.7 fine-tuning that reuses the existing GR00T OSMO training entry point.

  • Add training/vla/scripts/submit-azureml-groot-training.sh to register a digest-pinned Azure ML environment and submit N1.7 jobs against immutable, versioned Azure ML data assets on AmlCompute or Azure ML Kubernetes compute (--instance-type). Supports --preflight-only, --resume-checkpoint, --register-model, --assets-only, and --config-preview.
  • Add training/vla/scripts/groot/azureml-train-entry.sh to build a writable metadata overlay over the dataset input, link the latest resume checkpoint, and resolve the Hugging Face token from Key Vault through the Azure ML job identity endpoint (MSI_ENDPOINT, with MSI_SECRET when the compute provides it).
  • Add training/vla/workflows/azureml/groot-n17-train.yaml as the command-job template. Its environment version is derived from DEFAULT_GROOT_IMAGE and registered in scripts/update-image-digests.sh. Dataset and resume inputs use download mode.
  • Add tests/e2e/test_e2e_aml_groot_n17_training.py, an Azure ML lifecycle test covering Key Vault token retrieval, preflight, short training, checkpoint resume, and model registration.
  • Add training/vla/images/groot-n17/ with a dependency-only runtime image pinned to an Isaac-GR00T commit, plus a build-time check that writes the resolved runtime versions into the image.
  • Extend training/vla/scripts/groot/osmo-train-entry.sh with prebuilt-image detection, a NUM_GPUS multi-GPU torchrun launch, a preflight mode, deterministic N1.7 resume, installation of post-training dependencies into the active uv virtual environment, and model mirroring outside the Azure ML job MLflow run.

Scope changes after review:

  • The Hugging Face model cache moved to a follow-up PR together with the cache-specific review feedback.
  • The Data Viewer Dockerfile changes were removed from this PR.

Related to #1663, which tracks raw Hugging Face credential transport separately.

Type of Change

  • 🐛 Bug fix (non-breaking change fixing an issue)
  • ✨ New feature (non-breaking change adding functionality)
  • 💥 Breaking change (fix or feature causing existing functionality to change)
  • 📚 Documentation update
  • 🏗️ Infrastructure change (Terraform/IaC)
  • ♻️ Refactoring (no functional changes)

Component(s) Affected

  • infrastructure/terraform/prerequisites/ - Azure subscription setup
  • infrastructure/terraform/ - Terraform infrastructure
  • infrastructure/setup/ - OSMO control plane / Helm
  • workflows/ - Training and evaluation workflows
  • training/ - Training pipelines and scripts
  • docs/ - Documentation

Testing Performed

  • Terraform plan reviewed (no unexpected changes)
  • Terraform apply tested in dev environment
  • Training scripts tested locally with Isaac Sim
  • OSMO workflow submitted successfully
  • Smoke tests passed (smoke_test_azure.py)

Static validation on 6688943:

Command Result
uvx ruff==0.16.7 format --check training/vla tests/e2e Passed
uvx ruff==0.16.7 check training/vla tests/e2e Passed
npx cspell <changed files> Passed, 0 issues
shellcheck -S warning <changed shell scripts> Passed
bash -n <changed shell scripts> Passed
python3 scripts/security/check_hf_pins.py Passed
pytest --co tests/e2e/test_e2e_aml_groot_n17_training.py (default markers) Passed, E2E test deselected
Invoke-Pester scripts/tests/security/AzureMLEnvironmentPins.Tests.ps1, update-image-digests.Tests.ps1 (Pester 5.7.1, on e5ef1f1) Passed, 57 tests

Targeted checks:

  • The resume job-file rewrite was run with BSD awk on macOS against groot-n17-train.yaml, and the generated YAML was parsed to confirm the resume_checkpoint input and command reference.
  • The post-training dependency installer was exercised with stub uv and pip binaries to confirm that it installs into the uv virtual environment when one is active and falls back to pip otherwise.
  • The Key Vault token path was exercised against local fake identity endpoints for AmlCompute (MSI_ENDPOINT and MSI_SECRET), Kubernetes compute (MSI_ENDPOINT only), and a job without an identity, which fails explicitly.
  • The aml_mirror.py MLflow failure inside Azure ML jobs was reproduced with mlflow==2.22.5 and confirmed fixed by clearing the job run binding.

Azure ML E2E on 6688943:

E2E_GROOT_HF_TOKEN_SECRET_URL=https://<vault>.vault.azure.net/secrets/hf-token \
E2E_AML_INSTANCE_TYPE=gpuspot E2E_GROOT_NUM_GPUS=1 \
uv run pytest -vv -s -m e2e tests/e2e/test_e2e_aml_groot_n17_training.py

Result: 1 passed in 3706.74s (1:01:46) on Azure ML Kubernetes compute with one NVIDIA A100 80GB PCIe (Spot).

Phase Azure ML identifier Terminal state Evidence
Dataset groot-n17-e2e-dataset-1791971044-de880e58:1 Registered Synthetic GR00T LeRobot v2 dataset
Preflight great_store_2909ptbrgd Completed [azureml] loaded Hugging Face token from Key Vault, TRAIN_RESULT=PREFLIGHT_PASS
Training clever_vase_4ny3g2907x Completed 2 steps, checkpoint-1 and checkpoint-2, 54 files in the checkpoints output
Resume hungry_brain_0pyn7y3zx1 Completed Linked checkpoint-2, Current global step: 2, deterministic resume patch applied, checkpoint-3 and checkpoint-4 produced
Model registration groot-n17-e2e-aml-model-1791972576-2f641bdc:1 Registered 15 artifact files downloaded from the registered model

The test archives its data asset and model versions during teardown.

Documentation Impact

  • No documentation changes needed
  • Documentation updated in this PR
  • Documentation issue filed

Bug Fix Checklist

Complete this section for bug fix PRs. Skip for other contribution types.

  • Linked to issue being fixed
  • Regression test included, OR
  • Justification for no regression test:

Checklist

- add pinned runtime, model cache, preflight, and resume flows`n- support portable viewer images for smoke deployment

🚀 - Generated by Copilot`n`nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@maye-msft
Sean Ma (maye-msft) requested a review from a team as a code owner September 27, 2026 23:54
@codecov-commenter

Codecov Comments Bot (codecov-commenter) commented Sep 27, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 95.63107% with 9 lines in your changes missing coverage. Please review.
✅ Project coverage is 87.19%. Comparing base (e827083) to head (b4ea729).

Files with missing lines Patch % Lines
training/vla/scripts/groot/preflight_check.py 92.72% 3 Missing and 1 partial ⚠️
training/vla/scripts/groot/patch_n17_trainer.py 94.11% 3 Missing ⚠️
training/vla/scripts/groot/resolve_hf_token.py 98.50% 0 Missing and 1 partial ⚠️
...ining/vla/scripts/groot/verify_prebuilt_runtime.py 96.96% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1712      +/-   ##
==========================================
- Coverage   87.29%   87.19%   -0.11%     
==========================================
  Files         402      393       -9     
  Lines       52480    51592     -888     
  Branches     7309     7187     -122     
==========================================
- Hits        45814    44986     -828     
+ Misses       5365     5331      -34     
+ Partials     1301     1275      -26     
Flag Coverage Δ *Carryforward flag
docusaurus 89.13% <ø> (ø) Carriedforward from 401c68c
go 100.00% <ø> (ø)
pester 88.40% <ø> (ø)
pytest-accessibility 82.41% <ø> (ø)
pytest-curation 95.90% <ø> (ø) Carriedforward from 401c68c
pytest-data-pipeline 100.00% <ø> (ø) Carriedforward from 401c68c
pytest-dataset-io 97.48% <ø> (ø) Carriedforward from 401c68c
pytest-dataviewer 91.55% <ø> (ø) Carriedforward from 401c68c
pytest-dm-tools 93.20% <ø> (ø) Carriedforward from 401c68c
pytest-evaluation 79.45% <ø> (ø)
pytest-fuzz 1.74% <ø> (ø)
pytest-inference 99.16% <ø> (ø) Carriedforward from 401c68c
pytest-shared-ci 100.00% <ø> (ø) Carriedforward from 401c68c
pytest-training 91.98% <95.63%> (+0.18%) ⬆️
pytest-vlm-judge ∅ <ø> (∅) Carriedforward from 401c68c
vitest 92.78% <ø> (-0.05%) ⬇️ Carriedforward from 401c68c
vitest-app 87.50% <ø> (ø) Carriedforward from 401c68c
vitest-components 91.55% <ø> (ø) Carriedforward from 401c68c
vitest-features 90.33% <ø> (ø) Carriedforward from 401c68c
vitest-lib 96.06% <ø> (ø) Carriedforward from 401c68c
vitest-state 93.95% <ø> (-0.18%) ⬇️ Carriedforward from 401c68c

*This pull request uses carry forward flags. Click here to find out more.

Files with missing lines Coverage Δ
training/vla/scripts/groot/resolve_hf_token.py 98.50% <98.50%> (ø)
...ining/vla/scripts/groot/verify_prebuilt_runtime.py 96.96% <96.96%> (ø)
training/vla/scripts/groot/patch_n17_trainer.py 94.11% <94.11%> (ø)
training/vla/scripts/groot/preflight_check.py 92.72% <92.72%> (ø)

... and 14 files with indirect coverage changes

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@algattik Alexandre Gattiker (algattik) left a comment •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please provide evidence that the relevant e2e test paths were run: model-cache creation and registration or direct consumption, Azure ML preflight, short training, checkpoint resume, and model registration when in scope

Raw Hugging Face credential transport is a pre-existing issue and is tracked separately in #1663.

if os.environ.get("AZURE_CLIENT_ID"):
identity_query["client_id"] = os.environ["AZURE_CLIENT_ID"]
identity_url = (
"http://169.254.169.254/metadata/identity/oauth2/token?"

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This uses the VM IMDS endpoint, but Azure ML command jobs expose managed identity through the per-job MSI_ENDPOINT and MSI_SECRET. The recommended --hf-token-secret-url path will fail before training. Please obtain the Key Vault token through the Azure ML job MSI endpoint (or a credential implementation that uses it) and fail explicitly when those variables are unavailable.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 800a07d. The token request now uses the Azure ML job endpoint, MSI_ENDPOINT with the secret: MSI_SECRET header and api-version=2017-09-01, matching azure-identity's AzureMLCredential. The clientid value comes from AZURE_CLIENT_ID or DEFAULT_IDENTITY_CLIENT_ID. The script fails with an explicit error when MSI_ENDPOINT or MSI_SECRET is unavailable.

"nvidia/GR00T-N1.7-3B": sys.argv[2],
"nvidia/Cosmos-Reason2-2B": "9ce19a195e423419c349abfc86fd07178b230561",
}
actual = {model["repository"]: model["revision"] for model in manifest["models"]}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The cache producer records each file's size and SHA-256, but this consumer verifies only repository names and revisions before linking the cached files into the model load path. Please verify the exact file set, sizes, and hashes from the manifest before creating links, and reject missing, additional, symlinked, or mismatched entries.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Deferred with the model cache. The cache consumer was removed from this PR in 1c2f65d. The follow-up PR will verify the exact file set, sizes, and SHA-256 values from the manifest before linking, and will reject missing, extra, symlinked, or mismatched entries.

case "${model_cache_asset}" in
azureml://*/data/*/versions/*) model_asset_version="${model_cache_asset##*/versions/}" ;;
azureml:*:*) model_asset_version="${model_cache_asset##*:}" ;;
*) fatal "--model-cache-asset must use azureml:NAME:VERSION or a full versioned Azure ML data URI" ;;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The supplied cache workflow emits an anonymous command-job output, while this submitter accepts only a registered versioned data asset. Please make the producer-to-consumer path executable by registering and reporting the cache asset, or accept and validate an immutable Azure ML job-output URI directly.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Deferred with the model cache. --model-cache-asset and the cache workflow were removed from this PR in 1c2f65d. The follow-up PR will make the producer-to-consumer path executable end to end.

for name in EXPECTED_RUNTIME
}
if actual != EXPECTED_RUNTIME:
raise RuntimeError(f"locked runtime differs from Item 9: {actual}")

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Item 9 is task-relative and has no meaning to image builders or operators. Please state the runtime invariant directly and include the expected and actual version maps.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 800a07d. The error now states the runtime invariant directly and includes both version maps: GR00T N1.7 runtime must match the pinned CUDA, PyTorch, and NumPy ABI; expected {...}, found {...}.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for putting this together, a pinned GR00T N1.7 smoke path with model caching and resume is a nice addition, and I like that the cache script only accepts full 40 character revisions. Alexandre Gattiker (@algattik) already covered the main things to sort out, so I'll just add a couple.

The Ruff check is failing on formatting (write-image-metadata.py and cache-huggingface-models.py), so a quick ruff format on those should get it green. The PR description also has literal `n characters in it and an empty Closes #, so could you clean that up and add what you tested?

@algattik

Alexandre Gattiker (algattik) commented Sep 29, 2026 •

Copy link
Copy Markdown
Collaborator

The passing test_e2e_osmo_vla_finetune run validates the existing OSMO GR00T N1.5 path, but it does not exercise this PR’s net-new Azure ML N1.7 workflow.

Please add E2E coverage for the new Azure ML path introduced here. The test should exercise model-cache creation and registration or direct consumption; Key Vault secret retrieval using the Azure ML job identity; a short N1.7 training run that produces a checkpoint; resume from that checkpoint; and model registration when enabled.

The test should record the relevant Azure ML job and asset identifiers and assert successful terminal states.

import json
import os
import urllib.request
from datetime import UTC, datetime

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

datetime.UTC only exists on Python 3.11+, and this runs in the GR00T venv, which Isaac-GR00T pins to ==3.10.*, so the import fails before anything runs. timezone.utc works on both.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Deferred with the model cache. This script was removed from this PR in 1c2f65d. The follow-up PR will use timezone.utc.

compute: azureml:gpu-h100-train

command: >-
python -m pip install --quiet azure-identity==1.25.3 huggingface-hub==0.35.3 &&

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

python here resolves to the GR00T venv (the image puts /opt/Isaac-GR00T/.venv/bin first on PATH), and that venv comes from uv sync, which doesn't install pip. So this fails with "No module named pip" before the cache script runs. uv pip install --python "$VIRTUAL_ENV/bin/python" ... would work since uv is in the image, or the two packages could be baked into the image.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Deferred with the model cache. This workflow was removed from this PR in 1c2f65d. The follow-up PR will install through uv pip install --python "$VIRTUAL_ENV/bin/python".

The same root cause also affected this PR. The --register-model and ACR paths in osmo-train-entry.sh called pip install and then ran the venv Python. cfb9108 fixes that with a python_install helper that targets the active uv virtual environment and falls back to pip when no virtual environment is active.

PY
fi

if [ "${RESUME:-false}" = "true" ]; then

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This block sits outside the gr00t/experiment/experiment.py check above, so it also runs on the N1.5 layout. N1.5's trainer.py (at 796ca8d, the ref the OSMO E2E uses) doesn't have this text, so RESUME=true on N1.5, which works on main through --resume, now raises before training starts. Moving it inside the N1.7 check would fix it.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 800a07d. The deterministic-resume patch for gr00t/experiment/trainer.py now runs only when the N1.7 layout (gr00t/experiment/experiment.py) is present. N1.5 with RESUME=true goes through scripts/gr00t_finetune.py --resume as on main.

Sean Ma (maye-msft) and others added 4 commits October 14, 2026 11:59
- remove cache producer workflow and script
- drop model-cache input from Azure ML submitter and entry scripts

♻️ - Generated by Copilot

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
⏪ - Generated by Copilot

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- read Key Vault token via Azure ML job MSI endpoint
- scope deterministic resume patch to the N1.7 layout
- report expected and actual runtime versions on mismatch

🐛 - Generated by Copilot

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- install post-train deps into the uv venv that runs aml_mirror
- replace GNU-only sed with portable awk for resume input wiring
- invoke chained entry via bash; mark submitter executable

🔧 - Generated by Copilot

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sean Ma (maye-msft) and others added 3 commits October 14, 2026 18:43
- forward --instance-type as resources.instance_type
- read Key Vault token via MSI_ENDPOINT; MSI_SECRET is optional
- download dataset and resume inputs instead of ro_mount
- run aml_mirror outside the Azure ML job MLflow run binding

☸️ - Generated by Copilot

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- cover Key Vault token, preflight, short train, resume, model registration
- render N1.7 modality config from the synthetic GR00T dataset spec

🧪 - Generated by Copilot

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- resolve collections.abc import conflict in tests/e2e/_aml.py

🔀 - Generated by Copilot

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@maye-msft

Sean Ma (maye-msft) commented Oct 14, 2026 •

Copy link
Copy Markdown
Author

Thanks Alexandre Gattiker (@algattik) and Allen Greaves (@agreaves-ms) for the detailed review. I pushed e5ef1f1 and narrowed the scope of this PR:

  • The Hugging Face model cache moved to a follow-up PR. The follow-up will carry the cache-specific feedback: manifest verification, producer-to-consumer asset handoff, the Python 3.10 datetime.UTC import, and the missing pip in the cache job.
  • The Data Viewer Dockerfile changes were removed. They were unrelated to GR00T and caused the merge conflict with main.
  • 800a07d fixes the Key Vault identity path, the image metadata error message, and the N1.5 resume regression.
  • cfb9108 fixes two problems found during self-review: post-training dependencies now install into the uv virtual environment that runs aml_mirror.py, and the --resume-checkpoint job-file rewrite uses portable awk instead of GNU-only sed -i.
  • b35e577 adds Azure ML Kubernetes compute support and fixes two issues the E2E run exposed: Kubernetes compute provides MSI_ENDPOINT without MSI_SECRET, and aml_mirror.py failed because the Azure ML job's MLFLOW_RUN_ID bound it to the job run.
  • 6688943 adds tests/e2e/test_e2e_aml_groot_n17_training.py.
  • e5ef1f1 merges the latest main and registers groot-n17-train.yaml in scripts/update-image-digests.sh so the AzureML environment pin check passes.

Alexandre Gattiker (@algattik), the new E2E passed on Azure ML Kubernetes compute with one NVIDIA A100 80GB PCIe: 1 passed in 3706.74s (1:01:46).

Phase Azure ML identifier Terminal state Evidence
Dataset groot-n17-e2e-dataset-1791971044-de880e58:1 Registered Synthetic GR00T LeRobot v2 dataset
Preflight great_store_2909ptbrgd Completed [azureml] loaded Hugging Face token from Key Vault, TRAIN_RESULT=PREFLIGHT_PASS
Training clever_vase_4ny3g2907x Completed 2 steps, checkpoint-1 and checkpoint-2, 54 files in the checkpoints output
Resume hungry_brain_0pyn7y3zx1 Completed Linked checkpoint-2, Current global step: 2, deterministic resume patch applied, checkpoint-3 and checkpoint-4 produced
Model registration groot-n17-e2e-aml-model-1791972576-2f641bdc:1 Registered 15 artifact files downloaded from the registered model

Every job resolved the Hugging Face token from Key Vault through the job identity; the test passes no HF_TOKEN to the jobs. Model-cache coverage moves to the follow-up PR with the cache. All five phases ran on Azure ML Kubernetes compute. I did not run them on AmlCompute because this subscription has no AmlCompute quota for a GPU that meets GR00T N1.7's 40 GB memory requirement. AmlCompute uses the same MSI_ENDPOINT token code; its additional MSI_SECRET header is covered only by the local fake-endpoint test.

Sean Ma (maye-msft) and others added 2 commits October 14, 2026 21:38
- register groot-n17-train.yaml in azureml_pin_specs
- derive the environment version from DEFAULT_GROOT_IMAGE

📌 - Generated by Copilot

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- add resolve_hf_token, verify_prebuilt_runtime, patch_n17_trainer, preflight_check
- add unit tests for each helper in training/tests
- inject the OSMO helpers through groot-train.yaml like download_blob.py

♻️ - Generated by Copilot

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ights

🧪 - Generated by Copilot

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…nvironment

- stop writing the token to stdout for command substitution
- exec the training entry with HF_TOKEN set only in its environment

🔒 - Generated by Copilot

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@maye-msft

Copy link
Copy Markdown
Author

I moved the embedded Python that this PR added into standalone modules with unit tests.

5d85e50 adds four modules in training/vla/scripts/groot/. They follow the download_blob.py and aml_mirror.py conventions: typed functions and a main() entry point.

Module Replaces
resolve_hf_token.py Key Vault token lookup in azureml-train-entry.sh
verify_prebuilt_runtime.py Prebuilt image metadata check in osmo-train-entry.sh
patch_n17_trainer.py Both N1.7 trainer patches. Each patch still requires the exact upstream text, and the N1.5 layout is skipped
preflight_check.py Dataset and modality checks, plus the visible-GPU assertion

E2E results:

  • OSMO VLA (N1.5) on 5d85e50: 1 passed (15:47), workflow vla-finetune-e2e-osmo-1792049077-10367d0e-1. The task log shows the injected helpers ran ([preflight] visible GPUs: 1 (expected: 1)) and that the N1.5 checkout was left unpatched (not the N1.7 layout; trainer patches skipped).
  • Azure ML (N1.7) on 0f2222c: 1 passed (49:20).
    • Preflight khaki_vinegar_z9t0gpc573, training salmon_thread_j3m7kkqz33, and resume frank_wire_52c04bfsrp all ended Completed.
    • The registered model groot-n17-e2e-aml-model-1792051420-85faae8e:1 has 15 artifact files (11.72 GiB).

@@ -0,0 +1,105 @@
"""Patch the Isaac-GR00T N1.7 trainer for controlled, deterministic resume.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please consider submitting this fix upstream in GR00T, and then let's create an issue in our repo to track that upstream PR and remove this patching.

@@ -0,0 +1,105 @@
"""Patch the Isaac-GR00T N1.7 trainer for controlled, deterministic resume.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be good to have a CI test verifying the patch succeeds on the actual GR00T source (exact pinned ISAAC_GROOT_REF).

Can potentially be deferred to a follow-up PR. Please make sure we track such deferred work as Issues.

fi

python -c "import torch; p=torch.cuda.get_device_properties(0); print(f'[preflight] GPU: {p.name} total={p.total_memory/1024**3:.2f} GiB sm={p.major}.{p.minor}')"
python "${GROOT_HELPERS_DIR}/preflight_check.py" gpus --expected "${NUM_GPUS}"

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we propagate the OSMO workflow’s gpu value into the task environment as NUM_GPUS? The workflow requests {{gpu}}, but this script defaults NUM_GPUS to 1. With gpu=2, the new check therefore rejects the two allocated GPUs as “expected 1 visible GPUs, found 2”. Setting NUM_GPUS: "{{gpu}}" in the workflow would keep the allocation, preflight expectation, and training process count aligned. Please cover a non-default GPU count in a regression test.

az ml job create
--resource-group "${resource_group}"
--workspace-name "${workspace_name}"
--file "${resolved_job_file}"

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please propagate --subscription-id to compute-identity lookup, job creation and log streaming, not just environment registration. I ran the actual submitter against a fail-closed az stub with different active and selected subscriptions: environment registration used the selected subscription, while ml compute show, ml job create and ml job stream omitted it and therefore used the active subscription. The job environment and printed portal URL still pointed to the selected subscription. This can fail after environment registration or target the wrong workspace when names overlap. Pass --subscription "${subscription_id}" consistently, including through the identity helper, and add a regression test where selected and active subscriptions differ.

"""Replace both upstream resume call sites with an explicit resume flag."""
count = source.count(RESUME_CALL)
if count != RESUME_CALL_COUNT:
raise RuntimeError(f"expected {RESUME_CALL_COUNT} upstream resume call sites, found {count}")

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This patch cannot be reapplied when the entrypoint reuses a writable GR00T checkout. After resume=False, both True call sites become False, so a second invocation finds zero matches. Applying resume=True twice similarly fails because the original reseed block has already been replaced. I reproduced false-to-false, false-to-true and true-to-true failures in temporary checkouts. Fresh-container jobs are unaffected, but the entrypoint explicitly reuses an existing checkout, and checking out the same commit does not discard these edits. Could we use a fixed patch with resume selection handled at runtime in GR00T, consistent with the upstream-fix discussion? Otherwise, recognize the exact supported patched forms and add repeated-application tests while continuing to reject unexpected upstream changes.

[[ "${hf_token_secret_url}" =~ ^https://[A-Za-z0-9-]+\.vault\.azure\.net/secrets/[A-Za-z0-9-]+(/[A-Za-z0-9]+)?$ ]] || \
fatal "--hf-token-secret-url must be a versioned or unversioned Azure Key Vault secret URL"
fi
if [[ "${preflight_only}" != "true" && "${base_model}" == "nvidia/GR00T-N1.7-3B" ]]; then

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please exempt --assets-only from this training credential check. With the default model and no HF credentials, environment-only registration fails here before reaching its early exit, even though registering the environment does not fetch gated model weights. Either add an assets-only guard or move training-only validation after that branch. A stubbed CLI test should verify that --assets-only registers the environment without either HF credential and does not create a job.

az_args+=(--set "environment_variables.AZURE_CLIENT_ID=${managed_identity_client_id}")
[[ -n "${display_name}" ]] && az_args+=(--set "display_name=${display_name}")
[[ -n "${instance_type}" ]] && az_args+=(--set "resources.instance_type=${instance_type}")
[[ -n "${hf_token}" ]] && az_args+=(--set "environment_variables.HF_TOKEN=${hf_token}")

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This branch still puts the raw token into persisted Azure ML job properties, including when HF_TOKEN is inherited from the submitting shell. The runtime Key Vault/exec handoff does not protect this path: readers of the job definition or a saved job manifest may obtain the credential. I see the PR already defers credential transport to #1663; please explicitly include this new Azure ML GR00T submitter and its regression coverage there, since the issue's enumerated acceptance criteria currently name the existing pi0/evaluation and OSMO paths. The intended fix is to pass only a secret reference and resolve the token inside the workload, rather than forward its value in environment_variables.HF_TOKEN.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants