Repository navigation
feat(training): add GR00T N1.7 Azure ML smoke workflow - #1712
Sean Ma (maye-msft) wants to merge 15 commits into
Conversation
- add pinned runtime, model cache, preflight, and resume flows`n- support portable viewer images for smoke deployment 🚀 - Generated by Copilot`n`nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #1712 +/- ##
==========================================
- Coverage 87.29% 87.19% -0.11%
==========================================
Files 402 393 -9
Lines 52480 51592 -888
Branches 7309 7187 -122
==========================================
- Hits 45814 44986 -828
+ Misses 5365 5331 -34
+ Partials 1301 1275 -26
*This pull request uses carry forward flags. Click here to find out more.
🚀 New features to boost your workflow:
|
There was a problem hiding this comment.
Please provide evidence that the relevant e2e test paths were run: model-cache creation and registration or direct consumption, Azure ML preflight, short training, checkpoint resume, and model registration when in scope
Raw Hugging Face credential transport is a pre-existing issue and is tracked separately in #1663.
| if os.environ.get("AZURE_CLIENT_ID"): | ||
| identity_query["client_id"] = os.environ["AZURE_CLIENT_ID"] | ||
| identity_url = ( | ||
| "http://169.254.169.254/metadata/identity/oauth2/token?" |
There was a problem hiding this comment.
This uses the VM IMDS endpoint, but Azure ML command jobs expose managed identity through the per-job MSI_ENDPOINT and MSI_SECRET. The recommended --hf-token-secret-url path will fail before training. Please obtain the Key Vault token through the Azure ML job MSI endpoint (or a credential implementation that uses it) and fail explicitly when those variables are unavailable.
There was a problem hiding this comment.
Fixed in 800a07d. The token request now uses the Azure ML job endpoint, MSI_ENDPOINT with the secret: MSI_SECRET header and api-version=2017-09-01, matching azure-identity's AzureMLCredential. The clientid value comes from AZURE_CLIENT_ID or DEFAULT_IDENTITY_CLIENT_ID. The script fails with an explicit error when MSI_ENDPOINT or MSI_SECRET is unavailable.
| "nvidia/GR00T-N1.7-3B": sys.argv[2], | ||
| "nvidia/Cosmos-Reason2-2B": "9ce19a195e423419c349abfc86fd07178b230561", | ||
| } | ||
| actual = {model["repository"]: model["revision"] for model in manifest["models"]} |
There was a problem hiding this comment.
The cache producer records each file's size and SHA-256, but this consumer verifies only repository names and revisions before linking the cached files into the model load path. Please verify the exact file set, sizes, and hashes from the manifest before creating links, and reject missing, additional, symlinked, or mismatched entries.
There was a problem hiding this comment.
Deferred with the model cache. The cache consumer was removed from this PR in 1c2f65d. The follow-up PR will verify the exact file set, sizes, and SHA-256 values from the manifest before linking, and will reject missing, extra, symlinked, or mismatched entries.
| case "${model_cache_asset}" in | ||
| azureml://*/data/*/versions/*) model_asset_version="${model_cache_asset##*/versions/}" ;; | ||
| azureml:*:*) model_asset_version="${model_cache_asset##*:}" ;; | ||
| *) fatal "--model-cache-asset must use azureml:NAME:VERSION or a full versioned Azure ML data URI" ;; |
There was a problem hiding this comment.
The supplied cache workflow emits an anonymous command-job output, while this submitter accepts only a registered versioned data asset. Please make the producer-to-consumer path executable by registering and reporting the cache asset, or accept and validate an immutable Azure ML job-output URI directly.
There was a problem hiding this comment.
Deferred with the model cache. --model-cache-asset and the cache workflow were removed from this PR in 1c2f65d. The follow-up PR will make the producer-to-consumer path executable end to end.
| for name in EXPECTED_RUNTIME | ||
| } | ||
| if actual != EXPECTED_RUNTIME: | ||
| raise RuntimeError(f"locked runtime differs from Item 9: {actual}") |
There was a problem hiding this comment.
Item 9 is task-relative and has no meaning to image builders or operators. Please state the runtime invariant directly and include the expected and actual version maps.
There was a problem hiding this comment.
Fixed in 800a07d. The error now states the runtime invariant directly and includes both version maps: GR00T N1.7 runtime must match the pinned CUDA, PyTorch, and NumPy ABI; expected {...}, found {...}.
Allen Greaves (agreaves-ms)
left a comment
There was a problem hiding this comment.
Thanks for putting this together, a pinned GR00T N1.7 smoke path with model caching and resume is a nice addition, and I like that the cache script only accepts full 40 character revisions. Alexandre Gattiker (@algattik) already covered the main things to sort out, so I'll just add a couple.
The Ruff check is failing on formatting (write-image-metadata.py and cache-huggingface-models.py), so a quick ruff format on those should get it green. The PR description also has literal `n characters in it and an empty Closes #, so could you clean that up and add what you tested?
|
The passing Please add E2E coverage for the new Azure ML path introduced here. The test should exercise model-cache creation and registration or direct consumption; Key Vault secret retrieval using the Azure ML job identity; a short N1.7 training run that produces a checkpoint; resume from that checkpoint; and model registration when enabled. The test should record the relevant Azure ML job and asset identifiers and assert successful terminal states. |
| import json | ||
| import os | ||
| import urllib.request | ||
| from datetime import UTC, datetime |
There was a problem hiding this comment.
datetime.UTC only exists on Python 3.11+, and this runs in the GR00T venv, which Isaac-GR00T pins to ==3.10.*, so the import fails before anything runs. timezone.utc works on both.
There was a problem hiding this comment.
Deferred with the model cache. This script was removed from this PR in 1c2f65d. The follow-up PR will use timezone.utc.
| compute: azureml:gpu-h100-train | ||
|
|
||
| command: >- | ||
| python -m pip install --quiet azure-identity==1.25.3 huggingface-hub==0.35.3 && |
There was a problem hiding this comment.
python here resolves to the GR00T venv (the image puts /opt/Isaac-GR00T/.venv/bin first on PATH), and that venv comes from uv sync, which doesn't install pip. So this fails with "No module named pip" before the cache script runs. uv pip install --python "$VIRTUAL_ENV/bin/python" ... would work since uv is in the image, or the two packages could be baked into the image.
There was a problem hiding this comment.
Deferred with the model cache. This workflow was removed from this PR in 1c2f65d. The follow-up PR will install through uv pip install --python "$VIRTUAL_ENV/bin/python".
The same root cause also affected this PR. The --register-model and ACR paths in osmo-train-entry.sh called pip install and then ran the venv Python. cfb9108 fixes that with a python_install helper that targets the active uv virtual environment and falls back to pip when no virtual environment is active.
| PY | ||
| fi | ||
|
|
||
| if [ "${RESUME:-false}" = "true" ]; then |
There was a problem hiding this comment.
This block sits outside the gr00t/experiment/experiment.py check above, so it also runs on the N1.5 layout. N1.5's trainer.py (at 796ca8d, the ref the OSMO E2E uses) doesn't have this text, so RESUME=true on N1.5, which works on main through --resume, now raises before training starts. Moving it inside the N1.7 check would fix it.
There was a problem hiding this comment.
Fixed in 800a07d. The deterministic-resume patch for gr00t/experiment/trainer.py now runs only when the N1.7 layout (gr00t/experiment/experiment.py) is present. N1.5 with RESUME=true goes through scripts/gr00t_finetune.py --resume as on main.
- remove cache producer workflow and script - drop model-cache input from Azure ML submitter and entry scripts ♻️ - Generated by Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
⏪ - Generated by Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- read Key Vault token via Azure ML job MSI endpoint - scope deterministic resume patch to the N1.7 layout - report expected and actual runtime versions on mismatch 🐛 - Generated by Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- install post-train deps into the uv venv that runs aml_mirror - replace GNU-only sed with portable awk for resume input wiring - invoke chained entry via bash; mark submitter executable 🔧 - Generated by Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- forward --instance-type as resources.instance_type - read Key Vault token via MSI_ENDPOINT; MSI_SECRET is optional - download dataset and resume inputs instead of ro_mount - run aml_mirror outside the Azure ML job MLflow run binding ☸️ - Generated by Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- cover Key Vault token, preflight, short train, resume, model registration - render N1.7 modality config from the synthetic GR00T dataset spec 🧪 - Generated by Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- resolve collections.abc import conflict in tests/e2e/_aml.py 🔀 - Generated by Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
Thanks Alexandre Gattiker (@algattik) and Allen Greaves (@agreaves-ms) for the detailed review. I pushed
Alexandre Gattiker (@algattik), the new E2E passed on Azure ML Kubernetes compute with one NVIDIA A100 80GB PCIe:
Every job resolved the Hugging Face token from Key Vault through the job identity; the test passes no |
- register groot-n17-train.yaml in azureml_pin_specs - derive the environment version from DEFAULT_GROOT_IMAGE 📌 - Generated by Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- add resolve_hf_token, verify_prebuilt_runtime, patch_n17_trainer, preflight_check - add unit tests for each helper in training/tests - inject the OSMO helpers through groot-train.yaml like download_blob.py ♻️ - Generated by Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ights 🧪 - Generated by Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…nvironment - stop writing the token to stdout for command substitution - exec the training entry with HF_TOKEN set only in its environment 🔒 - Generated by Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
I moved the embedded Python that this PR added into standalone modules with unit tests.
E2E results:
|
| @@ -0,0 +1,105 @@ | |||
| """Patch the Isaac-GR00T N1.7 trainer for controlled, deterministic resume. | |||
There was a problem hiding this comment.
Please consider submitting this fix upstream in GR00T, and then let's create an issue in our repo to track that upstream PR and remove this patching.
| @@ -0,0 +1,105 @@ | |||
| """Patch the Isaac-GR00T N1.7 trainer for controlled, deterministic resume. | |||
There was a problem hiding this comment.
It would be good to have a CI test verifying the patch succeeds on the actual GR00T source (exact pinned ISAAC_GROOT_REF).
Can potentially be deferred to a follow-up PR. Please make sure we track such deferred work as Issues.
| fi | ||
|
|
||
| python -c "import torch; p=torch.cuda.get_device_properties(0); print(f'[preflight] GPU: {p.name} total={p.total_memory/1024**3:.2f} GiB sm={p.major}.{p.minor}')" | ||
| python "${GROOT_HELPERS_DIR}/preflight_check.py" gpus --expected "${NUM_GPUS}" |
There was a problem hiding this comment.
Could we propagate the OSMO workflow’s gpu value into the task environment as NUM_GPUS? The workflow requests {{gpu}}, but this script defaults NUM_GPUS to 1. With gpu=2, the new check therefore rejects the two allocated GPUs as “expected 1 visible GPUs, found 2”. Setting NUM_GPUS: "{{gpu}}" in the workflow would keep the allocation, preflight expectation, and training process count aligned. Please cover a non-default GPU count in a regression test.
| az ml job create | ||
| --resource-group "${resource_group}" | ||
| --workspace-name "${workspace_name}" | ||
| --file "${resolved_job_file}" |
There was a problem hiding this comment.
Please propagate --subscription-id to compute-identity lookup, job creation and log streaming, not just environment registration. I ran the actual submitter against a fail-closed az stub with different active and selected subscriptions: environment registration used the selected subscription, while ml compute show, ml job create and ml job stream omitted it and therefore used the active subscription. The job environment and printed portal URL still pointed to the selected subscription. This can fail after environment registration or target the wrong workspace when names overlap. Pass --subscription "${subscription_id}" consistently, including through the identity helper, and add a regression test where selected and active subscriptions differ.
| """Replace both upstream resume call sites with an explicit resume flag.""" | ||
| count = source.count(RESUME_CALL) | ||
| if count != RESUME_CALL_COUNT: | ||
| raise RuntimeError(f"expected {RESUME_CALL_COUNT} upstream resume call sites, found {count}") |
There was a problem hiding this comment.
This patch cannot be reapplied when the entrypoint reuses a writable GR00T checkout. After resume=False, both True call sites become False, so a second invocation finds zero matches. Applying resume=True twice similarly fails because the original reseed block has already been replaced. I reproduced false-to-false, false-to-true and true-to-true failures in temporary checkouts. Fresh-container jobs are unaffected, but the entrypoint explicitly reuses an existing checkout, and checking out the same commit does not discard these edits. Could we use a fixed patch with resume selection handled at runtime in GR00T, consistent with the upstream-fix discussion? Otherwise, recognize the exact supported patched forms and add repeated-application tests while continuing to reject unexpected upstream changes.
| [[ "${hf_token_secret_url}" =~ ^https://[A-Za-z0-9-]+\.vault\.azure\.net/secrets/[A-Za-z0-9-]+(/[A-Za-z0-9]+)?$ ]] || \ | ||
| fatal "--hf-token-secret-url must be a versioned or unversioned Azure Key Vault secret URL" | ||
| fi | ||
| if [[ "${preflight_only}" != "true" && "${base_model}" == "nvidia/GR00T-N1.7-3B" ]]; then |
There was a problem hiding this comment.
Please exempt --assets-only from this training credential check. With the default model and no HF credentials, environment-only registration fails here before reaching its early exit, even though registering the environment does not fetch gated model weights. Either add an assets-only guard or move training-only validation after that branch. A stubbed CLI test should verify that --assets-only registers the environment without either HF credential and does not create a job.
| az_args+=(--set "environment_variables.AZURE_CLIENT_ID=${managed_identity_client_id}") | ||
| [[ -n "${display_name}" ]] && az_args+=(--set "display_name=${display_name}") | ||
| [[ -n "${instance_type}" ]] && az_args+=(--set "resources.instance_type=${instance_type}") | ||
| [[ -n "${hf_token}" ]] && az_args+=(--set "environment_variables.HF_TOKEN=${hf_token}") |
There was a problem hiding this comment.
This branch still puts the raw token into persisted Azure ML job properties, including when HF_TOKEN is inherited from the submitting shell. The runtime Key Vault/exec handoff does not protect this path: readers of the job definition or a saved job manifest may obtain the credential. I see the PR already defers credential transport to #1663; please explicitly include this new Azure ML GR00T submitter and its regression coverage there, since the issue's enumerated acceptance criteria currently name the existing pi0/evaluation and OSMO paths. The intended fix is to pass only a secret reference and resolve the token inside the workload, rather than forward its value in environment_variables.HF_TOKEN.
Pull Request
Description
Add an Azure ML path for NVIDIA GR00T N1.7 fine-tuning that reuses the existing GR00T OSMO training entry point.
training/vla/scripts/submit-azureml-groot-training.shto register a digest-pinned Azure ML environment and submit N1.7 jobs against immutable, versioned Azure ML data assets on AmlCompute or Azure ML Kubernetes compute (--instance-type). Supports--preflight-only,--resume-checkpoint,--register-model,--assets-only, and--config-preview.training/vla/scripts/groot/azureml-train-entry.shto build a writable metadata overlay over the dataset input, link the latest resume checkpoint, and resolve the Hugging Face token from Key Vault through the Azure ML job identity endpoint (MSI_ENDPOINT, withMSI_SECRETwhen the compute provides it).training/vla/workflows/azureml/groot-n17-train.yamlas the command-job template. Its environment version is derived fromDEFAULT_GROOT_IMAGEand registered inscripts/update-image-digests.sh. Dataset and resume inputs usedownloadmode.tests/e2e/test_e2e_aml_groot_n17_training.py, an Azure ML lifecycle test covering Key Vault token retrieval, preflight, short training, checkpoint resume, and model registration.training/vla/images/groot-n17/with a dependency-only runtime image pinned to an Isaac-GR00T commit, plus a build-time check that writes the resolved runtime versions into the image.training/vla/scripts/groot/osmo-train-entry.shwith prebuilt-image detection, aNUM_GPUSmulti-GPUtorchrunlaunch, a preflight mode, deterministic N1.7 resume, installation of post-training dependencies into the active uv virtual environment, and model mirroring outside the Azure ML job MLflow run.Scope changes after review:
Related to #1663, which tracks raw Hugging Face credential transport separately.
Type of Change
Component(s) Affected
infrastructure/terraform/prerequisites/- Azure subscription setupinfrastructure/terraform/- Terraform infrastructureinfrastructure/setup/- OSMO control plane / Helmworkflows/- Training and evaluation workflowstraining/- Training pipelines and scriptsdocs/- DocumentationTesting Performed
planreviewed (no unexpected changes)applytested in dev environmentsmoke_test_azure.py)Static validation on
6688943:uvx ruff==0.16.7 format --check training/vla tests/e2euvx ruff==0.16.7 check training/vla tests/e2enpx cspell <changed files>shellcheck -S warning <changed shell scripts>bash -n <changed shell scripts>python3 scripts/security/check_hf_pins.pypytest --co tests/e2e/test_e2e_aml_groot_n17_training.py(default markers)Invoke-Pester scripts/tests/security/AzureMLEnvironmentPins.Tests.ps1, update-image-digests.Tests.ps1(Pester 5.7.1, one5ef1f1)Targeted checks:
awkon macOS againstgroot-n17-train.yaml, and the generated YAML was parsed to confirm theresume_checkpointinput and command reference.uvandpipbinaries to confirm that it installs into the uv virtual environment when one is active and falls back topipotherwise.MSI_ENDPOINTandMSI_SECRET), Kubernetes compute (MSI_ENDPOINTonly), and a job without an identity, which fails explicitly.aml_mirror.pyMLflow failure inside Azure ML jobs was reproduced withmlflow==2.22.5and confirmed fixed by clearing the job run binding.Azure ML E2E on
6688943:Result:
1 passed in 3706.74s (1:01:46)on Azure ML Kubernetes compute with one NVIDIA A100 80GB PCIe (Spot).groot-n17-e2e-dataset-1791971044-de880e58:1great_store_2909ptbrgdCompleted[azureml] loaded Hugging Face token from Key Vault,TRAIN_RESULT=PREFLIGHT_PASSclever_vase_4ny3g2907xCompletedcheckpoint-1andcheckpoint-2, 54 files in thecheckpointsoutputhungry_brain_0pyn7y3zx1Completedcheckpoint-2,Current global step: 2, deterministic resume patch applied,checkpoint-3andcheckpoint-4producedgroot-n17-e2e-aml-model-1791972576-2f641bdc:1The test archives its data asset and model versions during teardown.
Documentation Impact
Bug Fix Checklist
Complete this section for bug fix PRs. Skip for other contribution types.
Checklist