Skip to content

feat(dataviewer): optimize release preview preparation and address manual testing findings - #2249

Merged
Katrien De Graeve (katriendg) merged 6 commits into
mainfrom
feat/perf-changes-viewer
Nov 20, 2026
Merged

Katrien De Graeve (katriendg) merged 6 commits into
mainfrom
feat/perf-changes-viewer

Conversation

@katriendg

Copy link
Copy Markdown
Collaborator

Description

Optimize dataviewer release inventory, preview preparation and job access while
preserving source-change detection and publication safety checks.

Closes #2248.

This PR extends beyond the original issue's backend-only scope. Manual testing
identified additional release and viewer improvements, so it also includes saved
subtask publication, scratch recovery fixes, bounded worker diagnostics and UI
clarifications.

  • Enumerate actual source episode indices without annotation lookups or display
    record construction. Reuse verified preview metadata while rejecting changed
    source revisions, including same-length content changes.
  • Preserve JSON boolean registration flags in sorted destination responses.
  • Use job-specific reads for status and execution bindings, and job-local heartbeat
    transactions to avoid downloading unrelated Blob job records.
  • Publish saved subtask replacements and explicit clears, remap spans after frame
    removal, and preserve exact segment metadata when reopening releases.
  • Return needs_input for unsupported saved edits. Create scratch roots
    independently of ledger storage and recover demonstrably absent planned
    workspaces only after creator-exit checks.
  • Record bounded worker failure stages and causes without raw exception text.
    Use the asyncio server loop, clarify execution-time storage budgeting and
    distinguish pending, inconclusive and completed judge verdicts.
  • Update the release specification and regression coverage. No new endpoints or
    relaxed Parquet restrictions are introduced.

Type of Change

  • 🐛 Bug fix (non-breaking change fixing an issue)
  • ✨ New feature (non-breaking change adding functionality)
  • 💥 Breaking change (fix or feature causing existing functionality to change)
  • 📚 Documentation update
  • 🏗️ Infrastructure change (Terraform/IaC)
  • ♻️ Refactoring (no functional changes)

Component(s) Affected

  • infrastructure/terraform/prerequisites/ - Azure subscription setup
  • infrastructure/terraform/ - Terraform infrastructure
  • infrastructure/setup/ - OSMO control plane / Helm
  • workflows/ - Training and evaluation workflows
  • training/ - Training pipelines and scripts
  • docs/ - Documentation
  • data-management/ - Curation, dataset-I/O, viewer and release specification

Testing Performed

  • Terraform plan reviewed (no unexpected changes)
  • Terraform apply tested in dev environment
  • Training scripts tested locally with Isaac Sim
  • OSMO workflow submitted successfully
  • Smoke tests passed (smoke_test_azure.py)

Local preflight on this branch:

  • Release API and five affected curation test modules: 1,343 passed
  • JudgePanel, ReleaseDialog and ReleaseWorkspace component tests: 140 passed
  • Frontend type check and production build: passed; build reports a large-chunk warning
  • Frontend ESLint: passed with 45 warnings; Prettier check passed
  • Changed Python Ruff lint and format checks: passed (21 files)
  • Specification Markdown and spelling commands: passed; configured exclusions
    resulted in zero files linted/checked
  • Specification table-format check: passed after formatting-only repair
    a65c2436 (docs(data): align the release API table)

Documentation Impact

  • No documentation changes needed
  • Documentation updated in this PR
  • Documentation issue filed

The release specification documents subtask composition, unsupported-edit
refusals, diagnostics, scratch recovery and destination boolean flags.

Bug Fix Checklist

Complete this section for bug fix PRs. Skip for other contribution types.

  • Linked to issue being fixed
  • Regression test included, OR
  • Justification for no regression test:

Checklist

- add support for saved subtasks and their metadata in episode releases
- improve error handling for workspace creation and cleanup
- update tests to cover new functionality and edge cases
- refine API responses to include boolean flags for destination registration

🔧 - Generated by Copilot
📝 - Generated by Copilot

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@github-actions

github-actions Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

Snapshot Warnings

⚠️: No snapshots were found for the head SHA b3f3be6.
Ensure that dependencies are being submitted on PR branches and consider enabling retry-on-snapshot-warnings. See the documentation for more information and troubleshooting advice.

Scanned Files

None

@codecov-commenter

Codecov Comments Bot (codecov-commenter) commented Nov 20, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 98.93993% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 87.36%. Comparing base (4c3e500) to head (b3f3be6).
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
...tend/src/components/app-shell/DatasetWorkspace.tsx 86.66% 1 Missing and 1 partial ⚠️
...ion/physical_ai_curation/release/record_storage.py 97.29% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2249      +/-   ##
==========================================
+ Coverage   87.30%   87.36%   +0.06%     
==========================================
  Files         402      402              
  Lines       52500    52714     +214     
  Branches     7314     7358      +44     
==========================================
+ Hits        45836    46055     +219     
+ Misses       5363     5362       -1     
+ Partials     1301     1297       -4     
Flag Coverage Δ *Carryforward flag
docusaurus 89.13% <ø> (ø) Carriedforward from c211793
go 100.00% <ø> (ø)
pester 88.40% <ø> (ø) Carriedforward from c211793
pytest-accessibility 82.41% <ø> (ø)
pytest-curation 96.01% <99.56%> (+0.09%) ⬆️
pytest-data-pipeline 100.00% <ø> (ø) Carriedforward from c211793
pytest-dataset-io 97.57% <100.00%> (+0.09%) ⬆️
pytest-dataviewer 91.59% <100.00%> (+0.01%) ⬆️
pytest-dm-tools 93.20% <ø> (ø) Carriedforward from c211793
pytest-evaluation 79.45% <ø> (ø)
pytest-fuzz 1.74% <0.00%> (-0.01%) ⬇️
pytest-inference 99.16% <ø> (ø) Carriedforward from c211793
pytest-shared-ci 100.00% <ø> (ø) Carriedforward from c211793
pytest-training 91.79% <ø> (ø) Carriedforward from c211793
pytest-vlm-judge 93.23% <ø> (ø)
vitest 92.83% <86.66%> (+<0.01%) ⬆️
vitest-app 87.50% <ø> (ø)
vitest-components 91.56% <86.66%> (+<0.01%) ⬆️
vitest-features 90.33% <ø> (ø)
vitest-lib 96.06% <ø> (ø)
vitest-state 94.13% <ø> (ø)

*This pull request uses carry forward flags. Click here to find out more.

Files with missing lines Coverage Δ
...ration/physical_ai_curation/release/composition.py 99.23% <100.00%> (+1.36%) ⬆️
...ration/physical_ai_curation/release/coordinator.py 95.73% <100.00%> (+0.12%) ⬆️
...ment/curation/physical_ai_curation/release/jobs.py 97.45% <100.00%> (+0.05%) ⬆️
...tion/physical_ai_curation/release/native_inputs.py 94.78% <100.00%> (+0.10%) ⬆️
...nt/curation/physical_ai_curation/release/runner.py 94.82% <100.00%> (+0.53%) ⬆️
...nt/curation/physical_ai_curation/release/tables.py 98.82% <100.00%> (+0.21%) ⬆️
...ement/dataset-io/physical_ai_dataset_io/lerobot.py 97.18% <100.00%> (+0.12%) ⬆️
...agement/viewer/backend/src/api/routers/releases.py 98.44% <ø> (ø)
data-management/viewer/backend/src/api/server.py 85.89% <ø> (ø)
...ackend/src/api/services/dataset_service/service.py 86.91% <100.00%> (+0.08%) ⬆️
... and 7 more
🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@katriendg Katrien De Graeve (katriendg) changed the title perf(data): optimize release preparation and address manual testing findings feat(dataviewer): optimize release preview preparation and address manual testing findings Nov 20, 2026
@katriendg
Katrien De Graeve (katriendg) added this pull request to the merge queue Nov 20, 2026
Merged via the queue into main with commit 216f824 Nov 20, 2026
73 checks passed
@katriendg
Katrien De Graeve (katriendg) deleted the feat/perf-changes-viewer branch November 20, 2026 06:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(dataviewer): optimize release inventory and preview preparation

3 participants