Skip to content

fix(media): handle unset PyAV time_base after av 19 upgrade (#2134) - #2170

Open
Mangesh Raut (mangeshraut712) wants to merge 5 commits into
microsoft:mainfrom
mangeshraut712:fix/pyav19-time-base-guard
Open

Mangesh Raut (mangeshraut712) wants to merge 5 commits into
microsoft:mainfrom
mangeshraut712:fix/pyav19-time-base-guard

Conversation

@mangeshraut712

Copy link
Copy Markdown
Contributor

Description

Closes #2134

In PyAV >= 19.0.0, unset rational attributes (stream.time_base) return av.AVRational(0, 1) rather than None. Under if stream.time_base is not None:, this evaluated to True, resulting in ZeroDivisionError: float division by zero during seek timestamp calculation or returning an erroneous 0.0s duration.

This PR addresses the issue as suggested in #2134:

  • Replaces if stream.time_base is not None: with if stream.time_base: in lerobot_handler.py and lerobot_exporter.py to gracefully skip seek when time_base is unset.
  • Updates _stream_duration_s in frames.py with truthiness check so unset stream time base falls back to container duration.
  • Adds guards in _seek_and_decode to avoid divide-by-zero on unset time base.
  • Adds ZeroDivisionError to caught exceptions in extract_frames to translate unexpected division errors into MediaInputError.
  • Adds regression unit tests in evaluation/tests/vlm_judge/test_frames.py covering Fraction(0, 1), av.AVRational(0, 1), ZeroDivisionError translation, and decoding with unset time base.

Type of Change

  • 🐛 Bug fix (non-breaking change fixing an issue)

Component(s) Affected

  • workflows/ - Training and evaluation workflows

Testing Performed

  • Ran uv run pytest evaluation/tests/vlm_judge/test_frames.py --no-cov (23 passed)
  • Ran uv run pytest evaluation/vlm_judge (194 passed, 85.65% coverage)
  • Ran uv run pytest data-management/viewer/backend/tests/test_lerobot* --no-cov (142 passed)
  • Ran uv run ruff check and uv run ruff format --check (clean, 0 warnings)

Documentation Impact

  • No documentation changes needed

Bug Fix Checklist

Checklist

  • My code follows the project conventions
  • Commit messages follow conventional commit format
  • I have performed a self-review
  • Documentation impact assessed above
  • No new linting warnings introduced

@codecov-commenter

Codecov Comments Bot (codecov-commenter) commented Nov 17, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 88.88889% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 87.30%. Comparing base (e827083) to head (e72f257).

Files with missing lines Patch % Lines
...rc/api/services/dataset_service/lerobot_handler.py 0.00% 1 Missing ⚠️
Additional details and impacted files
@@           Coverage Diff           @@
##             main    #2170   +/-   ##
=======================================
  Coverage   87.29%   87.30%           
=======================================
  Files         402      402           
  Lines       52480    52482    +2     
  Branches     7309     7310    +1     
=======================================
+ Hits        45814    45819    +5     
+ Misses       5365     5363    -2     
+ Partials     1301     1300    -1     
Flag Coverage Δ *Carryforward flag
docusaurus 89.13% <ø> (ø) Carriedforward from e827083
go 100.00% <ø> (ø)
pester 88.40% <ø> (ø) Carriedforward from e827083
pytest-accessibility 82.41% <ø> (ø)
pytest-curation 95.92% <ø> (+0.01%) ⬆️
pytest-data-pipeline 100.00% <ø> (ø) Carriedforward from e827083
pytest-dataset-io 97.48% <ø> (ø)
pytest-dataviewer 91.55% <50.00%> (ø)
pytest-dm-tools 93.20% <ø> (ø) Carriedforward from e827083
pytest-evaluation 79.45% <ø> (ø)
pytest-fuzz 1.74% <0.00%> (ø)
pytest-inference 99.16% <ø> (ø) Carriedforward from e827083
pytest-shared-ci 100.00% <ø> (ø) Carriedforward from e827083
pytest-training 91.79% <ø> (ø) Carriedforward from e827083
pytest-vlm-judge 93.33% <100.00%> (+0.10%) ⬆️
vitest 92.83% <ø> (ø) Carriedforward from e827083
vitest-app 87.50% <ø> (ø) Carriedforward from e827083
vitest-components 91.55% <ø> (ø) Carriedforward from e827083
vitest-features 90.33% <ø> (ø) Carriedforward from e827083
vitest-lib 96.06% <ø> (ø) Carriedforward from e827083
vitest-state 94.13% <ø> (ø) Carriedforward from e827083

*This pull request uses carry forward flags. Click here to find out more.

Files with missing lines Coverage Δ
...iewer/backend/src/api/services/lerobot_exporter.py 90.51% <100.00%> (ø)
evaluation/vlm_judge/frames.py 98.21% <100.00%> (+0.94%) ⬆️
...rc/api/services/dataset_service/lerobot_handler.py 62.96% <0.00%> (ø)

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@jkim323 Jamie Kim (jkim323) left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for following up on #2134 so quickly, Mangesh Raut (@mangeshraut712). The truthiness guards are the right fix for av 19's AVRational(0, 1). The _stream_duration_s fallback works on both av 18 and 19, and the exporter now stops with a clear LeRobotExportError instead of an uncaught ZeroDivisionError.

We have a few suggested changes before merge:

  • _seek_and_decode: raise when time_base is unset, so the MediaInputError contract from #2134 still holds (frames.py 149-156).
  • _extract_frame_av: return None when time_base is unset, so the existing cv2 fallback still runs (lerobot_handler.py 395-397).
  • Tests: update the new _seek_and_decode test to match, and add backend tests for _extract_frame_av and _EpisodeFrames with an unset time_base.

Dataviewer Backend Pytest is still running. It's the job that exercises the new tests against av 19.0.1, so it'd be great to see that green too. Each inline comment includes a suggestion you can apply directly. Please comment if you have questions about any of these, and we're happy to discuss.

Comment thread evaluation/vlm_judge/frames.py Resolved
…ession tests

- Raise ValueError in evaluation/vlm_judge/frames.py:_seek_and_decode when
  stream has no valid time_base, preserving the MediaInputError contract.
- Return None early in lerobot_handler.py:_extract_frame_av when time_base is
  unset, allowing clean fallback to _extract_frame_cv2.
- Update test_frames.py with module-level imports and assert ValueError.
- Add backend regression tests for lerobot_handler._extract_frame_av and
  lerobot_exporter._EpisodeFrames with unset time_base.
@mangeshraut712

Copy link
Copy Markdown
Contributor Author

Thank you for the detailed and constructive review, Jamie Kim (@jkim323)!

I've updated the PR with all the requested changes in commit 6fc993e5:

  1. _seek_and_decode: Now raises a ValueError("Video stream has no time base; cannot sample frames by timestamp") when time_base is unset, preserving the MediaInputError contract.
  2. _extract_frame_av: Returns None immediately when time_base is unset, cleanly falling back to _extract_frame_cv2.
  3. Tests updated:
    • evaluation/tests/vlm_judge/test_frames.py: Refactored to use module-level imports and typed fakes, asserting the expected ValueError when seeking with an unset time base.
    • Added backend regression tests for lerobot_handler._extract_frame_av and lerobot_exporter._EpisodeFrames verifying that unset time bases avoid zero-division and trigger expected fallback / export errors.

All local tests pass cleanly. Ready for another look!

The new _extract_frame_av case used av.AVRational without importing av, so the test raised NameError before it could check the fallback.
@mangeshraut712

Mangesh Raut (mangeshraut712) commented Nov 21, 2026 •

Copy link
Copy Markdown
Contributor Author

Merged current main in 3c038f7. While rerunning the PR tests I also fixed a NameError in test_extract_frame_av_returns_none_when_time_base_unset: that case used av.AVRational without importing av. The import is in cadb22b.

Locally, pytest data-management/viewer/backend/tests/test_lerobot_export.py data-management/viewer/backend/tests/test_lerobot_handler.py passed (112 tests) and pytest evaluation/tests/vlm_judge/test_frames.py passed (23 tests).

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix: handle unset PyAV time_base after av 19 upgrade

3 participants