Skip to content

Meeting Notes 2026

Sam Freund edited this page May 11, 2026 · 2 revisions

May 9: RoboRIO is dead. Long live systemcore.

Meeting notes

In attendance: Sam, Chris, Charlotte, Jade, Jesse, Rikhil, Banks, Craig, carbotaniuman

  • Code-driven config:
    • Not gonna happen with full config, pipeline settings can be exposed but it gets very complicated very fast
  • Ix-nay jetson-ay
  • HEVC/RTSP stuff goes to Matt
  • MLtag: still in the apriltag pipeline, punt it out to another tab next to apriltag
    • Static cropping still needs to be tested
    • If you don't have MLtag, allow static/dynamic cropping (possibly robot driven)
  • wrt rapid pipeline switching, we're positing it's an issue with setting the camera params, the reason it worked with LL is because it's CSI
  • CI testing for model conversion notebooks (runs like once a week, we just wanna know it's still working)
  • Some sort of python wrapper (exploratory phase, nothing definite)
    • Blockly wrapper would probably go around python, that's blocked (get it) by wpilib finishing blockly stuff
  • FTC support will prolly come if coprocs are allowed
  • camera viz would be a useful thing to add (or give teams a way to figure it out in PV)
    • May be useful to have teams put the pose into the coproc
  • calib
    • wither will update calib docs, Sam is going to record videos
    • button to lazy load calib viz cause you get locked in otherwise (progress bar too)
    • allow for intermittent review while calibrating
    • snapshots on a timer
    • explore video guide for off coproc
  • OD backends
    • Ryzen AI is happening? maybe?
    • tflite CPU should be easy enough
    • opi mesa delegate coming soon:tm: :crossed_fingers:
    • HAILO is backlogged, needs someone to pick it up
    • tests! we got CPU, let's use that to test stuff
  • UI stuff
    • switching ports is priority
    • exporting calib is non-obvious
    • way to export calib pictures from calib menung
    • GPIO UI coming soon, but blocked by GPIO finalization
  • GTSAM
    • Jesse is interested
  • gyro on coproc
    • needs transform on coproc
    • eeshwar said they've got a PR 👀
  • recording
    • document OBS and give a way to sync with logs
    • current goal is recording on coproc (bandwidth limits)
  • Rubik NPU metrics
    • craig will poke 🙏
  • bootstrap coproc database using json config
  • improve diagnostics for calibration
    • consider adding the ability to export the raw camera model from calibration (in mrcal's format). this may require adding more things to the JNI
    • add other visualizations from https://mrcal.secretsauce.net/how-to-calibrate
    • cross validation between two calibrations to verify quality of the calibration
  • camera offsets documentation
    • how can you measure this, physically or in cad
    • visualization ideally

Preliminary notes

Matt's thoughts:

  • Code driven PhotonVision configuration is a thing that was not done
    • Should it ever? It's a neat idea.
  • Jetson support didn't happen. This was probably for the best in the long run.
  • H265/HEVC support is a thing I'd like to push through this year. 2027 has most of the right ingredients in place to make it happen for real this time, and we can finally be free of MJPEG
  • AI-bound-box apriltag detection is another priority this offseason
    • My major question marks here are just about how this integrates. Is it a unique pipeline with lots of duplicate code? Same pipeline with a checkbox?
      • This is a two way door. We can merge as is and change later, as long as its within the offseason.
  • Code cleanup. We aren't churning enough. Need to get more people developing new code (welcome!) and some old code getting refactored too
    • @thatcomputerguy0101 -- thanks for taking on moving away from Jackson + cleaning up database code!
  • ML model conversion notebooks seemed largely to work this year. Any additional feedback from users who trained a custom model?
  • Moving python to semiwrap is (to me) imparitive for reducing our code duplication in photon-lib
    • Explore how LLM tools can help maintain parity across our 3 languages
  • Blocky support! Anything we need to do for this to happen?
  • Docs -- we need to put more calories into docs
    • Robot to camera transform visualization
    • Calibration best practices -- still seeing teams do the dance poorly. Do our docs need more example videos?
  • Architecture discussion around https://github.com/PhotonVision/photonvision/pull/2440 . In general I've been hesitent to do this -- passing data from robot controller -> coprocessor exposes us to latency compensation and syncronization issues. Let's see if we can push to keep all of this logic in robot code.
  • Support for more ML backends -- thanks for championing this, @spacey-sooty ! Where did we land on Ryzen AI again?
    • Great opportunity for some new contributors to help grow the backend
  • UI QOL suggestions! While not breaking muscle memory is good, we should keep iterating and imrpoving. We've largely looked the same since 2021, but the scope has grown a lot. 2027 could be a radical redesign, if anyone is sufficiently motivated.
    • Little things -- like calibration dropdowns remembering how large your chessboard is (annoying pain points)
    • Also larger things -- what sucks about our UX? Not being able to switch cameras between physical ports for one
    • @thatcomputerguy0101 has expanded our GPIO support, but there is still no UI configurator for it. Should there be?
  • Looking for gtsam champions! github.com/mcm001/gtsam-playground is a good starting point. Would like additional investigation into latency comp/splicing observation factors in instead of snapping to nearest. WPILib has a thirdparty-gtsam repo working already -- just moving this into the robot process, playing around. Very good IIC bait/resume project

Jade thoughts:

  • More object detection platforms
    • Move to TFLite for opi
    • hailo didnt happen last year hopefully can this year?
    • cpu support for testing
    • other platforms? amd npu/gpu? cuda?
    • more automated testing !!
  • gyro on coprocessor estimation
    • matt mentioned above adds complexity for a few reasons
    • worth it with systemcore?
  • recording
    • hardware acceleration, links into what matt talked about
    • make it sync with a log in ascope
  • object pose estimation
    • could be good to look into things like 6d pose est models
    • obb?
  • hw accelerated tags
    • halide?
    • cuda tags?
  • calib videos?

@thatcomputerguy0101

  • Metric unit calibration - most ChArUco boards are metric
  • IPV4LL support - makes bench setup easier, although there are some (potentially outdated) concerns with OM5P-AC compatibility
    • USB ethernet gadget support? Allows connecting to the coprocessor's network over a UFP USB port, but configuration may be specific to each coprocessor family
  • More status LED support
    • Green/Yellow for LL up to 3G (mostly done)
    • Addressable (APA102/DotStar?) for future Luma
  • Button on LL cameras:
    • For reset IP address on long press
    • For capturing calibration images?
  • Is Libcamera 0.7 ISP support useful for hardware acceleration?

Sam thoughts

  • def want recording to improve LL parity
  • Churn pipeline indexing
  • Rubik NPU metrics
  • Limit apriltag IDs

Clone this wiki locally