stereo_mount assembly: SDK-free ZED + Mid-360 recording rig - #3070
Draft
jeff-hykin wants to merge 13 commits into
Draft
stereo_mount assembly: SDK-free ZED + Mid-360 recording rig#3070jeff-hykin wants to merge 13 commits into
jeff-hykin wants to merge 13 commits into
Conversation
… static tf and memory2 recording - ZedUvcCamera: ZED over plain UVC/OpenCV (no pyzed), splits the side-by-side stereo frame into color_image_left / color_image_right at 60 fps - stereo_mount.urdf: zed pitched up 10 deg in front, mid360 behind pitched down 30 deg; datasheet bounding boxes for zed body, zed lenses, mid360 - assembly.py: StereoMountStaticTf republishes the urdf fixed-joint tree on tf - record.py: StereoMountRecorder (memory2) for both eyes + livox lidar/imu - blueprints/stereo_mount_record.py registered as stereo-mount-record
- Swap raw Mid360 for PointLio(frame_id=world, sensor_frame_id=lidar_link): odometry publishes the moving world->lidar_link edge onto tf - Rename urdf frame lidar_frame->lidar_link so the odometry edge ties into the rig tree (frame names must match exactly for tf lookups) - StereoMountRecorder now extends PointlioRecorder (odometry-baked poses) Live-tested: all streams world-anchored (249/254 images with poses), zero 'No direct transform' warnings.
❌ 1 Tests Failed:
View the top 1 failed test(s) by shortest run time
To view more test analytics, go to the Test Analytics Dashboard |
Piggybacks on the pointlio_lidar pose setter as the liveness signal; if no lidar message lands for 5s the recorder repeats an unmissable error banner (no lidar -> no odometry -> no world poses) with hardware hints. Verified: banner fires/repeats with lidar traffic firewalled, silent on a healthy run (178/182 images world-posed).
# Conflicts: # dimos/robot/all_blueprints.py
Add ZedImu: opens the ZED-M HID interface and replays the official SDK's stream-start handshake (two output reports, captured via usbmon) instead of zed-open-capture's feature-report + 250ms ping. That handshake is what avoids the firmware's sporadic ~1s IMU stream suspensions — verified gap-free over a 16-min soak (774k samples @795Hz, max gap 3.75ms) with no SDK/CUDA. Wire it into StereoMountRecorder (new zed_imu In stream, autoconnected) and add a placeholder zed_imu_link frame to the rig URDF so recorded IMU resolves in tf. Values validated live: 795Hz, |accel|~10.4 m/s^2 (raw factory-uncalibrated), |gyro|~0.03 rad/s stationary.
Raw 2.6MB stereo frames over the pub/sub transport throttled the whole recording to ~5.7fps (and starved every other stream). Encode h264 in the capture process before publishing: ZedUvcCamera now outputs color_image_left/ right as CompressedVideo (PyAV, libx264 default / h264_nvenc option), a few KB per frame. That unclogs the pipeline — odometry/lidar return to full rate (29/10Hz) and stereo hits ~40fps (the ZED-M UVC caps at 56fps and the dimos coordinator ~40 under combined load; encode itself does 130fps/eye). Also publish camera_info_left/right from the factory .conf (auto-found in /usr/local/zed/settings), recorded alongside. Recorder Ins updated to CompressedVideo; decode downstream with H264Decoder. Verified live (90s db): all streams world-posed ~100%, h264 packets decode back to 1280x720 frames.
…lback libx264 on the CPU contends with the capture color-convert + Point-LIO + the recorder and caps stereo at ~40fps. Offloading to h264_nvenc frees those cores and the recording reaches the camera's own ceiling (56fps @2560x720 over USB). The blueprint requests nvenc; _make_encoder falls back to libx264 with a warning where nvenc isn't available, so it still runs on non-GPU boxes (~40fps). 90s db: stereo 56/56fps, odom 29.7Hz, lidar 10Hz, zed_imu 85Hz, camera_info 1Hz, tf 74Hz — all ~100%% world-posed.
The base recorder dispatches every message through a LATEST-coalescing mailbox and does a tf pose-lookup + committed insert per message, which capped the 800Hz IMU at ~70Hz (>90% of samples conflated away). Override _port_to_stream for the zed_imu port only: subscribe raw (no coalescing → no drops), skip the tf lookup (IMU has no per-sample world pose), buffer off the event loop, and flush every 50ms as one transaction (N inserts, single commit) straight through the stream backend. No new message type. Live 30s: zed_imu 25546 rows @799.3Hz (was ~70), median gap 1.25ms, 1 gap >15ms; video/odom/lidar unchanged; clean final flush on stop.
…b, simpler recorder - livox_sdk_config.hpp: pre-declare lidar_ip and auto-enable multicast for real (non-local) lidars so macOS discovery works without extra flags - sdkless_camera: publish camera_info from the factory SN*.conf (auto-detect or conf_path), with download instructions when missing - sdkless_imu: find homebrew hidapi via absolute paths on macOS - recorder: standalone Recorder with lidar/odometry ports matching PointLio outputs (no remappings), guard IMU flush loop against post-stop db writes
…ate quirk ZedMid360Recorder now just adds lidar/odometry ports on top of the reusable ZedRecorder (stereo h264 + camera_info + batched 800 Hz IMU path). The sdkless_imu docstring documents the Apple Silicon root-port ~533 Hz cap and the USB-hub workaround (verified 795-800 Hz).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
needed for stereo recording ( DIM-1196 ), blueprints+modules
ref DIM-1288