Skip to content

stereo_mount assembly: SDK-free ZED + Mid-360 recording rig - #3070

Draft
jeff-hykin wants to merge 13 commits into
mainfrom
jeff/feat/zed_rec
Draft

stereo_mount assembly: SDK-free ZED + Mid-360 recording rig#3070
jeff-hykin wants to merge 13 commits into
mainfrom
jeff/feat/zed_rec

Conversation

@jeff-hykin

@jeff-hykin jeff-hykin commented Jul 20, 2026

Copy link
Copy Markdown
Member

needed for stereo recording ( DIM-1196 ), blueprints+modules

DIMOS_POINTLIO_LIDAR_IP=192.168.1.107 DIMOS_POINTLIO_HOST_IP=192.168.1.5 dimos run zed-mid360-record

ref DIM-1288

… static tf and memory2 recording

- ZedUvcCamera: ZED over plain UVC/OpenCV (no pyzed), splits the side-by-side
  stereo frame into color_image_left / color_image_right at 60 fps
- stereo_mount.urdf: zed pitched up 10 deg in front, mid360 behind pitched
  down 30 deg; datasheet bounding boxes for zed body, zed lenses, mid360
- assembly.py: StereoMountStaticTf republishes the urdf fixed-joint tree on tf
- record.py: StereoMountRecorder (memory2) for both eyes + livox lidar/imu
- blueprints/stereo_mount_record.py registered as stereo-mount-record
- Swap raw Mid360 for PointLio(frame_id=world, sensor_frame_id=lidar_link):
  odometry publishes the moving world->lidar_link edge onto tf
- Rename urdf frame lidar_frame->lidar_link so the odometry edge ties into
  the rig tree (frame names must match exactly for tf lookups)
- StereoMountRecorder now extends PointlioRecorder (odometry-baked poses)

Live-tested: all streams world-anchored (249/254 images with poses), zero
'No direct transform' warnings.
@codecov

codecov Bot commented Jul 20, 2026

Copy link
Copy Markdown

❌ 1 Tests Failed:

Tests completed Failed Passed Skipped
3141 1 3140 176
View the top 1 failed test(s) by shortest run time
dimos.robot.test_all_blueprints_generation::test_all_blueprints_is_current
Stack Traces | 3.71s run time
def test_all_blueprints_is_current() -> None:
        root = DIMOS_PROJECT_ROOT / "dimos"
        all_blueprints, all_modules = _scan_for_blueprints(root)
    
        common = set(all_blueprints.keys()) & set(all_modules.keys())
        assert not common, (
            f"Names must be unique across blueprints and modules, "
            f"but these appear in both: {sorted(common)}"
        )
    
        generated_content = _generate_all_blueprints_content(all_blueprints, all_modules)
    
        file_path = root / "robot" / "all_blueprints.py"
    
        if "CI" in os.environ:
            if not file_path.exists():
                pytest.fail(f"all_blueprints.py does not exist at {file_path}")
    
            current_content = file_path.read_text()
            if current_content != generated_content:
                diff = difflib.unified_diff(
                    current_content.splitlines(keepends=True),
                    generated_content.splitlines(keepends=True),
                    fromfile="all_blueprints.py (current)",
                    tofile="all_blueprints.py (generated)",
                )
                diff_str = "".join(diff)
>               pytest.fail(
                    f"all_blueprints.py is out of date. Run "
                    f"`pytest dimos/robot/test_all_blueprints_generation.py` locally to update.\n\n"
                    f"Diff:\n{diff_str}"
                )
E               Failed: all_blueprints.py is out of date. Run `pytest dimos/robot/test_all_blueprints_generation.py` locally to update.
E               
E               Diff:
E               --- all_blueprints.py (current)
E               +++ all_blueprints.py (generated)
E               @@ -300,5 +300,6 @@
E                    "zed-imu": "dimos.hardware.sensors.camera.zed.sdkless_imu.ZedImu",
E                    "zed-mid360-recorder": "dimos.robot.assembly.zed_mid360.record.ZedMid360Recorder",
E                    "zed-mid360-static-tf": "dimos.robot.assembly.zed_mid360.assembly.ZedMid360StaticTf",
E               +    "zed-recorder": "dimos.hardware.sensors.camera.zed.sdkless_recorder.ZedRecorder",
E                    "zed-uvc-camera": "dimos.hardware.sensors.camera.zed.sdkless_camera.ZedUvcCamera",
E                }

all_blueprints = {'a1z-planner-coordinator': 'dimos.robot.manipulators.a1z.blueprints.basic:a1z_planner_coordinator', 'alfred-nav': 'di...rs.a1z.blueprints.basic:coordinator_a1z', 'coordinator-basic': 'dimos.control.blueprints.basic:coordinator_basic', ...}
all_modules = {'alfred-high-level': 'dimos.robot.diy.alfred.effector_high_level.AlfredHighLevel', 'arm-command-module': 'dimos.teleo...t_extensions.ArmTeleopModule', 'b-box-navigation-module': 'dimos.navigation.bbox_navigation.BBoxNavigationModule', ...}
common     = set()
current_content = '# Copyright 2025-2026 Dimensional Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the "License");\n# you m...embly.ZedMid360StaticTf",\n    "zed-uvc-camera": "dimos.hardware.sensors.camera.zed.sdkless_camera.ZedUvcCamera",\n}\n'
diff       = <generator object unified_diff at 0xffcb01cef920>
diff_str   = '--- all_blueprints.py (current)\n+++ all_blueprints.py (generated)\n@@ -300,5 +300,6 @@\n     "zed-imu": "dimos.hardw..._recorder.ZedRecorder",\n     "zed-uvc-camera": "dimos.hardware.sensors.camera.zed.sdkless_camera.ZedUvcCamera",\n }\n'
file_path  = PosixPath('.../dimos/robot/all_blueprints.py')
generated_content = '# Copyright 2025-2026 Dimensional Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the "License");\n# you m...ss_recorder.ZedRecorder",\n    "zed-uvc-camera": "dimos.hardware.sensors.camera.zed.sdkless_camera.ZedUvcCamera",\n}\n'
root       = PosixPath('.../dimos/dimos/dimos')

dimos/robot/test_all_blueprints_generation.py:76: Failed

To view more test analytics, go to the Test Analytics Dashboard
📋 Got 3 mins? Take this short survey to help us improve Test Analytics.

Piggybacks on the pointlio_lidar pose setter as the liveness signal; if no
lidar message lands for 5s the recorder repeats an unmissable error banner
(no lidar -> no odometry -> no world poses) with hardware hints.

Verified: banner fires/repeats with lidar traffic firewalled, silent on a
healthy run (178/182 images world-posed).
Add ZedImu: opens the ZED-M HID interface and replays the official SDK's
stream-start handshake (two output reports, captured via usbmon) instead of
zed-open-capture's feature-report + 250ms ping. That handshake is what avoids
the firmware's sporadic ~1s IMU stream suspensions — verified gap-free over a
16-min soak (774k samples @795Hz, max gap 3.75ms) with no SDK/CUDA.

Wire it into StereoMountRecorder (new zed_imu In stream, autoconnected) and add
a placeholder zed_imu_link frame to the rig URDF so recorded IMU resolves in tf.
Values validated live: 795Hz, |accel|~10.4 m/s^2 (raw factory-uncalibrated),
|gyro|~0.03 rad/s stationary.
Raw 2.6MB stereo frames over the pub/sub transport throttled the whole
recording to ~5.7fps (and starved every other stream). Encode h264 in the
capture process before publishing: ZedUvcCamera now outputs color_image_left/
right as CompressedVideo (PyAV, libx264 default / h264_nvenc option), a few KB
per frame. That unclogs the pipeline — odometry/lidar return to full rate
(29/10Hz) and stereo hits ~40fps (the ZED-M UVC caps at 56fps and the dimos
coordinator ~40 under combined load; encode itself does 130fps/eye).

Also publish camera_info_left/right from the factory .conf (auto-found in
/usr/local/zed/settings), recorded alongside. Recorder Ins updated to
CompressedVideo; decode downstream with H264Decoder.

Verified live (90s db): all streams world-posed ~100%, h264 packets decode
back to 1280x720 frames.
…lback

libx264 on the CPU contends with the capture color-convert + Point-LIO + the
recorder and caps stereo at ~40fps. Offloading to h264_nvenc frees those cores
and the recording reaches the camera's own ceiling (56fps @2560x720 over USB).
The blueprint requests nvenc; _make_encoder falls back to libx264 with a warning
where nvenc isn't available, so it still runs on non-GPU boxes (~40fps).

90s db: stereo 56/56fps, odom 29.7Hz, lidar 10Hz, zed_imu 85Hz, camera_info
1Hz, tf 74Hz — all ~100%% world-posed.
The base recorder dispatches every message through a LATEST-coalescing mailbox
and does a tf pose-lookup + committed insert per message, which capped the
800Hz IMU at ~70Hz (>90% of samples conflated away). Override _port_to_stream
for the zed_imu port only: subscribe raw (no coalescing → no drops), skip the
tf lookup (IMU has no per-sample world pose), buffer off the event loop, and
flush every 50ms as one transaction (N inserts, single commit) straight through
the stream backend. No new message type.

Live 30s: zed_imu 25546 rows @799.3Hz (was ~70), median gap 1.25ms, 1 gap
>15ms; video/odom/lidar unchanged; clean final flush on stop.
…b, simpler recorder

- livox_sdk_config.hpp: pre-declare lidar_ip and auto-enable multicast for
  real (non-local) lidars so macOS discovery works without extra flags
- sdkless_camera: publish camera_info from the factory SN*.conf (auto-detect
  or conf_path), with download instructions when missing
- sdkless_imu: find homebrew hidapi via absolute paths on macOS
- recorder: standalone Recorder with lidar/odometry ports matching PointLio
  outputs (no remappings), guard IMU flush loop against post-stop db writes
…ate quirk

ZedMid360Recorder now just adds lidar/odometry ports on top of the reusable
ZedRecorder (stereo h264 + camera_info + batched 800 Hz IMU path). The
sdkless_imu docstring documents the Apple Silicon root-port ~533 Hz cap and
the USB-hub workaround (verified 795-800 Hz).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant