feat(msgs): typed CompressedImage sensor msg

[spomichter]

Problem

dimos has no first-class compressed image type. jpeg support is bolted onto individual transports (JpegLcmTransport, JpegShmTransport, nothing for zenoh/webrtc), JpegShmTransport loses timestamps on decode, and every new transport needs its own jpeg variant. meanwhile raw images are big — 720p rgb is 2.76MB per frame, ~40MB/s at camera rate. thats fine for shared memory on one host but it kills lcm the moment you leave it: one dropped udp fragment loses the whole frame.

CompressedImage is a first-class msg (jpeg/png bytes + timestamp + frame_id, wraps the dimos_lcm binding that already existed):

CompressedImage.from_image(img, format="jpeg", quality=75, max_width=None)  # encode
ci.decode()                     # -> Image, ts/frame_id survive the round trip
ci.lcm_encode() / lcm_decode()  # typed wire msg on any transport
ci.to_rerun()                   # rr.EncodedImage — viewer renders jpeg with NO pixel decode

depth/16-bit images can't go through jpeg — use format="png" (lossless, covers GRAY16). float DEPTH raises on both.

Benchmark report: raw Image vs CompressedImage

(numbers produced with the bench harness in this PR; the codec pin is the bench mechanism, the wins are the type's)

full grid run on this dev box (x86_64, 32 cpus), 6 runs x 10 frames per cell, sequential round-trips. what "latency" measures: one broadcast() call until the subscriber callback fires, on localhost — so it includes the jpeg encode+decode cpu for the codec path but near-zero wire cost. thats why codec shows higher latency here: raw is two memcpys, codec pays ~5ms encode + ~5ms decode of real cpu. localhost is raw's best case — on a real 1Gbps link the raw frame pays ~22ms of transmit time alone vs <1ms for the jpeg, and over webrtc/wan its not close. what you buy for those ms: 8-40x less wire and the difference between frames arriving or not (see drops).

jpeg codec cpu cost (median of 20, ms)

                    raw        jpeg       ratio      encode      decode
480p  q75        0.92MB        49KB         19x         3.0         3.1
720p  q75        2.76MB       142KB         19x         5.1         5.2
1080p q75        6.22MB       315KB         20x         7.6         6.6
720p  q50        2.76MB        69KB         40x         4.7         4.6
720p  q90        2.76MB       345KB          8x         5.0         5.5

round-trip latency, median ms (LOST = zero frames ever arrived)

                     480p        720p       1080p
lcm   raw            LOST        LOST         5.5
lcm   codec           6.4        10.3        17.9
zenoh raw             0.7         0.9         6.9
zenoh codec           6.4         9.6        15.2
pshm  raw             0.2         0.6         1.6
pshm  codec           6.2         9.4        15.3

round-trip latency, p95 ms

                     480p        720p       1080p
lcm   raw            LOST        LOST         6.5
lcm   codec           7.0        11.8        21.2
zenoh raw             0.7         1.0         8.6
zenoh codec           6.8        11.4        18.8
pshm  raw             0.4         1.4         3.3
pshm  codec           6.8        11.3        17.4

dropped frames (of 60)

                     480p        720p       1080p
lcm   raw           60/60       60/60       43/60
lcm   codec          0/60        0/60       37/60
zenoh raw            0/60        0/60        0/60
zenoh codec          0/60        0/60        0/60
pshm  raw            0/60        0/60        0/60
pshm  codec          0/60        0/60        0/60

lcm raw 480p/720p lost every single frame on this box (untuned kernel udp buffers — CI=1 skips the lcm autoconf sysctl tuning, so this is what a fresh install sees). note the 1080p codec drops too: 315KB @ q75 is already past what untuned lcm handles reliably — thats what the quality/max_width knobs are for (q50 = 69KB, zero drops).

jpeg quality sweep (lcm, 720p)

             wire      median         fps       drops
q50          69KB         9.2         109        0/60
q75         142KB        10.3          97        0/60
q90         345KB        13.7          73       40/60

sustained 14Hz 720p camera feed for 8s (the go2 workload)

              delivered hz (target 14)
lcm   raw     ▊ 0.8
lcm   codec   ██████████████ 14.0
zenoh raw     ██████████████ 14.0
zenoh codec   ██████████████ 14.0

this is the headline: publish a go2-style camera feed at 14Hz over lcm and the raw pipeline delivers 0.8 fps. the codec pipeline delivers all 112 frames.

throughput capacity, fps (1 / median latency)

                     480p        720p       1080p
lcm   raw            LOST        LOST         183
lcm   codec           156          97          56
zenoh raw            1484        1098         145
zenoh codec           157         105          66
pshm  raw            6239        1765         639
pshm  codec           162         107          65

codec capacity is cpu-bound (~1/encode+decode time) and the same on every transport — ~100fps at 720p, 7x a 14Hz camera. pshm raw shows why shm stays the on-host raw escape hatch: 1765fps at 720p, zero encode cost.

end-to-end replay benchmark (full nav stack, jetson-class 4-core profile: raw delivers 6.5 of 14 fps, compressed delivers all of it, with LESS total cpu) is in #2831 with charts.

notes: webrtc isnt in the grid because its loopback provider is an in-memory dict (stack overhead only, no real SCTP — thats P1 of the webrtc backend spec) and the cloudflare path is cred-gated. the vlm/agent blueprints (spatial, temporal-memory, agentic) are the biggest indirect winners — they re-encode jpeg per VLM call today, with option 5 they get wire bytes for free.

follow-ups (not this PR): option 5 migration per #2831 (GO2Connection -> Out[CompressedImage] + consumers, needs a replay-dataset compat answer since recorded datasets store raw Image), migrating JpegLcmTransport/JpegShmTransport onto CompressedImage, webrtc ImageDecimation emitting this type.

Breaking Changes

None

How to Test

CI=1 uv run pytest dimos/msgs/sensor_msgs/test_CompressedImage.py dimos/protocol/pubsub/benchmark/test_replay_bench.py -s

14 tests incl. real lcm+zenoh round-trips (skipped where native libturbojpeg is missing — and ci.yml now installs it so CI cant skip silently). benchmark alone, prints one grid at the end:

CI=1 uv run pytest dimos/protocol/pubsub/benchmark/test_replay_bench.py -k benchmark -s

e2e replay bench: python -m dimos.protocol.pubsub.benchmark.tool_replay_bench --blueprint unitree-go2-basic --mode codec --duration 30 --out /tmp/bench (re-ran post-trim: 473 frames / 33s = 14.3fps, full delivery). full suite + mypy run clean locally.

Contributor License Agreement

• I have read and approved the CLA

[codecov]

Codecov Report

❌ Patch coverage is 76.48305% with 111 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
...mos/protocol/pubsub/benchmark/tool_replay_bench.py	31.91%	94 Missing and 2 partials ⚠️
dimos/msgs/sensor_msgs/CompressedImage.py	86.73%	7 Missing and 6 partials ⚠️
...mos/protocol/pubsub/benchmark/test_replay_bench.py	98.33%	1 Missing and 1 partial ⚠️

@@            Coverage Diff             @@
##             main    #2814      +/-   ##
==========================================
+ Coverage   75.34%   75.36%   +0.01%     
==========================================
  Files        1149     1153       +4     
  Lines      110476   110948     +472     
  Branches    10007    10627     +620     
==========================================
+ Hits        83234    83611     +377     
- Misses      24378    24465      +87     
- Partials     2864     2872       +8     

Flag	Coverage Δ
OS-ubuntu-24.04-arm	69.33% <74.78%> (+0.03%)	⬆️
OS-ubuntu-latest	71.40% <76.05%> (+0.03%)	⬆️
Py-3.10	71.39% <74.78%> (+0.02%)	⬆️
Py-3.11	71.39% <75.63%> (+0.03%)	⬆️
Py-3.12	71.39% <75.21%> (+0.02%)	⬆️
Py-3.13	71.39% <74.78%> (+0.03%)	⬆️
Py-3.14	71.39% <74.78%> (+0.02%)	⬆️
Py-3.14t	71.39% <74.78%> (+0.02%)	⬆️
SelfHosted-Large	29.36% <24.15%> (-0.03%)	⬇️
SelfHosted-Linux	35.75% <24.36%> (-0.07%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
dimos/conftest.py	81.36% <100.00%> (+0.97%)	⬆️
dimos/msgs/sensor_msgs/test_CompressedImage.py	100.00% <100.00%> (ø)
dimos/protocol/pubsub/impl/rospubsub_conversion.py	70.00% <ø> (ø)
...mos/protocol/pubsub/benchmark/test_replay_bench.py	98.33% <98.33%> (ø)
dimos/msgs/sensor_msgs/CompressedImage.py	86.73% <86.73%> (ø)
...mos/protocol/pubsub/benchmark/tool_replay_bench.py	31.91% <31.91%> (ø)

... and 4 files with indirect coverage changes

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[greptile-apps]

Greptile Summary

This PR adds a typed compressed-image message and benchmark support. The main changes are:

• JPEG and PNG CompressedImage encode/decode support.
• LCM serialization and Rerun encoded-image conversion.
• Benchmark-only compressed transport wrapper and CI TurboJPEG setup.

Confidence Score: 4/5

ROS bridge conversion and alpha-bearing PNG decoding can return invalid message data.

• ROS CompressedImage messages do not follow the bridge's flattened-header conversion contract.
• Valid PNG images with alpha are returned with a mismatched format tag.
• Benchmark shutdown can write incomplete or duplicated result records.

dimos/msgs/sensor_msgs/CompressedImage.py; dimos/protocol/pubsub/benchmark/tool_replay_bench.py

Important Files Changed

Filename	Overview
dimos/msgs/sensor_msgs/CompressedImage.py	Adds compressed image encoding, decoding, LCM serialization, and Rerun conversion; ROS conversion and alpha-PNG handling are incomplete.
dimos/protocol/pubsub/benchmark/tool_replay_bench.py	Adds benchmark codec and replay instrumentation; shutdown can race callback-side buffer flushing.
dimos/msgs/sensor_msgs/test_CompressedImage.py	Adds JPEG, PNG, LCM, metadata, resize, validation, and Rerun coverage.
.github/workflows/ci.yml	Installs the TurboJPEG system library for Ubuntu test jobs.
dimos/conftest.py	Adds TurboJPEG availability detection and test marker registration.

_{Reviews (1): Last reviewed commit: "Merge branch 'main' into feat/compressed..." | Re-trigger Greptile}

resolved by another PR