EncodedDepthImage archetype and integration with viewer and re_mcap #11877

makeecat · 2025-11-12T16:27:22Z

What

This is a draft PR to provide prototype for EncodedDepthImage. This PR needs to be split into sub-PRs once reviewed by @oxkitsune

Implemented the EncodedDepthImage archetype
Introduced a new utility crate re_depth_compression to host rvl codecs implementation
Integrated EncodedDepthImage archetype into viewer
Supported the EncodedDepthImage archetype in re_mcap.

github-actions

Hi! Thanks for opening this pull request.

Because this is your first time contributing to this repository, make sure you've read our Contributor Guide and Code of Conduct.

makeecat · 2025-11-13T13:08:09Z

Hi @oxkitsune , I believe I am comfortable with the prototype, the proof concept works well with lint and my local mcap bag with RVL-encoded depth image. I am ready to submit a separate PR for EncodedDepthImage archetype.

However, I am not very confident about my changes in re_view_spatial crate. I refactored the crates/viewer/re_view_spatial/src/visualizers/depth_images.rs to support crates/viewer/re_view_spatial/src/visualizers/encoded_depth_images.rs as a separate visualizer. I am looking for comments on that part.

If you are happy with the proof-of-concept demo and the design of the EncodedDepthImage archetype, I can prepare the first PR for EncodedDepthImage archetype.

oxkitsune · 2025-11-14T11:37:46Z

This demo looks great, thanks for putting it together!

Go ahead and open the PR for the EncodedDepthImage archetype; I’ll leave any implementation notes there.

makeecat · 2025-11-14T15:35:10Z

According to our discussion, we will not split this PR into small PRs.

There are a few decisions we need to make:

whether add mediatype
whether leave re_depth_compression as a crate or integrate it into exsting logic
fallback detection of different format for decoding if the data didn't provide type metadata
When RVL metadata disagrees with the logged ImageFormat, the code just logs a warning and still constructs an ImageInfo with the caller’s width/height.
I don't like the way I refactored the DepthImagevisualizer: viewer/re_view_spatial/src/visualizers/depth_image.rs, I would like to avoid changing too much of existing depth image visualizer, but I didn't come up with a solution. The reason I have to refactor it is to expose some components from the original depth image visualizer to support encoded depth image visualizer.

makeecat · 2025-11-18T13:41:35Z

After discussing with @oxkitsune

We plan to add support of rvl as a media_type
We plan to squash the logic of re_depth_completion crate into re_viewer_context.
We plan to support MediaType::guess_from_data to support fallback detection of RVL data
We need further discussion to avoid refactoring too much of viewer/re_view_spatial/src/visualizers/depth_image.rs

…her, add test for guess_from_data

oxkitsune · 2025-11-18T11:07:13Z

crates/utils/re_depth_compression/src/lib.rs

+pub use ros_rvl::{
+    RvlDecodeError, RvlMetadata, decode_ros_rvl_f32, decode_ros_rvl_u16, parse_ros_rvl_metadata,
+};


If the module's already pub, I think it's neater to avoid re-exporting. This means that in the future, when adding more formats, we can easily access it using re_depth_compression::<format>.

oxkitsune · 2025-11-26T10:59:36Z

crates/utils/re_mcap/src/parsers/ros2msg/sensor_msgs/compressed_image.rs

+    PayloadLengthMismatch { width: u32, height: u32 },
+}
+
+fn parse_ros_rvl_metadata(data: &[u8]) -> Result<RvlMetadata, RvlDecodeError> {


IIUC this is the same metadata parsing from crates/viewer/re_viewer_context/src/utils/depth.rs right? Is there any way we can avoid duplicating this?

done, refactored into re_depth_compression

oxkitsune · 2025-11-26T11:31:43Z

crates/viewer/re_viewer_context/src/cache/image_decode_cache.rs

+            format.datatype()
+        )));
+    }
+


Would be a good place to validate that the decoded_format is actually single channel data. This now support RGB pngs

now we have enforced single-channel PNG

oxkitsune · 2025-11-26T11:46:34Z

crates/store/re_types/src/components/media_type_ext.rs

+            // Read width and height from bytes 12-15 and 16-19 respectively
+            let width = u32::from_le_bytes([buf[12], buf[13], buf[14], buf[15]]);
+            let height = u32::from_le_bytes([buf[16], buf[17], buf[18], buf[19]]);
+


Given the structure given above, I think that this, in its current state, will lead to a lot of false positives.
I think we should also check that depth_quant_a and depth_quant_b are both valid floats with sane ranges (e.g. not NaN or inf and not absurdly large)

Suggested change

let depth_quant_a = f32::from_le_bytes([buf[4], buf[5], buf[6], buf[7]]);

let depth_quant_b = f32::from_le_bytes([buf[8], buf[9], buf[10], buf[11]]);

// Reject NaNs/infs

if !depth_quant_a.is_finite() || !depth_quant_b.is_finite() {

return false;

}

// Reject insane magnitudes

if depth_quant_a <= 0.0 || depth_quant_a > 1e4 || depth_quant_b.abs() > 1e4 {

return false;

}

oxkitsune · 2025-11-26T11:54:14Z

crates/store/re_types/src/components/media_type_ext.rs

 }
+
+#[test]
+fn test_guess_from_data() {


most of this test is for MediaType::rvl(), which I think is fine. No need to test the GLB/stl here in my opinion. Test can also be renamed to test_guess_from_data_rvl

oxkitsune · 2025-11-26T14:14:36Z

crates/store/re_types/definitions/rerun/archetypes/encoded_depth_image.fbs

+  /// Image format (width, height, datatype).
+  ///
+  /// Unlike generic [archetypes.EncodedImage], most depth codecs do not carry full headers,
+  /// so width/height/datatype must be logged explicitly.
+  format: rerun.components.ImageFormat ("attr.rerun.component_required", order: 1100);


Suggested change

/// Image format (width, height, datatype).

///

/// Unlike generic [archetypes.EncodedImage], most depth codecs do not carry full headers,

/// so width/height/datatype must be logged explicitly.

format: rerun.components.ImageFormat ("attr.rerun.component_required", order: 1100);

/// Image format (width, height, datatype).

///

/// Standard image formats like PNG or JPEG include this metadata in their headers,

/// allowing [archetypes.EncodedImage] to be self-describing. Depth images, however,

/// typically use headerless codecs, so this information must be provided explicitly.

format: rerun.components.ImageFormat ("attr.rerun.component_required", order: 1100);

crates/store/re_types/src/archetypes/encoded_depth_image_ext.rs

crates/store/re_types/src/components/media_type_ext.rs

oxkitsune · 2025-11-26T14:30:54Z

crates/utils/re_depth_compression/src/ros_rvl.rs

+        bytes
+    }
+
+    fn encode_rvl(values: &[u16]) -> Vec<u8> {


Since you essentially implement a full encoder here as well, maybe it makes sense to just bundle the decoder + encoder up in a separate rust crate, so that it can be used by others as well?

Just a suggestion as I was surprised to see an encoder here as well 😅

refactored the parsing/encoding/decoding logic back to re_depth_compression crate

docs/snippets/all/archetypes/encoded_depth.png

oxkitsune · 2025-11-26T14:46:43Z

crates/viewer/re_view_spatial/src/visualizers/encoded_depth_image.rs

as far as I can see, this is 99% the same as the depth_images visualizer. The main difference is the component identifiers (e.g. DepthImage::meter() vs EncodedDepthImage::meter()).

Can we move the implementation to a unified function that takes the component identifiers?

improve RVL media-type detection via the shared parser plus quantization sanity checks, trimmed the guess-from-data test to RVL coverage, and fixed the RVL doc link. improve depth decoding: enforce single-channel PNG depth, error on RVL/ImageFormat resolution mismatches, and validate decoded byte counts.

…mat and 1000.0 for integer format

…_context

…ion helper and remove duplicated rendering logic

makeecat added 8 commits November 11, 2025 18:41

Handle RVL compressed depth images in re_mcap

b9bbdf8

add encoded depth image archetype

6558704

add re_depth_compression (RVL only now)

aec5459

integrate EncodedDepthImage into viewer

360fcda

integrate EncodedDepthImage into re_mcap

37dc6df

add docs

dbf3622

fix codegen lint error of docs

0c52174

resolve lint error of re_depth_compression

ceda8bd

oxkitsune self-requested a review November 12, 2025 16:29

github-actions bot reviewed Nov 12, 2025

View reviewed changes

makeecat mentioned this pull request Nov 12, 2025

Add EncodedDepthImage archetype #9046

Open

makeecat added 15 commits November 12, 2025 12:12

Merge branch 'main' into users/yang/EncodedDepthImage

77deb1e

change version for re_depth_compression

5c300c3

add re_depth_compression in workspace.dependencies

a4e7668

fix README of re_depth_compression and pass lint

0c59b34

add EncodedDepthImage into rerun_py gen_common_index.py

50a7954

add empty example for encoded_depth_image to pass lint

2baf7e0

fix empty example for encoded_depth_image to pass lint

1ed9178

pass clippy lint

9a54e99

fix increment_count for EncodedDepthImage in view_2d

b3d6328

fmt

b6b7f91

use depth image visualizer and encoded depth image visualizer

da0acda

re-export EncodedDepthImage in rerun_py

203f027

add EncodedDepthImage.snap for re_viewer tests snapshot

aea216a

add example for EncodedDepthImage

5eeeb54

allow encoded_depth.png in check_large_files

cfd7c38

makeecat marked this pull request as ready for review November 13, 2025 19:30

add PNG decoding for EncodedDepthImage

013babb

makeecat added 3 commits November 14, 2025 13:29

fix compare_snippet_output lint

f7bfdfc

fix codegen lint

8100b1f

fix cpp-fmt

949c4af

makeecat mentioned this pull request Nov 18, 2025

CompressedDepth decoding for re_mcap #11702

Open

makeecat added 5 commits November 18, 2025 09:14

fix rs lint

888b0c3

support rvl as media_type

723d665

Refactor rvl logic into re_viewer_context instead of separte crate

7447e55

fix codegen lint

5446d7a

add rvl_matcher for MediaType::guess_from_data, fix typo for stl_matc…

d0e267e

…her, add test for guess_from_data

oxkitsune added 📺 re_viewer affects re_viewer itself include in changelog and removed do-not-merge Do not merge this PR labels Nov 24, 2025

oxkitsune requested changes Nov 26, 2025

View reviewed changes

oxkitsune reviewed Nov 26, 2025

View reviewed changes

makeecat added 14 commits November 26, 2025 15:36

fmt

83d7ae7

change from_encoded_bytes to from_file_contents

6129fe0

improve documentation for encoded_depth_image.fbs

ed7448e

ensure decoder always progresses forward

b32af94

avoid allocation of zeros in decode_ros_rvl_u16

453fdcf

add doc for depth image archetype: default to 1.0 for float depth for…

2d8d40b

…mat and 1000.0 for integer format

fix lint for doc of encoded_depth_image.fbs

aa9dbb1

add README for re_depth_compression

36e87ae

Merge branch 'main' into users/yang/EncodedDepthImage

68d331e

fix link to application/rvl

55a1da4

allow encoded depth images to guess media type from data in re_viewer…

6aa07a3

…_context

revert disparity initialization for ros_rvl

4a522b8

Refactor depth and encoded depth visualizers to share a common execut…

64a5b87

…ion helper and remove duplicated rendering logic

+let depth_quant_a = f32::from_le_bytes([buf[4], buf[5], buf[6], buf[7]]);
+let depth_quant_b = f32::from_le_bytes([buf[8], buf[9], buf[10], buf[11]]);
+// Reject NaNs/infs
+if !depth_quant_a.is_finite() || !depth_quant_b.is_finite() {
+    return false;
+}
+// Reject insane magnitudes
+if depth_quant_a <= 0.0 || depth_quant_a > 1e4 || depth_quant_b.abs() > 1e4 {
+    return false;
+}

EncodedDepthImage archetype and integration with viewer and re_mcap #11877

Are you sure you want to change the base?

EncodedDepthImage archetype and integration with viewer and re_mcap #11877

Uh oh!

Conversation

makeecat commented Nov 12, 2025

Related

What

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

makeecat commented Nov 13, 2025

Uh oh!

oxkitsune commented Nov 14, 2025

Uh oh!

makeecat commented Nov 14, 2025

Uh oh!

makeecat commented Nov 18, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants