[Question] How to record demos with image observations?

[ywang760]

I'm trying to use record_demos.py to record teleoperation demonstrations for imitation learning. How can I record demos with image observations?

One solution is to include the image as an ObsTerm, but that requires setting concatenate_terms=False in the environment, which breaks the observation structure. Another option I found in stack_ik_rel_blueprint_env_cfg.py implements custom save_image_to_file logic, but this saves to individual image files, instead of to the .hdf5 datasets. Therefore, I wonder what's the preferred way to record demo .hdf5 datasets with images.

[e-ndorfin]

What imitation learning are you aiming for? Are you looking to make a LeRobot dataset? If so, I believe they are saved under other folders so you won't need to add them to the .hdf5 dataset (see https://docs.phospho.ai/learn/lerobot-dataset#how-is-a-lerobot-dataset-organized-on-disk).

Otherwise, I believe you will need to make your own script to add them manually to the .hdf5 dataset. I don't believe this will be too hard though...? Let me know how it goes. (maybe you can adapt something like https://gist.github.com/Franklin-Yao/bf449cafb32334b300ee6b4ed1459019)