A question about GlobalStep and the Checkpoint Resumption

[Chloe-Yan-3465]

Hello Horizon Robotics, there is a question I'd like to ask.
Thankyou for your answer.
(Detailed in the issue.md)
issue.md

[linxuewu]

If you want to resume, set the config with resume_from='./workspace/checkpoints/checkpoint_11/'.

The item checkpoint is used for finetuning, not resume.

[Chloe-Yan-3465]

OK I got it. Thankyou!! =)

[Chloe-Yan-3465]

We evaluated the place_empty_cup task and obtained a 52.0% success rate (terminal screenshot attached).

We'd like to ask whether the camera depth configuration wasn't specifically set up in the eval process. The eval only takes the camera_config.yml as input, which doesn't specify rgbd settings - it only includes height and width parameters. During evaluation, we noticed many data points lack depth information (as shown in the terminal output).

even though we set depth=True during data collection. We suspect that the evaluation might not be fully utilizing depth information, which could explain why we can't reproduce the 85% success rate reported in the paper. Where can we get the depth? (Below is the sem_policy/deploy_policy.py,we make some adjustment of camera_data regarding code)Could you please provide some guidance on this?

def encode_obs(observation, data_transforms, instruction):
    images = []
    depths = []
    t_world2cam = []
    intrinsics = []
    joint_state = []

    missing_depth_cameras = []

    for cam_name, camera_data in observation["observation"].items():
        # 添加 RGB 图像
        images.append(camera_data["rgb"])

        # 处理 depth，有就用，没有就构造全零占位
        if "depth" in camera_data:
            depths.append(camera_data["depth"] / 1000)  # mm -> m
        else:
            print(f"[Warning] Camera '{cam_name}' has no depth, inserting zero placeholder.")
            h, w = camera_data["rgb"].shape[:2]
            fake_depth = np.zeros((h, w), dtype=np.float32)
            depths.append(fake_depth)
            missing_depth_cameras.append(cam_name)

        # 添加相机外参
        _ext = np.eye(4, dtype=np.float32)
        _ext[:3] = camera_data["extrinsic_cv"]
        t_world2cam.append(_ext)

        # 添加相机内参
        _int = np.eye(4, dtype=np.float32)
        _int[:3, :3] = camera_data["intrinsic_cv"]
        intrinsics.append(_int)

Thanks in advance for your help! we really admire this work and want to reproduce it faithfully!

[wzmsltw]

Hi, you can modify the task config to enable depth simulation. For example, if you eval model with task_config/demo_clean.yml, you should modify data_type/depth from false to true.

[Chloe-Yan-3465]

Okay! Thanks a lot for it !

This one ,right ?

data_type:
  rgb: true
  third_view: false
  depth: true # sem

Sorry that is indeed a silly mistake ( apologies for the noise >< ). I did not even realize to modifytask_config=demo_clean in sem_policy/eval.sh . What I actually used to collect data is sem_place_empty_cup.yml .

Really sorry for the bothering, and I sincerely appreciate your patience and support!

I'll go ahead and close this issue.

[bbbbbbbbly]

Thanks, i meet the load chepoint issues and the robot random action.

bug is

Building transforms...
Building model...
Loading checkpoint from: /home/kemove/data/test_project/RoboOrchardLab/projects/sem/robotwin/RoboTwin/policy/sem_policy/checkpoints/place_empty_cup/model.safetensors
Traceback (most recent call last):
  File "/home/kemove/data/test_project/RoboOrchardLab/projects/sem/robotwin/RoboTwin/script/eval_policy.py", line 362, in <module>
    main(usr_args)
  File "/home/kemove/data/test_project/RoboOrchardLab/projects/sem/robotwin/RoboTwin/script/eval_policy.py", line 165, in main
    model = get_model(usr_args)
  File "/home/kemove/data/test_project/RoboOrchardLab/projects/sem/robotwin/RoboTwin/./policy/sem_policy/deploy_policy.py", line 194, in get_model
    policy = SEMPolicy(usr_args["ckpt_path"], usr_args["config_file"])
  File "/home/kemove/data/test_project/RoboOrchardLab/projects/sem/robotwin/RoboTwin/./policy/sem_policy/deploy_policy.py", line 58, in __init__
    self._load_checkpoint()
  File "/home/kemove/data/test_project/RoboOrchardLab/projects/sem/robotwin/RoboTwin/./policy/sem_policy/deploy_policy.py", line 102, in _load_checkpoint
    load_safetensors(
  File "/home/kemove/miniconda3/envs/sem/lib/python3.10/site-packages/safetensors/torch.py", line 221, in load_model
    raise RuntimeError(error)
RuntimeError: Error(s) in loading state_dict for BIP3D:
    Missing key(s) in state_dict: "decoder.head.convs.0.bias", "decoder.head.convs.0.weight", "decoder.head.convs.1.bias", "decoder.head.convs.1.weight", "decoder.head.convs.2.bias", "decoder.head.convs.2.weight"
    Unexpected key(s) in state_dict: "decoder.head.upsamples.0.1.bias", "decoder.head.upsamples.0.1.weight", "decoder.head.upsamples.1.1.bias", "decoder.head.upsamples.1.1.weight", "decoder.head.upsamples.2.1.bias", "decoder.head.upsamples.2.1.weight"

and i change code to

    def _load_checkpoint(self):
        from safetensors.torch import load_file
        print(f"Loading checkpoint from: {self.ckpt_path}")
        state_dict = load_file(self.ckpt_path, device=str(self.device))

        # Remap keys if necessary
        remapped_state_dict = {}
        for k, v in state_dict.items():
            new_k = k.replace("upsamples", "convs") if "upsamples" in k else k
            remapped_state_dict[new_k] = v

        missing, unexpected = self.model.load_state_dict(remapped_state_dict, strict=False)
        if missing:
            print(f"[WARN] Missing keys: {missing}")
        if unexpected:
            print(f"[WARN] Unexpected keys: {unexpected}")

I have no idea how to change the network or checpoint from https://huggingface.co/HorizonRobotics/SEM-RoboTwin-Tiny/tree/main

episode0.mp4