Ray Clusters - updating cluster with new docker image should actually use new image

[uazafar]

Description

Hi,

I'm submitting this enhancement after discussion with Jiajun Yao on Slack.

I notice that when I update the docker image in my cluster configuration and run ray up..., the cluster continues to run the older image. I see a message in the logs that states:

"A container with name X is running image A instead of B (which was provided in the YAML)"

I see from the source code there is a function _check_if_container_restart_is_needed, which has the section:

        if running_image != image:
            cli_logger.error(
                "A container with name {} is running image {} instead "
                + "of {} (which was provided in the YAML)",
                self.container_name,
                running_image,
                image,
            )

It only sets re_init_required to True when differences in the mounts are detected. I'm wondering why this is not set to True when a new image is detected? It seems like specifying a new image should allow a cluster to use that new image, thereby requiring a container restart. Curious as to why this is the case and if there is a workaround that does not involve running ray down... and ray up....

Please let me know if you require more information.

Thanks
Usman

Use case

When updating my Ray cluster with a new docker image and running ray up..., I would expect the updated cluster to use that new image. At the moment it is being ignored and I don't consider this intuitive behaviour. This feature would make updating clusters simpler and possibly even allow such updated to be done without interrupting running jobs, which is very valuable.

[jjyao]

@uazafar contribution is welcome since you already dived into the code :)

[uazafar]

Thanks for the response @jjyao . I'll give it a go over the next week.

[cbelsole]

@jjyao @uazafar @hainesmichaelc

Is this the fix?

        if running_image != image:
            re_init_required = True # The fix?
            cli_logger.error(
                "A container with name {} is running image {} instead "
                + "of {} (which was provided in the YAML)",
                self.container_name,
                running_image,
                image,
            )

I tried to get ray running locally, but the setup never succeeded successfully.