[Ray Core] Progress bar showing with ray init local but not with an actual cluster

[sbucaille]

What happened + What you expected to happen

Hi,

I was trying to use ray_tqdm to have progress bars not to fill my logs with pyramids, but after switching from local mode to an actual cluster, the progress bar simply don't show up. I guess this is something about stdout or stderr (I'm not an expert on that) which in the case of local mode, I see everything, but in cluster mode, some of these are not returned by the cluster ?
What parameter or option am I missing ?
Thanks in advance for the help

Versions / Dependencies

2.35.0

Reproduction script

import logging
import random
from time import sleep

import ray
from ray.experimental import tqdm_ray

@ray.remote
def worker(n, desc):
    progress_bar = tqdm_ray.tqdm(desc=desc, total=n, position=0)
    for _ in range(n):
        sleep(random.random())
        progress_bar.update(1)

if __name__ == "__main__":
    # ray.init(address="ray://192.168.20.201:10001", runtime_env={"pip": ["tqdm"]}, logging_level=logging.ERROR)
    ray.init(address="local")

    n = 5
    m = 50
    result = ray.get([worker.remote(m, f"Bar {i}") for i in range(n)])

Issue Severity

Low: It annoys or frustrates me.

[jjyao]

cc @rynewang