[flaky] test_scheduling_2.py::test_demand_report_when_scale_up

[kevin85421]

What happened + What you expected to happen

The test is flaky, and the test is also super heavyweight. we should make it spawn fewer processes and run fewer tasks.

https://buildkite.com/ray-project/premerge/builds/41986#019769d1-4b12-43e5-b37b-81b1e27f56e8/6-158

Versions / Dependencies

TODO

Reproduction script

https://buildkite.com/ray-project/premerge/builds/41986#019769d1-4b12-43e5-b37b-81b1e27f56e8/6-158

Issue Severity

None

[kevin85421]

cc @edoakes

[kevin85421]

@rueian would you mind taking a look when you have time?

[rueian]

I have been trying to reproduce this for a couple of days. I did find the test case to be flaky on MacOS due to raylet failures caused by a GpuProfilingManager issue, which was fixed by #53807. After applying the fix, I haven't been able to reproduce the issue on my MacBook so far.

However, that was not the root cause in https://buildkite.com/ray-project/premerge/builds/41986#019769d1-4b12-43e5-b37b-81b1e27f56e8/6-158, because there was no raylet failure in its cluster logs.

I also looked into every log in the cluster logs, but everything looked good. I wonder if the flakiness could simply be due to the timeout for the current test workload is too short occasionally.

I think I am going to open a PR trying to reduce the tasks spawned by the test to see if the flakiness can be improved on CI runners. And we also need more information for debugging when it times out. I suggest we log all the current tasks when the timeout happens:

from ray.util.state import list_tasks
...
try:
    wait_for_condition(check_backlog_info, 660)
except Exception:
    tasks = list_tasks(limit=10000)
    logger.info(f"Total tasks: {len(tasks)}")
    for task in tasks:
        logger.info(task)
    raise