Skip to content

[Tests] Improve error message on some test-failures because of IndexError in test_utils.py #54343

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 6 commits into
base: master
Choose a base branch
from

Conversation

Daraan
Copy link
Contributor

@Daraan Daraan commented Jul 4, 2025

Why are these changes needed?

In the CI I saw a fleaky test failing, but its error is not reported correctly:

[2025-07-04T00:01:57Z] Traceback (most recent call last):
[2025-07-04T00:01:57Z]   File "/root/.cache/bazel/_bazel_root/1df605deb6d24fc8068f6e25793ec703/execroot/com_github_ray_project_ray/bazel-out/k8-opt/bin/rllib/examples/connectors/multi_agent_observation_preprocessor.runfiles/com_github_ray_project_ray/rllib/examples/connectors/multi_agent_observation_preprocessor.py", line 137, in <module>
[2025-07-04T00:01:57Z]     run_rllib_example_script_experiment(base_config, args)
[2025-07-04T00:01:57Z]   File "/rayci/python/ray/rllib/utils/test_utils.py", line 1354, in run_rllib_example_script_experiment
[2025-07-04T00:01:57Z]     f"{[e.args[0].args[2] for e in results.errors]}"
[2025-07-04T00:01:57Z]   File "/rayci/python/ray/rllib/utils/test_utils.py", line 1354, in <listcomp>
[2025-07-04T00:01:57Z]     f"{[e.args[0].args[2] for e in results.errors]}"  # <---
[2025-07-04T00:01:57Z] IndexError: tuple index out of range  # <---

In the logs the actual error does not appear because of the IndexError for f"{[e.args[0].args[2] for e in results.errors]}". I am not sure why e.args[0].args[2] is used, a nested Exception perhaps? This PR adds a guard to fallback to a simpler repr(e) in case the IndexError would accur again. This allows to investigate why this IndexError occures and raises the RuntimeError like expected.

Related issue number

NA

Checks

  • I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
  • I've run scripts/format.sh to lint the changes in this PR.
  • I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
  • Testing Strategy
    • This PR is not tested, but improves other tests :)

@Copilot Copilot AI review requested due to automatic review settings July 4, 2025 15:02
@Daraan Daraan requested a review from a team as a code owner July 4, 2025 15:02
Copy link
Contributor

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR improves error reporting in run_rllib_example_script_experiment by guarding against an IndexError when extracting nested exception messages and falling back to repr(e).

  • Adds a safe extraction of e.args[0].args[2] with a fallback to repr(e) if the tuple is too short.
  • Fixes a spelling typo in a comment (criteriscriteria).
  • Updates the raised RuntimeError to include the new errors list.
Comments suppressed due to low confidence (1)

rllib/utils/test_utils.py:1352

  • Add a unit test for this fallback branch to ensure that when nested exception args are too short, the code uses the fallback behavior as intended.
        # Might cause a Index error if Tuple is not long enough in that case use str(e)

if results.errors:
# Might cause a Index error if Tuple is not long enough in that case use str(e)
errors = [
e.args[0].args[2] if e.args and len(e.args[0]) > 2 else repr(e)
Copy link
Preview

Copilot AI Jul 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nitpick] Consider using str(e) instead of repr(e) for a clearer, human-readable error message.

Suggested change
e.args[0].args[2] if e.args and len(e.args[0]) > 2 else repr(e)
e.args[0].args[2] if e.args and len(e.args[0]) > 2 else str(e)

Copilot uses AI. Check for mistakes.

@Daraan Daraan changed the title [Tests] Improve error message on some test-failures [Tests] Improve error message on some test-failures because of IndexError in test_utils.py Jul 4, 2025
Daraan and others added 3 commits July 4, 2025 17:22
Co-authored-by: Copilot <[email protected]>
Signed-off-by: Daraan <[email protected]>
Co-authored-by: Copilot <[email protected]>
Signed-off-by: Daraan <[email protected]>
Signed-off-by: Daraan <[email protected]>
@cszhu cszhu added community-contribution Contributed by the community rllib RLlib related issues ci-test labels Jul 9, 2025
Copy link

This pull request has been automatically marked as stale because it has not had
any activity for 14 days. It will be closed in another 14 days if no further activity occurs.
Thank you for your contributions.

You can always ask for help on our discussion forum or Ray's public slack channel.

If you'd like to keep this open, just leave any comment, and the stale label will be removed.

@github-actions github-actions bot added the stale The issue is stale. It will be closed within 7 days unless there are further conversation label Jul 24, 2025
@Daraan
Copy link
Contributor Author

Daraan commented Jul 24, 2025

(ง •_•)ง stale

@github-actions github-actions bot added unstale A PR that has been marked unstale. It will not get marked stale again if this label is on it. and removed stale The issue is stale. It will be closed within 7 days unless there are further conversation labels Jul 24, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ci-test community-contribution Contributed by the community rllib RLlib related issues unstale A PR that has been marked unstale. It will not get marked stale again if this label is on it.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants