[AUTOCUT] Gradle Check Flaky Test Report for S3BlobContainerRetriesTests

[opensearch-ci-bot]

Flaky Test Report for S3BlobContainerRetriesTests

Noticed the S3BlobContainerRetriesTests has some flaky, failing tests that failed during post-merge actions and timer trigerred run on main branch.

Details

Git Reference	Merged Pull Request	Build Details	Test Name
4b091fc	18800	62789	org.opensearch.repositories.s3.S3BlobContainerRetriesTests.testReadRangeBlobWithRetries
2b62528	18308	58144	org.opensearch.repositories.s3.S3BlobContainerRetriesTests.classMethod
60aa7a5	18212	57844	org.opensearch.repositories.s3.S3BlobContainerRetriesTests.classMethod
47f9bdb	18165	57479	org.opensearch.repositories.s3.S3BlobContainerRetriesTests.classMethod
main	null	54463 57212 57451	org.opensearch.repositories.s3.S3BlobContainerRetriesTests.classMethod
6719b1f	18057	57116	org.opensearch.repositories.s3.S3BlobContainerRetriesTests.classMethod
1294506	18061	57060	org.opensearch.repositories.s3.S3BlobContainerRetriesTests.classMethod
816787d	18044	57010	org.opensearch.repositories.s3.S3BlobContainerRetriesTests.classMethod
8964f63	17873	56048	org.opensearch.repositories.s3.S3BlobContainerRetriesTests.classMethod
fc78200	17732	55229	org.opensearch.repositories.s3.S3BlobContainerRetriesTests.classMethod
e0a67fd	17674	55181	org.opensearch.repositories.s3.S3BlobContainerRetriesTests.classMethod
fcb0649	17668	54996	org.opensearch.repositories.s3.S3BlobContainerRetriesTests.classMethod
cc82be9	17515	54208	org.opensearch.repositories.s3.S3BlobContainerRetriesTests.classMethod

The other pull requests, besides those involved in post-merge actions, that contain failing tests with the S3BlobContainerRetriesTests class are:

• 17950
• 17718
• 17768
• 17490
• 17513
• 17711
• 17730
• 17769
• 17771
• 17792
• 17817
• 18093
• 18119
• 17753
• 17901
• 16731
• 17273
• 17299
• 17336
• 17344
• 17365
• 17447
• 17457
• 17521
• 17535
• 17559
• 17575
• 17607
• 17611
• 17612
• 17622
• 17631
• 17663
• 17670
• 17699
• 17712
• 17756
• 17774
• 17791
• 17796
• 17843
• 17849
• 17855
• 17868
• 17893
• 17894
• 17910
• 17912
• 17917
• 17922
• 17961
• 17978
• 17987
• 17989
• 18006
• 18043
• 18047
• 18049
• 18054
• 18067
• 18075
• 18085
• 18088
• 18097
• 18103
• 18134
• 18145
• 18148
• 18161
• 18163
• 18166
• 18218
• 18243
• 18247
• 18254
• 18263
• 18739
• 18779
• 19220
• 19383
• 12982
• 13568
• 14512
• 15181
• 16178
• 16670
• 17926
• 17927
• 17942
• 17953
• 18966
• 9151

For more details on the failed tests refer to OpenSearch Gradle Check Metrics dashboard.

[krisfreedain]

Catch All Triage - 1 2

[andrross]

Reopening as I saw the failure again even with this fix:

REPRODUCE WITH: ./gradlew ':plugins:repository-s3:test' --tests "org.opensearch.repositories.s3.S3BlobContainerRetriesTests" -Dtests.seed=7C03C4361DD3B7D5 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=en -Dtests.timezone=Etc/UTC -Druntime.java=21

S3BlobContainerRetriesTests > classMethod FAILED
    com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
       1) Thread[id=185, name=sdk-ScheduledExecutor-12-0, state=TIMED_WAITING, group=TGRP-S3BlobContainerRetriesTests]
            at java.****/jdk.internal.misc.Unsafe.park(Native Method)
            at java.****/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:269)
            at java.****/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1763)
            at java.****/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
            at java.****/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
            at java.****/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070)
            at java.****/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
            at java.****/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
            at java.****/java.lang.Thread.run(Thread.java:1583)
        at __randomizedtesting.SeedInfo.seed([7C03C4361DD3B7D5]:0)

https://build.ci.opensearch.org/job/gradle-check/57400/consoleText

[andrross]

  2> Apr 29, 2025 6:38:16 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
  2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
  2> Apr 29, 2025 6:38:21 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
  2> SEVERE: 1 thread leaked from SUITE scope at org.opensearch.repositories.s3.S3BlobContainerRetriesTests: 
  2>    1) Thread[id=185, name=sdk-ScheduledExecutor-12-0, state=TIMED_WAITING, group=TGRP-S3BlobContainerRetriesTests]
...
  2> Apr 29, 2025 6:38:21 PM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
  2> INFO: Starting to interrupt leaked threads:
  2>    1) Thread[id=185, name=sdk-ScheduledExecutor-12-0, state=TIMED_WAITING, group=TGRP-S3BlobContainerRetriesTests]
  2> Apr 29, 2025 6:38:24 PM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
  2> SEVERE: There are still zombie threads that couldn't be terminated:
  2>    1) Thread[id=185, name=sdk-ScheduledExecutor-12-0, state=TIMED_WAITING, group=TGRP-S3BlobContainerRetriesTests]
...
  2> com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.opensearch.repositories.s3.S3BlobContainerRetriesTests: 
       1) Thread[id=185, name=sdk-ScheduledExecutor-12-0, state=TIMED_WAITING, group=TGRP-S3BlobContainerRetriesTests]
...

The name of the leaked thread starts with sdk-ScheduledExecutor meaning it comes from the scheduled executor service in the client. The test finds the leaked thread, waits 5 seconds for it to stop, interrupts it, waits 3 more seconds, then fails the test. The fact the thread doesn't die with the interrupt means (I think) that the scheduled executor service is not being shut down. I can't get a reliable repro, and my reading of the code shows that this executor should be shut down when the client closes. I'm not sure what's going wrong.