Fix ListObjectsV2 pagination to support listing more than 1000 objects

[Copilot]

Problem

The AwsObjectStorage.doList() method only returned the first 1000 objects from S3's ListObjectsV2 API, which has a maximum response limit of 1000 objects per request. This caused issues when trying to list buckets with more than 1000 objects, as subsequent objects were silently ignored.

According to the AWS ListObjectsV2 documentation:

Returns some or all (up to 1,000) of the objects in a bucket with each request.

The implementation should support listing all objects with a given prefix, regardless of the total count.

Solution

Implemented proper pagination support by:

1. Added recursive pagination logic using listObjectsWithPagination() helper method
2. Checks response truncation using resp.isTruncated() to determine if more pages exist
3. Uses continuation tokens (resp.nextContinuationToken()) to fetch subsequent pages
4. Accumulates results from all pages into a single list
5. Maintains async behavior using CompletableFuture composition

Before:

@Override
CompletableFuture<List<ObjectInfo>> doList(String prefix) {
    return readS3Client.listObjectsV2(builder -> builder.bucket(bucket).prefix(prefix))
        .thenApply(resp -> /* process single page only */);
}

After:

@Override
CompletableFuture<List<ObjectInfo>> doList(String prefix) {
    return listObjectsWithPagination(prefix, null, new ArrayList<>());
}

private CompletableFuture<List<ObjectInfo>> listObjectsWithPagination(String prefix, String continuationToken, List<ObjectInfo> accumulator) {
    // Handles pagination recursively until all objects are retrieved
}

Testing

Added comprehensive test coverage:

• ✅ Pagination scenario: Multiple pages with continuation tokens
• ✅ Single page scenario: All objects fit in one response
• ✅ Empty results: No objects match the prefix
• ✅ Existing functionality: All existing tests continue to pass

Impact

This fix ensures that operations like log cleanup in LogUploader and other components that rely on listing objects will now work correctly with buckets containing more than 1000 objects.

Closes #2650

---

💬 Share your feedback on Copilot coding agent for the chance to win a $200 gift card! Click here to start the survey.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}