[BUG] controller node - failed to mark destroy compacted objects

[Aaron0930]

Version & Environment

• AutoMQ version : v1.2.1
• Operating System : Amazon Linux 2023

What went wrong?

some nodes of automq cluster faced issue due to run out of bandwidth. After that, one of controller and brokers keep reporting errors event the cluster is healthy now

Additional information

Please attach any relevant logs, backtraces, or metric charts.
broker error:
[2025-04-07 11:34:12,883] ERROR [STREAM_OBJECT_COMPACT_FAIL],[UNEXPECTED],5035,type=MINOR_V1,req=CompactStreamObjectRequest{objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, streamEpoch=14, attributes=0, sourceObjectIds=[6495919, 154573100], operations=[DEEP_DELETE, DEEP_DELETE]} (com.automq.stream.s3.compact.StreamObjectCompactor)

controller error:
[2025-04-07 11:34:12,883] ERROR [QuorumController id=0] [CommitStreamObject]: failed to mark destroy compacted objects. compactedObjects=[6495919, 154573100], req=CommitStreamObjectRequestData(nodeId=2, nodeEpoch=1743911169270, objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, sourceObjectIds=[6495919, 154573100], streamEpoch=14, attributes=0, operations=[2, 2]) (org.apache.kafka.controller.stream.StreamControlManager),

[Aaron0930]

try to restart impacted broker and controller nodes, it does not help. #765 look like similar issue, but the previous solution was for automq-for-rocketmq

[superhx]

Is there any other error log in controller.log?

[Aaron0930]

here is controller log
[2025-04-07 11:34:12,883] ERROR [QuorumController id=0] [CommitStreamObject]: failed to mark destroy compacted objects. compactedObjects=[6495919, 154573100], req=CommitStreamObjectRequestData(nodeId=2, nodeEpoch=1743911169270, objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, sourceObjectIds=[6495919, 154573100], streamEpoch=14, attributes=0, operations=[2, 2]) (org.apache.kafka.controller.stream.StreamControlManager)

[Aaron0930]

Other error controller logs:
[2025-04-07 12:08:59,961] ERROR [QuorumController id=0] Controller took 20164 µs to process write event: commitStreamObject (org.apache.kafka.controller.QuorumController)

[2025-04-07 12:20:10,312] ERROR [QuorumController id=0] object 23535896 not exist when mark destroy object (org.apache.kafka.controller.stream.S3ObjectControlManager)

[superhx]

Could you get more logs about object 6495919, 154573100

[jerome-j-20230331]

@superhx I am Aaron's colleague, here are some recent error logs.

message
[2025-04-15 12:48:08,792] ERROR [QuorumController id=0] object 6495919 not exist when mark destroy object (org.apache.kafka.controller.stream.S3ObjectControlManager)
[2025-04-15 12:47:08,260] ERROR [STREAM_OBJECT_COMPACT_FAIL],[UNEXPECTED],5035,type=CLEANUP_V1,req=CompactStreamObjectRequest{objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, streamEpoch=14, attributes=0, sourceObjectIds=[6495919, 154573100], operations=[DEEP_DELETE, DEEP_DELETE]} (com.automq.stream.s3.compact.StreamObjectCompactor)
java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found.
at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073)
at com.automq.stream.s3.compact.StreamObjectCompactor.cleanupExpiredObject(StreamObjectCompactor.java:250)
at com.automq.stream.s3.compact.StreamObjectCompactor.compact0(StreamObjectCompactor.java:168)
at com.automq.stream.s3.compact.StreamObjectCompactor.compact(StreamObjectCompactor.java:116)
at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:418)
at com.automq.stream.s3.S3StreamClient$StreamWrapper.compactV1(S3StreamClient.java:402)
at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:375)
at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$6(S3StreamClient.java:155)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$7(S3StreamClient.java:155)
at com.automq.stream.utils.ThreadUtils.lambda$wrapRunnable$0(ThreadUtils.java:74)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found.
[2025-04-15 12:47:08,260] ERROR [QuorumController id=0] [CommitStreamObject]: failed to mark destroy compacted objects. compactedObjects=[6495919, 154573100], req=CommitStreamObjectRequestData(nodeId=2, nodeEpoch=1743911169270, objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, sourceObjectIds=[6495919, 154573100], streamEpoch=14, attributes=0, operations=[2, 2]) (org.apache.kafka.controller.stream.StreamControlManager)
[2025-04-15 12:47:08,260] ERROR [QuorumController id=0] object 6495919 not exist when mark destroy object (org.apache.kafka.controller.stream.S3ObjectControlManager)
[2025-04-15 12:46:08,203] ERROR [STREAM_OBJECT_COMPACT_FAIL],[UNEXPECTED],5035,type=CLEANUP_V1,req=CompactStreamObjectRequest{objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, streamEpoch=14, attributes=0, sourceObjectIds=[6495919, 154573100], operations=[DEEP_DELETE, DEEP_DELETE]} (com.automq.stream.s3.compact.StreamObjectCompactor)
java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found.
at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073)
at com.automq.stream.s3.compact.StreamObjectCompactor.cleanupExpiredObject(StreamObjectCompactor.java:250)
at com.automq.stream.s3.compact.StreamObjectCompactor.compact0(StreamObjectCompactor.java:168)
at com.automq.stream.s3.compact.StreamObjectCompactor.compact(StreamObjectCompactor.java:116)
at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:418)
at com.automq.stream.s3.S3StreamClient$StreamWrapper.compactV1(S3StreamClient.java:402)
at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:375)
at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$6(S3StreamClient.java:155)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$7(S3StreamClient.java:155)
at com.automq.stream.utils.ThreadUtils.lambda$wrapRunnable$0(ThreadUtils.java:74)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found.
[2025-04-15 12:46:08,202] ERROR [QuorumController id=0] [CommitStreamObject]: failed to mark destroy compacted objects. compactedObjects=[6495919, 154573100], req=CommitStreamObjectRequestData(nodeId=2, nodeEpoch=1743911169270, objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, sourceObjectIds=[6495919, 154573100], streamEpoch=14, attributes=0, operations=[2, 2]) (org.apache.kafka.controller.stream.StreamControlManager)
[2025-04-15 12:46:08,202] ERROR [QuorumController id=0] object 6495919 not exist when mark destroy object (org.apache.kafka.controller.stream.S3ObjectControlManager)
[2025-04-15 12:45:08,148] ERROR [STREAM_OBJECT_COMPACT_FAIL],[UNEXPECTED],5035,type=CLEANUP_V1,req=CompactStreamObjectRequest{objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, streamEpoch=14, attributes=0, sourceObjectIds=[6495919, 154573100], operations=[DEEP_DELETE, DEEP_DELETE]} (com.automq.stream.s3.compact.StreamObjectCompactor)
java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found.
at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073)
at com.automq.stream.s3.compact.StreamObjectCompactor.cleanupExpiredObject(StreamObjectCompactor.java:250)
at com.automq.stream.s3.compact.StreamObjectCompactor.compact0(StreamObjectCompactor.java:168)
at com.automq.stream.s3.compact.StreamObjectCompactor.compact(StreamObjectCompactor.java:116)
at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:418)
at com.automq.stream.s3.S3StreamClient$StreamWrapper.compactV1(S3StreamClient.java:402)
at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:375)
at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$6(S3StreamClient.java:155)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$7(S3StreamClient.java:155)
at com.automq.stream.utils.ThreadUtils.lambda$wrapRunnable$0(ThreadUtils.java:74)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found.
[2025-04-15 12:45:08,147] ERROR [QuorumController id=0] [CommitStreamObject]: failed to mark destroy compacted objects. compactedObjects=[6495919, 154573100], req=CommitStreamObjectRequestData(nodeId=2, nodeEpoch=1743911169270, objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, sourceObjectIds=[6495919, 154573100], streamEpoch=14, attributes=0, operations=[2, 2]) (org.apache.kafka.controller.stream.StreamControlManager)
[2025-04-15 12:45:08,147] ERROR [QuorumController id=0] object 6495919 not exist when mark destroy object (org.apache.kafka.controller.stream.S3ObjectControlManager)
[2025-04-15 12:44:08,091] ERROR [STREAM_OBJECT_COMPACT_FAIL],[UNEXPECTED],5035,type=CLEANUP_V1,req=CompactStreamObjectRequest{objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, streamEpoch=14, attributes=0, sourceObjectIds=[6495919, 154573100], operations=[DEEP_DELETE, DEEP_DELETE]} (com.automq.stream.s3.compact.StreamObjectCompactor)
java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found.
at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073)
at com.automq.stream.s3.compact.StreamObjectCompactor.cleanupExpiredObject(StreamObjectCompactor.java:250)
at com.automq.stream.s3.compact.StreamObjectCompactor.compact0(StreamObjectCompactor.java:168)
at com.automq.stream.s3.compact.StreamObjectCompactor.compact(StreamObjectCompactor.java:116)
at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:418)
at com.automq.stream.s3.S3StreamClient$StreamWrapper.compactV1(S3StreamClient.java:402)
at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:375)
at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$6(S3StreamClient.java:155)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$7(S3StreamClient.java:155)
at com.automq.stream.utils.ThreadUtils.lambda$wrapRunnable$0(ThreadUtils.java:74)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found.
[2025-04-15 12:44:08,091] ERROR [QuorumController id=0] [CommitStreamObject]: failed to mark destroy compacted objects. compactedObjects=[6495919, 154573100], req=CommitStreamObjectRequestData(nodeId=2, nodeEpoch=1743911169270, objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, sourceObjectIds=[6495919, 154573100], streamEpoch=14, attributes=0, operations=[2, 2]) (org.apache.kafka.controller.stream.StreamControlManager)
[2025-04-15 12:44:08,091] ERROR [QuorumController id=0] object 6495919 not exist when mark destroy object (org.apache.kafka.controller.stream.S3ObjectControlManager)
[2025-04-15 12:43:08,031] ERROR [STREAM_OBJECT_COMPACT_FAIL],[UNEXPECTED],5035,type=CLEANUP_V1,req=CompactStreamObjectRequest{objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, streamEpoch=14, attributes=0, sourceObjectIds=[6495919, 154573100], operations=[DEEP_DELETE, DEEP_DELETE]} (com.automq.stream.s3.compact.StreamObjectCompactor)
java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found.
at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073)
at com.automq.stream.s3.compact.StreamObjectCompactor.cleanupExpiredObject(StreamObjectCompactor.java:250)
at com.automq.stream.s3.compact.StreamObjectCompactor.compact0(StreamObjectCompactor.java:168)
at com.automq.stream.s3.compact.StreamObjectCompactor.compact(StreamObjectCompactor.java:116)
at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:418)
at com.automq.stream.s3.S3StreamClient$StreamWrapper.compactV1(S3StreamClient.java:402)
at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:375)
at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$6(S3StreamClient.java:155)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$7(S3StreamClient.java:155)
at com.automq.stream.utils.ThreadUtils.lambda$wrapRunnable$0(ThreadUtils.java:74)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found.
[2025-04-15 12:43:08,031] ERROR [QuorumController id=0] [CommitStreamObject]: failed to mark destroy compacted objects. compactedObjects=[6495919, 154573100], req=CommitStreamObjectRequestData(nodeId=2, nodeEpoch=1743911169270, objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, sourceObjectIds=[6495919, 154573100], streamEpoch=14, attributes=0, operations=[2, 2]) (org.apache.kafka.controller.stream.StreamControlManager)
[2025-04-15 12:43:08,031] ERROR [QuorumController id=0] object 6495919 not exist when mark destroy object (org.apache.kafka.controller.stream.S3ObjectControlManager)
[2025-04-15 12:42:07,443] ERROR [STREAM_OBJECT_COMPACT_FAIL],[UNEXPECTED],5035,type=CLEANUP_V1,req=CompactStreamObjectRequest{objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, streamEpoch=14, attributes=0, sourceObjectIds=[6495919, 154573100], operations=[DEEP_DELETE, DEEP_DELETE]} (com.automq.stream.s3.compact.StreamObjectCompactor)
java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found.
at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073)
at com.automq.stream.s3.compact.StreamObjectCompactor.cleanupExpiredObject(StreamObjectCompactor.java:250)
at com.automq.stream.s3.compact.StreamObjectCompactor.compact0(StreamObjectCompactor.java:168)
at com.automq.stream.s3.compact.StreamObjectCompactor.compact(StreamObjectCompactor.java:116)
at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:418)
at com.automq.stream.s3.S3StreamClient$StreamWrapper.compactV1(S3StreamClient.java:402)
at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:375)
at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$6(S3StreamClient.java:155)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$7(S3StreamClient.java:155)
at com.automq.stream.utils.ThreadUtils.lambda$wrapRunnable$0(ThreadUtils.java:74)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found.
[2025-04-15 12:42:07,434] ERROR [QuorumController id=0] [CommitStreamObject]: failed to mark destroy compacted objects. compactedObjects=[6495919, 154573100], req=CommitStreamObjectRequestData(nodeId=2, nodeEpoch=1743911169270, objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, sourceObjectIds=[6495919, 154573100], streamEpoch=14, attributes=0, operations=[2, 2]) (org.apache.kafka.controller.stream.StreamControlManager)
[2025-04-15 12:42:07,434] ERROR [QuorumController id=0] object 6495919 not exist when mark destroy object (org.apache.kafka.controller.stream.S3ObjectControlManager)
[2025-04-15 12:41:07,372] ERROR [STREAM_OBJECT_COMPACT_FAIL],[UNEXPECTED],5035,type=CLEANUP_V1,req=CompactStreamObjectRequest{objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, streamEpoch=14, attributes=0, sourceObjectIds=[6495919, 154573100], operations=[DEEP_DELETE, DEEP_DELETE]} (com.automq.stream.s3.compact.StreamObjectCompactor)
java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found.
at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073)
at com.automq.stream.s3.compact.StreamObjectCompactor.cleanupExpiredObject(StreamObjectCompactor.java:250)
at com.automq.stream.s3.compact.StreamObjectCompactor.compact0(StreamObjectCompactor.java:168)
at com.automq.stream.s3.compact.StreamObjectCompactor.compact(StreamObjectCompactor.java:116)
at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:418)
at com.automq.stream.s3.S3StreamClient$StreamWrapper.compactV1(S3StreamClient.java:402)
at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:375)
at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$6(S3StreamClient.java:155)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$7(S3StreamClient.java:155)
at com.automq.stream.utils.ThreadUtils.lambda$wrapRunnable$0(ThreadUtils.java:74)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found.
[2025-04-15 12:41:07,371] ERROR [QuorumController id=0] [CommitStreamObject]: failed to mark destroy compacted objects. compactedObjects=[6495919, 154573100], req=CommitStreamObjectRequestData(nodeId=2, nodeEpoch=1743911169270, objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, sourceObjectIds=[6495919, 154573100], streamEpoch=14, attributes=0, operations=[2, 2]) (org.apache.kafka.controller.stream.StreamControlManager)
[2025-04-15 12:41:07,371] ERROR [QuorumController id=0] object 6495919 not exist when mark destroy object (org.apache.kafka.controller.stream.S3ObjectControlManager)
[2025-04-15 12:40:07,305] ERROR [STREAM_OBJECT_COMPACT_FAIL],[UNEXPECTED],5035,type=CLEANUP_V1,req=CompactStreamObjectRequest{objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, streamEpoch=14, attributes=0, sourceObjectIds=[6495919, 154573100], operations=[DEEP_DELETE, DEEP_DELETE]} (com.automq.stream.s3.compact.StreamObjectCompactor)
java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found.
at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073)
at com.automq.stream.s3.compact.StreamObjectCompactor.cleanupExpiredObject(StreamObjectCompactor.java:250)
at com.automq.stream.s3.compact.StreamObjectCompactor.compact0(StreamObjectCompactor.java:168)
at com.automq.stream.s3.compact.StreamObjectCompactor.compact(StreamObjectCompactor.java:116)
at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:418)
at com.automq.stream.s3.S3StreamClient$StreamWrapper.compactV1(S3StreamClient.java:402)
at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:375)
at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$6(S3StreamClient.java:155)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$7(S3StreamClient.java:155)
at com.automq.stream.utils.ThreadUtils.lambda$wrapRunnable$0(ThreadUtils.java:74)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found.
[2025-04-15 12:40:07,304] ERROR [QuorumController id=0] [CommitStreamObject]: failed to mark destroy compacted objects. compactedObjects=[6495919, 154573100], req=CommitStreamObjectRequestData(nodeId=2, nodeEpoch=1743911169270, objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, sourceObjectIds=[6495919, 154573100], streamEpoch=14, attributes=0, operations=[2, 2]) (org.apache.kafka.controller.stream.StreamControlManager)
[2025-04-15 12:40:07,304] ERROR [QuorumController id=0] object 6495919 not exist when mark destroy object (org.apache.kafka.controller.stream.S3ObjectControlManager)
[2025-04-15 12:39:07,250] ERROR [STREAM_OBJECT_COMPACT_FAIL],[UNEXPECTED],5035,type=MINOR_V1,req=CompactStreamObjectRequest{objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, streamEpoch=14, attributes=0, sourceObjectIds=[6495919, 154573100], operations=[DEEP_DELETE, DEEP_DELETE]} (com.automq.stream.s3.compact.StreamObjectCompactor)
java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found.
at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073)
at com.automq.stream.s3.compact.StreamObjectCompactor.cleanupExpiredObject(StreamObjectCompactor.java:250)
at com.automq.stream.s3.compact.StreamObjectCompactor.compact0(StreamObjectCompactor.java:168)
at com.automq.stream.s3.compact.StreamObjectCompactor.compact(StreamObjectCompactor.java:116)
at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:418)
at com.automq.stream.s3.S3StreamClient$StreamWrapper.compactV1(S3StreamClient.java:399)
at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:375)
at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$6(S3StreamClient.java:155)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$7(S3StreamClient.java:155)
at com.automq.stream.utils.ThreadUtils.lambda$wrapRunnable$0(ThreadUtils.java:74)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found.
[2025-04-15 12:39:07,249] ERROR [QuorumController id=0] [CommitStreamObject]: failed to mark destroy compacted objects. compactedObjects=[6495919, 154573100], req=CommitStreamObjectRequestData(nodeId=2, nodeEpoch=1743911169270, objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, sourceObjectIds=[6495919, 154573100], streamEpoch=14, attributes=0, operations=[2, 2]) (org.apache.kafka.controller.stream.StreamControlManager)
[2025-04-15 12:39:07,249] ERROR [QuorumController id=0] object 6495919 not exist when mark destroy object (org.apache.kafka.controller.stream.S3ObjectControlManager)
[2025-04-15 12:38:07,182] ERROR [STREAM_OBJECT_COMPACT_FAIL],[UNEXPECTED],5035,type=CLEANUP_V1,req=CompactStreamObjectRequest{objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, streamEpoch=14, attributes=0, sourceObjectIds=[6495919, 154573100], operations=[DEEP_DELETE, DEEP_DELETE]} (com.automq.stream.s3.compact.StreamObjectCompactor)
java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found.
at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073)
at com.automq.stream.s3.compact.StreamObjectCompactor.cleanupExpiredObject(StreamObjectCompactor.java:250)
at com.automq.stream.s3.compact.StreamObjectCompactor.compact0(StreamObjectCompactor.java:168)
at com.automq.stream.s3.compact.StreamObjectCompactor.compact(StreamObjectCompactor.java:116)
at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:418)
at com.automq.stream.s3.S3StreamClient$StreamWrapper.compactV1(S3StreamClient.java:402)
at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:375)
at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$6(S3StreamClient.java:155)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$7(S3StreamClient.java:155)
at com.automq.stream.utils.ThreadUtils.lambda$wrapRunnable$0(ThreadUtils.java:74)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found.
[2025-04-15 12:38:07,181] ERROR [QuorumController id=0] [CommitStreamObject]: failed to mark destroy compacted objects. compactedObjects=[6495919, 154573100], req=CommitStreamObjectRequestData(nodeId=2, nodeEpoch=1743911169270, objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, sourceObjectIds=[6495919, 154573100], streamEpoch=14, attributes=0, operations=[2, 2]) (org.apache.kafka.controller.stream.StreamControlManager)
[2025-04-15 12:38:07,181] ERROR [QuorumController id=0] object 6495919 not exist when mark destroy object (org.apache.kafka.controller.stream.S3ObjectControlManager)
[2025-04-15 12:37:07,107] ERROR [STREAM_OBJECT_COMPACT_FAIL],[UNEXPECTED],5035,type=CLEANUP_V1,req=CompactStreamObjectRequest{objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, streamEpoch=14, attributes=0, sourceObjectIds=[6495919, 154573100], operations=[DEEP_DELETE, DEEP_DELETE]} (com.automq.stream.s3.compact.StreamObjectCompactor)
java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found.
at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073)
at com.automq.stream.s3.compact.StreamObjectCompactor.cleanupExpiredObject(StreamObjectCompactor.java:250)
at com.automq.stream.s3.compact.StreamObjectCompactor.compact0(StreamObjectCompactor.java:168)
at com.automq.stream.s3.compact.StreamObjectCompactor.compact(StreamObjectCompactor.java:116)
at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:418)
at com.automq.stream.s3.S3StreamClient$StreamWrapper.compactV1(S3StreamClient.java:402)
at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:375)
at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$6(S3StreamClient.java:155)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$7(S3StreamClient.java:155)
at com.automq.stream.utils.ThreadUtils.lambda$wrapRunnable$0(ThreadUtils.java:74)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found.
[2025-04-15 12:37:07,106] ERROR [QuorumController id=0] [CommitStreamObject]: failed to mark destroy compacted objects. compactedObjects=[6495919, 154573100], req=CommitStreamObjectRequestData(nodeId=2, nodeEpoch=1743911169270, objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, sourceObjectIds=[6495919, 154573100], streamEpoch=14, attributes=0, operations=[2, 2]) (org.apache.kafka.controller.stream.StreamControlManager)
[2025-04-15 12:37:07,106] ERROR [QuorumController id=0] object 6495919 not exist when mark destroy object (org.apache.kafka.controller.stream.S3ObjectControlManager)
[2025-04-15 12:36:07,052] ERROR [STREAM_OBJECT_COMPACT_FAIL],[UNEXPECTED],5035,type=CLEANUP_V1,req=CompactStreamObjectRequest{objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, streamEpoch=14, attributes=0, sourceObjectIds=[6495919, 154573100], operations=[DEEP_DELETE, DEEP_DELETE]} (com.automq.stream.s3.compact.StreamObjectCompactor)
java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found.
at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073)
at com.automq.stream.s3.compact.StreamObjectCompactor.cleanupExpiredObject(StreamObjectCompactor.java:250)
at com.automq.stream.s3.compact.StreamObjectCompactor.compact0(StreamObjectCompactor.java:168)
at com.automq.stream.s3.compact.StreamObjectCompactor.compact(StreamObjectCompactor.java:116)
at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:418)
at com.automq.stream.s3.S3StreamClient$StreamWrapper.compactV1(S3StreamClient.java:402)
at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:375)
at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$6(S3StreamClient.java:155)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$7(S3StreamClient.java:155)
at com.automq.stream.utils.ThreadUtils.lambda$wrapRunnable$0(ThreadUtils.java:74)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found.
[2025-04-15 12:36:07,051] ERROR [QuorumController id=0] [CommitStreamObject]: failed to mark destroy compacted objects. compactedObjects=[6495919, 154573100], req=CommitStreamObjectRequestData(nodeId=2, nodeEpoch=1743911169270, objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, sourceObjectIds=[6495919, 154573100], streamEpoch=14, attributes=0, operations=[2, 2]) (org.apache.kafka.controller.stream.StreamControlManager)
[2025-04-15 12:36:07,051] ERROR [QuorumController id=0] object 6495919 not exist when mark destroy object (org.apache.kafka.controller.stream.S3ObjectControlManager)
[2025-04-15 12:35:06,998] ERROR [STREAM_OBJECT_COMPACT_FAIL],[UNEXPECTED],5035,type=CLEANUP_V1,req=CompactStreamObjectRequest{objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, streamEpoch=14, attributes=0, sourceObjectIds=[6495919, 154573100], operations=[DEEP_DELETE, DEEP_DELETE]} (com.automq.stream.s3.compact.StreamObjectCompactor)
java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found.
at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073)
at com.automq.stream.s3.compact.StreamObjectCompactor.cleanupExpiredObject(StreamObjectCompactor.java:250)
at com.automq.stream.s3.compact.StreamObjectCompactor.compact0(StreamObjectCompactor.java:168)
at com.automq.stream.s3.compact.StreamObjectCompactor.compact(StreamObjectCompactor.java:116)
at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:418)
at com.automq.stream.s3.S3StreamClient$StreamWrapper.compactV1(S3StreamClient.java:402)
at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:375)
at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$6(S3StreamClient.java:155)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$7(S3StreamClient.java:155)
at com.automq.stream.utils.ThreadUtils.lambda$wrapRunnable$0(ThreadUtils.java:74)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found.
[2025-04-15 12:35:06,992] ERROR [QuorumController id=0] object 6495919 not exist when mark destroy object (org.apache.kafka.controller.stream.S3ObjectControlManager)
[2025-04-15 12:35:06,992] ERROR [QuorumController id=0] [CommitStreamObject]: failed to mark destroy compacted objects. compactedObjects=[6495919, 154573100], req=CommitStreamObjectRequestData(nodeId=2, nodeEpoch=1743911169270, objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, sourceObjectIds=[6495919, 154573100], streamEpoch=14, attributes=0, operations=[2, 2]) (org.apache.kafka.controller.stream.StreamControlManager)
[2025-04-15 12:34:06,939] ERROR [STREAM_OBJECT_COMPACT_FAIL],[UNEXPECTED],5035,type=CLEANUP_V1,req=CompactStreamObjectRequest{objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, streamEpoch=14, attributes=0, sourceObjectIds=[6495919, 154573100], operations=[DEEP_DELETE, DEEP_DELETE]} (com.automq.stream.s3.compact.StreamObjectCompactor)
java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found.
at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073)
at com.automq.stream.s3.compact.StreamObjectCompactor.cleanupExpiredObject(StreamObjectCompactor.java:250)
at com.automq.stream.s3.compact.StreamObjectCompactor.compact0(StreamObjectCompactor.java:168)
at com.automq.stream.s3.compact.StreamObjectCompactor.compact(StreamObjectCompactor.java:116)
at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:418)
at com.automq.stream.s3.S3StreamClient$StreamWrapper.compactV1(S3StreamClient.java:402)
at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:375)
at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$6(S3StreamClient.java:155)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$7(S3StreamClient.java:155)
at com.automq.stream.utils.ThreadUtils.lambda$wrapRunnable$0(ThreadUtils.java:74)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found.
[2025-04-15 12:34:06,938] ERROR [QuorumController id=0] [CommitStreamObject]: failed to mark destroy compacted objects. compactedObjects=[6495919, 154573100], req=CommitStreamObjectRequestData(nodeId=2, nodeEpoch=1743911169270, objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, sourceObjectIds=[6495919, 154573100], streamEpoch=14, attributes=0, operations=[2, 2]) (org.apache.kafka.controller.stream.StreamControlManager)
[2025-04-15 12:34:06,938] ERROR [QuorumController id=0] object 6495919 not exist when mark destroy object (org.apache.kafka.controller.stream.S3ObjectControlManager)
[2025-04-15 12:33:06,881] ERROR [STREAM_OBJECT_COMPACT_FAIL],[UNEXPECTED],5035,type=CLEANUP_V1,req=CompactStreamObjectRequest{objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, streamEpoch=14, attributes=0, sourceObjectIds=[6495919, 154573100], operations=[DEEP_DELETE, DEEP_DELETE]} (com.automq.stream.s3.compact.StreamObjectCompactor)
java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found.
at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073)
at com.automq.stream.s3.compact.StreamObjectCompactor.cleanupExpiredObject(StreamObjectCompactor.java:250)
at com.automq.stream.s3.compact.StreamObjectCompactor.compact0(StreamObjectCompactor.java:168)
at com.automq.stream.s3.compact.StreamObjectCompactor.compact(StreamObjectCompactor.java:116)
at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:418)
at com.automq.stream.s3.S3StreamClient$StreamWrapper.compactV1(S3StreamClient.java:402)
at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:375)
at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$6(S3StreamClient.java:155)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$7(S3StreamClient.java:155)
at com.automq.stream.utils.ThreadUtils.lambda$wrapRunnable$0(ThreadUtils.java:74)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found.
[2025-04-15 12:33:06,880] ERROR [QuorumController id=0] [CommitStreamObject]: failed to mark destroy compacted objects. compactedObjects=[6495919, 154573100], req=CommitStreamObjectRequestData(nodeId=2, nodeEpoch=1743911169270, objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, sourceObjectIds=[6495919, 154573100], streamEpoch=14, attributes=0, operations=[2, 2]) (org.apache.kafka.controller.stream.StreamControlManager)
[2025-04-15 12:33:06,880] ERROR [QuorumController id=0] object 6495919 not exist when mark destroy object (org.apache.kafka.controller.stream.S3ObjectControlManager)
[2025-04-15 12:32:04,856] ERROR [STREAM_OBJECT_COMPACT_FAIL],[UNEXPECTED],5035,type=CLEANUP_V1,req=CompactStreamObjectRequest{objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, streamEpoch=14, attributes=0, sourceObjectIds=[6495919, 154573100], operations=[DEEP_DELETE, DEEP_DELETE]} (com.automq.stream.s3.compact.StreamObjectCompactor)
java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found.
at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073)
at com.automq.stream.s3.compact.StreamObjectCompactor.cleanupExpiredObject(StreamObjectCompactor.java:250)
at com.automq.stream.s3.compact.StreamObjectCompactor.compact0(StreamObjectCompactor.java:168)
at com.automq.stream.s3.compact.StreamObjectCompactor.compact(StreamObjectCompactor.java:116)
at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:418)
at com.automq.stream.s3.S3StreamClient$StreamWrapper.compactV1(S3StreamClient.java:402)
at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:375)
at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$6(S3StreamClient.java:155)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$7(S3StreamClient.java:155)
at com.automq.stream.utils.ThreadUtils.lambda$wrapRunnable$0(ThreadUtils.java:74)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found.
[2025-04-15 12:32:04,855] ERROR [QuorumController id=0] object 6495919 not exist when mark destroy object (org.apache.kafka.controller.stream.S3ObjectControlManager)
[2025-04-15 12:32:04,855] ERROR [QuorumController id=0] [CommitStreamObject]: failed to mark destroy compacted objects. compactedObjects=[6495919, 154573100], req=CommitStreamObjectRequestData(nodeId=2, nodeEpoch=1743911169270, objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, sourceObjectIds=[6495919, 154573100], streamEpoch=14, attributes=0, operations=[2, 2]) (org.apache.kafka.controller.stream.StreamControlManager)
[2025-04-15 12:31:03,988] ERROR [STREAM_OBJECT_COMPACT_FAIL],[UNEXPECTED],5035,type=CLEANUP_V1,req=CompactStreamObjectRequest{objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, streamEpoch=14, attributes=0, sourceObjectIds=[6495919, 154573100], operations=[DEEP_DELETE, DEEP_DELETE]} (com.automq.stream.s3.compact.StreamObjectCompactor)
java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found.
at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073)
at com.automq.stream.s3.compact.StreamObjectCompactor.cleanupExpiredObject(StreamObjectCompactor.java:250)
at com.automq.stream.s3.compact.StreamObjectCompactor.compact0(StreamObjectCompactor.java:168)
at com.automq.stream.s3.compact.StreamObjectCompactor.compact(StreamObjectCompactor.java:116)
at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:418)
at com.automq.stream.s3.S3StreamClient$StreamWrapper.compactV1(S3StreamClient.java:402)
at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:375)
at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$6(S3StreamClient.java:155)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$7(S3StreamClient.java:155)
at com.automq.stream.utils.ThreadUtils.lambda$wrapRunnable$0(ThreadUtils.java:74)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found.
[2025-04-15 12:31:03,987] ERROR [QuorumController id=0] object 6495919 not exist when mark destroy object (org.apache.kafka.controller.stream.S3ObjectControlManager)
[2025-04-15 12:31:03,987] ERROR [QuorumController id=0] [CommitStreamObject]: failed to mark destroy compacted objects. compactedObjects=[6495919, 154573100], req=CommitStreamObjectRequestData(nodeId=2, nodeEpoch=1743911169270, objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, sourceObjectIds=[6495919, 154573100], streamEpoch=14, attributes=0, operations=[2, 2]) (org.apache.kafka.controller.stream.StreamControlManager)
[2025-04-15 12:29:59,874] ERROR [QuorumController id=0] [CommitStreamObject]: failed to mark destroy compacted objects. compactedObjects=[6495919, 154573100], req=CommitStreamObjectRequestData(nodeId=2, nodeEpoch=1743911169270, objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, sourceObjectIds=[6495919, 154573100], streamEpoch=14, attributes=0, operations=[2, 2]) (org.apache.kafka.controller.stream.StreamControlManager)
[2025-04-15 12:29:59,874] ERROR [STREAM_OBJECT_COMPACT_FAIL],[UNEXPECTED],5035,type=CLEANUP_V1,req=CompactStreamObjectRequest{objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, streamEpoch=14, attributes=0, sourceObjectIds=[6495919, 154573100], operations=[DEEP_DELETE, DEEP_DELETE]} (com.automq.stream.s3.compact.StreamObjectCompactor)
java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found.
at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073)
at com.automq.stream.s3.compact.StreamObjectCompactor.cleanupExpiredObject(StreamObjectCompactor.java:250)
at com.automq.stream.s3.compact.StreamObjectCompactor.compact0(StreamObjectCompactor.java:168)
at com.automq.stream.s3.compact.StreamObjectCompactor.compact(StreamObjectCompactor.java:116)
at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:418)
at com.automq.stream.s3.S3StreamClient$StreamWrapper.compactV1(S3StreamClient.java:402)
at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:375)
at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$6(S3StreamClient.java:155)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$7(S3StreamClient.java:155)
at com.automq.stream.utils.ThreadUtils.lambda$wrapRunnable$0(ThreadUtils.java:74)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found.
[2025-04-15 12:29:59,874] ERROR [QuorumController id=0] object 6495919 not exist when mark destroy object (org.apache.kafka.controller.stream.S3ObjectControlManager)
[2025-04-15 12:28:42,227] ERROR [STREAM_OBJECT_COMPACT_FAIL],[UNEXPECTED],5035,type=MINOR_V1,req=CompactStreamObjectRequest{objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, streamEpoch=14, attributes=0, sourceObjectIds=[6495919, 154573100], operations=[DEEP_DELETE, DEEP_DELETE]} (com.automq.stream.s3.compact.StreamObjectCompactor)
java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found.
at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073)
at com.automq.stream.s3.compact.StreamObjectCompactor.cleanupExpiredObject(StreamObjectCompactor.java:250)
at com.automq.stream.s3.compact.StreamObjectCompactor.compact0(StreamObjectCompactor.java:168)
at com.automq.stream.s3.compact.StreamObjectCompactor.compact(StreamObjectCompactor.java:116)
at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:418)
at com.automq.stream.s3.S3StreamClient$StreamWrapper.compactV1(S3StreamClient.java:399)
at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:375)
at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$6(S3StreamClient.java:155)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$7(S3StreamClient.java:155)
at com.automq.stream.utils.ThreadUtils.lambda$wrapRunnable$0(ThreadUtils.java:74)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found.
[2025-04-15 12:28:42,226] ERROR [QuorumController id=0] [CommitStreamObject]: failed to mark destroy compacted objects. compactedObjects=[6495919, 154573100], req=CommitStreamObjectRequestData(nodeId=2, nodeEpoch=1743911169270, objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, sourceObjectIds=[6495919, 154573100], streamEpoch=14, attributes=0, operations=[2, 2]) (org.apache.kafka.controller.stream.StreamControlManager)
[2025-04-15 12:28:42,226] ERROR [QuorumController id=0] object 6495919 not exist when mark destroy object (org.apache.kafka.controller.stream.S3ObjectControlManager)
[2025-04-15 12:27:40,050] ERROR [STREAM_OBJECT_COMPACT_FAIL],[UNEXPECTED],5035,type=CLEANUP_V1,req=CompactStreamObjectRequest{objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, streamEpoch=14, attributes=0, sourceObjectIds=[6495919, 154573100], operations=[DEEP_DELETE, DEEP_DELETE]} (com.automq.stream.s3.compact.StreamObjectCompactor)
java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found.
at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073)
at com.automq.stream.s3.compact.StreamObjectCompactor.cleanupExpiredObject(StreamObjectCompactor.java:250)
at com.automq.stream.s3.compact.StreamObjectCompactor.compact0(StreamObjectCompactor.java:168)
at com.automq.stream.s3.compact.StreamObjectCompactor.compact(StreamObjectCompactor.java:116)
at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:418)
at com.automq.stream.s3.S3StreamClient$StreamWrapper.compactV1(S3StreamClient.java:402)
at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:375)
at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$6(S3StreamClient.java:155)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$7(S3StreamClient.java:155)
at com.automq.stream.utils.ThreadUtils.lambda$wrapRunnable$0(ThreadUtils.java:74)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found.
[2025-04-15 12:27:40,050] ERROR [QuorumController id=0] object 6495919 not exist when mark destroy object (org.apache.kafka.controller.stream.S3ObjectControlManager)
[2025-04-15 12:27:40,050] ERROR [QuorumController id=0] [CommitStreamObject]: failed to mark destroy compacted objects. compactedObjects=[6495919, 154573100], req=CommitStreamObjectRequestData(nodeId=2, nodeEpoch=1743911169270, objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, sourceObjectIds=[6495919, 154573100], streamEpoch=14, attributes=0, operations=[2, 2]) (org.apache.kafka.controller.stream.StreamControlManager)
[2025-04-15 12:26:39,281] ERROR [STREAM_OBJECT_COMPACT_FAIL],[UNEXPECTED],5035,type=CLEANUP_V1,req=CompactStreamObjectRequest{objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, streamEpoch=14, attributes=0, sourceObjectIds=[6495919, 154573100], operations=[DEEP_DELETE, DEEP_DELETE]} (com.automq.stream.s3.compact.StreamObjectCompactor)
java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found.
at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073)
at com.automq.stream.s3.compact.StreamObjectCompactor.cleanupExpiredObject(StreamObjectCompactor.java:250)
at com.automq.stream.s3.compact.StreamObjectCompactor.compact0(StreamObjectCompactor.java:168)
at com.automq.stream.s3.compact.StreamObjectCompactor.compact(StreamObjectCompactor.java:116)
at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:418)
at com.automq.stream.s3.S3StreamClient$StreamWrapper.compactV1(S3StreamClient.java:402)
at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:375)
at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$6(S3StreamClient.java:155)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$7(S3StreamClient.java:155)
at com.automq.stream.utils.ThreadUtils.lambda$wrapRunnable$0(ThreadUtils.java:74)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found.
[2025-04-15 12:26:39,280] ERROR [QuorumController id=0] [CommitStreamObject]: failed to mark destroy compacted objects. compactedObjects=[6495919, 154573100], req=CommitStreamObjectRequestData(nodeId=2, nodeEpoch=1743911169270, objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, sourceObjectIds=[6495919, 154573100], streamEpoch=14, attributes=0, operations=[2, 2]) (org.apache.kafka.controller.stream.StreamControlManager)
[2025-04-15 12:26:39,280] ERROR [QuorumController id=0] object 6495919 not exist when mark destroy object (org.apache.kafka.controller.stream.S3ObjectControlManager)
[2025-04-15 12:25:38,408] ERROR [STREAM_OBJECT_COMPACT_FAIL],[UNEXPECTED],5035,type=CLEANUP_V1,req=CompactStreamObjectRequest{objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, streamEpoch=14, attributes=0, sourceObjectIds=[6495919, 154573100], operations=[DEEP_DELETE, DEEP_DELETE]} (com.automq.stream.s3.compact.StreamObjectCompactor)
java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found.
at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073)
at com.automq.stream.s3.compact.StreamObjectCompactor.cleanupExpiredObject(StreamObjectCompactor.java:250)
at com.automq.stream.s3.compact.StreamObjectCompactor.compact0(StreamObjectCompactor.java:168)
at com.automq.stream.s3.compact.StreamObjectCompactor.compact(StreamObjectCompactor.java:116)
at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:418)
at com.automq.stream.s3.S3StreamClient$StreamWrapper.compactV1(S3StreamClient.java:402)
at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:375)
at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$6(S3StreamClient.java:155)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$7(S3StreamClient.java:155)
at com.automq.stream.utils.ThreadUtils.lambda$wrapRunnable$0(ThreadUtils.java:74)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found.
[2025-04-15 12:25:38,407] ERROR [QuorumController id=0] [CommitStreamObject]: failed to mark destroy compacted objects. compactedObjects=[6495919, 154573100], req=CommitStreamObjectRequestData(nodeId=2, nodeEpoch=1743911169270, objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, sourceObjectIds=[6495919, 154573100], streamEpoch=14, attributes=0, operations=[2, 2]) (org.apache.kafka.controller.stream.StreamControlManager)
[2025-04-15 12:25:38,407] ERROR [QuorumController id=0] object 6495919 not exist when mark destroy object (org.apache.kafka.controller.stream.S3ObjectControlManager)
[2025-04-15 12:24:38,350] ERROR [STREAM_OBJECT_COMPACT_FAIL],[UNEXPECTED],5035,type=CLEANUP_V1,req=CompactStreamObjectRequest{objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, streamEpoch=14, attributes=0, sourceObjectIds=[6495919, 154573100], operations=[DEEP_DELETE, DEEP_DELETE]} (com.automq.stream.s3.compact.StreamObjectCompactor)
java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found.
at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073)
at com.automq.stream.s3.compact.StreamObjectCompactor.cleanupExpiredObject(StreamObjectCompactor.java:250)
at com.automq.stream.s3.compact.StreamObjectCompactor.compact0(StreamObjectCompactor.java:168)
at com.automq.stream.s3.compact.StreamObjectCompactor.compact(StreamObjectCompactor.java:116)
at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:418)
at com.automq.stream.s3.S3StreamClient$StreamWrapper.compactV1(S3StreamClient.java:402)
at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:375)
at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$6(S3StreamClient.java:155)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$7(S3StreamClient.java:155)
at com.automq.stream.utils.ThreadUtils.lambda$wrapRunnable$0(ThreadUtils.java:74)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found.
[2025-04-15 12:24:38,348] ERROR [QuorumController id=0] [CommitStreamObject]: failed to mark destroy compacted objects. compactedObjects=[6495919, 154573100], req=CommitStreamObjectRequestData(nodeId=2, nodeEpoch=1743911169270, objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, sourceObjectIds=[6495919, 154573100], streamEpoch=14, attributes=0, operations=[2, 2]) (org.apache.kafka.controller.stream.StreamControlManager)
[2025-04-15 12:24:38,347] ERROR [QuorumController id=0] object 6495919 not exist when mark destroy object (org.apache.kafka.controller.stream.S3ObjectControlManager)
[2025-04-15 12:23:38,231] ERROR [STREAM_OBJECT_COMPACT_FAIL],[UNEXPECTED],5035,type=CLEANUP_V1,req=CompactStreamObjectRequest{objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, streamEpoch=14, attributes=0, sourceObjectIds=[6495919, 154573100], operations=[DEEP_DELETE, DEEP_DELETE]} (com.automq.stream.s3.compact.StreamObjectCompactor)
java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found.
at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073)
at com.automq.stream.s3.compact.StreamObjectCompactor.cleanupExpiredObject(StreamObjectCompactor.java:250)
at com.automq.stream.s3.compact.StreamObjectCompactor.compact0(StreamObjectCompactor.java:168)
at com.automq.stream.s3.compact.StreamObjectCompactor.compact(StreamObjectCompactor.java:116)
at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:418)
at com.automq.stream.s3.S3StreamClient$StreamWrapper.compactV1(S3StreamClient.java:402)
at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:375)
at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$6(S3StreamClient.java:155)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$7(S3StreamClient.java:155)
at com.automq.stream.utils.ThreadUtils.lambda$wrapRunnable$0(ThreadUtils.java:74)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found.
[2025-04-15 12:23:38,231] ERROR [QuorumController id=0] [CommitStreamObject]: failed to mark destroy compacted objects. compactedObjects=[6495919, 154573100], req=CommitStreamObjectRequestData(nodeId=2, nodeEpoch=1743911169270, objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, sourceObjectIds=[6495919, 154573100], streamEpoch=14, attributes=0, operations=[2, 2]) (org.apache.kafka.controller.stream.StreamControlManager)
[2025-04-15 12:23:38,231] ERROR [QuorumController id=0] object 6495919 not exist when mark destroy object (org.apache.kafka.controller.stream.S3ObjectControlManager)
[2025-04-15 12:22:37,526] ERROR [STREAM_OBJECT_COMPACT_FAIL],[UNEXPECTED],5035,type=CLEANUP_V1,req=CompactStreamObjectRequest{objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, streamEpoch=14, attributes=0, sourceObjectIds=[6495919, 154573100], operations=[DEEP_DELETE, DEEP_DELETE]} (com.automq.stream.s3.compact.StreamObjectCompactor)
java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found.
at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073)
at com.automq.stream.s3.compact.StreamObjectCompactor.cleanupExpiredObject(StreamObjectCompactor.java:250)
at com.automq.stream.s3.compact.StreamObjectCompactor.compact0(StreamObjectCompactor.java:168)
at com.automq.stream.s3.compact.StreamObjectCompactor.compact(StreamObjectCompactor.java:116)
at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:418)
at com.automq.stream.s3.S3StreamClient$StreamWrapper.compactV1(S3StreamClient.java:402)
at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:375)
at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$6(S3StreamClient.java:155)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$7(S3StreamClient.java:155)
at com.automq.stream.utils.ThreadUtils.lambda$wrapRunnable$0(ThreadUtils.java:74)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found.
[2025-04-15 12:22:37,525] ERROR [QuorumController id=0] [CommitStreamObject]: failed to mark destroy compacted objects. compactedObjects=[6495919, 154573100], req=CommitStreamObjectRequestData(nodeId=2, nodeEpoch=1743911169270, objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, sourceObjectIds=[6495919, 154573100], streamEpoch=14, attributes=0, operations=[2, 2]) (org.apache.kafka.controller.stream.StreamControlManager)
[2025-04-15 12:22:37,525] ERROR [QuorumController id=0] object 6495919 not exist when mark destroy object (org.apache.kafka.controller.stream.S3ObjectControlManager)
[2025-04-15 12:21:37,469] ERROR [QuorumController id=0] [CommitStreamObject]: failed to mark destroy compacted objects. compactedObjects=[6495919, 154573100], req=CommitStreamObjectRequestData(nodeId=2, nodeEpoch=1743911169270, objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, sourceObjectIds=[6495919, 154573100], streamEpoch=14, attributes=0, operations=[2, 2]) (org.apache.kafka.controller.stream.StreamControlManager)
[2025-04-15 12:21:37,472] ERROR [STREAM_OBJECT_COMPACT_FAIL],[UNEXPECTED],5035,type=CLEANUP_V1,req=CompactStreamObjectRequest{objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, streamEpoch=14, attributes=0, sourceObjectIds=[6495919, 154573100], operations=[DEEP_DELETE, DEEP_DELETE]} (com.automq.stream.s3.compact.StreamObjectCompactor)
java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found.
at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073)
at com.automq.stream.s3.compact.StreamObjectCompactor.cleanupExpiredObject(StreamObjectCompactor.java:250)
at com.automq.stream.s3.compact.StreamObjectCompactor.compact0(StreamObjectCompactor.java:168)
at com.automq.stream.s3.compact.StreamObjectCompactor.compact(StreamObjectCompactor.java:116)
at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:418)
at com.automq.stream.s3.S3StreamClient$StreamWrapper.compactV1(S3StreamClient.java:402)
at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:375)
at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$6(S3StreamClient.java:155)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$7(S3StreamClient.java:155)
at com.automq.stream.utils.ThreadUtils.lambda$wrapRunnable$0(ThreadUtils.java:74)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found.
[2025-04-15 12:21:37,469] ERROR [QuorumController id=0] object 6495919 not exist when mark destroy object (org.apache.kafka.controller.stream.S3ObjectControlManager)
[2025-04-15 12:20:37,391] ERROR [QuorumController id=0] [CommitStreamObject]: failed to mark destroy compacted objects. compactedObjects=[6495919, 154573100], req=CommitStreamObjectRequestData(nodeId=2, nodeEpoch=1743911169270, objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, sourceObjectIds=[6495919, 154573100], streamEpoch=14, attributes=0, operations=[2, 2]) (org.apache.kafka.controller.stream.StreamControlManager)
[2025-04-15 12:20:37,392] ERROR [STREAM_OBJECT_COMPACT_FAIL],[UNEXPECTED],5035,type=CLEANUP_V1,req=CompactStreamObjectRequest{objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, streamEpoch=14, attributes=0, sourceObjectIds=[6495919, 154573100], operations=[DEEP_DELETE, DEEP_DELETE]} (com.automq.stream.s3.compact.StreamObjectCompactor)
java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found.
at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073)
at com.automq.stream.s3.compact.StreamObjectCompactor.cleanupExpiredObject(StreamObjectCompactor.java:250)
at com.automq.stream.s3.compact.StreamObjectCompactor.compact0(StreamObjectCompactor.java:168)
at com.automq.stream.s3.compact.StreamObjectCompactor.compact(StreamObjectCompactor.java:116)
at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:418)
at com.automq.stream.s3.S3StreamClient$StreamWrapper.compactV1(S3StreamClient.java:402)
at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:375)
at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$6(S3StreamClient.java:155)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$7(S3StreamClient.java:155)
at com.automq.stream.utils.ThreadUtils.lambda$wrapRunnable$0(ThreadUtils.java:74)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found.
[2025-04-15 12:20:37,391] ERROR [QuorumController id=0] object 6495919 not exist when mark destroy object (org.apache.kafka.controller.stream.S3ObjectControlManager)

[superhx]

@jerome-j-20230331

1. Are there any logs related to object 6495919 before the first 'failed to mark destroy compacted' happens?
2. Could you try to rolling restart all the controller nodes and recheck whether the error recover

[jerome-j-20230331]

@superhx I restarted the three controller nodes one by one. The error "object 154377965 not exist when mark destroy object (org.apache.kafka.controller.stream.S3ObjectControlManager)" in controller.log has temporarily disappeared, but after manually rebalancing the cluster partitions, such errors still appear. Restarting the three controllers again can solve the problem, but other similar errors still occur:.
server.log:
[2025-04-16 07:19:57,393] INFO [CompactionManager id=2] Compaction started (com.automq.stream.s3.compact.CompactionManager)
[2025-04-16 07:19:57,394] INFO [CompactionManager id=2] Get 30 stream set objects from metadata (com.automq.stream.s3.compact.CompactionManager)
[2025-04-16 07:19:57,691] INFO [CompactionManager id=2] Throttle compaction read to 3492623 bytes/s, expect to complete in no less than 9min (com.automq.stream.s3.compact.CompactionManager)
[2025-04-16 07:19:57,691] INFO [CompactionManager id=2] Compact 30 stream set objects, total 30 objects to compact (com.automq.stream.s3.compact.CompactionManager)
[2025-04-16 07:19:57,691] INFO [CompactionManager id=2] 30 stream set objects as compact candidates, total compaction size: 1886016707 (com.automq.stream.s3.compact.CompactionManager)
[2025-04-16 07:19:57,694] INFO [CompactionAnalyzer id=2] 30 stream set objects to compact after filter (com.automq.stream.s3.compact.CompactionAnalyzer)
[2025-04-16 07:19:57,711] INFO [CompactionAnalyzer id=2] Current stream num in stream set: 310, max: 20000, stream object num: 42, max: 10000 (com.automq.stream.s3.compact.CompactionAnalyzer)
[2025-04-16 07:19:57,716] INFO [CompactionManager id=2] Analyze compaction plans complete, cost 25ms (com.automq.stream.s3.compact.CompactionManager)
[2025-04-16 07:19:57,716] INFO [CompactionManager id=2] Compaction plans: expect to generate 50 Stream Object, 1 stream set object with size 253553038 in 9 iterations, objects excluded: [] (com.automq.stream.s3.compact.CompactionManager)
[2025-04-16 07:19:57,717] INFO [CompactionManager id=2] Compaction progress 1/9, read from 30 stream set objects, total size: 209598660 (com.automq.stream.s3.compact.CompactionManager)
[2025-04-16 07:19:58,356] INFO [KafkaApi-2] produce cost (in microseconds), produce={count=6397, max=570.0, mean=107.1786775050805} callback={count=6397, max=2085.0, mean=87.62419884320775} ack={count=6397, max=65.0, mean=19.47412849773331} (kafka.server.streamaspect.ElasticKafkaApis)
[2025-04-16 07:20:05,996] INFO log append cost, permitAcquireFail={count=0, max=0.0, mean=0.0}, remainingPermit=104857600/104857600, append={count=6456, max=206926.0, mean=21550.96375464684}, callback={count=6456, max=1.1073328E7, mean=1262797.1220570013}, ack={count=6263, max=220723.0, mean=20970.326201500877} (kafka.log.streamaspect.ElasticLog)
[2025-04-16 07:20:57,342] WARN [GetStreamObjects]: stream: 4711, startOffset: 0, endOffset: 1454976172, limit: 2147483647, and search in metadataCache failed with exception: Cannot invoke "org.apache.kafka.metadata.stream.S3Object.getTimestamp()" because "objectMetadata" is null (kafka.log.stream.s3.metadata.StreamMetadataManager)
[2025-04-16 07:20:57,343] ERROR [STREAM_OBJECT_COMPACT_FAIL],[UNEXPECTED],4711,type=CLEANUP_V1,req=null (com.automq.stream.s3.compact.StreamObjectCompactor)
java.util.concurrent.ExecutionException: java.lang.NullPointerException: Cannot invoke "org.apache.kafka.metadata.stream.S3Object.getTimestamp()" because "objectMetadata" is null
at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073)
at com.automq.stream.s3.compact.StreamObjectCompactor.compact0(StreamObjectCompactor.java:156)
at com.automq.stream.s3.compact.StreamObjectCompactor.compact(StreamObjectCompactor.java:116)
at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:418)
at com.automq.stream.s3.S3StreamClient$StreamWrapper.compactV1(S3StreamClient.java:402)
at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:375)
at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$6(S3StreamClient.java:155)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$7(S3StreamClient.java:155)
at com.automq.stream.utils.ThreadUtils.lambda$wrapRunnable$0(ThreadUtils.java:74)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.lang.NullPointerException: Cannot invoke "org.apache.kafka.metadata.stream.S3Object.getTimestamp()" because "objectMetadata" is null
at kafka.log.stream.s3.metadata.StreamMetadataManager.lambda$getStreamObjects$8(StreamMetadataManager.java:182)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
at kafka.log.stream.s3.metadata.StreamMetadataManager.getStreamObjects(StreamMetadataManager.java:187)
at kafka.log.stream.s3.objects.ControllerObjectManager.getStreamObjects(ControllerObjectManager.java:274)
... 15 more
[2025-04-16 07:20:58,042] INFO [CompactionManager id=2] Compaction progress 2/9, read from 30 stream set objects, total size: 209385517 (com.automq.stream.s3.compact.CompactionManager)
[2025-04-16 07:20:58,432] INFO [KafkaApi-2] produce cost (in microseconds), produce={count=6236, max=551.0, mean=114.69611930724824} callback={count=6236, max=2091.0, mean=93.22370109044259} ack={count=6236, max=83.0, mean=20.784637588197562} (kafka.server.streamaspect.ElasticKafkaApis)
[2025-04-16 07:21:06,017] INFO log append cost, permitAcquireFail={count=0, max=0.0, mean=0.0}, remainingPermit=104857600/104857600, append={count=6295, max=159318.0, mean=22574.997776012708}, callback={count=6295, max=6848319.0, mean=1262421.6057188245}, ack={count=6147, max=159464.0, mean=21690.954612005855} (kafka.log.streamaspect.ElasticLog)
[2025-04-16 07:21:57,344] WARN [GetStreamObjects]: stream: 4711, startOffset: 0, endOffset: 1454984131, limit: 2147483647, and search in metadataCache failed with exception: Cannot invoke "org.apache.kafka.metadata.stream.S3Object.getTimestamp()" because "objectMetadata" is null (kafka.log.stream.s3.metadata.StreamMetadataManager)
[2025-04-16 07:21:57,344] ERROR [STREAM_OBJECT_COMPACT_FAIL],[UNEXPECTED],4711,type=CLEANUP_V1,req=null (com.automq.stream.s3.compact.StreamObjectCompactor)
java.util.concurrent.ExecutionException: java.lang.NullPointerException: Cannot invoke "org.apache.kafka.metadata.stream.S3Object.getTimestamp()" because "objectMetadata" is null
at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073)
at com.automq.stream.s3.compact.StreamObjectCompactor.compact0(StreamObjectCompactor.java:156)
at com.automq.stream.s3.compact.StreamObjectCompactor.compact(StreamObjectCompactor.java:116)
at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:418)
at com.automq.stream.s3.S3StreamClient$StreamWrapper.compactV1(S3StreamClient.java:402)
at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:375)
at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$6(S3StreamClient.java:155)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$7(S3StreamClient.java:155)
at com.automq.stream.utils.ThreadUtils.lambda$wrapRunnable$0(ThreadUtils.java:74)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.lang.NullPointerException: Cannot invoke "org.apache.kafka.metadata.stream.S3Object.getTimestamp()" because "objectMetadata" is null
at kafka.log.stream.s3.metadata.StreamMetadataManager.lambda$getStreamObjects$8(StreamMetadataManager.java:182)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
at kafka.log.stream.s3.metadata.StreamMetadataManager.getStreamObjects(StreamMetadataManager.java:187)
at kafka.log.stream.s3.objects.ControllerObjectManager.getStreamObjects(ControllerObjectManager.java:274)
... 15 more
[2025-04-16 07:21:57,564] INFO [CompactionManager id=2] Compaction progress 3/9, read from 30 stream set objects, total size: 209371118 (com.automq.stream.s3.compact.CompactionManager)
[2025-04-16 07:21:58,434] INFO [KafkaApi-2] produce cost (in microseconds), produce={count=6642, max=1138.0, mean=115.77220716651611} callback={count=6642, max=1964.0, mean=92.86013249021379} ack={count=6642, max=124.0, mean=21.08130081300813} (kafka.server.streamaspect.ElasticKafkaApis)

[superhx]

It seams the S3Object was deleted but the S3StreamObject wasn't deleted. It should not happen.

I will find the reason. You can delete streamId=4711 related topic to fix it.