Random io.grpc.StatusRuntimeException: CANCELLED exceptions in Server-Side Streaming RPC

[dennyjames]

gRPC Version : 1.38.0
Java Version : openjdk version "11.0.27" 2025-04-15 LTS

This issue is random and is not reproducible in a test environment. Even on the environments were this is seen, issue happens randomly.

I have a server side streaming gRPC method, That returns a Iterator to the client. While looping on the iterator, below error occurs on the client side.

6066f128-75d7-42a2-af96-40c98bca2137 orgId=649529784 user=55fcc549-5f6a-42d7-85a0-2786f11a580b null
io.grpc.StatusRuntimeException: CANCELLED: Server sendMessage() failed with Error
	at io.grpc.Status.asRuntimeException(Status.java:532)
	at io.grpc.stub.ClientCalls$BlockingResponseStream.hasNext(ClientCalls.java:662)
	at com.***.data.result.BatchedSqlResultSet.next(BatchedSqlResultSet.java:89)
	at com.***.common.sql.SqlParser.getFalconResultSet(SqlParser.java:95)
	at com.***.data.manager.exporter.DataExporterImpl.exportTableVisualization(DataExporterImpl.java:1064)
	at com.***.data.manager.exporter.DataExporterImpl.exportTableVisualization(DataExporterImpl.java:927)
	at com.***.data.manager.exporter.DataExporterImpl.exportDataForVisualization(DataExporterImpl.java:640)
	at com.***.data.manager.exporter.DataExporterImpl.exportDataForVisualizations(DataExporterImpl.java:538)

On the server side, I see the below error

java.lang.IllegalStateException: call already closed
        at com.google.common.base.Preconditions.checkState(Preconditions.java:502)
        at io.grpc.internal.ServerCallImpl.closeInternal(ServerCallImpl.java:218)
        at io.grpc.internal.ServerCallImpl.close(ServerCallImpl.java:211)
        at io.grpc.stub.ServerCalls$ServerCallStreamObserverImpl.onCompleted(ServerCalls.java:395)
        at com.***.server.DataManagerGrpcImpl.executeStreamQuery(DataManagerGrpcImpl.java:177)

The fact that the error just mentions CANCELLED, without a detailed error makes it hard to understand the error.

[kannanjgithub]

It looks like your service impl is unaware of the call being already cancelled from the server side (which could happen due to server side call timeout for example) but it still tries to write to the response stream. If your service impl uses async processing, the ServerCall's onCancel will indicate to the ServerCallStreamObserver's cancellation handler so you can stop writing to the stream. If your service impl blocks the calling thread, then ServerCall's onCancel will be called only after the onMessage returns and in this case you need to either use Context (polling or Listener) or ServerCallStreamObserver.isCancelled() (polling-only) to learn of the cancellation.

[ejona86]

Server sendMessage() failed with Error is pretty relevant. The IllegalStateException is unfortunate, and there may be something we could improve to make this error handling cleaner, but an Error just occurred, not an Exception. The most likely thing is your server got an OutOfMemoryError while trying to send a message. I suggest looking at the server logs for that or another Error.

We can't send more detailed error information to the client because we mustn't expose such detailed server application details with an arbitrary client.

[dennyjames]

@ejona86 We don't have a OutOfMemoryError on the server.

We are now seeing this error randomly even in Unary RPC calls. For Unary calls, we don't seejava.lang.IllegalStateException: call already closed on the server, I guess that is expected as there isn't a stream. A concrete pattern that we are observing whenever this error happens is the presence of a System.gc() event on the server's gc logs.

[398834.019s][info   ][gc,start       ] GC(480) Pause Young (System.gc())
[398834.032s][info   ][gc,heap        ] GC(480) PSYoungGen: 338650K->3648K(1212416K)
[398834.032s][info   ][gc,heap        ] GC(480) ParOldGen: 1809857K->1809857K(2162176K)
[398834.032s][info   ][gc,metaspace   ] GC(480) Metaspace: 115970K(118104K)->115970K(118104K) NonClass: 104893K(106456K)->104893K(106456K) Class: 11076K(11648K)->11076K(11648K)
[398834.032s][info   ][gc             ] GC(480) Pause Young (System.gc()) 2098M->1771M(3295M) 12.162ms
[398834.032s][info   ][gc,cpu         ] GC(480) User=0.03s Sys=0.00s Real=0.01s
[398834.032s][info   ][gc,start       ] GC(481) Pause Full (System.gc())

We are unaware of the trigger of this System.gc(), we don't explicitly do that from application code. To experiment a little we set the javaopt XX +DisableExplicitGC, to turn of explicit System.gc(). After this, we haven't seen this error so far. We are still monitoring, and I will update here later.

[ejona86]

There is absolutely some Error. If it isn't OutOfMemoryError, fine, but it is definitely something. I was just saying that as the most likely.