Help understanding connection.py event loop blocking

[atulenko]

I've search quite a bit and can't seem to find anything to be the likely culprit and so I was hoping someone here may have a suggestion about the root cause of an issue I am seeing.

I periodically see slow callback warnings stemming from what appears to be asyncpg/connection.py's task occupying the event loop for > 100ms:

WARNING:asyncio:Executing <Task pending name='Task-55' coro=<RequestResponseCycle.run_asgi() running at /usr/local/lib/python3.11/site-packages/uvicorn/protocols/http/httptools_impl.py:372> wait_for=<Future pending cb=[Protocol._on_waiter_completed(), Task.task_wakeup()] created at /usr/local/lib/python3.11/site-packages/asyncpg/connection.py:433> cb=[set.discard()] created at /usr/local/lib/python3.11/site-packages/uvicorn/protocols/http/httptools_impl.py:253> took 0.161 seconds

Typically in the 100-300ms range but can get as high as 3-5s.

This specific line in connection.py is the protocol prepare (I am still on 0.29 but a quick check with 0.30 did not seem to resolve the issue):
statement = await self._protocol.prepare(
stmt_name,
query,
timeout,
record_class=record_class,
ignore_custom_codec=ignore_custom_codec,
)

Which then is likely invoking the prepare at

asyncpg/asyncpg/protocol/protocol.pyx

Line 153 in 5b14653

waiter = self._new_waiter(timeout)

since that is where the _on_waiter_completed cb should be coming from.

That being said I am far from an expert at asyncpg so I am not certain what the underlying root cause could be.

I suspect it's possibly tied to my use of a NullPool in sqlalchemy which will open a new connection for each session. Possibly tied to the number of parallel connections. But even then I would have expected the connections to not block the event loop? Or is this occasional blocking of the event loop expected with asyncpg?

Would appreciate any guidance on what to investigate here if anyone has any suggestions/thoughts around this!