Evaluate if credit_flow block strategy can be changed

[videlalvaro]

It is my understanding that currently it is enough for a process to block a Pid, for that Pid to be blocked by credit_flow for sending messages to other processes:

https://github.com/rabbitmq/rabbitmq-server/blob/master/src/credit_flow.erl#L112
https://github.com/rabbitmq/rabbitmq-server/blob/master/src/credit_flow.erl#L150

Example: a rabbit_reader is sending messages to channels A, B and C. If channel A blocks the reader, then the reader will be blocked by B and C even tho perhaps those channels are able to take those messages.

Another situation: a channel is delivering messages to queues A, B and C. Queue A is receiving quite a lot of messages, so it blocks the channel via flow control. The channel will be blocked for sending messages to queues B and C, when perhaps those messages aren't even persistent, or those queues aren't mirrored and they might be ready to accept said messages.

Is the current implementation a conservative approach trying to block the channel as soon as one queue can't keep up?

[michaelklishin]

The ultimate problem is that it can be dangerous: e.g. we can't know what channel the next message is on, so if we let the reader read and frame it, what do we do if it is for the blocked channel? The next message can also push us past the RAM watermark limit. Plus credit_flow already does buffering.

So this is something that'd need a lot of investigation. Making credit flow values configurable is the first step to make experimentation straightforward.