Improve gossipsub

[sukhman-sukh]

What was wrong?

Current version of gossipsub did not have message queuing, priority message queuing, and also, message splitting based on maxMessageSize as discussed for interoperability with go-libp2p.
Issue # #891

How was it fixed?

• Added a rpc_queue which handles message sharing and separates it from pubsub.
• Added message size limit
• RPC message transfer similar to go-libp2p

Cute Animal Picture

[sukhman-sukh]

Hey @seetadev,
As per the discussion thread #719, I have shifted our gossipsub write_msg to a message queuing and splitting architecture. Please review this, and is there anything else to add to this PR.
(I am unable to tag luca).

[seetadev]

@sukhman-sukh : This is wonderful. Great work. Reviewing this in detail. Thank you so much for the contribution.

CCing @sumanjeet0012 , @lla-dane and @Winter-Soren , who are also working on different Gossipsub branches. Asking them to review this important PR with me.

@acul71 (Luca) had some github admin issues. They are now resolved and you can definitely tag Luca again :)

[lla-dane]

@sukhman-sukh: I had one question about the splitting logic, if an outgoing RPC is larger than maxMessageSize, it gets split into multiple smaller RPCs. On the receiving side, do we need to handle these fragments as parts of a larger message (i.e reassemble them), or is each split RPC a complete standalone message that can be processed independently?

[sukhman-sukh]

Hey @lla-dane, according to me, every RPC is standalone. We were first piggybacking multiple messages into a single RPC which might exceed it sometime, so now it is taking that big RPC and breaking it into a standalone list of RPCs.

[lla-dane]

Yeah I understand that. But suppose we break a big chunk RPC into individual RPCs, so isn't there a chamce that the content of those 2 smaller RPCs would be dependent, and since our ultimate goal was to somehow deliver the complete content of the big RPC on the receiver end, so the content of the 2 smaller RPCs need to agregated or something on the receiver end.
Is it not the case ?
@sukhman-sukh

[seetadev]

@sukhman-sukh : Hi Sukhman. Appreciate your efforts. Wish if you could resolve the CI/CD issues.

Wish if you could also respond to @lla-dane on his question/comment, whenever you get a chance.

CCing @Winter-Soren and @sumanjeet0012 for their thoughts and feedback too.

[sukhman-sukh]

Yeah I understand that. But suppose we break a big chunk RPC into individual RPCs, so isn't there a chamce that the content of those 2 smaller RPCs would be dependent, and since our ultimate goal was to somehow deliver the complete content of the big RPC on the receiver end, so the content of the 2 smaller RPCs need to agregated or something on the receiver end. Is it not the case ? @sukhman-sukh

Have a look here at go-libp2p impl (https://github.com/libp2p/go-libp2p-pubsub/blob/ed53c170f9a784c45f81d88dd6051218fbd5ece7/pubsub.go#L268).
Basically, acc, to my understanding. Let's say we have a queue of RPCs -> <1,2,3,4> now we first piggyback them into <1234> single RPC and then if size exceeds max then it will split it again into <12,34>. so the order is preserved for the same type of message. And also, afaik publish and then control message always works. If you have any edge cases in mind, please share so that we can improve the algorithm for greater correctness.

[lla-dane]

Oh I see, I was thinking it was splitting a single RPC message into parts. That's good then, now I understood.

So would it possible for a single RPC message to exceed max size ?

[sukhman-sukh]

Oh I see, I was thinking it was splitting a single RPC message into parts. That's good then, now I understood.

So would it possible for a single RPC message to exceed max size ?

Yes, it is possible. In that case:

for rpc in rpc_list:
                    if rpc.ByteSize() > max_message_size:
                        self.drop_rpc(rpc)
                        continue
                    await self.do_send_rpc(rpc, to_peer, urgent)

will be executed and that RPC will be dropped for now.

[sukhman-sukh]

@seetadev, please re run the ci

[seetadev]

@sukhman-sukh : We had some CI/CD performance issues over the last 7 hours.

Re-ran the CI/CD pipeline. We should now get test results soon.