WIP: libp2p-record module in reference with go-libp2p-record

[lla-dane]

WIP: The libp2p-record module for py-libp2p in reference with the go-libp2p-record repo.

Reference: https://github.com/libp2p/go-libp2p-record

[lla-dane]

@seetadev: The go-libp2p-record repo: https://github.com/libp2p/go-libp2p-record. has been reflected on the py-libp2p side with all the interfaces. ad860c0 is only about the go-libp2p-record repo, will included tests, and would be ready for a separate review and feedback.

The next step would be integrating the records routing and management in the services. Wish to confirm that now I have to look into these 3 protocols right? to integrate records:

• Kademlia DHT — for storing and retrieving key-value records, such as IPNS entries.
• IPNS (InterPlanetary Naming System) — for publishing and resolving mutable names.
• PubSub message signing and validation — for signed messages and peer validation.

[seetadev]

@lla-dane : Absolutely — you're on the right track. Thank you for sharing the updates.

Yes, please go ahead and start with Kademlia DHT integration first. This will be the foundational layer for key-value record storage and retrieval, including IPNS entries, and will naturally set the stage for the subsequent integration of IPNS and PubSub features.

Once Kademlia DHT is implemented and tested, the next steps would be:

1. PubSub message signing and validation – focusing on signing messages using libp2p-record and validating them appropriately at the receiving end, as per the spec.
2. IPNS – leveraging both DHT and record signing to enable publishing and resolving mutable names over the network.

Each of these steps builds progressively on the prior one, so the current plan and sequencing make perfect sense.

Also, your work in ad860c0 mapping out the interfaces from go-libp2p-record is much appreciated — please do include tests and push it for a dedicated review when ready. Looking forward to seeing the DHT integration next! Let us know if you need any pointers from the go/js reference implementations as you move forward.

CCing @sumanjeet0012 and @acul71 for their inputs and feedback as you proceed with kad-dht integration.

[lla-dane]

@seetadev: I explored integrating libp2p-record module in kad-dht just like the way, it is done in go-libp2p. I have made the following changes till now in kad_dht:

1. The Record schema was already in libp2p.kad_dht.pb.kademilia.proto so there is no need to create an extra Record.proto schema in libp2p.records module as it is done in go-libp2p.

2. Shifted some of the default config variables like the below from routing_table.py to common.py.

# Default parameters
BUCKET_SIZE = 20  # k in the Kademlia paper
MAXIMUM_BUCKETS = 256  # Maximum number of buckets (for 256-bit keys)
PEER_REFRESH_INTERVAL = 60  # Interval to refresh peers in seconds
STALE_PEER_THRESHOLD = 3600  # Time in seconds after which a peer is considered stale

3. Updated the kad_dht constructor function to this:

    def __init__(
        self,
        host: IHost,
        mode: DHTMode,
        validator: NamespacedValidator | None = None,
        validator_changed: bool = False,
        protocol_prefix: TProtocol = PROTOCOL_PREFIX,
        enable_providers: bool = True,
        enable_values: bool = True,

in the image of go-libp2p-kad-dht Config constructor.
Reference: https://github.com/libp2p/go-libp2p-kad-dht/blob/0532e1ca748c9ad45f39ed07fb3f74bcf64a3b9d/internal/config/config.go#L39C1-L56C1

4. Added functions:
   • apply_fallbacks: Aims to add the default pk and ipns validators if they are missing and the default validator set hasn't been overidden. Reference apply_fallback from go-libp2p
   • validate_config: This validates the kad_dht configuration to check if it complies with the /ipfs protocol-prefix. Referance: validate-config from go-libp2p
   • set_validator: Add a custom validator for our kad-dht
   • set_namespace_validator: Add a custom namespace_validator for our kad_dht.

Adding a few inline comments also

[lla-dane]

Added this code in the put_value function to validate the key/value pair that we receive in the put_value function's params. Reference from go-libp2p: PutValue

So what happens in go-libp2p PutValue function is we receive the key/value and key: str is in type string. and then we pass the key/value to the respective namespace_validator to get it validated.

So now the problem is, in py-libp2p side in put_value function, we receive key: bytes in bytes format which is different from go-libp2p.

So I would like some pointers on how to proceed @seetadev @sumanjeet0012

[lla-dane]

There is also a difference in the code design in the kad-dht utilities in py and go:

In go libp2p, we store the key/value pair in the Datastore in the form of Records{key, value, timeRecieved}, but on the py-libp2p side we store the key/value pair in this way:

    def put(self, key: bytes, value: bytes, validity: float = 0.0) -> None:
        ...
        self.store[key] = (value, validity)
        ...

Simply storing the value and validity corresponding to the key. But in the go-libp2p side we are storing like this:

    rec := record.MakePutRecord(key, value)
    rec.TimeReceived = internal.FormatRFC3339(time.Now())
    err = dht.putLocal(ctx, key, rec)
    if err != nil {
        return err
    }

Creating a record with key/value pair and then storing it in the local in the form of record corresponding to the key.

So this is the difference in code design. So what should I do, shall I refactor the py side as per the go-libp2p??

@seetadev

[acul71]

@lla-dane
Failed test:
/home/runner/work/py-libp2p/py-libp2p/docs/libp2p.records.rst: WARNING: document isn't included in any toctree
Just include the document in a doctree

[lla-dane]

This PR is continued at #890