Onion messaging support #68
Conversation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Summary of Changes
Hello @gijswijs, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!
This pull request significantly extends the lightning-onion package by integrating full support for onion messaging, a distinct type of onion routing. The changes involve fundamental modifications to how onion packets are structured, created, and processed, enabling greater flexibility in payload sizes and strict adherence to TLV encoding for messages. This enhancement is crucial for future-proofing the library and supporting advanced messaging capabilities within the Lightning Network.
Highlights
- Onion Messaging Support: This pull request introduces comprehensive support for 'onion messaging' within the
lightning-onionpackage, distinguishing it from traditional payment onions. This includes new packet structures, creation, and processing logic tailored for onion messages. - Flexible Routing Info Sizes: The fixed-size
RoutingInfofield inOnionPackethas been replaced with a dynamic slice ([]byte), enabling support for variable payload sizes. New constantsStandardRoutingInfoSize(1300 bytes) andJumboRoutingInfoSize(32768 bytes) are introduced to accommodate different onion message lengths, including those exceeding the previous 1300-byte limit. - Payload Type Enforcement: Onion messages are now strictly enforced to use TLV (Type-Length-Value) payloads, aligning with BOLT 7 specifications. The packet creation and processing logic has been updated to validate this, rejecting legacy payloads for onion messages.
- Updated Packet Creation and Processing: The
NewOnionPacketandProcessOnionPacketfunctions now accept functional options (OnionPacketOption,ProcessOnionOpt) to specify if the packet is an onion message. This allows for specialized handling, such as dynamic routing info sizing and payload type validation, based on the packet's purpose. - New Test Vectors and Comprehensive Testing: New test vectors from the BOLT 4 spec for onion messaging have been integrated. Dedicated tests (
TestBuildOnionMessageBlindedRoute,TestOnionMessageRouteBlinding,TestTLVPayloadMessagePacket) have been added to validate blinded route construction, packet processing, and TLV payload handling for onion messages. Tests for zero-length and larger than 1300-byte payloads are also included. - CLI Tool Enhancement: A new
--onion-messageflag has been added to thecmd/main.goCLI tool, allowing users to generate onion message packets directly. - Dependency Updates: Several Go module dependencies, including
btcd,btcec/v2,stretchr/testify, andgolang.org/x/crypto, have been updated to their latest versions. New indirect dependencies likelnd/tlvhave also been added.
Using Gemini Code Assist
The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.
Invoking Gemini
You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.
| Feature | Command | Description |
|---|---|---|
| Code Review | /gemini review |
Performs a code review for the current pull request in its current state. |
| Pull Request Summary | /gemini summary |
Provides a summary of the current pull request in its current state. |
| Comment | @gemini-code-assist | Responds in comments when explicitly tagged, both in issue comments and review comments. |
| Help | /gemini help |
Displays a list of available commands. |
Customization
To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.
Limitations & Feedback
Gemini Code Assist is currently in preview and may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments to provide feedback.
You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.
Footnotes
-
Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩
|
/gemini review |
There was a problem hiding this comment.
Code Review
This pull request introduces support for onion messaging, including handling of blinded routes and variable-sized payloads. The changes include new logic for packet creation and processing, and are supported by test vectors and new unit tests. I have a few suggestions to improve code clarity and robustness.
This commit removes an unused var and changes bytes.Compare to the idiomatic bytes.Equal.
This commit adds the spec test vector for blinded onion messages. It also adds a test that tests BuildBlindedRoute, decryptBlindedHopData and NextEphemeral against this vector.
We add TestOnionMessageRouteBlinding which verifies that the onion message packet from the test vector can be processed correctly by the nodes in a blinded route.
TestTLVPayloadMessagePacket creates a onion message with payload and the blinded route from the test vector. It then checks if the onion packet we create is equal to the one provided in the test vector.
Since the onion message payload can be zero-length, we need to decode it correctly. This commit adds a boolean flag to the HopPayload Decode that tells whether the payload is an onion message payload or not. If it is, the payload is decoded as a tlv payload also if the first byte is 0x00.
Onion messages allow for payloads that exceed 1300 bytes, in which case the payload should become 32768 bytes. This commit introduces support for those jumbo packets.
This commit adds a helper function to create onion messages of a specified length. This helper is then used to test the handling of packets larger than 1300 bytes specifically for onion messages.
When we are parsing onion messages, we must ensure that a blinding point is provided.
To temporarily use this package from my own repo, we redeclare the package path.
The field MaxPayloadSize is added to the sphinx package to allow for backwards compatibility with the old sphinx package. Removing it would have been a breaking change.
To provide backwards compatibility, the Decode function in sphinx.go has been modified to accept an optional isMessage parameter. This is now not a breaking change.
|
cc: @yyforyongyu @ellemouton for review |
| @@ -106,8 +106,7 @@ func newTestRoute(numHops int) ([]*Router, *PaymentPath, *[]HopData, *OnionPacke | |||
|
|
|||
| func TestBolt4Packet(t *testing.T) { | |||
| var ( | |||
| route PaymentPath | |||
| hopsData []HopData | |||
| route PaymentPath | |||
There was a problem hiding this comment.
nit: the commit msg is a bit off sphinx: remove dead code and
path_test.go
Outdated
| type onionMessageJsonTestCase struct { | ||
| Generate generateOnionMessageData `json:"generate"` | ||
| Route routeOnionMessageData `json:"route"` | ||
| // OnionMessage onionMessageData `json:"onionmessage"` |
| // ID as the next node id in the payload of the previous | ||
| // node, so we get that from the previous hop. | ||
| nodeIDStr, _ := hex.DecodeString( | ||
| currentHop, |
| } | ||
|
|
||
| // First, Dave will build a blinded path from Bob to itself. | ||
| DaveSessKey := privKeyFromString( |
| @@ -1,2 +1,3 @@ | |||
| vendor/ | |||
| .idea | |||
| .aider* | |||
There was a problem hiding this comment.
nit: can move .gitognore and go.mod changes into a new commit
| @@ -162,6 +163,89 @@ func TestBolt4Packet(t *testing.T) { | |||
| } | |||
| } | |||
|
|
|||
| func TestTLVPayloadMessagePacket(t *testing.T) { | |||
| pubKey, err := btcec.ParsePubKey(pubKeyBytes) | ||
| require.NoError(t, err) | ||
|
|
||
| EncryptedRecipientData, err := hex.DecodeString( |
| @@ -1,4 +1,4 @@ | |||
| module github.com/lightningnetwork/lightning-onion | |||
| module github.com/gijswijs/lightning-onion | |||
There was a problem hiding this comment.
I'm assuming this is changed so you can continue the development in lnd right? If that's the case, I would suggest using go.work instead locally, more details here.
Basically if you have the following dir tree struct,
> tree -L 1
.
├── btcsuite
├── lightning-infra
├── lightning-onion
├── lightning-terminal
├── lnd
...You can create a go.work file here,
> tree -L 1
.
├── btcsuite
├── go.work
├── itest_logs
├── itest-db
├── lightning-infra
├── lightning-onion
├── lightning-terminal
├── lnd
...Here's my go.work file,
go 1.23.10
toolchain go1.24.0
use (
./btcsuite/btcd
./btcsuite/btcd/btcec
./btcsuite/btcd/btcutil
./btcsuite/btcd/btcutil/psbt
./btcsuite/btcd/chaincfg/chainhash
./btcsuite/btcwallet
./btcsuite/btcwallet/wallet/txauthor
./btcsuite/btcwallet/wallet/txrules
./btcsuite/btcwallet/wallet/txsizes
./btcsuite/btcwallet/walletdb
./btcsuite/btcwallet/wtxmgr
./falafel
./go_temp
./lightning-onion
./lnd
./lnd/cert
./lnd/clock
./lnd/healthcheck
./lnd/kvdb
./lnd/queue
./lnd/ticker
./lnd/tor
./lnd/fn
./lnd/tlv
./neutrino
./lnd/sqldb
)
You may need to run go work sync and go work vendor to make it work. Once done, any local changes in one package (onion, btcd...) will be reflected and updated in lnd.
Finally say you've updated this lightning-onion package and made a few changes in lnd, and want to create a PR in lnd to check the CI, you can create a temp commit in lnd to update the package in lnd's go.mod file,
// note the space instead of an @
replace host.com/someone/pkg => host.com/you/pkg branch
So in this case we put this line in the end,
replace github.com/lightningnetwork/lightning-onion => github.com/gijswijs/lightning-onion onion-messaging
And run go mod tidy should make it work in lnd.
There was a problem hiding this comment.
Second round done. My main comments are,
- the
legacy vs empty tlv streamcheck, and wonder whether it's needed, and if so, can we do it in a higher layer instead. Also wondering if we could drop the support for legacy payload. - code format needs to be fixed.
- need to make it compile, ran linter and got
> golangci-lint run
cmd/main.go:1: : # github.com/gijswijs/lightning-onion/cmd
cmd/main.go:212:25: undefined: sphinx.OnionPacketOption
cmd/main.go:214:40: undefined: sphinx.WithOnionMessage
cmd/main.go:217:14: cannot use ... in call to non-variadic sphinx.NewOnionPacket (typecheck)
package mainI also noticed that we don't have any CI here, maybe a minimal CI that includes jobs like unit test, lint, and check commits would be a good starting point, and it should be easy to add as we can just copy files from lnd. Another PR tho.
| require.NoError(t, err) | ||
|
|
||
| // Once we have the raw file, we'll unpack it into our | ||
| // blindingJsonTestCase struct defined below. |
There was a problem hiding this comment.
should be onionMessageJsonTestCase
| path := make([]*HopInfo, len(h)) | ||
| currentHop := e | ||
| for i, hop := range h { | ||
| // The json test vector only specifies the current node |
There was a problem hiding this comment.
could we rephrase it a bit? Not sure I follow the sentence...
| currentHop, | ||
| ) | ||
| nodeID, _ := btcec.ParsePubKey(nodeIDStr) | ||
| payload, _ := hex.DecodeString(hop.EncryptedDataTlv) |
There was a problem hiding this comment.
all these errors should be checked via require.NoError(t, err)
| require.NoError(t, err) | ||
|
|
||
| // At this point, Dave will give his blinded path to the Sender who will | ||
| // then build its own blinded route from itself to Bob via Alice. The |
There was a problem hiding this comment.
Q: in other words Bob is the introduction node and Alice is the sender?
| genData := testCase.Generate.Hops[i] | ||
| priv := privKeyFromString(hop.PrivKey) | ||
| ephem := pubKeyFromString( | ||
| genData.EphemeralPubKey, |
| totalPayloadSize := paymentPath.TotalPayloadSize() | ||
|
|
||
| var routingInfoLen int | ||
| maxRoutingInfoErr := ErrStandardRoutingInfoSizeExceeded |
There was a problem hiding this comment.
var maxRoutingInfoErr error
|
|
||
| // Check whether total payload size doesn't exceed the hard maximum. | ||
| if totalPayloadSize > routingInfoLen { | ||
| return nil, maxRoutingInfoErr |
There was a problem hiding this comment.
I would do sth like,
// Assume it's the normal routing.
routingInfoLen := StandardRoutingInfoSize
maxRoutingInfoErr := ErrStandardRoutingInfoSizeExceeded
// Overwrite the values if it's an onion message and the payload
// exceeds the normal size.
if cfg.isOnionMessage && totalPayloadSize > StandardRoutingInfoSize {
routingInfoLen = JumboRoutingInfoSize
maxRoutingInfoErr = ErrJumboRoutingInfoSizeExceeded
}Or even simpler,
if totalPayloadSize > StandardRoutingInfoSize {
if !cfg.isOnionMessage {
return nil, ErrStandardRoutingInfoSizeExceeded
}
if totalPayloadSize > JumboRoutingInfoSize {
return nil, ErrJumboRoutingInfoSizeExceeded
}
}And create a dedicated method, sth like validatePayloadSize to perform this check.
|
|
||
| // If this is an onion message, we only expect TLV | ||
| // payloads. | ||
| if cfg.isOnionMessage && isLegacy { |
There was a problem hiding this comment.
Think this legacy vs empty tlv kinda increases the complexity a lot - I wonder if we need to check it here, like a higher layer should be able to catch and error out there?
| @@ -322,7 +372,8 @@ func rightShift(slice []byte, num int) { | |||
| // leaving only the original "filler" bytes produced by this function at the | |||
| // last hop. Using this methodology, the size of the field stays constant at | |||
| // each hop. | |||
| func generateHeaderPadding(key string, path *PaymentPath, sharedSecrets []Hash256) []byte { | |||
| func generateHeaderPadding(key string, path *PaymentPath, | |||
| sharedSecrets []Hash256, routingInfoLen int) []byte { | |||
| // header. The MaxPayloadSize bytes at the end are used to | ||
| // encrypt/decrypt the fillers when processing the packet of generating | ||
| // the HMACs when creating the packet. | ||
| numStreamBytes = routingInfoSize * 2 |
There was a problem hiding this comment.
I really like the docs here, and wonder if we could somehow retain it, like make numStreamBytes a method that takes routingInfoLen and returns it, otherwise I see routingInfoLen * 2 and will be wondering why the *2
|
/gemini review |
There was a problem hiding this comment.
Code Review
This pull request introduces comprehensive support for onion messaging, including blinded routes and variable-sized onion packets, which is a significant feature addition. The implementation is well-supported by new tests, including vectors from the BOLT specification.
My review focuses on enhancing the code quality. I've suggested improvements to test robustness by replacing ignored errors with explicit checks, and a simplification of some conditional logic to improve readability and maintainability. These suggestions align with general Go best practices for writing clear and robust code.
| nodeIDStr, _ := hex.DecodeString( | ||
| currentHop, | ||
| ) | ||
| nodeID, _ := btcec.ParsePubKey(nodeIDStr) | ||
| payload, _ := hex.DecodeString(hop.EncryptedDataTlv) |
There was a problem hiding this comment.
While ignoring errors with _ is common in tests for brevity, it can mask issues with test data or underlying functions. Using require.NoError(t, err) from the testify package would make the test more robust by failing immediately with a clear message if an error occurs. This is a good practice for ensuring test quality and reliability.
nodeIDStr, err := hex.DecodeString(
currentHop,
)
require.NoError(t, err)
nodeID, err := btcec.ParsePubKey(nodeIDStr)
require.NoError(t, err)
payload, err := hex.DecodeString(hop.EncryptedDataTlv)
require.NoError(t, err)| var routingInfoLen int | ||
| maxRoutingInfoErr := ErrStandardRoutingInfoSizeExceeded | ||
| if cfg.isOnionMessage { | ||
| switch { | ||
| case totalPayloadSize <= StandardRoutingInfoSize: | ||
| routingInfoLen = StandardRoutingInfoSize | ||
| default: | ||
| routingInfoLen = JumboRoutingInfoSize | ||
| maxRoutingInfoErr = ErrJumboRoutingInfoSizeExceeded | ||
| } | ||
| } else { | ||
| routingInfoLen = StandardRoutingInfoSize | ||
| } |
There was a problem hiding this comment.
The logic to determine routingInfoLen can be simplified for better readability and conciseness. The current implementation has some redundancy. By setting a default value and then conditionally overriding it for the jumbo case, the code becomes more direct and easier to follow.
routingInfoLen := StandardRoutingInfoSize
maxRoutingInfoErr := ErrStandardRoutingInfoSizeExceeded
if cfg.isOnionMessage && totalPayloadSize > StandardRoutingInfoSize {
routingInfoLen = JumboRoutingInfoSize
maxRoutingInfoErr = ErrJumboRoutingInfoSizeExceeded
}
This PR adds support for onion messaging to the
lightning-onionpackage. It does the following things:TestBuildOnionMessageBlindedRoutethat tests BuildBlindedRoute, decryptBlindedHopData and NextEphemeral against this vector.TestOnionMessageRouteBlindingwhich verifies that the onion message packet from the test vector can be processed correctly by the nodes in a blinded route.TestTLVPayloadMessagePacketthat creates a onion message with payload and the blinded route from the test vector.It also adds support for zero-length payloads and longer than 1300 bytes payloads that are specifically supported by the spec, but not tested by the test vectors supplied. Both features are accompanied by their own unit tests.