[epic]: Implement Onion Messaging (BOLT4) and Support Downstream BOLT12 Features

[gijswijs]

Background

This issue tracks the implementation of Onion Messaging (BOLT4) in lnd and related changes needed for future BOLT12 features (e.g., offers, invoice requests, invoices and invoice errors). Onion Messaging provides a protocol for sending and receiving general-purpose onion-encrypted messages across the Lightning Network, independent of HTLC payments.

Stage 1 - Forwarding Onion Messaging

The first step towards adoption of onion messaging is relaying onion messages sent by others. The node will announce the feature bit that indicates the node can forward onion messages (38/39), and the message type for onion messages (513).

Stage 1 is divided over 4 PRs.

#9868: The first pull request introduces the basic structures for onion messages into LND. It adds a new message type, OnionMessage, to the lnwire package. Additionally, it exposes functionality for handling these messages via new endpoints: SendOnionMessageand SubscribeOnionMessages.

lightning-onion Onion messaging support: This PR introduces support for onion messages in the lightning-onion package. This involves adding extensive tests using spec test vectors for creating, concatenating, and processing blinded onion routes. The code is updated to handle custom-sized onion packets, allowing for larger payloads typical of onion messages. Additionally, a change was made to correctly decode zero-length TLV payloads, which can occur in onion messages.

#10089: This PR implements the core onion message forwarding logic. It defines the structure of the onion's inner payload and updates the htlcswitch to decrypt and process it. The OnionEndpoint is refactored to handle these payloads: if the node is not the final destination, it decrypts its layer and forwards the message to the next hop. Finally, RPCs and integration tests are updated to support this new forwarding capability. This PR also depends on the lightning-onion PR

#10219: This PR makes onion message handling asynchronous to improve performance. It introduces a new BackpressureQueue to manage high message volumes by randomly dropping messages when the queue is nearly full. Incoming onion messages are now added to this queue instead of being processed immediately, with a dedicated background goroutine handling decryption and forwarding. This prevents high traffic from blocking peer communication and enhances node stability. This PR also depends on #9838.

Stage 2 - Path Finding for Onion Messages

A dedicated pathfinding functionality for onion messages is a necessary precursor for implementing BOLT 12 features.

1. Graph-Based Pathfinding:

   • Implement a new pathfinding algorithm that searches the channel graph for a route to a destination node.
   • This search would filter for nodes that have advertised the OnionMessages feature bit, ensuring all hops can forward the message.
   • Unlike HTLC pathfinding, this search would focus only on connectivity and not on channel capacity or fees.

2. RPC and API Modification:

   • The existing SendOnionMessage RPC would be updated to take a final destination public key instead of just a direct peer.
   • Internally, this would trigger the new pathfinding logic to find a route to that destination.

3. Fallback to Direct Connection:

   • If the pathfinding logic fails to find a suitable multi-hop route, the system would implement the fallback behavior : attempt to establish a direct connection with the destination peer to deliver the message.

Stage 3 - Offer Creation and Invoice Request Handling

• Implement the logic to create and store BOLT 12 offers.
• Add the ability to construct blinded paths to the local node for inclusion in offers.
• Expose this functionality via new RPC endpoints for creating and managing offers.
• Develop the application logic to handle incoming onion messages that contain invoice_request payloads.
• Validating the request against a stored offer, generating a corresponding BOLT 12 invoice, and sending it back using the onion message reply_path.