Skip to content
This repository was archived by the owner on Jul 18, 2025. It is now read-only.

Commit 403a80b

Browse files
dmcgowandmcgowan
committed
Protocol 0.2.0
Update to the protocol as a result of libchan meeting with Matteo Collina. Bump the version to 0.2.0 and name the set the previous version as 0.1.0. Protocol changes: - Define stream provider to support multiple multiplexing protocols - Support CBOR in addition to Msgpack for channel message encoding - Add extended type codes definition - Require byte-streams to send *"libchan-parent-ref"* - Allow byte-streams as duplex or half-duplex - Add channel synchronization through ack definition - Add channel errors - Update description of relationship to Go channels Other changes: - Add Derek and Matteo to authors - Reformatted to 80 character lines - Much cleanup and rewording Signed-off-by: Derek McGowan <[email protected]> (github: dmcgowan)
1 parent d5db941 commit 403a80b

File tree

1 file changed

+219
-106
lines changed

1 file changed

+219
-106
lines changed

‎PROTOCOL.md‎

Lines changed: 219 additions & 106 deletions
Original file line numberDiff line numberDiff line change
@@ -2,179 +2,292 @@
22

33
Extreme portability is a key design goal of libchan.
44

5-
This document specifies the libchan protocol to allow multiple implementations to co-exist with
6-
full interoperability.
5+
This document specifies the libchan protocol to allow multiple implementations
6+
to co-exist with full interoperability.
77

88
## Version
99

10-
No version yet.
10+
0.2.0
1111

12-
## Author
12+
## Authors
1313

1414
Solomon Hykes <[email protected]>
15+
Derek McGowan <[email protected]>
16+
Matteo Collina <[email protected]>
1517

1618
## Status
1719

18-
This specification is still work in progress. Things will change, probably in reverse-incompatible ways.
19-
We hope to reach full API stability soon.
20+
This specification is nearing a stable release. The protocol still may change in
21+
reverse-incompatible ways.
2022

2123
## Terminology
2224

2325
### Channel
2426

25-
A `channel` is an object which allows 2 concurrent programs to communicate with each other. The semantics
26-
of a libchan channel are very similar (but not identical) to those of Go's native channels.
27+
A `channel` is an object which allows 2 concurrent programs to communicate with
28+
each other. The semantics of a libchan channel are very similar (but not
29+
identical) to those of Go's native channels. A channel may be used
30+
synchronously, but do not support synchronization primitives such as Go's
31+
channel select semantics.
2732

28-
A channel has 2 ends: a `Sender` end and a `Receiver` end. The Sender can send messages and close the channel.
29-
The Receiver can receive messages. Messages arrive in the same order they were sent.
33+
A channel has 2 ends: a `Sender` end and a `Receiver` end. The Sender can send
34+
messages and close the channel. The Receiver can receive messages. Messages
35+
arrive in the same order they were sent.
3036

31-
A channel is uni-directional: messages can only flow in one direction. So channels are more similar to pipes
32-
than to sockets.
37+
A channel is uni-directional: messages can only flow in one direction. So
38+
channels are more similar to pipes than to sockets.
3339

3440
### Message
3541

36-
A message is a discrete packet of data which can be sent on a channel. Messages are structured into multiple
37-
fields. The protocol defines which data types can be carried by a message, and how transports should encode and
38-
decode them.
42+
A message is a discrete packet of data which can be sent on a channel. The
43+
protocol defines which data types can be carried sent as a message, and how
44+
transports should encode and decode them. A message is similar to a JSON object
45+
containing [custom types](#custom-types) to represent channels and byte streams.
3946

4047
### Byte stream
4148

42-
A byte stream is an object which implements raw IO with `Read`, `Write` and `Close` methods.
43-
Typical byte streams are text files, network sockets, memory buffers, pipes, and so on.
49+
A byte stream is an object which implements raw IO with `Read`, `Write` and
50+
`Close` methods. Typical byte streams are text files, network sockets, memory
51+
buffers, pipes, and so on. A byte stream may either be read only, write only, or
52+
full duplex.
4453

45-
One distinct characteristic of libchan is that it can encode byte streams as first class fields
46-
in a message, alongside more basic types like integers or strings.
54+
One distinct characteristic of libchan is that it can encode byte streams as
55+
first class fields in a message, alongside more basic types like integers or
56+
strings.
4757

4858
### Nesting
4959

50-
Libchan supports nesting. This means that a libchan message can include a channel, which itself
51-
can be used to send and receive messages, and so on all the way down.
60+
Libchan supports nesting. This means that a libchan message can include a
61+
channel, which itself can be used to send and receive messages, and so on all
62+
the way down.
5263

5364
Nesting is a fundamental property of libchan.
5465

5566
## Underlying transport
5667

57-
The libchan protocol requires a reliable, 2-way byte stream as a transport.
58-
The most popular options are TCP connections, unix stream sockets and TLS sessions.
68+
The libchan protocol requires a reliable, 2-way byte stream with support for
69+
multiplexing as a transport. The underlying byte stream protocol is abstracted
70+
to the libchan protocol through a simple multiplexed stream interface which may
71+
use SPDY/3.1, HTTP/2, or SSH over over TCP connections, unix stream sockets and
72+
TLS sessions.
5973

60-
It is also possible to use websocket as an underlying transport, which allows exposing
61-
a libchan endpoint at an HTTP1 url.
74+
When a reliable stream transport is not available but a non-multiplexed
75+
connection is available, a multiplexing protocol (such as SPDY or another
76+
simple multiplexing protocol) may be done on top of the existing connection.
77+
This also makes using websockets as an underlying byte stream transport
78+
possible, which allows exposing a libchan endpoint at an HTTP/1 url.
6279

6380
## Authentication and encryption
6481

65-
Libchan can optionally use TLS to authenticate and encrypt communications. After the initial
66-
handshake and protocol negotiation, the TLS session is simply used as the transport for
67-
the libchan wire protocol.
68-
69-
## Wire protocol
70-
71-
Libchan uses SPDY (protocol draft 3) as its wire protocol, with no modification.
82+
Libchan can optionally use TLS to authenticate and encrypt communications. After
83+
the initial handshake and protocol negotiation, the TLS session is simply used
84+
as the transport for the libchan multiplexed stream provider.
85+
86+
## Stream Provider
87+
88+
Libchan uses a stream provider to establish new channels and byte streams over
89+
an underlying byte stream. The stream provider must be able to send
90+
headers when creating new streams and retrieve headers for remotely created
91+
streams.
92+
93+
### Headers
94+
The stream provider must support sending key-value headers on stream creation.
95+
96+
- *"libchan-ref"* - String representation of a unique 64 bit integer identifier
97+
for the established stream.
98+
- *"libchan-parent-ref"* String representation of a unique 64 bit integer
99+
identifier for parent of the established stream. (see *"Sending nested
100+
channels"* and *"Sending byte streams"*)
101+
102+
### Streams
103+
The stream provider provides the functionality for creating new streams as
104+
well as accepting streams created remotely. A stream is create with
105+
a set of headers and an accepted stream has a method for returning the
106+
headers. Closing a stream must put the stream in a half-closed state and
107+
not allow anymore data to be written. If the remote side has already
108+
closed, the stream is fully closed. Reseting a stream forces the stream
109+
into a fully closed state and should only be used in error cases.
110+
Resetting does not give the remote a chance to finish sending data and
111+
cleanly close.
112+
113+
## Stream identifiers
114+
Libchan creates a unique identifier for every stream created by the stream
115+
provider. The identifiers are integer values and should never be reused.
116+
The identifier is only unique to a given endpoint, meaning both sides of a
117+
connection may have the same identifier for two different streams. The
118+
identifiers received from the remote endpoint should only be used to reference
119+
streams from that endpoint, and never streams created locally. A remote
120+
endpoint's stream identifier should never be sent in a libchan message.
121+
To send a stream created remotely, a new stream should be created
122+
locally, copied from the remote stream, and the identifier to the local copy
123+
should be used.
72124

73125
## Control protocol
74126

75-
Once 2 libchan endpoints have established a SPDY session, they communicate with the following
76-
control protocol.
127+
Once 2 libchan endpoints have established a multiplexed stream session, they
128+
communicate with the following control protocol.
77129

78130
### Top-level channels
79131

80-
Each SPDY session may carry multiple concurrent channels, in both directions, using standard
81-
SPDY framing and stream multiplexing. Each libchan channel is implemented by an underlying
82-
SPDY stream.
132+
Each libchan session may carry multiple concurrent channels, in both directions,
133+
using stream multiplexing. Each libchan channel is implemented by an underlying
134+
stream.
83135

84-
To use a SPDY session, either endpoint may initiate new channels, wait for its peer to
85-
initiate new channels, or a combination of both. Channels initiated in this way are called
86-
*top-level channels*.
136+
To use a libchan session, either endpoint may initiate new channels, wait for
137+
its peer to initiate new channels, or a combination of both. Channels initiated
138+
in this way are called *top-level channels*.
87139

88-
* To initiate a new top-level channel, either endpoint may initiate a new SPDY stream, then
89-
start sending messages to it (see *"sending messages"*).
140+
* To initiate a new top-level channel, either endpoint may initiate a new
141+
stream, then start sending messages to it (see *"sending messages"*).
90142

91-
* The endpoint initiating a top-level channel MAY NOT allow the application to receive messages
92-
from it and MUST ignore inbound messages received on that stream.
143+
* The endpoint initiating a top-level channel MAY NOT allow the application to
144+
receive messages from it and MUST interpret inbound messages received on that
145+
stream as an ack or error message.
93146

94-
* When an endpoint receives a new inbound SPDY stream, and the initial headers DO NOT include
95-
the key `libchan-ref`, it MUST queue a new `Receiver` channel to pass to the application.
147+
* The endpoint initiating the channel must create a unique identifier for the
148+
channel and include the value in the *"libchan-ref"* header when creating
149+
the new stream.
96150

97-
* The endpoint receiving a top-level channel MAY NOT allow the application to send messages to
98-
it.
151+
* When an endpoint receives a new stream without the header
152+
*"libchan-parent-ref"*, it MUST interpret the stream as an inbound top-level
153+
channel and queue a new `Receiver` channel to pass to the application.
99154

155+
* The endpoint receiving a top-level channel MAY NOT allow the application to
156+
send messages to it.
100157

101-
### Sending messages on a channel
102158

103-
Once a SPDY stream is initiated, it can be used as a channel, with the initiating endpoint holding
104-
the `Sender` end of the channel, and the recipient endpoint holding the `Receiver` end.
159+
### Sending messages on a channel
105160

106-
* To send a message, the sender MUST encode it using the [msgpack](https://msgpack.org) encoding format, and
107-
send a single data frame on the corresponding SPDY stream, with the encoded message as the exact content of
108-
the frame.
161+
Once a stream is initiated, it can be used as a channel, with the initiating
162+
endpoint holding the `Sender` end of the channel, and the recipient endpoint
163+
holding the `Receiver` end.
109164

110-
* When receiving a data frame on any active SPDY stream, the receiver MUST decode it using msgpack. If
111-
the decoding fails, the receiver MUST close the underlying stream, and future calls to `Receive` on that
112-
channel MUST return an error.
165+
* To send a message, the sender MUST encode it using the message encoding format
166+
(see *"message encoding"*), and send the encoded message on the corresponding
167+
stream.
113168

114-
* A valid msgpack decode operation with leftover trailing or leading data is considered an *invalid* msgpack
115-
decode operation, and MUST yield the corresponding error.
169+
* When receiving a data on any active stream, the receiver MUST decode it using
170+
the same message encoding format. If the decoding fails, the receiver MUST close
171+
the underlying stream, and future calls to `Receive` on that channel MUST return
172+
an error.
116173

117-
### Closing a channel
174+
* Every send message should have a corresponding receive of an ack message from
175+
the peer. The ack message is a map with at least one field named `code`. The
176+
`code` field should have an integer value, with an a value of zero considered
177+
a successful ack and non-zero as an error. An error should be accompanied with
178+
an additional `message` field of type string, describing the error. If an error
179+
is received, the sender should close and pass an error to the application.
118180

119-
The endpoint which initiated a channel MAY close it by closing the underlying SPDY stream.
181+
### Sending nested channels
120182

121-
*FIXME: provide more details*
183+
* When sending a nested channel, in addition to the *"libchan-ref"* header, the
184+
*"libchan-parent-ref"* header must be sent identifying the channel used to
185+
create the nested channel.
122186

123-
### Sending byte streams
187+
### Closing a channel
124188

125-
Libchan messages support a special type called *byte streams*. Unlike regular types like integers or strings,
126-
byte streams are not fully encoded in the message. Instead, the message encodes a *reference* which allows
127-
the receiving endpoint to reconstitute the byte stream after receiving the message, and pass it to the
128-
application.
189+
The endpoint which holds the send side of a channel MAY close it which will
190+
half-close the stream. The receive side should respond by closing the stream,
191+
putting the stream in a fully closed state. Any send or receive call from the
192+
application after close should return an error.
129193

130-
*FIXME: specify use of msgpack extended types to encode byte streams*
194+
When an error is received on a channel, the underlying stream should be
195+
closed by both ends.
131196

132-
Libchan supports 2 methods for sending byte streams: a default method which is supported on all transports,
133-
and an optional method which requires unix stream sockets. All implementations MUST support both methods.
197+
### Sending byte streams
134198

135-
#### Default method: SPDY streams
199+
Libchan messages support a special type called *byte streams*. Unlike regular
200+
types like integers or strings, byte streams are not fully encoded in the
201+
message. Instead, the message encodes a *reference* which allows the receiving
202+
endpoint to reconstitute the byte stream after receiving the message, and pass
203+
it to the application.
136204

137-
The primary method for sending a byte stream is to send it over a SPDY stream, with the following protocol:
205+
Byte streams use the raw stream returned by the stream provider.
138206

139-
* When encoding a message including 1 or more byte stream values, the sender MUST assign to each value
140-
an identifier unique to the session, and store these identifiers for future use.
207+
* When encoding a message including 1 or more byte stream values, the sender
208+
MUST assign to each value an identifier unique to the session, and store these
209+
identifiers for future use.
141210

142-
* After sending the encoded message, the sender MUST initiate 1 new SPDY stream for each byte stream value
143-
in the message.
211+
* After sending the encoded message, the sender MUST create 1 new stream for
212+
each byte stream value in the message.
144213

145-
* Each of those SPDY stream MUST include an initial header with as a key the string "*libchan-ref*", and
146-
as a value the identifier of the corresponding byte stream.
214+
* Each of new stream MUST include a header with the key *"libchan-ref"* and
215+
a value of the identifier of the corresponding byte stream. It must also
216+
include a header with the key *"libchan-parent-ref"* and a value of the
217+
stream identifier for the message channel which created the byte stream.
147218

148219
Conversely, the receiver must follow this protocol:
149220

150-
* When decoding a message including 1 or more byte stream values, the receiver MUST store the unique identifier
151-
of each value in a session-wide table of pending byte streams. It MAY then immediately pass the decoded message to the application.
152-
153-
* The sender SHOULD cap the size of its pending byte streams table to a reasonable value. It MAY make that value
154-
configurable by the application. If it receives a message with 1 or more byte stream references, and the table
155-
is full, the sender MAY suspend processing of the message until there is room in the table.
156-
157-
* When receiving new SPDY streams which include the header key "*libchan-ref*", the receiver MUST lookup that
158-
header value in the table of pending byte streams. If the value is registered in the table, that SPDY stream
159-
MUST be passed to the application.
160-
161-
On either end, once the SPDY stream for a byte-stream value is established, it MUST be exposed to the application
162-
as follows:
163-
164-
* After sending each of those SPDY streams, each write operation by the application to a byte-stream field MUST
165-
trigger the sending of a single data frame on the corresponding SPDY stream.
166-
167-
* Each read operation by the application from a byte-stream field MUST yield the content of the next
168-
data frame received on the corresponding SPDY stream. If the reading end of the SPDY stream is closed,
169-
the read operation MUST yield EOF.
170-
171-
* A close operation by the application on the a byte-stream field MUST trigger the closing of the writing end
172-
of the corresponding SPDY stream.
173-
174-
#### Optional method: file descriptor passing
221+
* When decoding a message including 1 or more byte stream values, the receiver
222+
MUST store the unique identifier of each value in a session-wide table of
223+
pending byte streams. It MAY then immediately pass the decoded message to the
224+
application.
175225

176-
*FIXME*
226+
* The sender SHOULD cap the size of its pending byte streams table to a
227+
reasonable value. It MAY make that value configurable by the application. If it
228+
receives a message with 1 or more byte stream references, and the table
229+
is full, the sender MAY suspend processing of the message until there is room in
230+
the table.
177231

178-
### Sending nested channels
232+
* When receiving new streams which include the header key "*libchan-ref*", the
233+
receiver MUST lookup that header value in the table of pending byte streams. If
234+
the value is registered in the table, that stream MUST be passed to the
235+
application.
179236

180-
*FIXME*
237+
On either end, once the stream for a byte-stream value is established, it MUST
238+
be exposed to the application as follows:
239+
240+
* After sending each of those streams, each write operation by the application
241+
to a byte-stream field MUST trigger the sending of a single data frame on the
242+
corresponding stream.
243+
244+
* Each read operation by the application from a byte-stream field MUST yield
245+
the content of the next data frame received on the corresponding stream. If the
246+
reading end of the stream is closed, the read operation MUST yield EOF.
247+
248+
* A close operation by the application on the a byte-stream field MUST trigger
249+
the closing of the writing end of the corresponding SPDY stream.
250+
251+
## Message encoding
252+
A message may be any type which supported by the libchan encoder. A libchan
253+
message encoder must support encoding raw byte stream types as well as channels.
254+
In addition to the libchan data types, time must also be encoded as a custom
255+
type to increase portability of the protocol.
256+
257+
Currently supported message encoders are msgpack5 and soon CBOR.
258+
259+
### Custom Types
260+
Each custom type defines a type code and the byte layout to represent that type.
261+
Directions of descriptions are from the point of view of the endpoint encoding.
262+
All multi-byte integers are encoded big endian. The length of bytes of the
263+
encoded value will be provided by the encoding format, allowing integer values
264+
to be variable length.
265+
266+
| Type | Code | Byte Layout|
267+
|---|---|---|
268+
| Duplex Byte Stream | 1 | 4 or 8 byte integer identifier |
269+
| Inbound Byte Stream | 2 | 4 or 8 byte integer identifier |
270+
| Outbound Byte Stream | 3 | 4 or 8 byte integer identifier |
271+
| Inbound channel | 4 | 4 or 8 byte integer identifier |
272+
| Outbound channel | 5 | 4 or 8 byte integer identifier |
273+
| time | 6 | 8 byte integer seconds + 4 byte integer nanoseconds |
274+
275+
## Version History
276+
277+
0.2.0
278+
- Define stream provider to support multiple multiplexing protocols
279+
- Support CBOR in addition to Msgpack for channel message encoding
280+
- Add extended type codes definition
281+
- Require byte-streams to send *"libchan-parent-ref"*
282+
- Allow byte-streams as duplex or half-duplex
283+
- Add channel synchronization through ack definition
284+
- Add channel errors
285+
- Update description of relationship to Go channels
286+
287+
0.1.0
288+
- Initial specification
289+
- Message channels
290+
- Nested message channels
291+
- Duplex byte streams
292+
- Msgpack channel message encoding
293+
- SPDY stream multiplexing

0 commit comments

Comments
 (0)