# Protocol Select (aka multistream 2.0 aka multiselect)

## Reflections on the past and future outlook

### What went wrong the last time?

A lot.

While everyone agrees that this is an important problem to solve, there has been significant disagreement about the concrete shape of the solution, especially what's in and what's out of scope. Pair that with poor prioritization and a poorly agreed-upon review process, and you get the mess that's multistream 2.0.

### Can we do better this time?

There's no reason we can't. Protocol Labs has solved far more complex problems.

### Why do we have a new name now?

We use the "multi" prefix for self-describing things. For example, a multihash is a self-describing hash in the sense that it tells you which hashing algorithm was used to derive the hash value.
This doesn't apply to multistream.

### What can we learn from the past?

1. There are **a lot** of optimizations we could build into the protocol. Even the MVP (minimum viable protocol) will be quite complex. We should aim to build the protocol in a way that will allow us to add features / optimizations later in a backwards-compatible way.
1. We need a clear review process with
1. Clearly communicated, binding deadlines.
1. A clear distinction between **blocking** review comments ("we shouldn't specifiy the protocol like this") and **non-blocking** comments ("it would be if ..., but the other way is fine too")
1. This probably needs a lot more thought.
1. We need to define who's in charge of making a final decision.

### Who should drive this?

Undoubtedly, defining Protocol Select is the prerogative of the Stewards Team. Improving the base protocols of the W3DT stack doesn't directly contribute to PMF as e.g. Project Nitro or Bedrock do, so it's probably out of scope for project teams.
It's the stewards who care about the performance and long-term maintainability of the libraries that the entire stack is built on.

## Prior Art

Multistream 1 spec: https://github.com/libp2p/specs/tree/master/connections
Long discussion document: https://docs.google.com/document/d/1jxd39Mvjyk-G1jdvZOoxSsQHtLSNBcYfxNrbjxETCH4
Design Document: https://github.com/libp2p/specs/pull/205
Specification: https://github.com/libp2p/specs/pull/227

### What could we agree on last time?

* Using protobufs can be a convenient way to build an extensible protocol, as protobuf ignores unknown fields.
* We can't negotiate multistream / protocol select itself. If we want to become censorship-resistant, the cryptographic handshake must be the first thing sent on the wire.
* This requires us to advertise the cryptographic protocol as part of the multiaddr (e.g. */ip4/1.2.3.4/tcp/noise/*).
* TCP simultaneous open must still work.
* As in multistream, it needs to be possible to suggest a protocol and optimistically start sending data. If the peer doesn't speak the protocol, it resets the stream.
* open question: When suggesting multiple protocols (possible use case: multiple versions of the DHT protocol), should it be possible to optimistically send data for one (or all?) versions as well?
* Stream Multiplexer negotiation must not assume that one specific peer is able to send data first. Depending on the handshake protocol used (or even the _implementation_ of that handshake protocol), the client or the server could be in the position to send data first (e.g. for TLS: the server can send 0.5-RTT data right after receiving the client's first flight).
* Commonly used protocols should be abbrevitated / compressed, so that we don't have to send the complete protocol string every time we open a new stream.
* We do this by building a "dynamic table", assigning an ID to a protocol name.

### What was contentious? / Open Questions

General:
* Packet Orientation: We've been planning to support packet-oriented for a long time. Does Protocol Select need to be designed with that in mind, or can we afford to punt this until we actually define how a packet transport would work in libp2p?
* How do we know that a peer supports Protocol Select?
* We could use the fact that the multiaddr contains a cryptographic protocol.
* On the other hand, this is making this project more complex. Trial-and-fallback might allow us to split up the multiaddr change into an independent project, and reduce overall complexity.

Stream muxer negotiation:
* Stream muxer advertisement vs. negotiation: Should the stream muxer be part of the multiaddr, or should it be negotiated at the beginning?
* If we decide for negotiation, do we want a `parallel-stream` protocol that allows the optimistic initialization and usage of multiple stream muxers at the same time?
* Other option: allow _one_ "embryonic stream" that allows the exchange of application data while stream muxer negotiation is running?

Abbreviating protocol names:
* Protocol abbreviations: Do we want to be able to ship abbreviation tables, or all abbreviations discovered / learned lazily?
* Who gets to define the ID for a protocol? One can make arguments in both directions:
* The initiator: the initiator will probably know better how frequently it's going to use a certain protocol. Problem: The receiver will have to keep state for each abbreviation.
* The receiver: The receiver is in control of how much state it wants to allocate for the mapping. Problem: It takes 1 round-trip until an abbreviation can be used. Can we live with that?
* Possible solution: allow the initiator to request an ID for a certain protocol. It's the receiver's decision if it honors that request.
* State Synchronization: Do we want to preserve the protocol abbreviation mappings across connections?
* State synchronization is hard. We might encounter tricky edge cases here.

Others:
* Compressors / Erasure Codes / Stream Transformations: Peers might want to apply e.g. gzip or some other arbitrary transformation of the bytes sent on a stream. How is this negotiated?
* List compressors in a dedicated `compressor` option? How do we determine which combinations are valid (are both `/protocol/gzip` and `/gzip/protocol` valid?)
* Advertise `/protocol`, `/gzip/protocol` (`/protocol/gzip`?) and let the peer choose?
* Is it still possible to optimistically send application data in the first flight? If not, applications might be discouraged to trade bandwidth savings for a one round-trip penalty.