# Infra how-to

[ToC]

## Node types

We can split nodes in two category:
- Messaging nodes: `peer` and `replication`
- Routing nodes: `relay`, `bootstrap` and `rdvp`

Messaging nodes can send messages:
- `peer` is a full Berty user that can register to a group by itself, send contact request, read and write messages on a group.
- `replication` is a server with an API allowing a `peer` to add it to a group. It will only provide high availability by forwarding messages on a group to peers without being able to read them.

Routing nodes allow Messaging nodes to connect to each other:
- Network:
- `relay` will provide NAT Traversal to a peer that can't accept inbound connection using a TURN-like protocol (see [libp2p doc](https://docs.libp2p.io/concepts/circuit-relay/))
- Discovery
- `bootstrap` will exchange peers addresses with any peer that connect to it, useful for DHT
- `rdvp` a kind of key/value store where any peer can register its address under an arbitrary name (e.g `Set({foo: <ip-address>})` and `Get(foo) // return <ip-address`)

## Instanciation flow

There is an order to instantiate the different node types, because `peer` and `replication` types (Messaging nodes) needs to know the addresses of the `relay`, `bootstrap` and `rdvp` types (Routing nodes).

So the order in which the different types of nodes should be instantiated is:
1. `relay` / `rdvp` / `bootstrap` (Routing nodes)
3. `replication` / `peer` (Messaging nodes)

Knowing also that after reflection on our side, the simplest for you would be to create the Berty groups (in the sense protocol / conversation group) and to add the replication servers to them via a temporary "setup daemon". This will require to implement a go program on your side talking with the daemon gRPC API (see [Daemon gRPC API](#Daemon-gRPC-API)).

So the complete flow should be:
1. instanciate `relay`, `rdvp` and `boostrap` -> save their addresses
2. instanciate `replication` (specify `relay`, `bootstrap` and `rdvp` addresses if needed)
3. create groups and add `replication` to them using a setup daemon -> save group invitations / kill setup daemon
4. instanciate `peer` (with group invitations and specify `relay`, `bootstrap` and `rdvp` addresses if needed)
5. run tests by sending gRPC calls to `peer`

## Daemon gRPC API

When you run a daemon using `go run ./go/cmd/berty daemon`, it will by default listen on `localhost:9091` and provide protocol and messenger APIs.

You can specify a custom listener using `-node.listeners` flag.

You can check the API protos in `api/protocoltypes` and `api/messengertypes`.

For example, to create a group and print an invit using a running daemon listening on default address, you will have to mix both API:

package main

import (
	"context"
	"fmt"

	"berty.tech/berty/v2/go/pkg/messengertypes"
	"berty.tech/berty/v2/go/pkg/protocoltypes"

	"google.golang.org/grpc"
)

func main() {
	ctx := context.Background()
	host := "127.0.0.1:9091" // default daemon gRPC listener

	// Init gRPC services (protocol and messenger)
	cc, err := grpc.DialContext(ctx, host, grpc.FailOnNonTempDialError(true), grpc.WithInsecure())
	if err != nil {
		panic(err)
	}

	protocol := protocoltypes.NewProtocolServiceClient(cc)
	messenger := messengertypes.NewMessengerServiceClient(cc)

	// Create group using protocol
	resCreate, err := protocol.MultiMemberGroupCreate(ctx, &protocoltypes.MultiMemberGroupCreate_Request{})
	if err != nil {
		panic(err)
	}

	// Print group invit on stdout using messenger
	resInvit, err := messenger.ShareableBertyGroup(ctx, &messengertypes.ShareableBertyGroup_Request{
		GroupPK:   resCreate.GroupPK,
		GroupName: "foo",
	})
	if err != nil {
		panic(err)
	}

	fmt.Println(resInvit.GetWebURL())
}

You can find a more detailled example [here](https://github.com/gfanton/berty/blob/7101a36f002212f25fcf66e3d0d0f6f00255f47c/tool/berty-client/client.go).
## Instanciation detail

### Relay

The easiest way to instantiate a relay is to use the rdvp command (berty/go/cmd/rdvp). **Note**: The node will not act as an rdvp as long as it is only specified in the relay config of a peer.

The address you'll get will need to be specified in node config that need a relay.

With libp2p/IPFS we use multiaddress to specify the address to reach a peer, more info here: https://docs.libp2p.io/concepts/addressing/

#### Instanciate a relay

It's up to you to do it using a pre-gen key or not, just choose the more convenient way.
Knowing that with a pre-gen key:
- you can persist the relay peerID across restart (so it's multiaddress will still be valid).
- you can get the full multiaddress by crafting it (since you have a persistent peerID) and without having to read the log

When you will use the `serve` subcommand (see examples below), by default, the `-announce` flag will be unspecified and the `-l` flag will be specified with an empty string:

`-l=''`

But depending on the configuration of the connections, you will have to adapt these flags to set one or more listeners.

##### Without a pre-gen key

###### TCP / UDP

# You first need to know the public IP of the relay
$ export PUBLIC_IP4=52.158.21.203

# Specify a protocol between UDP and TCP (for this example)
$ export PROTOC=tcp

# You can choose a port or set it at 0 to use a random one
$ export PORT=4040

# Start the rdvp in relay mode (announce)
$ rdvp -log.format=json -log.file=./log.json serve \
    -announce "/ip4/$PUBLIC_IP4/$PROTOC/$PORT" \
    -l "/ip4/$PUBLIC_IP4/$PROTOC/$PORT"

# Get full multiaddr from logs
$ cat ./log.json | jq .maddr
"/ip4/52.158.21.203/tcp/4040/p2p/12D3KooWCS8MMcM4GuyFa7vc59R9TChh522dEy6raJQR5ZNKEnzm"

# NOTE: the last segment after p2p is a peerID

###### QUIC (over UDP only)

# You first need to know the public IP of the relay
$ export PUBLIC_IP4=52.158.21.203

# You can choose a port or set it at 0 to use a random one
$ export PORT=4040

# Start the rdvp in relay mode (announce)
$ rdvp -log.format=json -log.file=./log.json serve
    -announce "/ip4/$PUBLIC_IP4/udp/$PORT/quic" \
    -l "/ip4/$PUBLIC_IP4/udp/$PORT/quic"

# Get full multiaddr from logs
$ cat ./log.json | jq.maddr
"/ip4/52.158.21.203/udp/4040/quic/p2p/12D3KooWCS8MMcM4GuyFa7vc59R9TChh522dEy6raJQR5ZNKEnzm"

# NOTE: the last segment after p2p is a peerID

###### Websocket (over TCP only)

# You first need to know the public IP of the relay
$ export PUBLIC_IP4=52.158.21.203

# You can choose a port or set it at 0 to use a random one
$ export PORT=4040

# Start the rdvp in relay mode (announce)
$ rdvp -log.format=json -log.file=./log.json serve
    -announce "/ip4/$PUBLIC_IP4/tcp/$PORT/ws" \
    -l "/ip4/$PUBLIC_IP4/tcp/$PORT/ws"

# Get full multiaddr from logs
$ cat ./log.json | jq .maddr
"/ip4/52.158.21.203/tcp/4040/ws/p2p/12D3KooWCS8MMcM4GuyFa7vc59R9TChh522dEy6raJQR5ZNKEnzm"

# NOTE: the last segment after p2p is a peerID

##### With a pre-gen key

**Note**: Refer to QUIC and Websocket section above to see the format of a multiaddr with these transports

# You first need to know the public IP of the relay
$ export PUBLIC_IP4=52.158.21.203

# Specify a protocol between UDP and TCP (for this example)
$ export PROTOC=tcp

# You can choose a port or set it at 0 to use a random one
$ export PORT=4040

# Generate a secret key
$ rdvp genkey > ./key.sk
# Get peer id from this secret key
$ rdvp sharekey -pk `cat ./key.sk` > ./peerID
# Export peerID
$ export PEER_ID=$(cat ./peerID)

# Start the rdvp in relay mode (announce) using secret key
$ rdvp serve  -pk `cat ./key.sk` \
    -anounce "/ip4/$PUBLIC_IP4/$PROTOC/$PORT" \
    -l "/ip4/$PUBLIC_IP4/$PROTOC/$PORT"

# The full multiaddr will be
$ echo "/ip4/$PUBLIC_IP4/$PROTOC/$PORT/p2p/$PEER_ID"
"/ip4/52.158.21.203/tcp/4040/p2p/12D3KooWCS8MMcM4GuyFa7vc59R9TChh522dEy6raJQR5ZNKEnzm"

### RDVP

Will be almost the same as relay section above with few differences:
- You can (optionnaly) persist the rdvp DB across restart by using the `-db file.db` flag
- You won't use `-announce <multiaddr>` as it's only useful in relay mode

When you will use the `serve` subcommand (see examples below), by default, the `-l` flag will be specified with an empty string:

`-l=''`

But depending on the configuration of the connections, you will have to adapt this flag to set one or more listeners.

#### Instanciate a RDVP

##### Without a pre-gen key

###### TCP / UDP

# You first need to know the public IP of the relay
$ export PUBLIC_IP4=52.158.21.203

# Specify a protocol between UDP and TCP (for this example)
$ export PROTOC=tcp

# You can choose a port or set it at 0 to use a random one
$ export PORT=4040

# Start the rdvp with persistent DB
$ rdvp -log.format=json -log.file=./log.json \
    serve -l "/ip4/$PUBLIC_IP4/$PROTOC/$PORT"

# Get full multiaddr from logs
$ cat ./log.json | jq .maddr
"/ip4/52.158.21.203/tcp/4040/p2p/12D3KooWCS8MMcM4GuyFa7vc59R9TChh522dEy6raJQR5ZNKEnzm"

# NOTE: the last segment after p2p is a peerID

###### QUIC (over UDP only)

# You first need to know the public IP of the relay
$ export PUBLIC_IP4=52.158.21.203

# You can choose a port or set it at 0 to use a random one
$ export PORT=4040

# Start the rdvp in relay mode (announce)
$ rdvp -log.format=json -log.file=./log.json \
   serve -l "/ip4/$PUBLIC_IP4/udp/$PORT/quic"

# Get full multiaddr from logs
$ cat ./log.json | jq.maddr
"/ip4/52.158.21.203/udp/4040/quic/p2p/12D3KooWCS8MMcM4GuyFa7vc59R9TChh522dEy6raJQR5ZNKEnzm"

# NOTE: the last segment after p2p is a peerID

###### Websocket (over TCP only)

# You first need to know the public IP of the relay
$ export PUBLIC_IP4=52.158.21.203

# You can choose a port or set it at 0 to use a random one
$ export PORT=4040

# Start the rdvp in relay mode (announce)
$ rdvp -log.format=json -log.file=./log.json \
   serve -l "/ip4/$PUBLIC_IP4/tcp/$PORT/ws"

# Get full multiaddr from logs
$ cat ./log.json | jq .maddr
"/ip4/52.158.21.203/tcp/4040/ws/p2p/12D3KooWCS8MMcM4GuyFa7vc59R9TChh522dEy6raJQR5ZNKEnzm"

# NOTE: the last segment after p2p is a peerID

##### With a pre-gen key

**Note**: Refer to QUIC and Websocket section above to see the format of a multiaddr with these transports

# You first need to know the public IP of the relay
$ export PUBLIC_IP4=52.158.21.203

# Specify a protocol between UDP and TCP (for this example)
$ export PROTOC=tcp

# You can choose a port or set it at 0 to use a random one
$ export PORT=4040

# Generate a secret key
$ rdvp genkey > ./key.sk
# Get peer id from this secret key
$ rdvp sharekey -pk `cat ./key.sk` > ./peerID
# Export peerID
$ export PEER_ID=$(cat ./peerID)

# Start the rdvp in relay mode (announce) using secret key
$ rdvp serve  -pk `cat ./key.sk` -l "/ip4/$PUBLIC_IP4/$PROTOC/$PORT"

# The full multiaddr will be
$ echo "/ip4/$PUBLIC_IP4/$PROTOC/$PORT/p2p/$PEER_ID"
"/ip4/52.158.21.203/tcp/4040/p2p/12D3KooWCS8MMcM4GuyFa7vc59R9TChh522dEy6raJQR5ZNKEnzm"

### Bootstrap

You can launch a bootstrap node by launching a Berty daemon with the right flags:
- `-p2p.mdns=false`
- `-p2p.bootstrap=':none:'`
- `-p2p.rdvp=':none'`
- `-p2p.static-relays=':none'`
- `-p2p.tinder-dht-driver=false`
- `-p2p.tinder-rdvp-driver=false`

By default, the swarm listener flag will be empty:

- `-p2p.swarm-listeners=':none:'`

But depending on the configuration of the connections, you will have to adapt this flag to set one or more listeners.

Example:

Bootstrap:
  type: bootstrap
  amount: 1
  connections:
    - to: internet
      transport: tcp
      bandwidth: 100Mbps
      reliability: 0,0

Will be run with:

berty daemon \
  -p2p.mdns=false \
  -p2p.bootstrap=':none:' \
  -p2p.rdvp=':none' \
  -p2p.static-relays=':none' \
  -p2p.tinder-dht-driver=false \
  -p2p.tinder-rdvp-driver=false \
  -p2p.swarm-listeners="/ip4/$PUBLIC_IP/tcp/$PORT"

### Group creation

**NOTE**: You could merge [Group creation](#Group-creation) and [Replication](#Replication) in one step if it seems better to you.

To create a group:
1. Run a berty daemon

berty daemon

2. Run a [custom go program](https://hackmd.io/6HsBjNkPTM6xvDo3WJq1bw?both#Daemon-gRPC-API) that:
2A. Create a group

gRPC call method:
protocoltypes.ProtocolService.MultiMemberGroupCreate

2B. Create an invitation to this group

gRPC call method:
protocoltypes.ProtocolService.MultiMemberGroupInvitationCreate

with:
protocoltypes.ProtocolService.MultiMemberGroupInvitationCreate.Request{group_pk: <value of protocoltypes.ProtocolService.MultiMemberGroupCreate.Reply.group_pk>}

2C. Save the output of `protocoltypes.ProtocolService.MultiMemberGroupInvitationCreate.Reply.group` as an invit

3. Kill the setup daemon and the go custom program

### Replication

A peer need a token to authenticate when using the API of a replication server.

So the flow should be:
1. Run a replication server

# The public IP of the replication server
$ export REPL_IP=1.1.1.1

# The port of the replication server API
$ export REPL_PORT=1111

# Run the replication server
berty repl-server \
  -node.listeners "/ip4/$REPL_IP/tcp/$PORT/grpc" \
  -node.auth-secret 0123456789abcdefghijklmnopqrstuvwxyzABCDEFG \
  -node.auth-pk sUpMtDYs4fSOMQ1EMZgkKXgz9l3U822ByT57liN4TG0

2. Run a token server connected to the replication server

# The public IP of the token server
$ export TOKEN_IP=2.2.2.2

# The port of the token server API
$ export TOKEN_PORT=2222

# Run the token server
berty token-server \
  -no-click \ # required for automation
  -svc "rpl@$REPL_IP:$REPL_PORT" \
  -http.listener "$TOKEN_IP:$TOKEN_PORT" \
  -auth.secret 0123456789abcdefghijklmnopqrstuvwxyzABCDEFG \
  -auth.sk ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789abcdefg 

3. Run a berty daemon

berty daemon

4. Run a [custom go program](https://hackmd.io/6HsBjNkPTM6xvDo3WJq1bw?both#Daemon-gRPC-API) that takes as parameter a group invit, the token server address and run this flow:

4A. Request a token to the token server

gRPC call method:
protocoltypes.ProtocolService.AuthServiceInitFlow

with:
protocoltypes.AuthServiceInitFlow.Request{auth_url: "https://$TOKEN_IP:$TOKEN_PORT"}

Then write a go function equivalent to:

export CALLBACK_URL=$(curl "<value of protocoltypes.AuthServiceInitFlow.Reply.url>" -s | grep href= | cut -d'"' -f2 | sed 's/&amp;/\&/')

4B. Register to the replication server

gRPC call method:
protocoltypes.ProtocolService.AuthServiceCompleteFlow

with:
protocoltypes.AuthServiceCompleteFlow.Request{callback_url: "$CALLBACK_URL"}

4C. Join the group to which you need to add this replication server (it will be auto-added on join)

gRPC call method:
protocoltypes.ProtocolService.MultiMemberGroupJoin

with:
protocoltypes.MultiMemberGroupJoin.Request{group: <invitation>}

5. Kill the setup daemon, go custom program and token server

### Peer

To instantiate a peer, you will have to run `berty daemon` command. One concern is that a lot of parameters are set by default and you will have to explicitly disable them.

The list of default flags a peer needs are:
- `-p2p.mdns=false`
- `-p2p.static-relays=':none'`
- `-p2p.bootstrap=':none:'`
- `-p2p.dht-randomwalk=false`
- `-p2p.tinder-dht-driver=false`
- `-p2p.rdvp=':none'`
- `-p2p.tinder-rdvp-driver=false`
- `-p2p.swarm-listeners=':none:'`

But depending on it's config, you will have adapt these flags.

#### Flags

##### Connections

Example_Peer:
  type: peer
  amount: 1
  connections:
    - to: lan_1
      transport: quic
      bandwidth: 100Mbps
      reliability: 0,0

Will be run using this flag:

berty daemon \
  -p2p.mdns=false \
  -p2p.static-relays=':none' \
  -p2p.bootstrap=':none:' \
  -p2p.dht-randomwalk=false \
  -p2p.tinder-dht-driver=false \
  -p2p.rdvp=':none' \
  -p2p.tinder-rdvp-driver=false \
  -p2p.swarm-listeners='/ip4/$PRIVATE_IP_LAN_1/udp/$PORT/quic'

##### Routers relay

Example_Peer:
  type: peer
  amount: 1
  connections:
    - to: internet
      transport: quic
      bandwidth: 100Mbps
      reliability: 0,0
  routers:
    - type: relay
      address: '/ip4/51.159.21.214/udp/4040/quic/p2p/QmdT7AmhhnbuwvCpa5PH1ySK9HJVB82jr3fo1bxMxBPW6p'

Will be run using this flag:

berty daemon \
  -p2p.mdns=false \
  -p2p.bootstrap=':none:' \
  -p2p.dht-randomwalk=false \
  -p2p.tinder-dht-driver=false \
  -p2p.rdvp=':none' \
  -p2p.tinder-rdvp-driver=false \
  -p2p.swarm-listeners='/ip4/$PUBLIC_IP/udp/$PORT/quic' \
  -p2p.static-relays='/ip4/51.159.21.214/udp/4040/quic/p2p/QmdT7AmhhnbuwvCpa5PH1ySK9HJVB82jr3fo1bxMxBPW6p'

##### Routers bootstrap

Example_Peer:
  type: peer
  amount: 1
  connections:
    - to: lan_1
      transport: quic
      bandwidth: 100Mbps
      reliability: 0,0
  routers:
    - type: bootstrap
      address: '/ip4/104.131.131.82/tcp/4001/p2p/QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ'

Will be run using this flag:

berty daemon \
  -p2p.mdns=false \
  -p2p.static-relays=':none' \
  -p2p.rdvp=':none' \
  -p2p.tinder-rdvp-driver=false \
  -p2p.swarm-listeners='/ip4/$PUBLIC_IP/udp/$PORT/quic' \
  -p2p.bootstrap='/ip4/104.131.131.82/tcp/4001/p2p/QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ' \
  -p2p.dht-randomwalk=true # Or unspecified because true is default \
  -p2p.tinder-dht-driver=true # Or unspecified because true is default

##### Routers rdvp

Example_Peer:
  type: peer
  amount: 1
  connections:
    - to: lan_1
      transport: quic
      bandwidth: 100Mbps
      reliability: 0,0
  routers:
    - type: rdvp
      address: '/ip4/104.131.131.82/tcp/4001/p2p/QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ'

Will be run using this flag:

berty daemon \
  -p2p.mdns=false \
  -p2p.static-relays=':none' \
  -p2p.bootstrap=':none:' \
  -p2p.dht-randomwalk=false \
  -p2p.tinder-dht-driver=false \
  -p2p.swarm-listeners='/ip4/$PUBLIC_IP/udp/$PORT/quic' \
  -p2p.rdvp='/ip4/51.159.21.214/udp/4040/quic/p2p/QmdT7AmhhnbuwvCpa5PH1ySK9HJVB82jr3fo1bxMxBPW6p' \
  -p2p.tinder-rdvp-driver=true # Or unspecified because true is default

#### Join group and run tests

For each group to join you will have to use the group invitation and connect to each `peer` (daemon) [using gRPC](#Daemon-gRPC-API).

1. Join a group

gRPC call method:
protocoltypes.ProtocolService.MultiMemberGroupJoin

with:
protocoltypes.MultiMemberGroupJoin.Request{group: <invitation>}

**TODO**: but you can check the code of Berty mini here: https://github.com/berty/berty/blob/master/go/cmd/berty/mini/view_group.go

2. Run a loop that receive messages and send back ACKs using these calls
3. Send text messages
4. Send media messages