New multi-router configuration system

Summary

Previously we only used DHT for content routing and content providing. After kubo-0.14.0 release we added support for delegated routing using Reframe protocol.

Now we need a better way to add different routers using different protocols like Reframe or DHT, and be able to configure them to cover different use cases.

Motivation

The actual routing implementation is not enough. Some users needs to have more options when configuring the routing system. The new implementations should be able to:

  • Be user-friendly and easy enough to configure, but also versatile
  • Configurable Router execution order
    • Delay some of the Router methods execution when they will be executed on parallel
  • Configure which method of a giving router will be used
  • Mark some router methods as mandatory to make the execution fails if that method fails

Detailed design

Configuration file description

The Routing configuration section will contain the following keys:

Type

Type will be still in use to avoid complexity for the user that only wants to use Kubo with the default behavior. We are going to add a new type, custom, that will use the new router systems. none type will deactivate all routers, default dht and delegated ones.

Routers

Routers will be a key-value list of routers that will be available to use. The key is the router name and the value is all the needed configurations for that router. the Type will define the routing kind. The main router types will be reframe and dht, but we will implement two special routers used to execute a set of routers in parallel or sequentially: parallel router and sequential router.

Depending on the routing type, it will use different parameters:

Reframe

Params:

  • "Endpoint": URL endpoint implementing Reframe protocol.
DHT

Params:

  • "Mode": Mode used by the DHT. Possible values: "server", "client", "auto"
  • "AcceleratedDHTClient": Set to true if you want to use the experimentalDHT.
  • "PublicIPNetwork": Set to true to create a WAN DHT. Set to false to create a LAN DHT.
Parallel

Params:

  • Routers: A list of routers that will be executed in parallel:
    • Name:string: Name of the router. It should be one of the previously added to Routers list.
    • Timeout:duration: Local timeout. It accepts strings compatible with Go time.ParseDuration(string). Time will start counting when this specific router is called, and it will stop when the router returns, or we reach the specified timeout.
    • ExecuteAfter:duration: Providing this param will delay the execution of that router at the specified time. It accepts strings compatible with Go time.ParseDuration(string).
    • IgnoreErrors:bool: It will specify if that router should be ignored if an error occurred.
  • Timeout:duration: Global timeout. It accepts strings compatible with Go time.ParseDuration(string).
Sequential

Params:

  • Routers: A list of routers that will be executed in order:
    • Name:string: Name of the router. It should be one of the previously added to Routers list.
    • Timeout:duration: Local timeout. It accepts strings compatible with Go time.ParseDuration(string). Time will start counting when this specific router is called, and it will stop when the router returns, or we reach the specified timeout.
    • IgnoreErrors:bool: It will specify if that router should be ignored if an error occurred.
  • Timeout:duration: Global timeout. It accepts strings compatible with Go time.ParseDuration(string).

Methods

Methods:map will define which routers will be executed per method. The key will be the name of the method: "provide", "find-providers", "find-peers", "put-ipns", "get-ipns". All methods must be added to the list. This will make configuration discoverable giving good errors to the user if a method is missing.

The value will contain:

  • RouterName:string: Name of the router. It should be one of the previously added to Routers list.

Configuration file example:

"Routing": {
  "Type": "custom",
  "Routers": {
    "storetheindex": {
      "Type": "reframe",
      "Parameters": {
        "Endpoint": "https://cid.contact/reframe"
      }
    },
    "dht-lan": {
      "Type": "dht",
      "Parameters": {
        "Mode": "server",
        "PublicIPNetwork": false,
        "AcceleratedDHTClient": false
      }
    },
    "dht-wan": {
      "Type": "dht",
      "Parameters": {
        "Mode": "auto",
        "PublicIPNetwork": true,
        "AcceleratedDHTClient": false
      }
    },
    "find-providers-router": {
      "Type": "parallel",
      "Parameters": {
        "Routers": [
          {
            "RouterName": "dht-lan",
            "IgnoreErrors": true
          },
          {
            "RouterName": "dht-wan"
          },
          {
            "RouterName": "storetheindex"
          }
        ]
      }
    },
    "provide-router": {
      "Type": "parallel",
      "Parameters": {
        "Routers": [
          {
            "RouterName": "dht-lan",
            "IgnoreErrors": true
          },
          {
            "RouterName": "dht-wan",
            "ExecuteAfter": "100ms",
            "Timeout": "100ms"
          },
          {
            "RouterName": "storetheindex",
            "ExecuteAfter": "100ms"
          }
        ]
      }
    },
    "get-ipns-router": {
      "Type": "sequential",
      "Parameters": {
        "Routers": [
          {
            "RouterName": "dht-lan",
            "IgnoreErrors": true
          },
          {
            "RouterName": "dht-wan",
            "Timeout": "300ms"
          },
          {
            "RouterName": "storetheindex",
            "Timeout": "300ms"
          }
        ]
      }
    },
    "put-ipns-router": {
      "Type": "parallel",
      "Parameters": {
        "Routers": [
          {
            "RouterName": "dht-lan"
          },
          {
            "RouterName": "dht-wan"
          },
          {
            "RouterName": "storetheindex"
          }
        ]
      }
    }
  },
  "Methods": {
    "find-providers": {
      "RouterName": "find-providers-router"
    },
    "provide": {
      "RouterName": "provide-router"
    },
    "get-ipns": {
      "RouterName": "get-ipns-router"
    },
    "put-ipns": {
      "RouterName": "put-ipns-router"
    }
  }
}

Added YAML for clarity:

---
Type: custom
Routers:
  storetheindex:
    Type: reframe
    Parameters:
      Endpoint: https://cid.contact/reframe
  dht-lan:
    Type: dht
    Parameters:
      Mode: server
      PublicIPNetwork: false
      AcceleratedDHTClient: false
  dht-wan:
    Type: dht
    Parameters:
      Mode: auto
      PublicIPNetwork: true
      AcceleratedDHTClient: false
  find-providers-router:
    Type: parallel
    Parameters:
      Routers:
      - RouterName: dht-lan
        IgnoreErrors: true
      - RouterName: dht-wan
      - RouterName: storetheindex
  provide-router:
    Type: parallel
    Parameters:
      Routers:
      - RouterName: dht-lan
        IgnoreErrors: true
      - RouterName: dht-wan
        ExecuteAfter: 100ms
        Timeout: 100ms
      - RouterName: storetheindex
        ExecuteAfter: 100ms
  get-ipns-router:
    Type: sequential
    Parameters:
      Routers:
      - RouterName: dht-lan
        IgnoreErrors: true
      - RouterName: dht-wan
        Timeout: 300ms
      - RouterName: storetheindex
        Timeout: 300ms
  put-ipns-router:
    Type: parallel
    Parameters:
      Routers:
      - RouterName: dht-lan
      - RouterName: dht-wan
      - RouterName: storetheindex
Methods:
  find-providers:
    RouterName: find-providers-router
  provide:
    RouterName: provide-router
  get-ipns:
    RouterName: get-ipns-router
  put-ipns:
    RouterName: put-ipns-router

Error cases

  • If any of the routers fails, the output will be an error by default.
  • You can use IgnoreErrors:true to ignore errors for a specific router output
  • To avoid any error at the output, you must ignore all router errors.

Implementation Details

Methods

All routers must implement the routing.Routing interface:








type Routing interface {
    ContentRouting
    PeerRouting
    ValueStore

    Bootstrap(context.Context) error
}

All methods involved:













type Routing interface {
    Provide(context.Context, cid.Cid, bool) error
    FindProvidersAsync(context.Context, cid.Cid, int) <-chan peer.AddrInfo
    
    FindPeer(context.Context, peer.ID) (peer.AddrInfo, error)

    PutValue(context.Context, string, []byte, ...Option) error
    GetValue(context.Context, string, ...Option) ([]byte, error)
    SearchValue(context.Context, string, ...Option) (<-chan []byte, error)

    Bootstrap(context.Context) error
}

We can configure which methods will be used per routing implementation. Methods names used in the configuration file will be:

  • Provide: "provide"
  • FindProvidersAsync: "find-providers"
  • FindPeer: "find-peers"
  • PutValue: "put-ipns"
  • GetValue, SearchValue: "get-ipns"
  • Bootstrap: It will be always executed when needed.

Routers

We need to implement the parallel and sequential routers and stop using routinghelpers.Tiered router implementation.

Add cycle detection to avoid to user some headaches.

Also we need to implement an internal router, that will define the router used per method.

Other considerations

  • We need to refactor how DHT routers are created to be able to use and add any amount of custom DHT routers.
  • We need to add a new custom router type to be able to use the new routing system.
  • Bitswap WANT broadcasting is not included on this document, but it can be added in next iterations.
  • This document will live in docs/design-notes for historical reasons and future reference.

Test fixtures

As test fixtures we can add different use cases here and see how the configuration will look like.

Mimic previous dual DHT config

"Routing": {
  "Type": "custom",
  "Routers": {
    "dht-lan": {
      "Type": "dht",
      "Parameters": {
        "Mode": "server",
        "PublicIPNetwork": false
      }
    },
    "dht-wan": {
      "Type": "dht",
      "Parameters": {
        "Mode": "auto",
        "PublicIPNetwork": true
      }
    },
    "parallel-dht-strict": {
      "Type": "parallel",
      "Parameters": {
        "Routers": [
          {
            "RouterName": "dht-lan"
          },
          {
            "RouterName": "dht-wan"
          }
        ]
      }
    },
    "parallel-dht": {
      "Type": "parallel",
      "Parameters": {
        "Routers": [
          {
            "RouterName": "dht-lan",
            "IgnoreError": true
          },
          {
            "RouterName": "dht-wan"
          }
        ]
      }
    }
  },
  "Methods": {
    "provide": {
      "RouterName": "dht-wan"
    },
    "find-providers": {
      "RouterName": "parallel-dht-strict"
    },
    "find-peers": {
      "RouterName": "parallel-dht-strict"
    },
    "get-ipns": {
      "RouterName": "parallel-dht"
    },
    "put-ipns": {
      "RouterName": "parallel-dht"
    }
  }
}

Yaml representation for clarity:

---
Type: custom
Routers:
  dht-lan:
    Type: dht
    Parameters:
      Mode: server
      PublicIPNetwork: false
  dht-wan:
    Type: dht
    Parameters:
      Mode: auto
      PublicIPNetwork: true
  parallel-dht-strict:
    Type: parallel
    Parameters:
      Routers:
      - RouterName: dht-lan
      - RouterName: dht-wan
  parallel-dht:
    Type: parallel
    Parameters:
      Routers:
      - RouterName: dht-lan
        IgnoreError: true
      - RouterName: dht-wan
Methods:
  provide:
    RouterName: dht-wan
  find-providers:
    RouterName: parallel-dht-strict
  find-peers:
    RouterName: parallel-dht-strict
  get-ipns:
    RouterName: parallel-dht
  put-ipns:
    RouterName: parallel-dht

Compatibility

We need to create a config migration using fs-repo-migrations. We should remove the Routing.Type param and add the configuration specified previously.

We don't need to create any config migration! To avoid to the users the hassle of understanding how the new routing system works, we are gonna keep the old behavior. We will add the Type custom to make available the new Routing system.

Security

No new security implications or considerations were found.

Alternatives

I got ideas from all of the following links to create this design document:

Copyright and related rights waived via CC0.

Select a repo