---
title: "Gateway architecture"
tags: wikidocs
teams:
- networkingWG
participants:
- Simon (SZ)
---

# Gateway Component

This page describes the general functionality of the gateway component.

The gateway component is part of the network manager (`node-net-manager`) deployed on-demand during the cluster operation on request of service exposure to the outside internet.

The gateway aims to enable controlled traffic flow for entities outside the Oakestra network towards services residing inside an Oakestra cluster. It functions as an entry point to the (private) Oakestra network by implementing basic port forwarding to ServiceIPs - thus we can keep the semantic addressing logic.

As an example: A developer has a monitoring interface (running on port 7080) of sensors deployed inside Oakestra and wants to access its web interface or API without being connected to the inside of Oakestra. By requesting a gateway towards the monitoring interface, he can receive a gateway IP and a desired port he wants it to be available at. By then connecting to the gateway on the forwarded port, the gateway component goes ahead, builds the connection towards the service and responds to the request, without ever exposing any information on the infrastructure behind the gateway.

## Architecture

This chapter describes the components of a gateway which are setup or managed for operation.

Below is a summary figure of a gateway edge node running the `net-manager`.


#### MQTT

The gateway uses MQTT as its main messaging protocol. It is used for deployment, management and operation of the gateway function. It is setup at startup of the `net-manager`.

#### goProxyTunnel

The tunnel is used for sending and receiving traffic to and from other nodes inside the Oakestra network. It is the main component used for tunneling traffic between worker nodes inside a Oakestra cluster. Its functionality is closely connected to the operation of the Net-Manager Proxy. It is setup on startup of the `net-manager`.

#### goProxyBridge

The bridge is currently only needed for operation of the gateway service translation, i.e. the [proxy translation](https://www.oakestra.io/docs/networking/service-ip-overlay-implementation/#proxy-translation). Since a gateway by design is never deploying any service instances, due to the need of all available ports, there is room for optimization in regards to address space exhaustion. In general, the proxyBridge is used to bridge the traffic generated by the containers running on a worker node. Since a gateway does not have any service containers, thie bridge should be made obsolete for gateway operation. It is setup on startup of the `net-manager`.

#### Net-Manager Proxy

The net-manager proxy is responsible for translating requests between the Oakestra Service and Virtual Layer. For further information on that look [here](https://www.oakestra.io/docs/networking/service-ip-overlay-implementation/#proxy-translation).

#### Firewall

The firewall is the key component for translating requests from the Internet to the inside of Oakestra. We explain the functionality in detail in the Operation chapter down below. It is setup at deployment of the gateway and managed during operation. Currently we only support iptables.

## Setup Process

This chapter describes the general setup of an environment for a gateway, as well as the setup process of a gateway node.

### Prerequisites

In order for a gateway to be able to be deployed, we need to have an Oakestra cluster running. This involves a `root-orchestrator` with running `root service-manager`, as well as one or many `cluster-orchestrators` with running `cluster service-managers`. Additionally, we can assume that a cluster has one or more running worker nodes, as well as one or more idle non-worker nodes for successfull gateway deployment.

An idle non-worker node can be described as a node with a running `net-manager`, waiting for `NodeEngine` bootup and registration. Thus we can assume that every node attached to a cluster has a `net-manager` running.

The figure below shows an example cluster.


### Configuration

In order to register a node to a cluster we first need to configure the needed configuration files located in `node-net-manager/config/`

For the `netcfg.json` we will need to set the ClusterURL, which is the IP under which the cluster-orchestrator is reachable.

The `tuncfg.json` can be left as is and do not need adjustment.

Place these configuration files under `/etc/netmanager/`, creating the directory if necessary.

At this point we can start up the net-manager.

## Deployment of a Gateway

This chapter describes the process resulting in a successfully deployed gateway inside a cluster.

### Net-Manager Setup

1. The net-manager registers with the cluster-orchestrator sending the system information over HTTP, receiving back its ID.

2. The ID is used to setup the MQTT client and subscribe to the following topics:
- nodes/{id}/net/subnetwork/result -- used for receiving the assigned namespace subnet
- nodes/{id}/net/gateway/deploy -- used for gateway deployment

3. The net-manager requests his namespace subnet over MQTT.
4. Upon receiving the subnet it proceeds to create the `goProxyBridge`, as well the `goProxyTun`, as described in `tuncfg.json`. These two new interfaces are key components for the net-manager functionality, are also needed for worker operation and thus only setup once throughout the lifetime of a worker node.

> The cluster-orchestrator keeps track of this new idle net-manager node in its mongoDB, which is later used to determine a candidate for gateway deployment.

At this point the net-manager is listening for either a worker registration request coming from a node-engine on the same host, or a gateway deployment request coming from the cluster-orchestrator.

In the following we describe the process to deploy a gateway on the newly registered net-manager. Since this is done on-demand we have to trigger this process by requesting the exposure of a service.

### Service Exposure Request

In order to request the exposure of a service to the Internet -- which in more general terms is just the forwarding of a port to inside the Oakestra cluster -- we need to prepare a short JSON document to send to the root orchestrators API endpoint `POST /api/gateway/`

The form should look like the following:

{
    sla_version: v2.0,
    customerID: <string>,
    microservices: [
        { 
            microserviceID: <string>,
            microservice_name: <string>,
            exposed_port: <int>
        },
        { ... }
    ]
}

To explain the fields:

- `sla_version`: use v2.0 API implementation
- `customerID`: your username registered with the cluster
- `microservices`: list of microservices you wish to expose
- `microserviceID`: unique service identifier
- `microservice_name`: optional, currently not used
- `exposed_port`: the port you wish to reach the service at

Upon `POST` of such a form the root orchestrator checks if the service is registered with the cluster, and if so, fetches all clusters the service is deployed at. It then goes ahead and notifies the clusters to go ahead and realize the service exposure.

The cluster checks if there is a gateway available in its cluster to realize the port forwarding and if so, notifies the gateway over MQTT to update the firewall. If not, it checks if there are idle workers (net-managers) in its cluster and notifies them over MQTT to set up the gateway function. If no idle worker is available the cluster manager reports an error to the root.

After successfull deployment the gateway gets a exposure request for the requested service, which then goes ahead and sets up the port forwarding to the service's assigned service IP. In particular, we set up three firewall rules:

1. `iptables -t nat -C PREROUTING -i gateway_interface -p tcp -d public_gateway_ip --dport exposed_service_port -j DNAT --to-destination service_ip:service_port`
2. `iptables -t filter -C FORWARD -i gateway_interface -o gateway_tunnel --dport service_port -m conntrack --ctstate NEW -j ACCEPT`
3. `iptables -t nat -C POSTROUTING -o gateway_tunnel -p tcp -d service_ip --dport service_port -j SNAT --to-source gateway_bridge_ip`

With these three rules we allow for a successfull translation and retranslation of requests. The details behind the rules are explained [in the next chapter](##Operation).

On successfull exposure the root orchestrator reports back a list of IP addresses where the service is reachable from the outside.

## Operation

This chapter describes the operation of a gateway with a exposed service, with focus on explaining the network flow of an incoming request.

The general idea for the gateway network flow was to utilize the already existing proxy mechanism for Oakestra-internal service-to-service communication. How that is done, can be read [here](https://www.oakestra.io/docs/networking/service-ip-overlay-implementation/). This allowed us to ignore further implementation of balancing strategies at gateway level, since we only hand over the request to the net-manager proxy, which handles the decision making and balancing of service requests.

The implementation of the gateway function thus only required us to translate incoming requests to the gateway IP address with exposed target port to a service IP request from inside Oakestra. The remaining work is then handled by the already working proxy. Please note that for we currently only support Round Robin Service IP exposure.

The translation of incoming requests is handled by ip- and ip6tables and its SNAT and DNAT capabilities. Since this program is part of most mainline Linux systems, we can avoid the need of 3rd party applications.


An example incoming request gets translated the following way step-by-step:

1. Request with Source IP 131.159.24.1 and Destination IP 59.58.1.3:80
2. Packet reaches PREROUTING chain in nat table -- translates Destination to 10.30.30.30:8080 which is the service's Round Robin service IP with container port 8080
3. Packet passes to the filter's FORWARD chain, where the packet is ACCEPTed, since there is a successfull mapping to a service. Bad requests get lost here, since they are not translated by the PREROUTING to an address from inside Oakestra.
4. Packets reaches POSTROUTING chain -- translates Source to gateways bridge IP

> At this point the packet looks like a regular service-to-service request in the Oakestra network. The tunneling towards the correct instance with respect to the balancing policy is handled by the net-managers proxy.

The response by the service is then retranslated by the proxy such that iptables SNAT and DNAT are able to retranslate and return the package to the client.