# How to add dynamic servers for host based routing

After working the awesome support team at Haproxy, I was able to come up with a unique configuration for my use case. I thought it would be of value to some to share it.

There are different ways to route layer 7 traffic. You can do it via hostname, fqdn, path, port or a combination of all of them. Here are some examples:
* mydomain.com
* myhost.mydomain.com
* myhost.mydomain.com/mypath
* myhost.mydomain.com:32000/mypath

Depending how you have structured your applications you will need to configure Haproxy for the type of routing you require. In this blog post we are going to talk exclusively about hostname routing. This means, that there will be a single domain name, however each route to a backend server will be based on the hostname.

In this discussion, I have two customers, Ralph and Jim. They have a application being hosted at https://ralph.domain.com and https://jim.doimain.com. The following configuration is how we route the incoming requests to each customers backend application.

The goal of this configuration is to have a dynamic backend that is managed by the Data Plane API. The second goal is to prevent the proxy server from reloading the configuration each time a new customer is added or deleted. This is especially important when dealing with long running websockets as is the case with my application.

Lets look at each section of the configuration and explain what is going on.

global
    maxconn            5000
    log                127.0.0.1 local0
    log                127.0.0.1 local1 notice
    user               hapee-lb
    group              hapee
    chroot             /var/empty
    pidfile            /var/run/hapee-2.6/hapee-lb.pid
    stats socket       /var/run/hapee-2.6/hapee-lb.sock user hapee-lb group hapee mode 660 level admin expose-fd listeners
    stats timeout      10m
    module-path        /opt/hapee-2.6/modules
    hard-stop-after    4h
    daemon
    log-send-hostname
    ssl-default-bind-options ssl-min-ver TLSv1.2
    ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384

There are three lines in particular that I want to explain. The first is `hard-stop-after 4h`. For long running websocket connections having the hard stop happen too soon can break the legitimate connections. This ensures we get at least 4 hours. This is values comes from the average time my users access their application server.

The last two lines, will set the ssl/tls environment for the whole proxy server and apply these directives to every front/backend/server in the configuration.

Next we are going to set the default values for all frontends. Each frontend can set their own values, but these will apply to all. They can be overwritten.

defaults fe_defaults
    mode               http
    log                global
    option             httpslog
    option             dontlognull
    option             forwardfor except 127.0.0.0/8
    timeout client     3600s
    maxconn            5000

Here the important bit is to set the log option to httpslog. This will help with debugging tls/ssl errors. I would recomend that you test your site against [https://www.ssllabs.com/ssltest/](https://). This will tell you instantanly if your tls/ssl is setup correctly.

Backend default, again applies to all backends and can be overwritten if needed.

defaults be_defaults
    mode               http
    option             redispatch
    retries            3
    timeout connect    10s
    timeout server     3600s
    timeout tunnel     4h
    errorfiles         custom_errors
    http-response return  status 404  default-errorfiles  if { status 404 }
    http-response return  status 503  default-errorfiles  if { status 503 }

Here we want to match the hard time out value with timeout tunnel. The tunnel value is specific for websockets. You should note, that having these long timeouts means the proxy server will use more ram on the host to keep track of these long connections. The last three error lines are so we can show a nice web page if we are missing a route.

This is the stats block. You will need this if you want prometheus to scrape the metrics along with the actual stats webpage Haproxy provides.

frontend stats from fe_defaults
    bind 10.1.1.1:1936
    mode http
    stats enable
    stats hide-version
    stats realm Hapee\ Statistics
    stats uri /stats
    stats refresh 5s
    stats admin if TRUE
    http-request set-log-level silent
    http-request use-service prometheus-exporter if { path /metrics }

The one point of note here is choose your bind ip address to ensure it is secure. You could use *:1936 but if you have a public address on the host it will then be available to the public interface. Here I only bind to a private address. I could bind this to localhost to make it even more secure, but in my use case I need to access it over the network.

Now we get to the meat of the hostname routing configuration. We need to create a frontend that will be the main entry point.

frontend fe_web from fe_defaults
    bind *:80
    bind *:443 ssl crt /etc/hapee-2.6/ssl/
    http-request redirect scheme https if !{ ssl_fc }
    http-request capture hdr(Host) len 128 # logs the host header
    http-request set-var(txn.username) req.hdr(Host),field(1,.)

    # static host
    use_backend be_mainsite if { req.hdr(host) -i domain.com www.domain.com }
    
    # all other hosts will be customer servers so we can use default.    
    default_backend be_customers 

The first three lines, simply deal with http/https. If we get something on port 80 (http) then redirect it to port 443 (https). We specify were our certificates are located on the file system so we can terminate the tls here at the proxy. Next you want to add all your static routes. These would be things like your main website, sites that won't change. For our first trick. We need to get the hostname of the incoming request. This will eventually be the name of the backend server so this line `http-request set-var(txn.username) req.hdr(Host),field(1,.)` will take the fqdn of `jim.domain.com` and pull the hostname `jim` and put it in a variable `txn.username`. By putting it in a txn object it will be availble during the whole request/response cycle of the request.

Instead of making a frontend for each customer (which will cause a reload) we simply assume any other request is a customer. So lets use the default_backend. The key word `default_backend` is special and will handle all other frontend requests that are not already handled.

Our only staic route needs a backend.

# static server for domain.com
backend be_mainsite from be_defaults
    option httpchk GET /health-check
    server s1_mainsite 10.1.1.1:9080 no-ssl check 

This will do health checking and forward the incoming request to a ip and port on the backside of the proxy. Also note, `no-ssl` this will terminate the tls at the proxy and forward only http to the application.

Our dynanic backend is really the whole point of the article.

backend be_customers from be_defaults
  acl server_found req.hdr(Host) -m found
  use-server %[var(txn.username)] if server_found
  
  server ralph 10.1.1.20:32000 check no-ssl weight 0
  server jim 10.1.1.20:32001 check no-ssl weight 0
  server default 10.1.1.1:8087 check no-ssl backup
  

The first line `acl server_found req.hdr(Host) -m found` is an acl that allows the use of a conditional to pick the approperiate sever line. This simply says, is there a host in the request url? Of course there always will be one. This will make sure the next line is evaluated true each and every time. `use-server %[var(txn.username)] if server_found`. The key part is `use-server %[var(txn.username)]`. We take the hostname from the frontend via the txn variable, and use it to select the appropriate server in this backend. You can see how we have named each server with the name of the customer (username). This is done via the Data Plane API.

Consider the following:

server ralph 10.1.1.20:32000 check no-ssl 
server jim 10.1.1.20:32001 check no-ssl 
server default 10.1.1.1:8087 check no-ssl 

This looks like it would do what we want. However the default behaviour is to load balance across all servers in a specific backend. Even using the `user-server` won't stop that behavour. We can use our next trick to stop this. If we sent each server line to `weight 0` this will tell haproxy not to consider it in the selection process. Now because there is only one server line that will match the server name, it will select it even though the weight is 0.

  server ralph 10.1.1.20:32000 check no-ssl weight 0
  server jim 10.1.1.20:32001 check no-ssl weight 0
  server default 10.1.1.1:8087 check no-ssl

Now this almost works. We are still getting the default server load balanced in with our specific customers' server. Lets talk about what the default servers purpose is. I want to show a nice webpage if there is no server available for a given hostname. Looking at this you would think that being at the end that would do the trick. Nope! There is a special directive just for this purpose. We would use `backup`.

  server ralph 10.1.1.20:32000 check no-ssl weight 0
  server jim 10.1.1.20:32001 check no-ssl weight 0
  server default 10.1.1.1:8087 check no-ssl backup

Notice the word `backup`. This tells haproxy to only use this server line if there are no other servers available, a failover if you will. Hence the word `backup`. So with `weight 0` and `backup` on the default server line we can effectively select a specific server based on the hostname. In our example the port doesn't play a role in the configuration. This is simply added via the Data Plane API.

You can do this exact same thing, with a unique frontend/backend pair for each customer. The one thing to realize, is that this transaction will force haproxy to do a complete configuration reload and it may screw with your long running websocket connections. Using the Data Plane API to add or remove a sever line DOES NOT force a configuration reload and is immediately available to be used.

I would like to thank the support team (Chad, Bruno, and Michel) for all their help. This is a unique use case, but I think it shows how flexable haproxy is and what is possible with a few tricks. ;) Happy hacking all.

## Links to relevant documentation
* https://docs.haproxy.org/2.6/configuration.html#5.2-backup
* https://docs.haproxy.org/2.6/configuration.html#5.2-weight
* https://docs.haproxy.org/2.6/configuration.html#4.2-timeout%20tunnel
* https://docs.haproxy.org/2.6/configuration.html#3.1-hard-stop-after
* https://docs.haproxy.org/2.6/configuration.html#4.2-use-server
* https://docs.haproxy.org/2.6/configuration.html#3.1-set-var
* https://docs.haproxy.org/2.6/configuration.html#5.2-no-ssl
* https://docs.haproxy.org/2.6/configuration.html#4.2-default_backend