HAProxy >= 1.9 is required for dataplane api
Except Photon 3, other distros default repositories don't have the latest HAProxy(2.1)
Ubuntu 19.10 contains 2.0.5
dataplaneapi needs to be built manually
Photon will soon include an RPM for dataplane api binary TKG-1372

Default repository

Ubuntu

For Ubuntu 18.04 Bionic

Install from default repositories

apt install -y haproxy

However, the default repositories usually don't contain the latest releases. To check the current haproxy version in default repository:

ubuntu@u3:~$ apt show haproxy 2>/dev/null | grep Version
Version: 1.8.8-1ubuntu0.10

For Ubuntu 19.10 Eoan

Install from default repositories

apt install -y haproxy

However, the default repositories usually don't contain the latest releases. To check the current haproxy version in default repository:

# apt show haproxy 2>/dev/null | grep Version
Version: 2.0.5-1ubuntu0.3

For Debian 10.3

Install from default repositories

apt install -y haproxy

However, the default repositories usually don't contain the latest releases. To check the current haproxy version in default repository:

# apt show haproxy 2>/dev/null | grep Version
Version: 1.8.19-1+deb10u2

For Photon 3.0

Install from default repositories

tdnf install haproxy-2.1.0

For CentOS

For CentOS 7

Install from default repositories

yum install -y haproxy.x86_64

However, the default repositories usually don't contain the latest releases. To check the current haproxy version in default repository:

# yum info haproxy | grep Version
Version     : 1.5.18

For CentOS 8

Install from default repositories

yum install -y haproxy.x86_64

However, the default repositories usually don't contain the latest releases. To check the current haproxy version in default repository:

# yum info haproxy | grep Version
Version      : 1.8.15

Build from Source

Preparation

Ubuntu

apt update
apt install -y make gcc perl libpcre3-dev zlib1g-dev libssl-dev libsystemd-dev

# Optional. will be used in validation section
apt install -y jq docker.io

Photon

tdnf update
# diffutils is for cmp command: /bin/sh: cmp: command not found
# glibc-devel provides the unix headers: unistd.h: No such file or directory
# binutils provides the as assembler, which is required by zlib
tdnf install -y tar git make gcc perl pcre-devel zlib-devel openssl-devel systemd-devel diffutils glibc-devel binutils

# Optional. will be used in validation section
tdnf install -y jq docker
systemctl start docker
iptables -P INPUT ACCEPT
iptables -P FORWARD ACCEPT
iptables -P OUTPUT ACCEPT

CentOS

yum update
yum install -y make git gcc perl pcre-devel.x86_64 zlib-devel.x86_64 openssl-devel.x86_64 systemd-devel.x86_64

# Optional. will be used in validation section
# ref: https://medium.com/@gchandra/install-jq-on-centos-7-459dd650baa3
yum -y install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm && yum install -y jq
systemctl start docker

Build

HAPROXY_VEERSION="2.1"
HAPROXY_TARBALL="haproxy-2.1.4"

curl -s -L http://www.haproxy.org/download/"${HAPROXY_VEERSION}"/src/"${HAPROXY_TARBALL}".tar.gz | tar xzv

pushd "${HAPROXY_TARBALL}"
make TARGET=linux-glibc USE_PCRE=1 USE_OPENSSL=1 USE_ZLIB=1 USE_SYSTEMD=1 EXTRA_OBJS="contrib/prometheus-exporter/service-prometheus.o"
make install
cp haproxy /usr/sbin/
popd

groupadd haproxy
useradd -s /usr/sbin/nologin --system -g haproxy --home-dir=/var/lib/haproxy haproxy

mkdir /var/lib/haproxy
chown -R haproxy:haproxy /var/lib/haproxy

curl -s https://raw.githubusercontent.com/horms/haproxy/master/doc/configuration.txt | gzip -c > configuration.txt.gz
mkdir -p /usr/share/doc/haproxy
mv configuration.txt.gz /usr/share/doc/haproxy/configuration.txt.gz

Configuration

systemd

cat > /lib/systemd/system/haproxy.service <<"EOF"
[Unit]
Description=HAProxy Load Balancer
Documentation=man:haproxy(1)
Documentation=file:/usr/share/doc/haproxy/configuration.txt.gz
# allows us to do millisecond level restarts without triggering alert in Systemd
StartLimitInterval=0
StartLimitBurst=0
After=network.target syslog.service
Wants=syslog.service

[Service]
EnvironmentFile=-/etc/default/haproxy
EnvironmentFile=-/etc/sysconfig/haproxy
Environment="CONFIG=/etc/haproxy/haproxy.cfg" "PIDFILE=/var/run/haproxy.pid" "EXTRAOPTS=-S /var/run/haproxy-master.sock"
ExecStartPre=/usr/sbin/haproxy -f $CONFIG -c -q $EXTRAOPTS
ExecStart=/usr/sbin/haproxy -Ws -f $CONFIG -p $PIDFILE $EXTRAOPTS
ExecReload=/usr/sbin/haproxy -f $CONFIG -c -q $EXTRAOPTS
ExecReload=/bin/kill -USR2 $MAINPID
KillMode=mixed
Restart=always
SuccessExitStatus=143
Type=notify

[Install]
WantedBy=multi-user.target
EOF

haproxy.cfg

mkdir -p /etc/haproxy

cat > /etc/haproxy/haproxy.cfg <<"EOF"
global
  master-worker
  log stdout format raw local0 info

  stats socket /run/haproxy.sock user haproxy group haproxy mode 660 level admin expose-fd listeners
  stats timeout 30s

  chroot /var/lib/haproxy
  user haproxy
  group haproxy

  # Default SSL material locations
  ca-base /etc/ssl/certs
  crt-base /etc/ssl/private

  # Default ciphers to use on SSL-enabled listening sockets.
  # For more information, see ciphers(1SSL). This list is from:
  #  https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
  # An alternative list with additional directives can be obtained from
  #  https://mozilla.github.io/server-side-tls/ssl-config-generator/?server=haproxy
  ssl-default-bind-options no-sslv3
  ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS

defaults
  mode tcp
  maxconn 4000
  log global
  option tcplog
  option dontlognull
  option tcp-smart-accept
  timeout check 5s
  timeout connect 9s
  timeout client 10s
  timeout queue 5m
  timeout server 10s
  timeout tunnel 1h
  timeout client-fin 10s

userlist controller
  user client insecure-password cert
EOF

systemctl enable haproxy.service
systemctl start haproxy.service

Dataplane

Build Dataplane

Build Dataplane from Source

OS=linux
ARCH=amd64
GOVERSION="1.14.2"
curl -L https://dl.google.com/go/"go${GOVERSION}.${OS}-${ARCH}".tar.gz | tar xzv -C /usr/local/

export GOROOT=/usr/local/go/
export GOPATH="$HOME/go"
export PATH="$GOPATH/bin:$GOROOT/bin:$PATH"
mkdir -p "$GOPATH"

DATAPLANEAPI_REF=0553265

git clone https://github.com/haproxytech/dataplaneapi.git
pushd dataplaneapi
git checkout -b build-me ${DATAPLANEAPI_REF}
make build
mv build/dataplaneapi /usr/local/bin/
popd

rm -rf dataplaneapi

Build using Docker

docker run -it --rm -v $(pwd):/out golang bash -c \
  "git clone https://github.com/haproxytech/dataplaneapi.git && \
  cd dataplaneapi && \
  git checkout -b 0553265 0553265 && \
  make && \
  mv build/dataplaneapi /out"

mv dataplaneapi /usr/local/bin/

Self Signed SSL Configuration

# If you have go installed
go get -u github.com/cloudflare/cfssl/cmd/cfssl
go get -u github.com/cloudflare/cfssl/cmd/cfssljson

# Otherwise
curl -sL https://github.com/cloudflare/cfssl/releases/download/v1.4.1/cfssl_1.4.1_linux_amd64 -o /usr/local/bin/cfssl
curl -sL https://github.com/cloudflare/cfssl/releases/download/v1.4.1/cfssljson_1.4.1_linux_amd64 -o /usr/local/bin/cfssljson
chmod +x /usr/local/bin/cfssl*

cat > ca.json <<EOF
{
  "CN": "HAproxy Root CA",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
  {
    "C": "US",
    "ST": "California",
    "L": "Palo Alto",
    "O": "VMware",
    "OU": "Load Balancer API"
  }
 ]
}
EOF
cat > cfssl.json <<EOF
{
  "signing": {
    "default": {
      "expiry": "8760h"
    },
    "profiles": {
      "server": {
        "usages": [
          "signing",
          "digital signing",
          "key encipherment",
          "server auth"
        ],
        "expiry": "8760h"
      },
      "client": {
        "usages": [
          "signing",
          "digital signature",
          "key encipherment",
          "client auth"
        ],
        "expiry": "8760h"
      }
    }
  }
}
EOF
cat > haproxy.json <<EOF
{
  "CN": "dataplaneapi",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
  {
    "C": "US",
    "ST": "California",
    "L": "Palo Alto",
    "O": "VMware",
    "OU": "Load Balancer API"
  }
  ],
  "hosts": [
    "haproxy.lbapi.run.tanzu.vmware.com",
    "localhost"
  ]
}
EOF

cfssl gencert -initca ca.json | cfssljson -bare ca
cfssl gencert -ca ca.pem -ca-key ca-key.pem -config cfssl.json -profile=server haproxy.json | cfssljson -bare server
cfssl gencert -ca ca.pem -ca-key ca-key.pem -config cfssl.json -profile=client haproxy.json | cfssljson -bare client


cp server.pem /etc/haproxy/server.crt
cp server-key.pem /etc/haproxy/server.key
cp ca.pem /etc/haproxy/ca.crt


cat >> /etc/haproxy/haproxy.cfg <<EOF
program api
  command dataplaneapi --scheme=https --haproxy-bin=/usr/sbin/haproxy --config-file=/etc/haproxy/haproxy.cfg --reload-cmd="systemctl reload haproxy" --reload-delay=5 --tls-host=0.0.0.0 --tls-port=5556 --tls-ca=/etc/haproxy/ca.crt --tls-certificate=/etc/haproxy/server.crt --tls-key=/etc/haproxy/server.key --userlist=controller
  no option start-on-reload
EOF

systemctl reload haproxy

Validation

Make sure HAProxy process is up and running

# systemctl status haproxy
● haproxy.service - HAProxy Load Balancer
   Loaded: loaded (/lib/systemd/system/haproxy.service; disabled; vendor preset: enabled)
   Active: active (running) since Tue 2020-04-21 22:37:46 UTC; 9min ago
     Docs: man:haproxy(1)
           file:/usr/share/doc/haproxy/configuration.txt.gz
 Main PID: 4879 (haproxy)
    Tasks: 11 (limit: 1152)
   CGroup: /system.slice/haproxy.service
           ├─4879 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -S /var/run/haproxy-master.sock
           ├─4881 dataplaneapi --scheme=https --haproxy-bin=/usr/sbin/haproxy --config-file=/etc/haproxy/haproxy.cfg --reload-cmd=/usr/bin/systemctl
           └─4882 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -S /var/run/haproxy-master.sock

Apr 21 22:37:46 u3 systemd[1]: Starting HAProxy Load Balancer...
Apr 21 22:37:46 u3 haproxy[4879]: [NOTICE] 111/223746 (4879) : New program 'api' (4881) forked
Apr 21 22:37:46 u3 haproxy[4879]: [NOTICE] 111/223746 (4879) : New worker #1 (4882) forked
Apr 21 22:37:46 u3 systemd[1]: Started HAProxy Load Balancer.
Apr 21 22:37:47 u3 haproxy[4879]: time="2020-04-21T22:37:47Z" level=info msg="HAProxy Data Plane API v1.2.4 0553265.dev"
Apr 21 22:37:47 u3 haproxy[4879]: time="2020-04-21T22:37:47Z" level=info msg="Build from: https://github.com/haproxytech/dataplaneapi.git"
Apr 21 22:37:47 u3 haproxy[4879]: time="2020-04-21T22:37:47Z" level=info msg="Build date: 2020-04-21T21:55:54"

Make sure the HAproxy Dataplane is up and running

# curl -s -u 'client:cert' --cacert ./ca.pem --key ./client-key.pem --cert ./client.pem https://localhost:5556/v2/services/haproxy | jq .
[
  {
    "description": "Returns an array of all configured sites.",
    "title": "Return an array of sites",
    "url": "/services/haproxy/sites"
  },
  {
    "description": "Returns a list of HAProxy configuration transactions. Transactions can be filtered by their status.",
    "title": "Return list of HAProxy configuration transactions.",
    "url": "/services/haproxy/transactions"
  },
  {
    "description": "Returns a list of endpoints to be used for advanced configuration of HAProxy objects.",
    "title": "Return list of HAProxy advanced configuration endpoints",
    "url": "/services/haproxy/configuration"
  },
  {
    "description": "Returns a list of endpoints to be used for advanced runtime settings of HAProxy objects.",
    "title": "Return list of HAProxy advanced runtime endpoints",
    "url": "/services/haproxy/runtime"
  },
  {
    "description": "Returns a list of HAProxy reloads.",
    "title": "Return list of HAProxy Reloads.",
    "url": "/services/haproxy/reloads"
  },
  {
    "description": "Returns a list of HAProxy stats endpoints.",
    "title": "Return list of HAProxy stats endpoints",
    "url": "/services/haproxy/stats"
  }
]

Make sure HAProxy is functioning

# Set AnyIP for local table
ip r add local 192.168.35.0/24 dev lo

# Deploy backends
docker run -d --rm --name "haproxy-backend-0" hashicorp/http-echo -listen ":12221" -text "This is backend-0 on port 12221"
docker run -d --rm --name "haproxy-backend-1" hashicorp/http-echo -listen ":12221" -text "This is backend-1 on port 12221"

# docker inspect haproxy-backend-0 | jq -cr '.[0].NetworkSettings.Networks.bridge.IPAddress'
172.17.0.2
# docker inspect haproxy-backend-1 | jq -cr '.[0].NetworkSettings.Networks.bridge.IPAddress'
172.17.0.3


# Update HAProxy config with frontend and backend
# Frontend will be listening on one IP from the Floating IP Range
cat >> /etc/haproxy/haproxy.cfg <<EOF
frontend echo
  mode tcp
  bind 192.168.35.23:12221 name lb
  option tcplog
  default_backend echo

backend echo
  mode tcp
  balance first
  default-server inter 10s downinter 10s rise 5 fall 3 slowstart 120s maxconn 1000 maxqueue 256 weight 100
  server s0 172.17.0.2:12221 check verify none
  server s1 172.17.0.3:12221 check verify none
  http-check expect status 200
EOF

systemctl reload haproxy

# Make sure the Floating IP is being served
root@u3:/home/ubuntu# curl 192.168.35.23:12221
This is backend-0 on port 12221


# Optional: If you havee another machine in the same subnet

root@u5:/home/ubuntu# ip r add 192.168.35.23/32 via [IP Address of the machine where HAProxy is deployed]
root@u5:/home/ubuntu# curl 192.168.35.23:12221
This is backend-0 on port 12221

Make sure HAProxy works with Prometheus

Add prometheus in frontend

cat > /etc/haproxy/haproxy.cfg <<EOF

frontend stats
   mode http
   bind *:8404
   option http-use-htx
   http-request use-service prometheus-exporter if { path /metrics }
   stats enable
   stats uri /stats
   stats refresh 10s
EOF


systemctl reload haproxy

Deploy Prometheus

cat > prometheus.yml <<EOF
global:
  scrape_interval:     15s # By default, scrape targets every 15 seconds.
scrape_configs:
  - job_name: 'haproxy'
    static_configs:
    - targets: ['localhost:8404']
EOF


docker run --network host -p 9090:9090 -d -v $PWD/prometheus.yml:/etc/prometheus/prometheus.yml prom/prometheus

Go to [ip]:9090 where IP belongs to the machine HAProxy and Prometheus are deployed.
Create a Graph with an expression like "haproxy_frontend_http_requests_total". You should be able to see something like the following

References

Installation: https://dev.to/tmidi/getting-started-with-haproxy-install-from-code-source-5c4g
- includes make Options
haproxy user/group setup: https://github.com/docker-library/haproxy/issues/6
haproxy configuration documentation: http://cbonte.github.io/haproxy-dconv/2.1/configuration.html
cfssl key generation: https://medium.com/@rob.blackbourn/how-to-use-cfssl-to-create-self-signed-certificates-d55f76ba5781
haproxy with prometheus: https://www.haproxy.com/blog/haproxy-exposes-a-prometheus-metrics-endpoint/

Install Manually

Images

Ubuntu Images

18.04 Bionic

19.10 Eoan

Debian Images

Debian 10.3

Photon Images

Photon 3.0

CentOS Images

CentOS 7

CentOS 8

Notes

Default repository

Ubuntu

For Ubuntu 18.04 Bionic

For Ubuntu 19.10 Eoan

For Debian 10.3

For Photon 3.0

For CentOS

For CentOS 7

For CentOS 8

Build from Source

Preparation

Build

Configuration

systemd

haproxy.cfg

Dataplane

Build Dataplane

Build Dataplane from Source

Build using Docker

Self Signed SSL Configuration

Validation

Make sure HAProxy process is up and running

Make sure the HAproxy Dataplane is up and running

Make sure HAProxy is functioning

Make sure HAProxy works with Prometheus

Add prometheus in frontend

Deploy Prometheus

References