Try   HackMD

Install Manually

This section describes how to install HAProxy and the DataPlane API manually.

Images

Ubuntu Images

Source: https://cloud-images.ubuntu.com/releases/

18.04 Bionic

ova: https://cloud-images.ubuntu.com/bionic/current/bionic-server-cloudimg-amd64.ova

19.10 Eoan

ova: https://cloud-images.ubuntu.com/releases/eoan/release-20200407/ubuntu-19.10-server-cloudimg-amd64.ova

Debian Images
Debian 10.3

Source: https://www.debian.org/distrib/netinst
iso: https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-10.3.0-amd64-netinst.iso

Photon Images
Photon 3.0

Source: https://github.com/vmware/photon/wiki/Downloading-Photon-OS
ova: http://dl.bintray.com/vmware/photon/3.0/Rev2/ova/photon-hw11-3.0-9355405.ova

CentOS Images

Source: https://wiki.centos.org/Download

CentOS 7

iso: http://mirror.fileplanet.com/centos/7.7.1908/isos/x86_64/CentOS-7-x86_64-Minimal-1908.iso

CentOS 8

iso: http://mirror.sjc02.svwh.net/centos/8.1.1911/isos/x86_64/CentOS-8.1.1911-x86_64-boot.iso

Notes

  1. HAProxy >= 1.9 is required for dataplane api
  2. Except Photon 3, other distros default repositories don't have the latest HAProxy(2.1)
  3. Ubuntu 19.10 contains 2.0.5
  4. dataplaneapi needs to be built manually
  5. Photon will soon include an RPM for dataplane api binary TKG-1372

Default repository

Ubuntu
For Ubuntu 18.04 Bionic

Install from default repositories

apt install -y haproxy

However, the default repositories usually don't contain the latest releases. To check the current haproxy version in default repository:

ubuntu@u3:~$ apt show haproxy 2>/dev/null | grep Version
Version: 1.8.8-1ubuntu0.10
For Ubuntu 19.10 Eoan

Install from default repositories

apt install -y haproxy

However, the default repositories usually don't contain the latest releases. To check the current haproxy version in default repository:

# apt show haproxy 2>/dev/null | grep Version
Version: 2.0.5-1ubuntu0.3
For Debian 10.3

Install from default repositories

apt install -y haproxy

However, the default repositories usually don't contain the latest releases. To check the current haproxy version in default repository:

# apt show haproxy 2>/dev/null | grep Version
Version: 1.8.19-1+deb10u2
For Photon 3.0

Install from default repositories

tdnf install haproxy-2.1.0
For CentOS
For CentOS 7

Install from default repositories

yum install -y haproxy.x86_64

However, the default repositories usually don't contain the latest releases. To check the current haproxy version in default repository:

# yum info haproxy | grep Version
Version     : 1.5.18
For CentOS 8

Install from default repositories

yum install -y haproxy.x86_64

However, the default repositories usually don't contain the latest releases. To check the current haproxy version in default repository:

# yum info haproxy | grep Version
Version      : 1.8.15

Build from Source

Preparation
  • Ubuntu
apt update
apt install -y make gcc perl libpcre3-dev zlib1g-dev libssl-dev libsystemd-dev

# Optional. will be used in validation section
apt install -y jq docker.io
  • Photon
tdnf update
# diffutils is for cmp command: /bin/sh: cmp: command not found
# glibc-devel provides the unix headers: unistd.h: No such file or directory
# binutils provides the as assembler, which is required by zlib
tdnf install -y tar git make gcc perl pcre-devel zlib-devel openssl-devel systemd-devel diffutils glibc-devel binutils

# Optional. will be used in validation section
tdnf install -y jq docker
systemctl start docker
iptables -P INPUT ACCEPT
iptables -P FORWARD ACCEPT
iptables -P OUTPUT ACCEPT
  • CentOS
yum update
yum install -y make git gcc perl pcre-devel.x86_64 zlib-devel.x86_64 openssl-devel.x86_64 systemd-devel.x86_64

# Optional. will be used in validation section
# ref: https://medium.com/@gchandra/install-jq-on-centos-7-459dd650baa3
yum -y install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm && yum install -y jq
systemctl start docker
Build
HAPROXY_VEERSION="2.1"
HAPROXY_TARBALL="haproxy-2.1.4"

curl -s -L http://www.haproxy.org/download/"${HAPROXY_VEERSION}"/src/"${HAPROXY_TARBALL}".tar.gz | tar xzv

pushd "${HAPROXY_TARBALL}"
make TARGET=linux-glibc USE_PCRE=1 USE_OPENSSL=1 USE_ZLIB=1 USE_SYSTEMD=1 EXTRA_OBJS="contrib/prometheus-exporter/service-prometheus.o"
make install
cp haproxy /usr/sbin/
popd

groupadd haproxy
useradd -s /usr/sbin/nologin --system -g haproxy --home-dir=/var/lib/haproxy haproxy

mkdir /var/lib/haproxy
chown -R haproxy:haproxy /var/lib/haproxy

curl -s https://raw.githubusercontent.com/horms/haproxy/master/doc/configuration.txt | gzip -c > configuration.txt.gz
mkdir -p /usr/share/doc/haproxy
mv configuration.txt.gz /usr/share/doc/haproxy/configuration.txt.gz

Configuration

systemd
cat > /lib/systemd/system/haproxy.service <<"EOF"
[Unit]
Description=HAProxy Load Balancer
Documentation=man:haproxy(1)
Documentation=file:/usr/share/doc/haproxy/configuration.txt.gz
# allows us to do millisecond level restarts without triggering alert in Systemd
StartLimitInterval=0
StartLimitBurst=0
After=network.target syslog.service
Wants=syslog.service

[Service]
EnvironmentFile=-/etc/default/haproxy
EnvironmentFile=-/etc/sysconfig/haproxy
Environment="CONFIG=/etc/haproxy/haproxy.cfg" "PIDFILE=/var/run/haproxy.pid" "EXTRAOPTS=-S /var/run/haproxy-master.sock"
ExecStartPre=/usr/sbin/haproxy -f $CONFIG -c -q $EXTRAOPTS
ExecStart=/usr/sbin/haproxy -Ws -f $CONFIG -p $PIDFILE $EXTRAOPTS
ExecReload=/usr/sbin/haproxy -f $CONFIG -c -q $EXTRAOPTS
ExecReload=/bin/kill -USR2 $MAINPID
KillMode=mixed
Restart=always
SuccessExitStatus=143
Type=notify

[Install]
WantedBy=multi-user.target
EOF
haproxy.cfg
mkdir -p /etc/haproxy

cat > /etc/haproxy/haproxy.cfg <<"EOF"
global
  master-worker
  log stdout format raw local0 info

  stats socket /run/haproxy.sock user haproxy group haproxy mode 660 level admin expose-fd listeners
  stats timeout 30s

  chroot /var/lib/haproxy
  user haproxy
  group haproxy

  # Default SSL material locations
  ca-base /etc/ssl/certs
  crt-base /etc/ssl/private

  # Default ciphers to use on SSL-enabled listening sockets.
  # For more information, see ciphers(1SSL). This list is from:
  #  https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
  # An alternative list with additional directives can be obtained from
  #  https://mozilla.github.io/server-side-tls/ssl-config-generator/?server=haproxy
  ssl-default-bind-options no-sslv3
  ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS

defaults
  mode tcp
  maxconn 4000
  log global
  option tcplog
  option dontlognull
  option tcp-smart-accept
  timeout check 5s
  timeout connect 9s
  timeout client 10s
  timeout queue 5m
  timeout server 10s
  timeout tunnel 1h
  timeout client-fin 10s

userlist controller
  user client insecure-password cert
EOF

systemctl enable haproxy.service
systemctl start haproxy.service

Dataplane

Build Dataplane
Build Dataplane from Source
OS=linux
ARCH=amd64
GOVERSION="1.14.2"
curl -L https://dl.google.com/go/"go${GOVERSION}.${OS}-${ARCH}".tar.gz | tar xzv -C /usr/local/

export GOROOT=/usr/local/go/
export GOPATH="$HOME/go"
export PATH="$GOPATH/bin:$GOROOT/bin:$PATH"
mkdir -p "$GOPATH"

DATAPLANEAPI_REF=0553265

git clone https://github.com/haproxytech/dataplaneapi.git
pushd dataplaneapi
git checkout -b build-me ${DATAPLANEAPI_REF}
make build
mv build/dataplaneapi /usr/local/bin/
popd

rm -rf dataplaneapi
Build using Docker
docker run -it --rm -v $(pwd):/out golang bash -c \
  "git clone https://github.com/haproxytech/dataplaneapi.git && \
  cd dataplaneapi && \
  git checkout -b 0553265 0553265 && \
  make && \
  mv build/dataplaneapi /out"

mv dataplaneapi /usr/local/bin/
Self Signed SSL Configuration
# If you have go installed
go get -u github.com/cloudflare/cfssl/cmd/cfssl
go get -u github.com/cloudflare/cfssl/cmd/cfssljson

# Otherwise
curl -sL https://github.com/cloudflare/cfssl/releases/download/v1.4.1/cfssl_1.4.1_linux_amd64 -o /usr/local/bin/cfssl
curl -sL https://github.com/cloudflare/cfssl/releases/download/v1.4.1/cfssljson_1.4.1_linux_amd64 -o /usr/local/bin/cfssljson
chmod +x /usr/local/bin/cfssl*

cat > ca.json <<EOF
{
  "CN": "HAproxy Root CA",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
  {
    "C": "US",
    "ST": "California",
    "L": "Palo Alto",
    "O": "VMware",
    "OU": "Load Balancer API"
  }
 ]
}
EOF
cat > cfssl.json <<EOF
{
  "signing": {
    "default": {
      "expiry": "8760h"
    },
    "profiles": {
      "server": {
        "usages": [
          "signing",
          "digital signing",
          "key encipherment",
          "server auth"
        ],
        "expiry": "8760h"
      },
      "client": {
        "usages": [
          "signing",
          "digital signature",
          "key encipherment",
          "client auth"
        ],
        "expiry": "8760h"
      }
    }
  }
}
EOF
cat > haproxy.json <<EOF
{
  "CN": "dataplaneapi",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
  {
    "C": "US",
    "ST": "California",
    "L": "Palo Alto",
    "O": "VMware",
    "OU": "Load Balancer API"
  }
  ],
  "hosts": [
    "haproxy.lbapi.run.tanzu.vmware.com",
    "localhost"
  ]
}
EOF

cfssl gencert -initca ca.json | cfssljson -bare ca
cfssl gencert -ca ca.pem -ca-key ca-key.pem -config cfssl.json -profile=server haproxy.json | cfssljson -bare server
cfssl gencert -ca ca.pem -ca-key ca-key.pem -config cfssl.json -profile=client haproxy.json | cfssljson -bare client


cp server.pem /etc/haproxy/server.crt
cp server-key.pem /etc/haproxy/server.key
cp ca.pem /etc/haproxy/ca.crt


cat >> /etc/haproxy/haproxy.cfg <<EOF
program api
  command dataplaneapi --scheme=https --haproxy-bin=/usr/sbin/haproxy --config-file=/etc/haproxy/haproxy.cfg --reload-cmd="systemctl reload haproxy" --reload-delay=5 --tls-host=0.0.0.0 --tls-port=5556 --tls-ca=/etc/haproxy/ca.crt --tls-certificate=/etc/haproxy/server.crt --tls-key=/etc/haproxy/server.key --userlist=controller
  no option start-on-reload
EOF

systemctl reload haproxy

Validation

Make sure HAProxy process is up and running
# systemctl status haproxy
● haproxy.service - HAProxy Load Balancer
   Loaded: loaded (/lib/systemd/system/haproxy.service; disabled; vendor preset: enabled)
   Active: active (running) since Tue 2020-04-21 22:37:46 UTC; 9min ago
     Docs: man:haproxy(1)
           file:/usr/share/doc/haproxy/configuration.txt.gz
 Main PID: 4879 (haproxy)
    Tasks: 11 (limit: 1152)
   CGroup: /system.slice/haproxy.service
           ├─4879 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -S /var/run/haproxy-master.sock
           ├─4881 dataplaneapi --scheme=https --haproxy-bin=/usr/sbin/haproxy --config-file=/etc/haproxy/haproxy.cfg --reload-cmd=/usr/bin/systemctl
           └─4882 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -S /var/run/haproxy-master.sock

Apr 21 22:37:46 u3 systemd[1]: Starting HAProxy Load Balancer...
Apr 21 22:37:46 u3 haproxy[4879]: [NOTICE] 111/223746 (4879) : New program 'api' (4881) forked
Apr 21 22:37:46 u3 haproxy[4879]: [NOTICE] 111/223746 (4879) : New worker #1 (4882) forked
Apr 21 22:37:46 u3 systemd[1]: Started HAProxy Load Balancer.
Apr 21 22:37:47 u3 haproxy[4879]: time="2020-04-21T22:37:47Z" level=info msg="HAProxy Data Plane API v1.2.4 0553265.dev"
Apr 21 22:37:47 u3 haproxy[4879]: time="2020-04-21T22:37:47Z" level=info msg="Build from: https://github.com/haproxytech/dataplaneapi.git"
Apr 21 22:37:47 u3 haproxy[4879]: time="2020-04-21T22:37:47Z" level=info msg="Build date: 2020-04-21T21:55:54"
Make sure the HAproxy Dataplane is up and running
# curl -s -u 'client:cert' --cacert ./ca.pem --key ./client-key.pem --cert ./client.pem https://localhost:5556/v2/services/haproxy | jq .
[
  {
    "description": "Returns an array of all configured sites.",
    "title": "Return an array of sites",
    "url": "/services/haproxy/sites"
  },
  {
    "description": "Returns a list of HAProxy configuration transactions. Transactions can be filtered by their status.",
    "title": "Return list of HAProxy configuration transactions.",
    "url": "/services/haproxy/transactions"
  },
  {
    "description": "Returns a list of endpoints to be used for advanced configuration of HAProxy objects.",
    "title": "Return list of HAProxy advanced configuration endpoints",
    "url": "/services/haproxy/configuration"
  },
  {
    "description": "Returns a list of endpoints to be used for advanced runtime settings of HAProxy objects.",
    "title": "Return list of HAProxy advanced runtime endpoints",
    "url": "/services/haproxy/runtime"
  },
  {
    "description": "Returns a list of HAProxy reloads.",
    "title": "Return list of HAProxy Reloads.",
    "url": "/services/haproxy/reloads"
  },
  {
    "description": "Returns a list of HAProxy stats endpoints.",
    "title": "Return list of HAProxy stats endpoints",
    "url": "/services/haproxy/stats"
  }
]
Make sure HAProxy is functioning
# Set AnyIP for local table
ip r add local 192.168.35.0/24 dev lo

# Deploy backends
docker run -d --rm --name "haproxy-backend-0" hashicorp/http-echo -listen ":12221" -text "This is backend-0 on port 12221"
docker run -d --rm --name "haproxy-backend-1" hashicorp/http-echo -listen ":12221" -text "This is backend-1 on port 12221"

# docker inspect haproxy-backend-0 | jq -cr '.[0].NetworkSettings.Networks.bridge.IPAddress'
172.17.0.2
# docker inspect haproxy-backend-1 | jq -cr '.[0].NetworkSettings.Networks.bridge.IPAddress'
172.17.0.3


# Update HAProxy config with frontend and backend
# Frontend will be listening on one IP from the Floating IP Range
cat >> /etc/haproxy/haproxy.cfg <<EOF
frontend echo
  mode tcp
  bind 192.168.35.23:12221 name lb
  option tcplog
  default_backend echo

backend echo
  mode tcp
  balance first
  default-server inter 10s downinter 10s rise 5 fall 3 slowstart 120s maxconn 1000 maxqueue 256 weight 100
  server s0 172.17.0.2:12221 check verify none
  server s1 172.17.0.3:12221 check verify none
  http-check expect status 200
EOF

systemctl reload haproxy

# Make sure the Floating IP is being served
root@u3:/home/ubuntu# curl 192.168.35.23:12221
This is backend-0 on port 12221


# Optional: If you havee another machine in the same subnet

root@u5:/home/ubuntu# ip r add 192.168.35.23/32 via [IP Address of the machine where HAProxy is deployed]
root@u5:/home/ubuntu# curl 192.168.35.23:12221
This is backend-0 on port 12221
Make sure HAProxy works with Prometheus
Add prometheus in frontend
cat > /etc/haproxy/haproxy.cfg <<EOF

frontend stats
   mode http
   bind *:8404
   option http-use-htx
   http-request use-service prometheus-exporter if { path /metrics }
   stats enable
   stats uri /stats
   stats refresh 10s
EOF


systemctl reload haproxy
Deploy Prometheus
cat > prometheus.yml <<EOF
global:
  scrape_interval:     15s # By default, scrape targets every 15 seconds.
scrape_configs:
  - job_name: 'haproxy'
    static_configs:
    - targets: ['localhost:8404']
EOF


docker run --network host -p 9090:9090 -d -v $PWD/prometheus.yml:/etc/prometheus/prometheus.yml prom/prometheus
  • Go to [ip]:9090 where IP belongs to the machine HAProxy and Prometheus are deployed.
  • Create a Graph with an expression like "haproxy_frontend_http_requests_total". You should be able to see something like the following
    HAproxy integration with Prometheus

References