# HAProxy troubleshooting for TSMC ## Options 1. **Adject VM configration - fewer core but higher frequency.** https://docs.openshift.com/container-platform/3.11/scaling_performance/routing_optimization.html#scaling-performance-optimizing-router-haproxy-cpu-affinity Use small scale VM and set-up cpu affinity for haproxy 2. Setup the dedicated router node only for HAProxy and use small scale VM sepc. 3. Gather more information to deep dive this issue check section above. 4. If current connection is higher than Maximum Incress max connection ``` oc adm router --max-connections=<MAX_CONNECTION> ``` 5. Incresse higher thread for haproxy ``` oc adm router --threads = 4 ``` The number may vary for different envrionment, should implement the performance test for different value. ## To gather more information... 1. Enable debug log on haproxy ``` oc set env dc/router ROUTER_SYSLOG_ADDRESS=<dest_ip:dest_port> ROUTER_LOG_LEVEL=<level> ``` [more information](https://docs.okd.io/3.11/admin_guide/router.html#viewing-logs) 2. Compare haproxy config template for different VM environment ``` oc rsh router-2-40fc3 cat haproxy-config.template > haproxy-config.template ``` 3. Compare haproxy config for different VM envrionment ``` oc rsh router-2-40fc3 cat haproxy.config > haproxy.config ``` 4. Compare environment variable for haproxy on different vm environment ``` oc rsh router-2-40fc3 printenv ``` 5. Enable HaProxy build-in stats page ``` STATS_USERNAME=$(oc set env deploymentconfigs/router --all --list | grep -e "STATS_USERNAME" | cut -d'=' -f2) STATS_PASSWORD=$(oc set env deploymentconfigs/router --all --list | grep -e "STATS_PASSWORD" | cut -d'=' -f2) $STATS_USERNAME:$STATS_PASSWORD@<router-ip>:1936/metrics ``` for more information: [link](https://access.redhat.com/articles/3510971) 6. Get the information about version and build options on different environment ``` oc rsh router-2-xsbk9 haproxy -vv > haproxy-vv.txt ``` 7. Export Router statistic into a csv file *On bation* ``` $ cmd="echo 'show stat' | socat - UNIX-CONNECT:/var/lib/haproxy/run/haproxy.sock" $ routerPod=$(oc get pods --selector="router=router" \ --template="{{with index .items 0}}{{.metadata.name}}{{end}}") $ oc exec $routerPod -- bash -c "$cmd" > haproxy_stats.csv ``` --- tune.maxrewrite 8192 tune.bufsize 32768 tune.bufsize指的是haproxy再處理每次session時候會預留的buffer size, 單位是byte 所以如果會用到http request size比較大的場景的話可以把這個數值加大 目前設定是32kb 應該足以應付大多數場景 tune.maxrewrite是指當haproxy轉送時如果需要進行headder改寫的操作時候，需要預留多少byte的空間，預設是bufsize的一半。單位是byte 設定太高的話會讓request除了header以外的空間被壓縮 而設定太低的話會造成後續如果要做headder rewrite或是append時出現異常 但通常1024就已經非常足夠，實務上並不會在header塞那麼多資料 跟據您提供的haproxy statistic wrew [LFBS]: cumulative number of failed header rewriting warnings 這個wrew數值都是零 所以沒有出現異常 maxconn 最大允許的連線數量 單位是個數 這三個參數沒辦法直接加減後比較來判斷haproxy出錯的原因 --- * Related KB https://access.redhat.com/solutions/5869341