How to set up Tyk Gateway with Redis CRDB

--- tags: devops --- # How to set up Tyk Gateway with Redis CRDB User sessions are managed via Redis in Tyk, and in order for this to work across regions, we need to set up a Conflict-Free Replicated Database using Redis first. - Installing Redis - create a redislabs.com account - download the binary for redis enterprise Ubuntu 18.04 - create two servers via AWS (r5.large, memory optimised since Redis uses memory, > 10 GB storage) - one in Singapore, one in US east/west - ensure that security groups allow connections from anywhere - ensure that nothing is running on port 53 - lsof -i :53, AND sudo lsof -i :53 - if there's something running (eg. systemd-resolved), stop it safely - sftp into each server and upload the binary - run the binary - this binary installs redis and nginx and other helper libs - Add an elastic IP - later on when you add the public IP you won't have to modify it in the cluster settings even if you spin down these nodes - If using a VPC - add each node to the vpc during creation - add internet gateway for the vpc - set the default route to point to internet gw in routing table - add elastic ip for each node - set VPC DNS hostnames true - Configuring Redis - For each cluster master server - go to the redis dashboard at public IP:8443 (eg. 13.229.182.48:8443) - click Setup - Under IP Address Usage, add node's Elastic IP (eg. 13.229.182.48) - Create a cluster with a DNS-readable name (eg. tyk-redis.singapore) - Enable private & public endpoints support - Add cluster key (if any, otherwise just skip) - Add master login credentials - Create a CRDB - Go to ONE server and login with master credentials - Create a Geo replicated DB - Add participating cluster - https://clustername:9443 (eg. https://tyk-redis.singapore:9443) - for each participating cluster, modify /etc/hosts to point to the correct IP for BOTH servers - eg. in Singapore server - 127.0.0.1 tyk-redis.singapore - 127.0.0.1 redis-11111.tyk-redis.singapore - 3.215.185.96 tyk-secondary-redis.us - 3.215.185.96 redis-11111.tyk-secondary.redis.us - eg. in US server - 127.0.0.1 tyk-redis-secondary.us - 127.0.0.1 redis-11111.tyk-redis-secondary.us - 13.229.182.48 tyk-redis.singapore - 13.229.182.48 redis-11111.tyk-redis.singapore - If everything is set up properly, you should see a green tick and a green synced icon in the redis dashboard - Test it by using redis cli, setting a key in one redis cluster, and checking if its readable in the other cluster - Point Route 53 to the redis clusters - eg. tyk-redis.gateway.tor.us - Installing Tyk Gateway - Create two Ubuntu 18.04 servers, one in Singapore and one in US, t3a.medium, 30 GB storage, security group accessible anywhere - Install docker, nginx, certbot plugin for nginx - Ensure docker can be run without sudo - clone https://github.com/torusresearch/tyk-docker - modify docker-compose.yml to exclude local tyk-redis and tyk-mongo instances - Ensure that only the Singapore server is running the dashboard docker container - Modify tyk-docker/confs/pump.conf - Point to the correct redis endpoint - eg. Host: tyk-redis.gateway.tor.us OR tyk-secondary-redis.gateway.tor.us, Port: 11111, Password: VerySecureRedisPassword - Point to the correct mongodb endpoint - eg. mongodb_url: mongodb://tyk:tyk123@52.77.243.83:27017/TYKANALYTICS - Modify tyk-docker/confs/settings.json - Point to the correct mongodb endpoint - Modify tyk-docker/confs/tyk.conf - Point to the correct redis endpoint - Install docker-compose - Run `docker-compose up -f docker-compose.prod.yml -f docker-prod.yml up -d` - Check logs for errors - Add nginx server conf with certbot to ensure that port 8080 (tyk-gateway) and port 5050 (torus-dashboard-backend, SG-only) are exposed and routed via their DNS names - this is required since the torus-dashboard front-end needs a https endpoint to avoid mixed content issues - also, point the torus-dashboard config to the torus-dashboard-backend DNS name - also make sure nginx.conf worker_connections is 100000 - Point Route 53 to the gateways - eg. us.gateway.tor.us - Configure AWS route 53 latency and health based routing - Point tyk.gateway.tor.us to both the Singapore and US gateway servers - Use latency based routing - Use health endpoints - If doesn't exist, create health endpoints that point http://us.gateway.tor.us/hello and http://sg.gateway.tor.us/hello -