# C2C Communication Implementation Notes ## 1. Introduction ### 1.1. [App Service Discovery](https://docs.cloudfoundry.org/devguide/deploy-apps/cf-networking.html#discovery) > With app service discovery, apps pushed to Cloud Foundry Application Runtime can establish container-to-container communications through a known route served by internal BOSH DNS. This allows front end apps to easily connect with back end apps. ### 1.2. [**Reference Implementation**](https://github.com/cloudfoundry-attic/cf-networking-examples/blob/master/docs/c2c-with-service-discovery.md) | Purpose | Command | Reference Use Case | | ------------------------------------------------------------ | ------------------------------ | ------------------ | | Frontend connects to a specific backend. | `cf add-network-policy` | Use case 1 | | Frontend connects to multiple backends for **load balance**. | `cf create-route`, `map-route` | Use case 2 | ### 1.3. Concept - [Microservice Architecture](https://microservices.io/) - Route: An app’s route is the URL that it runs at (or an application address). - Domain: A group of routes, default internal domain: `apps.internal` --- ## 2. Before pushing applications ### 2.1. Check `domains`, `routes`, `network-policies` #### Domains ```bash ~$ cf domains Getting domains in org brilliankingsman-main-org as brilliankingsman@gmail.com... name status type details cfapps.io shared cf-tcpapps.io shared tcp apps.internal shared internal ``` #### Routes ```bash ~$ cf routes Getting routes for org brilliankingsman-main-org / space development as brilliankingsman@gmail.com ... space host domain port path type apps service No routes found ``` #### Network Policy ```bash ~$ cf network-policies Listing network policies in org brilliankingsman-main-org / space development as brilliankingsman@gmail.com... source destination protocol ports destination space destination org ``` ### 2.2. Default Configuration of 3 Applications: #### Check `env` and `routes` | Application Name | `env` | `routes` | | ---------------- | ------------------------------------------------------------ | -------------------------------- | | `frontend` | `GOPACKAGENAME: example-apps/cats-and-dogs/frontend` | | | `backend-b` | `GOPACKAGENAME: example-apps/cats-and-dogs-with-service-discovery/backend-b`<br />`CATS_PORTS: 7007,7008,7009`<br />`UDP_PORTS: 9001,9002,9003` | | | `backend-a` | `GOPACKAGENAME: example-apps/cats-and-dogs-with-service-discovery/backend-a`<br />`CATS_PORTS: 7007,7008,7009`<br />`UDP_PORTS: 9001,9002,9003` | `route: backend-a.apps.internal` | #### Linked Pictures: | Application Name | Picture url | | ---------------- | ------------------------------------------------------------ | | `backend-a` | https://i.imgur.com/1uYroRF.gif | | `backend-b` | https://media.giphy.com/media/FElR3ylsj6PTy/giphy.gif | | `frontend` | https://i2.kym-cdn.com/photos/images/original/000/234/765/b7e.jpg | #### In-app routing in `frondend`: ```go func main() { systemPortString := os.Getenv("PORT") systemPort, err := strconv.Atoi(systemPortString) if err != nil { log.Fatal("invalid required env var PORT") } mux := http.NewServeMux() mux.Handle("/proxy/", &HttpDemoHandler{}) mux.Handle("/udp-test/", &UDPDemoHandler{}) mux.Handle("/", &HomePageHandler{}) http.ListenAndServe(fmt.Sprintf("0.0.0.0:%d", systemPort), mux) } ``` --- ## 3. After pushing the applications ### 3.1. Default `env` of 3 applications (focus on key features) #### backend-b ```bash System-Provided: { "application_name": "backend-b-bkm", "application_uris": [ "backend-b-bkm.apps.internal" ], ... "uris": [ "backend-b-bkm.apps.internal" ], ... } User-Provided: CATS_PORTS: 7007,7008,7009 GOPACKAGENAME: example-apps/cats-and-dogs-with-service-discovery/backend-b UDP_PORTS: 9001,9002,9003 No running env variables have been set No staging env variables have been set ``` - if there is no domain specified during the push, the `uri`/`application_uris` would be `backend-b-bkm.cfapps.io` #### backend-a ```bash System-Provided: { "application_name": "backend-a-bkm", "application_uris": [ "backend-a.apps.internal" ], ... "uris": [ "backend-a.apps.internal" ], ... } User-Provided: CATS_PORTS: 7007,7008,7009 GOPACKAGENAME: example-apps/cats-and-dogs-with-service-discovery/backend-b UDP_PORTS: 9001,9002,9003 No running env variables have been set No staging env variables have been set ``` #### frontend ```bash System-Provided: { "application_name": "frontend-bkm", "application_uris": [ "frontend-bkm.cfapps.io" ], ... "uris": [ "frontend-bkm.cfapps.io" ], ... } User-Provided: GOPACKAGENAME: example-apps/cats-and-dogs/frontend No running env variables have been set No staging env variables have been set ``` ### 3.2. Routes ```bash ~$ cf routes Getting routes for org brilliankingsman-main-org / space development as brilliankingsman@gmail.com ... space host domain port path type apps service development frontend-bkm cfapps.io frontend-bkm development backend-a apps.internal backend-a-bkm development backend-b-bkm apps.internal backend-b-bkm ``` ### 3.3. Network Policy ```bash ~$ cf network-policies Listing network policies in org brilliankingsman-main-org / space development as brilliankingsman@gmail.com... source destination protocol ports destination space destination org ``` --- ## 4. After setting ports in environment variables Specifying ports for c2c communication based on the allowed ports. #### backend-a - Command ```bash cf set-env backend-a-bkm CATS_PORTS "7007,7008" cf set-env backend-a-bkm UDP_PORTS "9003,9004" cf restage bakend-a-bkm ``` - `env` ```bash User-Provided: CATS_PORTS: 7007,7008 GOPACKAGENAME: example-apps/cats-and-dogs-with-service-discovery/backend-a UDP_PORTS: 9003,9004 ``` #### backend-b - Command ```bash cf set-env backend-b-bkm CATS_PORTS "7007,7008" cf set-env backend-b-bkm UDP_PORTS "9003,9004" cf restage bakend-b-bkm ``` - `env` ```bash User-Provided: CATS_PORTS: 7007,7008 GOPACKAGENAME: example-apps/cats-and-dogs-with-service-discovery/backend-a UDP_PORTS: 9003,9004 ``` #### Trying the first connect - HTTP ``` request failed: Get http://backend-a.apps.internal:7007: dial tcp 10.248.52.205:7007: connect: connection refused request failed: Get http://backend-b-bkm.apps.internal:7007: dial tcp 10.246.253.193:7007: connect: connection refused ``` - UDP ``` request failed: read udp data: read udp 10.255.127.69:37806->10.248.52.205:9003: recvfrom: connection refused request failed: read udp data: read udp 10.255.127.69:50315->10.255.208.229:9003: recvfrom: connection refused ``` --- ## 5. Adding Network Policy: let frontend connect to a specific backend ```bash ~$ cf add-network-policy frontend-bkm --destination-app backend-a-bkm --port 7007 --protocol tcp Adding network policy from app frontend-bkm to app backend-a-bkm in org brilliankingsman-main-org / space development as brilliankingsman@gmail.com... OK ~$ cf add-network-policy frontend-bkm --destination-app backend-a-bkm --port 9003 --protocol udp Adding network policy from app frontend-bkm to app backend-a-bkm in org brilliankingsman-main-org / space development as brilliankingsman@gmail.com... OK ``` ### 5.1. Check Network Policy ```bash ~$ cf network-policies Listing network policies in org brilliankingsman-main-org / space development as brilliankingsman@gmail.com... source destination protocol ports destination space destination org frontend-bkm backend-a-bkm tcp 7007 development brilliankingsman-main-org frontend-bkm backend-a-bkm udp 9003 development brilliankingsman-main-org ``` ### 5.2. Check Domains ```bash ~$ cf domains Getting domains in org brilliankingsman-main-org as brilliankingsman@gmail.com... name status type details cfapps.io shared cf-tcpapps.io shared tcp apps.internal shared internal ``` ### 5.3. Check Routes ```bash ~$ cf routes Getting routes for org brilliankingsman-main-org / space development as brilliankingsman@gmail.com ... space host domain port path type apps service development frontend-bkm cfapps.io frontend-bkm development backend-a apps.internal backend-a-bkm development backend-b-bkm apps.internal backend-b-bkm ``` After adding the network policy, the connection should be avaliable. --- ## 6. let frontend connect to multiple backends ### 6.1. Create a route with a new hostname to contain multiple backends [command reference](https://cli.cloudfoundry.org/en-US/v6/create-route.html): `cf create-route SPACE DOMAIN [--hostname HOSTNAME] [--path PATH]` ```bash ~$ cf create-route development apps.internal --hostname nbackend-bkm Creating route nbackend-bkm.apps.internal for org brilliankingsman-main-org / space development as brilliankingsman@gmail.com... Route nbackend-bkm.apps.internal has been created. OK ``` **Check Routes:** ```bash $ cf routes Getting routes for org brilliankingsman-main-org / space development as brilliankingsman@gmail.com ... space host domain port path type apps service development frontend-bkm cfapps.io frontend-bkm development backend-a apps.internal backend-a-bkm development backend-b-bkm apps.internal backend-b-bkm development nbackend-bkm apps.internal ``` ### 6.2. Map routes to backends #### Map route to backend-a ```bash ~$ cf map-route backend-a-bkm apps.internal --hostname nbackend-bkm Creating route nbackend-bkm.apps.internal for org brilliankingsman-main-org / space development as brilliankingsman@gmail.com... OK Route nbackend-bkm.apps.internal already exists Adding route nbackend-bkm.apps.internal to app backend-a-bkm in org brilliankingsman-main-org / space development as brilliankingsman@gmail.com... OK ``` - Check after mapping route to backend-a ```bash ~$ cf apps Getting apps in org brilliankingsman-main-org / space development as brilliankingsman@gmail.com... OK name requested state instances memory disk urls backend-a-bkm started 1/1 32M 32M backend-a.apps.internal, nbackend-bkm.apps.internal backend-b-bkm started 1/1 32M 32M backend-b-bkm.apps.internal frontend-bkm started 1/1 32M 32M frontend-bkm.cfapps.io ~$ cf routes Getting routes for org brilliankingsman-main-org / space development as brilliankingsman@gmail.com ... space host domain port path type apps service development frontend-bkm cfapps.io frontend-bkm development backend-a apps.internal backend-a-bkm development backend-b-bkm apps.internal backend-b-bkm development nbackend-bkm apps.internal backend-a-bkm ``` #### Map route to backend-b - before mapping route to `backend-b-bkm`, the network policies for tcp/udp connections should be allowed first: ```bash ~$ cf add-network-policy frontend-bkm --destination-app backend-b-bkm --port 7007 --protocol tcp Adding network policy from app frontend-bkm to app backend-b-bkm in org brilliankingsman-main-org / space development as brilliankingsman@gmail.com... OK ~$ cf add-network-policy frontend-bkm --destination-app backend-b-bkm --port 9003 --protocol udp Adding network policy from app frontend-bkm to app backend-b-bkm in org brilliankingsman-main-org / space development as brilliankingsman@gmail.com... OK ``` ```bash ~$ cf map-route backend-b-bkm apps.internal --hostname nbackend-bkm Creating route nbackend-bkm.apps.internal for org brilliankingsman-main-org / space development as brilliankingsman@gmail.com... OK Route nbackend-bkm.apps.internal already exists Adding route nbackend-bkm.apps.internal to app backend-b-bkm in org brilliankingsman-main-org / space development as brilliankingsman@gmail.com... OK ``` - Check after mapping route to backend-a ```bash ~$ cf routes Getting routes for org brilliankingsman-main-org / space development as brilliankingsman@gmail.com ... space host domain port path type apps service development frontend-bkm cfapps.io frontend-bkm development backend-a apps.internal backend-a-bkm development backend-b-bkm apps.internal backend-b-bkm development nbackend-bkm apps.internal backend-a-bkm,backend-b-bkm ~$ cf apps Getting apps in org brilliankingsman-main-org / space development as brilliankingsman@gmail.com... OK name requested state instances memory disk urls backend-a-bkm started 1/1 32M 32M backend-a.apps.internal, nbackend-bkm.apps.internal backend-b-bkm started 1/1 32M 32M backend-b-bkm.apps.internal, nbackend-bkm.apps.internal frontend-bkm started 1/1 32M 32M frontend-bkm.cfapps.io ``` (End of reference implemantation) --- ## 7. Keep backend applicaitons only accessible internally ### 7.1. **Add backend applications to default internal domain `apps.internal`**: - By `manifest.yml`: (befor pushing the app) ```yaml --- applications: - name: backend-a-bkm memory: 32M disk_quota: 32M buildpack: go_buildpack env: GOPACKAGENAME: example-apps/cats-and-dogs-with-service-discovery/backend-a CATS_PORTS: 7007,7008,7009 UDP_PORTS: 9001,9002,9003 routes: - route: backend-a.apps.internal ``` - By `cf push`: (during pushing the app) ```bash ~$ cf push -d apps.internal ``` - By `cf map-route`: (after pushing the app) ``` cf map-route APP_NAME DOMAIN [--hostname HOSTNAME] [--path PATH] ``` - Reference: - [Creating a Route on the Internal Domain](https://cfcd-prep.cloudfoundry.org/routing/domains/#internal-domains) - [Configuring Routes and Domains - Internal Domain](https://docs.cloudfoundry.org/devguide/deploy-apps/routes-domains.html#internal-domains) ### 7.2. **Craete network policy to allow specified connections.** ### 7.3. **Delete** additional routes. - Command: ```bash ~$ cf delete-route apps.internal --hostname backend-a Really delete the route backend-a.apps.internal?> y Deleting route backend-a.apps.internal... OK ``` ```bash ~$ cf delete-route apps.internal --hostname backend-b-bkm Really delete the route backend-b-bkm.apps.internal?> y Deleting route backend-b-bkm.apps.internal... OK ``` - Check routes: ```bash ~$ cf routes Getting routes for org brilliankingsman-main-org / space development as brilliankingsman@gmail.com ... space host domain port path type apps service development frontend-bkm cfapps.io frontend-bkm development nbackend-bkm apps.internal backend-a-bkm,backend-b-bkm ~$ cf apps Getting apps in org brilliankingsman-main-org / space development as brilliankingsman@gmail.com... OK name requested state instances memory disk urls backend-a-bkm started 1/1 32M 32M nbackend-bkm.apps.internal backend-b-bkm started 1/1 32M 32M nbackend-bkm.apps.internal frontend-bkm started 1/1 32M 32M frontend-bkm.cfapps.io ``` ---