# Equinix Metal Cloud2Cloud

## Pain Point

Traffic egress from GCP to Amazon takes up a very large amount of egress costs.

- https://metal.equinix.com/product/interconnection/
- 10 Gbps = $1/hour
- Machine n3.xlarge = $4.50/haur
- vrouter enabled or custom pxe router?
- https://cloud.google.com/network-connectivity/docs/interconnect
- https://cloud.google.com/network-connectivity/docs/interconnect/pricing
- 10 Gbps = $2.36/hour
- **EGRESS COST = $0.02/GB** need to find a way to reduce this
- https://aws.amazon.com/directconnect/
- https://aws.amazon.com/directconnect/pricing/
- 10 Gbps = $2.48/hour

## How do we get Amazon Customers to hit the Amazon ASN that forwards to Google GCP

- Anycast IPs for the reg.k8s.io & registry.k8s.io towards AWS pipe.
- Have Amazon route the IPs for k8s.gcr.io and registry.k8s.io over the pipe
- Override reg.k8s.io via split DNS
- https://www.jhanley.com/blog/google-cloud-private-dns-zones/
- https://cloud.google.com/blog/products/networking/introducing-private-dns-zones-resolve-to-keep-internal-networks-concealed
- Override k8s.gcr.io DNS (via )
## High Level Solution

Use dual Equinix Metal Fabric Infrasruction for Cloud connections to both AWS and Amazon and hairpin them together.

If possible at a software level, expose an IP directly on AWS fabric that connects directly to k8s.gcr.io over the Equinix Fabric.

Split DNS would allow us to supply the Equinix Fabric IP address for registry.k8s.io, and possibly even k8s.gcr.io, but only for Amazon customers.

## Initial Solution Thoughts/Brain Dump
Using this section to gather initial, untested brain ideas and thoughts about potential limitations, etc.

**Basic Outline**:

- Metal Org:
- metro(s) ??:
- Contains Device using some Router Image
- L2 Networking Enabled
- Connected to all VLANs
- VLANs
- AWS-Primary
- AWS-Redundant
- GCP-Primary
- GCP-Redundant
- Metal Internal/mgmt(?)
- Fabric VC (Metal Billed) x2(w/ redundant)
- Can't remember if it forces redundancy on the connection setup, have to double check
- AWS (w/ redundant)
- GCP (w/ redundant)
- Jumpbox(? for prod?)
- L2: Metal Internal/Mgmt
- AWS Account:
- Direct Connect:
- Connections:
- Metal-Primary (initiated from Equinix Fabric, accepted in AWS)
- Metal-Secondary (initiated in Equinix Fabric, accepted in AWS)
- Virtual Interface
- AWS-Metal-Primary
- Peered w/ Metal Router via BGP
- AWS-Metal-Secondary
- Peered w/ Metal Router via BGP
- VP/DC Gateway as appropriate to connect back to existing AWS Infrastructure
- GCP Account:
- I actually haven't played enough in GCP land to know the equivalent, but roughly the same as AWS but using the correct Googli-fied terminology

**Thoughts that occur in no particular order:**
- Some financial considerations that may be impacting the architecture:
- Trying to keep billing all under "metal" entity, may mean favoring building things in metal that might be covered under existing Network Edge
- (Some secrets re: Metal Network Edge, can we get access? Is it stable?)
- Means some sort of router image on a metal device
- I don't have an offhand comparison of if ther's a real cost difference
- Each Fabric VC Connection max speed 10gbps
- Drops into Metal on Layer2
- Back to above re: network edge, we need to fully build routing on metal side inclusive of like, "do we want this on a VM and what type of device should we deploy" and "what router OS should we use?"
- Network Edge could solve the (physical) device issues and give us a handful of router images to choose from.
- (again only really useful if we have available inside metal which is not GA at this time)
- Redundant connections still come into the same metro/city (on metal) which should be treated the same as a single AZ for all intents and purposes
- Relevant to production solution more than prototype
- BGP advertised remote routes should allow traffic to route from AWS to GCP via Metal with Metal router/ASN as hinge to the BGP polycule
- Some combination of local DNS and BGP should ensure routes go the way we want them to
- Could split DNS to serve local IPs known only through the fabric connections
- BGP route preferences might be able to route folks from AWS through the Fabric connection with out having to rely on DNS?
- We'll have to chat more about the DNS routing and traffic flow

----
Co-Authored-By: Aaron Aldrich
Co-Authored-By: Hippie Hacker