# vanderlab

## Networking

### Physical Networking

| Portgroup | Subnet | Description |
|--------------|--------------|-----------------------------|
| VM Network | 192.168.1.0/24 | Primary home LAN network |
| lab-pg | 192.168.100.0/23 | Internal Lab network¹ |

All portgroups, except for `VM Network` are available on all hosts. `VM Network` is not to be used without good reason as testing and installations degrade the quality of service for everyone else in the house :( .

¹ IP range 192.168.101.1-250 is statically assigned.

With the exception of the storage network, all portgroups uplink to a 1Gbps switch. The storage network uplinks to a 2.5Gbps switch. Jumbo frames are enabled for the storage network.

### Network Services

vyos is the provider of DHCP, routing, and NAT'ing for the lab-pg port group. The vyos VM can be accessed via ssh at 192.168.100.1.

~~~
ssh vyos@192.168.100.1
~~~

Additionally, DNS forwarding is enabled in the disconnected-pg. This allows external DNS to be used, but no other traffic is allowed beyond the vyos VM.

### DNS

DNS is provided by Synology at 192.168.1.215 for the vanderlab.net for the lab-pg portgroup. Other lookups are forwarded from Synology to upstream DNS.

## Compute

Compute capacity is distributed among 4 clusters.

| Name | Cores | RAM | Storage |
| -------- | -------- | -------- | -----|
| management-cluster | 8/16T | 64GB | Datastores: <br>- mgmtDatastore: 1.69TB NVMe Flash Memory<br> - raidiscsi: 1.9TB Spinning Rust hosted by NAS
| workload-cluster-1² | 16/32T | 128GB | Datastores: <br> - workloadDatastore: 931GB NVMe Flash Storage<br> - raidiscsi: 1.9TB Spinning Rust hosted by NAS
| workload-cluster-2¹ | 8/16T | 32GB | Datastores: <br> - workloadDatastore2: 299GB NVMe Flash Memory

¹ Nested ESXi host running on ThinkPad P1 Gen 4i
² 6 vCPUs/32GB of RAM reserved

## Internal Registry

A Quay image registry is setup in the VM named `bastion`. This VM is accessible from `lab-pg`. The registry has 500GB of persistent storage. This storage is frequently wiped as new nightly releases are tested. The registry hostname is `registry.vanderlab.net:8443` and has no auth.

To login to Quay:
username: init
password: 52F9BXMtb41C0T3muSY6UQ7cfGgHx8kj

Push credentials:
~~~json
{
"auths": {
"registry.vanderlab.net:8443": {
"auth": "aW5pdCtwdXNoZXI6VEFDMVNLV09DOUtBMFUzVjdTRUxMMjhYNE5VNE9LMjcyRVVFNldHWTNHSU5VRUs3SDdJWVA0VklBT0NQMVRZMA==",
"email": ""
}
}
}
~~~

## MinIO Registry Storage

... need to update ...

## Accessing the Lab

Access to the lab is provided the [Twingate](https://vanderlab.twingate.com/) VPN which is attached to vcenter.vanderlab.net, `lab-pg`.

The bastion host is bastion.vanderlab.net.

## Performing Installations

All installations should be performed against the internal mirror registry. This is done to reduce the amount of traffic traversing `home-pg`.

### Mirroring

~~~shell
RELEASE_IMAGE=quay.io/openshift-release-dev/ocp-release:4.13.0-ec.3-x86_64
oc adm release mirror -a <pull-secret> --from=$RELEASE_IMAGE --to=registry.vanderlab.net:8443/init/openshift --to-release-image=registry.vanderlab.net:8443/init/openshift:4.13.0-ec.3-x86_64
~~~

### Installing from the mirror

~~~
export OPENSHIFT_INSTALL_RELEASE_IMAGE_OVERRIDE=$RELEASE_IMAGE
./openshift-install create cluster
~~~

### Sample `install-config.yaml` for multi-zone

~~~yaml
apiVersion: v1
baseDomain: vanderlab.net
compute:
- architecture: amd64
hyperthreading: Enabled
name: worker
platform:
vsphere:
zones:
- zone1
- zone2
- zone3
cpus: 4
coresPerSocket: 2
memoryMB: 16384
osDisk:
diskSizeGB: 100
replicas: 0
controlPlane:
architecture: amd64
hyperthreading: Enabled
name: master
platform:
vsphere:
zones:
- zone1
- zone2
- zone3
cpus: 4
coresPerSocket: 2
memoryMB: 16384
osDisk:
diskSizeGB: 60
replicas: 3
metadata:
creationTimestamp: null
name: rvanderp5-dev
networking:
clusterNetwork:
- cidr: 10.128.0.0/14
hostPrefix: 23
machineNetwork:
- cidr: 192.168.100.0/23
serviceNetwork:
- 172.30.0.0/16
platform:
vsphere:
datacenter: vanderlab
apiVIP: 192.168.100.100
ingressVIP: 192.168.100.201
network: "lab-pg"
defaultDatastore: nvme
password: <password>
cluster: cluster1
username: <user>@vsphere.local
vCenter: vcenter.vanderlab.net
failureDomains:
- name: zone1
zone: "zone-a"
region: "vanderlab"
topology:
computeCluster: /vanderlab/host/workload-cluster-1
datastore: workloadDatastore
folder: /vanderlab/vm/rvanderp5-dev-5dhsb
networks:
- "lab-pg"
- name: zone2
zone: "zone-b"
region: "vanderlab"
topology:
computeCluster: /vanderlab/host/workload-cluster-1
datastore: workloadDatastore2
folder: /vanderlab/vm/rvanderp5-dev-5dhsb
networks:
- "lab-pg"
- name: zone3
zone: "zone-c"
region: "vanderlab"
topology:
computeCluster: /vanderlab/host/management-cluster
datastore: mgmtDatastore
folder: /vanderlab/vm/rvanderp5-dev-5dhsb
networks:
- "lab-pg"
publish: External
imageContentSources:
- mirrors:
- registry.vanderlab.net:8443/init/openshift
source: quay.io/openshift-release-dev/ocp-release
- mirrors:
- registry.vanderlab.net:8443/init/openshift
source: quay.io/openshift-release-dev/ocp-v4.0-art-dev
pullSecret: <your secret>
~~~

## vyos configuration

~~~json
interfaces {
ethernet eth0 {
address dhcp
hw-id 00:50:56:be:1f:2e
}
ethernet eth1 {
address 192.168.100.1/23
address 2001::100/64
hw-id 00:50:56:be:00:b7
}
loopback lo {
}
}
nat {
source {
rule 1000 {
outbound-interface eth0
source {
address 192.168.100.0/23
}
translation {
address masquerade
}
}
}
}
protocols {
static {
route 0.0.0.0/0 {
next-hop 192.168.1.1 {
}
}
}
}
service {
dhcp-server {
shared-network-name lab {
name-server 192.168.1.215
subnet 192.168.100.0/23 {
default-router 192.168.100.1
range lab-dhcp {
start 192.168.101.1
stop 192.168.101.250
}
}
}
}
ntp {
allow-client {
address 0.0.0.0/0
address ::/0
}
server time1.vyos.net {
}
server time2.vyos.net {
}
server time3.vyos.net {
}
}
ssh {
port 22
}
}
system {
config-management {
commit-revisions 100
}
conntrack {
modules {
ftp
h323
nfs
pptp
sip
sqlnet
tftp
}
}
console {
device ttyS0 {
speed 115200
}
}
host-name vyos
login {
user vyos {
authentication {
encrypted-password ****************
plaintext-password ****************
}
}
}
syslog {
global {
facility all {
level info
}
facility protocols {
level debug
}
}
}
}
~~~

## Mirrored Release Image

~~~
Update image: registry.vanderlab.net:8443/init/openshift:4.13.0-ec.3-x86_64
Mirror prefix: registry.vanderlab.net:8443/init/openshift
Mirror prefix: registry.vanderlab.net:8443/init/openshift:4.13.0-ec.3-x86_64

To use the new mirrored repository to install, add the following section to the install-config.yaml:

imageContentSources:
- mirrors:
- registry.vanderlab.net:8443/init/openshift
source: quay.io/openshift-release-dev/ocp-release
- mirrors:
- registry.vanderlab.net:8443/init/openshift
source: quay.io/openshift-release-dev/ocp-v4.0-art-dev

To use the new mirrored repository for upgrades, use the following to create an ImageContentSourcePolicy:

apiVersion: operator.openshift.io/v1alpha1
kind: ImageContentSourcePolicy
metadata:
name: example
spec:
repositoryDigestMirrors:
- mirrors:
- registry.vanderlab.net:8443/init/openshift
source: quay.io/openshift-release-dev/ocp-release
- mirrors:
- registry.vanderlab.net:8443/init/openshift
source: quay.io/openshift-release-dev/ocp-v4.0-art-dev
~~~