# Reproducing SPLAT-876

In the reproduction steps below, I found that during a Y-stream upgrade (4.9.51 -> 4.10.39) that the Service Account token is updated on disk inside of a test Pod. However, the old token (before ugprading) is still valid *after* the upgrade. This leads me to believe there's another issue going on in SPLAT-876 (or possible the reproducer below is not accurate).

## Steps
1. Installed cluster as 4.9.51 on AWS
2. Created a paused MCP that is connected to a single Node.

$ oc label node ip-10-0-203-168.us-west-2.compute.internal node-role.kubernetes.io/tests=""
$ oc adm taint node ip-10-0-203-168.us-west-2.compute.internal node-role.kubernetes.io/tests="":NoSchedule

$ cat paused.yaml 
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfigPool
metadata:
  labels:
    machineconfiguration.openshift.io/mco-built-in: ""
    pools.operator.machineconfiguration.openshift.io/paused: ""
  name: paused
spec:
  machineConfigSelector:
    matchLabels:
      machineconfiguration.openshift.io/role: worker
  nodeSelector:
    matchLabels:
      node-role.kubernetes.io/tests: ""
  paused: true

$ oc create -f paused.yaml

$ oc get mcp 
NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
master   rendered-master-25fb91f13879018d25c5213665e9e572   True      False      False      3              3                   3                     0                      37m
paused                                                      False     False      False      1              0                   0                     0                      17s
worker   rendered-worker-8b899d10fa354def8b36623484845c4a   True      False      False      3              3                   3                     0                      37m

$ oc get nodes
NAME                                         STATUS   ROLES          AGE   VERSION
ip-10-0-129-31.us-west-2.compute.internal    Ready    worker         48m   v1.22.8+9e95cb9
ip-10-0-129-53.us-west-2.compute.internal    Ready    master         53m   v1.22.8+9e95cb9
ip-10-0-165-140.us-west-2.compute.internal   Ready    master         53m   v1.22.8+9e95cb9
ip-10-0-179-34.us-west-2.compute.internal    Ready    worker         47m   v1.22.8+9e95cb9
ip-10-0-195-15.us-west-2.compute.internal    Ready    master         53m   v1.22.8+9e95cb9
ip-10-0-203-168.us-west-2.compute.internal   Ready    tests,worker   48m   v1.22.8+9e95cb9

3. Deployed a test Pod that will run on the isolated node:

$ cat pod.yaml 
apiVersion: v1
kind: Pod
metadata:
  name: ubuntu
  labels:
    app: ubuntu
spec:
  tolerations:
  - key: "node-role.kubernetes.io/tests"
    operator: "Exists"
    effect: "NoSchedule"
  nodeSelector: 
    node-role.kubernetes.io/tests: ""
  containers:
  - image: ubuntu
    command:
      - "sleep"
      - "604800"
    imagePullPolicy: IfNotPresent
    name: ubuntu
  restartPolicy: Always

$ oc create -f pod.yaml

$ oc get pods -o wide 
NAME     READY   STATUS              RESTARTS   AGE   IP       NODE                                         NOMINATED NODE   READINESS GATES
ubuntu   0/1     ContainerCreating   0          4s    <none>   ip-10-0-203-168.us-west-2.compute.internal   <none>           <none>

4. Captured service account secret information:

$ oc rsh ubuntu
$ bash
$ cd /root
$ mkdir original
$ cp /var/run/secrets/kubernetes.io/serviceaccount/* original/

5. Initiated Y-stream update

# Edited cluster version to stable-4.10
$ oc adm upgrade --to 4.10.39

6. Wait for upgrade to complete...

$ oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.10.39   True        False         58m     Cluster version is 4.10.39

$ oc get clusterversion version -o yaml | yq '.status.history[] | .version'
4.10.39
4.9.51

7. Capture new service account secret information:

$ oc rsh ubuntu
$ mkdir after-upgrade-4.10
$ cp /var/run/secrets/kubernetes.io/serviceaccount/* after-upgrade-to-4.10/

8. Note the new token (second one is new. check near end of tokens when comparing.):

$ cat original/token && echo && echo && cat after-upgrade-to-4.10/token  && echo
eyJhbGciOiJSUzI1NiIsImtpZCI6IjNPSXVxcmZoVUt1VUFGRF96NDMzdmkzR1l1WVNJaXpueFNUNnVkSmhPUmcifQ.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjIl0sImV4cCI6MTcwMTM2NDk1MiwiaWF0IjoxNjY5ODI4OTUyLCJpc3MiOiJodHRwczovL2t1YmVybmV0ZXMuZGVmYXVsdC5zdmMiLCJrdWJlcm5ldGVzLmlvIjp7Im5hbWVzcGFjZSI6ImRlZmF1bHQiLCJwb2QiOnsibmFtZSI6InVidW50dSIsInVpZCI6ImY4NDM5MzcyLTA3YTgtNDMyNS05N2I1LWMyNzc3NjJkY2MxMCJ9LCJzZXJ2aWNlYWNjb3VudCI6eyJuYW1lIjoiZGVmYXVsdCIsInVpZCI6ImMyM2JjNjFiLWU4NWMtNDdhNi04YjhlLWJkZmNkZGZiYzNjMCJ9LCJ3YXJuYWZ0ZXIiOjE2Njk4MzI1NTl9LCJuYmYiOjE2Njk4Mjg5NTIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpkZWZhdWx0OmRlZmF1bHQifQ.I2nS3E-moH_G4JzOaWX1msXR9CGU6O_MV5cnObltLpkDJzJLpK9mR1adySerNppP1eeAoZOGOXK-M2R5dNk9dSSJowTUAYFBn9i1IWYS0EID0IhPUu_P-NNEYTk_cFQ7tu4KWmH3B_yCmfHUIu4XszNAOWEuwJ0rXTcL539KqwhyWFjrby2v7lEwT6TIgPtAX2GR8w4p4irhT9ZhxEuIbG93uLHLvEy4mppzmSOp22NS-LNwmDL5LN9Oct-mAAmWDiGhYqfx97Y7aoZdB1S-nBqwg86cHXcxiDAnvO8tw9B_g65snIxD5GZyqUoPKrerk0E-kPQL1WoMFLkPERAT9F-nAjcdjggE6MhgP_oMoFlnTKR7dk8Lu6VNVTt8DfsY0dUDI5-R5kWE32n5fTM0PskGhJbDY-iBkdNnN0vQh71q0jaabYfhslyTD1LrF255tA1eczJnOJG0Fr2Ywy4p6VidwmxfrmoVCk2qL982MKhJqOpS6t1czQvGjOwKy8HPWBO8ZUui69kX_ipgn4OjnGI783roO5CEjNts2H4IiORcL0Gxfqbb-PA-POIRQ2IYUwvjCn6V_BgbSPTJnVPXGTb0luPudWRnn00pw72XHDOAzO4c4_7wR3lc9_hdChaAJjp3_j07C4Z54pIVm-8MgO6EuJUbaY5OufYEI-AIrAk

eyJhbGciOiJSUzI1NiIsImtpZCI6IjNPSXVxcmZoVUt1VUFGRF96NDMzdmkzR1l1WVNJaXpueFNUNnVkSmhPUmcifQ.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjIl0sImV4cCI6MTcwMTM3MDczOSwiaWF0IjoxNjY5ODM0NzM5LCJpc3MiOiJodHRwczovL2t1YmVybmV0ZXMuZGVmYXVsdC5zdmMiLCJrdWJlcm5ldGVzLmlvIjp7Im5hbWVzcGFjZSI6ImRlZmF1bHQiLCJwb2QiOnsibmFtZSI6InVidW50dSIsInVpZCI6ImY4NDM5MzcyLTA3YTgtNDMyNS05N2I1LWMyNzc3NjJkY2MxMCJ9LCJzZXJ2aWNlYWNjb3VudCI6eyJuYW1lIjoiZGVmYXVsdCIsInVpZCI6ImMyM2JjNjFiLWU4NWMtNDdhNi04YjhlLWJkZmNkZGZiYzNjMCJ9LCJ3YXJuYWZ0ZXIiOjE2Njk4MzgzNDZ9LCJuYmYiOjE2Njk4MzQ3MzksInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpkZWZhdWx0OmRlZmF1bHQifQ.REB5V3OpLmS3YX7lBlSH-su90rAoyJzc_FfeEQyKWL-Izf15Dxe19A1QNrEMnhc8mO3U5TJQq0v6KOmEdwV8TYeLCYsCjiSQBT2ZDrKlaoZYcwAO6m18sPMiLt7VkApUFZFB8Ywsn5pdX7msHW5Bz2FwBuAlw2XLRrunXUHkrSAqOi9qeTRCMqTBYxkvGcG5Baq0Idv0IIv5iIbplPsqwCGoQeMHwZu3Zv1fzuJCwIRvH1kZ3qYGxvX2EGm8KSi2U6xsF_gNPzimSjNOS2ydniMaXld0cJvtiQtq8J9dDYNbEi2abzxNS-O9w6FirUJNsoFJJ2uGMD2IanSDrBY2rahakQ08xDr4i4lZMNhi4ngVDVWdwQuT3s1_T4fMP1Oh-TRrU3wns_FKOb0ZlKHXXaBJWv1Ql-xn4YvWo7mCJRGC74CMKaW4gANClUHZ2wp3XO1fkVJ3CkZOzjFHs9baHHQfPPGhYdDquoEG66QI9lhG83ep482VZ4EPerSk-u8XN7j298QZa14IpQdd35sUjfxRJuhWx7oDTALkAhUU58UOEmHQ419y1iIefK9sOYiEl2aMEbKuN2j4E7BUGgfLpuW5snXA2RFswFzuD0WY_R9ikoM1Zi1GZR832C9NauFLLS895kUstlj_wnhYWNjYne7-o8HehwocaNemA8tmQcg

9. Try using old and new tokens with Kube API:

$ apt-get update
$ apt-get install curl
#### OLD TOKEN
$ curl -k -H "Authorization: Bearer $(cat original/token)" https://api.rbost.devcluster.openshift.com:6443/apis/user.openshift.io/v1/users/~
{
  "kind": "User",
  "apiVersion": "user.openshift.io/v1",
  "metadata": {
    "name": "system:serviceaccount:default:default",
    "creationTimestamp": null
  },
  "groups": [
    "system:authenticated",
    "system:serviceaccounts",
    "system:serviceaccounts:default"
  ]
}

#### NEW TOKEN
$ curl -k -H "Authorization: Bearer $(cat after-upgrade-to-4.10/token)" https://api.rbost.devcluster.openshift.com:6443/apis/user.openshift.io/v1/users/~
{
  "kind": "User",
  "apiVersion": "user.openshift.io/v1",
  "metadata": {
    "name": "system:serviceaccount:default:default",
    "creationTimestamp": null
  },
  "groups": [
    "system:authenticated",
    "system:serviceaccounts",
    "system:serviceaccounts:default"
  ]
}