```shell=sh mkdir testRegistrySnapshotting cd testRegistrySnapshotting export BUCKET=$(oc get deployments -n openshift-image-registry image-registry -ojsonpath="{.spec.template.spec.containers[?(@.name=='registry')].env[?(@.name=='REGISTRY_STORAGE_S3_BUCKET')].value}") # need to scale down first cluster-image-registry-operator to 0 # Then you can scale down image-registry aws s3 ls s3://$BUCKET # Snapshot bucket aws s3 cp --recursive s3://$BUCKET . # Create disaster to bucket aws s3 rm --recursive s3://$BUCKET # check bucket is now empty aws s3 ls s3://$BUCKET # oc get image still return images even ones stored in internal registry prior to disaster. Maybe this is due to this data being pulled from etcd which is not a live data from s3 bucket oc get image # deleting a pod dependent on internal registry results in a new pod which still works with following event text # > Container image "image-registry.openshift-image-registry.svc:5000/wordpress/wordpress@sha256:8f3a949a1e1dcab7bc03283c0740ae441b6f401d9aade8ce74bd6f7073b46cf1" already present on machine # Generated from kubelet on ip-10-0-139-99.ec2.internal # this suggests that internal registry is not always pulled and there are cache in the node/kubelet to clear before pods cannot start. ``` ```shell=sh # On OpenShift Web Console go to the node which had the image # enter terminal chroot /host podman images | grep wordpress podman image rm <imageID> ``` we should now be unable to restart pod due to imagepullbackoff. - scale up wordpress - ImagePullBackOff YAY - Registry pod contain logs with response errors # Restore bucket ```shell=sh aws s3 cp --recursive . s3://$BUCKET ``` Image pull backoff no longer occurs. ### Old notes below Registry pod logs after starting up when s3 bucket is empty ``` time="2022-03-30T20:51:52.510557704Z" level=info msg="start registry" distribution_version=v2.7.1+unknown go.version=go1.16.12 openshift_version=4.9.0-202202131427.p0.gab2eaa5.assembly.stream-ab2eaa5 time="2022-03-30T20:51:52.511407025Z" level=info msg="caching project quota objects with TTL 1m0s" go.version=go1.16.12 time="2022-03-30T20:51:52.512561973Z" level=info msg="redis not configured" go.version=go1.16.12 time="2022-03-30T20:51:52.512630476Z" level=info msg="Starting upload purge in 27m0s" go.version=go1.16.12 time="2022-03-30T20:51:52.522620002Z" level=info msg="using openshift blob descriptor cache" go.version=go1.16.12 time="2022-03-30T20:51:52.522659032Z" level=warning msg="Registry does not implement RempositoryRemover. Will not be able to delete repos and tags" go.version=go1.16.12 time="2022-03-30T20:51:52.523852319Z" level=info msg="Using \"image-registry.openshift-image-registry.svc:5000\" as Docker Registry URL" go.version=go1.16.12 time="2022-03-30T20:51:52.524043387Z" level=info msg="listening on :5000, tls" go.version=go1.16.12 time="2022-03-30T20:51:53.451760385Z" level=info msg=response go.version=go1.16.12 http.request.host="10.128.3.128:5000" http.request.id=a1420293-e0a2-4585-95c7-b77613f0acc9 http.request.method=GET http.request.remoteaddr="10.128.2.1:33190" http.request.uri=/healthz http.request.useragent=kube-probe/1.22 http.response.duration="160.632µs" http.response.status=200 http.response.written=0 time="2022-03-30T20:52:00.091261128Z" level=info msg=response go.version=go1.16.12 http.request.host="10.128.3.128:5000" http.request.id=89ed8aae-7e2a-46a1-84a4-25d2e7bdcc50 http.request.method=GET http.request.remoteaddr="10.128.2.1:33330" http.request.uri=/healthz http.request.useragent=kube-probe/1.22 http.response.duration="67.193µs" http.response.status=200 http.response.written=0 time="2022-03-30T20:52:10.097722303Z" level=info msg=response go.version=go1.16.12 http.request.host="10.128.3.128:5000" http.request.id=5a6d15fe-7b3d-47d1-b755-ce3e4ec7d0eb http.request.method=GET http.request.remoteaddr="10.128.2.1:33518" http.request.uri=/healthz http.request.useragent=kube-probe/1.22 http.response.duration="55.139µs" http.response.status=200 http.response.written=0 time="2022-03-30T20:52:10.099971655Z" level=info msg=response go.version=go1.16.12 http.request.host="10.128.3.128:5000" http.request.id=1f930ed5-33d6-4f69-9d2c-dfd75663cd80 http.request.method=GET http.request.remoteaddr="10.128.2.1:33520" http.request.uri=/healthz http.request.useragent=kube-probe/1.22 http.response.duration="51.331µs" http.response.status=200 http.response.written=0 time="2022-03-30T20:52:10.150127049Z" level=info msg="authorized request" go.version=go1.16.12 http.request.host="10.128.3.128:5000" http.request.id=6929cdf4-e081-472f-b001-5fac5e5bd7c7 http.request.method=GET http.request.remoteaddr="10.131.0.28:49102" http.request.uri=/extensions/v2/metrics http.request.useragent=Prometheus/2.29.2 openshift.auth.user="system:serviceaccount:openshift-monitoring:prometheus-k8s" time="2022-03-30T20:52:10.153333758Z" level=info msg="response completed" go.version=go1.16.12 http.request.host="10.128.3.128:5000" http.request.id=6929cdf4-e081-472f-b001-5fac5e5bd7c7 http.request.method=GET http.request.remoteaddr="10.131.0.28:49102" http.request.uri=/extensions/v2/metrics http.request.useragent=Prometheus/2.29.2 http.response.contenttype="text/plain; version=0.0.4; charset=utf-8" http.response.duration=61.419437ms http.response.status=200 http.response.written=2056 ``` Still able to push to registry after it is empty PHP:wordpress S2I Build log when pushing new build after bucket disaster ``` Cloning "https://github.com/wordpress/wordpress" ... Commit: f99c38efd54d910fb122a3694f8d7457fe8c8dad (Code Modernization: Rename parameters that use reserved keywords in `wp-admin/..) Author: Sergey Biryukov <sergeybiryukov.ru@gmail.com> Date: Wed Mar 30 10:45:06 2022 +0000 time="2022-03-30T21:09:17Z" level=info msg="Not using native diff for overlay, this may cause degraded performance for building images: kernel has CONFIG_OVERLAY_FS_REDIRECT_DIR enabled" I0330 21:09:17.344772 1 defaults.go:102] Defaulting to storage driver "overlay" with options [mountopt=metacopy=on]. Caching blobs under "/var/cache/blobs". Trying to pull image-registry.openshift-image-registry.svc:5000/openshift/php@sha256:2c7e706dcdd4829542d5a936b2a086ab7f3bb957182b4d8e0908e1e8dd54de14... Getting image source signatures Copying blob sha256:3b3e40b753e90aa16191234fa3f0e581b846f4e550d0cb39dbd2a2e8ab76c2f9 Copying blob sha256:4f7d33d0190851970fcf23430ca871284880c7b2f22bbf185ae865117d052be2 Copying blob sha256:effc4ea612c8cb531b45192df865c42f91ac1fa5da56a4af992e019934c442b9 Copying blob sha256:22ad6e077244916883d9242fb846e29fbd3cc70477693309001ee238772b6234 Copying blob sha256:de9bc33d733791f5dcdc296c0b8f06dcefd136c34b8f263716dc590adea53850 Copying config sha256:f74cf1a88766d73c010dd6928417645c010d7bb035b5dd77a96b63653a53522c Writing manifest to image destination Storing signatures Generating dockerfile with builder image image-registry.openshift-image-registry.svc:5000/openshift/php@sha256:2c7e706dcdd4829542d5a936b2a086ab7f3bb957182b4d8e0908e1e8dd54de14 Adding transient rw bind mount for /run/secrets/rhsm STEP 1/9: FROM image-registry.openshift-image-registry.svc:5000/openshift/php@sha256:2c7e706dcdd4829542d5a936b2a086ab7f3bb957182b4d8e0908e1e8dd54de14 STEP 2/9: LABEL "io.openshift.build.source-location"="https://github.com/wordpress/wordpress" "io.openshift.build.image"="image-registry.openshift-image-registry.svc:5000/openshift/php@sha256:2c7e706dcdd4829542d5a936b2a086ab7f3bb957182b4d8e0908e1e8dd54de14" "io.openshift.build.commit.author"="Sergey Biryukov <sergeybiryukov.ru@gmail.com>" "io.openshift.build.commit.date"="Wed Mar 30 10:45:06 2022 +0000" "io.openshift.build.commit.id"="f99c38efd54d910fb122a3694f8d7457fe8c8dad" "io.openshift.build.commit.ref"="master" "io.openshift.build.commit.message"="Code Modernization: Rename parameters that use reserved keywords in `wp-admin/.." STEP 3/9: ENV OPENSHIFT_BUILD_NAME="wordpress-2" OPENSHIFT_BUILD_NAMESPACE="wordpress" OPENSHIFT_BUILD_SOURCE="https://github.com/wordpress/wordpress" OPENSHIFT_BUILD_COMMIT="f99c38efd54d910fb122a3694f8d7457fe8c8dad" STEP 4/9: USER root STEP 5/9: COPY upload/src /tmp/src STEP 6/9: RUN chown -R 1001:0 /tmp/src STEP 7/9: USER 1001 STEP 8/9: RUN /usr/libexec/s2i/assemble ---> Installing application source... => sourcing 20-copy-config.sh ... ---> 21:10:02 Processing additional arbitrary httpd configuration provided by s2i ... => sourcing 00-documentroot.conf ... => sourcing 50-mpm-tuning.conf ... => sourcing 40-ssl-certs.sh ... STEP 9/9: CMD /usr/libexec/s2i/run COMMIT temp.builder.openshift.io/wordpress/wordpress-2:949cb23d time="2022-03-30T21:10:02Z" level=warning msg="Adding metacopy option, configured globally" Getting image source signatures Copying blob sha256:101e6c3495512ae1ef4c2d3c006fbb4b84b55a83bc8506076898ba612180987c Copying blob sha256:aadc47c09f66ee89512816702bc9d52b3ba07218c1439fe6db90073baf239acb Copying blob sha256:06bfbc2615fb283c646d1fabecae90231be0dec2ebd8b076e1608a63a8028e25 Copying blob sha256:9303f305b342bb9c394c28e3602bbfba7d2b11070121049a7db700daf9ef74c4 Copying blob sha256:5502af5fc0fcbf4078fd7defd0a6a751d5a1cb40253bf232dd7410e31625f0ba Copying blob sha256:06d048276bd2c3313e0acce6c7534d45a907fe82e9e014234894c093053057f2 Copying config sha256:f2299b3d05902bb4faa673887de8b9145c32e6af85258b54e462beaf4ba83711 Writing manifest to image destination Storing signatures --> f2299b3d059 Successfully tagged temp.builder.openshift.io/wordpress/wordpress-2:949cb23d f2299b3d05902bb4faa673887de8b9145c32e6af85258b54e462beaf4ba83711 Pushing image image-registry.openshift-image-registry.svc:5000/wordpress/wordpress:latest ... Getting image source signatures Copying blob sha256:effc4ea612c8cb531b45192df865c42f91ac1fa5da56a4af992e019934c442b9 Copying blob sha256:4f7d33d0190851970fcf23430ca871284880c7b2f22bbf185ae865117d052be2 Copying blob sha256:de9bc33d733791f5dcdc296c0b8f06dcefd136c34b8f263716dc590adea53850 Copying blob sha256:06d048276bd2c3313e0acce6c7534d45a907fe82e9e014234894c093053057f2 Copying blob sha256:22ad6e077244916883d9242fb846e29fbd3cc70477693309001ee238772b6234 Copying blob sha256:3b3e40b753e90aa16191234fa3f0e581b846f4e550d0cb39dbd2a2e8ab76c2f9 Copying config sha256:f2299b3d05902bb4faa673887de8b9145c32e6af85258b54e462beaf4ba83711 Writing manifest to image destination Storing signatures Successfully pushed image-registry.openshift-image-registry.svc:5000/wordpress/wordpress@sha256:f01429f38ce9db1367aab46ab60b14f1da02ee487a36aa3703ab1f24fb9ae263 Push successful ```