---
# System prepended metadata

title: Restore bucket

---

```shell=sh
mkdir testRegistrySnapshotting
cd testRegistrySnapshotting
export BUCKET=$(oc get deployments -n openshift-image-registry image-registry -ojsonpath="{.spec.template.spec.containers[?(@.name=='registry')].env[?(@.name=='REGISTRY_STORAGE_S3_BUCKET')].value}")
# need to scale down first cluster-image-registry-operator to 0
# Then you can scale down image-registry
aws s3 ls s3://$BUCKET
# Snapshot bucket
aws s3 cp --recursive s3://$BUCKET .
# Create disaster to bucket
aws s3 rm --recursive s3://$BUCKET
# check bucket is now empty
aws s3 ls s3://$BUCKET
# oc get image still return images even ones stored in internal registry prior to disaster. Maybe this is due to this data being pulled from etcd which is not a live data from s3 bucket
oc get image
# deleting a pod dependent on internal registry results in a new pod which still works with following event text
# > Container image "image-registry.openshift-image-registry.svc:5000/wordpress/wordpress@sha256:8f3a949a1e1dcab7bc03283c0740ae441b6f401d9aade8ce74bd6f7073b46cf1" already present on machine
#   Generated from kubelet on ip-10-0-139-99.ec2.internal
# this suggests that internal registry is not always pulled and there are cache in the node/kubelet to clear before pods cannot start.
```
```shell=sh
# On OpenShift Web Console go to the node which had the image
# enter terminal
chroot /host
podman images | grep wordpress
podman image rm <imageID>
```
we should now be unable to restart pod due to imagepullbackoff.
- scale up wordpress
- ImagePullBackOff YAY
- Registry pod contain logs with response errors

# Restore bucket
```shell=sh
aws s3 cp --recursive . s3://$BUCKET
```
Image pull backoff no longer occurs.



### Old notes below

Registry pod logs after starting up when s3 bucket is empty
```
time="2022-03-30T20:51:52.510557704Z" level=info msg="start registry" distribution_version=v2.7.1+unknown go.version=go1.16.12 openshift_version=4.9.0-202202131427.p0.gab2eaa5.assembly.stream-ab2eaa5
time="2022-03-30T20:51:52.511407025Z" level=info msg="caching project quota objects with TTL 1m0s" go.version=go1.16.12
time="2022-03-30T20:51:52.512561973Z" level=info msg="redis not configured" go.version=go1.16.12
time="2022-03-30T20:51:52.512630476Z" level=info msg="Starting upload purge in 27m0s" go.version=go1.16.12
time="2022-03-30T20:51:52.522620002Z" level=info msg="using openshift blob descriptor cache" go.version=go1.16.12
time="2022-03-30T20:51:52.522659032Z" level=warning msg="Registry does not implement RempositoryRemover. Will not be able to delete repos and tags" go.version=go1.16.12
time="2022-03-30T20:51:52.523852319Z" level=info msg="Using \"image-registry.openshift-image-registry.svc:5000\" as Docker Registry URL" go.version=go1.16.12
time="2022-03-30T20:51:52.524043387Z" level=info msg="listening on :5000, tls" go.version=go1.16.12
time="2022-03-30T20:51:53.451760385Z" level=info msg=response go.version=go1.16.12 http.request.host="10.128.3.128:5000" http.request.id=a1420293-e0a2-4585-95c7-b77613f0acc9 http.request.method=GET http.request.remoteaddr="10.128.2.1:33190" http.request.uri=/healthz http.request.useragent=kube-probe/1.22 http.response.duration="160.632µs" http.response.status=200 http.response.written=0
time="2022-03-30T20:52:00.091261128Z" level=info msg=response go.version=go1.16.12 http.request.host="10.128.3.128:5000" http.request.id=89ed8aae-7e2a-46a1-84a4-25d2e7bdcc50 http.request.method=GET http.request.remoteaddr="10.128.2.1:33330" http.request.uri=/healthz http.request.useragent=kube-probe/1.22 http.response.duration="67.193µs" http.response.status=200 http.response.written=0
time="2022-03-30T20:52:10.097722303Z" level=info msg=response go.version=go1.16.12 http.request.host="10.128.3.128:5000" http.request.id=5a6d15fe-7b3d-47d1-b755-ce3e4ec7d0eb http.request.method=GET http.request.remoteaddr="10.128.2.1:33518" http.request.uri=/healthz http.request.useragent=kube-probe/1.22 http.response.duration="55.139µs" http.response.status=200 http.response.written=0
time="2022-03-30T20:52:10.099971655Z" level=info msg=response go.version=go1.16.12 http.request.host="10.128.3.128:5000" http.request.id=1f930ed5-33d6-4f69-9d2c-dfd75663cd80 http.request.method=GET http.request.remoteaddr="10.128.2.1:33520" http.request.uri=/healthz http.request.useragent=kube-probe/1.22 http.response.duration="51.331µs" http.response.status=200 http.response.written=0
time="2022-03-30T20:52:10.150127049Z" level=info msg="authorized request" go.version=go1.16.12 http.request.host="10.128.3.128:5000" http.request.id=6929cdf4-e081-472f-b001-5fac5e5bd7c7 http.request.method=GET http.request.remoteaddr="10.131.0.28:49102" http.request.uri=/extensions/v2/metrics http.request.useragent=Prometheus/2.29.2 openshift.auth.user="system:serviceaccount:openshift-monitoring:prometheus-k8s"
time="2022-03-30T20:52:10.153333758Z" level=info msg="response completed" go.version=go1.16.12 http.request.host="10.128.3.128:5000" http.request.id=6929cdf4-e081-472f-b001-5fac5e5bd7c7 http.request.method=GET http.request.remoteaddr="10.131.0.28:49102" http.request.uri=/extensions/v2/metrics http.request.useragent=Prometheus/2.29.2 http.response.contenttype="text/plain; version=0.0.4; charset=utf-8" http.response.duration=61.419437ms http.response.status=200 http.response.written=2056
```

Still able to push to registry after it is empty
PHP:wordpress S2I Build log when pushing new build after bucket disaster
```
Cloning "https://github.com/wordpress/wordpress" ...
	Commit:	f99c38efd54d910fb122a3694f8d7457fe8c8dad (Code Modernization: Rename parameters that use reserved keywords in `wp-admin/..)
	Author:	Sergey Biryukov <sergeybiryukov.ru@gmail.com>
	Date:	Wed Mar 30 10:45:06 2022 +0000
time="2022-03-30T21:09:17Z" level=info msg="Not using native diff for overlay, this may cause degraded performance for building images: kernel has CONFIG_OVERLAY_FS_REDIRECT_DIR enabled"
I0330 21:09:17.344772       1 defaults.go:102] Defaulting to storage driver "overlay" with options [mountopt=metacopy=on].
Caching blobs under "/var/cache/blobs".
Trying to pull image-registry.openshift-image-registry.svc:5000/openshift/php@sha256:2c7e706dcdd4829542d5a936b2a086ab7f3bb957182b4d8e0908e1e8dd54de14...
Getting image source signatures
Copying blob sha256:3b3e40b753e90aa16191234fa3f0e581b846f4e550d0cb39dbd2a2e8ab76c2f9
Copying blob sha256:4f7d33d0190851970fcf23430ca871284880c7b2f22bbf185ae865117d052be2
Copying blob sha256:effc4ea612c8cb531b45192df865c42f91ac1fa5da56a4af992e019934c442b9
Copying blob sha256:22ad6e077244916883d9242fb846e29fbd3cc70477693309001ee238772b6234
Copying blob sha256:de9bc33d733791f5dcdc296c0b8f06dcefd136c34b8f263716dc590adea53850
Copying config sha256:f74cf1a88766d73c010dd6928417645c010d7bb035b5dd77a96b63653a53522c
Writing manifest to image destination
Storing signatures
Generating dockerfile with builder image image-registry.openshift-image-registry.svc:5000/openshift/php@sha256:2c7e706dcdd4829542d5a936b2a086ab7f3bb957182b4d8e0908e1e8dd54de14
Adding transient rw bind mount for /run/secrets/rhsm
STEP 1/9: FROM image-registry.openshift-image-registry.svc:5000/openshift/php@sha256:2c7e706dcdd4829542d5a936b2a086ab7f3bb957182b4d8e0908e1e8dd54de14
STEP 2/9: LABEL "io.openshift.build.source-location"="https://github.com/wordpress/wordpress"       "io.openshift.build.image"="image-registry.openshift-image-registry.svc:5000/openshift/php@sha256:2c7e706dcdd4829542d5a936b2a086ab7f3bb957182b4d8e0908e1e8dd54de14"       "io.openshift.build.commit.author"="Sergey Biryukov <sergeybiryukov.ru@gmail.com>"       "io.openshift.build.commit.date"="Wed Mar 30 10:45:06 2022 +0000"       "io.openshift.build.commit.id"="f99c38efd54d910fb122a3694f8d7457fe8c8dad"       "io.openshift.build.commit.ref"="master"       "io.openshift.build.commit.message"="Code Modernization: Rename parameters that use reserved keywords in `wp-admin/.."
STEP 3/9: ENV OPENSHIFT_BUILD_NAME="wordpress-2"     OPENSHIFT_BUILD_NAMESPACE="wordpress"     OPENSHIFT_BUILD_SOURCE="https://github.com/wordpress/wordpress"     OPENSHIFT_BUILD_COMMIT="f99c38efd54d910fb122a3694f8d7457fe8c8dad"
STEP 4/9: USER root
STEP 5/9: COPY upload/src /tmp/src
STEP 6/9: RUN chown -R 1001:0 /tmp/src
STEP 7/9: USER 1001
STEP 8/9: RUN /usr/libexec/s2i/assemble
---> Installing application source...
=> sourcing 20-copy-config.sh ...
---> 21:10:02     Processing additional arbitrary httpd configuration provided by s2i ...
=> sourcing 00-documentroot.conf ...
=> sourcing 50-mpm-tuning.conf ...
=> sourcing 40-ssl-certs.sh ...
STEP 9/9: CMD /usr/libexec/s2i/run
COMMIT temp.builder.openshift.io/wordpress/wordpress-2:949cb23d
time="2022-03-30T21:10:02Z" level=warning msg="Adding metacopy option, configured globally"
Getting image source signatures
Copying blob sha256:101e6c3495512ae1ef4c2d3c006fbb4b84b55a83bc8506076898ba612180987c
Copying blob sha256:aadc47c09f66ee89512816702bc9d52b3ba07218c1439fe6db90073baf239acb
Copying blob sha256:06bfbc2615fb283c646d1fabecae90231be0dec2ebd8b076e1608a63a8028e25
Copying blob sha256:9303f305b342bb9c394c28e3602bbfba7d2b11070121049a7db700daf9ef74c4
Copying blob sha256:5502af5fc0fcbf4078fd7defd0a6a751d5a1cb40253bf232dd7410e31625f0ba
Copying blob sha256:06d048276bd2c3313e0acce6c7534d45a907fe82e9e014234894c093053057f2
Copying config sha256:f2299b3d05902bb4faa673887de8b9145c32e6af85258b54e462beaf4ba83711
Writing manifest to image destination
Storing signatures
--> f2299b3d059
Successfully tagged temp.builder.openshift.io/wordpress/wordpress-2:949cb23d
f2299b3d05902bb4faa673887de8b9145c32e6af85258b54e462beaf4ba83711

Pushing image image-registry.openshift-image-registry.svc:5000/wordpress/wordpress:latest ...
Getting image source signatures
Copying blob sha256:effc4ea612c8cb531b45192df865c42f91ac1fa5da56a4af992e019934c442b9
Copying blob sha256:4f7d33d0190851970fcf23430ca871284880c7b2f22bbf185ae865117d052be2
Copying blob sha256:de9bc33d733791f5dcdc296c0b8f06dcefd136c34b8f263716dc590adea53850
Copying blob sha256:06d048276bd2c3313e0acce6c7534d45a907fe82e9e014234894c093053057f2
Copying blob sha256:22ad6e077244916883d9242fb846e29fbd3cc70477693309001ee238772b6234
Copying blob sha256:3b3e40b753e90aa16191234fa3f0e581b846f4e550d0cb39dbd2a2e8ab76c2f9
Copying config sha256:f2299b3d05902bb4faa673887de8b9145c32e6af85258b54e462beaf4ba83711
Writing manifest to image destination
Storing signatures
Successfully pushed image-registry.openshift-image-registry.svc:5000/wordpress/wordpress@sha256:f01429f38ce9db1367aab46ab60b14f1da02ee487a36aa3703ab1f24fb9ae263
Push successful
```