---
# System prepended metadata

title: containerd failed to prepare extraction snapshot
tags: [trouble]

---

# containerd failed to prepare extraction snapshot


## 報錯
* 手動拉 image 時發現以下報錯
```
$ crictl pull 192.168.11.90/suse-observability/hadoop:3.4.1-java11-8-90a9d727
E0317 14:33:26.388176    8105 log.go:32] "PullImage from image service failed" err="rpc error: code = Unknown desc = failed to pull and unpack image \"192.168.11.90/suse-observability/hadoop:3.4.1-java11-8-90a9d727\": failed to prepare extraction snapshot \"extract-289854180-amZ- sha256:0bd96a6a52b13edad37e66c3e91231992ee46a4801ffde00f47058b14fbea34e\": failed to stat parent: stat /var/lib/rancher/rke2/agent/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/121/fs: no such file or directory" image="192.168.11.90/suse-observability/hadoop:3.4.1-java11-8-90a9d727"
FATA[0000] pulling image: failed to pull and unpack image "192.168.11.90/suse-observability/hadoop:3.4.1-java11-8-90a9d727": failed to prepare extraction snapshot "extract-289854180-amZ- sha256:0bd96a6a52b13edad37e66c3e91231992ee46a4801ffde00f47058b14fbea34e": failed to stat parent: stat /var/lib/rancher/rke2/agent/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/121/fs: no such file or directory
```
![image](https://hackmd.io/_uploads/Sk788BB21e.png)

* 查看 containerd log 

![image](https://hackmd.io/_uploads/B1lw8rS3kl.png)

## 解決
* 檢查 snapshot 有發現 extract 代表這個 snapshot 有異常
```
$ ctr --address /run/k3s/containerd/containerd.sock -n k8s.io snapshots ls | grep extract
WARN[0000] DEPRECATION: The `configs` property of `[plugins."io.containerd.grpc.v1.cri".registry]` is deprecated since containerd v1.5 and will be removed in containerd v2.1. Use `config_path` instead.
extract-274496232-mJ-Z sha256:0507e189b1b568447fd80c777b80b2f7911f4c3082ab599d836a51bab3573e3d sha256:3d9824a6b05ece6f0b450dbca8416cece5d4906739d71a8c93972382c4cad249 Active
extract-723398845-17rV sha256:0bd96a6a52b13edad37e66c3e91231992ee46a4801ffde00f47058b14fbea34e sha256:1eef8b11f2f42810e7909b4de7b3d8bcf6232598d4e052f2b7094ab67ed8257f Active
```

* 停止 rke2-agent or rke2-server 服務，刪除 `/var/lib/rancher/rke2/agent/containerd` 目錄讓他重新生成
```
$ systemctl stop rke2-agent.service

$ rm -rf /var/lib/rancher/rke2/agent/containerd

$ systemctl start rke2-agent.service
```
* 異常的 snapshot 已經消失
```
$ ctr --address /run/k3s/containerd/containerd.sock -n k8s.io snapshots ls | grep extract
WARN[0000] DEPRECATION: The `configs` property of `[plugins."io.containerd.grpc.v1.cri".registry]` is deprecated since containerd v1.5 and will be removed in containerd v2.1. Use `config_path` instead.
```
* 並且可以正常拉到 image
```
$ crictl pull 192.168.11.90/suse-observability/hadoop:3.4.1-java11-8-90a9d727
Image is up to date for sha256:abd67bc1122167df5881d6f8ecfc27e711a04d49743edbb3e91916f69960df3d
```

## 參考
https://serverfault.com/questions/1166840/k3s-failed-to-prepare-extraction-snapshot