---
# System prepended metadata

title: Untitled

---

oc get cm -n nvidia-gpu-operator | grep device-plugin
oc get cm device-plugin-config -n gpu-operator-resources -o yaml


Check what happens on device plugin

```bash!
oc logs -n nvidia-gpu-operator -l app=nvidia-device-plugin-daemonset | egrep -i 'Registered|Discovered'

[ekhiaev@stocpbastion ~]$ oc logs -n nvidia-gpu-operator -l app=nvidia-device-plugin-daemonset | egrep -i 'Registered|Discovered'
Defaulted container "nvidia-device-plugin" out of: nvidia-device-plugin, toolkit-validation (init)
Defaulted container "nvidia-device-plugin" out of: nvidia-device-plugin, toolkit-validation (init)
I1016 13:24:05.908226       1 server.go:146] Registered device plugin for 'nvidia.com/gpu' with Kubelet
Error from server: Get "https://172.19.248.110:10250/containerLogs/nvidia-gpu-operator/nvidia-device-plugin-daemonset-77m7k/nvidia-device-plugin?tailLines=10": dial tcp 172.19.248.110:10250: connect: no route to host
[ekhiaev@stocpbastion ~]$

```

```bash
# see ip who's who and if all is running
oc get pods -n nvidia-gpu-operator -o wide | grep nvidia-device-plugin

# see if node is healthy
oc get node <FOR-NODE-5> 
oc describe node <FOR-NODE-5> | sed -n '/Conditions:/,/Addresses:/p'

```

# Kubelete issue


let's check kubelet reachability and try fix

- get status and logs of kubelet

debug node and hop to /host

```
systemctl status kubelet
journalctl -u kubelet -n 200 --no-pager
```

- check CRI-O health

```bash=
systemctl status crio
journalctl -u crio -n 200 --no-pager
```

- is 10250 listening 

```bash=
ss -tlpn | grep 10250
```

- Any host fw blocking?
I am lost at this stage without eyes


```
oc get clusterpolicy.nvidia.com -A -o yaml | sed -n '/devicePlugin:/,/^\s*[^ ]/p'

```


# Docs for timeslicing on 24.9.2 version of operator

[link to docs](https://docs.nvidia.com/datacenter/cloud-native/openshift/24.9.2/time-slicing-gpus-in-openshift.html)


nvidia.com/gpu.slices.ci=1


oc -n nvidia-gpu-operator get cm default-mig-parted-config -o yaml | less



[ekhiaev@stocpbastion MIG]$ oc get nodes -o custom-columns="NODE:.metadata.name,GPU:.status.capacity.nvidia\.com/gpu,MIG:.status.capacity.nvidia\.com/mig-1g\.24gb"
NODE                                 GPU      MIG
stocpmaster01.stocp.isracard.co.il   <none>   <none>
stocpmaster02.stocp.isracard.co.il   <none>   <none>
stocpmaster03.stocp.isracard.co.il   <none>   <none>
stocpworker01.stocp.isracard.co.il   <none>   <none>
stocpworker02.stocp.isracard.co.il   <none>   <none>
stocpworker03.stocp.isracard.co.il   <none>   <none>
stocpworker04.stocp.isracard.co.il   <none>   <none>
stocpworker05.stocp.isracard.co.il   2        8
stocpworker06.stocp.isracard.co.il   2        <none>
[ekhiaev@stocpbastion MIG]$


===


kube_pod_container_resource_requests{
  resource="nvidia_com_gpu",
  namespace!=""
} > 0





```
(
  max by (namespace, pod, container, instance, gpu_uuid, gpu_instance_id, compute_instance_id, mig_profile)
  (nvidia_gpu_memory_used_bytes{pod!="", mig_profile!=""} > 0)
)
* on (namespace, pod) group_left(node)
kube_pod_info
```


# steps

oc adm cordon  stocpworker05.stocp.isracard.co.il
oc adm drain stocpworker05.stocp.isracard.co.il --ignore-daemonsets --delete-emptydir-data

# all-disabled
oc label node stocpworker05.stocp.isracard.co.il nvidia.com/mig.config=all-disabled --overwrite


oc adm uncordon  stocpworker05.stocp.isracard.co.il

# strategy: mixed
oc edit clusterpolicies.nvidia.com gpu-cluster-policy-isracard

oc adm cordon  stocpworker05.stocp.isracard.co.il
oc adm drain stocpworker05.stocp.isracard.co.il --ignore-daemonsets --delete-emptydir-data

# all-1g.24gb
oc label node stocpworker05.stocp.isracard.co.il nvidia.com/mig.config=all-1g.24gb --overwrite
oc adm uncordon  stocpworker05.stocp.isracard.co.il



