---
# System prepended metadata

title: Ray Cluster

---

# Ray Cluster

## Bare metal Ray cluster
How to create a bare metal Cluster for parallel processing

### Update Ubuntu to 20.04 if needed
```
sudo apt update && sudo apt upgrade
sudo apt autoremove
sudo apt install update-manager-core
sudo do-release-upgrade
sudo reboot
```

### Remove previous cuda installation
```
sudo apt-get --purge remove "*cuda*" "*cublas*" "*cufft*" "*cufile*" "*curand*" \
 "*cusolver*" "*cusparse*" "*gds-tools*" "*npp*" "*nvjpeg*" "nsight*"
sudo apt-get --purge remove "*nvidia*" "libxnvctrl*"
sudo apt-get autoremove
```
### install cuda-11.7
```
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-ubuntu2004.pin
sudo mv cuda-ubuntu2004.pin /etc/apt/preferences.d/cuda-repository-pin-600
wget https://developer.download.nvidia.com/compute/cuda/11.7.0/local_installers/cuda-repo-ubuntu2004-11-7-local_11.7.0-515.43.04-1_amd64.deb
sudo dpkg -i cuda-repo-ubuntu2004-11-7-local_11.7.0-515.43.04-1_amd64.deb
sudo cp /var/cuda-repo-ubuntu2004-11-7-local/cuda-*-keyring.gpg /usr/share/keyrings/
sudo apt-get update
sudo apt-get -y install cuda
```

edit ~/.bashrc to use new cuda path
```
#export PATH=/usr/local/cuda-10.1/bin${PATH:+:${PATH}}
#export LD_LIBRARY_PATH=/usr/local/cuda-10.1/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
#export CUDA_HOME=/usr/local/cuda-10.1
export PATH=/usr/local/cuda-11.7/bin${PATH:+:${PATH}}
export LD_LIBRARY_PATH=/usr/local/cuda-11.7/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
export CUDA_HOME=/usr/local/cuda-11.7
```



### Prepare Ray using Anaconda environment
get the env file from [here](https://gist.github.com/darwinharianto/d05a32389b0a89ff3588b22046573120)


get cluster.yaml template from [here](https://gist.github.com/darwinharianto/2667fd526ee6115bb7593fd2d63fd471)


In your own pc, create anaconda env and start ray cluster
```
conda env create -q -n ray_cluster -f /home/doors/environment.yaml || conda activate ray_cluster &&  ray up cluster.yaml 
```




## Docker Ray Cluster
How to create a containerized Cluster for parallel processing


### Install Docker and make sure it is working
```
sudo apt-get remove docker docker-engine docker.io containerd runc
sudo mkdir -p /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo   "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
  $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-compose-plugin
```

To check if it is working
```
sudo docker run hello-world
```


### Install nvidia docker
```
distribution=$(. /etc/os-release;echo $ID$VERSION_ID)       && curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg       && curl -s -L https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list |             sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' |             sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update
sudo apt-get install -y nvidia-docker2
sudo systemctl restart docker
```



To check if it is working
```
sudo docker run --rm --gpus all nvidia/cuda:11.6.2-base-ubuntu20.04 nvidia-smi
```


get cluster_docker.yaml template from [here](https://gist.github.com/darwinharianto/2667fd526ee6115bb7593fd2d63fd471)


In your own pc, create anaconda env and start ray cluster
```
conda env create -q -n ray_cluster -f /home/doors/environment.yaml || conda activate ray_cluster && ray up cluster_docker.yaml 
```



## Sending script command

sample python script from [here](https://gist.github.com/darwinharianto/2840cf9d370d42d9a95e8749efceabf1)


need to ssh to one of the worker/head
```
python training_pytorch_distributed.py --address=HEAD_IP_ADDRESS:6379 --num_workers=2 --model_name_or_path=textattack/roberta-base-CoLA --task_name=cola --use_gpu --per_device_train_batch_size=16
```
 
#### ray client cli for pc outside of cluster

```
python training_pytorch_distributed.py --address=ray://HEAD_IP_ADDRESS:10001 --num_workers=2 --model_name_or_path=textattack/roberta-base-CoLA --task_name=cola --use_gpu --per_device_train_batch_size=16
```



## Stopping cluster

```
ray down cluster.yaml
```
or 
```
ray down cluster_docker.yaml
```


[aa]\(bb)

＞データ容量　2GB超
~20GB：永年 500円/月~
