# Train LoRA Models on Apple Silicon

## Prerequisites
- git
- a pre-trained model, for example `anything-v4.5-pruned.safetensors`
- Homebrew x64 version

## Environment Setup

### Install pyenv
`brew install pyenv`

Add these 3 lines to your `~/.bashrc` or `~/.zshrc`

export PYENV_ROOT="$HOME/.pyenv"
command -v pyenv >/dev/null || export PATH="$PYENV_ROOT/bin:$PATH"
eval "$(pyenv init -)"

Ref: https://github.com/pyenv/pyenv#set-up-your-shell-environment-for-pyenv

### Clone the scripts
`git clone https://github.com/kohya-ss/sd-scripts`
`cd sd-scripts`

### Install Python 3.10.6

echo '3.10.6' > .python-version
pyenv install $(cat .python-version)

After the installation finishes, run `python --version` and it should return `3.10.6`.

### Install Dependencies

python -m venv venv
source venv/bin/activate
pip install torch==1.12.1 torchvision==0.13.1
python -m pip install tensorflow-macos

Then open `requirements.txt` and add a `#` at the start of `tensorflow==2.10.1`

This prevents pip from installing the tensorflow that does not work on MacOS. (In the previous step, we've already installed tensorflow for MacOS.)

pip install -r requirements.txt --prefer-binary

Note that you might see it complains it cannot install tensorflow correctly or tensorboard version not match. Just ignore it.

(optional) To use Lion as the optimiser, you need to install it by running `pip install lion_pytorch`.

## Training

Prepare your datasets and the config file.

For more information about the config file, please see: https://github.com/kohya-ss/sd-scripts/blob/main/train_README-ja.md#step-2-%E8%A8%AD%E5%AE%9A%E3%83%95%E3%82%A1%E3%82%A4%E3%83%AB%E3%81%AE%E8%A8%98%E8%BF%B0

This colab is also a great reference of writing the config file: https://github.com/Linaqruf/kohya-trainer/blob/main/kohya-LoRA-dreambooth.ipynb

Here's my `config.toml` file for your reference.

[general]
enable_bucket = true

[[datasets]]
resolution = 512
batch_size = 1 # you might want to have a bigger batch_size to make the training faster.
caption_extension = '.txt'

  [[datasets.subsets]]
  image_dir = '/Users/jean/aiart/datasets/testmacos/2_testmac'
  class_tokens = 'testmac'
  num_repeats = 2

You are all set!

Then just run the command below, under the `sd-scripts` root directory.

python ./train_network.py --pretrained_model_name_or_path=<path of your pre-trained model> \
--dataset_config=<path of your config file> \
--network_module=networks.lora \
--output_dir=<path of the output LoRA model file(s)>

For more available args, please see: the https://github.com/kohya-ss/sd-scripts/blob/main/train_network_README-ja.md

Again, this colab is a great reference for setting up your args: https://github.com/Linaqruf/kohya-trainer/blob/main/kohya-LoRA-dreambooth.ipynb

Here's mine, just for your reference:

python ./train_network.py --pretrained_model_name_or_path=/Users/jean/aiart/models/anything-v4.5-pruned.safetensors \
--vae=/Users/jean/aiart/vae/kl-f8-anime.ckpt \
--dataset_config=/Users/jean/aiart/datasets/testmacos/config.toml \
--network_module=networks.lora \
--save_every_n_epochs=5 \
--output_dir=/Users/jean/aiart/my_lora_models/ --output_name=testmac \
--noise_offset=0.1 --optimizer_type=Lion \
--clip_skip=2

Let's train =)

### Annex
- If you want to have both x86 & arm Pythons on your mac
- https://qiita.com/niccari/items/50b930dd5de65d0c71af