Train LoRA Models on Apple Silicon

Prerequisites

git
a pre-trained model, for example anything-v4.5-pruned.safetensors
Homebrew x64 version

Environment Setup

Install pyenv

brew install pyenv

Add these 3 lines to your ~/.bashrc or ~/.zshrc

export PYENV_ROOT="$HOME/.pyenv"
command -v pyenv >/dev/null || export PATH="$PYENV_ROOT/bin:$PATH"
eval "$(pyenv init -)"

Ref: https://github.com/pyenv/pyenv#set-up-your-shell-environment-for-pyenv

Clone the scripts

git clone https://github.com/kohya-ss/sd-scripts
cd sd-scripts

Install Python 3.10.6

echo '3.10.6' > .python-version
pyenv install $(cat .python-version)

After the installation finishes, run python --version and it should return 3.10.6.

Install Dependencies

python -m venv venv
source venv/bin/activate
pip install torch==1.12.1 torchvision==0.13.1
python -m pip install tensorflow-macos

Then open requirements.txt and add a # at the start of tensorflow==2.10.1

This prevents pip from installing the tensorflow that does not work on MacOS. (In the previous step, we've already installed tensorflow for MacOS.)

pip install -r requirements.txt --prefer-binary

Note that you might see it complains it cannot install tensorflow correctly or tensorboard version not match. Just ignore it.

(optional) To use Lion as the optimiser, you need to install it by running pip install lion_pytorch.

Training

Prepare your datasets and the config file.

For more information about the config file, please see: https://github.com/kohya-ss/sd-scripts/blob/main/train_README-ja.md#step-2-設定ファイルの記述

This colab is also a great reference of writing the config file: https://github.com/Linaqruf/kohya-trainer/blob/main/kohya-LoRA-dreambooth.ipynb

Here's my config.toml file for your reference.

[general]
enable_bucket = true

[[datasets]]
resolution = 512
batch_size = 1 # you might want to have a bigger batch_size to make the training faster.
caption_extension = '.txt'

  [[datasets.subsets]]
  image_dir = '/Users/jean/aiart/datasets/testmacos/2_testmac'
  class_tokens = 'testmac'
  num_repeats = 2

You are all set!

Then just run the command below, under the sd-scripts root directory.

python ./train_network.py --pretrained_model_name_or_path=<path of your pre-trained model> \
--dataset_config=<path of your config file> \
--network_module=networks.lora \
--output_dir=<path of the output LoRA model file(s)>

For more available args, please see: the https://github.com/kohya-ss/sd-scripts/blob/main/train_network_README-ja.md

Again, this colab is a great reference for setting up your args: https://github.com/Linaqruf/kohya-trainer/blob/main/kohya-LoRA-dreambooth.ipynb

Here's mine, just for your reference:

python ./train_network.py --pretrained_model_name_or_path=/Users/jean/aiart/models/anything-v4.5-pruned.safetensors \
--vae=/Users/jean/aiart/vae/kl-f8-anime.ckpt \
--dataset_config=/Users/jean/aiart/datasets/testmacos/config.toml \
--network_module=networks.lora \
--save_every_n_epochs=5 \
--output_dir=/Users/jean/aiart/my_lora_models/ --output_name=testmac \
--noise_offset=0.1 --optimizer_type=Lion \
--clip_skip=2

Let's train =)

Annex

If you want to have both x86 & arm Pythons on your mac
- https://qiita.com/niccari/items/50b930dd5de65d0c71af