# Train Stable Diffusion LoRA from VRM Avatar

In this guide I'll explain how you can train a [LoRA](https://replicate.com/blog/lora-faster-fine-tuning-of-stable-diffusion) of your VRM avatar that you can use to draw images of your 3D avatar in any Stable Diffusion Model that you can find. In this Tutorial we are going to train the LoRA for the tokenized identity [Nature](https://twitter.com/naturevrm) and I'm going to provide the file so you can play around with it in your own Stable Diffusion installation.

Via the images below you can get a sense of the LoRA training results of my own VRM avatar. Find some vtubing content on my [Youtube Channel](https://www.youtube.com/@reneil1337) to get an impression about the actual 3D avatar. A LoRA captures and conserves a concept or an idea in a way that it can be aggregated into larger models to be part of its outputs. A LoRA can be anything but in this case it is a character.


## Install the Stack

You'll need the following 3 ingrediences to train a Stable Diffusion model on your VRM avatar. All links that you need are provided in the step-by-step guide below. The Stable Diffusion and Kohya resources are linked in the descriptions of the Youtube Videos and outlined in that content.

- VRM Live Viewer
- Stable Diffusion
- Kohya

## Step-by-Step Guide

Install the tools linked above and learn the entire pipeline from data set recording, data set curation, LoRA model training up to prompting final outputs from your local stable diffusion instance. We are going to do all of these things step by step. Let's go!

### 1. Caputure your avatar shots via VRM Live Viewer

Download [VRM Live Viewer](https://booth.pm/ja/items/1783082) and extract the files to your computer. Launch the exe to launch the software. VRM Live Viewer allows for the configuration and playback of dance choreographies involving avatars, visuals and most importantly animations.


The UI can be pretty overwhelming if you are not used to it but lets go through it step by step.

1. Load your VRM avatar into a 3D world. The app has a couple of predefined worlds and performances that you can launch right away. However we're going to change the surrounding to white which ensures that the LoRA doesn't contain background artifacts.


2. So once loaded, the first choreography will start right away. The default stage is pretty wild when it comes to the VFX so change the stage to "Plane" in the bottom of the right menu. Set the floor (2nd select) to None/3D and the sky to (3rd select) to None/360. Click that 360 icon right next to that last select box which opens this popup.

Click the box right next to "BackgroundColor" and set the color to #FFFFFF so that we have a completely clean environments which will help to isolate our avatar in the next steps.


3. In the top right you can choose between 3 preinstalled dance choreographies. You can also load custom bvh animation files from your computer via the blue folder icon above but for this guide we'll stick to the 3 dances as these are very long animations which leaves us enough time for the camera positioning to take the shots of our avatar.


4. Hit "tab" on your keyboard to toggle the side menus. Now you take screenshots of our avatar, cut them to square orientation and save 512x512 px jpg files on your computer. Every one in a while hit "tab" to display the menu again as you shoot different positions on various stages from all sorts of angles. The more variety you give, the better the AI understands your avatar.

- Hold left mouse button to turn the camera
- Hold right mouse button to move the camera
- Use scroll wheel to zoom in and out


When the animation is over, just reselect another choreography via the dropdown indicated in step 3. The stop + play buttons at the bottom of the right menu are sometimes buggy for me. Camera flights with XBOX Controller works great - give it a try. I've decided to go for 30 images for the full body capture of the Nature avatar.

::: info

VRM Live Viewer helps to capture different facial expressions as the avatar is singing. The first training for Reneil was done with only 20 head shots from VRM Live Viewer which were not clean. Depending on your goals you can focus on the face - for example if you want to wear different clothes. For the Nature LoRA we'll capture the entire avatar.

:::

### 2. Prepare your Avatar Shots via Stable Diffusion and Kohya

1. Watch this tutorial and install Stable Diffusion quick and easy within 15 minutes. It is an essential component of the stack that we'll be using here so there is no way around it.

<iframe style="width:100%;display:inline-block;padding:0px" height="420"src="https://www.youtube.com/embed/onmqbI5XPH8" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>

2. Now start to watch this tutorial on how to install Kohya on your local machine and learn how to prepare the LoRA training data from the images that we've prepared in the first step. This tutorial contains everything that you need to know related to preprocessing and training. Open it in a new tab and keep it open as it helps you to understand the next steps.

<iframe style="width:100%;display:inline-block;padding:0px" height="420"src="https://www.youtube.com/embed/70H03cv57-o" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>

3. Now we'll use an AI to automatically analyze the content of the images of our avatar that we've captured. Create a folder called "processed" inside the folder in which you've stored your training imgs. Then launch Stable Diffusion (close Kohya before if you still have it running due to the installation in step 2) and navigate to "Train > Preprocess images".


Insert your folder path into "Source directory" and the path of the newly created processed folder into "Destination directory". Check "Use BLIP for caption" and hit the Preprocess button to start,

4. After the preprocessing step, the folder you use for training should look like this. I've used "NatureVRM" as character name instead of Nature to ensure that there won't be confusion about the already existing definition of Nature when I'm prompting the outputs later on.


5. When you are optimizing the preprocessed txt files make ensure to strip out everything that describes the essence your character. Any information in the txt is substracted from the LoRA that will be created during the training. The more clear you are, the less artifacts will be in your LoRA.

In my case the files originally says "in a costume with flowers on her arm" but I stripped that from the txt as the roses are an essential part of Natures visual characteristic and this is not a custome. Make sure to always copy+paste the name of your LoRA in the beginning of any txt file. You need to do this for every image in the folder.

::: info

Txt input: ``NatureVRM a character with arms outstretched, lifting one hand in the air, on white background``
:::

6. Now prepare the final folder structure for your LoRA training. As all of this is explained step-by-step in the LoRA tutorial by Aitrepreneur I won't explain this here. Your final folder structure before we begin the training should look just like this. In the image folder sits the "100_NatureVRM" folder. With 15 imgs or less that number differs (see the LoRA video).


All the preprocessed images that we have prepared during the previous steps are located inside the image/100_NatureVRM and the model + log folders are still empty. From here we can start to train the LoRA with the Stable Diffusion model inside Kohya.


### 3. Start the LoRA Training inside Kohya

To start the actual training you can now launch Kohya - make sure to close Stable Diffusion in case you had that still open. Also close VRM Live Viewer if this is still open as you'll need every bit of VRAM from your GPU during the training. First we'll train the LoRA with SD 1.5 as this ensures compatibility with most models on [CivitAI](https://civitai.com/) as most of them were trained with that SD version.

You should repeat the training with SD 2.1 so that you also have a LoRA that works with the upcoming models that might be trained with that newer version of Stable Diffusion. The end result of the training is that you'll have two .safetensor files - one of each SD version.

1. Switch into the "Dreambooth LoRA" tab to start. The regular Dreambooth View looks very similar so double check that you are in the LoRA tab! Click the "Configuration file" section which pops out an uploader area.


Open the [LoRA settings json file](https://mega.nz/file/ndFRwTIB#-T-6AKXtK8Xa8VZGzNAjARNUyGTj2qEYSLp7zePDOIs) that you've downloaded during the tutorial (you have less than 7 GB VRAM? Use [this json file](https://mega.nz/file/zFNjlJAL#uB2uTAvcqLohSUzYBgtuBcAMt4Jnclg6jVV5YE4s0F4)) as this contains the basic training setup and click "Load".

2. Navigate to the "Folders" tab where you overwrite the 3 folders paths (regularisation stays empty) according to the structure that you've prepared in step 2 of this tutorial. Then set "Model output name" according to the Folder that you've prepared and add v15 to indicate that this LoRA was trained with SD 1.5 as a base model.


Then click train model to start the training process. Depending on your hardware and the number of training images this is going to take a while. You can go AFK to touch some grass during that process.


3. Once this is done you should see the safetensors file in the model folder that you've created. Copy+Paste it into your "stable-diffusion-webui\models\Lora" folder so that you can access it from the webUI of your Stable Diffusion installation. Do the same with the SD v2.1 LORA


4. Now We repeat the process for SD 2.1 so we close the Kohya cmd and boot it up again. Refresh the WebUI or jump into it again via http://127.0.0.1:7860/ - again you load your json config file. This time you select stable-diffusion-2-1 in the Model Quick Pick which automatically checks the v2 and v_parameterization checkboxes.



Don't forget to copy the created .safetensors file into your Stable Diffusion folder once the second training is done. Now we're ready to start with the creating of outputs.

## Prompt Images with your Avatar LoRA

Launch Stable Diffusion and click the red icon below the Generate button to show your extra networks. Click the Lora tab and you should now see the LORAs that you copied into your SD installation. When you click one of those it will add the lora tag into your prompt text area. You can embed this into more complex prompts to embed your avatar into your creation.


You can download a few models to play around. Lets start prompting with [Protogen V22 Anime](https://civitai.com/models/3627/protogen-v22-anime-official-release) which was also used to create the images in the intro of this article. You can scroll down the linked model page to get some example prompts that work particular well for the selected model. Adjust those and don't forget your LoRA tag in the beginning of your prompt.


You don't have to keep the LoRA tab open. Click the red icon again to hide it. The anime model is adding a face and haircut into the nature avatar. To prevent this you can increase the weight of your LoRA by replacing the 1 in the pointed brackets to 1.3 just play around with all these things as you prompt your way towards all sorts of configurations and models.


The Nature crew prompted a crew cool prompts with various models. The [Protogen x5.8](https://civitai.com/models/3867/protogen-x58-rebuilt-scifianime-official-release) and [Deliberate](https://civitai.com/models/4823/deliberate) models work great with this LoRA. Download LoRA file at the end of the guide.


Bonus: Lets get a bit advanced and bring in ControlNet. If you don't have that extension installed yet [watch this video by Aitrepreneur](https://www.youtube.com/watch?v=OxFcIv8Gq8o) and after you've installed and understood ControlNet feel free to follow along. Controlnet allows you to position your avatar according to the characters on the input image. It's an entirely new rabbit hole to explore and Aitrepreneur explains how.


I hope you learned a couple of things in this guide. You can [follow me on Twitter](https://twitter.com/reneil1337) in case you're interested in these topics and want to be notified about more guides like this.

## Find the NatureVRM LoRA on CivitAI

The LoRA that was trained in this guide was [uploaded to CivitAI](https://civitai.com/models/31462/naturevrm) under cc0 licence and you can download it for free. Play around with it in your own stable diffusion instance. Post your creations on CivitAI - when posting on socials make sure to tag [Nature's Twitter](https://twitter.com/naturevrm) so that she can retweet.


Drop your questions in [this Reddit thread](https://www.reddit.com/r/StableDiffusion/comments/12bpkqn/guide_convenient_process_to_train_lora_from_your/) will try to reply periodically over there.