SVA Computational Art - Spring 22

In this course we will explore:

• AI techniques and best practices for artists
• Generative / procedural art techiques and best practices for artists

We won't be writing code

However, we will be running some deep learning models using iPython Notebooks (both online using Google Colab and locally Jupyter Notebooks). So you will do some basic editing like changing example file names to match the names of the files you upload.

Early-access mode unlocked
When you are comfortable running an editing Notebooks code, you will be able to test and create content using the latest and greatest AI models– without needing to wait for anyone to create a UI for you.

Class 1

Discussion

What are the use cases for AI in creative productions?

• concept art

  • characters
  • environments
  • textures

• product design

  • information architecture
  • user interfaces
  • code

• mocap / rigging

  • human pose estimation
  • rigging and retargeting

Most popular deep learning libraries

• PyTorch (Facebook)
• Tensorflow (Google)
  • https://www.tensorflow.org/hub/tutorials

Workshop

In this workshop, we run and edit our first ML models using TensorflowHub

First, we look at MoveNet, a pose estimation model

Second, we look at Arbitrary Image Stylization, a style transfer model

Finally, we look at the web UI for Stable Diffision and watch an explainer video on how stable diffusion works

Pose Estimation

Movenet estimates 17 key joints in a human (or other biped character)

Movenet Tutorial
Movenet Google Colab

Results and Limitations

In our tests, we noticed that occlusion can hamper the model from detecting all 17 joints.

In this example, Captain America's shield prevents the model from identifying the joints in his right arm.

Style Transfer

https://colab.research.google.com/github/tensorflow/hub/blob/master/examples/colab/tf2_arbitrary_image_stylization.ipynb

Stable Diffusion

Explainer from AI Coffee Break

Class 2

How to run Stable Diffusion Locally

Install Miniconda

1. Download Miniconda

2. Install it for Just Me

3. Change Destination Folder to a writeable folder

   Note In our lab, only the U:\ drive is writeable, so we use U:\miniconda for the install location.

4. Run Anaconda Prompt

5. create a conda environment
   conda create --name art

6. Activate the environment
   conda activate art

7. Install python 3.10.2

   conda install python=3.10.2 -c conda-forge

8. Install Cuda Toolkit

   Warning The remaining instructions are for machines that have NVIDIA GPUs. Before installing, confirm that your machine has an NVIDIA GPU with CUDA of version 11.7 or later installed.

   run this command to check the version of Cuda
   nvidia-smi
   

   Warning If your CUDA version is less than 11.7, you will need to update your graphics drivers before continuing.

   Warning If the nvidia-smi command returns an error, your system most likely does not have a CUDA compatible NVIDIA GPU. Do not continue with the steps below. Instead, select the Conda installation options for your system on the Pytorch installation website.

   If nvidia-smi returns a CUDA Version >= 11.7, run this command to install CUDA Toolkit

   conda install -c "nvidia/label/cuda-11.7.1" cuda-toolkit

9. Install Pytorch

   conda install pytorch torchvision torchaudio pytorch-cuda=11.7 -c pytorch -c conda-forge -c nvidia

---

Install Stable Diffusion Web UI

Warning Before running any of these commands, make sure your terminal has activated the conda environment named art environment that we set up in the previous section.

Run conda activate art if you are not sure.

1. Clone the automatic1111 repo

   ​​​​u:
   ​​​​md src
   ​​​​cd src
   ​​​​git clone https://github.com/AUTOMATIC1111/stable-diffusion-webui.git
   ​​​​cd stable-diffusion-webui
   

2. Clone the stable diffusion repo inside of the automatic1111 repo

   ​​​​git clone https://github.com/Stability-AI/stablediffusion repositories/stable-diffusion-stability-ai
   

3. Install the necessary dependencies

   ​​​​pip install -r .\requirements.txt
   ​​​​pip install open_clip_torch
   ​​​​pip install git+https://github.com/crowsonkb/k-diffusion/
   

---

Use Custom Checkpoints With Stable Diffusion Web UI

Note We can use Stable Diffusion model checkpoints that were trained and shared by other artists. In this example, we'll use a checkpoint from CivitAI

1. Find a checkpoint that produces results in your desired style.

   In this example, I use https://civitai.com/models/2540/elldreths-stolendreams-mix

2. Move the checkpoint into the folder .\stable-diffusion-webui\models\Stable-diffusion\

---

Run the Stable Diffusion Web UI

Warning Before running any of these commands, make sure your terminal has activated the conda environment named art environment that we set up in the previous section.

Run conda activate art if you are not sure.

1. Run the python command from the stable-diffusion-webui directory

   ​​​​python .\webui.py
   

2. Open the local URL in your browser

   ​​​​Model loaded in 30.8s (1.5s create model, 22.7s load weights).
   ​​​​Running on local URL:  http://127.0.0.1:7860
   
   ​​​​To create a public link, set `share=True` in `launch()`.
   

   

---

Stable Diffusion Workshop

In this workshop we run this notebook: Stable Diffusion in Google Colab

Tutorial from Chris Johnston

• txt2img parameters
  

  

• img2img parameters
  

  

• tips

  • Check lexica prompts (especially the end)
  • Check hugging face for models
    • waifu diffusion for anime
    • protogen model
  • Check civitai.com
    • download checkpoints
  • Check huggingface for textual inversion
    • VikingPunk

Theory behind stable difffusion

Class 3

Installing Dreambooth

Save the 1.5-pruned checkpoint from
https://huggingface.co/runwayml/stable-diffusion-v1-5/resolve/main/v1-5-pruned-emaonly.ckpt
to your directory: .\models\Stable-diffusion

Dreambooth Setup

1. Create Model

2. Settings
   

   
   
   

3. Concepts

   • We use our 512x512 cropped images of our face in the dataset directory
   • We make an empty directory for the classification dataset directory (we will generate 300 images here in a later step)
   
   • The instance prompt uses the rare token ohwx and our classifier man
   • The class prompt uses a comparible image photo of a man
   • The sample image prompt uses our rare token combined with the class prompt photo of ohwx man
   
   • These settings use best practices for number of images as specified by the research papers
   

4. Saving

   

5. Generate Class images

   

   (you should see images genearating in the output tab and in the console)

   

   (note that I am getting around 4 iterations per second)

   

Class 4 Posing with ControlNet

tutorial:

Instructions for installing ControlNet

• Enable the sd-webui-controlnet stable diffusion extension
• Save the checkpoints from this link to the folder stable-diffusion-webui\extensions\sd-webui-controlnet\models

  • Click the model and then click the "download" link
    

    

  • In class we used the models:

    • control_sd15_openpose.pth
    • control_sd15_canny.pth

Using ControlNet

1. Generate an image
   
   
2. Send to Img2Img
3. Put in the new prompts
   
   
4. Modify ControlNet settings
   • Before clicking generate:
     1. Enable ControlNet
     2. Select the OpenPose Preprocessor
     3. Select the OpenPose Model
        
        
5. You should see both the image and the detected pose generate in the output section
   
   
   
   

Class 5

Training subject and style with dreambooth

For the subject (60 images)

• 30% Headshot
• 30% Portrait
• 30% Full body
• 10% Wacky shots

For the aesthetic (105 images)

• vary characters and environments to that only the style is consistent