# SVA Computational Art - Spring 22
In this course we will explore:

* AI techniques and best practices for artists
* Generative / procedural art techiques and best practices for artists

:::warning

:nerd_face: **We won't be writing code**

However, we will be running some deep learning models using iPython Notebooks (both online using Google Colab and locally Jupyter Notebooks). So you **will** do some basic editing like changing example file names to match the names of the files you upload.

:rocket: **Early-access mode unlocked**
When you are comfortable running an editing Notebooks code, you will be able to test and create content using the latest and greatest AI models-- without needing to wait for anyone to create a UI for you.
:::

## Class 1

### Discussion

:::info

:thinking_face: **What are the use cases for AI in creative productions?**

:::

- concept art
- characters
- environments
- textures

- product design
- information architecture
- user interfaces
- code

- mocap / rigging
- human pose estimation
- rigging and retargeting

#### Most popular deep learning libraries

- PyTorch (Facebook)
- Tensorflow (Google)
- https://www.tensorflow.org/hub/tutorials

### Workshop

:::info
:bulb: **In this workshop, we run and edit our first ML models using TensorflowHub**

First, we look at [MoveNet](#Pose-Estimation), a pose estimation model

Second, we look at [Arbitrary Image Stylization](#Style-Transfer), a style transfer model

Finally, we look at the web UI for Stable Diffision and watch an explainer video on how stable diffusion works
:::

#### Pose Estimation

Movenet estimates 17 key joints in a human (or other biped character)

[Movenet Tutorial](https://www.tensorflow.org/hub/tutorials/movenet)
[Movenet Google Colab](https://colab.research.google.com/github/tensorflow/hub/blob/master/examples/colab/movenet.ipynb)

##### Results and Limitations

In our tests, we noticed that occlusion can hamper the model from detecting all 17 joints.

In this example, Captain America's shield prevents the model from identifying the joints in his right arm.

#### Style Transfer
https://colab.research.google.com/github/tensorflow/hub/blob/master/examples/colab/tf2_arbitrary_image_stylization.ipynb

#### Stable Diffusion
Explainer from AI Coffee Break
{%youtube J87hffSMB60 %}

## Class 2

### How to run Stable Diffusion Locally

#### Install Miniconda

1. [Download Miniconda](https://repo.anaconda.com/miniconda/Miniconda3-latest-Windows-x86_64.exe)

2. Install it for `Just Me`


3. Change Destination Folder to a writeable folder
::: info
:bulb: **Note** In our lab, only the `U:\` drive is writeable, so we use `U:\miniconda` for the install location.
:::


4. Run `Anaconda Prompt`


5. create a conda environment
`conda create --name art`

6. Activate the environment
`conda activate art`

6. Install python 3.10.2

`conda install python=3.10.2 -c conda-forge`

7. Install Cuda Toolkit

:::warning
:warning: **Warning** The remaining instructions are for machines that have NVIDIA GPUs. Before installing, confirm that your machine has an NVIDIA GPU with CUDA of version 11.7 or later installed.
:::

run this command to check the version of Cuda
`nvidia-smi`

:::warning
:warning: **Warning** If your CUDA version is less than 11.7, you will need to update your graphics drivers before continuing.
:::

:::danger
:heavy_exclamation_mark: **Warning** If the `nvidia-smi` command returns an error, your system most likely does not have a CUDA compatible NVIDIA GPU. Do not continue with the steps below. Instead, select the Conda installation options for your system on the [Pytorch installation website](https://pytorch.org/get-started/locally/).
:::
If `nvidia-smi` returns a CUDA Version >= 11.7, run this command to install CUDA Toolkit

`conda install -c "nvidia/label/cuda-11.7.1" cuda-toolkit`

8. Install Pytorch

`conda install pytorch torchvision torchaudio pytorch-cuda=11.7 -c pytorch -c conda-forge -c nvidia`

---

#### Install Stable Diffusion Web UI

:::warning
:warning: **Warning** Before running any of these commands, make sure your terminal has activated the conda environment named `art` environment that we set up in the previous section.

Run `conda activate art` if you are not sure.
:::

1. Clone the automatic1111 repo

    u:
    md src
    cd src
    git clone https://github.com/AUTOMATIC1111/stable-diffusion-webui.git
    cd stable-diffusion-webui

2. Clone the stable diffusion repo inside of the automatic1111 repo

    git clone https://github.com/Stability-AI/stablediffusion repositories/stable-diffusion-stability-ai

3. Install the necessary dependencies

    pip install -r .\requirements.txt
    pip install open_clip_torch
    pip install git+https://github.com/crowsonkb/k-diffusion/

---

#### Use Custom Checkpoints With Stable Diffusion Web UI

::: info
:bulb: **Note** We can use Stable Diffusion model checkpoints that were trained and shared by other artists. In this example, we'll use a checkpoint from [CivitAI](https://civitai.com/)
:::

1. Find a checkpoint that produces results in your desired style.

In this example, I use https://civitai.com/models/2540/elldreths-stolendreams-mix


2. Move the checkpoint into the folder `.\stable-diffusion-webui\models\Stable-diffusion\`


---

#### Run the Stable Diffusion Web UI

:::warning
:warning: **Warning** Before running any of these commands, make sure your terminal has activated the conda environment named `art` environment that we set up in the previous section.

Run `conda activate art` if you are not sure.
:::

1. Run the python command from the `stable-diffusion-webui` directory

    python .\webui.py

2. Open the local URL in your browser

    Model loaded in 30.8s (1.5s create model, 22.7s load weights).
    Running on local URL:  http://127.0.0.1:7860

    To create a public link, set `share=True` in `launch()`.


---

### Stable Diffusion Workshop

:::info

:nerd_face: ***In this workshop we run this notebook: [Stable Diffusion in Google Colab](https://github.com/TheLastBen/fast-stable-diffusion)***

:::

##### Tutorial from Chris Johnston
{%youtube QdExCik6sww %}

* txt2img parameters

* img2img parameters

* tips
* Check lexica prompts (especially the end)
* Check hugging face for models
* waifu diffusion for anime
* [protogen model](https://huggingface.co/spaces/darkstorm2150/Stable-Diffusion-Protogen-x3.4-webui)
* Check [civitai.com](https://civitai.com)
* download checkpoints
* Check huggingface for textual inversion
* [VikingPunk](https://huggingface.co/Conflictx/VikingPunk)

###### Theory behind stable difffusion

{%youtube -lz30by8-sU %}
{%youtube 1CIpzeNxIhU %}

## Class 3

Installing Dreambooth

Save the 1.5-pruned checkpoint from
https://huggingface.co/runwayml/stable-diffusion-v1-5/resolve/main/v1-5-pruned-emaonly.ckpt
to your directory: `.\models\Stable-diffusion`

### Dreambooth Setup

1. Create Model

2. Settings

3. Concepts

* We use our 512x512 cropped images of our face in the `dataset directory`
* We make an empty directory for the classification dataset directory (we will generate 300 images here in a later step)


* The `instance prompt` uses the rare token `ohwx` and our classifier `man`
* The `class prompt` uses a comparible image `photo of a man`
* The `sample image prompt` uses our rare token combined with the class prompt `photo of ohwx man`


* These settings use best practices for number of images as specified by the research papers


4. Saving


5. Generate Class images


(you should see images genearating in the output tab and in the console)


(note that I am getting around 4 iterations per second)


## Class 4 Posing with ControlNet

tutorial:
{%youtube 5z71oxf8kh4 %}

### Instructions for installing ControlNet

* Enable the `sd-webui-controlnet` stable diffusion extension
* Save the checkpoints from [this link](https://huggingface.co/lllyasviel/ControlNet/tree/main/models
) to the folder `stable-diffusion-webui\extensions\sd-webui-controlnet\models`
* Click the model and then click the "download" link

* In class we used the models:
* `control_sd15_openpose.pth`
* `control_sd15_canny.pth
`

#### Using ControlNet

1. Generate an image
2. Send to Img2Img
3. Put in the new prompts
4. Modify ControlNet settings
* Before clicking generate:
1. Enable ControlNet
2. Select the OpenPose Preprocessor
3. Select the OpenPose Model
5. You should see both the image and the detected pose generate in the output section

## Class 5

### Training subject and style with dreambooth

#### For the subject (60 images)
- 30% Headshot
- 30% Portrait
- 30% Full body
- 10% Wacky shots

#### For the aesthetic (105 images)
- vary characters and environments to that only the style is consistent