# SD from virtual environment

## Pips
`pip install safetensors`
`pip install diffusers --upgrade`
`pip install invisible_watermark transformers accelerate safetensors
`
For apple silicon(or devices not compatible with CUDA):
`pip3 install torch torchvision torchaudio
`

## Variations

#### Original - hello world

from diffusers import DiffusionPipeline
import torch

pipe = DiffusionPipeline.from_pretrained(
	"stabilityai/stable-diffusion-xl-base-1.0", 
	torch_dtype=torch.float16, 
	use_safetensors=True, 
	variant="fp16"
)
pipe.to("cuda")

# if using torch < 2.0
# pipe.enable_xformers_memory_efficient_attention()

prompt = "An astronaut riding a green horse"

images = pipe(prompt=prompt).images[0]

* [documentation](https://huggingface.co/docs/transformers/main_classes/model)
* `torch_dtype=`
* Under Pytorch a model normally gets instantiated with torch.float32 format. This can be an issue if one tries to load a model whose weights are in fp16, since it’d require twice as much memory. To overcome this limitation, you can explicitly pass the desired dtype using torch_dtype argument:
* [docs](https://pytorch.org/docs/stable/tensor_attributes.html)
* `torch.float16`
* `torch.float32`
* `"auto"`
* `use_safetensors=` boolean
* `variant=` Optional
* change `cuda` to `mps` to run on Metal Shaders
* these are pretty slow and a little janky on Apple Silicon. Take a look at the CoreML as an alternative approach

### Performance help(seems to need at least one of these -- massively slows down)
* [docs](https://huggingface.co/docs/diffusers/main/en/optimization/memory)
* `pipe.enable_attention_slicing()`
* Enable sliced attention computation. When this option is enabled, the attention module splits the input tensor in slices to compute attention in several steps. For more than one attention head, the computation is performed sequentially over each head. This is useful to save some memory in exchange for a small speed decrease.
* `pipe.enable_vae_slicing()`
* Enable sliced VAE decoding. When this option is enabled, the VAE will split the input tensor in slices to compute decoding in several steps. This is useful to save some memory and allow larger batch sizes.
* `pipe.enable_vae_tiling()`
* Enable tiled VAE decoding. When this option is enabled, the VAE will split the input tensor into tiles to compute decoding and encoding in several steps. This is useful for saving a large amount of memory and to allow processing larger images.

# CoreML Builds

## Downloading models

`brew install huggingface-cli`
* download repo: `huggingface-cli download HuggingFaceH4/zephyr-7b-beta`
* download specifics: `huggingface-cli download stabilityai/stable-diffusion-xl-base-1.0 --include "*.safetensors" --exclude "*.fp16.*"*`

When you need to download models, keep in mind the method you're using to render images.
* CoreML models for Python are "Packages"
* while CoreML models for Swift are "Compiled" models
* these will also be the ones that are compatible with the various UIs(currently only tested on Mochi)
* you should be able to find both variations in your model repos

#### [Github](https://github.com/apple/ml-stable-diffusion#-converting-models-to-core-ml)

Apple's approach to running image generators on Apple Silicon(since MPS isn't keeping up all that well).

`pip install coremltools`

#### Requires either:
- every .safetensors package to be converted into a coreml directory/restructured data
- NOTE: this process is extremely resource-intensive, and I had it fail a number of times due to not enough ram
- start with
- `huggingface-cli login` and API key for your account
- and find an text-to-img/img-to-img/text-to-vid stable diffusion model
- then:

python -m python_coreml_stable_diffusion.torch2coreml --convert-unet --convert-text-encoder --convert-vae-decoder --convert-safety-checker --model-version <model-version-string-from-hub> -o <output-mlpackages-directory>

- the `<model-version-string-from-hub>` is the `developer/package` endpoint on a huggingface repo: `stability/stable-diffusion-xl-1.0`

This creates a CoreML directory out of the models -- along with all the variations.

OR you can download pre-built libraries

- you can download pre-formatted CoreML directories from Huggingface
- you'll also need
- find models from https://huggingface.co/apple
- then make a directory to work in and create a .py file with this:

from huggingface_hub import snapshot_download
from pathlib import Path

repo_id = "apple/coreml-stable-diffusion-v1-4"
variant = "original"

model_path = Path("./models") / (repo_id.split("/")[-1] + "_" + variant.replace("/", "_"))
snapshot_download(repo_id, allow_patterns=f"{variant}/*", local_dir=model_path, local_dir_use_symlinks=False)
print(f"Model downloaded at {model_path}")

- Give it the right `repo_id` then run it
- it should create a models folder and deposit it in
- you might need to customize the variant to reflect what's in the Huggingface repo

### That's all you need to start generating

The simple plug-and-run:

python -m python_coreml_stable_diffusion.pipeline --prompt "a photo of an astronaut riding a horse on mars" -i <core-ml-model-directory> -o </path/to/output/image> --compute-unit ALL --seed 93

Adjustments:
- compute-unit
- determines cpu/gpu/ram use in the process. Using all three does not always increase speeds
- `ALL`
- `CPU_AND_GPU`
- `CPU_ONLY`
- `CPU_AND_NE`