Installing the Stable Diffusion v1 model and Hugging Face's Diffusers library on a Fedora workstation

Diffusers is an open-source Python library provided by Hugging Face (a US company). The library allows you to easily use various pre-trained diffusion models.

In this article we describe how to install the Stable Diffusion model and the Diffusers library on a Fedora (36) Linux workstation.

Conda package and environment management

We start with the installation of Conda (the package and environment management system underneath the Anaconda suite), which allows us to set up a sandbox environment for the Python libraries that Stable Diffusion depends on.

dnf install conda

The Stable Diffusion v1 software

Next we create a new directory in which we clone the original Stable Diffusion software.

mkdir huggingface-diffusers/
cd huggingface-diffusers/
git clone https://github.com/CompVis/stable-diffusion.git
cd stable-diffusion/

Setting up the Conda environment

Now we can use the file 'environment.yaml' to set up our Conda environment. But first we update the versions of the libraries that will be downloaded as part of this environment. These are the changes we made to this setup file:

python=3.8.16
pip=20.3.4
cudatoolkit=11.8
pytorch=1.11.0  [as is, updated later to version 2.0.1; see below]
torchvision=0.12.0  [as is, updated later to version 0.15.2; see below]
numpy=1.19.5  [updated later to version 1.20.3; see below]
pytorch-lightning==1.4.9
transformers==4.30.2

Running Conda now creates a new environment 'ldm', and downloads and installs all specified packages and their dependencies:

conda env create -f environment.yaml

The configuration is saved in the directory '.conda/envs/ldm/'. It can be activated like this (note the changed command prompt):

conda activate ldm

and deactivated like this:

conda deactivate

If you want to start over again, simply deactivate the environment and trash its directory '~/.conda/envs/ldm/'.

Installing the Stable Diffusion model

Now that the software is there, it's time to install the Stable Diffusion model itself, that is the weights that are the result of training the model. These weights can and are being used to further train the model, thereby incorporating new input, which is why a set of weights is often called a checkpoint.

In this case, we downloaded and installed Stable Diffusion version 1.5, which is the latest checkpoint of Stable Diffusion v1 and available through Hugging Face. The details of this checkpoint are described in its so-called Model Card, where you can read how version 1.5 itself is built on version 1.2. Similarly, there is the Stable-Diffusion-Inpainting model, which is based on Stable Diffusion version 1.5 and additionally trained for inpainting.

Testing Stable Diffusion

After downloading and installing the checkpoint file 'v1-5-pruned-emaonly.ckpt' in the (new) directory 'huggingface.co-models/', we can test the Stable Diffusion software like this:

conda activate ldm
cd stable-diffusion/
python scripts/txt2img.py --prompt "a photograph of an astronaut riding a horse" --plms \
    --ckpt ../huggingface.co-models/v1-5-pruned-emaonly.ckpt

Which results in this error message:

ERROR: AssertionError: Torch not compiled with CUDA enabled

It turns out that the version of 'pytorch' that was installed, does not come with CUDA support. Instead it relies on the CPU to do its calculations.

We can confirm this problem this way:

(ldm) [user@host stable-diffusion]$ conda list
...
pytorch    1.11.0    py3.8_cpu_0    pytorch
...

Or we can use this little Python program:

import torch

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
print('Using device:', device)

which produces this output:

Using device: cpu

To remedy this problem, we need to uninstall 'pytorch' and install the version that does have CUDA-support built in:

conda uninstall pytorch

(which removes 'diffusers-0.17.1-pyhd8ed1ab_0', 'pytorch-1.11.0-py3.8_cpu_0', 'torchvision-0.12.0-py38_cpu' and all depending packages)

conda install pytorch=2.0.1 torchvision==0.15.2 torchaudio==2.0.2 \
    cudatoolkit=11.8 pytorch-cuda=11.8 \
    -c pytorch -c conda-forge -c nvidia

(which installs 'pytorch-2.0.1', 'torchaudio-2.0.2', 'torchvision-0.15.2')

Oddly, we had to uninstall en reinstall a second time to actually get the CUDA-enabled software in place.

Upgrading 'numpy'

Another run of the 'txt2img.py' script resulted in a new error:

ImportError: this version of pandas is incompatible with numpy < 1.20.3
your numpy version is 1.19.5.
Please upgrade numpy to >= 1.20.3 to use this pandas version

Which we simply solved by upgrading 'numpy':

conda install numpy=1.20.3

CUDA out of memory

The errror message we receive after we have resolved these issues, is no longer related to any software dependency issues, but simply to the fact that the video adapter of this workstation does not have enough VRAM available to load the full model:

torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 1.50 GiB
(GPU 0; 11.76 GiB total capacity; 8.63 GiB already allocated; 1.13 GiB free; 8.78 GiB reserved in total by PyTorch)
If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.
See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

This particular workstation is equiped with a GeForce RTX 3060 graphics adapter, featuring 12 GB of VRAM, while double this size is required to work with the full model, e.g. a GeForce RTX 3090 or an RTX 4090.

We'll leave this issue for now, and work around it using Diffusers later on.

Installing Diffusers

After activating the 'ldm' environment, we install Diffusers like this:

conda activate ldm
conda install diffusers -c conda-forge

Transformers version mismatch

The problem we ran into when running the 'diffusers-cli' command, is a version mismatch:

ImportError: cannot import name 'SAFE_WEIGHTS_NAME' from 'transformers.utils'

The best solution was to upgrade 'transformers' to version 4.30.2, which we already anticipated in the changes to the 'environment.yaml' setup file described above.

Testing the Diffusers library

Now it's time to test the actual Diffusers library. We do that using this little Python program to create our first image:

from diffusers import StableDiffusionPipeline
import torch

model_id = "runwayml/stable-diffusion-v1-5"
device = "cuda"
output = "output.png"
prompt = "a cat looking through a glass fish bowl"

pipe = StableDiffusionPipeline.from_pretrained(model_id, revision="fp16", torch_dtype=torch.float16)
pipe = pipe.to(device)

image = pipe(prompt).images[0]  
    
image.save(output)

Note that Diffusers does not use the '.ckpt' file format, but the '.safetensors' format created by Hugging Face. That means that we cannot use the 'v1-5-pruned-emaonly.ckpt' file we installed and used with Stable Fusion before. Instead, the 'StableDiffusionPipeline' function will download the 'v1-5-pruned-emaonly.safetensors' file the first time it's called.

Also note that we have added the parameters 'revision="fp16", torch_dtype=torch.float16' to the 'StableDiffusionPipeline' call. These make sure that the model with half precision weights (float16) is downloaded and used instead of the full float32 model. This prevents CUDA from running out of VRAM like we experienced before when testing the original Stable Fusion scripts.

If everything has been installed correctly, running the Python program above now (finally) results in the output file 'output.png' being created.

What's next

Note that in the end the install of Diffusers turns out to be completely independent of Stable Diffusion. You can use both independently, each in its own Conda sandbox. We'll check out Stable Diffusion v2 (which generates images measuring 768x768 instead of 512x512) and Stable Diffusion web UI (a web front-end for Stable Diffusion based on Gradio) next.