How to generate images with the same seed but with different kind of noise schedulers using Diffusers

Question

I'm trying to use the Diffusers library to generate images with different schedulers (Just generate images I don't want to have to prompt). For this I follow this tutorial: https://huggingface.co/docs/diffusers/using-diffusers/write_own_pipeline. As well as: https://huggingface.co/docs/diffusers/using-diffusers/schedulers.

Here is the code:

import torch
from diffusers import DDPMScheduler, EulerAncestralDiscreteScheduler, EulerDiscreteScheduler, UNet2DModel
from PIL import Image
import numpy as np

I first defined my schedulers and the model, according to the tutorial:

schedulers = [DDPMScheduler.from_pretrained("google/ddpm-cat-256"), EulerAncestralDiscreteScheduler.from_pretrained("google/ddpm-cat-256"), EulerDiscreteScheduler.from_pretrained("google/ddpm-cat-256")]

model = UNet2DModel.from_pretrained("google/ddpm-cat-256", use_safetensors=True)

Setting timesteps to 50 for every scheduler:

for scheduler in schedulers:
    scheduler.set_timesteps(50)

Getting the initial noise:

sample_size = model.config.sample_size
noise = torch.randn((1, 3, sample_size, sample_size))

Denoising the initial noise for each scheduler:

denoised_dict = {}
input = noise

for i, scheduler in enumerate(schedulers):
    denoised_dict[i] = []
    
    for t in scheduler.timesteps:
        with torch.no_grad():
            noisy_residual = model(input, t).sample
        previous_noisy_sample = scheduler.step(scheduler.scale_model_input(noisy_residual, t), t, input).prev_sample
        input = previous_noisy_sample
        
    denoised_dict[i] = input

Plotting a resulting image:

image = (denoised_dict[0] / 2 + 0.5).clamp(0, 1).squeeze()
image = (image.permute(1, 2, 0) * 255).round().to(torch.uint8).cpu().numpy()
image = Image.fromarray(image)
image

The problem is that images generated using EulerDiscreteScheduler and EulerAncestralDiscreteScheduler are just noise, meaningless, not even displayed, etc.

I tried several ways, including using only EulerAncestralDiscreteScheduler and following exactly the tutorial. I got a message: The scale_model_inputfunction should be called beforestepto ensure correct denoising. SeeStableDiffusionPipeline for a usage example.

I also called EulerAncestralDiscreteScheduler as suggested in this tutorial: https://huggingface.co/docs/diffusers/using-diffusers/schedulers:

scheduler = DDPMScheduler.from_pretrained("google/ddpm-cat-256")

scheduler = EulerAncestralDiscreteScheduler.from_config(scheduler.config)

Anyway, nothing works and no matter how many times I read the tutorials, I just can't do it.

Anyone could help me out of this situation?

Answer 1

Use the built in ones to make it easier.

from diffusers import EulerDiscreteScheduler

euler= EulerDiscreteScheduler(
    beta_start=0.001775,
    beta_end=0.012,
    beta_schedule="linear",
    num_train_timesteps=1000,
    prediction_type="epsilon",
    steps_offset=1,
    trained_betas=None,
    interpolation_type="linear",
    use_karras_sigmas=False,
    timestep_spacing="linspace",
)

# load your model
pipe.scheduler = euler

image = pipe(prompt).image[0]