Stable Diffusion XL Inpainting struggles with masked background areas, often failing to generate new content there

Question

I am working on an Inpainting task and I am using Stable diffusion XL 1.0 Inpainting. It is working very well when I mask people, animals or objects, but when I give it a mask of background, it doesn't generate anything and give me the original input image. I have tried changing parameters as well but nothing could make it work. Below is the approach I am using:

model_id = "diffusers/stable-diffusion-xl-1.0-inpainting-0.1"

model = AutoPipelineForInpainting.from_pretrained(
     model_id, 
     torch_dtype = torch.float16,
     cache_dir = "model_cache",
)

model = pipeline.to("cuda")

if base64_img is not None:
     img_data = base64.b64decode(base64_img)
     img = Image.open(io.BytesIO(img_data))

     print("Recieved image of size: ", img.size)
        
if base64_mask is not None:
     mask_data = base64.b64decode(base64_mask)
     mask = Image.open(io.BytesIO(mask_data))

     print("Recieved mask of size: ",mask.size)

img = img.convert("RGB")
mask = mask.convert("L")
        
generator = torch.Generator(device="cuda").manual_seed(0)
        
image = model(
     prompt=prompt,
     negative_prompt=IMG_INPAINTING_NEG_PROMPT,
     image=img,
     mask_image=mask,
     guidance_scale=8.0,
     num_inference_steps=25,
     strength=strength,
     generator=generator,
).images[0]