What causes noise in SSIM for detecting and showing image difference?

Question

I'm trying to write a script to detect the difference between two architectural floor plan images (but it has to be a big enough difference). So far, I've used non-ML methods and SSIM works the best but the results are not optimal for some reason. I crop and align both images using feature matching and I've checked this is done right manually, and erase any text so ideally, my code should detect the differences correctly but it's sensitive to noise such as different textures in the image (e.g. spots/speckles to denote a speckled floor). Any clue what's causing the noise?

My code for the SSIM bit is below:

def find_diff(before, after):
before_grey = cv2.cvtColor(before, cv2.COLOR_BGR2GRAY)
after_grey = cv2.cvtColor(after, cv2.COLOR_BGR2GRAY)

# Compute the Structural Similarity Index (SSIM) between the two images, ensuring that the difference image is returned
(score, diff) = compare_ssim(before_grey, after_grey, full=True)
diff = (diff * 255).astype("uint8")
print("SSIM: {}".format(score))

# Threshold the difference image, then find contours to obtain the regions of the two input images that differ
thresh = cv2.threshold(diff, 0, 255, cv2.THRESH_BINARY_INV | cv2.THRESH_OTSU)[1]
contours = cv2.findContours(thresh.copy(), cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
contours = imutils.grab_contours(contours)

for c in contours:
    # Compute the bounding box of the contour and then draw the bounding box on both input images to represent where the two images differ
    (x, y, w, h) = cv2.boundingRect(c)
    cv2.rectangle(before, (x, y), (x + w, y + h), (0, 0, 255), 2)
    cv2.rectangle(after, (x, y), (x + w, y + h), (0, 0, 255), 2)

return before, after, diff

I've tried searching online for ML methods (which I think may produce better results) but I can't find any suitable resources other than to give a similarity score between two images. Do you have any ideas for how I could go about this?

The results I'm currently getting can be seen below where the first drawing is the before, the second is the after, and the third is the difference detected. The black sections on the side are caused by the cropping and the blurry sections are caused by the text erasure but this is something I don't need to worry about as of right now. The blue rectangles are the differences that are being detected.

Result 1:

Result 2:

Result 3: