Cropping Images in Python/OpenCV so Image Doesn't Include Dead/White/Black Space (Different Size Height/Width)

Question

Background

I have been working to stitch together multiple images in a grid (2 by 2 in this case) using SIFT and ANN. When all of the images are the same orientation, translation scale factor, my images are coming out perfectly.

Input Images: 4 Input Images

Stitched Image (Grayscale):
Stitched Image

The issue arises when I translate the images (this algorithm needs to run on actual datasets where every image is not perfect). When I translate the input images even 1 pixel from their original orientation, I begin seeing black lines appear all over the images.

Top-Left Image (Translated 20 pixels in the x and y directions):
Image with Thick Horizontal Black Line
(The line here is 20 pixels thick).

Top-Left and Top-Right Images (Translated 5 pixels in the x and y directions):
Image with Horizontal and Vertical Black Line.
(The vertical line here is 5 pixels thick and the horizontal line here is 1 pixel thick).

What I've Done So Far

I did some investigating and the black lines show up during the stitching process, when parts of the image are lost, due to the translation. The cropping algorithm includes these missing pieces in the final image (showing up as black).

My overall stitching algorithm involves stitching all the horizontal images together first and then loops vertically to stitch the images vertically. This means that if there is any blank space from one of the previous stitched images, it carries forward and ends up appearing somewhere along the final image:
See this image for the first iteration of the algorithm, keeping the black space at the bottom: Top Image

Here's the algorithm I incorporated to remove the seams and extra black space:

# CLEAN UP IMAGE: The following code will clean up the merged image, removing any seams that may arise, ensuring the image stays grayscale, and cropping the image to scale. 
def CleanImage(result):
    # Remove Seams, Ensure Grayscale, Crop Image to Scale
    
    # grayscale images
    finalG = result
    thresh = cv.threshold(finalG, 0, 255, cv.THRESH_BINARY|cv.THRESH_OTSU)[1]
    #OTSU's binarization determines the best value to separate the peaks as certain pixels/colors appear

    cnts = cv.findContours(thresh.copy(), cv.RETR_EXTERNAL, cv.CHAIN_APPROX_SIMPLE)
    print(cnts)
    cnts = imutils.grab_contours(cnts)
    print("Contours", len(cnts))

    # find maximum contour area
    c = max(cnts, key=cv.contourArea)
    (x, y, w, h) = cv.boundingRect(c)

    # crop image to smallest box coordinates
    finalG = finalG[y:y + h, x:x + w]
    
    cnts = cv.findContours(thresh.copy(), cv.RETR_EXTERNAL, cv.CHAIN_APPROX_SIMPLE)

    #output final result
    printF(finalG)
    
    return finalG

The issue is that this code crops to the maximum x and y values, meaning that if parts of one side of the image are lost but the other side is not (see this image above), then the crop still keeps the dead space.

My Question

My question here is: How do I crop these images so that the dead space is removed (e.g. how do I crop these images using a coordinate corresponding to my desired edge)? Or, if there is a better solution to my problem, that would be very much appreciated!!

Thank you so much!

Code for the rest of the algorithm (context):

def runAlgo(ImgArray):

    while (len(ImgArray) >= 2):
        (kpsA, featuresA, annoImgA) = SIFTAlgo(ImgArray[0])
        (kpsB, featuresB, annoImgB) = SIFTAlgo(ImgArray[1])
        matches = ANN(annoImgA, annoImgB, featuresA, featuresB, kpsA, kpsB)
        H = Homography(kpsA, kpsB, featuresA, featuresB, matches)
        result = CreateImage(ImgArray[0], ImgArray[1], H)
        finalG = CleanImage(result)
        ImgArray.pop(0)
        ImgArray[0] = finalG
    
    return ImgArray

#main
rows = int(input("Please insert number of rows: "))
columns = int(input("Please insert number of columns: "))

readIm()
# printIm(Images)

for row in range(rows):
    newImages = []
    for col in range(columns):
        newImages.append(Images[0])
        Images.pop(0)
    runAlgo(newImages)
    finalImages.append(newImages[0])

for row in range(rows):
    runAlgo(finalImages)  

printIm(finalImages)