CODEWITHSUNDEEP
pythonpython-imaging-libraryout-of-memorygisimage-stitching
Joseph T. Richardson
Joseph T. Richardson

Reputation: 113

How to stitch together a huge image without memory issues?

I am trying to stitch together (too) many images -- for what it's worth, PNG map tiles downloaded from a Tile Map Service -- into one large image. Perhaps naïvely, I'm using Python Pillow. It worked up to zoom level 9, but now I'm getting a MemoryError. What is a better way to do this? My system has 16 GB of RAM, and ought to be able to do this in virtual memory I supposed, but I guess not.

The code I'm using (this is from Stamen's the-ultimate-tile-stitcher):

    out_img = Image.new('RGBA', (out_w, out_h), (0, 0, 255, 0))
    for filepath in tqdm.tqdm(filepaths):
        x, y = xy(filepath)
        x, y = x - x_0, y - y_0
        tile = Image.open(filepath)
        out_img.paste(tile, box=(x * tile_w, y * tile_h, (x + 1) * tile_w, (y + 1) * tile_h))

At zoom level 9, I have 512 x 512 tiles, each 512 x 512 pixels -- resulting in a 131,072 x 131,072-pixel image. The stack trace for the error I get:

Traceback (most recent call last):
  File "stitcher.py", line 65, in <module>
    main()
  File "stitcher.py", line 53, in main
    out_img = Image.new('RGBA', (out_w, out_h), (0, 0, 255, 0))
  File "C:\Anaconda3\envs\gis\lib\site-packages\PIL\Image.py", line 2671, in new
    return im._new(core.fill(mode, size, color))
MemoryError

Is there some way to make PIL swallow this huge pill? Or some better way to do what I'm trying to do?

Upvotes: 1

Views: 856

Answers (1)

Thomas Weller
Thomas Weller

Reputation: 59302

My system has 16 GB of RAM, and ought to be able to do this in virtual memory I supposed, but I guess not.

You are mixing two concepts here: 16 GB of RAM is physical RAM and is only barely related to virtual memory. Your virtual memory may be as small as 2 GB (for a 32 bit application that is not large address aware) or as big as 8 TB, if your pagefile has a large enough disk to expand to.

131,072 x 131,072-pixel image

The calculation is easy: 131072 * 131072 pixels * 4 bytes/pixel = 68719476736 bytes = 64 GB, because you need at least 4 bytes to store R, G, B and A.

So, along with your 16 GB physical RAM, you need at least 48 GB of additional memory in the pagefile. This calculation does neither include the size for the 260000 small images nor other applications or the operating system.

What is a better way to do this?

It's hard to say and depends on what you want to do with that image. Saving it as a file will hardly be possible, because many image formats have limits for the amout of pixels.

BMP version 2 allows a maximum of 32k pixels in size. JPG is limited to 64k pixels. BMP version 3 should be 2G pixels and PNG should go up to 4G pixels, but you'll really have trouble finding an application that smoothly displays those images. IMHO, GDI on Windows will start complaining at 2 GB.

Basically that's one of the reasons why tiles are used.

Upvotes: 1

Related Questions