CODEWITHSUNDEEP
pythonimageimage-processinglarge-datalarge-files
LostXOR
LostXOR

Reputation: 198

Generate image file without keeping the whole image in memory

I wrote a program to generate RGB color values using three equations and write them to a data file:

from struct import pack

#User-set variables
size = [16384, 16384]
center = [0, 0]
r_equation = "x ^ y"
g_equation = "x - y"
b_equation = "x + y"
data_path = "data.dat"

#Calculate x and y offset
offset = [center[0] - (size[0] // 2), center[1] - (size[1] // 2)]

#Compile code for calculating RGB values
r_code = compile(r_equation, "r_equation", "eval")
g_code = compile(g_equation, "g_equation", "eval")
b_code = compile(b_equation, "b_equation", "eval")

data_file = open(data_path, "wb")

#Pack image size as first 4 bytes
data_file.write(pack("HH", *size))

#Iterate through the graph
for y in range(offset[1], size[1] + offset[1]):
    for x in range(offset[0], size[0] + offset[0]):
        #Calculate the RGB values
        rgb = [int(eval(r_code)) % 256,
               int(eval(g_code)) % 256,
               int(eval(b_code)) % 256]
        #Pack the RGB values as 3 bytes
        data_file.write(pack("BBB", *rgb))

data_file.close()

I want to create an image from the data file, but I don't have enough RAM to load the entire image. I wrote this program to read pieces of the file:

from struct import unpack

#User-set variables
data_path = "data.dat"
image_path = "image.xxx"

data_file = open(data_path, "rb")

#Unpack first 4 bytes to get image size
size = unpack("HH", data_file.read(4))

#create_image(size, image_path)

for data in data_file.read(1048576 * 3): #Read 1M pixels at a time
    #write_to_image(data)

If I try to use PIL to write the entire image file, it soon runs out of memory. Is there a way to write the image file in pieces so one piece is in memory at a time?

Upvotes: 0

Views: 459

Answers (2)

jcupitt
jcupitt

Reputation: 11190

Here's your program (I think!) done with pyvips:

#!/usr/bin/python3

import sys
import pyvips

# make a huge two-band image where each pixel has the value of its (x, y)
# coordinate
xy = pyvips.Image.xyz(16384, 16384)

# subtract the half way point, so each pixel is now -8192 to +8191
xy -= [8192, 8192]

# compute three one-band images from (x, y) ... image[n] gets the nth band
r = xy[0] ^ xy[1]
g = xy[0] + xy[1]
b = xy[0] - xy[1]

# join as an RGB image, modulo 256
rgb = r.bandjoin([g, b]).copy(interpretation="srgb") & 0xff

rgb.write_to_file(sys.argv[1])

On this modest laptop, I see:

$ /usr/bin/time -f %M:%e ./pattern.py x.jpg
136712:5.81s

6s and 137mb of memory.

Upvotes: 1

mprostak
mprostak

Reputation: 246

You could try to use this python library: https://github.com/libvips/pyvips

Programs that use pyvips don't manipulate images directly, instead they create pipelines of image processing operations building on a source image. When the end of the pipe is connected to a destination, the whole pipeline executes at once, streaming the image in parallel from source to destination a section at a time.

Vips makes processing big images possible and it does it pretty fast.

Because pyvips is parallel, it's quick, and because it doesn't need to keep entire images in memory, it's light.

Upvotes: 3

Related Questions