How could I improve my data structure for superpixel coordinates storage in image segmentation algorithms?

Question

I am attempting to produce a python implementation of a certain superpixel algorithm. To do so, I would need a variable sk which stores the coordinates of each pixel in different superpixels.

To be more specific, my current approach is sk = np.empty([N, N, 2], dtype=int), in which N is the number of total pixels; the first dimension refers to a specific superpixel label; the second dimension refers to the index of a specific pixel; the last "2" is for x and y coordinates for that pixel. So sk[3, 2] = [10, 10] means that the 2nd pixel in superpixel 3 is the pixel at (10, 10). I am also using numba for enhancing numpy array calculations, so speed is a major concern here.

As you can see, the above sk would take around 7TB if I were to do so for a 1080p image. However, I am currently using Mac, and OS doesn't allocate memory for sk until I try to access it. And thanks to paging, only the pages associated with sections I wish to access is affected by my modifications. I am wondering if numpy array is the best data structure I could use for this, or if perhaps there are better alternatives out there.

My current approach:

sk = np.empty([N, N, 2], dtype=int)
inside_index = [0]  # inside_index[x] is the number of pixels in superpixel x

I have tried scipy.sparse.csr_matrix and scipy.sparse.lil_matrix. They're fixed 2-dimensional, so I attempted to store coordinates for a single superpixel in one row like this: [x1 y1 x2 y2 x3 y3...]. Issues are, the first one is created for matrix calculations, so its inefficient when it comes to changing the sparity of it (adding stuff in); the second one is not meant for accessing the data within, so getting a specific value from sk is once again inefficient. Lists are simply too slow for other complex, vector-based calculations.