How is Spark reading my image using the image format?

Question

It might be a silly question but I can't figure out how Spark read my image using the spark.read.format("image").load(....) argument.

After importing my image which gives me the following:

>>> image_df.select("image.height","image.width","image.nChannels", "image.mode", "image.data").show()
+------+-----+---------+----+--------------------+
|height|width|nChannels|mode|                data|
+------+-----+---------+----+--------------------+
|   430|  470|        3|  16|[4D 55 4E 4C 54 4...|
+------+-----+---------+----+--------------------+

I arrive to the conclusion that:

• my image is 430x470 pixels,
• my image is colored (RGB due to nChannels = 3) which is an openCV compatible-type,
• my image mode is 16 which corresponds to a particular openCV byte-order.
  • Does someone knows which website/documentation I could browse to know more about it?
• the data in the data column is of type Binary but:
  • when I run image_df.select("image.data").take(1) I got an output which seems to be only one array (see below).

>>> image_df.select("image.data").take(1)

# **1/** Here are the last elements of the result
....<<One Eternity Later>>....x92\x89\x8a\x8d\x84\x86\x89\x80\x84\x87~'))]

# 2/ I got also several part of the result which looks like:
.....\x89\x80\x80\x83z|\x7fvz}tpsjqtkrulsvmsvmsvmrulrulrulqtkpsjnqhnqhmpgmpgmpgnqhnqhn
qhnqhnqhnqhnqhnqhmpgmpgmpgmpgmpgmpgmpgmpgnqhnqhnqhnqhnqhnqhnqhnqhknejmdilcilchkbh
kbilcilckneloflofmpgnqhorioripsjsvmsvmtwnvypx{ry|sz}t{~ux{ry|sy|sy|sy|sz}tz}tz}tz}
ty|sy|sy|sy|sz}t{~u|\x7fv|\x7fv}.....

What come next are linked to the results displayed above. Those might be due to my lack of knowledge concerning openCV (or else). Nonetheless:

• 1/ I don't understand the fact that if I got an RGB image, I should have 3 matrix but the output finishes by .......\x84\x87~'))]. I was more thinking on obtaining something like [(...),(...),(...\x87~')].
• 2/ Is this part has a special meaning? Like those are the separator between each matrix or something?

To be more clear about what I'm trying to achieve, I want to process images to do pixel comparison between each images. Therefore, I want to know the pixel values for a given position in my image (I assume that if I have an RGB image, I shall have 3 pixel values for a given position).

Example: let's say that I have a webcam pointing to the sky only during the day and I want to know the values of a pixel at a position corresponding to the top left sky part, I found out that the concatenation of those values gives the colour Light Blue which says that the photo was taken on a sunny day. Let's say that the only possibility is that a sunny day takes the colour Light Blue.
Next I want to compare the previous concatenation with another concat of pixel values at the exact same position but from a picture taken the next day. If I found out that they are not equal then I conclude that the given picture was taken on a cloudy/rainy day. If equal then sunny day.

Any help on that would be highly appreciated. I have vulgarized my example for a better understanding but my goal is pretty much the same. I know that ML model can exist to achieve those stuff but I would be happy to try this first. My first goal is to split this column into 3 columns corresponding to each color code: a red matrix, a green matrix, a blue matrix

Answer 1

I think I have the logic. I used the keras.preprocessing.image.img_to_array() function to understand how the values are classified (since I have an RGB image, I must have 3 matrix: one for each color R G B). Posting that if someone wonder how it works, I might be wrong but I think I have something :

from keras.preprocessing import image
import numpy as np
from PIL import Image

# Using spark built-in data source
first_img = spark.read.format("image").schema(imageSchema).load(".....")
raw = first_img.select("image.data").take(1)[0][0]
np.shape(raw)
(606300,) # which is 470*430*3



# Using keras function
img = image.load_img(".../path/to/img")
yy = image.img_to_array(img)
>>> np.shape(yy)
(430, 470, 3) # the form is good but I have a problem of order since:

>>> raw[0], raw[1], raw[2]
(77, 85, 78)
>>> yy[0][0]
array([78., 85., 77.], dtype=float32)

# Therefore I used the numpy reshape function directly on raw 
# to have 470 matrix of 3 lines and 470 columns:

array = np.reshape(raw, (430,470,3))
xx = image.img_to_array(array)     # OPTIONAL and not used here

>>> array[0][0] == (raw[0],raw[1],raw[2])
array([ True,  True,  True])

>>> array[0][1] == (raw[3],raw[4],raw[5])
array([ True,  True,  True])

>>> array[0][2] == (raw[6],raw[7],raw[8])
array([ True,  True,  True])

>>> array[0][3] == (raw[9],raw[10],raw[11])
array([ True,  True,  True])

So if I understood well, spark will read the image as a big array - (606300,) here - where in fact each element are ordered and corresponds to their respective color shade (R G B).
After doing my little transformations, I obtain 430 matrix of 3 columns x 470 lines. Since my image is (470x430) for (WidthxHeight), each matrix corresponds to a pixel heigth position and inside each: 3 columns for each color and 470 lines for each width position.

Hope that helps someone :)!