CODEWITHSUNDEEP
pythontensorflow
Tane van Wifferen
Tane van Wifferen

Reputation: 13

How to map sequence to sequence in tensorflow?

I have a 3 dimensional matrix of shape (height, width, 4). In fact it is a bitmap with RGBA values per pixel. I would like to reduce each RGBA set to a set with two values, say [x,y].

see image at imgur com / Blr2EQC

I have tried using map_fn

import cv2
import tensorflow as tf

def map_pixel_to_vector(elt):
    b = elt[0] - 127
    g = elt[1] - 127
    r = elt[2] - 127
    a = elt[3] - 127

    dx = (g * 127) + r
    dy = (a * 127) + b
    return [dx,dy]

file = "example.png"
frame = cv2.imread(file, cv2.IMREAD_UNCHANGED
s = tf.shape(frame)

# reshape to list of pixels
elts = tf.reshape(frame, (s[0]*s[1],4))

# cast from uint8 to int32 to support negative output
elts = tf.dtypes.cast(elts, tf.int32)

# map each pixel to output
elts = tf.map_fn(map_pixel_to_vector, elts)

# reshape back to image resolution
elts = tf.reshape(elts, (s[0], s[1], 2)

Now I would expect this to work, each [rgba] pixel would be reduced to [xy] pixel, but instead I get 

ValueError: The two structures don't have the same nested structure.

First structure: type=DType str=<dtype: 'int32'>

Second structure: type=list str=[<tf.Tensor: id=262537, shape=(), dtype=int32, numpy=98>, <tf.Tensor: id=262540, shape=(), dtype=int32, numpy=210>]

More specifically: Substructure "type=list str=[<tf.Tensor: id=262537, shape=(), dtype=int32, numpy=98>, <tf.Tensor: id=262540, shape=(), dtype=int32, numpy=210>]" is a sequence, while substructure "type=DType str=<dtype: 'int32'>" is not

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "main.py", line 97, in <module>
    loss = loss_fn(exc, [outputs[-1]], [inputs[-1]])
  File "main.py", line 36, in loss_fn
    elts = tf.map_fn(reduce_pixel_to_vector, elts)
  File "/usr/lib/python3.7/site-packages/tensorflow_core/python/ops/map_fn.py", line 268, in map_fn
    maximum_iterations=n)
  File "/usr/lib/python3.7/site-packages/tensorflow_core/python/ops/control_flow_ops.py", line 2714, in while_loop
    loop_vars = body(*loop_vars)
  File "/usr/lib/python3.7/site-packages/tensorflow_core/python/ops/control_flow_ops.py", line 2705, in <lambda>
    body = lambda i, lv: (i + 1, orig_body(*lv))
  File "/usr/lib/python3.7/site-packages/tensorflow_core/python/ops/map_fn.py", line 258, in compute
    nest.assert_same_structure(dtype or elems, packed_fn_values)
  File "/usr/lib/python3.7/site-packages/tensorflow_core/python/util/nest.py", line 313, in assert_same_structure
    % (str(e), str1, str2))

Any help would be greatly appreciated.

Upvotes: 1

Views: 256

Answers (1)

javidcf
javidcf

Reputation: 59691

Your function map_pixel_to_vector is returning a list, not a tensor. You can make it into a tensor for example with tf.stack or tf.convert_to_tensor:

def map_pixel_to_vector(elt):
    b = elt[0] - 127
    g = elt[1] - 127
    r = elt[2] - 127
    a = elt[3] - 127

    dx = (g * 127) + r
    dy = (a * 127) + b
    return tf.stack([dx, dy])

However, you can do the same operation without tf.map_fn more simply and efficiently like this:

import tensorflow as tf
import cv2

file = "example.png"
frame = tf.constant(cv2.imread(file, cv2.IMREAD_UNCHANGED))
elts = tf.dtypes.cast(frame, tf.int32)
r, g, b, a = tf.unstack(elts - 127, num=4, axis=-1)
elts = tf.stack([(g * 127) + r, (a * 127) + b], axis=-1)

Upvotes: 1

Related Questions