CODEWITHSUNDEEP
pythontensorflow
Aeteros
Aeteros

Reputation: 673

Dramatically slow copying(?) to tensorflow

I'm working with hyperspectral image of size [610, 340, 102] from here saved in .mat file. After loading it with scipy, it is represented as array. Prior to make computations I have to reshape it to [610 * 340, 102]. The code below is running on desktop machine with i5-6400 processor:

import tensorflow as tf
import scipy.io as spio
import time

pavia_u = spio.loadmat("./../data/PaviaU.mat")["paviaU"]
original_shape = tf.shape(pavia_u)
pavia_u_reshaped = tf.to_float(tf.squeeze(tf.reshape(pavia_u, [-1, original_shape[0] * original_shape[1], original_shape[2]])))

with tf.Session() as session:
    start = time.clock()
    pavia_u_reshaped_value = session.run(pavia_u_reshaped)
    end = time.clock()
    print("Reshape time: {}".format(end - start))

and it prints: Reshape time: 157.7449091748693

It seems that reshape operation on so relative small dataset should not be so slow. Anyway, I moved reshaping to numpy a

pavia_u = spio.loadmat("./../data/PaviaU.mat")["paviaU"]
original_shape = pavia_u.shape
pavia_u_reshaped = pavia_u.reshape([-1, original_shape[0] * original_shape[1], original_shape[2]]).squeeze())

with tf.Session() as session:
    start = time.clock()
    session.run(tf.identity(pavia_u_reshaped))
    end = time.clock()
    print("Test time: {}".format(end - start))

which prints: Test time: 143.26529380869727

What consumes so much time now? How to fix this.

Upvotes: 0

Views: 515

Answers (1)

Vijay Mariappan
Vijay Mariappan

Reputation: 17201

Reshape is not the reason for the slow time, its the initialisations that is taking time. Changing the code to :

original_shape = tf.shape(pavia_u)
X = tf.Variable(pavia_u, dtype=tf.float32)

pavia_u_reshaped = tf.to_float(tf.squeeze(tf.reshape(X, [-1, original_shape[0] * original_shape[1], original_shape[2]])))


sess = tf.InteractiveSession()
start = time.clock()
tf.global_variables_initializer().run()
end = time.clock()
print("Initialization time: {}".format(end - start))

start = time.clock()
pavia_u_reshaped_value = sess.run(pavia_u_reshaped)
end = time.clock()
print("Reshape time: {}".format(end - start))

# Initialization time: 734.495346
# Reshape time: 1.897106

Upvotes: 1

Related Questions