CODEWITHSUNDEEP
pythonnumpytensorflowmachine-learningkeras
Charlie
Charlie

Reputation: 316

Does it matter what dtype a numpy array is for input into a tensorflow/keras neural network?

If I take a tensorflow.keras model and call model.fit(x, y) (wherex and y are numpy arrays) does it matter what dtype the numpy array is? Am I best to just make the dtype as small as possible (e.g. int8 for binary data) or does this give tensorflow/keras extra work to cast it to a float?

Upvotes: 3

Views: 2422

Answers (1)

Nicolas Gervais
Nicolas Gervais

Reputation: 36674

You should cast your input to np.float32, that's the default dtype for Keras. Look it up:

import tensorflow as tf
tf.keras.backend.floatx()

'float32'

If you give Keras input in np.float64, it will complain:

import tensorflow as tf
from tensorflow.keras.layers import Dense 
from tensorflow.keras import Model
from sklearn.datasets import load_iris
iris, target = load_iris(return_X_y=True)

X = iris[:, :3]
y = iris[:, 3]

ds = tf.data.Dataset.from_tensor_slices((X, y)).shuffle(25).batch(8)

class MyModel(Model):
  def __init__(self):
    super(MyModel, self).__init__()
    self.d0 = Dense(16, activation='relu')
    self.d1 = Dense(32, activation='relu')
    self.d2 = Dense(1, activation='linear')

  def call(self, x):
    x = self.d0(x)
    x = self.d1(x)
    x = self.d2(x)
    return x

model = MyModel()

_ = model(X)

WARNING:tensorflow:Layer my_model is casting an input tensor from dtype float64 to the layer's dtype of float32, which is new behavior in TensorFlow 2. The layer has dtype float32 because it's dtype defaults to floatx. If you intended to run this layer in float32, you can safely ignore this warning. If in doubt, this warning is likely only an issue if you are porting a TensorFlow 1.X model to TensorFlow 2. To change all layers to have dtype float64 by default, call tf.keras.backend.set_floatx('float64'). To change just this layer, pass dtype='float64' to the layer constructor. If you are the author of this layer, you can disable autocasting by passing autocast=False to the base Layer constructor.

It is possible to use Tensorflow for training with 8bit input, which is called quantization. But it is challenging and unnecessary in most cases (i.e., unless you need to deploy your models on edge devices).

tl;dr keep your input in np.float32. See also this post.

Upvotes: 2

Related Questions