CODEWITHSUNDEEP
tensorflowdistributed-computingkaggletpu
Saish Reddy
Saish Reddy

Reputation: 61

InvalidArgumentError while initializing ttpu

I actually ran into this issue while using tf-2.3.0 stable build, while initializing tpu in kaggle from the following code:

try:
tpu_name = os.getenv('TPU_NAME')
tpu = tf.distribute.cluster_resolver.TPUClusterResolver(tpu_name)
print("running on tpu: ", tpu.master())
except ValueError:
tpu = None
if tpu:
    tf.config.experimental_connect_to_cluster(tpu)
    tf.tpu.experimental.initialize_tpu_system(tpu)
    strategy = tf.distribute.experimental.TPUStrategy(tpu)
else:
    strategy = tf.distribute.get_strategy()

with output:

running on tpu:  grpc://10.0.0.2:8470

with error:

---------------------------------------------------------------------------
InvalidArgumentError                      Traceback (most recent call last)
<ipython-input-9-5394dc1b79b0> in <module>
      7 if tpu:
      8     tf.config.experimental_connect_to_cluster(tpu)
----> 9     tf.tpu.experimental.initialize_tpu_system(tpu)
     10     strategy = tf.distribute.experimental.TPUStrategy(tpu)
     11 else:

/opt/conda/lib/python3.7/site-packages/tensorflow/python/tpu/tpu_strategy_util.py in initialize_tpu_system(cluster_resolver)
    109     context.context()._clear_caches()  # pylint: disable=protected-access
    110 
--> 111     serialized_topology = output.numpy()
    112 
    113     # TODO(b/134094971): Remove this when lazy tensor copy in multi-device

/opt/conda/lib/python3.7/site-packages/tensorflow/python/framework/ops.py in numpy(self)
   1061     """
   1062     # TODO(slebedev): Consider avoiding a copy for non-CPU or remote tensors.
-> 1063     maybe_arr = self._numpy()  # pylint: disable=protected-access
   1064     return maybe_arr.copy() if isinstance(maybe_arr, np.ndarray) else maybe_arr
   1065 

/opt/conda/lib/python3.7/site-packages/tensorflow/python/framework/ops.py in _numpy(self)
   1029       return self._numpy_internal()
   1030     except core._NotOkStatusException as e:  # pylint: disable=protected-access
-> 1031       six.raise_from(core._status_to_exception(e.code, e.message), None)  # pylint: disable=protected-access
   1032 
   1033   @property

/opt/conda/lib/python3.7/site-packages/six.py in raise_from(value, from_value)

InvalidArgumentError: NodeDef expected inputs 'string' do not match 0 inputs specified; Op<name=_Send; signature=tensor:T -> ; attr=T:type; attr=tensor_name:string; attr=send_device:string; attr=send_device_incarnation:int; attr=recv_device:string; attr=client_terminated:bool,default=false; is_stateful=true>; NodeDef: {{node _Send}}
----------

if anyone had this issue or know any way to resolve it please put it up thank you!

Upvotes: 1

Views: 504

Answers (1)

Gagik
Gagik

Reputation: 486

If you have manually updated Tensorflow in kaggle (or colab) notebook though pip install your TPU machine might have a different version of Tensorflow. Try using the following code to make TPU to use current version of Tensorflow:

!pip install cloud-tpu-client

import tensorflow as tf
from cloud_tpu_client import Client
print(tf.__version__)

Client().configure_tpu_version(tf.__version__, restart_type='ifNeeded')

Upvotes: 3

Related Questions