CODEWITHSUNDEEP
apache-sparktensorflowhadooppexdistributed-tensorflow
SANKET KINAGE
SANKET KINAGE

Reputation: 1

Getting error in creating pex from TF-YARN library for distributed training

We are trying out TF-YARN library for training DL on tendorflow since our data is in Hadoop. But we are getting error in cluster_pack.upload_env()

Following is the complete error:

ERROR:cluster_pack.packaging:Cannot create pex Traceback (most recent call last): File "/data1/python3.6.10/lib/python3.6/site-packages/cluster_pack/packaging.py", line 144, in pack_in_pex indexes=[CRITEO_PYPI_URL] if _is_criteo() else None) File "/data1/python3.6.10/lib/python3.6/site-packages/pex/resolver.py", line 803, in resolve_multi return list(resolve_request.resolve_distributions(ignore_errors=ignore_errors)) File "/data1/python3.6.10/lib/python3.6/site-packages/pex/resolver.py", line 500, in resolve_distributions raise_type=Unsatisfiable): File "/data1/python3.6.10/lib/python3.6/site-packages/pex/resolver.py", line 370, in _run_parallel max_jobs=self._max_parallel_jobs File "/data1/python3.6.10/lib/python3.6/site-packages/pex/jobs.py", line 219, in execute_parallel raise error pex.resolver.Unsatisfiable: pid: 6749 -> /data1/python3.6.10/bin/python3.6 /tmp/tmpirzknr9r --disable-pip-version-check --isolated --exists-action i -q --no-cache-dir download --dest /tmp/tmp1ezcnpuj/resolved_dists/cp36-cp36m absl-py==0.9.0 alembic==1.4.2 astor==0.8.1 astunparse==1.6.3 async-generator==1.10 attrs==19.3.0 backcall==0.1.0 bleach==3.1.5 cachetools==4.1.1 certifi==2020.4.5.1 certipy==0.1.3 cffi==1.14.0 chardet==3.0.4 cloudpickle==1.3.0 cluster-pack==0.0.9 conda-pack==0.4.0 cryptography==2.9.2 cx-Oracle==7.3.0 cycler==0.10.0 decorator==4.4.2 defusedxml==0.6.0 entrypoints==0.3 gast==0.3.3 google-auth==1.18.0 google-auth-oauthlib==0.4.1 google-pasta==0.2.0 graphframes==0.6 grpcio==1.30.0 h5py==2.10.0 icc-rt==2020.0.133 idna==2.9 importlib-metadata==1.6.0 intel-openmp==2020.0.133 ipykernel==5.3.0 ipython==7.14.0 ipython-genutils==0.2.0 ipywidgets==7.5.1 jedi==0.17.0 Jinja2==2.11.2 joblib==0.16.0 json5==0.9.4 jsonschema==3.2.0 jupyter-client==6.1.3 jupyter-core==4.6.3 jupyter-telemetry==0.1.0 jupyter-tensorboard==0.2.0 jupyterhub==1.1.0 jupyterlab==2.1.2 jupyterlab-server==1.1.4 Keras==2.4.3 Keras-Applications==1.0.8 Keras-Preprocessing==1.1.2 kiwisolver==1.2.0 Mako==1.1.2 Markdown==3.2.2 MarkupSafe==1.1.1 matplotlib==3.2.2 mistune==0.8.4 mkl==2019.0 mkl-random==1.0.1.1 nbconvert==5.6.1 nbformat==5.0.6 networkx==2.4 nose==1.3.7 notebook==6.0.3 numpy==1.18.5 oauthlib==3.1.0 opt-einsum==3.2.1 packaging==20.4 pamela==1.0.0 pandas==1.0.4 pandocfilters==1.4.2 parso==0.7.0 pex==2.1.1 pexpect==4.8.0 pickleshare==0.7.5 prometheus-client==0.7.1 prompt-toolkit==3.0.5 protobuf==3.12.2 ptyprocess==0.6.0 py4j==0.10.7 pyarrow==1.0.0 pyasn1==0.4.8 pyasn1-modules==0.2.8 pycparser==2.20 Pygments==2.6.1 pyOpenSSL==19.1.0 pyparsing==2.4.7 pyrsistent==0.16.0 pyspark==2.4.6 python-dateutil==2.8.1 python-editor==1.0.4 python-json-logger==0.1.11 pytz==2020.1 PyYAML==5.3.1 pyzmq==19.0.1 requests==2.23.0 requests-oauthlib==1.3.0 rsa==4.6 ruamel.yaml==0.16.10 ruamel.yaml.clib==0.2.0 scikit-learn==0.23.1 scipy==1.4.1 seaborn==0.10.1 Send2Trash==1.5.0 six==1.15.0 skein==0.8.0 sklearn==0.0 SQLAlchemy==1.3.17 tbb==2019.0 tbb4py==2019.0 tensorboard==2.2.2 tensorboard-plugin-wit==1.7.0 tensorflow==2.2.0 tensorflow-estimator==2.2.0 tensorflowonspark==2.2.1 termcolor==1.1.0 terminado==0.8.3 testpath==0.4.4 tf-yarn==0.5.1 threadpoolctl==2.1.0 tornado==6.0.4 traitlets==4.3.3 urllib3==1.25.9 wcwidth==0.1.9 webencodings==0.5.1 Werkzeug==1.0.1 widgetsnbextension==3.5.1 wrapt==1.12.1 zipp==3.1.0 raised Executing /data1/python3.6.10/bin/python3.6 /tmp/tmpirzknr9r --disable-pip-version-check --isolated --exists-action i -q --no-cache-dir download --dest /tmp/tmp1ezcnpuj/resolved_dists/cp36-cp36m absl-py==0.9.0 alembic==1.4.2 astor==0.8.1 astunparse==1.6.3 async-generator==1.10 attrs==19.3.0 backcall==0.1.0 bleach==3.1.5 cachetools==4.1.1 certifi==2020.4.5.1 certipy==0.1.3 cffi==1.14.0 chardet==3.0.4 cloudpickle==1.3.0 cluster-pack==0.0.9 conda-pack==0.4.0 cryptography==2.9.2 cx-Oracle==7.3.0 cycler==0.10.0 decorator==4.4.2 defusedxml==0.6.0 entrypoints==0.3 gast==0.3.3 google-auth==1.18.0 google-auth-oauthlib==0.4.1 google-pasta==0.2.0 graphframes==0.6 grpcio==1.30.0 h5py==2.10.0 icc-rt==2020.0.133 idna==2.9 importlib-metadata==1.6.0 intel-openmp==2020.0.133 ipykernel==5.3.0 ipython==7.14.0 ipython-genutils==0.2.0 ipywidgets==7.5.1 jedi==0.17.0 Jinja2==2.11.2 joblib==0.16.0 json5==0.9.4 jsonschema==3.2.0 jupyter-client==6.1.3 jupyter-core==4.6.3 jupyter-telemetry==0.1.0 jupyter-tensorboard==0.2.0 jupyterhub==1.1.0 jupyterlab==2.1.2 jupyterlab-server==1.1.4 Keras==2.4.3 Keras-Applications==1.0.8 Keras-Preprocessing==1.1.2 kiwisolver==1.2.0 Mako==1.1.2 Markdown==3.2.2 MarkupSafe==1.1.1 matplotlib==3.2.2 mistune==0.8.4 mkl==2019.0 mkl-random==1.0.1.1 nbconvert==5.6.1 nbformat==5.0.6 networkx==2.4 nose==1.3.7 notebook==6.0.3 numpy==1.18.5 oauthlib==3.1.0 opt-einsum==3.2.1 packaging==20.4 pamela==1.0.0 pandas==1.0.4 pandocfilters==1.4.2 parso==0.7.0 pex==2.1.1 pexpect==4.8.0 pickleshare==0.7.5 prometheus-client==0.7.1 prompt-toolkit==3.0.5 protobuf==3.12.2 ptyprocess==0.6.0 py4j==0.10.7 pyarrow==1.0.0 pyasn1==0.4.8 pyasn1-modules==0.2.8 pycparser==2.20 Pygments==2.6.1 pyOpenSSL==19.1.0 pyparsing==2.4.7 pyrsistent==0.16.0 pyspark==2.4.6 python-dateutil==2.8.1 python-editor==1.0.4 python-json-logger==0.1.11 pytz==2020.1 PyYAML==5.3.1 pyzmq==19.0.1 requests==2.23.0 requests-oauthlib==1.3.0 rsa==4.6 ruamel.yaml==0.16.10 ruamel.yaml.clib==0.2.0 scikit-learn==0.23.1 scipy==1.4.1 seaborn==0.10.1 Send2Trash==1.5.0 six==1.15.0 skein==0.8.0 sklearn==0.0 SQLAlchemy==1.3.17 tbb==2019.0 tbb4py==2019.0 tensorboard==2.2.2 tensorboard-plugin-wit==1.7.0 tensorflow==2.2.0 tensorflow-estimator==2.2.0 tensorflowonspark==2.2.1 termcolor==1.1.0 terminado==0.8.3 testpath==0.4.4 tf-yarn==0.5.1 threadpoolctl==2.1.0 tornado==6.0.4 traitlets==4.3.3 urllib3==1.25.9 wcwidth==0.1.9 webencodings==0.5.1 Werkzeug==1.0.1 widgetsnbextension==3.5.1 wrapt==1.12.1 zipp==3.1.0 failed with 120

Upvotes: 0

Views: 273

Answers (1)

Fabian
Fabian

Reputation: 2086

What is failing a the pex creation with one of your dependencies. You really have a lot of dependencies. The best would be to isolate your dependencies for each use case you have and create a smaller virtual environment or just try it out with tensorflow only.

What you can try is to execute pex cli command with different requirements and see which requirement exactly is creating issues. What is also important to check is if it works with a more recent pex version (currently tf-yarn uses pex==2.1.1)

pex -r requirements -o myarchive.pex

As an alternative you could also try if it works with conda.

If you create a requirements.txt file with the absolutely needed requirements and enter an issue in https://github.com/criteo/tf-yarn/issues I can have a look.

Upvotes: 0

Related Questions