install python-tk using apt-get before running tensorflow on Google Cloud-ML

Question

I’m working on an object detector using Cloud Machine Learning Engine via a Cloud-VM instance. Following the tutorial (https://cloud.google.com/blog/big-data/2017/06/training-an-object-detector-using-cloud-machine-learning-engine).

I get a module import error on Google Cloud Platform when I submit the training job below:

gcloud ml-engine jobs submit training `whoami`_object_detection_`date +%s` \
    --job-dir=${YOUR_GCS_BUCKET}/train \
    --packages dist/object_detection-0.1.tar.gz,slim/dist/slim-0.1.tar.gz \
    --module-name object_detection.train \
    --region us-central1 \
    --config object_detection/samples/cloud/cloud.yml \
    -- \
    --train_dir=${YOUR_GCS_BUCKET}/train \
    --pipeline_config_path=${YOUR_GCS_BUCKET}/data/faster_rcnn_resnet101_coco.config

The error is below:

...object_detection/utils/visualization_utils.py", line 24, in <module>
import matplotlib.pyplot as plt
ImportError: No module named matplotlib.pyplot

I've installed matplotlib using pip install. This code works fine python2.7 -c 'import matplotlib.pyplot as plt'.

The matplotlib error is resolved by adding package names in the REQUIRED_PACKAGES list inside the setup.py program file.

Also, Please see my setup.py file..

"""Setup script for object_detection."""

from setuptools import find_packages
from setuptools import setup
import subprocess

subprocess.check_call(['apt-get', 'update'])
subprocess.check_call(['apt-get', 'install', 'python-tk'])

REQUIRED_PACKAGES = ['Pillow>=1.0', 'matplotlib']

setup(
    name='object_detection',
    version='0.1',
    install_requires=REQUIRED_PACKAGES,
    include_package_data=True,
    packages=[p for p in find_packages() if p.startswith('object_detection')],
    description='Tensorflow Object Detection Library',
)

But, even after resolving this, there are some other errors coming in this scenario as matplotlib has a dependency on python-tk package.

ps-replica-0   Could not find a version that satisfies the requirement python-tk (from object-detection==0.1) (from versions: ) ps-replica-0 
ps-replica-0 No matching distribution found for python-tk (from object-detection==0.1) ps-replica-0 
ps-replica-0 Command '['pip', 'install', '--user', u'object_detection-0.1.tar.gz']' returned non-zero exit status 1 ps-replica-0 
ps-replica-0 Module completed; cleaning up. ps-replica-0 

But python-tk/python3-tk is not available in the pip package. In order to do so, we need to do sudo apt-get install python-tk or sudo apt-get install python3-tk

Google Cloud-ML runs python 2.7. Hence, we need install python-tk before running out tensorflow training program.

Now, can someone please help me in order to command Cloud ML to install python-tk using apt-get before running tensorflow.

Update_01:*

I'm getting another set of errors. It seems it's caused by the failure of python setup.py egg_info. Also, there's this..

Command '['apt-get', 'install', 'python-tk']' returned non-zero exit status 1

The error log is shown below. Thanks in advance for your help.

ps-replica-2
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-BhSDtP-build/
ps-replica-2
Command '['pip', 'install', '--user', '--upgrade', '--force-reinstall', '--no-deps', u'object_detection-0.1.tar.gz']' returned non-zero exit status 1
The replica ps 0 exited with a non-zero status of 1. Termination reason: Error. 
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/tmp/pip-C3hdCp-build/setup.py", line 8, in <module>
subprocess.check_call(['apt-get', 'install', 'python-tk'])
  File "/usr/lib/python2.7/subprocess.py", line 540, in check_call
raise CalledProcessError(retcode, cmd)
CalledProcessError: Command '['apt-get', 'install', 'python-tk']' returned non-zero exit status 1

The replica ps 1 exited with a non-zero status of 1. Termination reason: Error. 
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/tmp/pip-iR0TqP-build/setup.py", line 8, in <module>
subprocess.check_call(['apt-get', 'install', 'python-tk'])
  File "/usr/lib/python2.7/subprocess.py", line 540, in check_call
raise CalledProcessError(retcode, cmd)
CalledProcessError: Command '['apt-get', 'install', 'python-tk']' returned non-zero exit status 1

The replica ps 2 exited with a non-zero status of 1. Termination reason: Error. 
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/tmp/pip-BhSDtP-build/setup.py", line 8, in <module>
subprocess.check_call(['apt-get', 'install', 'python-tk'])
  File "/usr/lib/python2.7/subprocess.py", line 540, in check_call
raise CalledProcessError(retcode, cmd)
CalledProcessError: Command '['apt-get', 'install', 'python-tk']' returned non-zero exit status 1

To find out more about why your job exited please check the logs: https://console.cloud.google.com/logs/viewer?project=640992742297&resource=ml_job%2Fjob_id%2Froot_object_detection_1510462119&advancedFilter=resource.type%3D%22ml_job%22%0Aresource.labels.job_id%3D%22root_object_detection_1510462119%22" 

The job submission code:

gcloud ml-engine jobs submit training `whoami`_object_detection_`date +%s` \
    --job-dir=${YOUR_GCS_BUCKET}/train \
    --packages dist/object_detection-0.1.tar.gz,slim/dist/slim-0.1.tar.gz \
    --module-name object_detection.train \
    --config object_detection/samples/cloud/cloud.yml \
    -- \
    --train_dir=${YOUR_GCS_BUCKET}/train \
    --pipeline_config_path=${YOUR_GCS_BUCKET}/data/faster_rcnn_resnet101_coco.config

Thanks in advance..

Update_02: Solution Thanks to @Dennis Liu for the solution. No need to install python-tk package. Apart from that there will be one more error, which can be solved by changing tf.train.get_or_create_global_step() to tf.contrib.framework.get_or_create_global_step() at line 103 in object_detection/builders/optimizer_builder.py. Solution Link

Answer 1

Use matplotlib.use('agg') right after importing matplotlib

I changed the backend of matplotlib from python-tk to agg, this did the trick. Here is the answer that I found this at:

https://stackoverflow.com/a/47077614

Answer 2

Add the following lines to your setup.py:

import subprocess
subprocess.check_call(['apt-get', 'install', 'python-tk'])

and remove python-tk from REQUIRED_PACKAGES.