CODEWITHSUNDEEP
python-2.7google-cloud-platformgoogle-cloud-dataflowapache-beam
Doug
Doug

Reputation: 232

ImportError: No module named idlelib" when running Google Dataflow worker

I have a python 2.7 script I run locally to launch a Apache Beam / Google Dataflow job (SDK 2.12.0). The job takes a csv file from a Google storage bucket, processes it and then creates an entity in Google Datastore for each row. The script ran fine for years ...but now it is failing:

INFO:root:2019-05-15T22:07:11.481Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
    INFO:root:2019-05-15T21:47:13.370Z: JOB_MESSAGE_ERROR: Traceback (most recent call last):
      File "/usr/local/lib/python2.7/dist-packages/dataflow_worker/batchworker.py", line 773, in run
        self._load_main_session(self.local_staging_directory)
      File "/usr/local/lib/python2.7/dist-packages/dataflow_worker/batchworker.py", line 489, in _load_main_session
        pickler.load_session(session_file)
      File "/usr/local/lib/python2.7/dist-packages/apache_beam/internal/pickler.py", line 280, in load_session
        return dill.load_session(file_path)
      File "/usr/local/lib/python2.7/dist-packages/dill/_dill.py", line 410, in load_session
        module = unpickler.load()
      File "/usr/lib/python2.7/pickle.py", line 864, in load
        dispatch[key](self)
      File "/usr/lib/python2.7/pickle.py", line 1139, in load_reduce
        value = func(*args)
      File "/usr/local/lib/python2.7/dist-packages/dill/_dill.py", line 827, in _import_module
        return __import__(import_name)
    ImportError: No module named idlelib

I believe this error is happening at the worker level (not locally). I don't make reference to it in my script. To make sure it wasn't me I have installed updates for all google-cloud packages, apache-beam[gcp] etc locally -just in case. I tried importing idlelib into my script I get the same error. Any suggestions?

Upvotes: 0

Views: 563

Answers (1)

Ruoyun Huang
Ruoyun Huang

Reputation: 173

It has been fine for years and started failing from SDK 2.12.0 release.

What was the last release that this script succeeding on? 2.11?

Upvotes: 0

Related Questions