CODEWITHSUNDEEP
pythonairflow
MT467
MT467

Reputation: 698

airflow error:AttributeError: module 'airflow.utils.log' has no attribute 'file_processor_handler'

My local airflow instant was up and running, but now when I run airflow webserver or any other airflow command I got the below error (with some traceback):

  Unable to load the config, contains a configuration error.
  Traceback (most recent call last):
  File "/anaconda3/lib/python3.6/logging/config.py", line 382, in resolve
  found = getattr(found, frag)
  AttributeError: module 'airflow.utils.log' has no attribute 'file_processor_handler'

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
  File "/anaconda3/lib/python3.6/logging/config.py", line 384, in resolve
self.importer(used)
   File "/anaconda3/lib/python3.6/site-packages/airflow/utils/log/file_processor_handler.py", line 25, in <module>
from airflow.utils.helpers import parse_template_string
   File "/anaconda3/lib/python3.6/site-packages/airflow/utils/helpers.py", line 25, in <module>
import psutil
 File "/anaconda3/lib/python3.6/site-packages/psutil/__init__.py", line 134, in <module>
 from . import _psosx as _psplatform
  File "/anaconda3/lib/python3.6/site-packages/psutil/_psosx.py", line 14, in <module>
from . import _psutil_osx as cext
  ImportError: dlopen(/anaconda3/lib/python3.6/site-packages/psutil/_psutil_osx.cpython-36m-darwin.so, 2): Symbol not found: _kIOMasterPortDefault
  Referenced from: /anaconda3/lib/python3.6/site-packages/psutil/_psutil_osx.cpython-36m-darwin.so
  Expected in: flat namespace
 in /anaconda3/lib/python3.6/site-packages/psutil/_psutil_osx.cpython-36m-darwin.so

  The above exception was the direct cause of the following exception:

   Traceback (most recent call last):
   File "/anaconda3/lib/python3.6/logging/config.py", line 558, in configure
   handler = self.configure_handler(handlers[name])
 ValueError: Cannot resolve 'airflow.utils.log.file_processor_handler.FileProcessorHandler': dlopen(/anaconda3/lib/python3.6/site-packages/psutil/_psutil_osx.cpython-36m-darwin.so, 2): Symbol not found: _kIOMasterPortDefault
  Referenced from: /anaconda3/lib/python3.6/site-packages/psutil/_psutil_osx.cpython-36m-darwin.so
  Expected in: flat namespace
 in /anaconda3/lib/python3.6/site-packages/psutil/_psutil_osx.cpython-36m-darwin.so

   During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
 File "/anaconda3/bin/airflow", line 21, in <module>
from airflow import configuration
  File "/anaconda3/lib/python3.6/site-packages/airflow/__init__.py", line 36, in <module>
from airflow import settings
 File "/anaconda3/lib/python3.6/site-packages/airflow/settings.py", line 259, in <module>
configure_logging()
 File "/anaconda3/lib/python3.6/site-packages/airflow/logging_config.py", line 72, in configure_logging
raise e
 File "/anaconda3/lib/python3.6/site-packages/airflow/logging_config.py", line 67, in configure_logging
dictConfig(logging_config)
 File "/anaconda3/lib/python3.6/logging/config.py", line 795, in dictConfig
dictConfigClass(config).configure()

I tried to upgrade the airflow but that didn't help

Upvotes: 3

Views: 9282

Answers (3)

Ash Berlin-Taylor
Ash Berlin-Taylor

Reputation: 4058

I think the problem is that the install of pustil has a mix of old and new files.

The "Cannot resolve 'airflow.utils.log.file_processor_handler.FileProcessorHandler" part of the error message is just Airflow saying while trying to FileProcessorHandler we got this other error.

You should see the exact same error form running python -c "import psutil" which would entirely remove Airflow from the equation.

I think the fix is to totally uninstall psutil (if conda wasn't involved I'd say pip uninstall psutil) and then reinstall it. I think in this case I'd try a pip-uninstall and to uninstall it from however you install conda packages.

Upvotes: 0

ransy
ransy

Reputation: 11

See my comment in this PR https://github.com/apache/airflow/pull/3585

essentially pinning psutil to 4.4.2 will fix, as the latest version of psutil at 5.6.1 in this context fails for me on osx

Upvotes: 0

Yong Wang
Yong Wang

Reputation: 1313

I just encountered the same issue as your mentioned.

I will facing the issue even I just import pandas lib.

The reason is I have some dag files name email.py. which is same as python internal lib name. the conflict lead to python stop work. After I rename the email.py to email_test.py. The problem solved.

You can refer to the similar question. It is same issue and same solution. ImportError : cannot import DAG airflow

If you like my answer , pls vote it.

Upvotes: 2

Related Questions