Apache Airflow on Ubuntu 20.04 not running after fresh install

Question

I followed the steps from Running Airflow on Ubuntu 20.04 (TypeError: required field "type_ignores" missing from Module```), and received the following:

(airflow-uGvev7QO) root@testing2:/opt/airflow# airflow db init
DB: sqlite:////root/airflow/airflow.db
[2021-03-30 21:17:43,978] {db.py:674} INFO - Creating tables
/root/.local/share/virtualenvs/airflow-uGvev7QO/lib/python3.7/site-packages/sqlalchemy/orm/relationships.py:3454 SAWarning: relationship 'DagRun.serialized_dag' will copy column serialized_dag.dag_id to column dag_run.dag_id, which conflicts with relationship(s): 'TaskInstance.dag_run' (copies task_instance.dag_id to dag_run.dag_id), 'DagRun.task_instances' (copies task_instance.dag_id to dag_run.dag_id). If this is not the intention, consider if these relationships should be linked with back_populates, or if viewonly=True should be applied to one or more if they are read-only. For the less common case that foreign key constraints are partially overlapping, the orm.foreign() annotation can be used to isolate the columns that should be written towards.   The 'overlaps' parameter may be used to remove this warning.
/root/.local/share/virtualenvs/airflow-uGvev7QO/lib/python3.7/site-packages/sqlalchemy/orm/relationships.py:3454 SAWarning: relationship 'SerializedDagModel.dag_runs' will copy column serialized_dag.dag_id to column dag_run.dag_id, which conflicts with relationship(s): 'TaskInstance.dag_run' (copies task_instance.dag_id to dag_run.dag_id), 'DagRun.task_instances' (copies task_instance.dag_id to dag_run.dag_id). If this is not the intention, consider if these relationships should be linked with back_populates, or if viewonly=True should be applied to one or more if they are read-only. For the less common case that foreign key constraints are partially overlapping, the orm.foreign() annotation can be used to isolate the columns that should be written towards.   The 'overlaps' parameter may be used to remove this warning.
INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
Traceback (most recent call last):
  File "/root/.local/share/virtualenvs/airflow-uGvev7QO/bin/airflow", line 8, in <module>
    sys.exit(main())
  File "/root/.local/share/virtualenvs/airflow-uGvev7QO/lib/python3.7/site-packages/airflow/__main__.py", line 40, in main
    args.func(args)
  File "/root/.local/share/virtualenvs/airflow-uGvev7QO/lib/python3.7/site-packages/airflow/cli/cli_parser.py", line 48, in command
    return func(*args, **kwargs)
  File "/root/.local/share/virtualenvs/airflow-uGvev7QO/lib/python3.7/site-packages/airflow/cli/commands/db_command.py", line 31, in initdb
    db.initdb()
  File "/root/.local/share/virtualenvs/airflow-uGvev7QO/lib/python3.7/site-packages/airflow/utils/db.py", line 549, in initdb
    upgradedb()
  File "/root/.local/share/virtualenvs/airflow-uGvev7QO/lib/python3.7/site-packages/airflow/utils/db.py", line 684, in upgradedb
    command.upgrade(config, 'heads')
  File "/root/.local/share/virtualenvs/airflow-uGvev7QO/lib/python3.7/site-packages/alembic/command.py", line 294, in upgrade
    script.run_env()
  File "/root/.local/share/virtualenvs/airflow-uGvev7QO/lib/python3.7/site-packages/alembic/script/base.py", line 490, in run_env
    util.load_python_file(self.dir, "env.py")
  File "/root/.local/share/virtualenvs/airflow-uGvev7QO/lib/python3.7/site-packages/alembic/util/pyfiles.py", line 97, in load_python_file
    module = load_module_py(module_id, path)
  File "/root/.local/share/virtualenvs/airflow-uGvev7QO/lib/python3.7/site-packages/alembic/util/compat.py", line 182, in load_module_py
    spec.loader.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/root/.local/share/virtualenvs/airflow-uGvev7QO/lib/python3.7/site-packages/airflow/migrations/env.py", line 108, in <module>
    run_migrations_online()
  File "/root/.local/share/virtualenvs/airflow-uGvev7QO/lib/python3.7/site-packages/airflow/migrations/env.py", line 102, in run_migrations_online
    context.run_migrations()
  File "<string>", line 8, in run_migrations
  File "/root/.local/share/virtualenvs/airflow-uGvev7QO/lib/python3.7/site-packages/alembic/runtime/environment.py", line 813, in run_migrations
    self.get_context().run_migrations(**kw)
  File "/root/.local/share/virtualenvs/airflow-uGvev7QO/lib/python3.7/site-packages/alembic/runtime/migration.py", line 548, in run_migrations
    for step in self._migrations_fn(heads, self):
  File "/root/.local/share/virtualenvs/airflow-uGvev7QO/lib/python3.7/site-packages/alembic/command.py", line 283, in upgrade
    return script._upgrade_revs(revision, rev)
  File "/root/.local/share/virtualenvs/airflow-uGvev7QO/lib/python3.7/site-packages/alembic/script/base.py", line 365, in _upgrade_revs
    revs = list(revs)
  File "/root/.local/share/virtualenvs/airflow-uGvev7QO/lib/python3.7/site-packages/alembic/script/revision.py", line 916, in _iterate_revisions
    uppers = util.dedupe_tuple(self.get_revisions(upper))
  File "/root/.local/share/virtualenvs/airflow-uGvev7QO/lib/python3.7/site-packages/alembic/script/revision.py", line 457, in get_revisions
    resolved_id, branch_label = self._resolve_revision_number(id_)
  File "/root/.local/share/virtualenvs/airflow-uGvev7QO/lib/python3.7/site-packages/alembic/script/revision.py", line 640, in _resolve_revision_number
    self._revision_map
  File "/root/.local/share/virtualenvs/airflow-uGvev7QO/lib/python3.7/site-packages/alembic/util/langhelpers.py", line 234, in __get__
    obj.__dict__[self.__name__] = result = self.fget(obj)
  File "/root/.local/share/virtualenvs/airflow-uGvev7QO/lib/python3.7/site-packages/alembic/script/revision.py", line 156, in _revision_map
    for revision in self._generator():
  File "/root/.local/share/virtualenvs/airflow-uGvev7QO/lib/python3.7/site-packages/alembic/script/base.py", line 115, in _load_revisions
    script = Script._from_filename(self, vers, file_)
  File "/root/.local/share/virtualenvs/airflow-uGvev7QO/lib/python3.7/site-packages/alembic/script/base.py", line 904, in _from_filename
    module = util.load_python_file(dir_, filename)
  File "/root/.local/share/virtualenvs/airflow-uGvev7QO/lib/python3.7/site-packages/alembic/util/pyfiles.py", line 97, in load_python_file
    module = load_module_py(module_id, path)
  File "/root/.local/share/virtualenvs/airflow-uGvev7QO/lib/python3.7/site-packages/alembic/util/compat.py", line 182, in load_module_py
    spec.loader.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/root/.local/share/virtualenvs/airflow-uGvev7QO/lib/python3.7/site-packages/airflow/migrations/versions/2c6edca13270_resource_based_permissions.py", line 29, in <module>
    from airflow.www.app import create_app
  File "/root/.local/share/virtualenvs/airflow-uGvev7QO/lib/python3.7/site-packages/airflow/www/app.py", line 38, in <module>
    from airflow.www.extensions.init_views import (
  File "/root/.local/share/virtualenvs/airflow-uGvev7QO/lib/python3.7/site-packages/airflow/www/extensions/init_views.py", line 29, in <module>
    from airflow.www.views import lazy_add_provider_discovered_options_to_connection_form
  File "/root/.local/share/virtualenvs/airflow-uGvev7QO/lib/python3.7/site-packages/airflow/www/views.py", line 96, in <module>
    from airflow.www import auth, utils as wwwutils
  File "/root/.local/share/virtualenvs/airflow-uGvev7QO/lib/python3.7/site-packages/airflow/www/utils.py", line 27, in <module>
    from flask_appbuilder.models.sqla.interface import SQLAInterface
  File "/root/.local/share/virtualenvs/airflow-uGvev7QO/lib/python3.7/site-packages/flask_appbuilder/models/sqla/interface.py", line 16, in <module>
    from sqlalchemy_utils.types.uuid import UUIDType
  File "/root/.local/share/virtualenvs/airflow-uGvev7QO/lib/python3.7/site-packages/sqlalchemy_utils/__init__.py", line 1, in <module>
    from .aggregates import aggregated  # noqa
  File "/root/.local/share/virtualenvs/airflow-uGvev7QO/lib/python3.7/site-packages/sqlalchemy_utils/aggregates.py", line 372, in <module>
    from .functions.orm import get_column_key
  File "/root/.local/share/virtualenvs/airflow-uGvev7QO/lib/python3.7/site-packages/sqlalchemy_utils/functions/__init__.py", line 1, in <module>
    from .database import (  # noqa
  File "/root/.local/share/virtualenvs/airflow-uGvev7QO/lib/python3.7/site-packages/sqlalchemy_utils/functions/database.py", line 11, in <module>
    from .orm import quote
  File "/root/.local/share/virtualenvs/airflow-uGvev7QO/lib/python3.7/site-packages/sqlalchemy_utils/functions/orm.py", line 14, in <module>
    from sqlalchemy.orm.query import _ColumnEntity
ImportError: cannot import name '_ColumnEntity' from 'sqlalchemy.orm.query' (/root/.local/share/virtualenvs/airflow-uGvev7QO/lib/python3.7/site-packages/sqlalchemy/orm/query.py)
(airflow-uGvev7QO) root@testing2:/opt/airflow# 

Advice is welcome.

Thank you

Answer 1

Please use below

pip install "apache-airflow[celery]==2.2.2" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.2.2/constraints-3.6.txt"

It will definitely work.

Answer 2

I was able to reproduce the error and the reason seems to be the release of sqlalchemy 1.4 that introduces breaking changes. Airflow by default depends on the latest version of sqlalchemy, but cannot work with version 1.4. A workaround is to downgrade sqlalchem to a version < 1.4.0:

pipenv uninstall sqlalchemy
pipenv install 'sqlalchemy < 1.4.0'

When installing Airflow fresh:

pipenv install 'sqlalchemy < 1.4.0' apache-airflow