LatentDirichletAllocation was not installed in decomposition module of SciKit-Learn

Question

I got some strange problem at SciKit-Learn package. There is "decomposition" module inside SciKit-Learn package, which should contain LatentDirichletAllocation([…]) function. See documentation here: "https://scikit-learn.org/stable/modules/classes.html#module-sklearn.decomposition"

When I tried to import "decomposition" module: from sklearn import as decomposition it gives error:

Traceback (most recent call last):
  File "tf_1_day_scikit_dnn.py", line 12, in <module>
    from sklearn import decomposition
  File "/home/developer1/.local/lib/python3.6/site-packages/sklearn/decomposition/__init__.py", line 19, in <module>
    from ._online_lda import LatentDirichletAllocation
ImportError: cannot import name 'LatentDirichletAllocation'

Command: ls -al ~/.local/lib/python3.6/site-packages/sklearn/decomposition shows:

drwxr-xr-x  4 developer1 developer1   4096 Dec  9 00:45 .
drwxr-xr-x 33 developer1 developer1   4096 Dec  9 00:45 ..
-rw-r--r--  1 developer1 developer1   5490 Dec  9 00:44 _base.py
-rw-r--r--  1 developer1 developer1    480 Dec  9 00:44 base.py
-rwxr-xr-x  1 developer1 developer1 179440 Dec  9 00:44 _cdnmf_fast.cpython-36m-x86_64-linux-gnu.so
-rwxr-xr-x  1 developer1 developer1 175344 Dec  3 00:09 cdnmf_fast.cpython-36m-x86_64-linux-gnu.so
-rw-r--r--  1 developer1 developer1    498 Dec  9 00:44 cdnmf_fast.py
-rw-r--r--  1 developer1 developer1  54528 Dec  9 00:44 _dict_learning.py
-rw-r--r--  1 developer1 developer1    507 Dec  9 00:44 dict_learning.py
-rw-r--r--  1 developer1 developer1  12572 Dec  9 00:44 _factor_analysis.py
-rw-r--r--  1 developer1 developer1    513 Dec  9 00:44 factor_analysis.py
-rw-r--r--  1 developer1 developer1  20866 Dec  9 00:44 _fastica.py
-rw-r--r--  1 developer1 developer1    490 Dec  9 00:44 fastica_.py
-rw-r--r--  1 developer1 developer1  14076 Dec  9 00:44 _incremental_pca.py
-rw-r--r--  1 developer1 developer1    513 Dec  9 00:44 incremental_pca.py
-rw-r--r--  1 developer1 developer1   1401 Dec  9 00:44 __init__.py
-rw-r--r--  1 developer1 developer1  13597 Dec  9 00:44 _kernel_pca.py
-rw-r--r--  1 developer1 developer1    498 Dec  9 00:44 kernel_pca.py
-rw-r--r--  1 developer1 developer1  47255 Dec  9 00:44 _nmf.py
-rw-r--r--  1 developer1 developer1    477 Dec  9 00:44 nmf.py
-rwxr-xr-x  1 developer1 developer1  62056 Dec  3 00:09 _online_lda.cpython-36m-x86_64-linux-gnu.so
-rwxr-xr-x  1 developer1 developer1  62064 Dec  9 00:44 _online_lda_fast.cpython-36m-x86_64-linux-gnu.so
-rw-r--r--  1 developer1 developer1    513 Dec  9 00:44 online_lda_fast.py
-rw-r--r--  1 developer1 developer1  30471 Dec  9 00:44 _online_lda.py
-rw-r--r--  1 developer1 developer1    498 Dec  9 00:44 online_lda.py
-rw-r--r--  1 developer1 developer1  22807 Dec  9 00:44 _pca.py
-rw-r--r--  1 developer1 developer1    477 Dec  9 00:44 pca.py
drwxr-xr-x  2 developer1 developer1   4096 Dec  9 00:45 __pycache__
-rw-r--r--  1 developer1 developer1    855 Dec  9 00:44 setup.py
-rw-r--r--  1 developer1 developer1  13654 Dec  9 00:44 _sparse_pca.py
-rw-r--r--  1 developer1 developer1    498 Dec  9 00:44 sparse_pca.py
drwxr-xr-x  3 developer1 developer1   4096 Dec  9 00:45 tests
-rw-r--r--  1 developer1 developer1   8346 Dec  9 00:44 _truncated_svd.py
-rw-r--r--  1 developer1 developer1    507 Dec  9 00:44 truncated_svd.py

Most of the functions are here but there are no traces of the "LatentDirichletAllocation" function. Yet I did see LatentDirichletAllocation class defined in the _online_lda.py file.

Command python3 -c "import sklearn; sklearn.show_versions()" prints all versions required:

System:
    python: 3.6.8 (default, Oct  7 2019, 12:59:55)  [GCC 8.3.0]
executable: /usr/bin/python3
   machine: Linux-4.15.0-65-generic-x86_64-with-Ubuntu-18.04-bionic

Python dependencies:
       pip: 9.0.1
setuptools: 42.0.2
   sklearn: 0.22
     numpy: 1.17.4
     scipy: 1.3.3
    Cython: None
    pandas: 0.25.3
matplotlib: 3.1.2
    joblib: 0.14.0

Built with OpenMP: True

So I assume all requirements were met here. (BTW SciKit-Learn worked fine till I tried to import "decomposition" module).

I installed the SciKit-Learn package as it was described here: "https://scikit-learn.org/stable/install.html" using command pip3 install -U scikit-learn I did reinstall that several times but got same result.

What do I do wrong? Do I need to add missing "LatentDirichletAllocation" function into the "decomposition" module? If so - how? Should I install the whole package somehow different way?

Thanks.

Answer 1

I had the same problem today. I solved it by going back to the previous version 0.21 of scikit-learn:

pip3 install scikit-learn==0.21

EDIT: I think the answer from glemaitre (https://stackoverflow.com/a/59328446/10429267) shows a better solution.

Answer 2

The validated "answered" is not the right fix since it only downgrades scikit-learn. You will not be able to benefit from new features and bug fixes.

It will be helpful instead to report and give feedback in the issue tracker: https://github.com/scikit-learn/scikit-learn/issues/15884

It will allow us to find the root of the problem and propose the proper fix which seems to affect several persons.

EDIT: After some investigation, you need to remove the following files:

• ~/.local/lib/python3.6/site-packages/sklearn/decomposition/_online_lda.cpython-36m-x86_64-linux-gnu.so
• ~/.local/lib/python3.6/site-packages/sklearn/feature_extraction/_hashing.cpython-36m-x86_64-linux-gnu.so
• ~/.local/lib/python3.6/site-packages/sklearn/datasets/_svmlight_format.cpython-36m-x86_64-linux-gnu.so

Apparently, the so files do not get removed when updating to scikit-learn. This might be due to an old pip version.

Answer 3

Try:

---

For Python 3

pip3 uninstall scikit-learn
pip3 install -U scikit-learn==0.21.3

---

For Python 2

pip uninstall scikit-learn
pip install -U scikit-learn==0.20.4