CODEWITHSUNDEEP
pythonpython-3.xpysparkcassandraanaconda
Coder94
Coder94

Reputation: 1045

Issue in Python version after installing PySpark

I have installed PySpark 3.1.2 along with OpenJDK-1.8 to connect with a docker instance of Cassandra 4.0.1. I followed the instructions as in https://towardsdatascience.com/installing-pyspark-with-java-8-on-ubuntu-18-04-6a9dea915b5b and successfully installed the required versions.

I'm using anaconda environment, after installation I noticed that my Python version got automatically downgraded to 3.5 which is not supported by Pyspark(even in all environments where I had different python versions earlier, it's now 3.5). I read that Pyspark needs python3.6+. I tried everything possible to upgrade the python version to 3.6+ but it's not happening. When I try conda upgrade python some upgrades and removals happen but python is still 3.5.

conda update python gives:

Collecting package metadata (current_repodata.json): done
Solving environment: \ 

Updating python is constricted by 

anaconda -> requires python==3.8.3=hcff3b4d_2

If you are sure you want an update of your package either try `conda update --all` or install a specific version of the package you want using `conda install <pkg>=<version>`

done

# All requested packages already installed.

While using conda install python=3.7 after installing/upgrading so many packages the following error message is obtained:

Downloading and Extracting Packages
traitlets-4.3.3      | 140 KB    | ##################################### | 100% 
conda-token-0.3.0    | 10 KB     | ##################################### | 100% 
testpath-0.4.4       | 82 KB     | ##################################### | 100% 
click-7.1.2          | 71 KB     | ##################################### | 100% 
singledispatch-3.4.0 | 16 KB     | ##################################### | 100% 
conda-package-handli | 881 KB    | ##################################### | 100% 
astropy-4.0.1.post1  | 6.1 MB    | ##################################### | 100% 
greenlet-0.4.16      | 24 KB     | ##################################### | 100% 
navigator-updater-0. | 691 KB    | ##################################### | 100% 
defusedxml-0.6.0     | 23 KB     | ##################################### | 100% 
importlib-metadata-1 | 51 KB     | ##################################### | 100% 
ipython-7.16.1       | 987 KB    | ##################################### | 100% 
numba-0.50.1         | 3.1 MB    | ##################################### | 100% 
mkl_fft-1.1.0        | 143 KB    | ##################################### | 100% 
pycurl-7.43.0.5      | 68 KB     | ##################################### | 100% 
scikit-learn-0.23.1  | 5.0 MB    | ##################################### | 100% 
sympy-1.6.1          | 8.6 MB    | ##################################### | 100% 
matplotlib-base-3.2. | 5.4 MB    | ##################################### | 100% 
pandocfilters-1.4.2  | 13 KB     | ##################################### | 100% 
pkginfo-1.5.0.1      | 44 KB     | ##################################### | 100% 
ipykernel-5.3.2      | 179 KB    | ##################################### | 100% 
anaconda-navigator-2 | 5.1 MB    | ##################################### | 100% 
pylint-2.5.3         | 441 KB    | ##################################### | 100% 
kiwisolver-1.2.0     | 84 KB     | ##################################### | 100% 
setuptools-49.2.0    | 743 KB    | ##################################### | 100% 
bokeh-2.1.1          | 5.4 MB    | ##################################### | 100% 
cryptography-2.9.2   | 552 KB    | ##################################### | 100% 
zope.interface-4.7.1 | 204 KB    | ##################################### | 100% 
ujson-1.35           | 25 KB     | ##################################### | 100% 
_ipyw_jlab_nb_ext_co | 4 KB      | ##################################### | 100% 
asn1crypto-1.3.0     | 164 KB    | ##################################### | 100% 
pyzmq-19.0.1         | 459 KB    | ##################################### | 100% 
terminado-0.8.3      | 26 KB     | ##################################### | 100% 
conda-content-trust- | 56 KB     | ##################################### | 100% 
jsonschema-3.2.0     | 91 KB     | ##################################### | 100% 
bitarray-1.4.0       | 88 KB     | ##################################### | 100% 
beautifulsoup4-4.9.1 | 165 KB    | ##################################### | 100% 
certifi-2020.6.20    | 156 KB    | ##################################### | 100% 
nbconvert-5.6.1      | 451 KB    | ##################################### | 100% 
zope.event-4.4       | 10 KB     | ##################################### | 100% 
cffi-1.14.0          | 224 KB    | ##################################### | 100% 
path-13.1.0          | 35 KB     | ##################################### | 100% 
scikit-image-0.16.2  | 23.1 MB   | ##################################### | 100% 
python-language-serv | 81 KB     | ##################################### | 100% 
werkzeug-1.0.1       | 240 KB    | ##################################### | 100% 
conda-repo-cli-1.0.4 | 47 KB     | ##################################### | 100% 
anaconda-2020.07     | 17 KB     | ##################################### | 100% 
numpy-1.18.5         | 5 KB      | ##################################### | 100% 
flask-1.1.2          | 78 KB     | ##################################### | 100% 
psutil-5.7.0         | 319 KB    | ##################################### | 100% 
jinja2-2.11.2        | 103 KB    | ##################################### | 100% 
gevent-20.6.2        | 1.6 MB    | ##################################### | 100% 
six-1.15.0           | 13 KB     | ##################################### | 100% 
distributed-2.20.0   | 992 KB    | ##################################### | 100% 
wheel-0.34.2         | 51 KB     | ##################################### | 100% 
numpy-base-1.18.5    | 4.1 MB    | ##################################### | 100% 
lxml-4.5.2           | 1.2 MB    | ##################################### | 100% 
pip-20.1.1           | 1.7 MB    | ##################################### | 100% 
isort-4.3.21         | 69 KB     | ##################################### | 100% 
spyder-kernels-1.9.2 | 97 KB     | ##################################### | 100% 
pandas-1.0.5         | 7.8 MB    | ##################################### | 100% 
spyder-4.1.4         | 5.6 MB    | ##################################### | 100% 
python-3.7.7         | 45.1 MB   | ##################################### | 100% 
pytest-5.4.3         | 388 KB    | ##################################### | 100% 
backports-1.0        | 160 KB    | ##################################### | 100% 
conda-build-3.21.5   | 549 KB    | ##################################### | 100% 
sqlalchemy-1.3.18    | 1.5 MB    | ##################################### | 100% 
cytoolz-0.10.1       | 376 KB    | ##################################### | 100% 
pillow-7.2.0         | 617 KB    | ##################################### | 100% 
notebook-6.0.3       | 4.0 MB    | ##################################### | 100% 
keyring-21.2.1       | 58 KB     | ##################################### | 100% 
conda-4.10.3         | 2.9 MB    | ##################################### | 100% 
statsmodels-0.11.1   | 7.8 MB    | ##################################### | 100% 
decorator-4.4.2      | 14 KB     | ##################################### | 100% 
scipy-1.5.0          | 14.4 MB   | ##################################### | 100% 
llvmlite-0.33.0      | 17.3 MB   | ##################################### | 100% 
pywavelets-1.1.1     | 3.5 MB    | ##################################### | 100% 
regex-2020.6.8       | 324 KB    | ##################################### | 100% 
pyrsistent-0.16.0    | 94 KB     | ##################################### | 100% 
Preparing transaction: failed

# >>>>>>>>>>>>>>>>>>>>>> ERROR REPORT <<<<<<<<<<<<<<<<<<<<<<

    Traceback (most recent call last):
      File "/home/xx/anaconda3/lib/python3.8/site-packages/conda/exceptions.py", line 1079, in __call__
        return func(*args, **kwargs)
      File "/home/xx/anaconda3/lib/python3.8/site-packages/conda/cli/main.py", line 84, in _main
        exit_code = do_call(args, p)
      File "/home/xx/anaconda3/lib/python3.8/site-packages/conda/cli/conda_argparse.py", line 83, in do_call
        return getattr(module, func_name)(args, parser)
      File "/home/xx/anaconda3/lib/python3.8/site-packages/conda/cli/main_install.py", line 20, in execute
        install(args, parser, 'install')
      File "/home/xx/anaconda3/lib/python3.8/site-packages/conda/cli/install.py", line 317, in install
        handle_txn(unlink_link_transaction, prefix, args, newenv)
      File "/home/xx/anaconda3/lib/python3.8/site-packages/conda/cli/install.py", line 346, in handle_txn
        unlink_link_transaction.execute()
      File "/home/xx/anaconda3/lib/python3.8/site-packages/conda/core/link.py", line 245, in execute
        self.verify()
      File "/home/xx/anaconda3/lib/python3.8/site-packages/conda/common/io.py", line 88, in decorated
        return f(*args, **kwds)
      File "/home/xx/anaconda3/lib/python3.8/site-packages/conda/core/link.py", line 222, in verify
        self.prepare()
      File "/home/xx/anaconda3/lib/python3.8/site-packages/conda/core/link.py", line 211, in prepare
        grps = self._prepare(self.transaction_context, stp.target_prefix,
      File "/home/xx/anaconda3/lib/python3.8/site-packages/conda/core/link.py", line 288, in _prepare
        packages_info_to_link = tuple(read_package_info(prec, pcrec)
      File "/home/xx/anaconda3/lib/python3.8/site-packages/conda/core/link.py", line 288, in <genexpr>
        packages_info_to_link = tuple(read_package_info(prec, pcrec)
      File "/home/xx/anaconda3/lib/python3.8/site-packages/conda/gateways/disk/read.py", line 89, in read_package_info
        package_metadata = read_package_metadata(epd)
      File "/home/xx/anaconda3/lib/python3.8/site-packages/conda/gateways/disk/read.py", line 144, in read_package_metadata
        data = json.loads(f.read())
      File "/home/xx/anaconda3/lib/python3.8/json/__init__.py", line 357, in loads
        return _default_decoder.decode(s)
      File "/home/xx/anaconda3/lib/python3.8/json/decoder.py", line 337, in decode
        obj, end = self.raw_decode(s, idx=_w(s, 0).end())
      File "/home/xx/anaconda3/lib/python3.8/json/decoder.py", line 355, in raw_decode
        raise JSONDecodeError("Expecting value", s, err.value) from None
    json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

`$ /home/xx/anaconda3/bin/conda install python=3.7`

  environment variables:
                 CIO_TEST=<not set>
          COMPIZ_BIN_PATH=/usr/bin/
        CONDA_DEFAULT_ENV=base
                CONDA_EXE=/home/xx/anaconda3/bin/conda
             CONDA_PREFIX=/home/xx/anaconda3
    CONDA_PROMPT_MODIFIER=(base)
         CONDA_PYTHON_EXE=/home/xx/anaconda3/bin/python
               CONDA_ROOT=/home/xx/anaconda3
              CONDA_SHLVL=1
           CURL_CA_BUNDLE=<not set>
            DEFAULTS_PATH=/usr/share/gconf/ubuntu.default.path
           MANDATORY_PATH=/usr/share/gconf/ubuntu.mandatory.path
                     PATH=/home/xx/anaconda3/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/u
                          sr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/xx/s
                          park-3.1.2-bin-
                          hadoop3.2/bin:/home/xx/anaconda3/bin:/usr/lib/jvm/java-8-openjdk-
                          amd64/jre/bin
               PYTHONPATH=/home/xx/spark-3.1.2-bin-hadoop3.2/python:
       REQUESTS_CA_BUNDLE=<not set>
            SSL_CERT_FILE=<not set>
            XDG_SEAT_PATH=/org/freedesktop/DisplayManager/Seat0
         XDG_SESSION_PATH=/org/freedesktop/DisplayManager/Session0

     active environment : base
    active env location : /home/xx/anaconda3
            shell level : 1
       user config file : /home/xx/.condarc
 populated config files : 
          conda version : 4.10.3
    conda-build version : 3.18.11
         python version : 3.8.3.final.0
       virtual packages : __linux=4.15.0=0
                          __glibc=2.23=0
                          __unix=0=0
                          __archspec=1=x86_64
       base environment : /home/xx/anaconda3  (writable)
      conda av data dir : /home/xx/anaconda3/etc/conda
  conda av metadata url : None
           channel URLs : https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /home/xx/anaconda3/pkgs
                          /home/xx/.conda/pkgs
       envs directories : /home/xx/anaconda3/envs
                          /home/xx/.conda/envs
               platform : linux-64
             user-agent : conda/4.10.3 requests/2.24.0 CPython/3.8.3 Linux/4.15.0-142-generic ubuntu/16.04.7 glibc/2.23
                UID:GID : 1000:1000
             netrc file : None
           offline mode : False


An unexpected error has occurred. Conda has prepared the above report.

Can someone help me figure out the issue? How can I resolve this and upgrade the Python package?

Upvotes: 0

Views: 1484

Answers (1)

Coder94
Coder94

Reputation: 1045

I resolved the issue by manually installing pyspark and making a minor change in the environment variables.

After downloading the required version of spark, you need to configure environment variables. There are a few Spark home paths you need to add to the user profile as follows,

echo "export SPARK_HOME=/opt/spark" >> ~/.profile
echo "export PATH=$PATH:$SPARK_HOME/bin:$SPARK_HOME/sbin" >> ~/.profile
echo "export PYSPARK_PYTHON=/usr/bin/python3" >> ~/.profile

Here, for me just after adding the spark home path and other parameters my python version downgrades to 3.5 in anaconda.

So I changed the Python path in the user profile to: PYSPARK_PYTHON=/usr/bin/python3.7

which resolved the issue since pyspark is compatible with python3.6+

Upvotes: 1

Related Questions