CODEWITHSUNDEEP
pysparkamazon-emr
Michel Hua
Michel Hua

Reputation: 1777

Python version running on EMR 6.8

What version of Python does EMR 6.8 support?

It looks like previous versions of EMR supported Python 3.7 which will be deprecated in 6 months.

EMR 6.8 runs Spark 3.3.0 which supports Python up to 3.9.

I can't find the answer on the official release guide: https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-680-release.html

Upvotes: 4

Views: 6624

Answers (3)

charles carlson
charles carlson

Reputation: 11

I also hit this issue of needing to run a different Python version on EMR. The following code block should install Python 3.9 and then let you install additional dependencies if you add pip install commands afterwards. Then when trying to trigger Spark jobs in an EMR step, use Spark Submit with the PYSPARK_PYTHON arg pointing to Python 3.9, and you'll be fine. I think my PYSPARK_PYTHON arg is set with --conf spark.pyspark.python=/usr/local/bin/python3.9

# Set Python to autolocate Python 3.9
PYTHON='$(which python3.9)'

# Install necessary Python and Spark Dependencies
# This code works on EC2 instances by default.
sudo yum -y install gcc openssl-devel bzip2-devel libffi-devel git

# Collect Python 3.9.16
wget https://www.python.org/ftp/python/3.9.16/Python-3.9.16.tgz
sudo tar xzf Python-3.9.16.tgz

# Install Python 3.9.16
cd Python-3.9.16
sudo ./configure --enable-optimizations
sudo make altinstall

# Setup Python 3.9 Pip to install future dependencies added
sudo $PYTHON -m pip install --upgrade pip

Upvotes: 1

Wesley Womack
Wesley Womack

Reputation: 342

Since this question was asked, they have updated the page at https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-release-app-versions-6.x.html with the information you are looking for. There is now a line item for included Python version.

emr-6.8.0 includes Python versions 2.7, 3.7.

As of today, the latest EMR version is emr-6.14.0 which still only includes Python 3.7.

Upvotes: 2

kojiro
kojiro

Reputation: 77167

Which Python is available to you is a question about which compute you're using in EMR to run PySpark. If you want another Python, you may need to pick another AMI, a different container image, or talk to AWS directly.

Upvotes: 2

Related Questions