CODEWITHSUNDEEP
azureapache-sparkanacondacondaazure-synapse
Kamil Banaszczyk
Kamil Banaszczyk

Reputation: 1153

Azure synapse apache spark pool can't install geopandas due to recent update

We follow this steps for package management for apache spark pool in our synapse environment: https://learn.microsoft.com/en-us/azure/synapse-analytics/spark/apache-spark-manage-pool-packages#yml-format Due to recent update for conda environment on azure synapse described as a purple note on this page: https://learn.microsoft.com/en-us/azure/synapse-analytics/spark/apache-spark-24-runtime We can't anymore start our apache spark pool by providing this environment.yml file:

name: root
channels:
  - anaconda
dependencies:
  - geopandas=0.9.0

With error at the end of startup of apache spark pool:

Determining conflicts:   0%|          | 0/5 [00:00<?, ?it/s]
Examining conflict for geopandas python:   0%|          | 0/5 [00:00<?, ?it/s]
Examining conflict for __glibc python:  20%|██        | 1/5 [00:00<00:00, 12.91it/s]
                                                                                    
UnsatisfiableError: The following specifications were found to be incompatible with each other:

Output in format: Requested package -> Available versionsThe following specifications were found to be incompatible with your system:

  - feature:/linux-64::__glibc==2.27=0
  - feature:|@/linux-64::__glibc==2.27=0

Your installed version is: 2.27

Is there any way to interfere with conda environment directly on azure apache spark pool? If not is there any dependency that we can install, or force downgrade of updated dependencies using environment.yml file?

Upvotes: 0

Views: 501

Answers (1)

CHEEKATLAPRADEEP
CHEEKATLAPRADEEP

Reputation: 12768

Yes last month, as part of upgrading conda package manager, dependent libgcc libraries were upgraded as well. I can see this upgrade is causing cx package geopandas to conflict.

This is an expected impact of the conda upgrade. Make sure to use this env.yml file instead. I tried this on my test workspace and it works.

 name: root
 channels:
 - anaconda
 dependencies:
 - pip:
   - geopandas==0.9.0

Upvotes: 1

Related Questions