CODEWITHSUNDEEP
apache-sparkpycharmdatabricksdelta-lake
Siete
Siete

Reputation: 368

How to reliably obtain partition columns of delta table

I need to obtain the partitioning columns of a delta table, but the returned result of a DESCRIBE delta.`my_table` returns different results on databricks and locally on PyCharm.

Minimal example:

from pyspark.sql.types import StructType, StructField, StringType, IntegerType

delta_table_path = "c:/temp_delta_table"
partition_column = ["rs_nr"]

schema = StructType([
        StructField("rs_nr", StringType(), False),
        StructField("event_category", StringType(), True),
        StructField("event_counter", IntegerType(), True)])

data = [{'rs_nr': '001', 'event_category': 'event_01', 'event_counter': 1},
 {'rs_nr': '002', 'event_category': 'event_02', 'event_counter': 2},
 {'rs_nr': '003', 'event_category': 'event_03', 'event_counter': 3},
 {'rs_nr': '004', 'event_category': 'event_04', 'event_counter': 4}]

sdf = spark.createDataFrame(data=data, schema=schema)

sdf.write.format("delta").mode("overwrite").partitionBy(partition_column).save(delta_table_path)

df_descr = spark.sql(f"DESCRIBE delta.`{delta_table_path}`")

df_descr.toPandas()

Shows, on databricks, the partition column(s):

    col_name                data_type     comment
0   rs_nr                      string        None
1   event_category             string        None
2   event_counter                 int        None
3   # Partition Information
4   # col_name              data_type     comment
5   rs_nr                      string        None

But when running this locally in PyCharm, I get the following different output:

         col_name data_type comment
0           rs_nr    string        
1  event_category    string        
2   event_counter       int        
3                                  
4  # Partitioning                  
5          Part 0     rs_nr

Parsing both types of return value seems ugly to me, so is there a reason that this is returned like this?

Setup:

In Pycharm:

In DataBricks:

In PyCharm, I create the connection using:

def get_spark():
    spark = SparkSession.builder.appName('schema_checker')\
        .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension")\
        .config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog")\
        .config("spark.jars.packages", "io.delta:delta-core_2.12:2.0.0")\
        .config("spark.sql.catalogImplementation", "in-memory")\
        .getOrCreate()

    return spark

Upvotes: 3

Views: 3023

Answers (1)

Alex Ott
Alex Ott

Reputation: 87369

If you're using Python, then instead of executing SQL command that is harder to parse, it's better to use Python API. The DeltaTable instance has a detail function that returns a dataframe with details about the table (doc), and this dataframe has the partitionColumns column that is array of strings with partition columns names. So you can just do:

from delta.tables import *

detailDF = DeltaTable.forPath(spark, delta_table_path).detail()
partitions = detailDF.select("partitionColumns").collect()[0][0]

Upvotes: 5

Related Questions