StackOverflow Questions for Tag: pyspark

Jim Macaulay

Reputation: 5165

Extract specific dictionary value from dataframe in PySpark

dictionarypysparkapache-spark-sqlextract

Score: 0

Answers: 1

Jim Macaulay

Reputation: 5165

Pyspark - Convert array type to string type for ARRAY<MAP<STRING, STRING>> type

pyspark

Score: 0

Answers: 1

Mads

Reputation: 27

Databricks dataframe join - ambiguous columns

dataframejoinpysparkdatabricksdelta-live-tables

Score: 0

Answers: 2

Burak Turan

Reputation: 11

Py4JJavaError : An error occurred while calling o745.save.\n: org.apache.spark.SparkException

apache-sparkpyspark

Score: 0

Answers: 0

Shiva Ram

Reputation: 61

How to write data in the dataframe into single .parquet file(both data & metadata in single file) in HDFS?

apache-sparkpysparkapache-spark-sqlbiginsights

Score: 6

Answers: 2

Rcheologist

Reputation: 303

DLT pipeline inserts pipeline Update ID into the table name and raises permission denied error on it

pysparkdatabricksdlt

Score: 0

Answers: 0

Mohammad

Reputation: 1013

Join two PySpark DataFrames and get some of the columns from one DataFrame when column names are similar

dataframeapache-sparkjoinpyspark

Score: 0

Answers: 2

Matthew Thomas

Reputation: 861

partitionOverwriteMode dynamic and "logical" partitions

apache-sparkpysparkparquet

Score: 0

Answers: 1

Michael

Reputation: 1

Allocation of Executors in Callee/Caller Spark Notebooks

apache-sparkpysparkazure-synapse

Score: 0

Answers: 0

shri

Reputation: 47

Trouble connecting pyspark application to Azure Eventhub

azurepysparkapache-kafkaazure-eventhub

Score: 0

Answers: 0

Saugat Mukherjee

Reputation: 1000

Suppress py4j.clientserver logs in pyspark (databricks)

apache-sparkpysparkdatabricksazure-databrickspy4j

Score: 0

Answers: 0

coding

Reputation: 167

Accessing Azure Key Vault Secrets list from Fabric Notebook using Managed Private Endpoint

azurepysparkazure-keyvaultmicrosoft-fabric

Score: 0

Answers: 0

Climbs_lika_Spyder

Reputation: 6754

How do I access the fields within a VARIANT column while reading from Kafka using Spark?

apache-sparkpysparkapache-kafkadatabricksvariant-format

Score: 0

Answers: 2

Jason

Reputation: 1130

Spark Physical Plan: meaning of number of input batches in ColumnarToRow operator

apache-sparkpysparkapache-spark-sql

Score: 0

Answers: 2

DataEngineer_LBP

Reputation: 1

<class 'TypeError'>: Object of type 'DataFrame' is not JSON serializable

pythonapache-sparkpyspark

Score: 0

Answers: 1

Kania

Reputation: 31

Error on saving pyspark dataframe to parquet

pythonapache-sparkpysparkapache-spark-sql

Score: 1

Answers: 1

Andres Ospina

Reputation: 138

Pyspark connection to the Microsoft SQL server?

sql-serverjdbcpysparkodbcpyodbc

Score: 4

Answers: 3

Anuj Patil

Reputation: 1

TypeError: 'Column' object is not callable in pyspark file reading

pythonmatplotlibpysparkjupyter-notebookseaborn

Score: -5

Answers: 0

chaitra k

Reputation: 421

Delta merge logic whenMatchedDelete case

pysparkdelta-lake

Score: 4

Answers: 3

Matej N

Reputation: 21

Too many .parquet files in the output folder of pyspark app

apache-sparkpysparkparquet

Score: 0

Answers: 0

PreviousPage 1Next

StackOverflow Questions for Tag: pyspark

Extract specific dictionary value from dataframe in PySpark

Pyspark - Convert array type to string type for ARRAY&lt;MAP&lt;STRING, STRING&gt;&gt; type

Databricks dataframe join - ambiguous columns

Py4JJavaError : An error occurred while calling o745.save.\n: org.apache.spark.SparkException

How to write data in the dataframe into single .parquet file(both data &amp; metadata in single file) in HDFS?

DLT pipeline inserts pipeline Update ID into the table name and raises permission denied error on it

Join two PySpark DataFrames and get some of the columns from one DataFrame when column names are similar

partitionOverwriteMode dynamic and &quot;logical&quot; partitions

Allocation of Executors in Callee/Caller Spark Notebooks

Trouble connecting pyspark application to Azure Eventhub

Suppress py4j.clientserver logs in pyspark (databricks)

Accessing Azure Key Vault Secrets list from Fabric Notebook using Managed Private Endpoint

How do I access the fields within a VARIANT column while reading from Kafka using Spark?

Spark Physical Plan: meaning of number of input batches in ColumnarToRow operator

&lt;class &#39;TypeError&#39;&gt;: Object of type &#39;DataFrame&#39; is not JSON serializable

Error on saving pyspark dataframe to parquet

Pyspark connection to the Microsoft SQL server?

TypeError: &#39;Column&#39; object is not callable in pyspark file reading

Delta merge logic whenMatchedDelete case

Too many .parquet files in the output folder of pyspark app

Pyspark - Convert array type to string type for ARRAY<MAP<STRING, STRING>> type

How to write data in the dataframe into single .parquet file(both data & metadata in single file) in HDFS?

partitionOverwriteMode dynamic and "logical" partitions

<class 'TypeError'>: Object of type 'DataFrame' is not JSON serializable

TypeError: 'Column' object is not callable in pyspark file reading