CODEWITHSUNDEEP
rsparklyr
tjerkie
tjerkie

Reputation: 29

How to get original filename in column using sparklyr spark_read_csv?

In pyspark I am able to get the filename in a column using:

df = spark.read.option("delimiter", ";").load(inlees_pad, format='csv', header=True)
df = df.withColumn("filename", input_file_name())

I try the same using sparklyr in R:

sigma_raw <- spark_read_csv(
  sc,
  name = "comma_decimal_df",
  path = inlees_pad,
  delimiter = ";",            # Use ";" as the delimiter
  header = TRUE               # Include headers if the file has them
) %>% mutate(filename = input_file_name())

However in sparklyr the column filename remains empty without an error. Does anyone know how to fix this?

Upvotes: 0

Views: 20

Answers (0)

Related Questions