CODEWITHSUNDEEP
rapache-spark-sqlsparkrsparklyr
Ruth van Dongen
Ruth van Dongen

Reputation: 31

How to convert from SparkR to sparklyr?

How do I convert a SparkDataFrame from SparkR into a tbl_spark from sparklyr?

A similar question was asked here: Convert spark dataframe to sparklyR table "tbl_spark".

The suggestion was to use the sdf_copy_to function, however, the input of this function must be an R object in instead of a SparkDataFrame.

Any suggestions to solve this problem?

Thanks!

Upvotes: 3

Views: 1914

Answers (1)

Paul
Paul

Reputation: 9097

Use a temp Spark table to convert from SparkR::SparkDataFrame to sparklyr::tbl_spark.

Starting with a SparkDataFrame in SparkR

df_sparkr <- SparkR::createDataFrame(data.frame(
  x = 1:10
))

Create a temp table in Spark

SparkR::registerTempTable(df_sparkr, "temp_df")

Read the table using sparklyr

sc <- sparklyr::spark_connect(master = "local")
df_sparklyr <- dplyr::tbl(sc, "temp_df")

Here is a second method if your data is small. You can convert to a normal R data frame then copy into sparklyr. This is not recommended if the data frame is large.

df_normal <- SparkR::collect(df_sparkr)
df_sparklyr <- dplyr::copy_to(sc, df_normal)

Upvotes: 1

Related Questions