CODEWITHSUNDEEP
rsparkr
Ole Petersen
Ole Petersen

Reputation: 680

To use sparkR columns

SparkR Column provides a long list of useful methods for example 'isNull' but in sparkR I have an issue using them. I run sparkR in R like this

cd /home/ole/R/spark-1.4.0 ./bin/sparkR --packages com.databricks:spark-csv_2.10:1.0.3 sqlContext

When I for example type this u=c() isNull(u) I get this message Error in (function (classes, fdef, mtable) : unable to find an inherited method for function ‘isNull’ for signature ‘"NULL"

Upvotes: 0

Views: 646

Answers (1)

Wannes Rosiers
Wannes Rosiers

Reputation: 1690

That is because isNull expects a column of a DataFrame and not a vector. It checks wether entries are NULL, it works as follows:

a   <- createDataFrame(sqlContext,data.frame(b=c("a","b",NA,"c"),c=c(1,2,3,4)))
a$d <- isNull(a$b)
collect(a)

It also returns a (logical) column, that's why I appended it to the DataFrame. You will notice however that SparkR did not store the NA as NULL, so all logicals are FALSE, but you already see how the function is working.

Upvotes: 2

Related Questions