CODEWITHSUNDEEP
javaapache-sparkuser-defined-functions
YerivanLazerev
YerivanLazerev

Reputation: 375

Set field of java UDF instance in spark driver and use in executors

Is it possible to set field on the UDF instance in driver and use it by the executors when call() is invoked?

public class SomeUDF implements UDF2<String, String, String> {
    private String val = "foo";

    public void init(String st){
        val = st;
    }
    @Override
    public String call(String a, String b) {
        return val;
    }
}

py spark:

jvm_udf = spark._jvm.com.example.demo.SomeUDF()
jvm_udf.init("bla")

spark.udf.registerJavaFunction("foo", jvm_udf.getClass().getName(), T.StringType())
df_single_row.withColumn("val", expr(f"foo('a','b')"))

output is:

"foo"

I want:

"bla"

Upvotes: 0

Views: 23

Answers (0)

Related Questions