CODEWITHSUNDEEP
javaapache-sparkmachine-learning
Bhagwati Malav
Bhagwati Malav

Reputation: 3549

Drop null value column in spark

i am having below given code

ataset.select("Lead Owner").show();
        dataset.filter(dataset.col("Lead Owner").isNotNull());
        dataset.select("Lead Owner").show();

But it is not removing rows which has null value for column Lead Owner. Can anyone tell what wrong i am doing here ?

Upvotes: 1

Views: 1511

Answers (1)

Alex Naspo
Alex Naspo

Reputation: 2102

Datasets are Immutable. dataset.filter is a transformation and will return a new dataset, rather than modifying the original. Please refer to the DataSet Docs

val filteredData = dataset.filter(dataset.col("Lead Owner").isNotNull());
filteredData.select("Lead Owner").show();

Upvotes: 2

Related Questions