CODEWITHSUNDEEP
apache-sparkpysparkapache-spark-sqlprefixes
Nele
Nele

Reputation: 91

How to subset spark dataframe by prefixes of the column names?

The column names of my spark dataframe df are: A_x1, A_x2, B_x1, B_x2, C_x1, C_x2.

How do I create 3 new spark dataframes from df by using the prefixes? The output should look like this:

Thank you!

Upvotes: 1

Views: 1301

Answers (1)

mck
mck

Reputation: 42352

You can use colRegex to filter the columns:

A_ = df.select(df.colRegex('`A_.*`'))
B_ = df.select(df.colRegex('`B_.*`'))
C_ = df.select(df.colRegex('`C_.*`'))

Upvotes: 5

Related Questions