CODEWITHSUNDEEP
pyspark
user15649753
user15649753

Reputation: 523

I have 2 columns, how I can fill one of them based on the other one?

I have 2 columns, how I can fill one of them based on the other one?

if col2 was non_null make the col1==col2 and keep col1 everywhere:

col1     col2
null       'us'
'us'       null
'us'       null 
null       'us'
null       'us'
null       null

output

col1     col2
'us'       'us'
'us'       null
'us'       null 
'us'       'us'
'us'       'us'
null       null

Upvotes: 0

Views: 44

Answers (1)

Linus
Linus

Reputation: 669

pyspark.sql.functions.coalesce(*cols): Returns the first column that is not null.

data = spark.createDataFrame([
    (None, 'us'),
    ('us', None),
    (None, 'us'),
    (None, None),
], ['col1', 'col2'])

data.withColumn('col1', coalesce(col('col1'), col('col2'))).show(10)
# +----+----+
# |col1|col2|
# +----+----+
# |  us|  us|
# |  us|null|
# |  us|  us|
# |null|null|
# +----+----+

Upvotes: 1

Related Questions