CODEWITHSUNDEEP
apache-sparkpysparkapache-spark-sql
Roger Steinberg
Roger Steinberg

Reputation: 1604

Pass a list of values from a variable into IN statement sparksql

How to pass a list list1:

list1 = [1,2,3]

into a spark.sql statement

spark.sql(f'select * from tbl where id IN list1')

Upvotes: 2

Views: 5052

Answers (2)

Alex Ott
Alex Ott

Reputation: 87144

To do that you need to format it correctly, and will be more trickier if you use the strings instead of numbers.

If you use PySpark, then it's better to use isin function instead, with something like this:

import pyspark.sql.functions as F
list1 = [1,2,3]
df = spark.read.table("tbl").filter(F.col("id").isin(*list1)

then you can use different data types the same way

Upvotes: 2

Bala
Bala

Reputation: 11244

One way is to convert the list to tuple.

>>> list1
[1, 2, 3]
>>> str = r'select * from tbl where id in ' + format(tuple(list1))
>>> str
'select * from tbl where id in (1, 2, 3)'

Upvotes: 3

Related Questions