scala mongodb IN filter use

Question

I have been struggling with the Filters for MongoDB in the Scala mongo Driver. I am trying to get all documents in a collection where a specific string in the document is in a list of strings. I create a list of strings in Scala and then I build the query with the IN operator:

val find3 = in(EventItem, ListDelayedItems.toList)

According to the documentation, the in has two overloaded methods:

public static <TItem> Bson in(String fieldName,
                              TItem... values)
public static <TItem> Bson in(String fieldName,
                              Iterable<TItem> values)

The problem is, I have no idea how to construct Iterable values. If I pass the values as a list of values (see the first overloaded method),

in(EventItem, "ABCDE", "DEF", "ESO4SAP")

it works. This is what I see when I print the in operator filter

Operator Filter{fieldName='billing_event.action.resources.item', operator='$in', value=[ABCDE, DEF, ESO4SAP]}

But as soon as I try to pass a list in the second parameter to use the second overloaded method, it does not work. The list I am trying to use to filter is dynamic. I have tried scala Lists, ListBuffers, java.util.Lists but every time I use the variable like this:

in(EventItem, ListValidItems)

It stops working. I defined the ListValidItems as every collection I know, but when I see the log I see this:

Operator Filter{fieldName='billing_event.action.resources.item', operator='$in', value=[List(ABCDE, EFGH, ESO4SAP)]}

So it changed from value=[ABCDE, DEF, ESO4SAP] to value=[List(ABCDE, EFGH, ESO4SAP)] and the filter does not work.

As I mentioned, I tried a lot of different list, tried list.toIterator but nothing seems to work for converting the list data structure to a list of values TItem... or Iterable

This is the list of Imports for the mongoDB driver

import org.mongodb.scala.connection.ClusterSettings
import com.mongodb.BasicDBObject
import com.mongodb.client.model.{FindOneAndReplaceOptions, FindOneAndUpdateOptions}
import org.mongodb.scala.ServerAddress
import org.mongodb.scala._
import org.mongodb.scala.bson.{BsonDocument, conversions}
import org.mongodb.scala.model.Filters._
import org.mongodb.scala.model.Updates._

Any ideas?

Thanks!!

Answer 1

import java.util.concurrent.TimeUnit
import org.bson.Document
import org.mongodb.scala.bson.BsonDocument
import org.mongodb.scala.bson.conversions.Bson
import org.mongodb.scala.{FindObservable, MongoClient, MongoCollection, MongoDatabase, Observer, Subscription}

val client: MongoClient = MongoClient("mongodb://127.0.0.1:27017")
val db: MongoDatabase = client.getDatabase("test")
val dbColl: MongoCollection[Document] = db.getCollection[Document]("md" + TaskContext.getPartitionId)
val ids: List[String] = List("A","B","C")  
val listStr: String = ids.map(o => "\"" + o + "\"").distinct.mkString(",")
val filterStr = """{_id:{$in:""" + s"""[$listStr]}}"""
val filterDoc: Document = Document.parse(filterStr)
val obs: FindObservable[Document] = dbColl.find(filterDoc)
val documents: Seq[Document] = Await.result(obs.toFuture(), Duration(10, TimeUnit.SECONDS))

This works.

Answer 2

According to the signature of Filters#in, the values parameter is varargs. This is how it supports syntax passing an arbitrary number of arguments, like this:

scala> in(EventItem, "ABCDE", "DEF", "ESO4SAP")
res1: org.mongodb.scala.bson.conversions.Bson = Operator Filter{fieldName='EventItem', operator='$in',
value=[ABCDE, DEF, ESO4SAP]}

However, in your case, you have a collection, such as a Scala List, that you want to build up with variable values, and then pass to in. Doing that directly, in ends up interpreting values as a single value, which is the List itself containing the elements.

scala> val ListValidItems = List("ABCDE", "DEF", "ESO4SAP")
ListValidItems: List[String] = List(ABCDE, DEF, ESO4SAP)

scala> in(EventItem, ListValidItems)
res2: org.mongodb.scala.bson.conversions.Bson = Operator Filter{fieldName='EventItem', operator='$in',
value=[List(ABCDE, DEF, ESO4SAP)]}

It requires some extra syntax (ListValidItems:_*) to "unroll" the collection for passing its individual elements as varargs:

scala> in(EventItem, ListValidItems:_*)
res4: org.mongodb.scala.bson.conversions.Bson = Operator Filter{fieldName='EventItem', operator='$in',
value=[ABCDE, DEF, ESO4SAP]}

The Scala documentation on Types discusses this, in particular the section on Ascription.

Earlier, it appeared you were attempting to call the Java API from within the Scala code, indicated by the presence of the Java method signatures in your question statement:

public static <TItem> Bson in(String fieldName,
                              TItem... values)

public static <TItem> Bson in(String fieldName,
                              Iterable<TItem> values)

If that was what you were trying to do, then you could prepare the ListValidItems as your Scala collection of choice and then convert to an equivalent Java collection before passing to in. The Java collection will implement the Java Iterable interface.

The easiest way to do this is using the JavaConverters object that is included in Scala. If you import it, then it will add extension methods to the collection classes to support the conversion. Here is a demonstration in the REPL.

scala> // Import converters.

scala> import collection.JavaConverters._
import collection.JavaConverters._

scala> // Create Scala List.

scala> val ListValidItems = List("ABCDE", "DEF", "ESO4SAP")
ListValidItems: List[String] = List(ABCDE, DEF, ESO4SAP)

scala> // Convert to Java List.

scala> val ListValidItemsConverted = ListValidItems.asJava
ListValidItemsConverted: java.util.List[String] = [ABCDE, DEF, ESO4SAP]

scala> // Prove that it is a Java Iterable.

scala> ListValidItemsConverted.asInstanceOf[java.lang.Iterable[java.lang.String]]
res8: Iterable[String] = [ABCDE, DEF, ESO4SAP]

For more details, see Conversions Between Java and Scala Collections in the Scala Guides and Overviews.