CODEWITHSUNDEEP
scala
Veerapat Boonvanich
Veerapat Boonvanich

Reputation: 251

how to cross product of list without OutOfMemory

this is my code

def main(args: Array[String]): Unit = {
    genSomeThing.grouped(100).grouped(1000)
    // save to MongoDB
  }

  def genSomeThing: Stream[String] = {
    for {
      a <- ('A' to 'Z').toStream
      b <- ('0' to '9').toStream
      c <- ('A' to 'Z').toStream
      d <- ('0' to '9').toStream
      e <- ('A' to 'Z').toStream
      f <- ('A' to 'Z').toStream
    } yield s"$a$b$c$d$e$f"
  }

and I make bulk insert to MongoDB (100 col, 1000 record), I get exception OutofMemory java heap space or GC limit exceed

how to save to MongoDB without exception (I change Stream to Iterator but useless and get the same exception)

Upvotes: 0

Views: 54

Answers (1)

jwvh
jwvh

Reputation: 51271

This is what I came up with, trying to follow your desired result strings.

class dataStr extends Iterator[String] {
  private val itr: Iterator[BigInt] = Iterator.iterate(0:BigInt)(_ + 1)

  override def hasNext = true
  override def next = {
    val na = itr.next ; val ca: Char = ('A' + na%26).toChar
    val nb = na/26    ; val cb: Char = ('A' + nb%26).toChar
    val nc = nb/26    ; val cc: Char = ('0' + nc%10).toChar
    val nd = nc/10    ; val cd: Char = ('A' + nd%26).toChar
    val ne = nd/26    ; val ce: Char = ('0' + ne%10).toChar
    val nf = ne/10    ; val cf: Char = ('A' + nf%26).toChar
    s"$cf$ce$cd$cc$cb$ca"
  }
}

From here you can group and insert as needed.

val ds = new dataStr
ds.next                       // res0: String = A0A0AA
ds.grouped(100).grouped(1000) // res1: GroupedIterator[Seq[String]] = non-empty iterator
ds.next                       // res2: String = A0O7YF

Upvotes: 1

Related Questions