CODEWITHSUNDEEP
scalacassandra
Knows Not Much
Knows Not Much

Reputation: 31546

Read java.util.Iterator parallely

I am using the Cassandra java driver to read data from a Cassandra table. after i execute the query, I get a java.util.Iterator and I am reading it in a for loop.

I wonder if there is any possibility that I could read this iterator in-parallel rather than one item at a time in a for loop.

Can any of the N number of streaming APIs (FS2?) help me here? (No I can't use Spark or Hadoop at this point of time.)

Upvotes: 0

Views: 128

Answers (1)

prayagupadhyay
prayagupadhyay

Reputation: 31192

Did you try Java 8 Spliterator.

Spliterator is an object for traversing and partitioning elements of a source. The source of elements covered by a Spliterator could be, for example, an array, a Collection, an IO channel, or a generator function.

If this is more of question to parallelize the given collection, as you can create partitions and process each partition.

eg.

Spliterator<CassandraRecord> cassandraRecordsSpliterator = recordsIterator.spliterator();

Spliterator<CassandraRecord> eachPartition = cassandraRecordsSpliterator.trySplit();

System.out.println(" " + eachPartition.estimateSize()); //will give you x
System.out.println(" " + cassandraRecordsSpliterator.estimateSize()); // will have N - x

Upvotes: 1

Related Questions