CODEWITHSUNDEEP
scalaapache-sparkapache-spark-sql
Cassie
Cassie

Reputation: 3099

Saving JDBC db data as shared state Spark

I have an MSSQL table as a data source and I would like to save some kind of the processing offset in the form of the timestamp (it is one of the table's columns). So it would be possible to process the data from the latest offset. I would like to save as some kind of shared state between Spark sessions. I have researched shared state in Spark session, however, I did not find the way to store this offset in the shared state. So is it possible to use existing Spark constructs to perform this task?

Upvotes: 0

Views: 132

Answers (1)

abiratsis
abiratsis

Reputation: 7316

As far as I know there is no official built-in feature supporting passing data between sessions in Spark. As alternative I would consider the following options/suggestions:

  1. First the offset column must be an indexed field in MSSQL in order to be able to query it fast.
  2. If there is already an in-memory (i.e Redis, Apache Ignite) system installed and used by your project I would store there the offset.
  3. I wouldn't use a message queue system such as Kafka because once you consume one message you will need to resend it therefore that would't make sense.
  4. As solution I would prefer to save it in the filesystem or in Hive even if it would add extra overhead since you will have only one value in that table. In the case of the filesystem of course the performance would be much better.

Let me know if further information is needed

Upvotes: 1

Related Questions