CODEWITHSUNDEEP
scalajvmapache-beambeam
Dasph
Dasph

Reputation: 446

State in Singleton object changes between workers in Scala

First of all: I'm aware this is not an ideal way to approach this situation, but I have a Beam pipeline in Scala with a singleton object which has a state, and the state is accessed by multiple workers. I have to use this approach because it's a non-serializable object, but the question isn't about the approach itself.

The object, lets say, look like this

object Foo { 
   var bar: Option[MyConfig] = None
   
   def init(myConf: MyConfig) = this.bar = Some(myConf)
}

this method is first called at the entrypoint of the pipeline. It is the second step after loading the config, let's say its like this:

    Object Main {
    
    def main(args: Array[String): Unit = {
      */ elided config load */
      */ elided pipeline creation */
      Foo.init(myConfig)
      pipeline.apply("SomeTransform", ParDo.of(new SomeTransform))
    }
}

And SomeTransform has a method which has to execute if Foo.bar is defined, however, every time a SomeTransform is created and I reach this point, the state is seemingly lost and the conditional step is skipped.

I've added logging to confirm that:

  1. The state IS changing at the base of the pipeline, and this is a singleton object, so it should remain so through the same JVM.
  2. The state (Foo.bar) equals to None when SomeTransform checks for its status

I could also call the init method inside the transform, but I'm afraid about concurrency issues.

Upvotes: 0

Views: 37

Answers (0)

Related Questions