CODEWITHSUNDEEP
hazelcast-jet
Harshad Murtekar
Harshad Murtekar

Reputation: 57

Hazelcast Jet 0.6.1 - Dag Definition

The Hazelcast Jet prints the DAG definition on the console,once started

This converts the Pipeline definition to the DAG.

Here is a Pipeline definition.

    private Pipeline buildPipeline() {
 Pipeline p = Pipeline.create();
 p.drawFrom(Sources.<String, Record>remoteMapJournal("record", getClientConfig(), START_FROM_OLDEST))
          .addTimestamps((v) ->  getTimeStamp(v), 3000)
          .peek()
          .groupingKey((v) -> Tuple2.tuple2(getUserID(v),getTranType(v)))
    .window(WindowDefinition.sliding(SLIDING_WINDOW_LENGTH_MILLIS, SLIDE_STEP_MILLIS))
    .aggregate(counting())
    .map((v)-> getMapKey(v))
    .drainTo(Sinks.remoteMap("Test", getClientConfig()));
    return p;
  }

and here is a DAG definition printed on console.

 .vertex("remoteMapJournalSource(record)").localParallelism(1)
.vertex("sliding-window-step1").localParallelism(4)
.vertex("sliding-window-step2").localParallelism(4)
.vertex("map").localParallelism(4)
.vertex("remoteMapSink(Test)").localParallelism(1)
.edge(between("remoteMapJournalSource(record)", "sliding-window-step1").partitioned(?))
.edge(between("sliding-window-step1", "sliding-window-step2").partitioned(?).distributed())
.edge(between("sliding-window-step2", "map"))
.edge(between("map", "remoteMapSink(Test)"))

Is there any way to get the DAG definition with all the details like sliding window details, aggregation APIs etc ?

Upvotes: 1

Views: 64

Answers (1)

Oliv
Oliv

Reputation: 10812

No, it's technically impossible. If you write a lambda (for example for a key extractor), there's no way to display the code that defined the lambda. The only way for you to get more information is to embed that information into the vertex name.

In Jet 0.7, this printout will be changed to the graphviz format so that you can copy-paste it to a tool and see the DAG as an image.

Upvotes: 1

Related Questions