CODEWITHSUNDEEP
apache-sparkapache-kafkaapache-stormspark-streamingstream-processing
Shades88
Shades88

Reputation: 8360

Port existing php application in spark streaming

We have a huge existing application in php which

  1. Accepts a log file
  2. Initialises all the database, in-memory store resources
  3. Processes every line
  4. Creates a set of output files

Above process happens per input file. Input files are written by a kafka consumer. Is it possible to fit this application in spark streaming by somehow not porting all the code in java? For example in following manner

  1. get a message from kafka topic
  2. Pass this message to spark streaming
  3. Spark streaming somehow interacts with legacy app and generates output
  4. spark then writes output again in kafka

Whatever I have just mentioned is too high level. I just want to know whether there's a possibility of doing this by not recoding existing app in java? And can anyone please tell me roughly how this can be done?

Upvotes: 0

Views: 640

Answers (2)

Alma Alma
Alma Alma

Reputation: 1722

I think Storm is an excellent choice in this case because it offers non-jvm language integration through Thrift. Also I am sure that there is a PHP Thrift client.

So basically what you have to do is finding a ShellSpout and ShellBolt written in PHP (this is the integration part needed to interact with Storm in your application) and then write your own spouts and bolts which are consuming Kafka and processing each line.

You can use this library for your need: https://github.com/Lazyshot/storm-php

Then you will also have to find a PHP Thrift client to interact with the Storm cluster.

The Storm Thrift definition can be found here: https://github.com/apache/storm/blob/master/storm-core/src/storm.thrift

And a PHP Thrift client example can be found here: https://thrift.apache.org/tutorial/php

Now putting these things together you can write your own Apache Storm app in PHP.

Information sources: http://storm.apache.org/about/multi-language.html http://storm.apache.org/releases/current/Using-non-JVM-languages-with-Storm.html

Upvotes: 0

mariusz-s
mariusz-s

Reputation: 1766

I think there is no possibility to use PHP in Spark directly. According to documentation (http://spark.apache.org/) and my knowledge it supports only Java, Scala, R and Python.

However you can change an architecture of your app and create some external services (ws, rest etc) and use them from Spark (you can use whichever library you want) - not all modules from old app must be rewritten to Java. I would try to go in that way :)

Upvotes: 1

Related Questions