CODEWITHSUNDEEP
scalaamazon-web-servicesapache-sparkamazon-sqsspark-structured-streaming
djWann
djWann

Reputation: 2117

How to read streaming datasets from AWS SQS?

For Spark streaming there are connectors that can achieve this functionality.

Unfortunately for Spark structured streaming I couldn't find any as it's a newer technology. Is there a way to connect to a source using a Spark streaming connector? Or is there a way to create a custom connector similar to the way one can be created in a spark streaming application?

Upvotes: 2

Views: 2075

Answers (2)

Jacek Laskowski
Jacek Laskowski

Reputation: 74599

Is there a way to connect to a source using a spark streaming connector?

No idea.

Or is there a way to create a custom connector similar to the way one can be created in a spark streaming application?

With no experience in AWS SQS I however think that if Spark Streaming supports it, Spark Structured Streaming should too.

The streaming Source interface allows you to implement a custom data source to load streaming datasets, possibly from AWS SQS, too.

Use KafkaSource or TextSocketSource as the templates.

Upvotes: 1

Ashan
Ashan

Reputation: 19728

First of all, AWS SQS might not fit for streaming requirements compared to using AWS Kinesis Streams, Kinesis Firehorse and Kinesis Analytics which are focused on Streaming data processing, storage and analytics.

You can event connect Spark Streaming + Kinesis Integration to configure Spark Streaming to receive data from Kinesis.

I don't think there is a spark streaming connector at the moment for AWS SQS but you should be able to write a custom connector using AWS SQS SDK to Poll the Data from Queue and Push to the Spark Stream.

Upvotes: 1

Related Questions