CODEWITHSUNDEEP
pythonamazon-s3apache-beam
Mohit Kumar
Mohit Kumar

Reputation: 740

how to read s3 files from apache beam python?

I am using Apache Beam python SDK to read s3 file data.

code I am using

    ip = (pipe
          | beam.io.ReadFromText("s3://bucket_name/file_path")
          | beam.Map(lambda x: x.split(","))
          | beam.Map(print)

          )

but it gives a error

 S3ClientError('Unable to locate credentials', None)}")}

Anybody have any idea how to give credentials to s3 client thorugh apache beam python file

Upvotes: 0

Views: 853

Answers (1)

robertwb
robertwb

Reputation: 5104

You need to ensure the credentials are available on your worker machines. One way to do this is installing them in a custom container.

Upvotes: 1

Related Questions