CODEWITHSUNDEEP
ruby-on-railsrubyamazon-web-servicesamazon-s3
Demian Sims
Demian Sims

Reputation: 921

Getting a datafile from AWS S3 Bucket and parse in Rails?

I'm creating a Ruby script in Rails that will:

1) create an S3 object with AWS S3 SDK 2) iterate through bucket and download (get) each file 3) through iteration, store the file in memory and then convert to a string 4) parse the string for data and re-upload the file based on parsed data to an appropriate folder in same bucket

So the code I have so far in Rails jobs: 

  def aws_get
    io = IO.new(1)
    bucket_col = []
    s3 = Aws::S3::Resource.new(region: 'us-east-1', access_key_id: Rails.application.credentials.dig(:aws, :access_key_id), secret_access_key: Rails.application.credentials.dig(:aws, :secret_access_key))
    s3.bucket('missouridata').objects.each do |object| 
      obj = s3.bucket('missouridata').object(object.key)
      file = obj.get(response_target: io) 
      ???
    end
  end

The questions marks is where I don't know what to do next. How do I take the file stored in memory and convert it to a string to be parsed?

Upvotes: 0

Views: 1064

Answers (1)

T.H.Mra
T.H.Mra

Reputation: 27

I have the perfect solution for you. I have beein using fog gem to manipulate S3 bucket for a while. It can do pretty much anything for you.

Here is the reference link.

https://www.ironin.it/blog/manipulating-files-on-amazon-s3-storage-with-rubys-fog-gem.html

Upvotes: 1

Related Questions