CODEWITHSUNDEEP
.netamazon-s3f#
Thomas
Thomas

Reputation: 12127

Saving files to a S3 bucket with a directory structure, in .NET

I am trying to save files to a S3 bucket using the AWSSDK.S3 nuget package. My code is in F#, but answers in C# would work too.

I have a series of files I want to organize by folder. From what I understand, there is not really a folder structure in the S3 buckets, but folders are rather 'keys' which are part of the filename. I could be wrong, but that's my understanding at this point.

The code is taking crypto coin data and making one file a day, per coin. This was working until I wanted to make one folder per coin.

Filenames have the following format: "coinname-YYMMDD.csv', for example: "BTCUSDT-20200820.csv'. and I want a structure like:

coin1/
    coin1-20200819.csv
    coin1-20200820.csv
coin2/
    coin2-20200819.csv
    coin2-20200820.csv

From the documentation, I need to set the folder as a 'key', so here is the code:

// upload a file
let uploadFile (filename: string) =

    // create the key
    let coinName = Path.GetFileName(filename).Split('-').[0]
    let keyName  = Path.Combine(coinName, filename)

    // create the send request
    let putObjectRequest = Amazon.S3.Model.PutObjectRequest()
    putObjectRequest.FilePath     <- filename
    putObjectRequest.Key          <- keyName
    putObjectRequest.BucketName   <- bucketName
    putObjectRequest.CannedACL    <- S3CannedACL.PublicRead
    putObjectRequest.StorageClass <- S3StorageClass.IntelligentTiering
    putObjectRequest.ContentType  <- "text/csv"

    // send the object
    let response =
        s3client.PutObjectAsync(putObjectRequest)
        |> Async.AwaitTask
        |> Async.RunSynchronously

    // DEBUG OUTPUT
    printfn "trying to save %s" keyName
    printfn "object request: %s" (putObjectRequest |> JsonConvert.SerializeObject)
    printfn "response: %s" (response |> JsonConvert.SerializeObject)

    // handle the response
    match response.HttpStatusCode with
    | HttpStatusCode.OK -> Ok()
    | _                 -> let message = sprintf "error uploading file %s to S3: %s" filename (response.ToString())
                           Error(message)

The debug output shows this:

trying to save /app/leechdata/ATOMUSDT/ATOMUSDT-20200819.csv object request: {"CannedACL":{"Value":"public-read"},"InputStream":null,"FilePath":"/app/leechdata/ATOMUSDT/ATOMUSDT-20200819.csv","ContentBody":null,"AutoCloseStream":true,"AutoResetStreamPosition":true,"UseChunkEncoding":true,"BucketName":"sunnybot-cap ture","Headers":{"Count":1,"Keys":["Content-Type"],"CacheControl":null,"ContentDisposition":null,"ContentEncoding":null,"ContentLength":-1,"ContentMD5":null,"ContentType":"text/csv","ExpiresUtc":null,"Expires":null},"Metadata":{"Count":0,"Keys":[]},"Key" :"/app/leechdata/ATOMUSDT/ATOMUSDT-20200819.csv","ServerSideEncryptionMethod":null,"ServerSideEncryptionCustomerMethod":null,"ServerSideEncryptionCustomerProvidedKey":null,"ServerSideEncryptionCustomerProvidedKeyMD5":null,"ServerSideEncryptionKeyManageme ntServiceKeyId":null,"ServerSideEncryptionKeyManagementServiceEncryptionContext":null,"StorageClass":{"Value":"INTELLIGENT_TIERING"},"ObjectLockLegalHoldStatus":null,"ObjectLockMode":null,"ObjectLockRetainUntilDate":"0001-01-01T00:00:00","WebsiteRedirect Location":null,"StreamTransferProgress":null,"ContentType":"text/csv","MD5Digest":null,"RequestPayer":null,"TagSet":[],"Grants":[]} response: {"Expiration":null,"ServerSideEncryptionMethod":null,"ETag":""6a614845e94ffa3fc1bfb8aad40bdb5b"","VersionId":null,"ServerSideEncryptionKeyManagementServiceKeyId":null,"ServerSideEncryptionCustomerMethod":null,"ServerSideEncryptionCustomerProv idedKeyMD5":null,"ServerSideEncryptionKeyManagementServiceEncryptionContext":null,"RequestCharged":null,"ResponseMetadata":{"RequestId":"81892F3E28F07562","Metadata":{"x-amz-id-2":"/mTevk7iATTIKlvIs6eGGc3QK9+rHdzgPu6SzZ36eNfEzyjOS2BjXnEazTchL6z8P9kqU+eZJ N8="}},"ContentLength":0,"HttpStatusCode":200}

so, it looks like everything is fine, but the files do not appear! if I don't specify the Key field, all the files get properly uploaded, without a folder.

What am I missing?

Upvotes: 0

Views: 2789

Answers (1)

Jose Vu
Jose Vu

Reputation: 696

To upload files to S3, I suggest to use TransferUtility, your key should contain both "folder" and "filename", e.g. coin1/coin1-20200819.csv. The TransferUtility.UploadAsync method would take 3 arguments:

  • Path to your file, e.g. @"C:\coin1-20200819.csv"
  • bucket name in your aws, e.g. crypto
  • key name including folder in bucket and filename, e.g. "coin1/coin1-20200819.csv" Your code to upload a file would be something like this
let fileTransferUtil = new TransferUtility( s3client )  // create a transfer utility
fileTransferUtil.UploadAsync(@"C:\coin1-20200819.csv", "crypto", "coin1/coin1-20200819.csv" ) |> Async.AwaitTask |> Async.RunSynchronously

Upvotes: 2

Related Questions