CODEWITHSUNDEEP
amazon-web-servicesamazon-s3
MayoMan
MayoMan

Reputation: 4917

Updating a file in Amazon S3 bucket

I am trying to append a string to the end of a text file stored in S3. Currently I just read the contents of the file into a String, append my new text and resave the file back to S3. Is there a better way to do this. I am thinkinig when the file is >>> 10MB then reading the entire file would not be a good idea so how should I do this correctly?

Current code [code]

private void saveNoteToFile( String p_note ) throws IOException, ServletException    
{
    String str_infoFileName =  "myfile.json"; 

    String existingNotes = s3Helper.getfileContentFromS3( str_infoFileName  ); 
    existingNotes += p_note;
    writeStringToS3( str_infoFileName , existingNotes );        
}

public void writeStringToS3(String p_fileName, String p_data) throws IOException 
{
  ByteArrayInputStream byteArrayInputStream = new ByteArrayInputStream( p_data.getBytes());

  try {
      streamFileToS3bucket(  p_fileName, byteArrayInputStream, p_data.getBytes().length);
  } 
  catch (AmazonServiceException e)
  {
      e.printStackTrace();
  } catch (AmazonClientException e)
  {
      e.printStackTrace();
  }
}

public void streamFileToS3bucket( String p_fileName,  InputStream input, long size)
{
    //Create sub folders if there is any in the file name.
    p_fileName = p_fileName.replace("\\", "/");
    if( p_fileName.charAt(0) == '/')
    {
        p_fileName = p_fileName.substring(1, p_fileName.length());
    }
    String folder = getFolderName( p_fileName );
    if( folder.length() > 0)
    {
        if( !doesFolderExist(folder))
        {
            createFolder( folder );
        }
    }
    ObjectMetadata metadata =  new ObjectMetadata();
    metadata.setContentLength(size);
    AccessControlList acl = new AccessControlList();
    acl.grantPermission(GroupGrantee.AllUsers, Permission.Read);

    s3Client.putObject(new PutObjectRequest(bucket, p_fileName , input,metadata).withAccessControlList(acl));
}

[/code]

Upvotes: 11

Views: 57587

Answers (4)

antigenius
antigenius

Reputation: 131

You can do it with s3api put-object. First download the version you want and use below commend. it will upload as the latest version.

ᐅ aws s3api put-object --bucket $BUCKET --key $FOLDER/$FILE --body $YOUR_LOCAL_DOWNLOADED_VERSION_FILE

Upvotes: 0

Siddhartha
Siddhartha

Reputation: 1563

It's not possible to append to an existing file on AWS S3.

You can delete existing file and upload new file with same name.

Configuration

    private string bucketName = "my-bucket-name-123";
    private static string awsAccessKey = "AKI............";
    private static string awsSecretKey = "+8Bo..................................";

    IAmazonS3 client = new AmazonS3Client(awsAccessKey, awsSecretKey, 
                                           RegionEndpoint.APSoutheast2);

    string awsFile = "my-folder/sub-folder/textFile.txt";

    string localFilePath = "my-folder/sub-folder/textFile.txt";

To Delete

    public void DeleteRefreshTokenFile()
    {
        try
        {
            var deleteFileRequest = new DeleteObjectRequest
            {
                BucketName = bucketName,
                Key = awsFile
            };
            DeleteObjectResponse fileDeleteResponse = client.DeleteObject(deleteFileRequest);
        }
        catch (Exception ex)
        {
            throw new Exception(ex.Message);
        }
    }

To Upload

        public void UploadRefreshTokenFile()
        {
            FileInfo file = new FileInfo(localFilePath);
            try
            {
                PutObjectRequest request = new PutObjectRequest()
                {
                    InputStream = file.OpenRead(),
                    BucketName = bucketName,
                    Key = awsFile
                };
                PutObjectResponse response = client.PutObject(request);
            }
            catch (Exception ex)
            {
                throw new Exception(ex.Message);
            }
        }

Upvotes: 1

J Roysdon
J Roysdon

Reputation: 402

One option is to write the new lines/information to a new version of the file. This would create a LARGE number of versions. But, essentially, whatever program you are using the file for could read ALL the versions and append them back together when reading it (this seems like a really bad idea as I write it out).

Another option would be to write a new object each time with a time stamp appended to the object name. my-log-file-date-time . Then whatever program is reading from it could append them all together after downloading my-log-file-*.

You would want to delete objects older than a certain time just like log rotation.

Depending on how busy your events are this might work. If you have thousands per second, I don't think this would work. But if you just have a few events per minute it may be reasonable.

Upvotes: 0

Volkan Paksoy
Volkan Paksoy

Reputation: 6957

It's not possible to append to an existing file on AWS S3. When you upload an object it creates a new version if it already exists:

If you upload an object with a key name that already exists in the bucket, Amazon S3 creates another version of the object instead of replacing the existing object

Source: http://docs.aws.amazon.com/AmazonS3/latest/UG/ObjectOperations.html

The objects are immutable.

It's also mentioned in these AWS Forum threads:

https://forums.aws.amazon.com/message.jspa?messageID=179375 https://forums.aws.amazon.com/message.jspa?messageID=540395

Upvotes: 18

Related Questions