patrick
patrick

Reputation: 9742

I seem to have something uploading a million things to amazon s3?

I was just putting together a script to do a backup of a mysql database and upload it to s3... I used "s3cmd la", to see the contents of what was currently on this s3 bucket, and was suprised to see a million files like this:

    2012-10-23 12:19       411   s3://xv_backups/xv.backups_2012-10-23-12-19-55-F6DC46D0F13059F7
    2012-10-23 12:30       410   s3://xv_backups/xv.backups_2012-10-23-12-30-27-76B099AC21454FED
    2012-10-23 13:21       411   s3://xv_backups/xv.backups_2012-10-23-13-21-53-0A339BDD587ECACD
    2012-10-23 13:28       410   s3://xv_backups/xv.backups_2012-10-23-13-28-23-65B7B3AEAAAD0FDF
    2012-10-23 14:18       410   s3://xv_backups/xv.backups_2012-10-23-14-18-07-29FFF4BD0AE9AB02
    2012-10-23 14:26       410   s3://xv_backups/xv.backups_2012-10-23-14-26-43-3BC4C3011FB93627
    2012-10-23 14:28       410   s3://xv_backups/xv.backups_2012-10-23-14-28-19-CAC78643A195BFB5
    2012-10-23 15:21       410   s3://xv_backups/xv.backups_2012-10-23-15-21-34-BABF49A52A8BA824
    2012-10-23 16:00       411   s3://xv_backups/xv.backups_2012-10-23-16-00-30-072D0A74DBA71008
    2012-10-23 16:21       410   s3://xv_backups/xv.backups_2012-10-23-16-21-12-B81605B136D338F3
    2012-10-23 16:28       378   s3://xv_backups/xv.backups_2012-10-23-16-28-15-1BDAA5F42E3E5EAE
    2012-10-23 16:57       410   s3://xv_backups/xv.backups_2012-10-23-16-57-50-CF7662865C347D1D
    2012-10-23 17:20       410   s3://xv_backups/xv.backups_2012-10-23-17-20-19-798E3C0312C959A0
    2012-10-23 17:31       411   s3://xv_backups/xv.backups_2012-10-23-17-31-31-18C3CD91ACE9D6ED
    2012-10-23 17:56       410   s3://xv_backups/xv.backups_2012-10-23-17-56-06-8B197E8EE680A9F5
    2012-10-23 18:19       410   s3://xv_backups/xv.backups_2012-10-23-18-19-54-486BC0B861DF196E
    2012-10-23 19:24       411   s3://xv_backups/xv.backups_2012-10-23-19-24-03-129C8E3227876BDA

I notice they are all relatively small files-- but... where in the world are they coming from? I id "s3cmd get" to download one to inspect it, and I see:

06859b77eb778603ede74dfa6b3444948d577d4c5bca27bfaa1fadd6ed56d35a xv_backups [23/Oct/2012:18:19:54 +0000] 10.186.158.42 3272ee65a908a7677109fedda345db8d9554ba26398b2ca10581de88777e2b61 88B05F67852EAA0D REST.PUT.OBJECT bb.backups_2012-10-23-18-19-54-486BC0B861DF196E "PUT /xv_backups/xv.backups_2012-10-23-18-19-54-486BC0B861DF196E HTTP/1.1" 200 - - 410 48 28 "-" "Jakarta Commons-HttpClient/3.0" -

So--- something is making a put request somewhere and it's putting this funky file on s3? does anyone know what this is, how it got here, how I can stop it, should I be concerned?

Upvotes: 0

Views: 361

Answers (2)

Chip
Chip

Reputation: 195

TL;DR - Don't log to the same bucket in Amazon S3

I had a similar issue.

Basically, it’s logging that it wrote a log file. Writing this line to a log file triggers another log, and another, etc. Fortunately S3’s logging isn’t in real-time - it seems to only write a few hundred files per day.

http://kylecronin.me/blog/2011/11/30/dont-log-to-the-same-bucket-in-amazon-s3.html

Upvotes: 1

Wade Matveyenko
Wade Matveyenko

Reputation: 4420

Those look like they are coming from S3 bucket logging. Have you turned on logging for the bucket in question?

Upvotes: 0

Related Questions