Reputation: 9742
I was just putting together a script to do a backup of a mysql database and upload it to s3... I used "s3cmd la", to see the contents of what was currently on this s3 bucket, and was suprised to see a million files like this:
2012-10-23 12:19 411 s3://xv_backups/xv.backups_2012-10-23-12-19-55-F6DC46D0F13059F7
2012-10-23 12:30 410 s3://xv_backups/xv.backups_2012-10-23-12-30-27-76B099AC21454FED
2012-10-23 13:21 411 s3://xv_backups/xv.backups_2012-10-23-13-21-53-0A339BDD587ECACD
2012-10-23 13:28 410 s3://xv_backups/xv.backups_2012-10-23-13-28-23-65B7B3AEAAAD0FDF
2012-10-23 14:18 410 s3://xv_backups/xv.backups_2012-10-23-14-18-07-29FFF4BD0AE9AB02
2012-10-23 14:26 410 s3://xv_backups/xv.backups_2012-10-23-14-26-43-3BC4C3011FB93627
2012-10-23 14:28 410 s3://xv_backups/xv.backups_2012-10-23-14-28-19-CAC78643A195BFB5
2012-10-23 15:21 410 s3://xv_backups/xv.backups_2012-10-23-15-21-34-BABF49A52A8BA824
2012-10-23 16:00 411 s3://xv_backups/xv.backups_2012-10-23-16-00-30-072D0A74DBA71008
2012-10-23 16:21 410 s3://xv_backups/xv.backups_2012-10-23-16-21-12-B81605B136D338F3
2012-10-23 16:28 378 s3://xv_backups/xv.backups_2012-10-23-16-28-15-1BDAA5F42E3E5EAE
2012-10-23 16:57 410 s3://xv_backups/xv.backups_2012-10-23-16-57-50-CF7662865C347D1D
2012-10-23 17:20 410 s3://xv_backups/xv.backups_2012-10-23-17-20-19-798E3C0312C959A0
2012-10-23 17:31 411 s3://xv_backups/xv.backups_2012-10-23-17-31-31-18C3CD91ACE9D6ED
2012-10-23 17:56 410 s3://xv_backups/xv.backups_2012-10-23-17-56-06-8B197E8EE680A9F5
2012-10-23 18:19 410 s3://xv_backups/xv.backups_2012-10-23-18-19-54-486BC0B861DF196E
2012-10-23 19:24 411 s3://xv_backups/xv.backups_2012-10-23-19-24-03-129C8E3227876BDA
I notice they are all relatively small files-- but... where in the world are they coming from? I id "s3cmd get" to download one to inspect it, and I see:
06859b77eb778603ede74dfa6b3444948d577d4c5bca27bfaa1fadd6ed56d35a xv_backups [23/Oct/2012:18:19:54 +0000] 10.186.158.42 3272ee65a908a7677109fedda345db8d9554ba26398b2ca10581de88777e2b61 88B05F67852EAA0D REST.PUT.OBJECT bb.backups_2012-10-23-18-19-54-486BC0B861DF196E "PUT /xv_backups/xv.backups_2012-10-23-18-19-54-486BC0B861DF196E HTTP/1.1" 200 - - 410 48 28 "-" "Jakarta Commons-HttpClient/3.0" -
So--- something is making a put request somewhere and it's putting this funky file on s3? does anyone know what this is, how it got here, how I can stop it, should I be concerned?
Upvotes: 0
Views: 361
Reputation: 195
TL;DR - Don't log to the same bucket in Amazon S3
I had a similar issue.
Basically, it’s logging that it wrote a log file. Writing this line to a log file triggers another log, and another, etc. Fortunately S3’s logging isn’t in real-time - it seems to only write a few hundred files per day.
http://kylecronin.me/blog/2011/11/30/dont-log-to-the-same-bucket-in-amazon-s3.html
Upvotes: 1
Reputation: 4420
Those look like they are coming from S3 bucket logging. Have you turned on logging for the bucket in question?
Upvotes: 0