CODEWITHSUNDEEP
javaamazon-web-servicesjenkinsamazon-s3
oggmonster
oggmonster

Reputation: 4872

AWS S3 local server for integration testing

I have some code that uploads and downloads files using AWS S3 (using the Java AWS SDK). I want to be able to write some tests for it, I was wondering if anyone has any good options. Ideally I would like a light-weight S3 server that runs locally that can be started fast and requires no system configuration (the tests need to be run by Jenkins).

Some options I have looked at so far:

  1. FakeS3 - Almost exactly what I'm looking for, however, when using the Java AWS SDK, you must edit your /etc/hosts file and restart networking, not something I can do in Jenkins. Also when trying it out there seems to be a bug with the creation date field being formatted wrong which makes my client throw an exception, which doesn't inspire me with much confidence in the project.
  2. Ceph - Implements S3 API but takes several minutes to install

Upvotes: 47

Views: 78246

Answers (8)

Bar Hemo
Bar Hemo

Reputation: 11

ive tried both minio and localstack, and the problem with localstack is that the storage in the s3 bucket is not persistence. I think only if you have the pro version it will support percistency. minio was very easy to use, and it is persistent for free.

Upvotes: 0

Ewoks
Ewoks

Reputation: 12445

Minio offers (in my opinion) the best set of features, flexibility and ease of use. It is available as a docker container or binary for major OSes.

To start with minio, it is as easy as:

  • Download
  • Start the binary minio server /data
  • Use it

It works flawlessly with s3cmd and it has nice documentation for popular programming languages.

Upvotes: 6

Flux
Flux

Reputation: 10940

You can try localstack, which is an open source local AWS cloud stack made for testing. It provides implementations of several of AWS services, including S3.

It looks like a very popular open source project on GitHub.

Upvotes: 25

Artur Gajowy
Artur Gajowy

Reputation: 2038

findify/s3mock - an in-process, Java S3 server aimed at testing. Didn't test it - just stumbled upon it. Needs no docker, which might be an advantage. HTH! :)

Upvotes: 3

Ian Bytchek
Ian Bytchek

Reputation: 9085

Late answer, will be useful mostly for Docker users. There's a great S3 compatible storage software called Riak CS and there's docker-riak-cs image that allows to quickly launch the server.

I've been using it for nearly 2 years for local development and integration testing with great success. It has some limitations, but nothing major that comes in the way, see api / compatibility documentation.

If you need Docker-less solution, you can set it up locally for each build, all setup and configuration scripts are available in docker-riak-cs repository.

Upvotes: 9

RalfU
RalfU

Reputation: 41

I started a S3 Server API project for Ladon, it contains a simple File System Repository. Its a Java Project and contains a Spring Boot Starter for simple testing. Not all S3 API features are supported yet but I will add them on request. Its on Github: Ladon S3 Server

Upvotes: 4

koolhead17
koolhead17

Reputation: 1964

You can try installing minio server on your laptop/system, its open source & single static binary. Server is S3 compatible. Then you can try minio-java client library for all operations, following is basic operations example.

Installing minio server [GNU/Linux]

$ wget https://dl.minio.io/server/minio/release/linux-amd64/minio
$ chmod 755 minio
$ ./minio --help

Hope it helps Disclaimer: I work for Minio

Upvotes: 18

Anthony Hobbs
Anthony Hobbs

Reputation: 44

I created different buckets to use for the different use cases. For example: my-dev-bucket and my-prod-bucket. I don't know if this meets your use criteria but you might want to consider it. The side benefit is it makes your pre production and production code follow the exact same flows.

Upvotes: -9

Related Questions