CODEWITHSUNDEEP
amazon-web-serviceselasticsearchamazon-ecs
Stefano Giacone
Stefano Giacone

Reputation: 2153

ElasticSearch with Docker: how to persist data with AWS

I'm trying to run ElasticSearch on Docker (actually on AWS ECS). If I don't configure the volume it's working correctly, but every time I restart the container I lose all the data. I can't figure out how to configure the volume.

What I tried:

  1. in the task definition I configured volume "Name=esdata1" and "source path=/usr/share/elasticsearch/data"
  2. inside the container definition in the "storage and logging" section I configured the mount point "source volume= esdata1" and "container path=/usr/share/elasticsearch/data"

Now when I launch the container it fail with error "access denied" when elasticsearch try to write in "/usr/share/elasticsearch/data". So in the section Security I configured "user=ec2-user" but then the container will not even launch (stay in "status=created"). What should I do? I guess the issue is due to the fact that the user of the container must be the same of the one on the host. The user on the host is "ec2-user", I don't know how to proceed.

Edit:

I'm now able to persist data with this configuration: enter image description here enter image description here

docker inspect:

"Mounts": [
            {
                "Name": "elasticsearch_data",
                "Source": "/var/lib/docker/volumes/elasticsearch_data/_data",
                "Destination": "/usr/share/elasticsearch/data",
                "Driver": "local",
                "Mode": "z",
                "RW": true,
                "Propagation": "rprivate"
            }
        ]

Now data persist if I stop the container or I reboot the host. My only last concern is that this folder "/var/lib/docker/volumes/elasticsearch_data/_data" is located on the OS volume and not on the bigger docker volume. From Aws doc:

Amazon ECS-optimized AMIs from version 2015.09.d and later launch with an 8 GiB volume for the operating system that is attached at /dev/xvda and mounted as the root of the file system. There is an additional 22 GiB volume that is attached at /dev/xvdcz that Docker uses for image and metadata storage. The volume is configured as a Logical Volume Management (LVM) device and it is accessed directly by Docker via the devicemapper back end.

How can I persist data on /dev/xvdcz?

Thanks very much

Upvotes: 2

Views: 2796

Answers (1)

doorstuck
doorstuck

Reputation: 2308

Your sourcepath is the path on the host instance where the data is written. In your case elasticsearch_data. You need to point sourcepath to a folder that exists and that is on the disk you want on the EC2 instance.

So attach an EBS disk to the instance. Mount the disk in a place like /data/es and set your source path to that folder.

But remember that to properly run ES you would probably need a cluster of machines that are connected and automated backups. Consider using the managed ES from Amazon if you plan to host critical data. It does not sound like you have a very robust setup here.

Upvotes: 1

Related Questions