CODEWITHSUNDEEP
dockerelasticsearch
Shadi
Shadi

Reputation: 10335

Copy data from elastic.co cloud cluster to local docker elasticsearch service

My goal is to move data from an elasticsearch cluster hosted in the elastic.co cloud to a local elasticsearch service defined in a docker-compose.yml file.

This SO question discusses how to add nodes to a cluster defined on the same machine. The documentation states

When you run a second node on the same machine, it automatically discovers and joins the cluster as long as it has the same cluster.name as the first node. However, for nodes running on different machines to join the same cluster, you need to configure a list of unicast hosts the nodes can contact to join the cluster. For more information, see Prefer Unicast over Multicast.

The elasticsearch service defined in the docker-compose.yml file is

  elasticsearch1:
    image: docker.elastic.co/elasticsearch/elasticsearch:6.1.1
    ports:
      - "9200:9200"
      - "9300:9300"
    mem_limit: 1g
    environment:
      - cluster.name=97ec5e0ea90e0016e26f078f767b4ea4
      # - bootstrap.memory_lock=true
      - node.name=ec2-001
      # - discovery.zen.ping.multicast.enabled=false # with elastic
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
      - "discovery.zen.ping.unicast.hosts=ip.of.elastic.co.cluster"

Here are the logs of the docker image

Attaching to dockersicnamecheck_elasticsearch1_1
elasticsearch1_1  | [2018-03-30T09:39:42,601][INFO ][o.e.n.Node               ] [ec2-001] initializing ...
elasticsearch1_1  | [2018-03-30T09:39:42,711][INFO ][o.e.e.NodeEnvironment    ] [ec2-001] using [1] data paths, mounts [[/ (overlay)]], net usable_space [12.8gb], net total_space [19.3gb], types [overlay]
elasticsearch1_1  | [2018-03-30T09:39:42,713][INFO ][o.e.e.NodeEnvironment    ] [ec2-001] heap size [495.3mb], compressed ordinary object pointers [true]
elasticsearch1_1  | [2018-03-30T09:39:42,716][INFO ][o.e.n.Node               ] [ec2-001] node name [ec2-001], node ID [btVubyQGQWCvek5bNwIMBg]
elasticsearch1_1  | [2018-03-30T09:39:42,717][INFO ][o.e.n.Node               ] [ec2-001] version[6.1.1], pid[1], build[bd92e7f/2017-12-17T20:23:25.338Z], OS[Linux/4.4.0-1052-aws/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_151/25.151-b12]
elasticsearch1_1  | [2018-03-30T09:39:42,717][INFO ][o.e.n.Node               ] [ec2-001] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.cgroups.hierarchy.override=/, -Xms512m, -Xmx512m, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config]
elasticsearch1_1  | [2018-03-30T09:39:44,926][INFO ][o.e.p.PluginsService     ] [ec2-001] loaded module [aggs-matrix-stats]
elasticsearch1_1  | [2018-03-30T09:39:44,927][INFO ][o.e.p.PluginsService     ] [ec2-001] loaded module [analysis-common]
elasticsearch1_1  | [2018-03-30T09:39:44,927][INFO ][o.e.p.PluginsService     ] [ec2-001] loaded module [ingest-common]
elasticsearch1_1  | [2018-03-30T09:39:44,927][INFO ][o.e.p.PluginsService     ] [ec2-001] loaded module [lang-expression]
elasticsearch1_1  | [2018-03-30T09:39:44,932][INFO ][o.e.p.PluginsService     ] [ec2-001] loaded module [lang-mustache]
elasticsearch1_1  | [2018-03-30T09:39:44,932][INFO ][o.e.p.PluginsService     ] [ec2-001] loaded module [lang-painless]
elasticsearch1_1  | [2018-03-30T09:39:44,932][INFO ][o.e.p.PluginsService     ] [ec2-001] loaded module [mapper-extras]
elasticsearch1_1  | [2018-03-30T09:39:44,932][INFO ][o.e.p.PluginsService     ] [ec2-001] loaded module [parent-join]
elasticsearch1_1  | [2018-03-30T09:39:44,932][INFO ][o.e.p.PluginsService     ] [ec2-001] loaded module [percolator]
elasticsearch1_1  | [2018-03-30T09:39:44,932][INFO ][o.e.p.PluginsService     ] [ec2-001] loaded module [reindex]
elasticsearch1_1  | [2018-03-30T09:39:44,932][INFO ][o.e.p.PluginsService     ] [ec2-001] loaded module [repository-url]
elasticsearch1_1  | [2018-03-30T09:39:44,932][INFO ][o.e.p.PluginsService     ] [ec2-001] loaded module [transport-netty4]
elasticsearch1_1  | [2018-03-30T09:39:44,932][INFO ][o.e.p.PluginsService     ] [ec2-001] loaded module [tribe]
elasticsearch1_1  | [2018-03-30T09:39:44,933][INFO ][o.e.p.PluginsService     ] [ec2-001] loaded plugin [ingest-geoip]
elasticsearch1_1  | [2018-03-30T09:39:44,933][INFO ][o.e.p.PluginsService     ] [ec2-001] loaded plugin [ingest-user-agent]
elasticsearch1_1  | [2018-03-30T09:39:44,933][INFO ][o.e.p.PluginsService     ] [ec2-001] loaded plugin [x-pack]
elasticsearch1_1  | [2018-03-30T09:39:48,918][INFO ][o.e.x.m.j.p.l.CppLogMessageHandler] [controller/84] [Main.cc@128] controller (64 bit): Version 6.1.1 (Build c508cf991ee61c) Copyright (c) 2017 Elasticsearch BV
elasticsearch1_1  | [2018-03-30T09:39:49,453][INFO ][o.e.d.DiscoveryModule    ] [ec2-001] using discovery type [zen]
elasticsearch1_1  | [2018-03-30T09:39:50,366][INFO ][o.e.n.Node               ] [ec2-001] initialized
elasticsearch1_1  | [2018-03-30T09:39:50,366][INFO ][o.e.n.Node               ] [ec2-001] starting ...
elasticsearch1_1  | [2018-03-30T09:39:50,553][INFO ][o.e.t.TransportService   ] [ec2-001] publish_address {172.18.0.1:9300}, bound_addresses {[::]:9300}
elasticsearch1_1  | [2018-03-30T09:39:50,596][INFO ][o.e.b.BootstrapChecks    ] [ec2-001] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
elasticsearch1_1  | [2018-03-30T09:39:53,713][INFO ][o.e.c.s.MasterService    ] [ec2-001] zen-disco-elected-as-master ([0] nodes joined), reason: new_master {ec2-001}{btVubyQGQWCvek5bNwIMBg}{YumJMzVvQAScwJO8kLp5nA}{172.18.0.1}{172.18.0.1:9300}{ml.machine_memory=1073741824, ml.max_open_jobs=20, ml.enabled=true}
elasticsearch1_1  | [2018-03-30T09:39:53,722][INFO ][o.e.c.s.ClusterApplierService] [ec2-001] new_master {ec2-001}{btVubyQGQWCvek5bNwIMBg}{YumJMzVvQAScwJO8kLp5nA}{172.18.0.1}{172.18.0.1:9300}{ml.machine_memory=1073741824, ml.max_open_jobs=20, ml.enabled=true}, reason: apply cluster state (from master [master {ec2-001}{btVubyQGQWCvek5bNwIMBg}{YumJMzVvQAScwJO8kLp5nA}{172.18.0.1}{172.18.0.1:9300}{ml.machine_memory=1073741824, ml.max_open_jobs=20, ml.enabled=true} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)]])
elasticsearch1_1  | [2018-03-30T09:39:53,749][INFO ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [ec2-001] publish_address {172.18.0.1:9200}, bound_addresses {[::]:9200}
elasticsearch1_1  | [2018-03-30T09:39:53,750][INFO ][o.e.n.Node               ] [ec2-001] started
elasticsearch1_1  | [2018-03-30T09:39:53,847][INFO ][o.e.g.GatewayService     ] [ec2-001] recovered [0] indices into cluster_state
elasticsearch1_1  | [2018-03-30T09:39:54,453][INFO ][o.e.l.LicenseService     ] [ec2-001] license [7da059ea-edbb-4ec5-a199-cc9374b546d2] mode [basic] - valid
elasticsearch1_1  | [2018-03-30T09:40:00,503][INFO ][o.e.c.m.MetaDataCreateIndexService] [ec2-001] [.monitoring-es-6-2018.03.30] creating index, cause [auto(bulk api)], templates [.monitoring-es], shards [1]/[1], mappings [doc]

The local docker node doesn't see the elastic.co cloud node. Any advice?

Related: Elasticsearch in docker container cluster

Upvotes: 0

Views: 301

Answers (1)

dadoonet
dadoonet

Reputation: 14492

You should not do that. It's not supported to have a cluster split on multiple datacenters and specifically with cloud. To do that, it would mean that cloud has access to your local machine which I doubt you exposed on internet directly.

But, coming back to the need: you would like to export data available in cloud to your local machine.

You can start a local cluster as you did, but this cluster is running on its own. Then use the reindex from remote API where you can read from remote (cloud) and reindex locally.

That'd be much easier in my opinion.

Upvotes: 1

Related Questions