Reputation: 1124

Building 3 node working Ignite cluster on AWS

did anyone successfully launch a three-node ignite cluster on AWS without any issues? if so can some one help with step by step process how to workaround.

Ignite documentation doesn't make sense and its way to little information with no screenshots, the documentation just explains docker instance running on one Ec2 but I need at least three-node ignite cluster on AWS or EMR.

I did try this blog (https://aws.amazon.com/blogs/big-data/real-time-in-memory-oltp-and-analytics-with-apache-ignite-on-aws/) and with the cloud formation json (https://github.com/aws-samples/aws-big-data-blog/blob/master/aws-blog-real-time-in-memory-oltp-and-analytics-with-apache-ignite/cloudformation/configignite.json), below is the configureIgnite.sh script which cloud formation template refers but ignite setup is failing with property syntax error in default-config.xml file

#!/bin/bash

#
# This is a modified version of the file stored at s3://publicbucketbabupe/ignitelibrary/configureIgnite.sh
# which changes the config to use the instance provided credentials rather than requiring access/secret to be passed in
#
# Parameters are 
# 1 - Cache Name
# 2 - Number of replicas
# 3 - S3 Bucket Name
#

echo "<?xml version=\"1.0\" encoding=\"UTF-8\"?>

<!--
            Licensed to the Apache Software Foundation (ASF) under one or more
  contributor license agreements.  See the NOTICE file distributed with
  this work for additional information regarding copyright ownership.
  The ASF licenses this file to You under the Apache License, Version 2.0
  (the \"License\"); you may not use this file except in compliance with
  the License.  You may obtain a copy of the License at

       http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an \"AS IS\" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License.
-->

<beans xmlns=\"http://www.springframework.org/schema/beans\"
       xmlns:util=\"http://www.springframework.org/schema/util\"
       xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\"
       xsi:schemaLocation=\"
       http://www.springframework.org/schema/beans
       http://www.springframework.org/schema/beans/spring-beans.xsd
       http://www.springframework.org/schema/util
       http://www.springframework.org/schema/util/spring-util.xsd\">
    <!--
                          Alter configuration below as needed.
    -->
  <bean id=\"grid.cfg\" class=\"org.apache.ignite.configuration.IgniteConfiguration\">
    <property name=\"cacheConfiguration\">
      <list>
        <bean class=\"org.apache.ignite.configuration.CacheConfiguration\">
          <property name=\"name\" value=\"$1\"/>" > /tmp/igniteconfig.xml

echo "    <property name=\"cacheMode\" value=\"PARTITIONED\"/>
          <property name=\"atomicWriteOrderMode\" value=\"PRIMARY\"/>
          <property name=\"writeSynchronizationMode\" value=\"PRIMARY_SYNC\"/>" >> /tmp/igniteconfig.xml

availfreeMemory=$(cat /proc/meminfo|grep MemTotal|awk '{print $2}')
memoryOverhead=$((availfreeMemory/1024/1024/10))
availfreeMemoryinGB=$((availfreeMemory/1024/1024 - memoryOverhead))
if [[ $availfreeMemoryinGB -gt 8 ]]; then
    offheapmemoryinGB=$((availfreeMemoryinGB-8))
    echo "    <property name=\"memoryMode\" value=\"ONHEAP_TIERED\" />
      <property name=\"offHeapMaxMemory\" value=\"#{$offheapmemoryinGB * 1024L * 1024L * 1024L}\" />" >> /tmp/igniteconfig.xml
    echo "8g" > /tmp/heapsize.log
else
    echo "${availfreeMemoryinGB}g" > /tmp/heapsize.log
fi
echo "    <property name=\"evictionPolicy\">
        <bean class=\"org.apache.ignite.cache.eviction.lru.LruEvictionPolicy\">
            <property name=\"maxSize\" value=\"100000000\"/>
        </bean>
      </property>" >> /tmp/igniteconfig.xml
echo "    <property name=\"swapEnabled\" value=\"false\"/>" >> /tmp/igniteconfig.xml
echo "    <property name=\"atomicityMode\" value=\"ATOMIC\" />" >> /tmp/igniteconfig.xml
echo "    <property name=\"backups\" value=\"$2\" />" >> /tmp/igniteconfig.xml
echo "  </bean>" >> /tmp/igniteconfig.xml
echo " </list>" >> /tmp/igniteconfig.xml
echo "</property>" >> /tmp/igniteconfig.xml
echo "<property name=\"discoverySpi\">
        <bean class=\"org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi\">
          <property name=\"ipFinder\">
            <bean class=\"org.apache.ignite.spi.discovery.tcp.ipfinder.s3.TcpDiscoveryS3IpFinder\">
              <property name=\"awsCredentials\" ref=\"aws.creds\"/>
              <property name=\"bucketName\" value=\"$3\"/>
             </bean>
          </property>
        </bean>
    </property>
    <property name=\"communicationSpi\">
    <bean class=\"org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi\">
      <property name=\"slowClientQueueLimit\" value=\"1000\"/>
    </bean>
  </property>
  </bean>
<!-- AWS credentials. Provide your access key ID and secret access key. -->
<bean id="aws.creds" class="com.amazonaws.auth.BasicAWSCredentials">
  <constructor-arg value="" />
  <constructor-arg value="" />
</bean>

</beans>" >> /tmp/igniteconfig.xml

Also I tried below steps mentioned in this blog (https://www.gridgain.com/docs/8.7.6//installation-guide/manual-install-on-ec2) I am getting Spring XML error.

content in my aws-static-ip-finder.xml file is below.

<bean class="org.apache.ignite.configuration.IgniteConfiguration" >
    <!-- other properties -->
    <!-- Explicitly configure TCP discovery SPI to provide a list of nodes. -->
    <property name="discoverySpi">
        <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
            <property name="ipFinder">
                <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
                    <property name="addresses">
                        <list>
                            <value>172.31.81.211</value>
                            <value>172.31.82.21</value>
                        </list>
                    </property>
                </bean>
            </property>
        </bean>
    </property>
</bean>

class org.apache.ignite.IgniteException: Failed to instantiate Spring XML application context [springUrl=file:/home/ec2-user/aws-static-ip-finder_2.xml, err=Line 1 in XML document from URL [file:/home/ec2-user/aws-static-ip-finder_2.xml] is invalid; nested exception is org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 67; cvc-elt.1: Cannot find the declaration of element 'bean'.]
    at org.apache.ignite.internal.util.IgniteUtils.convertException(IgniteUtils.java:1052)
    at org.apache.ignite.Ignition.start(Ignition.java:350)
    at org.apache.ignite.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:300)
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to instantiate Spring XML application context [springUrl=file:/home/ec2-user/aws-static-ip-finder_2.xml, err=Line 1 in XML document from URL [file:/home/ec2-user/aws-static-ip-finder_2.xml] is invalid; nested exception is org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 67; cvc-elt.1: Cannot find the declaration of element 'bean'.]
    at org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.applicationContext(IgniteSpringHelperImpl.java:391)
    at org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.loadConfigurations(IgniteSpringHelperImpl.java:103)
    at org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.loadConfigurations(IgniteSpringHelperImpl.java:97)
    at org.apache.ignite.internal.IgnitionEx.loadConfigurations(IgnitionEx.java:750)
    at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:951)
    at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:860)
    at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:730)
    at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:699)
    at org.apache.ignite.Ignition.start(Ignition.java:347)
    ... 1 more
Caused by: org.springframework.beans.factory.xml.XmlBeanDefinitionStoreException: Line 1 in XML document from URL [file:/home/ec2-user/aws-static-ip-finder_2.xml] is invalid; nested exception is org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 67; cvc-elt.1: Cannot find the declaration of element 'bean'.
    at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadBeanDefinitions(XmlBeanDefinitionReader.java:399)
    at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:336)
    at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:304)
    at org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.applicationContext(IgniteSpringHelperImpl.java:378)
    ... 9 more
Caused by: org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 67; cvc-elt.1: Cannot find the declaration of element 'bean'.
    at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.createSAXParseException(ErrorHandlerWrapper.java:203)
    at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.error(ErrorHandlerWrapper.java:134)
    at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:396)
    at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:327)
    at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:284)
    at com.sun.org.apache.xerces.internal.impl.xs.XMLSchemaValidator.handleStartElement(XMLSchemaValidator.java:1901)
    at com.sun.org.apache.xerces.internal.impl.xs.XMLSchemaValidator.startElement(XMLSchemaValidator.java:741)
    at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.scanStartElement(XMLNSDocumentScannerImpl.java:374)
    at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl$NSContentDriver.scanRootElementHook(XMLNSDocumentScannerImpl.java:613)
    at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:3132)
    at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$PrologDriver.next(XMLDocumentScannerImpl.java:852)
    at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:602)
    at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:112)
    at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:505)
    at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:842)
    at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:771)
    at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
    at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:243)
    at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:339)
    at org.springframework.beans.factory.xml.DefaultDocumentLoader.loadDocument(DefaultDocumentLoader.java:76)
    at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadDocument(XmlBeanDefinitionReader.java:429)
    at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadBeanDefinitions(XmlBeanDefinitionReader.java:391)
    ... 12 more
Failed to start grid: Failed to instantiate Spring XML application context [springUrl=file:/home/ec2-user/aws-static-ip-finder_2.xml, err=Line 1 in XML document from URL [file:/home/ec2-user/aws-static-ip-finder_2.xml] is invalid; nested exception is org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 67; cvc-elt.1: Cannot find the declaration of element 'bean'.]

Thanks Sri

Upvotes: 0

Answers (3)

charlb

Reputation: 1249

I got it working (embedded Ignite) using TcpDiscoveryAlbIpFinder using a target group.

Added ingress security group rules to allow port 47500-47600 (discovery), 47100-47200 (communication) and 10800 (thin client).

Added AWS policy to allow load balancer access.

   {
   "Sid": "AllowLoadBalancer",
   "Effect": "Allow",
   "Action": [
     "elasticloadbalancing:Describe*",
     "elasticloadbalancing:DeregisterInstancesFromLoadBalancer",
     "elasticloadbalancing:RegisterInstancesWithLoadBalancer"
   ],
   "Resource": "arn:aws:elasticloadbalancing:*:*:loadbalancer/*"
 }

See @antkr's answer

Upvotes: 0

dmagda

Reputation: 1785

Here is detailed documentation for GridGain: https://www.gridgain.com/docs/latest/installation-guide/manual-install-on-ec2

Just replace GridGain with Ignite to complete the deployment.

Upvotes: 0

antkr

Reputation: 414

Can you please shared what you have tried to make the cluster formation work? And which version of Ignite are you on?

Basically you must create Ignite configurations with matching IP Finders in Discovery SPI, check out this documentation:

https://apacheignite-mix.readme.io/docs/amazon-aws

You can also go with simple static IP Finder that contains public IP adresses of EC2 instances:

https://apacheignite.readme.io/docs/tcpip-discovery#section-static-ip-finder

To do this, open your EC2 instances listed, check for "IPv4 Public IP" field.

Another thing you should consider is Security Group:

https://docs.aws.amazon.com/en_us/vpc/latest/userguide/VPC_SecurityGroups.html#CreatingSecurityGroups

Make sure the following TCP ports are open bidirectionally (I've included defaults):

Discovery: 47500-47600 (port range from static IP finder)

Communication: 47100-47200

Thin client connection port: 10800

REST (optional): 8080

Make sure that these connections are also available via Outbound, you must be available to receive messages from other EC2 instances.

Upvotes: 2

Building 3 node working Ignite cluster on AWS

Answers (3)

Related Questions