Dynamically create cache in Ignite server node and link it to PostgreSql table

Question

I have one Ignite server node and one thick java client. I'm able to create new caches in the server node by calling the Client node (I have exposed REST APIs).

I have a PostgreSQL DB where schema separated multitenancy is implemented, meaning:

Here Table1 in Schema1 and Schema2 is same by the properties. As it belongs to different schemas it will hold values for different tenants. . Here in PostgreSQL dynamically new schemas and tables could be created when new tenant is part of the project. .

I was able to create a configuration where Table1 from all the existing schemas are loaded to Ignite Server node and values are loaded into tables from DB. Through the client, I am able to get values as well.

Problem: I'm not able to create a cache (for the newly created table in the new schema) from the Client node and link it to PostgreSQL.

I couldn't get a straightforward solution to my issue in Ignite developer documents. Can anyone help me what should be the proper way to tackle this, and also link to an example where dynamically cache is created and linked to DB.

I get the following exception, I know I haven't attached code, if you need code to understand the problem better then I will push the code to github and link here in the question.

Dynamic cache creation: https://www.gridgain.com/docs/latest/developers-guide/key-value-api/basic-cache-operations Here using ccfg.setCacheStoreFactory(cacheStoreFactory) I have linked DB.

NOTE: If I run the client in server mode "cfg.setClientMode(false)" then I'm able to successfully create a new cache and link to DB. Does that mean new caches can be created only in Servers.?

    2021-07-23 00:03:39.574 ERROR 8036 --- [nio-8081-exec-1] o.a.c.c.C.[.[.[/].[dispatcherServlet]    : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is javax.cache.CacheException: class org.apache.ignite.IgniteCheckedException: Failed to complete exchange process.] with root cause

org.apache.ignite.IgniteCheckedException: Failed to complete exchange process.
    at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.createExchangeException(GridDhtPartitionsExchangeFuture.java:3372) ~[ignite-core-8.7.9.jar:8.7.9]
    at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.sendExchangeFailureMessage(GridDhtPartitionsExchangeFuture.java:3400) ~[ignite-core-8.7.9.jar:8.7.9]
    at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.finishExchangeOnCoordinator(GridDhtPartitionsExchangeFuture.java:3496) ~[ignite-core-8.7.9.jar:8.7.9]
    at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onAllReceived(GridDhtPartitionsExchangeFuture.java:3477) ~[ignite-core-8.7.9.jar:8.7.9]
    at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.distributedExchange(GridDhtPartitionsExchangeFuture.java:1608) ~[ignite-core-8.7.9.jar:8.7.9]
    at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:929) ~[ignite-core-8.7.9.jar:8.7.9]
    at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body0(GridCachePartitionExchangeManager.java:3251) ~[ignite-core-8.7.9.jar:8.7.9]
    at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:3097) ~[ignite-core-8.7.9.jar:8.7.9]
    at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:119) ~[ignite-core-8.7.9.jar:8.7.9]
    at java.lang.Thread.run(Thread.java:748) ~[na:na]
    Suppressed: org.apache.ignite.IgniteCheckedException: Failed to initialize exchange locally [locNodeId=2753929a-755e-4243-b8d0-693e42b1a078]
        at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onCacheChangeRequest(GridDhtPartitionsExchangeFuture.java:1345) ~[ignite-core-8.7.9.jar:8.7.9]
        at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:855) ~[ignite-core-8.7.9.jar:8.7.9]
        ... 4 common frames omitted
    Caused by: org.apache.ignite.IgniteException: Failed to enrich cache configuration [cacheName=Users_cuddle_nand]
        at org.apache.ignite.internal.processors.cache.CacheConfigurationEnricher.enrich(CacheConfigurationEnricher.java:128)
        at org.apache.ignite.internal.processors.cache.CacheConfigurationEnricher.enrich(CacheConfigurationEnricher.java:61)
        at org.apache.ignite.internal.processors.cache.GridCacheProcessor.prepareCacheContext(GridCacheProcessor.java:1881)
        at org.apache.ignite.internal.processors.cache.GridCacheProcessor.prepareCacheStart(GridCacheProcessor.java:1849)
        at org.apache.ignite.internal.processors.cache.GridCacheProcessor.lambda$prepareStartCaches$55a0e703$1(GridCacheProcessor.java:1724)
        at org.apache.ignite.internal.processors.cache.GridCacheProcessor.lambda$prepareStartCachesIfPossible$14(GridCacheProcessor.java:1694)
        at org.apache.ignite.internal.processors.cache.GridCacheProcessor.prepareStartCaches(GridCacheProcessor.java:1721)
        at org.apache.ignite.internal.processors.cache.GridCacheProcessor.prepareStartCachesIfPossible(GridCacheProcessor.java:1692)
        at org.apache.ignite.internal.processors.cache.CacheAffinitySharedManager.processCacheStartRequests(CacheAffinitySharedManager.java:971)
        at org.apache.ignite.internal.processors.cache.CacheAffinitySharedManager.onCacheChangeRequest(CacheAffinitySharedManager.java:857)
        at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onCacheChangeRequest(GridDhtPartitionsExchangeFuture.java:1334)
        ... 5 common frames omitted
    Caused by: org.apache.ignite.IgniteException: Failed to deserialize field storeFactory
        at org.apache.ignite.internal.processors.cache.CacheConfigurationEnricher.deserialize(CacheConfigurationEnricher.java:153)
        at org.apache.ignite.internal.processors.cache.CacheConfigurationEnricher.enrich(CacheConfigurationEnricher.java:121)
        ... 15 common frames omitted
    Caused by: org.apache.ignite.IgniteCheckedException: Failed to deserialize object with given class loader: sun.misc.Launcher$AppClassLoader@18b4aac2
        at org.apache.ignite.marshaller.jdk.JdkMarshaller.unmarshal0(JdkMarshaller.java:148)
        at org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:92)
        at org.apache.ignite.marshaller.jdk.JdkMarshaller.unmarshal0(JdkMarshaller.java:162)
        at org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:80)
        at org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10478)
        at org.apache.ignite.internal.processors.cache.CacheConfigurationEnricher.deserialize(CacheConfigurationEnricher.java:150)
        ... 16 common frames omitted
    Caused by: java.lang.ClassCastException: cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.ignite.cache.store.jdbc.CacheJdbcPojoStoreFactory.dataSrcFactory of type javax.cache.configuration.Factory in instance of org.apache.ignite.cache.store.jdbc.CacheJdbcPojoStoreFactory
        at java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2301)
        at java.io.ObjectStreamClass.setObjFieldValues(ObjectStreamClass.java:1431)
        at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2411)
        at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2329)
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2187)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1667)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:503)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:461)
        at org.apache.ignite.marshaller.jdk.JdkMarshaller.unmarshal0(JdkMarshaller.java:140)
        ... 21 common frames omitted

Answer 1

My guess is that you cannot easily create new caches in Postgres via Ignite because all caches must be provided in the config and they must match the base table in Postgres as well.

Some time ago I integrate Ignite with Postgres and I was able to use the following configuration:

    <?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xmlns:util="http://www.springframework.org/schema/util"
       xsi:schemaLocation="
     http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-3.1.xsd
     http://www.springframework.org/schema/util  http://www.springframework.org/schema/util/spring-util-3.1.xsd">

    <bean id="postgre_con" class="org.springframework.jdbc.datasource.DriverManagerDataSource">
        <property name="driverClassName" value="org.postgresql.Driver" />
        <property name="url" value="jdbc:postgresql://localhost:5432/postgres" />
        <property name="username" value="postgres" />
        <property name="password" value="qwerty" />
        <property name="connectionProperties">
            <props>
                <prop key="socketTimeout">10</prop>
            </props>
        </property>
    </bean>

    <bean id="grid.cfg"
          class="org.apache.ignite.configuration.IgniteConfiguration">

        <property name="failureDetectionTimeout" value="60000"/>
        <property name="clientFailureDetectionTimeout" value="60000"/>
        <property name="cacheConfiguration">
            <list>
                <bean class="org.apache.ignite.configuration.CacheConfiguration">
                    <property name="name" value="Person"/>
                    <property name="cacheMode" value="PARTITIONED"/>
                    <property name="atomicityMode" value="TRANSACTIONAL"/>
                    <property name="sqlSchema" value="PUBLIC"/>

                    <property name="keyConfiguration">
                        <list>
                            <bean class="org.apache.ignite.cache.CacheKeyConfiguration">
                                <constructor-arg name="keyCls" value="org.gridgain.essilor.model.Person"/>
                            </bean>
                        </list>
                    </property>

                    <property name="cacheStoreFactory">
                        <bean class="org.apache.ignite.cache.store.jdbc.CacheJdbcPojoStoreFactory">
                            <property name="dataSourceBean" value="postgre_con"/>
                            <property name="dialect">
                                <bean class="org.apache.ignite.cache.store.jdbc.dialect.BasicJdbcDialect"/>
                            </property>

                            <property name="types">
                                <list>
                                    <bean class="org.apache.ignite.cache.store.jdbc.JdbcType">
                                        <property name="cacheName" value="Person"/>
                                        <property name="keyType" value="java.lang.Integer"/>
                                        <property name="valueType" value="org.gridgain.essilor.model.Person"/>
                                        <property name="databaseSchema" value="PUBLIC"/>
                                        <property name="databaseTable" value="Person"/>

                                        <property name="keyFields">
                                            <list>
                                                <bean class="org.apache.ignite.cache.store.jdbc.JdbcTypeField">
                                                    <constructor-arg>
                                                        <util:constant static-field="java.sql.Types.INTEGER"/>
                                                    </constructor-arg>
                                                    <constructor-arg value="person_id"/>
                                                    <constructor-arg value="int"/>
                                                    <constructor-arg value="person_id"/>
                                                </bean>
                                            </list>
                                        </property>

                                        <property name="valueFields">
                                            <list>
                                                <bean class="org.apache.ignite.cache.store.jdbc.JdbcTypeField">
                                                    <constructor-arg>
                                                        <util:constant static-field="java.sql.Types.INTEGER"/>
                                                    </constructor-arg>
                                                    <constructor-arg value="company_id"/>
                                                    <constructor-arg value="int"/>
                                                    <constructor-arg value="company_id"/>
                                                </bean>
                                                <bean class="org.apache.ignite.cache.store.jdbc.JdbcTypeField">
                                                    <constructor-arg>
                                                        <util:constant static-field="java.sql.Types.VARCHAR"/>
                                                    </constructor-arg>
                                                    <constructor-arg value="person_name"/>
                                                    <constructor-arg value="java.lang.String"/>
                                                    <constructor-arg value="person_name"/>
                                                </bean>
                                            </list>
                                        </property>
                                    </bean>
                                </list>
                            </property>
                        </bean>
                    </property>

                    <property name="readThrough" value="true"/>
                    <property name="writeThrough" value="true"/>

                    <!-- Configure type metadata to enable queries. -->
                    <property name="queryEntities">
                        <list>
                            <bean class="org.apache.ignite.cache.QueryEntity">
                                <property name="keyType" value="java.lang.Integer"/>
                                <property name="valueType" value="org.gridgain.essilor.model.Person"/>
                                <property name="tableName" value="Person"/>
                                <property name="keyFieldName" value="person_id"/>

                                <property name="fields">
                                    <map>
                                        <entry key="person_id" value="java.lang.Integer"/>
                                        <entry key="company_id" value="java.lang.Integer"/>
                                        <entry key="person_name" value="java.lang.String"/>
                                    </map>
                                </property>
                            </bean>
                        </list>
                    </property>
                </bean>
                <bean class="org.apache.ignite.configuration.CacheConfiguration">
                    <property name="name" value="Company"/>
                    <property name="cacheMode" value="PARTITIONED"/>
                    <property name="atomicityMode" value="TRANSACTIONAL"/>
                    <property name="sqlSchema" value="PUBLIC"/>

                    <property name="keyConfiguration">
                        <list>
                            <bean class="org.apache.ignite.cache.CacheKeyConfiguration">
                                <constructor-arg name="keyCls" value="org.gridgain.essilor.model.Company"/>
                            </bean>
                        </list>
                    </property>

                    <property name="cacheStoreFactory">
                        <bean class="org.apache.ignite.cache.store.jdbc.CacheJdbcPojoStoreFactory">
                            <property name="dataSourceBean" value="postgre_con"/>
                            <property name="dialect">
                                <bean class="org.apache.ignite.cache.store.jdbc.dialect.BasicJdbcDialect"/>
                            </property>

                            <property name="types">
                                <list>
                                    <bean class="org.apache.ignite.cache.store.jdbc.JdbcType">
                                        <property name="cacheName" value="Company"/>
                                        <property name="keyType" value="java.lang.Integer"/>
                                        <property name="valueType" value="org.gridgain.essilor.model.Company"/>
                                        <property name="databaseSchema" value="PUBLIC"/>
                                        <property name="databaseTable" value="Company"/>

                                        <property name="keyFields">
                                            <list>
                                                <bean class="org.apache.ignite.cache.store.jdbc.JdbcTypeField">
                                                    <constructor-arg>
                                                        <util:constant static-field="java.sql.Types.INTEGER"/>
                                                    </constructor-arg>
                                                    <constructor-arg value="company_id"/>
                                                    <constructor-arg value="int"/>
                                                    <constructor-arg value="company_id"/>
                                                </bean>
                                            </list>
                                        </property>

                                        <property name="valueFields">
                                            <list>
                                                <bean class="org.apache.ignite.cache.store.jdbc.JdbcTypeField">
                                                    <constructor-arg>
                                                        <util:constant static-field="java.sql.Types.VARCHAR"/>
                                                    </constructor-arg>
                                                    <constructor-arg value="company_name"/>
                                                    <constructor-arg value="java.lang.String"/>
                                                    <constructor-arg value="company_name"/>
                                                </bean>
                                            </list>
                                        </property>
                                    </bean>
                                </list>
                            </property>
                        </bean>
                    </property>

                    <property name="readThrough" value="true"/>
                    <property name="writeThrough" value="true"/>

                    <property name="queryEntities">
                        <list>
                            <bean class="org.apache.ignite.cache.QueryEntity">
                                <property name="keyType" value="java.lang.Integer"/>
                                <property name="valueType" value="org.gridgain.essilor.model.Company"/>
                                <property name="tableName" value="Company"/>
                                <property name="keyFieldName" value="company_id"/>

                                <property name="fields">
                                    <map>
                                        <entry key="company_id" value="java.lang.Integer"/>
                                        <entry key="company_name" value="java.lang.String"/>
                                    </map>
                                </property>
                            </bean>
                        </list>
                    </property>
                </bean>
            </list>
        </property>

        <property name="dataStorageConfiguration">
            <bean class="org.apache.ignite.configuration.DataStorageConfiguration">
                <property name="defaultDataRegionConfiguration">
                    <bean class="org.apache.ignite.configuration.DataRegionConfiguration">
                        <property name="persistenceEnabled" value="true"/>
                        <property name="name" value="Default_Region"/>
                        <property name="initialSize" value="#{500L * 1024 * 1024}"/>
                        <property name="maxSize" value="#{1L * 1024 * 1024 * 1024}"/>
                    </bean>
                </property>


                <property name="walMode" value="LOG_ONLY"/>
                <property name="writeThrottlingEnabled" value="true"/>
                <property name="pageSize" value="4096"/>
            </bean>
        </property>

        <property name="discoverySpi">
            <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
                <property name="ipFinder">
                    <bean
                        class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
                        <property name="addresses">
                            <list>
                                <value>127.0.0.1:47500..47501</value>
                            </list>
                        </property>
                    </bean>
                </property>
            </bean>
        </property>

        <property name="communicationSpi">
            <bean
                class="org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi">
                <property name="localPort" value="48100"/>
                <property name="localPortRange" value="10"/>
                <property name="socketWriteTimeout" value="300000"/>
            </bean>
        </property>
    </bean>
</beans>

My guess is that when you try to change the config in your client, then on the server side it cannot be merged with its config. The only way that seems to work here is to apply the new configuration to the entire cluster, discarding all cached data (you should just load it again from Postgres after the upgrade).