Ignite search query not returning results after Cluster restart with native persistence enabled

Question

We are using GridGain Community edition : 8.8.10 and have created Ignite Cluster in Kubernetes using the Apache Ignite operator. We have enabled native persistence also.

https://ignite.apache.org/docs/latest/installation/kubernetes/gke-deployment

In the development environment we shutdown our cluster during the night and bring it up during morning hours. When the cluster comes up it contains the data which was stored earlier. If we search the cache using key then it returns the result, but if we use the Query API for partial search then it is not returning results. We checked the cache size and it matches the datasource record size. Also after we search the Cache using the cache key, then that entry is available in the Query search results.

If we shut-down one of the nodes of the Ignite Cluster or client nodes. The TextSearch still works. TextSearch doesn't works only when all the nodes of the cluster are scaled-down and then scaled up using the existing disk.

Is there any configuration required to enable Query search after cold restart of the Ignite cluster ?

<bean id="ignite.cfg" class="org.apache.ignite.configuration.IgniteConfiguration">
        <property name="metricsLogFrequency" value="300000"/>
        <property name="peerClassLoadingEnabled" value="true"/>
        <property name="clientMode" value="true"/>
        <property name="sqlConfiguration">
            <bean class="org.apache.ignite.configuration.SqlConfiguration">
                <property name="sqlGlobalMemoryQuota" value="300M"/>
                <property name="sqlQueryMemoryQuota" value="30M"/>
                <property name="sqlOffloadingEnabled" value="true"/>
            </bean>
        </property>
        <property name="workDirectory" value="/gridgain/work"/>

        <property name="dataStorageConfiguration">
            <bean class="org.apache.ignite.configuration.DataStorageConfiguration">
                <!-- set the size of wal segments to 128MB -->
                <property name="walSegmentSize" value="#{128 * 1024 * 1024}"/>
                <!-- Set the page size to 8 KB -->
                <property name="pageSize" value="#{8 * 1024}"/>
                <property name="defaultDataRegionConfiguration">
                    <bean class="org.apache.ignite.configuration.DataRegionConfiguration">
                        <property name="name" value="Default_Region"/>
                        <!-- Memory region of 20 MB initial size. -->
                        <property name="initialSize" value="#{20 * 1024 * 1024}"/>
                        <!-- Memory region of 8 GB max size. -->
                        <property name="maxSize" value="#{8L * 1024 * 1024 * 1024}"/>
                        <!-- Enabling eviction for this memory region. -->
                        <property name="pageEvictionMode" value="RANDOM_2_LRU"/>
                        <property name="persistenceEnabled" value="true"/>
                        <property name="warmUpConfiguration">
                            <bean class="org.apache.ignite.configuration.LoadAllWarmUpConfiguration"/>
                        </property>
                        <!-- Increasing the buffer size to 1 GB. -->
                        <property name="checkpointPageBufferSize" value="#{1024L * 1024 * 1024}"/>
                    </bean>
                </property>
                <property name="walPath" value="/gridgain/wal"/>
                <property name="walArchivePath" value="/gridgain/wal"/>
            </bean>
        </property>
        <property name="discoverySpi">
            <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
                <property name="ipFinder">
                    <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.kubernetes.TcpDiscoveryKubernetesIpFinder">
                        <property name="namespace" value="cache"/>
                        <property name="serviceName" value="cache-service"/>
                    </bean>
                </property>
                <property name="NetworkTimeout" value="30000"/>
            </bean>
        </property>
        <property name="communicationSpi">
            <bean class="org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi">
                <property name="slowClientQueueLimit" value="2000"/>
            </bean>
        </property>
        <property name="cacheConfiguration" ref="igniteCacheDefinition"/>
    </bean>

Answer 1

Apache Ignite uses Apache Lucene (currently it's 7.4.0) for text queries under the hood. In general Lucene-based indexes leverage various implementations of org.apache.lucene.store.Directory. In Apache Ignite it's a custom one. In turn it uses RAM-based GridLuceneOutputStream. Basically it means that Ignite native persistence doesn't come into play for those kinds of indexes at the moment.

UPD: in case of configured backups for a partitioned cache it should work as a regular index. For example if you add an additional node to the baseline topology you would see a rebalance happening. Rebalance uses regular cache operations to insert entries. Lucene index would be built on the new node. On the contrary if you remove a node from a cluster you would still have a full copy of data including text indexes.