CODEWITHSUNDEEP
apache-spark-sqlignite
zfy
zfy

Reputation: 4597

How to configure cache for ignite-spark-dataframe?

I managed to save and load spark dataframe from ignite by the example here: https://apacheignite-fs.readme.io/docs/ignite-data-frame

By following the code example when the cache is created in ignite it automatically has a name like "SQL_PUBLIC_name_of_table_in_spark".

One the other hand if I want to change some cache configuration I need to specify the same cache name in xml or code before creating ignite cache. Because cache configuration can not be changed after cache is created. See following code. 

<bean class="org.apache.ignite.configuration.IgniteConfiguration">
    <property name="cacheConfiguration">
        <bean class="org.apache.ignite.configuration.CacheConfiguration">
            <!-- Set a cache name. -->
            <property name="name" value="SQL_PUBLIC_name_of_table_in_spark"/>
            <!-- Set cache mode. -->
            <property name="cacheMode" value="PARTITIONED"/>
        </bean>
    </property>
</bean>

Then one of them will be reject by "cache already exists" The result is I could't change any cache configuration by xml/code.

Is this expected? And how can I change the cache configuration in this case?

Upvotes: 0

Views: 282

Answers (1)

Stanislav Lukyanov
Stanislav Lukyanov

Reputation: 2157

The doc page you've link contains a code piece that creates an SQL table:

CREATE TABLE person ( 
    id LONG,  
    name VARCHAR,  
    city_id LONG,  
    PRIMARY KEY (id, city_id) 
) WITH "backups=1, affinityKey=city_id”;

This SQL command is what actually creates the cache. You can change this command to change the parameters of the cache that will be created. Refer to the CREATE TABLE doc.

In particular, the parameter that gives the most flexibility is WITH template=mytemplate. It lets you create a cache from a pre-existing template configuration. To register a template you can specify it in your cacheConfiguration with a name ending with asterisk, like

<bean class="org.apache.ignite.configuration.IgniteConfiguration">
    <property name="cacheConfiguration">
        <bean class="org.apache.ignite.configuration.CacheConfiguration">
            <property name="name" value="mytemplate*"/>
            <!-- your parameters. -->
        </bean>
    </property>
</bean>

You can also specify the WITH parameters for CREATE TABLE in the OPTION_CREATE_TABLE_PARAMETERS setting if the table is being created automatically by Spark.

Upvotes: 1

Related Questions