OrientDB Embedded and Distributed error: Distributed Storage was not installed

Question

Hello thank you for reading this question. I have orient db set up and would like to add a second server and have them replicate between each other. When one goes on its own it runs fine and I have been using it for months. When enabling the hazelcast plugin, the servers start communicating, and I can see they start to talk to each other. An error happens when they try to write to each other though. This is the same issue discussed here:

https://groups.google.com/forum/#!topic/orient-database/QpZPG4y_KpU

For what its worth, I have both of these servers deployed on the same machine, each with their own embedded database. The database paths are plocal:/home/chris/dbs/db2 and plocal:/home/chris/dbs/db1

2015-10-08 08:56:14:048 INFO  [db2-orient] Saving distributed configuration file for database 'db' to: ./databases/db/distributed-config.json [OHazelcastPlugin]
2015-10-08 08:56:14:049 INFO  [db2-orient] received new status idp2-orient.idp=SYNCHRONIZING [OHazelcastPlugin]
2015-10-08 08:56:18:054 WARNING [db2-orient]->[[db1-orient]] requesting deploy of database 'db' on local server... [OHazelcastPlugin]

Then on the other server, the one that started first, I see

[OHazelcastPlugin]{db=db} [db1-orient]<-[db2-orient] error on executing distributed request 0: deploy_db
com.orientechnologies.orient.server.distributed.ODistributedException: Distributed storage was not installed for database 'db'. Implementation found: com.orientechnologies.orient.core.storage.impl.local.paginated.OLocalPaginatedStorage
at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.executeOnLocalNode(OHazelcastPlugin.java:745)
at com.orientechnologies.orient.server.hazelcast.ODistributedWorker.onMessage(ODistributedWorker.java:298)
at com.orientechnologies.orient.server.hazelcast.ODistributedWorker.run(ODistributedWorker.java:121)

I put a breakpoint on the line that throws that exception, and the storage type present at that time is indeed OLocalPaginatedStorage. My orientDB version is 2.0.15.

My distributed config. (Same on both servers)

{
"autoDeploy": true,
"hotAlignment": false,
"executionMode": "undefined",
"readQuorum": 1,
"writeQuorum": 2,
"failureAvailableNodesLessQuorum": false,
"readYourWrites": true,
"clusters": {
    "internal": {
    },
    "index": {
    },
    "*": {
        "servers" : [ "<NEW_NODE>" ]
    }
}

}

This is how I start my server. It is embedded, and started via a java application.

OServer server = OServerMain.create(true);
OPartitionedDatabasePool pool = server.startup(config.toString()).activate().getDatabasePoolFactory().get(dbPath, OUser.ADMIN, OUser.ADMIN);

The config the server uses to start up is this

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<orient-server>
<handlers>
    <handler class="com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin">
        <parameters>
            <parameter name="nodeName" value="db2-orient" />
            <parameter name="enabled" value="true" />
            <parameter name="configuration.db.default" value="${orientDBConfigs}/orientdb-default-distributed-db-config.json" />
            <parameter name="configuration.hazelcast" value="${orientDBConfigs}/orientdb-hazelcast.xml" />
            <parameter name="conflict.resolver.impl" value="com.orientechnologies.orient.server.distributed.conflict.ODefaultReplicationConflictResolver" />

            <parameter name="sharding.strategy.round-robin" value="com.orientechnologies.orient.server.hazelcast.sharding.strategy.ORoundRobinPartitioninStrategy" />
        </parameters>
    </handler>
    <handler class="com.orientechnologies.orient.server.handler.OAutomaticBackup">
        <parameters>
            <parameter name="enabled" value="false" />
            <parameter name="delay" value="4h" />
            <parameter name="target.directory" value="backup" />
            <parameter name="target.fileName" value="${DBNAME}-${DATE:yyyyMMddHHmmss}.json" />
            <parameter name="db.include" value="" />
            <parameter name="db.exclude" value="" />
        </parameters>
    </handler>
    <handler class="com.orientechnologies.orient.server.plugin.mail.OMailPlugin">
        <parameters>
            <parameter name="enabled" value="false" />
            <parameter name="profile.default.mail.smtp.host" value="localhost" />
            <parameter name="profile.default.mail.smtp.port" value="25" />
            <parameter name="profile.default.mail.smtp.auth" value="true" />
            <parameter name="profile.default.mail.smtp.starttls.enable" value="true" />
            <parameter name="profile.default.mail.smtp.user" value="" />
            <parameter name="profile.default.mail.smtp.password" value="" />
            <parameter name="profile.default.mail.date.format" value="yyyy-MM-dd HH:mm:ss" />
        </parameters>
    </handler>
    <handler class="com.orientechnologies.orient.server.handler.OServerSideScriptInterpreter">
        <parameters>
            <parameter name="enabled" value="false" />
        </parameters>
    </handler>
</handlers>
<network>
    <protocols>
        <protocol name="binary" implementation="com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary" />
    </protocols>
     <listeners>
        <listener protocol="binary" ip-address="0.0.0.0" port-range="2424-2430" />

    </listeners> 
    <cluster>
    </cluster>
</network>
<storages>
    <storage name="${dbName}" path="${dbPath}" loaded-at-startup="true" />
</storages>
<users>
    <user name="root" password="root" resources="*"/>
</users>
<properties>

    <entry name="db.pool.min" value="1" />
    <entry name="db.pool.max" value="20" />

    <entry name="cache.level1.enabled" value="false" />
    <entry name="cache.level1.size" value="1000" />
    <entry name="cache.level2.enabled" value="true" />
    <entry name="cache.level2.size" value="1000" />
    <entry name="profiler.enabled" value="true" />

    <entry name="log.console.level" value="info" />
    <entry name="log.file.level" value="fine" />

    <entry name="plugin.dynamic" value="false"/>
</properties>

Thanks again.

Answer 1

as wolf4ood pointed out, the storage type is replaced by the hazelcast plugin in the hazelcast plugin's onOpen method. Paginated storage gets switched to Distributed. This replacement does not happen if the path doesnt start with "plocal:./databases". Solution: make the path start with that. I have no idea if this is a good idea or not. That check seems to be in there for a reason, the comments in the code seem to indicate something about running on the same jvm.