OrientDB corruption state in Nexus Repository version 3.2.0-01

Question

We are using Nexus Repository version 3.2.0-01 and started getting the following error in nexus.log. OrientDB has got corrupted. Nexus is no longer starting.

2017-03-21 13:00:36,329+0000 INFO  [FelixStartLevel] *SYSTEM org.sonatype.nexus.internal.orient.DatabaseServerImpl - OrientDB version: 2.2.13
2017-03-21 13:00:36,348+0000 INFO  [FelixStartLevel] *SYSTEM com.orientechnologies.orient.server.OServer - OrientDB Server v2.2.13 is starting up...
2017-03-21 13:00:36,355+0000 INFO  [FelixStartLevel] *SYSTEM com.orientechnologies.orient.server.OServer - Databases directory: /c9/setup/apps/nexus/sonatype-work/nexus3/db
2017-03-21 13:00:36,647+0000 WARN  [FelixStartLevel] *SYSTEM com.orientechnologies - Not enough physical memory available for DISKCACHE: 1,873MB (heap=1,161MB direct=2,048MB). Set lower Maximum Heap (-Xmx setting on JVM) and restart OrientDB. Now running with DISKCACHE=256MB
2017-03-21 13:00:36,648+0000 INFO  [FelixStartLevel] *SYSTEM com.orientechnologies - OrientDB config DISKCACHE=256MB (heap=1,161MB direct=2,048MB os=1,873MB)
2017-03-21 13:00:36,735+0000 WARN  [FelixStartLevel] *SYSTEM com.orientechnologies.orient.server.OServer - Found ORIENTDB_ROOT_PASSWORD variable, using this value as root's password
2017-03-21 13:00:36,899+0000 INFO  [FelixStartLevel] *SYSTEM com.orientechnologies.orient.server.handler.OJMXPlugin - JMX plugin installed and active: 

profilerManaged=true
2017-03-21 13:00:36,901+0000 INFO  [FelixStartLevel] *SYSTEM com.orientechnologies.orient.server.OServer - OrientDB Studio available at $ANSI{blue 

http://localhost:2480/studio/index.html}
2017-03-21 13:00:36,902+0000 INFO  [FelixStartLevel] *SYSTEM com.orientechnologies.orient.server.OServer - $ANSI{green:italic OrientDB Server is active} v2.2.13.
2017-03-21 13:00:36,902+0000 INFO  [FelixStartLevel] *SYSTEM org.sonatype.nexus.internal.orient.DatabaseServerImpl - Activated
2017-03-21 13:00:36,904+0000 INFO  [FelixStartLevel] *SYSTEM org.sonatype.nexus.extender.NexusLifecycleManager - Start UPGRADE
2017-03-21 13:00:37,632+0000 ERROR [FelixStartLevel] *SYSTEM org.sonatype.nexus.orient.DatabaseInstanceImpl - Lifecycle operation start failed
com.orientechnologies.orient.core.exception.OSecurityAccessException: User or password not valid for database: 'config'^M
        DB name="config"
        at com.orientechnologies.orient.core.metadata.security.OSecurityShared.authenticate(OSecurityShared.java:237) [na:na]
        at com.orientechnologies.orient.core.metadata.security.OSecurityShared.authenticate(OSecurityShared.java:237) [na:na]
        at com.orientechnologies.orient.core.metadata.security.OSecurityExternal.authenticate(OSecurityExternal.java:62) [na:na]
        at com.orientechnologies.orient.core.metadata.security.OSecurityProxy.authenticate(OSecurityProxy.java:107) [na:na]
        at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.open(ODatabaseDocumentTx.java:274) [na:na]
        at org.sonatype.nexus.orient.DatabaseManagerSupport.connect(DatabaseManagerSupport.java:142) [org.sonatype.nexus.orient:3.2.0.01]
        at org.sonatype.nexus.orient.DatabaseInstanceImpl.doStart(DatabaseInstanceImpl.java:56) [na:na]
        at org.sonatype.goodies.lifecycle.LifecycleSupport.start(LifecycleSupport.java:104) [org.sonatype.goodies.lifecycle:2.2.3]
        at org.sonatype.goodies.lifecycle.Lifecycles.start(Lifecycles.java:44) [org.sonatype.goodies.lifecycle:2.2.3]
        at org.sonatype.nexus.orient.DatabaseManagerSupport.createInstance(DatabaseManagerSupport.java:258) [org.sonatype.nexus.orient:3.2.0.01]
        at org.sonatype.nexus.orient.DatabaseManagerSupport.instance(DatabaseManagerSupport.java:232) [org.sonatype.nexus.orient:3.2.0.01]
        at org.sonatype.nexus.internal.orient.ConfigDatabase$ProviderImpl.get(ConfigDatabase.java:61) [org.sonatype.nexus.base:3.2.0.01]
        at org.sonatype.nexus.internal.orient.ConfigDatabase$ProviderImpl.get(ConfigDatabase.java:1) [org.sonatype.nexus.base:3.2.0.01]
        at com.google.inject.internal.ProviderInternalFactory.provision(ProviderInternalFactory.java:81) [com.google.inject:4.1.0]
        at com.google.inject.internal.BoundProviderFactory.provision(BoundProviderFactory.java:72) [com.google.inject:4.1.0]
        at com.google.inject.internal.ProviderInternalFactory$1.call(ProviderInternalFactory.java:65) [com.google.inject:4.1.0]
        at com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:115) [com.google.inject:4.1.0]
        at com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:133) [com.google.inject:4.1.0]
        at com.google.inject.internal.ProvisionListenerStackCallback.provision(ProvisionListenerStackCallback.java:68) [com.google.inject:4.1.0]
        at com.google.inject.internal.ProviderInternalFactory.circularGet(ProviderInternalFactory.java:63) [com.google.inject:4.1.0]
        at com.google.inject.internal.BoundProviderFactory.get(BoundProviderFactory.java:62) [com.google.inject:4.1.0]
        at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46) [com.google.inject:4.1.0]
        at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092) [com.google.inject:4.1.0]
        at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) [com.google.inject:4.1.0]
        at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:194) [com.google.inject:4.1.0]
        at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41) [com.google.inject:4.1.0]
        at com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1019) [com.google.inject:4.1.0]
        at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1085) [com.google.inject:4.1.0]
        at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1015) [com.google.inject:4.1.0]
         at org.eclipse.sisu.inject.LazyBeanEntry.getValue(LazyBeanEntry.java:81) [org.eclipse.sisu.inject:0.3.2]
        at org.eclipse.sisu.wire.BeanProviders.firstOf(BeanProviders.java:179) [org.eclipse.sisu.inject:0.3.2]
        at org.eclipse.sisu.wire.PlaceholderBeanProvider.lookup(PlaceholderBeanProvider.java:122) [org.eclipse.sisu.inject:0.3.2]
        at org.eclipse.sisu.wire.PlaceholderBeanProvider.get(PlaceholderBeanProvider.java:90) [org.eclipse.sisu.inject:0.3.2]
        at com.google.inject.internal.ProviderInternalFactory.provision(ProviderInternalFactory.java:81) [com.google.inject:4.1.0]
        at com.google.inject.internal.InternalFactoryToInitializableAdapter.provision(InternalFactoryToInitializableAdapter.java:53) [com.google.inject:4.1.0]
        at com.google.inject.internal.ProviderInternalFactory$1.call(ProviderInternalFactory.java:65) [com.google.inject:4.1.0]
        at com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:115) [com.google.inject:4.1.0]
        at org.eclipse.sisu.bean.BeanScheduler$Activator.onProvision(BeanScheduler.java:176) [org.eclipse.sisu.inject:0.3.2]
        at com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:126) [com.google.inject:4.1.0]
        at com.google.inject.internal.ProvisionListenerStackCallback.provision(ProvisionListenerStackCallback.java:68) [com.google.inject:4.1.0]
        at com.google.inject.internal.ProviderInternalFactory.circularGet(ProviderInternalFactory.java:63) [com.google.inject:4.1.0]
        at com.google.inject.internal.InternalFactoryToInitializableAdapter.get(InternalFactoryToInitializableAdapter.java:45) [com.google.inject:4.1.0]
        at com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1019) [com.google.inject:4.1.0]
        at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1085) [com.google.inject:4.1.0]
        at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1015) [com.google.inject:4.1.0]
        at org.sonatype.nexus.upgrade.internal.ModelVersionStore.doStart(ModelVersionStore.java:82) [org.sonatype.nexus.upgrade:3.2.0.01]
        at org.sonatype.nexus.common.stateguard.StateGuardLifecycleSupport.start(StateGuardLifecycleSupport.java:67) [org.sonatype.nexus.common:3.2.0.01]
        at org.sonatype.nexus.upgrade.internal.ModelVersionStore$$EnhancerByGuice$$b30c182f.CGLIB$start$5(<generated>) [3.2.0-01:na]
        at org.sonatype.nexus.upgrade.internal.ModelVersionStore$$EnhancerByGuice$$b30c182f$$FastClassByGuice$$94ba58c4.invoke(<generated>) [3.2.0-01:na]
        at com.google.inject.internal.cglib.proxy.$MethodProxy.invokeSuper(MethodProxy.java:228) [com.google.inject:4.1.0]
        at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:76) [com.google.inject:4.1.0]
        at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39) [org.sonatype.nexus.common:3.2.0.01]
        at org.sonatype.nexus.common.stateguard.StateGuard$TransitionImpl.run(StateGuard.java:191) [org.sonatype.nexus.common:3.2.0.01]
        at org.sonatype.nexus.common.stateguard.TransitionsInterceptor.invoke(TransitionsInterceptor.java:56) [org.sonatype.nexus.common:3.2.0.01]
        at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:77) [com.google.inject:4.1.0]
        at com.google.inject.internal.InterceptorStackCallback.intercept(InterceptorStackCallback.java:55) [com.google.inject:4.1.0]
        at org.sonatype.nexus.upgrade.internal.ModelVersionStore$$EnhancerByGuice$$b30c182f.start(<generated>) [3.2.0-01:na]
        at org.sonatype.nexus.upgrade.internal.UpgradeServiceImpl.doStart(UpgradeServiceImpl.java:80) [org.sonatype.nexus.upgrade:3.2.0.01]
        at org.sonatype.goodies.lifecycle.LifecycleSupport.start(LifecycleSupport.java:104) [org.sonatype.goodies.lifecycle:2.2.3]
        at org.sonatype.nexus.extender.NexusLifecycleManager.startComponent(NexusLifecycleManager.java:154) [org.sonatype.nexus.extender:3.2.0.01]
        at org.sonatype.nexus.extender.NexusLifecycleManager.to(NexusLifecycleManager.java:94) [org.sonatype.nexus.extender:3.2.0.01]
        at org.sonatype.nexus.extender.NexusContextListener.frameworkEvent(NexusContextListener.java:185) [org.sonatype.nexus.extender:3.2.0.01]
        at org.apache.felix.framework.Felix.setActiveStartLevel(Felix.java:1429) [org.apache.felix.framework-5.4.0.jar:na]
        at org.apache.felix.framework.FrameworkStartLevelImpl.run(FrameworkStartLevelImpl.java:308) [org.apache.felix.framework-5.4.0.jar:na]
        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]

Root cause Root cause of this was found to be an unclean shutdown of Nexus because of disk space getting full, this resulted in DB corruption.

2017-03-17 07:09:49,480+0000 WARN  [elasticsearch[A82A95AC-5D396A2B-337B8736-5017E8FC-915232E4][local_transport][T#1]] *SYSTEM org.elasticsearch.cluster.action.shard - [A82A95AC-5D396A2B-337B8736-5017E8FC-915232E4] [cab22f445c1892acef64d17ac2da6d0086023e21][0] received shard failed for [cab22f445c1892acef64d17ac2da6d0086023e21][0], node[HBjARpI7RyqqqruVqEJ8Ig], [P], v[11], s[INITIALIZING], a[id=FmxOHS7nQcGNz_93VlTpKg], unassigned_info[[reason=ALLOCATION_FAILED], at[2017-03-17T07:09:49.101Z], details[failed recovery, failure IndexShardRecoveryException[failed to recovery from gateway]; nested: EngineCreationFailureException[failed to recover from translog]; nested: EngineException[failed to recover from translog]; nested: IndexFailedEngineException[Index failed for [component#36e3dec8de528c9b41c5c6ac3b18e3f7]]; nested: NotSerializableExceptionWrapper[No space left on device]; ]], indexUUID [fLN4hdE8QPKH7Pj5RpLtmw], message [engine failure, reason [index]], failure [NotSerializableExceptionWrapper[No space left on device]]
org.elasticsearch.common.io.stream.NotSerializableExceptionWrapper: No space left on device
at sun.nio.ch.FileDispatcherImpl.write0(Native Method) [na:1.8.0_121]
at sun.nio.ch.FileDispatcherImpl.write(FileDispatcherImpl.java:60) [na:1.8.0_121]
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) [na:1.8.0_121]
at sun.nio.ch.IOUtil.write(IOUtil.java:65) [na:1.8.0_121]
at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:211) [na:1.8.0_121]
at org.elasticsearch.common.io.Channels.writeToChannel(Channels.java:211) [na:na]
at org.elasticsearch.common.bytes.PagedBytesReference.writeTo(PagedBytesReference.java:132) [na:na]
at org.elasticsearch.index.translog.BufferingTranslogWriter.add(BufferingTranslogWriter.java:57) [na:na]

No backup We have not backed up Nexus VM, therefore we wont be able to restore it back. What is the possible way out - recreating the database - with least damage. Appreciate any help.

Answer 1

Removing the file frozen.marker from the db folder solved it for us.

Answer 2

I had the same mistake:

Cannot open local storage '/nexus-data/db/config' with mode=rw

and when starting the nexus console (java -jar /opt/sonatype/nexus/lib/support/nexus-orient-console.jar) the following message appeared:

The reason was that Nexus was out of memory and had created some ".wal" files with 0 bytes in the folder "/nexus-data/db/config/". After I deleted them, everything worked again.

Many thanks also to yobert https://gist.github.com/marcelmaatkamp/123e8793e07a72a382d8d0e8d66bbd8f

Answer 3

Deleting the cache folder (/opt/sonatype-work/nexus3/cache) solved the issue for me

Answer 4

We had similar issue with Nexus OSS verison 3.2.0-01 and OrientDB corrupted state because of unclean shutdown.

In our case the corrupted storage was component located in

$install-dir/sonatype-work/nexus3/db/component

2017-04-14 15:09:20,007+0200 ERROR [qtp18266400-74] *UNKNOWN com.google.common.eventbus.EventBus.reentrant - Could not dispatch event AssetUpdatedEvent{metadata=AttachedEntityMetadata{schema=asset, document=asset#18:91856{bucket:#13:3,format:maven2,last_updated:Fri Apr 14 15:09:19 CEST 2017,attributes:[5],component:null,name:org/apache/maven/plugins/maven-jar-plugin/maven-metadata.xml,size:743,content_type:application/xml,blob_ref:default@ECF4439E-D7F184F8-578BAB14-ACFC0FB8-5589B163:3fb95745-4aed-460f-b711-b5f5b74b837c,last_accessed:Fri Apr 14 15:09:19 CEST 2017} v466}, remoteNodeId=null} to subscriber org.sonatype.nexus.repository.maven.internal.group.MavenGroupFacet$$EnhancerByGuice$$a88d7327@83eca7 method [public void org.sonatype.nexus.repository.maven.internal.group.MavenGroupFacet.on(org.sonatype.nexus.repository.storage.AssetEvent)]
com.orientechnologies.orient.core.exception.OStorageException: Error during transaction commit
        DB name="component"
        at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.makeRollback(OAbstractPaginatedStorage.java:2210) [na:na]
        at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.commit(OAbstractPaginatedStorage.java:1433) [na:na]
        at com.orientechnologies.orient.core.tx.OTransactionOptimistic.doCommit(OTransactionOptimistic.java:539) [com.orientechnologies.orientdb-core:2.2.13]
        at com.orientechnologies.orient.core.tx.OTransactionOptimistic.commit(OTransactionOptimistic.java:104) [com.orientechnologies.orientdb-core:2.2.13]
        at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.commit(ODatabaseDocumentTx.java:2756) [com.orientechnologies.orientdb-core:2.2.13]
        at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.commit(ODatabaseDocumentTx.java:2725) [com.orientechnologies.orientdb-core:2.2.13]
        at org.sonatype.nexus.repository.storage.StorageTxImpl.commit(StorageTxImpl.java:172) [org.sonatype.nexus.repository:3.2.0.01]
        at sun.reflect.GeneratedMethodAccessor105.invoke(Unknown Source) [na:na]

---

We managed to solve our problem with the following strategy

1. Scheduling Nexus tasks, particullary:

   • Rebuild Maven repository metadata.
   • Rebuild repository index.
   • Compact blob store.

2. Orient DB manipulation:

   • Rebuilding indexes.
   • Repairing database.

According Nexus support you can access Orient DB console with Nexus OSS verison 3.2.1-01. Everything is explained here. Make sure your Nexus application is properly shut down before you apply any operations on database.

Once in Orient DB console you connect to local database with

> CONNECT PLOCAL:${your_install_dir}/sonatype-work/nexus3/db/${database} admin admin

And you perform the following operations:

> REBUILD INDEX *
> REPAIR DATABASE --fix-graph
> REPAIR DATABASE --fix-links
> REPAIR DATABASE --fix-ridbags
> REPAIR DATABASE --fix-bonsai
> DISCONNECT

I am absolutely aware my answer probably won't solve your concrete problem but I still wanted to give you some directions because you don't have any backups.

Answer 5

With Nexus 3.16 there is now a task: Create Repair - Reconcile component database from blob store

super slow, but it helps recreating a accidentally deleted blobstore still on disk.

Answer 6

The database write ahead logs are corrupt.

Make a backup of $install-dir/sonatype-work/nexus3/db/component, and then remove files with extension .wal from that directory. Then try the startup again

if the issue still persist, try clearing all .wal files under the "DB" directory, will resolve the issue,

I did tried this method and it worked like charm.

regards vinay

Answer 7

Just in case that actually matters for somebody else - the above did not help me, but the json export / import one.

In my case nexus runs in the official docker image, so

docker exec -it nexus bash

Or however you access your nexus shell

cd /tmp
/opt/jdk1.8.0_141/bin/java -jar /opt/sonatype/nexus/lib/support/nexus-orient-console.jar

CONNECT plocal:/nexus-data/db/component admin admin

export database component-export
drop database
create database plocal:/nexus-data/db/component
import database component-export.json.gz

In your case the java-path could vary as also the data-path (nexus data) which we use to connect/create the database.

After that, restarting the service and your component database should be up and running again