CODEWITHSUNDEEP
csvimportarangodbarangoimport
Hendrik
Hendrik

Reputation: 93

Error while importing very large dataset to arangoDB

I'm trying to import a huge dataset to ArangoDB via arangoimp. It's about 55 million edges. I alreasy increased the size of the wal.logfiles ( --wal.logfile-size to 1024k from 32k). This solved the last error. But now i get the following error:

WARNING {collector} got unexpected error in MMFilesCollectorThread::collect: no journal
ERROR cannot create datafile '/usr/local/var/lib/arangodb3/databases/database-1/collection-2088918365-385765492/temp-2153337069.db': Too many open files

Importstatement was: arangoimp --file links_de.csv --type csv --collection links

Is there a way to let arangoimp work more iterativ like doing chunks or something? I would be quite complicated to split the CSV in some parts because of its size...

thanks a lot!

Upvotes: 2

Views: 470

Answers (1)

Hendrik
Hendrik

Reputation: 93

i finally solved it: Too many open files was the hint. You need to raise the limit for open file handles. So before starting arango on unix / mac os run ulimit -n 16384 or higher to raise the limit for this shell session. Afterwards the import still takes several hours but worked for me. You can also increase the journal size of the collection in arango to reduce the number of needed file handles.

Also see there: ArangoDB Too many open files

Upvotes: 1

Related Questions