CODEWITHSUNDEEP
indexingelasticsearchimportbigdata
Taimoor Khan
Taimoor Khan

Reputation: 625

Uploading large 800gb json file from remote server to elasticsearch

I'm trying to upload an 800gb json file from a remote server to my local server but elasticsearch keeps getting killed. Im using this code to upload data

curl -XPOST http://localhost:9200/carrier/data/ [email protected]

Is this because a post request cant handle 800 gb or a configuration ive missed somewhere. Ive also mapped everything appropriately as smaller files upload easily.

Upvotes: 0

Views: 2253

Answers (1)

imotov
imotov

Reputation: 30163

In order to index a document, elasticsearch needs to allocate this document in memory first and then buffer it in an analyzed form again. So, you typically looking at double the size of the memory for the documents that you are indexing (it's more complex than that, but 2x is a good approximation). So, unless you have 1.6tb of memory on your machine I shouldn't try to index 800gb documents. If you have several documents in this json, you need to split them into chunk and send to elasticsearch using multiple Bulk Requests.

Upvotes: 2

Related Questions