Indexing bulk data with Elastic Search Javascript API

Question

Background

In my node application I am trying to index the returned values from an SQL table. I have been reading the docs and I think I should be doing this with the bulk function. I am trying to use the nested data type and I find the documentation to be a little unclear.

There are three functions that come into play here.

1. index
2. putMapping
3. bulk

When I try and add one index it works fine. But when I try and add the bulk indexes it says the index does not exist. This leads me to assume that I need to create the index and then add the nested mappings with putMapping. Either way I try I get an error. Most recently I am getting the error,

"request body is required"},"status":400}',

Problem

I am getting this error when I try and create the index which I am trying to do without the body key. I was thinking I needing to create the index with mappings first so I could than enter bulk data. I think my problem is that I am not using the correct functions.

Example

index

exports.createIndex = () => {
    esClient.index({
        index: "products",
        type: "product",
        refresh: "true"
    }, (error, response) => {
        if(error) {
            console.log('Put Index Error ', error);
        } else {
            this.createMapping();
            console.log('Put Index Response ', response);
        }
    });
};

putMapping

exports.createMapping = () => {
    esClient.indices.putMapping({
        index: "products",
        type: "product",
        body: {
            mappings: {
                product: {
                    properties: {
                        variantId: { type: "text" },
                        productId: { type: "text" },
                        keyStrengths: { type: "text" },
                        something: {
                            type: "nested",
                            properties: {
                                type: { type: "text" },
                                label: { type: "text" },
                                items: {
                                    type: "nested",
                                    properties: {
                                        value: {
                                            type: "text"
                                        },
                                        characteristic: {
                                            type: "text"
                                        }
                                    }
                                }
                            }
                        }
                    }
                }
            }
        }
    }, (error, response) => {
        if(error) {
            console.log('Put Index Error ', error);
        } else {
            console.log('Put Index Response ', response);
        }
    });
};

bulk

   esClient.bulk({})

Question

Please tell me the proper way to create a new index, and then bulk insert the data returned from a database. I am not clear if I need to use all three of those functions or where I am going wrong with this.

My goal is to bulk insert an array of objects that have an array of objects inside of them.

Answer 1

You're not using the right functions. esClient.index() is meant to index a document, not to create an index. The correct thing to do in your case is to call esClient.indices.create() in order to create your index with the appropriate mapping and then you can call esClient.bulk(). It goes like this:

exports.createIndex = () => {
    esClient.indices.create({
        index: "products",
        body: {
            mappings: {
                product: {
                    properties: {
                        variantId: { type: "text" },
                        productId: { type: "text" },
                        keyStrengths: { type: "text" },
                        something: {
                            type: "nested",
                            properties: {
                                type: { type: "text" },
                                label: { type: "text" },
                                items: {
                                    type: "nested",
                                    properties: {
                                        value: {
                                            type: "text"
                                        },
                                        characteristic: {
                                            type: "text"
                                        }
                                    }
                                }
                            }
                        }
                    }
                }
            }
        }
    }, (error, response) => {
        if(error) {
            console.log('Create Index Error ', error);
        } else {
            console.log('Create Index Response ', response);
        }
    });
};

Once that's done, you can call esClient.bulk() with your data extracted from the database.