Mati Yunhyuk Lee
Mati Yunhyuk Lee

Reputation: 1

The way to save Object field type to index by using ingest pipeline

I want to save the Object field type

when I made a index without any option.and I send it in Dashboard.

POST /test-index/_doc
{
  "name": "name",
  "birth-day": "1995-10-04",
  "age": 28,
  "occupation": "Software Engineer"
}

So, type is made like this, through dynamic mapping.

"mappings": {
      "properties": {
        "age": {
          "type": "long"
        },
        "birth-day": {
          "type": "date"
        },
        "name": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "occupation": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        }
      }
    }

I wanted to iterate through the fields, check if the type differs from the existing index and if the types do not match, add it as a subfield. Now, I want to save this data in Opensearch Index by using Ingest pipeline. ↓

POST /test-index/_doc
{
  "name": "name",
  "birth-day": "1995-10-04",
  "age": 28,
  "occupation": {
    "name": "Software Engineer",
    "jobcode": "ED26P",
  }
}

It is hard to build that function by using "convert" in "processor" for me. So, I tried "painless script" at "Ingest Pipeline".

PUT /_ingest/pipeline/test-pipeline
{
  "description": "field add by type",
  "processors": [
    {
      "script": {
        "lang": "painless",
        "source": """
          // iterate field Key
          for (def field : ctx.keySet()) {
            def original_value = ctx[field];
            
            // add all type value depends on field type
            if (original_value != null){
              if (original_value instanceof String) {
                ctx[field] = [
                  'value': original_value,
                  'as_double': null,
                  'as_int': null,
                  'as_bool': null,
                  'as_string': original_value
                ];
                try {
                  ctx[field]['as_double'] = Double.parseDouble(original_value);
                } catch (Exception e) {}
                try {
                  ctx[field]['as_int'] = Integer.parseInt(original_value);
                } catch (Exception e) {}
                try {
                  ctx[field]['as_bool'] = Boolean.parseBoolean(original_value);
                } catch (Exception e) {}
              } else if (original_value instanceof Integer) {
                ctx[field] = [
                  'value': original_value,
                  'as_double': original_value.doubleValue(),
                  'as_int': original_value,
                  'as_bool': null,
                  'as_string': original_value.toString()
                ];
              } else if (original_value instanceof Double) {
                ctx[field] = [
                  'value': original_value,
                  'as_double': original_value,
                  'as_int': original_value.intValue(),
                  'as_bool': null,
                  'as_string': original_value.toString()
                ];
              } else if (original_value instanceof Boolean) {
                ctx[field] = [
                  'value': original_value,
                  'as_double': null,
                  'as_int': original_value ? 1 : 0,
                  'as_bool': original_value,
                  'as_string': original_value.toString()
                ];
              } else if (original_value instanceof ZonedDateTime) {
                ctx[field] = [
                  'value': original_value.toString(),
                  'as_double': null,
                  'as_int': null,
                  'as_bool': null,
                  'as_string': original_value.toString()
                ];
              } else if (original_value instanceof Map) {
                // *******************************
                // ***** this is the problem *****
                // *******************************

                StringBuilder sb = new StringBuilder();
                sb.append("{");
                for (Map.Entry entry : original_value.entrySet()) {
                    sb.append(entry.getKey()).append(": ").append(entry.getValue()).append(", ");
                }
                sb.append("}");
                
                String mapAsString = sb.toString();
                ctx[field] = [
                  'value': mapAsString.toString(),
                  'as_double': null,
                  'as_int': null,
                  'as_bool': null,
                  'as_string': mapAsString.toString()
                ];
              }else {
                // the others to string
                ctx[field] = [
                  'value': original_value.toString(),
                  'as_double': null,
                  'as_int': null,
                  'as_bool': null,
                  'as_string': original_value.toString()
                ];
              }
            }
          }
        """
      }
    }
  ]
}

I want type cast from Object("{key : value}") to String. But, I couldn't. any other solution for me? thank you.

Upvotes: 0

Views: 81

Answers (0)

Related Questions