How I can transform a field of the Jsonfile witn NiFi?

Question

Good morning

I am new in NiFi and I want modify a field in a JSON file (I am using NiFi v.1.12.0) and save it in other PATH.

This is and example of my JSON file:

    "Id": "2b2ef24a-f3ce-4249-ad92-db9a565b5b66",
    "Tipo": "AuditEvent",
    "SubTipo": "Plataforma",
    "Accion": "Audit.Middleware.EventData.HttpResponseSentEvent",
    "IDCorrelacion": "7af48a20-587d-4e60-9c3b-02cc6a074662",
    "TiempoEvento": "2020-07-30 11:45:08.315",
    "Resultado": "No informado",
    "ResultadoDesc": "No informado",
    "Origen": {
        "IDOrigen": "132403308182038429",
        "Tipo": "Backend",
        "Aplicacion": "fabric:/Omnicanalidad.Canales.Muro_v1",
        "Servicio": "fabric:/Omnicanalidad.Canales.Muro_v1/Muro",
        "Maquina": "ibsfabbe02pru",
        "IP": "ibsfabbe02pru"
    },
    "OrigenInterno": "Audit.Middleware.AuditMiddleware",
    "Agente": {
        "Rol": "Sin rol asignado",
        "IDUsuario": "1428",
        "AltIDUsuario": "20141115",
        "Localizador": "197.183.27.17",
        "PropositoUso": "No informado",
        "IDSession": "",
        "XForwardedPort": "443",
        "XForwardedFor": "162.37.0.100:30279, 162.37.0.5:10158, 172.37.0.5",
        "XForwardedHost": "ebeprate.es",
        "XForwardedProto": "https",
        "XOriginalURL": "/test/v1/Relation/ObtieneGestor?IdUser=4355625&NiciTitular=43485326",
        "XOriginalHost": "ebeprate.es",
        "Referer": null,
        "AuthenticationType": "AuthenticationTypes.Federation",
        "UserAgent": "HttpApplicationGateway",
        "Claims": "Hello World",
        "AcceptedLanguage": null
    },
    "DatosEvento": {
        "Headers": ["Content-Length: 0", "Request-Context: appId=cid-v1:d8b40be1-4838-4a94-a4f8-3ec374989b27"],
        "StatusCode": 204,
        "Body": ""
    }
}

I want modify the field TiempoEvento from date to timestamp.

In this case 2020-07-30 11:45:08.315 convert to 1596109508

So I use this procedure:

1.- I used the GetFile Processor for take the file. I configure the properties (without any problems) and everything it is ok.

2.- I used UpdateRecord Processor to modify the field. (The problems appears)

In properties I have 3 properties:

I read that I need configure a schema-registry if I want to work with any data in NiFi (I don't know if it is totaly true). In this case how I am working with a JsonFile I supposed that I need it, so I did it.

In controller service I configure JsonPathReader, JsonRecordSetWriter and AvroSchemaRegistry.

I started with AvroSchemaRegistry.

SETTING

Name: Test

PROPERTIES

Validate Field Names -> true
test-schema -> 
{
  "name": "MyFirstNiFiTest",
  "type": "record",
  "namespace": "test.example",
  "fields": [
    {
      "name": "Id",
      "type": "string"
    },
    {
      "name": "Tipo",
      "type": "string"
    },
    {
      "name": "SubTipo",
      "type": "string"
    },
    {
      "name": "Accion",
      "type": "string"
    },
    {
      "name": "IDCorrelacion",
      "type": "string"
    },
    {
      "name": "TiempoEvento",
      "type": "string"
    },
    {
      "name": "Resultado",
      "type": "string"
    },
    {
      "name": "ResultadoDesc",
      "type": "string"
    },
    {
      "name": "Origen",
      "type": {
        "name": "Origen",
        "type": "record",
        "fields": [
          {
            "name": "IDOrigen",
            "type": "string"
          },
          {
            "name": "Tipo",
            "type": "string"
          },
          {
            "name": "Aplicacion",
            "type": "string"
          },
          {
            "name": "Servicio",
            "type": "string"
          },
          {
            "name": "Maquina",
            "type": "string"
          },
          {
            "name": "IP",
            "type": "string"
          }
        ]
      }
    },
    {
      "name": "OrigenInterno",
      "type": "string"
    },
    {
      "name": "Agente",
      "type": {
        "name": "Agente",
        "type": "record",
        "fields": [
          {
            "name": "Rol",
            "type": "string"
          },
          {
            "name": "IDUsuario",
            "type": "string"
          },
          {
            "name": "AltIDUsuario",
            "type": "string"
          },
          {
            "name": "Localizador",
            "type": "string"
          },
          {
            "name": "PropositoUso",
            "type": "string"
          },
          {
            "name": "IDSession",
            "type": "string"
          },
          {
            "name": "XForwardedPort",
            "type": "string"
          },
          {
            "name": "XForwardedFor",
            "type": "string"
          },
          {
            "name": "XForwardedHost",
            "type": "string"
          },
          {
            "name": "XForwardedProto",
            "type": "string"
          },
          {
            "name": "XOriginalURL",
            "type": "string"
          },
          {
            "name": "XOriginalHost",
            "type": "string"
          },
          {
            "name": "Referer",
            "type": [
              "string",
              "null"
            ]
          },
          {
            "name": "AuthenticationType",
            "type": [
              "string",
              "null"
            ]
          },
          {
            "name": "UserAgent",
            "type": "string"
          },
          {
            "name": "Claims",
            "type": "string"
          },
          {
            "name": "AcceptedLanguage",
            "type": [
              "string",
              "null"
            ]
          }
        ]
      }
    },
    {
      "name": "DatosEvento",
      "type": {
        "name": "DatosEvento",
        "type": "record",
        "fields": [
          {
            "name": "Name",
            "type": "string"
          },
          {
            "name": "Category",
            "type": "string"
          },
          {
            "name": "EventType",
            "type": "int"
          },
          {
            "name": "Id",
            "type": "int"
          },
          {
            "name": "ApiName",
            "type": "string"
          },
          {
            "name": "Token",
            "type": "string"
          },
          {
            "name": "ApiScopes",
            "type": {
              "type": "array",
              "items": "string"
            }
          },
          {
            "name": "TokenScopes",
            "type": {
              "type": "array",
              "items": "string"
            }
          },
          {
            "name": "Message",
            "type": "string"
          },
          {
            "name": "ActivityId",
            "type": "string"
          },
          {
            "name": "TimeStamp",
            "type": "int",
            "logicalType": "date"
          },
          {
            "name": "ProcessId",
            "type": "int"
          },
          {
            "name": "LocalIpAddress",
            "type": "string"
          },
          {
            "name": "RemoteIpAddress",
            "type": "string"
          }
        ]
      }
    }
  ]
}

I converted the JSON file to avroSchema

I enable it and everything it is OK.

Then I configure the JsonRecordSetWrite:

SETTING

Name: TestRecordSetWriter

PROPERTIES

I enable it and everything it is OK.

and then I configue de JsonPathReader

SETTING

Name: TestPathReader

PROPERTIES

And in this point I have and alert that said:

'JSON paths' is invalid bacause No JSON Paths were specified

and I can't enable this controller services, and I don't know what am I missing?

I don't know if there are another way to do it easier. I don't know if I am totally wrong. So I need some help.

Thank you

Answer 1

I found the answer. I has a bad configuration in JsonPathreader, because I had not configured the records of the schema in the properties.