CODEWITHSUNDEEP
amazon-web-servicesaws-glue
confusedpunter
confusedpunter

Reputation: 157

Use a custom classifier in Glue for multi line records

I have some files in the following format

AB1|STUFF|1234|

AB2|SF|STUFF|

AB1|STUFF|45670|

AB2|AF|STUFF|

Each bit of data is delimited by '|' and a record is made up of the data in lines AB1 and AB2.

I would like to use a custom grok classifier in Glue something like the following:

?<LINE1>(?:AB1)?|%{WORD:ignore1}|%{NUMBER:id}\|\n%{WORD:LINE2}|%{WORD:make}|%{WORD:stuff2}\|

That is a multi line grok expression to extract the data from a multi line record as shown above. I am unsure how the classifiers in Glue work any comments or advice would be very helpful.

Upvotes: 0

Views: 947

Answers (1)

Robert Kossendey
Robert Kossendey

Reputation: 7028

According to the Glue Documentation:

Grok patterns can process only one line at a time. Multiple-line patterns are not supported. Also, line breaks within a pattern are not supported.

I am not sure what the actual question is, if you need general guidance on how to create your own classifier, I would advise you to read this and this.

Upvotes: 1

Related Questions