CODEWITHSUNDEEP
rencoding
Pj_
Pj_

Reputation: 824

Working with log files in R

I have a .log file that has an inconsistent data format.

The data looks something like this and is stored as "Little-endian UTF-16 Unicode" text: 

2017-06-21 00:00:30.483 START THIS THING
2017-06-21 00:00:56.400 SOMETHING ELSE HAPPENS
     [XYZ 1000 T1]:1
2017-06-22 01:15:17.945 NOTHING 'D': 989
     [CASE] IN: [ID: 1010]33
     [CASE] IN: [ID: 2010]8
2017-06-21 00:00:30.483 START THIS THING
2017-06-21 00:00:56.400 SOMETHING ELSE HAPPENS
2017-06-21 00:00:30.483 START THIS THING
2017-06-21 00:00:56.400 SOMETHING ELSE HAPPENS
2017-06-21 00:00:30.483 START THIS THING
2017-06-21 00:00:56.400 SOMETHING ELSE HAPPENS

323133.....238813   76378    989899 000000000000

Now, I have several log files that follow this kind of pattern. I have tried scan() and read.table(), they both don't return data back in the format I expect it to do.

The data format I am expecting looks like this: 

Date                          String
2017-06-21 00:00:30.483       START THIS THING

But, I have these line multiple times in the log files:

 [CASE] IN: [ID: 1010]33
 [CASE] IN: [ID: 2010]8

And this,

323133.....238813   76378    989899 000000000000

What would be the best way to approach this solution? Thanks!

Upvotes: 1

Views: 1547

Answers (1)

R Yoda
R Yoda

Reputation: 8760

Just a raw sketch (ignoring the time part of your timestamp and column names) using base R without any performance optimisation (like using data.table::fread and the package lubridate):

log.data <- "2017-06-21 00:00:30.483 START THIS THING
2017-06-21 00:00:56.400 SOMETHING ELSE HAPPENS
     [XYZ 1000 T1]:1
2017-06-22 01:15:17.945 NOTHING 'D': 989
     [CASE] IN: [ID: 1010]33
     [CASE] IN: [ID: 2010]8
2017-06-21 00:00:30.483 START THIS THING
2017-06-21 00:00:56.400 SOMETHING ELSE HAPPENS
2017-06-21 00:00:30.483 START THIS THING
2017-06-21 00:00:56.400 SOMETHING ELSE HAPPENS
2017-06-21 00:00:30.483 START THIS THING
2017-06-21 00:00:56.400 SOMETHING ELSE HAPPENS

323133.....238813   76378    989899 000000000000"

log <- read.csv(text = log.data, sep = "\n", header = F)
log$timestamp <- as.Date(log[,1])

This results in:

> log
                                                 V1  timestamp
1    2017-06-21 00:00:56.400 SOMETHING ELSE HAPPENS 2017-06-21
2                                   [XYZ 1000 T1]:1       <NA>
3          2017-06-22 01:15:17.945 NOTHING 'D': 989 2017-06-22
4                           [CASE] IN: [ID: 1010]33       <NA>
5                            [CASE] IN: [ID: 2010]8       <NA>
6          2017-06-21 00:00:30.483 START THIS THING 2017-06-21
7    2017-06-21 00:00:56.400 SOMETHING ELSE HAPPENS 2017-06-21
8          2017-06-21 00:00:30.483 START THIS THING 2017-06-21
9    2017-06-21 00:00:56.400 SOMETHING ELSE HAPPENS 2017-06-21
10         2017-06-21 00:00:30.483 START THIS THING 2017-06-21
11   2017-06-21 00:00:56.400 SOMETHING ELSE HAPPENS 2017-06-21
12 323133.....238813   76378    989899 000000000000       <NA>

Update 1:

Since you found out that your log file uses the UTF-16 little-endian file encoding (checked with the file command of Linux/OSX in a terminal) you have to add the file encoding to read.csv to let R convert the file content correctly during reading:

log <- read.csv(file = "my.log", sep = "\n", header = F, fileEncoding = "UTF-16LE", encoding = "UTF-8")

Upvotes: 1

Related Questions