CODEWITHSUNDEEP
shellsedgrep
paolo9785
paolo9785

Reputation: 114

How to optimize grep

I'm trying to optimize a concatenation of greps/sed since it's becaming very slow for the use i need.

Here's my command:

LANG=C tail -50 /var/log/user.log | LANG=C grep -E 'LivingLight.*STATE =' | tail -1 | cut -f2- -d= | sed -e 's/^\s*//' -e '/^$/d' | grep -E -o '([OFF|ON]*)'

Wanted result: latest ON or OFF reported state

here some source lines:

2019-11-11T17:04:12.757889+01:00 KitchenFan-7873 ESP-RSL: STATE = {"Time":"2019-11-11T17:04:12","Uptime":"4T20:12:27","Heap":22,"SleepMode":"Dynamic","Sleep":50,"LoadAvg":19,"POWER":"OFF","Wifi":{"AP":1,"SSId":"Home9785","BSSId":"10:13:31:C8:D9:D3","Channel":6,"RSSI":80,"LinkCount":61,"Downtime":"0T00:19:04"}}
2019-11-11T17:04:13.680766+01:00 4CHdoorManager ESP-RSL: STATE = {"Time":"2019-11-11T17:04:13","Uptime":"0T01:46:13","Heap":22,"SleepMode":"Dynamic","Sleep":50,"LoadAvg":19,"POWER1":"OFF","POWER2":"OFF","POWER3":"OFF","POWER4":"OFF","Wifi":{"AP":1,"SSId":"Home9785","BSSId":"10:13:31:C8:D9:D3","Channel":6,"RSSI":100,"LinkCount":1,"Downtime":"0T00:00:04"}}
2019-11-11T17:04:13.681045+01:00 4CHdoorManager ESP-RSL: SENSOR = {"Time":"2019-11-11T17:04:13","AM2301":{"Temperature":null,"Humidity":null},"TempUnit":"C"}
2019-11-11T17:04:15.124453+01:00 LivingLight ESP-RSL: STATE = {"Time":"2019-11-11T17:04:14","Uptime":"4T20:12:26","Heap":21,"SleepMode":"Dynamic","Sleep":50,"LoadAvg":19,"POWER":"OFF","Wifi":{"AP":1,"SSId":"Home9785","BSSId":"10:13:31:C8:D9:D3","Channel":6,"RSSI":100,"LinkCount":26,"Downtime":"0T00:17:20"}}
2019-11-11T17:04:18.503329+01:00 EmergencyLight ESP-RSL: STATE = {"Time":"2019-11-11T17:04:18","Uptime":"4T20:11:20","Heap":21,"SleepMode":"Dynamic","Sleep":50,"LoadAvg":19,"POWER":"ON","Wifi":{"AP":1,"SSId":"Home9785","BSSId":"10:13:31:C8:D9:D3","Channel":6,"RSSI":80,"LinkCount":26,"Downtime":"0T00:17:25"}}
2019-11-11T17:04:18.503783+01:00 EmergencyLight ESP-RSL: SENSOR = {"Time":"2019-11-11T17:04:18","AM2301":{"Temperature":19.9,"Humidity":48.7},"TempUnit":"C"}
2019-11-11T17:04:21.187349+01:00 BedroomBlind ESP-RSL: STATE = {"Time":"2019-11-11T17:04:21","Uptime":"4T20:11:34","Heap":21,"SleepMode":"Dynamic","Sleep":50,"LoadAvg":19,"POWER1":"OFF","POWER2":"OFF","Wifi":{"AP":1,"SSId":"Home9785","BSSId":"10:13:31:C8:D9:D3","Channel":6,"RSSI":66,"LinkCount":33,"Downtime":"0T00:17:50"}}
2019-11-11T17:04:36.187722+01:00 MasterSwitch ESP-RSL: tele/sonoff/STATE = {"Time":"2019-11-11T17:04:35","Uptime":"4T20:11:41","Heap":21,"SleepMode":"Dynamic","Sleep":50,"LoadAvg":19,"POWER":"ON","Wifi":{"AP":1,"SSId":"Home9785","BSSId":"10:13:31:C8:D9:D3","Channel":6,"RSSI":100,"LinkCount":26,"Downtime":"0T00:17:15"}}
2019-11-11T17:04:36.197470+01:00 MasterSwitch ESP-RSL: tele/sonoff/SENSOR = {"Time":"2019-11-11T17:04:35","ENERGY":{"TotalStartTime":"2019-11-05T21:43:42","Total":24.195,"Yesterday":4.350,"Today":2.068,"Period":1,"Power":66,"ApparentPower":170,"ReactivePower":157,"Factor":0.39,"Voltage":223,"Current":0.762}}
2019-11-11T17:04:36.321760+01:00 LivingBlind ESP-RSL: tele/sonoff/STATE = {"Time":"2019-11-11T17:04:36","Uptime":"4T20:11:15","Heap":21,"SleepMode":"Dynamic","Sleep":50,"LoadAvg":19,"POWER1":"OFF","POWER2":"OFF","Wifi":{"AP":1,"SSId":"Home9785","BSSId":"10:13:31:C8:D9:D3","Channel":6,"RSSI":76,"LinkCount":28,"Downtime":"0T00:17:41"}}
2019-11-11T17:04:36.542349+01:00 BathroomFan ESP-RSL: STATE = {"Time":"2019-11-11T17:04:36","Uptime":"4T06:41:33","Heap":21,"SleepMode":"Dynamic","Sleep":50,"LoadAvg":19,"POWER":"OFF","Wifi":{"AP":1,"SSId":"Home9785","BSSId":"10:13:31:C8:D9:D3","Channel":6,"RSSI":50,"LinkCount":32,"Downtime":"0T00:17:16"}}
2019-11-11T17:04:36.542956+01:00 BathroomFan ESP-RSL: SENSOR = {"Time":"2019-11-11T17:04:36","AM2301":{"Temperature":21.2,"Humidity":48.2},"TempUnit":"C"}
2019-11-11T17:04:37.519628+01:00 BedroomLight ESP-RSL: STATE = {"Time":"2019-11-11T17:04:38","Uptime":"4T20:12:26","Heap":22,"SleepMode":"Dynamic","Sleep":50,"LoadAvg":19,"POWER":"OFF","Wifi":{"AP":1,"SSId":"Home9785","BSSId":"10:13:31:C8:D9:D3","Channel":6,"RSSI":58,"LinkCount":38,"Downtime":"0T00:18:00"}}
2019-11-11T17:04:43.241885+01:00 BadroomAuxLeft ESP-RSL: tele/sonoff/STATE = {"Time":"2019-11-11T17:04:43","Uptime":"0T12:53:06","Heap":21,"SleepMode":"Dynamic","Sleep":50,"LoadAvg":19,"POWER":"OFF","Wifi":{"AP":1,"SSId":"Home9785","BSSId":"10:13:31:C8:D9:D3","Channel":6,"RSSI":44,"LinkCount":4,"Downtime":"0T00:02:04"}}
2019-11-11T17:04:43.245286+01:00 BadroomAuxLeft ESP-RSL: tele/sonoff/SENSOR = {"Time":"2019-11-11T17:04:43","AM2301":{"Temperature":null,"Humidity":null},"TempUnit":"C"}
2019-11-11T17:04:46.721135+01:00 EntranceLight ESP-RSL: STATE = {"Time":"2019-11-11T17:04:46","Uptime":"5T09:11:18","UptimeSec":465078,"Heap":28,"SleepMode":"Dynamic","Sleep":50,"LoadAvg":19,"MqttCount":0,"POWER":"OFF","Wifi":{"AP":1,"SSId":"Home9785","BSSId":"10:13:31:C8:D9:D3","Channel":6,"RSSI":100,"LinkCount":27,"Downtime":"0T00:19:25"}}
2019-11-11T17:04:46.787822+01:00 BadroomAuxRight ESP-RSL: tele/sonoff/STATE = {"Time":"2019-11-11T17:04:46","Uptime":"4T20:10:09","Heap":21,"SleepMode":"Dynamic","Sleep":50,"LoadAvg":19,"POWER":"OFF","Wifi":{"AP":1,"SSId":"Home9785","BSSId":"10:13:31:C8:D9:D3","Channel":6,"RSSI":56,"LinkCount":26,"Downtime":"0T00:16:53"}}
2019-11-11T17:04:59.495380+01:00 GuestRoomLight ESP-RSL: STATE = {"Time":"2019-11-11T17:05:00","Uptime":"4T20:12:22","Heap":21,"SleepMode":"Dynamic","Sleep":50,"LoadAvg":19,"POWER":"OFF","Wifi":{"AP":1,"SSId":"Home9785","BSSId":"10:13:31:C8:D9:D3","Channel":6,"RSSI":80,"LinkCount":34,"Downtime":"0T00:17:40"}}

Thanks in advance for your support.

Already put LANG=C which improved a bit the execution time but still not enough.

Upvotes: 0

Views: 87

Answers (2)

James Brown
James Brown

Reputation: 37464

Using awk with default field-separators (space, tabs):

$ awk '                                 # using awk, 
$2=="LivingLight" {                     # if second field is LivingLight
    b=$NF                               # buffer last field
}
END {                                   # in the end
    if(match(b,/(ON|OFF)/))             # look for ON/OFF in last buffered entry
        print substr(b,RSTART,RLENGTH)  # output if found
}' file

Output this time:

OFF

Feel free to use with tail if it's any faster. One-liner:

$ awk '$2=="LivingLight"{b=$NF}END{if(match(b,/(ON|OFF)/))print substr(b,RSTART,RLENGTH)}' file

Or tac as suggested in another answer:

$ tac file | awk '
$2=="LivingLight" {                       # if match
    if(match($NF,/(ON|OFF)/)) {           # find ON/OFF
        print substr($NF,RSTART,RLENGTH)  # output
        exit                              # and exit
    }
}'

One-liner:

$ tac file | awk '$2=="LivingLight"{if(match($NF,/(ON|OFF)/)){print substr($NF,RSTART,RLENGTH);exit}}'

Upvotes: 1

KamilCuk
KamilCuk

Reputation: 142080

A good implementation of tac should use fseek(... SEEK_END). With mix with grep -m1 should allows for quite a fast parsing the file from the end to the first matching string. With grep -P we can remove all the unneeded parsing and processes - we can extract the OFF/ON in grep.

tac /var/log/user.log | grep -m1 -o -P 'LivingLight.*STATE =.*"POWER":"\K[^"]*'

Upvotes: 3

Related Questions