jq select filter chain

Question

I am having the following array of objects that I would like to filter down. Like this:

1. LOGGEDIN == 0
2. Timestamp older than 5 minutes
3. IDLETIME > 60 && CPULOAD < 200

So for the second filter I’d like not to consider the objects filtered out on the first filter. And for the third filter I’d like not to consider the objects filtered out on the second filter. I tried to get the selection with jq:

1. jq '.[] | select(.LOGGEDIN=="0")'
2. jq '.[] | select(.TIMESTAMP | fromdateiso8601 < '$FIVEMINAGO')'
3. jq '.[] | select(.IDLETIME |tonumber > 60) | select(.CPULOAD |tonumber < 200)'

I’d like to wrap these up so that I end up with one array of objects, matching the filters and another array of objects, that do not. I’m on a Mac, zsh.

[
  {
    "SERIAL": "XXXSERIAL1XXX",
    "TIMESTAMP": "2020-12-17 18:45:14",
    "EMAIL": "email1@mydomain.com",
    "LOGGEDIN": "0",
    "IDLETIME": "122",
    "CPULOAD": "2",
    "BLOCKED": "0"
  },
  {
    "SERIAL": "XXXSERIAL2XXX",
    "TIMESTAMP": "2020-12-17 18:43:29",
    "EMAIL": "email2@mydomain.com",
    "LOGGEDIN": "1",
    "IDLETIME": "0",
    "CPULOAD": "0",
    "BLOCKED": "0"
  },
  {
    "SERIAL": "XXXSERIAL3XXX",
    "TIMESTAMP": "2020-12-17 18:46:37",
    "EMAIL": "email1@mydomain.com",
    "LOGGEDIN": "1",
    "IDLETIME": "0",
    "CPULOAD": "0",
    "BLOCKED": "0"
  },
  {
    "SERIAL": "XXXSERIAL4XXX",
    "TIMESTAMP": "2020-12-17 18:45:23",
    "EMAIL": "email3@mydomain.com",
    "LOGGEDIN": "0",
    "IDLETIME": "0",
    "CPULOAD": "13",
    "BLOCKED": "0"
  },
  {
    "SERIAL": "XXXSERIAL5XXX",
    "TIMESTAMP": "2020-12-17 18:47:02",
    "EMAIL": "email2@mydomain.com",
    "LOGGEDIN": "1",
    "IDLETIME": "0",
    "CPULOAD": "0",
    "BLOCKED": "0"
  },
  {
    "SERIAL": "XXXSERIAL6XXX",
    "TIMESTAMP": "2020-12-17 18:43:42",
    "EMAIL": "email3@mydomain.com",
    "LOGGEDIN": "1",
    "IDLETIME": "10",
    "CPULOAD": "20",
    "BLOCKED": "0"
  },
  {
    "SERIAL": "XXXSERIAL7XXX",
    "TIMESTAMP": "2020-12-17 18:43:29",
    "EMAIL": "email4@mydomain.com",
    "LOGGEDIN": "1",
    "IDLETIME": "0",
    "CPULOAD": "0",
    "BLOCKED": "0"
  },
  {
    "SERIAL": "XXXSERIAL8XXX",
    "TIMESTAMP": "2020-12-17 18:46:02",
    "EMAIL": "email4@mydomain.com",
    "LOGGEDIN": "0",
    "IDLETIME": "0",
    "CPULOAD": "0",
    "BLOCKED": "0"
  },
  {
    "SERIAL": "XXXSERIAL9XXX",
    "TIMESTAMP": "2020-12-17 18:45:23",
    "EMAIL": "email1@mydomain.com",
    "LOGGEDIN": "0",
    "IDLETIME": "443",
    "CPULOAD": "666",
    "BLOCKED": "0"
  }
]

Answer 1

Problems with the snippets you posted:

• Don't try to generate code in the shell! Use --arg (or some other mechanism) to pass values to your program instead.
• Your timestamps are not valid ISO8601 timestamps, much less what fromdateiso8601 expects.
• | has the lowest precedence other than ;, so
  .IDLETIME | tonumber > 60 means
  .IDLETIME | ( tonumber > 60 ) but you want
  ( .IDLETIME | tonumber ) > 60.

---

We can start with this:

jq --arg TSCUT "$( date --date='5 minutes ago' +%s )" '
   group_by(
      .LOGGEDIN == "0" and
      ( .TIMESTAMP | sub(" "; "T") + "Z" | fromdateiso8601 ) < $TSCUT and
      ( .IDLETIME | tonumber ) > 60 and
      ( .CPULOAD | tonumber ) < 200
   )
'

jqplay

The above segregates the matching records from those that don't, but we could end up with any of the following:

• [ ]
• [ [...matches...] ]
• [ [...non-matches...] ]
• [ [...non-matches...], [...matches...] ]

This isn't very useful. As such, I propose the following:

jq --arg TSCUT "$( date --date='5 minutes ago' +%s )" '
   map(
      ._f = (
         .LOGGEDIN == "0" and
         ( .TIMESTAMP | sub(" "; "T") + "Z" | fromdateiso8601 ) < $TSCUT and
         ( .IDLETIME | tonumber ) > 60 and
         ( .CPULOAD | tonumber ) < 200
      )
   ) |
   . as $a |
   {
      "matches":     [ $a[] | select( ._f       ) | del(._f) ],
      "non-matches": [ $a[] | select( ._f | not ) | del(._f) ]
   }
'

jqplay

---

I assumed that "$( ... )" means the same thing in zsh as it does in the POSIX shell. Adjust as needed.

Thanks to @oguz ismail for pointing out group_by, even though I retain my original solution.