CODEWITHSUNDEEP
linuxbashgrep
Nice Guy
Nice Guy

Reputation: 509

Using grep to find lines that each contain ALL search strings

I have a file which contains a lot of lines similar to this:

{"id": 2796, "some_model": "Profile", "message_type": "MODEL_SAVE", "fields": {"account": 14, "address": null, "modification_timestamp": "2014-03-19T10:46:33.543Z", "was_deleted": false}}

But then I want to find all the lines which will contain some pieces of the respective lines I want only. An example that will be applied in the example line above would be:

~$ grep '2796' file.log | grep 'Profile' | grep 'another_more' | grep 'so_on'

I tried doing the same way as above, EDITED: It did work, but was not quite enough to bring all the necessary data. I mean, there were missing data in the results of the search. :(

Following the idea of grep 'word' filename it works, but just one word in a mountain of data is not enough. So, how to pass multiple 'word' to match what I really want? What I really want is a search by the 'ID', '*some_model*' and 'account' using grep at the same time.

How do I do that search to match all the possible lines with those arguments in the prompt? This is more like a doubt, is possible to use conditions like if and else o while too combining with grep for example?

If the questions are not clear, please let me know to right it. Thanks to all.

Upvotes: 3

Views: 4417

Answers (5)

dimo414
dimo414

Reputation: 48884

This question specifically asks about grep, but really sed or awk are much cleaner for 'a AND b'-style matches, see How to run grep with multiple AND patterns?

This answer covers how to use grep to match all lines which match all inputs - my other answer covers matching any input.

Note that grep is more powerful than simple word-matching, it can match arbitrary patterns, including multiple words.

Consider the following simplified version of the example you provide:

$ cat file
{"id": 2796, "some_model": "Profile", "was_deleted": false}
{"id": 2797, "some_model": "Profile", "was_deleted": true}
{"id": 2798, "some_model": "Another", "was_deleted": false}

You could find item 2796 like so:

$ grep '"\?id"\? *: *2796 *,\?' file
{"id": 2796, "some_model": "Profile", "was_deleted": false}

Or find all non-deleted items:

$ grep '"\?was_deleted"\? *: *false *[,}]' file
{"id": 2796, "some_model": "Profile", "was_deleted": false}
{"id": 2798, "some_model": "Another", "was_deleted": false}

You can even combine the two, to only get item 2796 if it's not deleted (change false to true and the line no longer matches):

$ grep '"\?id"\? *: *2796 *,\?.*"\?was_deleted"\? *: *false *[,}]' file
{"id": 2796, "some_model": "Profile", "was_deleted": false}

Or roughly equivalent, using the grep-piping syntax you use above:

$ grep '"\?id"\? *: *2796 *,\?' file | grep '"\?was_deleted"\? *: *false *[,}]'
{"id": 2796, "some_model": "Profile", "was_deleted": false}

These examples hopefully look tricky to get right, because this is not a good idea!

The data you're working with looks to be JSON, a structured data format which grep is not well suited for processing. Valid JSON could be split across multiple lines or have fields in arbitrary order, which will break the above patterns. Not to mention arbitrary white-space (*), semi-optional quotes ("\?), and end of field vs. end of object markers ([,}]) which the above patterns should handle, but are easy to get wrong.

If you are trying to query JSON data, you need a JSON parser, which grep is not. http://www.json.org/ offers links to several popular JSON parsers in many languages, see if any of those will suite your needs. You will have much better success with a real tool than trying to construct complicated regular expressions.

Upvotes: 1

dimo414
dimo414

Reputation: 48884

This answer covers how to use grep to match all lines which match one of many inputs - my other answer covers matching all input, which the OP was actually looking for. I suspect this answer is more commonly what people are looking for, so I'm leaving it here.

The -e parameter lets you search for multiple different matches:

$ cat file
Hello World
Nope
Foo Bar

$ grep -e Hello -e Foo file
Hello World
Foo Bar

You can also use the | character to delimit multiple matches in a single query, but you have to escape it with a \ and quote the query string, like so:

$ grep 'Hello\|Foo' file
Hello World
Foo Bar

Or use the f flag to have grep use patterns specified in a file:

$ cat patterns
Hello
Foo

$ grep -f patterns file
Hello World
Foo Bar

Needless to say, I personally prefer using -e, but there are many options.

Upvotes: 0

jimm-cl
jimm-cl

Reputation: 5422

I believe this is answered in the official GNU grep documentation:

http://www.gnu.org/savannah-checkouts/gnu/grep/manual/grep.html#Usage

10.  I can do “OR” with ‘|’, but what about “AND”?

    grep 'paul' /etc/motd | grep 'franc,ois' 

finds all lines that contain both ‘paul’ and ‘franc,ois’.

Looks like the only way to achieve a logical AND using grep, that is not dependent of the order in where the matching patterns are found.

Upvotes: 0

Jotne
Jotne

Reputation: 41460

An awk version. This is restrictive in the find order, so string1 comes before string2

awk '/string1.*string2.*string3.*string4/' filename

Upvotes: 0

devnull
devnull

Reputation: 123698

You could use sed:

sed '/string1/!d; /string2/!d; /string3/!d; /string4/!d' filename

This would produce only the lines containing all the strings in any order.

Same thing using awk:

awk '/string1/ && /string2/ && /string3/ && /string4/' filename

Upvotes: 3

Related Questions