CODEWITHSUNDEEP
bashshellawksed
dig_123
dig_123

Reputation: 2358

parsing a json file with awk or sed by dividing contents into lines based on start and end match

I have a .json file which I need to parse: 

 {
    "schemaVersion":16,
    "addons":[
      {"id":"[email protected]","syncGUID":"pc9SmMiK6Peu","location":"app-profile","version":"1.2.3.1-signed","type":"extension","internalName":null,"updateURL":null,"updateKey":null,"optionsURL":null,"optionsType":null,"aboutURL":null,"iconURL":"chrome://testpilot/skin/testpilot_32x32.png","icon64URL":null,"defaultLocale":{"name":"Test Pilot","description":"Help make Firefox better by running user studies.","creator":"Mozilla Corporation","homepageURL":"http://testpilot.mozillalabs.com/"},"visible":true,"active":true,"userDisabled":false,"appDisabled":false,"descriptor":"/Users/kiamlaluno/Library/Application Support/Firefox/Profiles/a9wsbury.default/extensions/[email protected]","installDate":1431796864000,"updateDate":1432784122000,"applyBackgroundUpdates":1,"bootstrap":false,"size":1278820,"sourceURI":"https://addons.cdn.mozilla.net/user-media/addons/13661/test_pilot-1.2.3-fx.xpi?filehash=sha256%3A813e3080c355e98091fa321529cf3949739ef3a1497a4b45a3b5244aee443fac","releaseNotesURI":"https://addons.mozilla.org/versions/updateInfo/1501117/en-US/","softDisabled":false,"foreignInstall":false,"hasBinaryComponents":false,"strictCompatibility":false,"locales":[],"targetApplications":[{"id":"{ec8030f7-c20a-464f-9b0e-13a3a9e97384}","minVersion":"3.5","maxVersion":"17.0a1"}],"targetPlatforms":[],"multiprocessCompatible":false},
      {"id":"[email protected]","syncGUID":"Lu-OEFQCHDuJ","location":"app-profile","version":"1.2.14","type":"extension","internalName":null,"updateURL":null,"updateKey":null,"optionsURL":null,"optionsType":null,"aboutURL":null,"iconURL":null,"icon64URL":null,"defaultLocale":{"name":"Dreditor","description":"An extension for Drupal.org that enhances user experience and functionality. Original author: Daniel F. Kudwien (sun).","creator":"Mark Carver","homepageURL":"https://dreditor.org"},"visible":true,"active":true,"userDisabled":false,"appDisabled":false,"descriptor":"/Users/kiamlaluno/Library/Application Support/Firefox/Profiles/a9wsbury.default/extensions/[email protected]","installDate":1432790689000,"updateDate":1432790689000,"applyBackgroundUpdates":1,"bootstrap":true,"size":124907,"sourceURI":"https://dreditor.org/dreditor.xpi?1.2.14","releaseNotesURI":null,"softDisabled":false,"foreignInstall":false,"hasBinaryComponents":false,"strictCompatibility":false,"locales":[],"targetApplications":[{"id":"{ec8030f7-c20a-464f-9b0e-13a3a9e97384}","minVersion":"21.0","maxVersion":"29.0a1"}],"targetPlatforms":[],"multiprocessCompatible":false},
      {"id":"[email protected]","syncGUID":"n8s_OF7ZCuw1","location":"app-profile","version":"7.0.1","type":"dictionary","internalName":null,"updateURL":null,"updateKey":null,"optionsURL":null,"optionsType":null,"aboutURL":null,"iconURL":null,"icon64URL":null,"defaultLocale":{"name":"United States English Spellchecker","description":"English United States (en-US) Spellchecker","creator":"Giuliano Masseroni","homepageURL":"https://addons.mozilla.org/en-US/firefox/addon/3497","contributors":["Kevin Atkinson","Geoff Kuenning"]},"visible":true,"active":true,"userDisabled":false,"appDisabled":false,"descriptor":"/Users/kiamlaluno/Library/Application Support/Firefox/Profiles/a9wsbury.default/extensions/[email protected]","installDate":1433038761000,"updateDate":1433038761000,"applyBackgroundUpdates":1,"bootstrap":true,"size":701806,"sourceURI":"https://addons.mozilla.org/firefox/downloads/latest/3497/addon-3497-latest.xpi?src=dp-btn-primary","releaseNotesURI":null,"softDisabled":false,"foreignInstall":false,"hasBinaryComponents":false,"strictCompatibility":false,"locales":[],"targetApplications":[{"id":"{ec8030f7-c20a-464f-9b0e-13a3a9e97384}","minVersion":"17.0","maxVersion":"42.0"},{"id":"{3550f703-e582-4d05-9a08-453d09bdfdc6}","minVersion":"5.0","maxVersion":"23.0"},{"id":"{92650c4d-4b8e-4d2a-b7eb-24ecf4f6b63a}","minVersion":"2.0","maxVersion":"2.20"}],"targetPlatforms":[]},
      {"id":"[email protected]","syncGUID":"0HLJG3tK5S0H","location":"app-profile","version":"3.4.0","type":"dictionary","internalName":null,"updateURL":null,"updateKey":null,"optionsURL":null,"optionsType":null,"aboutURL":null,"iconURL":null,"icon64URL":null,"defaultLocale":{"name":"Italian dictionary","description":"Italian dictionary based on version 3.4.0 (2013-09-05) of the Italian dictionary created by project PLIO.","creator":"Francesco Lodolo","homepageURL":"http://www.mozillaitalia.org"},"visible":true,"active":true,"userDisabled":false,"appDisabled":false,"descriptor":"/Users/kiamlaluno/Library/Application Support/Firefox/Profiles/a9wsbury.default/extensions/[email protected]","installDate":1433038989000,"updateDate":1433038989000,"applyBackgroundUpdates":1,"bootstrap":true,"size":1412342,"sourceURI":"https://addons.mozilla.org/firefox/downloads/latest/3053/platform:3/addon-3053-latest.xpi?src=dp-btn-primary","releaseNotesURI":null,"softDisabled":false,"foreignInstall":false,"hasBinaryComponents":false,"strictCompatibility":false,"locales":[{"locales":["it"],"name":"Dizionario italiano","description":"Dizionario italiano basato sulla versione 3.4.0 (2013-09-05) del dizionario realizzato dal progetto PLIO.","creator":null,"homepageURL":null}],"targetApplications":[{"id":"{ec8030f7-c20a-464f-9b0e-13a3a9e97384}","minVersion":"18.0a1","maxVersion":"41.0"},{"id":"{3550f703-e582-4d05-9a08-453d09bdfdc6}","minVersion":"18.0a1","maxVersion":"34.0"},{"id":"{92650c4d-4b8e-4d2a-b7eb-24ecf4f6b63a}","minVersion":"2.7a1","maxVersion":"2.31"},{"id":"{aa3c5121-dab2-40e2-81ca-7ea25febc110}","minVersion":"18.0a1","maxVersion":"34.0"},"id":"[email protected]","minVersion":"18.0a1","maxVersion":"34.0"}],"targetPlatforms":[]},
      {"id":"[email protected]","syncGUID":"fqAFQIuqP6rc","location":"app-profile","version":"4.4.2","type":"extension","internalName":null,"updateURL":"https://d13itkw33a7sus.cloudfront.net/dist/1P/ext/autoupdate_firefox4.xml","updateKey":null,"optionsURL":null,"optionsType":null,"aboutURL":null,"iconURL":null,"icon64URL":null,"defaultLocale":{"name":"1Password","description":"Password and identity manager for Mac, Windows, iOS and Android.","creator":"AgileBits Inc","homepageURL":"https://agilebits.com/onepassword"},"visible":true,"active":true,"userDisabled":false,"appDisabled":false,"descriptor":"/Users/kiamlaluno/Library/Application Support/Firefox/Profiles/a9wsbury.default/extensions/[email protected]","installDate":1431759097000,"updateDate":1438291043000,"applyBackgroundUpdates":1,"bootstrap":true,"size":248521,"sourceURI":"https://d13itkw33a7sus.cloudfront.net/dist/1P/ext/1Password-4.4.2.xpi","releaseNotesURI":null,"softDisabled":false,"foreignInstall":false,"hasBinaryComponents":false,"strictCompatibility":false,"locales":[],"targetApplications":[{"id":"{ec8030f7-c20a-464f-9b0e-13a3a9e97384}","minVersion":"24.0","maxVersion":"27.0"}],"targetPlatforms":[],"multiprocessCompatible":false},
      {"id":"[email protected]","syncGUID":"hbdMIPDTjJjU","location":"app-profile","version":"39.0","type":"locale","internalName":null,"updateURL":null,"updateKey":null,"optionsURL":null,"optionsType":null,"aboutURL":null,"iconURL":null,"icon64URL":null,"defaultLocale":{"name":"English (US) Language Pack","description":null,"creator":"mozilla.org","homepageURL":null},"visible":true,"active":true,"userDisabled":false,"appDisabled":false,"descriptor":"/Users/kiamlaluno/Library/Application Support/Firefox/Profiles/a9wsbury.default/extensions/[email protected]","installDate":1433038780000,"updateDate":1438297982000,"applyBackgroundUpdates":1,"bootstrap":true,"size":1107183,"sourceURI":"https://addons.cdn.mozilla.net/user-media/addons/407142/english_us_language_pack-39.0-fx.xpi?filehash=sha256%3Ae970523c118125330029522253119290be0c3fae810304b6f8851d5b1a08bb2a","releaseNotesURI":null,"softDisabled":false,"foreignInstall":false,"hasBinaryComponents":false,"strictCompatibility":true,"locales":[],"targetApplications":[{"id":"{ec8030f7-c20a-464f-9b0e-13a3a9e97384}","minVersion":"39.0","maxVersion":"39.*"}],"targetPlatforms":[]},
      {"id":"{972ce4c6-7e08-4474-a285-3208198ce6fd}","syncGUID":"4aQXjLIYT8-y","location":"app-global","version":"39.0.3","type":"theme","internalName":"classic/1.0","updateURL":null,"updateKey":null,"optionsURL":null,"optionsType":null,"aboutURL":null,"iconURL":null,"icon64URL":null,"defaultLocale":{"name":"Default","description":"The default theme.","creator":"Mozilla","homepageURL":null,"contributors":["Mozilla Contributors"]},"visible":true,"active":true,"userDisabled":false,"appDisabled":false,"descriptor":"/Applications/Firefox.app/Contents/Resources/browser/extensions/{972ce4c6-7e08-4474-a285-3208198ce6fd}","installDate":1428098726000,"updateDate":1439001657000,"applyBackgroundUpdates":1,"skinnable":true,"size":3175,"sourceURI":null,"releaseNotesURI":null,"softDisabled":false,"foreignInstall":false,"hasBinaryComponents":false,"strictCompatibility":true,"locales":[],"targetApplications":[{"id":"{ec8030f7-c20a-464f-9b0e-13a3a9e97384}","minVersion":"39.0.3","maxVersion":"39.0.3"}],"targetPlatforms":[]},{"id":"{4ED1F68A-5463-4931-9384-8FFF5ED91D92}","syncGUID":"mtfF_fBR6ym-","location":"app-system-local","version":"2.0","type":"extension","internalName":null,"updateURL":null,"updateKey":null,"optionsURL":null,"optionsType":null,"aboutURL":null,"iconURL":null,"icon64URL":null,"defaultLocale":{"name":"McAfee SiteAdvisor","description":null,"creator":"McAfee Inc.","homepageURL":"http://www.siteadvisor.com/"},"visible":true,"active":true,"userDisabled":false,"appDisabled":false,"descriptor":"/Library/Application Support/Mozilla/Extensions/{ec8030f7-c20a-464f-9b0e-13a3a9e97384}/{4ED1F68A-5463-4931-9384-8FFF5ED91D92}","installDate":1417806531000,"updateDate":1417806531000,"applyBackgroundUpdates":1,"bootstrap":false,"size":18013,"sourceURI":null,"releaseNotesURI":null,"softDisabled":false,"foreignInstall":true,"hasBinaryComponents":false,"strictCompatibility":false,"locales":[],"targetApplications":[{"id":"{ec8030f7-c20a-464f-9b0e-13a3a9e97384}","minVersion":"4.0","maxVersion":"15.*"}],"targetPlatforms":[],"multiprocessCompatible":false}
    ]
}

I need to take each {"id": line from the "addons" section and get the "version":, "name": under "defaultLocale" and "descriptor": sections.

I need the same to be done in only awk or sed (basically any standard linux commands or a combination of these). I'll not be able to use jq or any other tools to parse the .json file.

Please help

Upvotes: 0

Views: 424

Answers (3)

Ed Morton
Ed Morton

Reputation: 203635

Your question isn't clear wrt requirements, the lines in the example are WAY too long to wade through trying to find the context for the strings you're interested in, and you haven't shown us the expected output so there's not much we can do to help but here's a start that you can hopefully adapt to do whatever it is you're trying to do:

$ cat tst.awk
{
    while ( match($0,/"(version|name)":("[^"]*"|[^,}]+)/) ) {
        print NR, substr($0,RSTART,RLENGTH)
        $0 = substr($0,RSTART+RLENGTH)
    }
}

.

$ awk -f tst.awk file
4 "version":"1.2.3.1-signed"
4 "name":"Test Pilot"
5 "version":"1.2.14"
5 "name":"Dreditor"
6 "version":"7.0.1"
6 "name":"United States English Spellchecker"
7 "version":"3.4.0"
7 "name":"Italian dictionary"
7 "name":"Dizionario italiano"
8 "version":"4.4.2"
8 "name":"1Password"
9 "version":"39.0"
9 "name":"English (US) Language Pack"
10 "version":"39.0.3"
10 "name":"Default"
10 "version":"2.0"
10 "name":"McAfee SiteAdvisor"

.

$ cat tst.awk
{
    while ( match($0,/"defaultLocale":{[^{}]+}|"descriptor":("[^"]*"|[^,}]+)/) ) {
        sect = substr($0,RSTART,RLENGTH)
        $0 = substr($0,RSTART+RLENGTH)
        print NR, sect
        while ( match(sect,/"(version|name)":("[^"]*"|[^,}]+)/) ) {
            print "\t" substr(sect,RSTART,RLENGTH)
            sect = substr(sect,RSTART+RLENGTH)
        }
    }
    print ""
}

.

$ awk -f tst.awk file



4 "defaultLocale":{"name":"Test Pilot","description":"Help make Firefox better by running user studies.","creator":"Mozilla Corporation","homepageURL":"http://testpilot.mozillalabs.com/"}
        "name":"Test Pilot"
4 "descriptor":"/Users/kiamlaluno/Library/Application Support/Firefox/Profiles/a9wsbury.default/extensions/[email protected]"

5 "defaultLocale":{"name":"Dreditor","description":"An extension for Drupal.org that enhances user experience and functionality. Original author: Daniel F. Kudwien (sun).","creator":"Mark Carver","homepageURL":"https://dreditor.org"}
        "name":"Dreditor"
5 "descriptor":"/Users/kiamlaluno/Library/Application Support/Firefox/Profiles/a9wsbury.default/extensions/[email protected]"

6 "defaultLocale":{"name":"United States English Spellchecker","description":"English United States (en-US) Spellchecker","creator":"Giuliano Masseroni","homepageURL":"https://addons.mozilla.org/en-US/firefox/addon/3497","contributors":["Kevin Atkinson","Geoff Kuenning"]}
        "name":"United States English Spellchecker"
6 "descriptor":"/Users/kiamlaluno/Library/Application Support/Firefox/Profiles/a9wsbury.default/extensions/[email protected]"

7 "defaultLocale":{"name":"Italian dictionary","description":"Italian dictionary based on version 3.4.0 (2013-09-05) of the Italian dictionary created by project PLIO.","creator":"Francesco Lodolo","homepageURL":"http://www.mozillaitalia.org"}
        "name":"Italian dictionary"
7 "descriptor":"/Users/kiamlaluno/Library/Application Support/Firefox/Profiles/a9wsbury.default/extensions/[email protected]"

8 "defaultLocale":{"name":"1Password","description":"Password and identity manager for Mac, Windows, iOS and Android.","creator":"AgileBits Inc","homepageURL":"https://agilebits.com/onepassword"}
        "name":"1Password"
8 "descriptor":"/Users/kiamlaluno/Library/Application Support/Firefox/Profiles/a9wsbury.default/extensions/[email protected]"

9 "defaultLocale":{"name":"English (US) Language Pack","description":null,"creator":"mozilla.org","homepageURL":null}
        "name":"English (US) Language Pack"
9 "descriptor":"/Users/kiamlaluno/Library/Application Support/Firefox/Profiles/a9wsbury.default/extensions/[email protected]"

10 "defaultLocale":{"name":"Default","description":"The default theme.","creator":"Mozilla","homepageURL":null,"contributors":["Mozilla Contributors"]}
        "name":"Default"
10 "descriptor":"/Applications/Firefox.app/Contents/Resources/browser/extensions/{972ce4c6-7e08-4474-a285-3208198ce6fd}"
10 "defaultLocale":{"name":"McAfee SiteAdvisor","description":null,"creator":"McAfee Inc.","homepageURL":"http://www.siteadvisor.com/"}
        "name":"McAfee SiteAdvisor"
10 "descriptor":"/Library/Application Support/Mozilla/Extensions/{ec8030f7-c20a-464f-9b0e-13a3a9e97384}/{4ED1F68A-5463-4931-9384-8FFF5ED91D92}"

The above will work using any awk in any shell on any UNIX box. Note that it does what it does with your posted sample input only, don't expect it to work robustly for JSON in general.

Upvotes: 1

Chris Maes
Chris Maes

Reputation: 37742

jq should get you a long way:

# some JSON resembling yours
JSON='{"schemaVersion":16, "addons":[{"id": "id_1"}, {"id": "id_2"}, {"id": "id_3"}]}'
jq -r '.addons[].id' <<< $JSON

which gives: 

id_1
id_2
id_3

Upvotes: 3

Anubis
Anubis

Reputation: 7435

Maybe you should consider using a tool specialized for json processing (like jq as suggested in the comments) if you are going to parse a more complicated json.

However, if your json has the exact same format as mentioned, you may filter it as a normal text file using sed as follows.

sed '0,/"id"/ d; /}/,/"id"/ d;s/\s*//' input.txt

Output:

//data 1    # suffixed with a number to indicate different lines
//data 2
//data 3
//data 4

Upvotes: 0

Related Questions