CODEWITHSUNDEEP
regexawk
Hancy
Hancy

Reputation: 563

How to use awk variables in regular expressions?

I have a file called domain which contains some domains. For example:

google.com
facebook.com
...
yahoo.com

And I have another file called site which contains some sites URLs and numbers. For example:

image.google.com   10
map.google.com     8
...
photo.facebook.com  22
game.facebook.com   15
..

Now I'm going to count the url number each domain has. For example: google.com has 10+8. So I wrote an awk script like this:

BEGIN{
  while(getline dom < "./domain" > 0) {
    domain[dom]=0;
  }
  for(dom in domain) {
    while(getline < "./site" > 0) {
      if($1 ~/$dom$)   #if $1 end with $dom {
        domain[dom]+=$2;
      }
    }
  }
}

But the code if($1 ~/$dom$) doesn't run like I want. Because the variable $dom in the regular expression was explained literally. So, the first question is:

Is there any way to use variable $dom in a regular expression?

Then, as I'm new to writing script

Is there any better way to solve the problem I have?

Upvotes: 46

Views: 53773

Answers (5)

FrankL
FrankL

Reputation: 404

The problem of the answers above is that you cannot use the "metacharacters" (e.g. \< for a word boundary at the beginning of a word) if you use a string instead of a regular expression /.../. If you had a domain xyz.com and two sites ab.xyz.com and cd.prefix_xyz.com, the numbers of the two site entries would be added to xyz.com

Here's a solution using awk's pipe and the sed command: ...

for(dom in domain) {
    while(getline < "./site" > 0) {
        # let sed replaces occurence of the domain at the end of the site
        cmd = "echo '" $1 "' | sed 's/\\<'" dom "'$/NO_VALID_DOM/'"
        cmd | getline x
        close(cmd)
        if (match(x, "NO_VALID_DOM")) { 
          domain[dom]+=$2;
        }
    }
    close("./site") # this misses in original code
}

...

Upvotes: 0

Matt
Matt

Reputation: 74919

awk can match against a variable if you don't use the // regex markers.

if ( $0 ~ regex ){ print $0; }

In this case, build up the required regex as a string 

regex = dom"$"

Then match against the regex variable

if ( $1 ~ regex ) {
  domain[dom]+=$2;
}

Upvotes: 60

Jonathan Leffler
Jonathan Leffler

Reputation: 755114

You clearly want to read the site file once, not once per entry in domain. Fixing that, though, is trivial.

Equally, variables in awk (other than fields $0 .. $9, etc) are not prefixed with $. In particular, $dom is the field number identified by the variable dom (typically, that's going to be 0 since domain strings don't convert to any other number).

I think you need to find a way to get the domain from the data read from the site file. I'm not sure if you need to deal with sites with country domains such as bbc.co.uk as well as sites in the GTLDs (google.com etc). Assuming you are not dealing with country domains, you can use this:

BEGIN {
    while (getline dom < "./domain" > 0) domain[dom] = 0
    FS = "[ .]+"
    while (getline  < "./site" > 0)
    {
        topdom = $(NF-2) "." $(NF-1)
        domain[topdom] += $NF          
    }
    for (dom in domain) print dom "  " domain[dom]
}

In the second while loop, there are NF fields; $NF contains the count, and $1 .. $(NF-1) contain components of the domain. So, topdom ends up containing the top domain name, which is then used to index into the array initialized in the first loop.

Given the data in the question (minus the lines of dots), the output is:

yahoo.com  0
facebook.com  37
google.com  18

Upvotes: 1

Steve
Steve

Reputation: 54592

One way using an awk script:

BEGIN {
    FS = "[. ]"
    OFS = "."
}

FNR == NR {
    domain[$1] = $0
    next
}

FNR < NR {
    if ($2 in domain) {
        for ( i = 2; i < NF; i++ ) {
            if ($i != "") {
                line = (line ? line OFS : "") $i
            }
        }
        total[line] += $NF
        line = ""
    }
}

END {
    for (i in total) {
        printf "%s\t%s\n", i, total[i]
    }
}

Run like:

awk -f script.awk domain.txt site.txt

Results:

facebook.com    37
google.com  18

Upvotes: 2

glenn jackman
glenn jackman

Reputation: 247260

First of all, the variable is dom not $dom -- consider $ as an operator to extract the value of the column number stored in the variable dom

Secondly, awk will not interpolate what's between // -- that is just a string in there.

You want the match() function where the 2nd argument can be a string that is treated as the regular expression:

if (match($1, dom "$")) {...}

I would code a solution like:

awk '
  FNR == NR {domain[$1] = 0; next}
  {
    for (dom in domain) {
      if (match($1, dom "$")) {
        domain[dom] += $2
        break
      }
    }
  }
  END {for (dom in domain) {print dom, domain[dom]}}
' domain site

Upvotes: 27

Related Questions