Find and Replace Incrementally Across Multiple Files - Bash

Question

I apologize in advance if this belongs in SuperUser, I always have a hard time discerning whether these scripting in bash questions are better placed here or there. Currently I know how to find and replace strings in multiple files, and how to find and replace strings within a single file incrementally from searching for a solution to this issue, but how to combine them eludes me.

Here's the explanation:

• I have a few hundred files, each in sets of two: a data file (.data), and a message file (data.ms).
• These files are linked via a key value unique to each set of two that looks like: ab.cdefghi

Here's what I want to do:

• Step through each .data file and do the following:

• Find:

  MessageKey ab.cdefghi
  

• Replace:

  MessageKey xx.aaa0001
  MessageKey xx.aaa0002
  ...
  MessageKey xx.aaa0010
  etc.
  

  Incrementing by 1 every time I get to a new file.

Clarifications:

• For reference, there is only one instance of "MessageKey" in every file.
• The paired files have the same name, only their extensions differ, so I could simply step through all .data files and then all .data.ms files and use whatever incremental solution on both and they'd match fine, don't need anything too fancy to edit two files in tandem or anything.
• For all intents and purposes whatever currently appears on the line after each MessageKey is garbage and I am completely throwing it out and replacing it with xx.aaa####
• String length does matter, so I need xx.aa0009, xx.aaa0010 not xx.aa0009, xx.aa00010
• I'm using cygwin.

Answer 1

@Carl.Anderson got me started on the right track and after a little tweaking, I ended up implementing his solution but with some syntax tweaks.

First of all, this solution only works if all of your files are located in the same directory. I'm sure anyone with even slightly more experience with UNIX than me could modify this to work recursively, but here goes:

First I ran:

-hr "MessageKey" . | sort -u | awk '{ printf("%s:xx.aaa%04d\n", $2, ++i); }' > MessageKey

This command was used to create a find and replace map file called "MessageKey."

The contents of which looked like:

In.Rtilyd1:aa.xxx0087
In.Rzueei1:aa.xxx0088
In.Sfricf1:aa.xxx0089
In.Slooac1:aa.xxx0090
etc...

Then I ran:

MessageKey | while IFS=: read old new; do sed -i -e "s/MessageKey $old/MessageKey $new/" *Data ; done

I had to use IFS=: (or I could have alternatively find and replaced all : in the map file with a space, but the former seemed easier.

Anyway, in the end this worked! Thanks Carl for pointing me in the right direction.

Answer 2

I would approach this by creating a mapping from old key to new and dumping that into a temp file.

grep MessageKey *.data \
  | sort -u \
  | awk '{ printf("%s:xx.aaa%04d\n", $1, ++i); }' \
  > /tmp/key_mapping

From there I would confirm that the file looks right before I applied the mapping using sed to the files.

cat /tmp/key_mapping \
  | while read old new; do
      sed -i -e "s:MessageKey $old:MessageKey $new:" * \
    done

This will probably work for you, but it's neither elegant or efficient. This is how I would do it if I were only going to run it once. If I were going to run this regularly and efficiency mattered, I would probably write a quick python script.