6

This is a text processing question. I have 2 files:

joeblogs
johnsmith
chriscomp
12:00:00 (AAA) OUT: "string" joeblogs@hostname
12:00:00 (AAA) OUT: "string" joeblogs@hostname
12:00:00 (AAA) OUT: "string" johnsmith@hostname
12:00:00 (AAA) OUT: "string" joeblogs@hostname
12:00:00 (AAA) OUT: "string" chriscomp@hostname

File 1 contains a list of unique usernames that appear in a log (file 2).

Desired output

12:00:00 (AAA) OUT: "string" USER1@hostname
12:00:00 (AAA) OUT: "string" USER1@hostname
12:00:00 (AAA) OUT: "string" USER2@hostname
12:00:00 (AAA) OUT: "string" USER1@hostname
12:00:00 (AAA) OUT: "string" USER3@hostname

I guess I don't need both files. File 1 is generated by parsing file 2 for the unique usernames. My logic was to get a list of usernames that I know appear in file 2, and loop through it, replacing with sed.

Something like:

for i in $(cat file1);do sed -e 's/$i/USER[X]';done

Where USER[X] increments with each unique username.

However I can't do this. I don't even think that logic is sound. Can I have help to achieve the desired output? awk/sed/grep/bash are all welcome.

9

As you have realized that you "don't need the 2 files", use the following awk solution to process the initial log file in one pass:

awk '{
         u_name = substr($5, 1, index($5, "@"));
         if (!(u_name in users)) users[u_name] = ++c;
         sub(/^[^@]+/, "USER" users[u_name], $5)
     }1' file.log

The output:

12:00:00 (AAA) OUT: "string" USER1@hostname
12:00:00 (AAA) OUT: "string" USER1@hostname
12:00:00 (AAA) OUT: "string" USER2@hostname
12:00:00 (AAA) OUT: "string" USER1@hostname
12:00:00 (AAA) OUT: "string" USER3@hostname
6

Another awk

awk '!($5 in a){a[$5]=++i}{sub("[^@]*","USER"a[$5],$5)}1' infile
5

With bash you could do:

n=0
declare -A users=()
while IFS= read -r line; do
    if [[ $line =~ ([^[:blank:]]+)@ ]]; then
        user=${BASH_REMATCH[1]}
        if [[ -z ${users[$user]} ]]; then
            users[$user]=USER$((++n))
        fi
        line=${line/$user/${users[$user]}}
    fi 
    echo "$line"
done < File2

or a perl one-liner

perl -pe 's/(\S+)(?=@)/ $users{$1} ||= "USER".++$n /e' File2
2

With sed, you could do this:

$ sed "$(sed '=' File1 | sed -r 'N;s/(.*)\n(.*)/s%\2@hostname%USER\1@hostname%/')" File2
12:00:00 (AAA) OUT: "string" USER1@hostname
12:00:00 (AAA) OUT: "string" USER1@hostname
12:00:00 (AAA) OUT: "string" USER2@hostname
12:00:00 (AAA) OUT: "string" USER1@hostname
12:00:00 (AAA) OUT: "string" USER3@hostname
$ 

There are 3 sed commands here. sed commands 2 and 3 generate a sed expression out of File1 that is in turn used by command 1 to process File2:

  • Command 2 simply appends the line number after each line of File1
  • Command 3 rearranges each line of File1 and its following line number into a sed expression to replace e.g. joeblogs@hostname with USER1@hostname, and so on for all the users in File1
  • Command 1 then uses the generated sed expression to process the all the replacements in File2.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.