Remove usernames in a file and replace with a incremental number

Question

This is a text processing question. I have 2 files:

joeblogs
johnsmith
chriscomp

12:00:00 (AAA) OUT: "string" joeblogs@hostname
12:00:00 (AAA) OUT: "string" joeblogs@hostname
12:00:00 (AAA) OUT: "string" johnsmith@hostname
12:00:00 (AAA) OUT: "string" joeblogs@hostname
12:00:00 (AAA) OUT: "string" chriscomp@hostname

File 1 contains a list of unique usernames that appear in a log (file 2).

Desired output

12:00:00 (AAA) OUT: "string" USER1@hostname
12:00:00 (AAA) OUT: "string" USER1@hostname
12:00:00 (AAA) OUT: "string" USER2@hostname
12:00:00 (AAA) OUT: "string" USER1@hostname
12:00:00 (AAA) OUT: "string" USER3@hostname

I guess I don't need both files. File 1 is generated by parsing file 2 for the unique usernames. My logic was to get a list of usernames that I know appear in file 2, and loop through it, replacing with sed.

Something like:

for i in $(cat file1);do sed -e 's/$i/USER[X]';done

Where USER[X] increments with each unique username.

However I can't do this. I don't even think that logic is sound. Can I have help to achieve the desired output? awk/sed/grep/bash are all welcome.

Answer 1

As you have realized that you "don't need the 2 files", use the following awk solution to process the initial log file in one pass:

awk '{
         u_name = substr($5, 1, index($5, "@"));
         if (!(u_name in users)) users[u_name] = ++c;
         sub(/^[^@]+/, "USER" users[u_name], $5)
     }1' file.log

The output:

12:00:00 (AAA) OUT: "string" USER1@hostname
12:00:00 (AAA) OUT: "string" USER1@hostname
12:00:00 (AAA) OUT: "string" USER2@hostname
12:00:00 (AAA) OUT: "string" USER1@hostname
12:00:00 (AAA) OUT: "string" USER3@hostname

Answer 2

Another awk

awk '!($5 in a){a[$5]=++i}{sub("[^@]*","USER"a[$5],$5)}1' infile

Answer 3

With bash you could do:

n=0
declare -A users=()
while IFS= read -r line; do
    if [[ $line =~ ([^[:blank:]]+)@ ]]; then
        user=${BASH_REMATCH[1]}
        if [[ -z ${users[$user]} ]]; then
            users[$user]=USER$((++n))
        fi
        line=${line/$user/${users[$user]}}
    fi 
    echo "$line"
done < File2

or a perl one-liner

perl -pe 's/(\S+)(?=@)/ $users{$1} ||= "USER".++$n /e' File2

Answer 4

With sed, you could do this:

$ sed "$(sed '=' File1 | sed -r 'N;s/(.*)\n(.*)/s%\2@hostname%USER\1@hostname%/')" File2
12:00:00 (AAA) OUT: "string" USER1@hostname
12:00:00 (AAA) OUT: "string" USER1@hostname
12:00:00 (AAA) OUT: "string" USER2@hostname
12:00:00 (AAA) OUT: "string" USER1@hostname
12:00:00 (AAA) OUT: "string" USER3@hostname
$ 

There are 3 sed commands here. sed commands 2 and 3 generate a sed expression out of File1 that is in turn used by command 1 to process File2:

• Command 2 simply appends the line number after each line of File1
• Command 3 rearranges each line of File1 and its following line number into a sed expression to replace e.g. joeblogs@hostname with USER1@hostname, and so on for all the users in File1
• Command 1 then uses the generated sed expression to process the all the replacements in File2.