This was my original command line:

awk 'FNR==NR{a[$2]=$1;next}($2 in a){if(a[$2]!=$1)print $2}' file1 file2

This is located within a bash script and I originally output each section to a file and then on the next line I used that file as input. However, now I am switching to the use of variables and I have got down to this line and am stuck because of the double input.

So I am no longer trying to feed files to awk, I am wanting to feed variables.

If I was dealing with one variable, I know I would do:

echo "$var1" | awk '{$1=$2=""; print $0}'

But I can't find an example of how to pass a second variable/stream into awk while in a bash script.

examples I've tried:

awk 'FNR==NR{a[$2]=$1;next}($2 in a){if(a[$2]!=$1)print $2}' "$var1" "$var2"

echo "${var1}" | awk 'FNR==NR{a[$2]=$1;next}($2 in a){if(a[$2]!=$1)print $2}' "${var2}"

awk -v s1="$var1" -v s2="$var2" 'FNR==NR{a[$2]=$1;next}($2 in a){if(a[$2]!=$1)print $2}' "$s1" "$s2"


For some background, this is a very simplified version, but I have a set of files somewhat like this:

:~/test$ ls -lR ?/file*
-rw-rw-r-- 1 madivad madivad 11 Apr 19 23:17 1/file.0
-rw-rw-r-- 1 madivad madivad  6 Apr 19 23:04 1/file.1
-rw-rw-r-- 1 madivad madivad  8 Apr 19 23:04 1/file.2
-rw-rw-r-- 1 madivad madivad  8 Apr 19 23:04 1/file.3
-rw-rw-r-- 1 madivad madivad  6 Apr 19 23:05 2/file.1
-rw-rw-r-- 1 madivad madivad 10 Apr 19 23:06 2/file.3
-rw-rw-r-- 1 madivad madivad  6 Apr 19 23:05 2/file.4
-rw-rw-r-- 1 madivad madivad 11 Apr 19 23:17 2/file.5

I also have two audit files which contain the results of a hashing of each directory structure which can span very deeply and contains 500MB+ of data each.

On the command line I have written several tools to look for dupes, sort of age and size, find files that are missing etc. I'm trying to more automate the process and have it self contained within a batch file. I have actually put it all in a batch file already and use the HDD as a scratch place, i.e., I save each output to the hard drive as a file, and the next batch line loads that file and continues. My goal is to get rid of all the superfluous files.

The file structure is in standard hashdeep output but I have already manipulated it to contain just a fixed string of hashes, a couple of spaces and then the full path and filename of the file. It starts out like this:

~/test$ head 2/audit?
==> 2/audit1 <==
%%%% HASHDEEP-1.0
%%%% size,md5,sha256,filename
## Invoked from: /home/madivad/test/1
## $ hashdeep -l file.0 file.1 file.2 file.3

==> 2/audit2 <==
%%%% HASHDEEP-1.0
%%%% size,md5,sha256,filename
## Invoked from: /home/madivad/test/2
## $ hashdeep -l file.1 file.3 file.4 file.5

and by the time I have got to the awk processing I already have it like this:

~/test$ head 2/so*
==> 2/sort1 <==
3213c6d334141924ab1454f0349a0ccb6f92601344e16851316e7cda90d053c0ad234e047ccf81ce6fe89e78bbdb111e    file.0
736e5da8b598eec84d4ec0e1be63b6d1eeda2f21a9320e751d98a92ceb903bdf5e9e5b6f6ca3690fdc0fb67b70a5efb2    file.2
e487aeec573e0936c1ac9f091130c20b1bbb337a707d059be503510600b59292bbc9b1a171fe2b0b07ec122bc2bbff4f    file.3
fa8f294721ab3fbb37793c68ff2cf09b32c66107f0f4f2053128e519681fc8e88806d0d2b17607ce9f2362aff66ad6c7    file.1

==> 2/sort2 <==
3213c6d334141924ab1454f0349a0ccb6f92601344e16851316e7cda90d053c0ad234e047ccf81ce6fe89e78bbdb111e    file.5
b7b9e4419c0708f401d45f061ac9a231fcfecc7079ad08912d7a0ebcb24816442067b101a0da0c252533b78e00c19c52    file.3
9e87b0fde0cf6b641bd4a27aee5f5536247ed9d62df1b8b2da7dedf48b0795ef302d21811885180384c3f560c819d361    file.4
fa8f294721ab3fbb37793c68ff2cf09b32c66107f0f4f2053128e519681fc8e88806d0d2b17607ce9f2362aff66ad6c7    file.1

With the specific line I am trying to run as per this question, I would expect the above input to output the filename $2 only if the filename appears in both files, but the hashes a[$2] and$1` don't match.

So for the above input I would only expect to see:

  • Can you give an example of desired input (files and variables both) and output? – DopeGhoti Apr 19 '17 at 18:35
  • 2
    Would awk ... <(echo "$var1") <(echo "$var2") do it? – Kusalananda Apr 19 '17 at 18:41
  • @Kusalananda You did have this comment up first so if you care to put this into an answer I will accept it. – Madivad Apr 20 '17 at 6:15
  • I'm saddened that considering I put a great deal of time into this problem before coming to SEUL, and then the time in researching it on the various sites (I had literally dozens of tabs open and tried so many solutions to which none worked), that I already have neigh sayers down voting my question. It's hard enough to learn this at the best of times without at least someone telling me why my question is no good :/ – Madivad Apr 20 '17 at 6:19

Considering these variables:

$ a=$'one\ntwo';echo "$a"
$ b=$'three\nfour';echo "$b"

These are some possible options:

$ printf '%s\n%s\n' "$a" "$b" |awk '{print NR,$0}'
$ awk '{print NR,$0}' <(echo "$a") <(echo "$b")   #process substitution as per Kusalananda's comment
$ echo "$a" |awk '{print NR,$0}' - <(echo "$b")
$ { echo "$a" && echo "$b";} |awk '{print NR,$0}'  #or even { echo "$a";echo "$b";}
$ awk '{print NR,$0}' <(echo "$a";echo "$b")
$ awk '{print NR,$0}' <<<$(echo "$a";echo "$b")

All above examples yields to:

1 one
2 two
3 three
4 four

If variables end with a new line (i.e variables contains contents of files) like this pattern:

$ a=$'one\ntwo\n'
$ b=$'three\nfour\n'

Then also this works:

$ awk '{print NR,$0}' <<<"$a$b"
1 one
2 two
3 three
4 four
| improve this answer | |
  • 1
    Process substitution works by, more or less, providing two temporary files, each one containing the result of the corresponding command. – Kusalananda Apr 19 '17 at 20:23
  • @Kusalananda Thanks again, I wish I saw this before going to bed last night. You hit the nail on the head. Although George, you had the first complete answer up, Kusalananda's comment got me over the line with this project. I will accept your answer if he hasn't put anything up in the next 24hrs. Cheers both of you – Madivad Apr 20 '17 at 6:17
  • @Kusalananda done. cheers for you help! thanks again guys – Madivad Apr 20 '17 at 6:59
  • I just came back to this George, and I thought I would check out your other examples. The last one with the <<< outputs 1 one two three four on one line. Of the others, only the second one <(echo "$a") <(echo "$b") and third echo "$a" |awk '{print NR,$0}' - <(echo "$b") provide what I'm after since I need to utilise FNR to differentiate between the inputs. I went with the dual echos as that is the most natural looking and easier to understand version for me (although there was nothing wrong with the third one, I just liked #2 better) :) – Madivad Apr 25 '17 at 9:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.