1

I am having a three different set of files as shown below:

cat "110001_test file_first_file.csv"
ID,NAME,LOCATION
1,Vikrant,Gurgaon
2,Bharat,Noida
3,Raju,Hyderabad

cat "110001_test file_second_file.csv"
ID,NAME,LOCATION
1,Vikrant,Gurgaon
22,abcd,Noida
3,Raju,Hyderabad

cat "110001_test file_third_file.csv"
ID,NAME,LOCATION
1,Vikrant,Gurgaon
2,Bharat,Noida
33,xyz,Hyderabad

and I am using below command to merge these files together and keeping header once at the top and also remove duplicates if any.

find . -type f -name '*test file*.csv' -exec cat {} + | awk 'NR == 1; NR > 1 {print $0 | "sort -u"}' > output.file

It's giving me an output as:

ID,NAME,LOCATION
1,Vikrant,Gurgaon
22,abcd,Noida
2,Bharat,Noida
33,xyz,Hyderabad
3,Raju,Hyderabad
ID,NAME,LOCATION

I understand what's happening here and what command is doing. It's basically neglecting the first line and picking other records to sort and removing duplicates among them. due to which I see an extra header at the bottom of the output file.

I was expecting output like this.

ID,NAME,LOCATION
1,Vikrant,Gurgaon
22,abcd,Noida
2,Bharat,Noida
33,xyz,Hyderabad
3,Raju,Hyderabad

4 Answers 4

2

With bash using head, tail and sort:

You could save the filenames in an array and

  • print the header line of the first file
  • output the content of all files starting at line 2 and sort the result with the unique option
  • redirect the output to a file
files=( *test*.csv )
{
  head -n1 "${files[0]}"
  for i in "${files[@]}"; do
    tail -n+2 "$i"
  done | sort -u
} > output
1

You can't rely using NR>1 and concatenating before all files with cat {} +
Like this, I try to reuse your code as possible:

{ 
    awk 'NR==1{print;exit}' *test*file*csv
    find . -type f -name '*test*file.csv' -exec awk 'NR>1' {} \; |
    sort -u
} > output.file

Output file

ID,NAME,LOCATION
1,Vikrant,Gurgaon
22,abcd,Noida
2,Bharat,Noida
33,xyz,Hyderabad
3,Raju,Hyderabad

find -exec {} +

-exec command {} +
This variant of the -exec action runs the specified command on the selected files, but the command line is built by appending each selected file name at the end

So already concatenated

4
  • Thanks. what if I have to pass "test file" file name as a variable to above script. second AWK statement is working but not sure how to pass in first awk statement. find . -type f -name "$myvar.csv" -exec awk 'NR>1' {} \; | sort - u Jun 17, 2020 at 21:46
  • 1
    Not clear what you are asking, better post another question with input(s)/expected output/code Jun 17, 2020 at 21:51
  • what I meant was like suppose passing a file name as a variable instead of hard coding in above code. Jun 17, 2020 at 21:53
  • Still unclear, if the variable belongs to 'find', what the variable contains and the general context Jun 17, 2020 at 22:02
1

simply do:

find . -type f -name '*test*file.csv' -exec awk '!seen[$0]++' {} +

for each line that awk reads, if it was not (!) previously set in associated array called seen that will be added to the array and since condition result resolves to "true" so that line goes to output and if any same line read by awk then condition result to "false", so it will skip duplicates;

In awk $0 is referring to entire line and here we used as the array keys and values will be incremented for each key (line) read. for a line if it was not set in the array, the value is 0 and will be printed, and if value for a key was >0 it will be skipped.

2
  • 1
    Thanks a ton.. I was not aware of this and really liked this approach a lot and can do lot of things further with this array. Jun 18, 2020 at 15:46
  • you are welcome! : ) Jun 18, 2020 at 15:49
-1

Modifying your command line slightly, we will get the desired output:

awk 'NR<2||FNR>1' ./*test_file*.csv > temp ;
( head -n 1; sort -u; ) < temp > output.file

Your results are in the file output.file

7
  • The < temp means to take input from temp file. The previous temp is on a separate line. Jun 18, 2020 at 2:13
  • tesh.sh: line 6: syntax error near unexpected token (' tesh.sh: line 6: < temp (head -n 1; sort -u;) > output.file' Jun 18, 2020 at 2:24
  • You forgot a semicolon. #!/bin/bash find . -type f -name 'test file.csv' -exec cat {} + | awk 'NR<2||FNR>1' > temp; < temp (head -n 1; sort -u;) > output.file Jun 18, 2020 at 2:59
  • same error. syntax error near unexpected token `(' .. could be please check at ur side and post the correct answer. I will upvote Jun 18, 2020 at 3:04
  • Edited. my post . now it should work at your end. Jun 18, 2020 at 3:57

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .