5

I am looking for a way to sort a list and print all lines, whose first column appears only once - i.e., match only on the first column. For example, I have a file where the first column is a path and the second column contains a 'type'

/path/foo/1 footsy
/path/foo/1 barsy
/path/foo/X barsy
/path/bar/2 footsy
/path/bar/2 barsy
/path/foo/Y footsy

(the file is actually sorted -k1,1)

Now, I would like to extract only cases like

/path/foo/X barsy
/path/foo/Y footsy

I am thinking about some way with awk, where I would have to store the previous line and compare the first field of the previous line to the corresponding field in the current line. But I have not yet an idea how to get it done :( I tried to adapt a solution found in another question but it is not really working as hoped

awk '{
  prev=$0; path=$1; type=$2
  getline
  if ($1 != $path) {
    print prev
  }
}'
  • foo/1 is unique from foo/X and foo/Y same as bar/2 so result should contain 4 lines – Costas Oct 12 '15 at 15:24
  • Try sort -u -k 1,1? – Tom Hunt Oct 12 '15 at 15:25
  • Please edit your question and clarify. Do you mean that you want the path minus the file name? Which of the footsy files in /path/foo should be kept? The first? The last? Any? What operating system are you using? Do you have access to GNU tools? – terdon Oct 12 '15 at 15:38
  • Thomas: You say, “the file is actually sorted -k1,1”, but the data you present isn’t sorted. – G-Man Oct 13 '15 at 3:10
  • This is hilarious (and I mean that ironically).  We insist that question-askers present the code that they’re using, and then, when one does, we bombard him with answers that don’t correct or build upon his attempt. – G-Man Oct 13 '15 at 3:11
1
  1. awk normally reads each line of the input and invokes the script on it.  The cases where you would use getline are few and far between.  When your script is run with six lines of input, this is an overview of what happens:

    Read line 1 normally

    Set variables
    Call getline, which reads line 2
    Compare variables

    Read line 3 normally

    Set variables
    Call getline, which reads line 4
    Compare variables

    Read line 5 normally

    Set variables
    Call getline, which reads line 6
    Compare variables

    Obviously this isn’t going to work.

  2. Secondly, you made a common mistake in your awk code.  In awk, fields from the input are referenced as $number and variables are referenced as variable_name.  This is different from shell scripts, where command line arguments are referenced as $number and variables are referenced as $variable_name.  Your test

    if ($1 != $path)
    

    should be

    if ($1 != path)
    
  3. Your overall approach is flawed.  You can’t identify strings that occur only once in the file by looking at two lines at a time.  I believe that you can do it by looking at three lines at a time (i.e., by keeping the two previous lines in variables), but things like that get complicated and messy.  It’s probably simpler to count occurrences.  Here’s a minimal modification on your script to do that.

    awk '{
      if ($1 != path) {
        if (count == 1) {
          print prev
        }
        count=1
      }
      else count++
      prev=$0; path=$1
    }
    END {
        if (count == 1) {
          print prev
        }
    }'
    

    I deleted type, since you never used it.

    Disclosure: This is essentially the same as the last part of glenn’s answer.

  • yes, you are right - while sleeping over it I also noticed that I need a window with essential three lines for a proper comparison, i.e., keeping the two previous lines in addition to the current one – THX Oct 13 '15 at 8:18
  • actually, I was thinking now about comm to handle it, but I really like your approach to use an occurence counter to store the info about common lines! – THX Oct 13 '15 at 8:28
2

These answers don't require the input to be sorted:

Store the count and last-line-seen in arrays. Requires a lot of memory for large files, and requires GNU awk

gawk '
    {count[$1]++; line[$1]=$0} 
    END {
        PROCINFO["sorted_in"]="@val_str_asc"
        for (key in line) if (count[key] == 1) print line[key]
    }
' file

Scan the file twice, first to get the count, next to print the lines with count 1

awk 'NR == FNR {count[$1]++; next} count[$1]==1' file file

This will be the fastest and require the least memory, taking advantage of the sorted input:

awk '
    prev_key && prev_key != $1 {if (count==1) print prev_line; count=0}
    {prev_key=$1; prev_line=$0; count++}
    END {if (count==1) print prev_line}
' file
1

If your shell support Process Substitution, and X and Y doesn't contain spaces, tabs:

$ grep -Ff <(awk '{print $1" "}' <file | LC_ALL=C uniq -u) <file
/path/foo/X barsy
/path/foo/Y footsy
  • If there are 2 or more lines with /path/foo/XX, grep will pick them up too. You probably need grep -f <(awk '{print "^" $1 "[[:blank:]]"} ... – glenn jackman Oct 13 '15 at 12:47
  • @glennjackman: Well, good point. But don't use -F also break if X contain meta characters. – cuonglm Oct 13 '15 at 12:51
  • seems that grep isn't a good fit for this question. – glenn jackman Oct 13 '15 at 13:20
  • @glennjackman: Append a space to the end of 1st field can prevent X and XX case. – cuonglm Oct 13 '15 at 13:25
  • still doesn't anchor it to the start of the line, but that does not appear to be a problem given the sample input. – glenn jackman Oct 13 '15 at 13:29
0

You could try with this:

cat text.tx | sort | uniq -c -w11 | fgrep '1 /' | awk '{print $2" "$3}'

with your text.txt like this

]#cat text.txt
/path/foo/1 footsy
/path/foo/1 barsy
/path/foo/X barsy
/path/bar/2 footsy
/path/bar/2 barsy
/path/foo/Y footsy
  • (1) printf "%s\n" "/path/foo/X barsy" "/path/foo/Y footsy" will also produce the requested output.  But that would be stupid; that just produces the correct result for the example input, not the problem in general.  By the same token, your answer is inappropriately tailored to the example input; it assumes that the data in the first column always begins with / and is 11 characters long, and that there are only two columns — none of which was specified in the problem description.  … (Cont’d) – G-Man Oct 13 '15 at 1:27
  • (Cont’d) …  (2) Even making those assumptions, your answer reports any path that appears once — or 11 times, or 21, 31, 451, 2001, etc.  (3) Overall, this is unnecessarily complicated.  For starters, read about UUOC.  But also, you’re missing the point of uniq — keeping the assumption about the field length, you can simplify the command to sort text.tx | uniq -u -w11. – G-Man Oct 13 '15 at 1:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.