Is there a way to output file's contents using custom patterns?

For instance, having a file myfile with following contents:


..how would one sort it using following pattern: print lines starting with "b" first, then print lines starting with "d" and then print lines in normal alphabetical order, so expected output is:


You would need to use something more than just the sort command. First grep the b lines, then the d lines and then sort anything without the b or d at the end of that.

grep '^b' myfile > outfile
grep '^d' myfile >> outfile
grep -v '^b' myfile | grep -v '^d' | sort >> outfile
cat outfile

will result in:


This is assuming that the lines start with the 'pattern' b and d if that is the whole pattern or something inside the line you can leave out the caret (^)

A one-line equivalent would be:

(grep '^b' myfile ; grep '^d' myfile ; grep -v '^b' myfile | grep -v '^d' | sort)
  • This method works, but can it be refactored into a one-liner that only outputs to STDOUT instead of writing to a file in chunks? – Sergey Lukin May 12 '13 at 11:45
  • @Sergey Just don't use redirections. – Rany Albeg Wein May 12 '13 at 12:07
  • I see, so: grep '^b' myfile; grep '^d' myfile; cat myfile| grep -v '^b' | grep -v '^d' | sort – Sergey Lukin May 12 '13 at 12:20
  • @Sergey you would want to inclose the stuff in ( and ) otherwise you cannot redirect the combined output – Anthon May 12 '13 at 12:52
  • Yes, also I can see that cat was unnecessary, thanks! I think we've got a bit overwhelmed but a perfectly working one-liner. Can we utilize awk to make it more efficient? – Sergey Lukin May 12 '13 at 13:13

When you need data to be sorted beyond sort's capability, a common approach is to pre-process the data to prepend a sort key, then sort, and finally remove the extra sort key. For example, here, add a 0 if a line starts with b, a 1 if a line starts with d, and a 2 otherwise.

sed -e 's/^b/0&/' -e t -e 's/^d/1&/' -e 't' -e 's/^/2/' |
sort |
sed 's/^.//'

Note that this sorts all the b and d lines. If you want these lines in the original order, then the easiest approach is to split off the lines that you want to leave unsorted. You can, however, work the original line into a sort key with nl — but here it's more complicated. (Replace \t by a literal tab character throughout if your sed doesn't understand that syntax.)

nl -ba -nln |
sed 's/^[0-9]* *\t\([bd]\)/\1\t&/; t; s/^[0-9]* *\t/z\t0\t/' |
sort -k1,1 -k2,2n |
sed 's/^[^\t]*\t[^\t]*\t//'

Alternatively, use a language such as Perl, Python or Ruby that lets you easily specify a custom sort function.

perl -e 'print sort {($b =~ /^[bd]/) - ($a =~ /^[bd]/) ||
                     $a cmp $b} <>'
python -c 'import sys; sys.stdout.write(sorted(sys.stdin.readlines(), key=lambda s: (0 if s[0]=="b" else 1 if s[0]=="d" else 2), s))'

or if you want to leave the b and d lines in the original order:

perl -e 'while (<>) {push @{/^b/ ? \@b : /^d/ ? \@d : \@other}, $_}
         print @b, @d, sort @other'
python -c 'import sys
b = []; d = []; other = []
for line in sys.stdin.readlines():
    if line[0]=="b": b += line
    elif line[0]=="d": d += line
    else: other += line
sys.stdout.writelines(b); sys.stdout.writelines(d); sys.stdout.writelines(other)'
  • The idea of pre-processing sounds promising, however neither of your sed examples really work for me – Sergey Lukin May 13 '13 at 7:54
  • The second sed example was nonsense and I've fixed it, but the first works for me. What's wrong with it? – Gilles 'SO- stop being evil' May 13 '13 at 21:55
  • With first example I get an error when t; is inside, when I replace t with \t, literal tab or take it off I get following output: 0b\n 1d\n a\n c\n which is close but b and d are left with integers – Sergey Lukin May 14 '13 at 17:59
  • @Sergey, you must have copied the code wrong. The first sed example is an excellent and efficient solution to this problem. Just copy and paste the first example into your terminal, type in the lines a, b, c, d, and then control-d to see output. It would help if you can tell us the content of the error message. It sounds like you have a missing quote or other syntax. – RobertL Oct 26 '15 at 5:21
  • @RobertL So I figured it out. I'm getting sed: 1: "s/^b/0&/; t; s/^d/1&/; ...": undefined label '; s/^d/1&/; t; s/^/2/' on BSD version of SED (used by default on FreeBSD and OS X), however it indeed works perfectly fine on GNU SED 4.2.2 – Sergey Lukin Oct 27 '15 at 15:05

One way of solving this using awk would be:

sort myfile | awk '$0 ~ /^b/ || $0 ~ /^d/ {print} $0 !~ /^b/ && $0 !~ /^d/ { a[f++] = $0 } END { for (word = 0; word < f; word++) { print a[word] } }'
  • +1 have you tried if this runs faster than the (multiple) grep invocations? – Anthon May 12 '13 at 14:28
  • @Anthon Yes, according to my results AWK method runs in average 2 times faster. Check out this gist for details: gist.github.com/sergeylukin/5564524 – Sergey Lukin May 12 '13 at 18:57
cat file | tr bd '\1\2' | LANG=C sort | tr '\1\2' bd

Where the intermediary contents are: (printing CrtA, CrtB as , )

file  | tr-1  | sort  | tr-2
cat     cat     Ⓐat     bat
bed     ⒶeⒷ     ⒶeⒷ     bed
fog     fog     Ⓑay     day
dog     Ⓑog     Ⓑog     dog
egg     egg     cat     cat
day     Ⓑay     egg     egg
kin     kin     fog     fog
lay     lay     get     get
in      in      in      in
bat     Ⓐat     kin     kin
get     get     lay     lay

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.