I'm wanting to replace each character of white space at the end of each line with '_'. I found a similar question and answer for leading whitespace. But have failed to reconstruct it for trailing whitespace. Here's the link: https://stackoverflow.com/questions/9222281/replace-leading-whitespace-with-sed-or-similar

If anyone can think of a quicker or better way, that would also be great. I also appreciate good explanations, as that way I learn quicker :)

foo bar
 foo bar oof
  line 3a  
  line fo a

foo bar_____
 foo bar oof
  line 3a___
  line fo a_
  • 1
    If I cut and paste your sample input, the "foo bar" line has no trailing whitespace. Do you want to add some so all the lines are the same length? – glenn jackman Mar 28 at 22:11
  • yes, sorry, whitespace should be there and replaced as per the output example – Giles Mar 29 at 23:51

With GNU awk for the 3rd arg to match() and gensub():

$ awk 'match($0,/(.*[^ ])(.*)/,a){$0=a[1] gensub(/ /,"_","g",a[2])} 1' file
foo bar_____
 foo bar oof
  line 3a___
  line fo a_

With any awk:

$ awk 'match($0,/ +$/){tail=substr($0,RSTART,RLENGTH); gsub(/ /,"_",tail); $0=substr($0,1,RSTART-1) tail} 1' file
foo bar_____
 foo bar oof
  line 3a___
  line fo a_

To replace leading blanks too by tweaking the above gawk solution:

$ awk 'match($0,/^( *)(.*[^ ])(.*)/,a){$0=gensub(/ /,"_","g",a[1]) a[2] gensub(/ /,"_","g",a[3])} 1' file
foo bar_____
_foo bar oof
__line 3a___
__line fo a_
| improve this answer | |
  • The OP is apparently trying to print lines that all line up on the right side so I doubt if they want a tab to be replaced with a single _ but their input doesn't contain any tabs so we're guessing. ln any case, why not post that as an answer as it's a reasonable approach? – Ed Morton Mar 28 at 14:45
  • I don't think posting different awk scripts as comments under one answer is as useful to the community as posting them as different answers but it's every posters prerogative I suppose. – Ed Morton Mar 28 at 15:37
  • @mosvy . I switched out the tab for space and that awk command is so much faster than the sed command. Because of this I also tried using it for the leading spaces, editing sections I thought I understood to make it work for leading.... I failed miserably. Could you explain how to implement this for leading spaces too? – Giles Mar 29 at 18:56
  • @mosvy here's another good reason to post your own answer - so I don't get messages in my Inbox when people try to talk to you about your answer as I did just now (and presumably will get alerted when you respond to them too). It'd also be nice if the OP could accept your answer if they like, which they can't do if it's buried in a list of comments. – Ed Morton Mar 29 at 18:58
  • 1
    I didn't know that. Well as the sed command wasn't practical time wise for the file size I'm dealing with, this really is a far more suitable answer, so I'm going to switch it. – Giles Mar 29 at 19:25

With GNU sed, replacing all spaces at eol by underscore:

sed ':x;s/ \( *\)$/_\1/;tx'  
| improve this answer | |
  • Probably since Oracle took it over. – Gerard H. Pille Mar 28 at 15:21
  • Was meant to be funny. What gives "sed ':x s/ \( *\)$/_\1/;tx'" on Solaris? But even with the --posix parameter, my original solution still works. – Gerard H. Pille Mar 28 at 17:06
  • 1
    I read you LOUD and CLEAR. – Gerard H. Pille Mar 28 at 18:57

More efficient to use perl:

perl -lpe 's/(\s+)$/"_" x length($1)/e' input.txt

which only has to do one substitution per line with trailing whitespace, instead of looping.

| improve this answer | |
  • But needs loading a 3Mb executable vs. a 120kb one. – Gerard H. Pille Mar 28 at 9:06
  • @GerardH.Pille modern systems (ie newer > 198x) are using on-demand loading and a unified buffer cache. – mosvy Mar 28 at 13:55

With awk

awk -F '[ \t]+$' 'NF>1{t=substr($0,length($1)+1);gsub(/./,"_",t); $0=$1 t} 1'

This also handles a mixture of trailing tabs and spaces. The field separator (-F, FS) can be easily adjusted to only match spaces or also match other kind of invisible characters, provided that it's kept anchored at the end with $.

To make that work for leading blanks, everything should be mirrored not just $ to ^:

awk -F '^[ \t]+' 'NF>1{h=substr($0,1,length()-length($2));gsub(/./,"_",h); $0=h $2} 1'

To make it work for both leading and trailing blanks, the logic should be inverted; set the field separator to a pattern not matching leading and trailing blanks:

awk -F '[^ \t](.*[^ \t]|$)' '{s=$0; h=gsub(/./,"_",$1); t=gsub(/./,"_",$2); print $1 substr(s,h+1, length(s)-h-t) $2}'

Or the same with adjustable pattern:

awk -v ns='[^ \t]' 'BEGIN{FS=ns"(.*"ns"|$)"}{s=$0; h=gsub(/./,"_",$1); t=gsub(/./,"_",$2); print $1 substr(s,h+1, length(s)-h-t) $2}'

Different from @EdMorton's solutions, these handle correctly lines which contain only spaces and will work with any implementation of awk, not just GNU awk (gawk): eg. with mawk or bwk ("original-awk"), which are both much faster than gawk. But even when used with gawk, the last solution will be almost twice as fast as @EdMorton's.

With sed

With sed, the only solution I can think of is to substitute repeatedly in a loop; if there are many trailing spaces and long lines, this can get slow fast:

sed -e :x -e 's/ \( *\)$/_\1/;tx'

Notice that sed ':x;s/ \( *\)$/_\1/;tx' is not standard sed; :label is not one of the commands which can be terminated by a ;:

Editing commands other than {...}, a, b, c, i, r, t, w, :, and # can be followed by a <semicolon>, optional <blank> characters, and another editing command.

With perl

Here is an alternate perl solution, which is NOT really an improvement upon the existing perl answer, but which, since it doesn't use the e flag of s///, could be theoretically adapted to some other tool providing a sed-like s/// and perl/pcre-like zero-width assertions in its regexes:

perl -ple 's/\s(?=\s*$)/_/g'
| improve this answer | |
  • I've never seen a fancy awk field separator like that before. Cool! – Joe Apr 8 at 12:20

If you're trying to add whitespace to even out the lines:

$ cat -A file
foo bar$
 foo bar oof$
  line 3a  $
  line fo a$

Lines are not even

perl -MList::Util=max -lne '
    push @lines, $_
    $wid = max map {length} @lines;
    for $line (@lines) {
        $padded = sprintf "%-*s", $wid, $line;
        $padded =~ s/(\s+)$/"_" x length($1)/e;
        print $padded
' file
foo bar_____
 foo bar oof
  line 3a___
  line fo a_
| improve this answer | |
  • You can also use pack "A$length", $line instead of sprintf. Example: perl -MList::Util=max -e '$p="_"x($l=max map{s/\s*$//;length}@l=<>); print map {pack"A$l a",$_.$p,"\n"}@l' – mosvy Mar 28 at 23:27
  • or working like your example when the longest line also had trailing spaces: perl -MList::Util=max -le '$p="_"x($l=max map{chomp;length}@l=<>); print for map {pack"A$l",s/\s+$//r.$p}@l' – mosvy Mar 28 at 23:37

You can do this via the following simple methods also :

Perl splits on nonwhitespace and perform the transliteration of space to underscore on the last field. Then print all fields.

$ perl -F'(\S+)'  -lane '
    $F[-1] =~ tr/ /_/ if @F;
    print @F;
' file 

Using GNU sed we perform the equivalent of lookahead:

$ sed -re '
    y/ /\n/
      s/\n(\S| )/ \1/
' file 

This will work posixly (change \S to [^[:space:]])

$ sed -e '
    y/ /_/;x;G;s/\n//
' file 

This method will work only in gnu sed as it makes use of s///e extended flag.

$ sed -re '
    s/(.*\S)(.*)/echo "\1""$(echo "\2" | tr " " _)"/e
' file 
| improve this answer | |
  • Your perl will choke on empty lines with Modification of non-creatable array value attempted. You sed s///e (the last) will probably take an hour to finish on an input that the perl will handle in 1 or 2 seconds ;-) – mosvy Mar 30 at 2:49
  • It is no big deal, just make the tranliteration conditional. Well the OP mentioned no need for speed in his query. – Rakesh Sharma Mar 30 at 3:37
  • 1. It's a bug. 2. Should they mention it? The sed s///e is not twice, or thrice, or ten times slower, but 3000 times slower (and that assuming a multicore system and an OS with very fast process creation, like Linux -- otherwise it can easily get 30000 or 100000 times slower). – mosvy Mar 30 at 3:49
  • Hello, we are not making production code here. The OP gave a data we tried on it. If you want to nitpick then ask the Op to provide comprehensive input. – Rakesh Sharma Mar 30 at 3:56
  • 1. Don't royal-we me. 2. Welcome to my kill-list. – mosvy Mar 30 at 3:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.