15

I want to replace occurrences of "|" EXCEPT the last one in every line of a file with a space using sed only. I want to avoid doing this:

 sed -e "s/[|]/ /1" -e "s/[|]/ /1" -e "s/[|]/ /1" -e "s/[|]/ /1" -e "s/[|]/ /1"  -e "s/[|]/ /1" -e "s/[|]/ /1" mydata.txt

File input:

FLD1     |SFK TK |FLD2   |FLD4 |FLD5 |-          |20200515 |NNNN |406   RCO 301
FLD1     |SFK TK |FLD2   |FLD4 |FLD5 |-          |20200515 |NNNN |0
FLD1     |SFK TK |FLD2   |FLD4 |FLD5 |-          |20200515 |NNNN |0     

File output:

FLD1      SFK TK  FLD2    FLD4  FLD5  -           20200515  NNNN |406   RCO 301
FLD1      SFK TK  FLD2    FLD4  FLD5  -           20200515  NNNN |0
FLD1      SFK TK  FLD2    FLD4  FLD5  -           20200515  NNNN |0
4
  • 3
    What if there is only one | on the line? So the first one will also be the last one. Should that be removed or left alone?
    – terdon
    May 17, 2020 at 22:40
  • @terdon, never occurred to me, a good point, but a single "|" would never be the case for my input file. May 18, 2020 at 2:19
  • 2
    Are the number of pipe delimited fields fixed? If yes, sed 'y/|/\n/;s/\n/|/8;y/\n/ /' file can be used. May 18, 2020 at 14:33
  • If the number of | is fixed, a nicely chosen combination of cut and paste would probably do it. May 19, 2020 at 13:56

8 Answers 8

20
sed ':a;/[|].*[|]/s/[|]/ /;ta' file
  • /[|].*[|]/: If line has two pipes,
  • s/[|]/ /: Substitute the first with a space.
  • ta: If a substitution was made, go back to :a.

Output:

$ sed ':a;/[|].*[|]/s/[|]/ /;ta' file
FLD1      SFK TK  FLD2    FLD4  FLD5  -           20200515  NNNN |406   RCO 301
FLD1      SFK TK  FLD2    FLD4  FLD5  -           20200515  NNNN |0
FLD1      SFK TK  FLD2    FLD4  FLD5  -           20200515  NNNN |0

As @steeldriver has remarked, you can use simply | instead of [|] in a basic regular expression (BRE), as is the case above. If you add the -E flag to sed, extended regular expression (ERE) is enabled and then you need to write [|] or \|.


Just for completeness, POSIX sed specification says that "Editing commands other than {...}, a, b, c, i, r, t, w, :, and # can be followed by a semicolon". Then, a compliant alternative to the above is:

sed -e ':a' -e '/[|].*[|]/s/[|]/ /;t a' file
2
  • 2
    Slightly different, but very similar: sed -e ':again' -e 's/|\(.*\)|/ \1|/' -e 't again' file (also POSIX, as you need a newline after a label ordinarily).
    – Kusalananda
    May 17, 2020 at 19:43
  • The POSIX spec also indicates that the t command verb shall be separated from its argument by a single space: t a rather than ta
    – guest
    May 18, 2020 at 5:09
10

A different approach from Quasímodo's explicit loop in sed:

$ sed 'h; s/.*|//; x; s/|[^|]*$//; y/|/ /; G; y/\n/|/' file
FLD1      SFK TK  FLD2    FLD4  FLD5  -           20200515  NNNN |406   RCO 301
FLD1      SFK TK  FLD2    FLD4  FLD5  -           20200515  NNNN |0
FLD1      SFK TK  FLD2    FLD4  FLD5  -           20200515  NNNN |0

For each line, this saves the line in the hold space with h and then removes everything on the line up to and including the last |. It then swaps in the original copy of the line and removes the last | and everything after it.

The pattern space now contains the original first portion of the line, and the hold space contains the last portion of the line.

The first y/// command replaces all remaining | with spaces. G appends the hold space to the end of the pattern space with a newline character in-between. The second y/// command transforms that newline character to a |, and we're done.

Doing a limited (fixed) number of s/// substitutions and using the faster y/// command when possible means that this runs quicker than the explicit loop variation (~2.3 seconds on 50 MiB data, compared to ~7.8 seconds on the same data with the loop, using GNU sed on my machine).

Interestingly, using a back reference in the explicit loop variation, like both me and Isaac did, slows it down even more (~33s with Isaac's variation, and ~29s with mine (in comments), on the same data set and under the same conditions as above).


Using awk, this almost replaces all | delimiters except the last one by a space. "Almost" since it inserts a space before the last |.

$ awk -F '|' 'BEGIN { OFS = " " } { $NF = "|" $NF; print }' file
FLD1      SFK TK  FLD2    FLD4  FLD5  -           20200515  NNNN  |406   RCO 301
FLD1      SFK TK  FLD2    FLD4  FLD5  -           20200515  NNNN  |0
FLD1      SFK TK  FLD2    FLD4  FLD5  -           20200515  NNNN  |0

It reads each line as a set of |-delimited fields, prepends a | character to the start of the last field, and prints the resulting record with spaces for field delimiters.

With taking the default behavior of awk into account (space is the default output field delimiter, and the input field delimiter is available as FS):

awk -F '|' '{ $NF = FS $NF; print }' file

or, slightly shorter, courtesy of @Isaac,

awk -F '|' '{ $NF = FS $NF }; 1' file
0
8

Using Perl you could run something along the lines of

perl -pe 's/\|(?=.*\|)/ /g'     ex

where:

  • perl -pe action -- executes action and print
  • \|(?=.*\|) is a regular expression that matches | with a not consumed lookeahed (?=.*|) containing another |
5

Posixly:

sed -e ':a' -e '/|\(.*|\)/s// \1/;ta' file
  • -e ':a' define a label (a) to jump.
  • -e ' start another script section.
  • /|\(.*|\)/ one regex to match two |, everything in the middle and to capture all in between and the last |.
  • s// \1/ replace everyting matched above with what was captured.
  • ;ta loop again.
  • ' file on the filename given.

To measure the speed of all the options (from faster to slower) you can use:

#! /bin/bash
TIMEFORMAT='run : %lR sec'

read -d '' str <<\END
FLD1     |SFK TK |FLD2   |FLD4 |FLD5 |-          |20200515 |NNNN |406   RCO 301
FLD1     |SFK TK |FLD2   |FLD4 |FLD5 |-          |20200515 |NNNN |0
FLD1     |SFK TK |FLD2   |FLD4 |FLD5 |-          |20200515 |NNNN |0'
END

n=${1:-100}; printf "$str"'%.0s\n' $(seq "$n") > file

time perl -pe 's/\|(?=.*\|)/ /g' file >/dev/null
time sed -E ':a;/\|.{1,}\|/s/\|/ /1;ta' file >/dev/null
time sed 'h; s/.*|//; x; s/|[^|]*$//; y/|/ /; G; y/\n/|/' file >/dev/null
time sed 's/\(.*\)|/\1\x00/;y/|/ /;s/\x00/|/1' file >/dev/null
time sed 's/\(.*\)|/\1\n/;y/|/ /;s/\n/|/1' file >/dev/null

To be used as:

$ ./testbash.sh 235000
run    : 0m07.676s sec
run    : 0m17.753s sec
run    : 0m22.074s sec
run    : 0m24.036s sec
run    : 0m24.047s sec
2

Here are a couple of alternatives you can use.

$ sed -e '
   s/|[^|]*$/\n&/
   s/\n|/\n/
   y/\n|/| /
' file

$ perl -pe 's/\|/ / until tr/|/|/ == 1' file

$ perl -pe 'my $k=tr/|/|/; s/\|/ / while $k-->1' file
2

What about cheating this way ? No one said we needed just one sed:

sed -r 's/\|([^|]+)$/##\1/' | sed 's/|/ /g' | sed 's/##/|/'

Here, I used ## as a placeholder, replace only the last | of the line with it, replace all the other |, and then put back the last one. Just make sure you have no ## in your input.

1
  • This is functionally the same as my answer, except mine replaces the last | with a newline, which cannot possibly occur within a line (so you don't need to include a disclaimer).  Note that you can have multiple sed commands within a single sed process / invocation.  It's very rare to need to pipe one sed process into another, let alone two others. Jun 5, 2020 at 23:22
1

Without a loop:

sed 's/\(.*\)|/\1\
/; s/|/ /g; s/\
/|/'
  • Replace the last vertical bar in each line with a newline.  (This may be inserted into the pattern space, but is guaranteed not to be there initially.)
  • Replace all remaining vertical bars with spaces.
  • Replace the newline (inserted with the first command) with a vertical bar.
1
  • That is exactly the last approach (the slowest) in my answer, sorry. :-)
    – user232326
    May 19, 2020 at 1:47
1

If number of fields is always the same, cut +tr & paste with a shell that supports <():

paste -d\| <(cut -d\| -f1-8 file | tr \| ' ') <(cut -d\| -f9 file)

Or with GNU cut&paste:

paste -d\| <(cut -d\| --output-delimiter=' ' -f1-8 file) <(cut -d\| -f9 file)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.