15

I want to replace occurrences of "|" EXCEPT the last one in every line of a file with a space using sed only. I want to avoid doing this:

 sed -e "s/[|]/ /1" -e "s/[|]/ /1" -e "s/[|]/ /1" -e "s/[|]/ /1" -e "s/[|]/ /1"  -e "s/[|]/ /1" -e "s/[|]/ /1" mydata.txt

File input:

FLD1     |SFK TK |FLD2   |FLD4 |FLD5 |-          |20200515 |NNNN |406   RCO 301
FLD1     |SFK TK |FLD2   |FLD4 |FLD5 |-          |20200515 |NNNN |0
FLD1     |SFK TK |FLD2   |FLD4 |FLD5 |-          |20200515 |NNNN |0

File output:

FLD1      SFK TK  FLD2    FLD4  FLD5  -           20200515  NNNN |406   RCO 301
FLD1      SFK TK  FLD2    FLD4  FLD5  -           20200515  NNNN |0
FLD1      SFK TK  FLD2    FLD4  FLD5  -           20200515  NNNN |0
Improve this question
4
  • 3
    What if there is only one | on the line? So the first one will also be the last one. Should that be removed or left alone?
    – terdon ♦
    Commented May 17, 2020 at 22:40
  • @terdon, never occurred to me, a good point, but a single "|" would never be the case for my input file.
    – et_phonehome
    Commented May 18, 2020 at 2:19
  • 2
    Are the number of pipe delimited fields fixed? If yes, sed 'y/|/\n/;s/\n/|/8;y/\n/ /' file can be used.
    – Rakesh Sharma
    Commented May 18, 2020 at 14:33
  • If the number of | is fixed, a nicely chosen combination of cut and paste would probably do it.
    – Vincent Fourmond
    Commented May 19, 2020 at 13:56

8 Answers 8

Reset to default
20 
sed ':a;/[|].*[|]/s/[|]/ /;ta' file
  • /[|].*[|]/: If line has two pipes,
  • s/[|]/ /: Substitute the first with a space.
  • ta: If a substitution was made, go back to :a.

Output:

$ sed ':a;/[|].*[|]/s/[|]/ /;ta' file
FLD1      SFK TK  FLD2    FLD4  FLD5  -           20200515  NNNN |406   RCO 301
FLD1      SFK TK  FLD2    FLD4  FLD5  -           20200515  NNNN |0
FLD1      SFK TK  FLD2    FLD4  FLD5  -           20200515  NNNN |0

As @steeldriver has remarked, you can use simply | instead of [|] in a basic regular expression (BRE), as is the case above. If you add the -E flag to sed, extended regular expression (ERE) is enabled and then you need to write [|] or \|.

Just for completeness, POSIX sed specification says that "Editing commands other than {...}, a, b, c, i, r, t, w, :, and # can be followed by a semicolon". Then, a compliant alternative to the above is:

sed -e ':a' -e '/[|].*[|]/s/[|]/ /;t a' file
Improve this answer
2
  • 2
    Slightly different, but very similar: sed -e ':again' -e 's/|\(.*\)|/ \1|/' -e 't again' file (also POSIX, as you need a newline after a label ordinarily).
    – Kusalananda ♦
    Commented May 17, 2020 at 19:43
  • The POSIX spec also indicates that the t command verb shall be separated from its argument by a single space: t a rather than ta
    – guest
    Commented May 18, 2020 at 5:09
10

A different approach from Quasímodo's explicit loop in sed:

$ sed 'h; s/.*|//; x; s/|[^|]*$//; y/|/ /; G; y/\n/|/' file
FLD1      SFK TK  FLD2    FLD4  FLD5  -           20200515  NNNN |406   RCO 301
FLD1      SFK TK  FLD2    FLD4  FLD5  -           20200515  NNNN |0
FLD1      SFK TK  FLD2    FLD4  FLD5  -           20200515  NNNN |0

For each line, this saves the line in the hold space with h and then removes everything on the line up to and including the last |. It then swaps in the original copy of the line and removes the last | and everything after it.

The pattern space now contains the original first portion of the line, and the hold space contains the last portion of the line.

The first y/// command replaces all remaining | with spaces. G appends the hold space to the end of the pattern space with a newline character in-between. The second y/// command transforms that newline character to a |, and we're done.

Doing a limited (fixed) number of s/// substitutions and using the faster y/// command when possible means that this runs quicker than the explicit loop variation (~2.3 seconds on 50 MiB data, compared to ~7.8 seconds on the same data with the loop, using GNU sed on my machine).

Interestingly, using a back reference in the explicit loop variation, like both me and Isaac did, slows it down even more (~33s with Isaac's variation, and ~29s with mine (in comments), on the same data set and under the same conditions as above).

Using awk, this almost replaces all | delimiters except the last one by a space. "Almost" since it inserts a space before the last |.

$ awk -F '|' 'BEGIN { OFS = " " } { $NF = "|" $NF; print }' file
FLD1      SFK TK  FLD2    FLD4  FLD5  -           20200515  NNNN  |406   RCO 301
FLD1      SFK TK  FLD2    FLD4  FLD5  -           20200515  NNNN  |0
FLD1      SFK TK  FLD2    FLD4  FLD5  -           20200515  NNNN  |0

It reads each line as a set of |-delimited fields, prepends a | character to the start of the last field, and prints the resulting record with spaces for field delimiters.

With taking the default behavior of awk into account (space is the default output field delimiter, and the input field delimiter is available as FS):

awk -F '|' '{ $NF = FS $NF; print }' file

or, slightly shorter, courtesy of @Isaac,

awk -F '|' '{ $NF = FS $NF }; 1' file
Improve this answer
0
8

Using Perl you could run something along the lines of

perl -pe 's/\|(?=.*\|)/ /g'     ex

where:

  • perl -pe action -- executes action and print
  • \|(?=.*\|) is a regular expression that matches | with a not consumed lookeahed (?=.*|) containing another |
Improve this answer
5

Posixly:

sed -e ':a' -e '/|\(.*|\)/s// \1/;ta' file
  • -e ':a' define a label (a) to jump.
  • -e ' start another script section.
  • /|\(.*|\)/ one regex to match two |, everything in the middle and to capture all in between and the last |.
  • s// \1/ replace everyting matched above with what was captured.
  • ;ta loop again.
  • ' file on the filename given.

To measure the speed of all the options (from faster to slower) you can use:

#! /bin/bash
TIMEFORMAT='run : %lR sec'

read -d '' str <<\END
FLD1     |SFK TK |FLD2   |FLD4 |FLD5 |-          |20200515 |NNNN |406   RCO 301
FLD1     |SFK TK |FLD2   |FLD4 |FLD5 |-          |20200515 |NNNN |0
FLD1     |SFK TK |FLD2   |FLD4 |FLD5 |-          |20200515 |NNNN |0'
END

n=${1:-100}; printf "$str"'%.0s\n' $(seq "$n") > file

time perl -pe 's/\|(?=.*\|)/ /g' file >/dev/null
time sed -E ':a;/\|.{1,}\|/s/\|/ /1;ta' file >/dev/null
time sed 'h; s/.*|//; x; s/|[^|]*$//; y/|/ /; G; y/\n/|/' file >/dev/null
time sed 's/\(.*\)|/\1\x00/;y/|/ /;s/\x00/|/1' file >/dev/null
time sed 's/\(.*\)|/\1\n/;y/|/ /;s/\n/|/1' file >/dev/null

To be used as:

$ ./testbash.sh 235000
run    : 0m07.676s sec
run    : 0m17.753s sec
run    : 0m22.074s sec
run    : 0m24.036s sec
run    : 0m24.047s sec
Improve this answer
2

Here are a couple of alternatives you can use. 

$ sed -e '
   s/|[^|]*$/\n&/
   s/\n|/\n/
   y/\n|/| /
' file

$ perl -pe 's/\|/ / until tr/|/|/ == 1' file

$ perl -pe 'my $k=tr/|/|/; s/\|/ / while $k-->1' file
Improve this answer
2

What about cheating this way ? No one said we needed just one sed:

sed -r 's/\|([^|]+)$/##\1/' | sed 's/|/ /g' | sed 's/##/|/'

Here, I used ## as a placeholder, replace only the last | of the line with it, replace all the other |, and then put back the last one. Just make sure you have no ## in your input.

Improve this answer
1
  • This is functionally the same as my answer, except mine replaces the last | with a newline, which cannot possibly occur within a line (so you don't need to include a disclaimer).  Note that you can have multiple sed commands within a single sed process / invocation.  It's very rare to need to pipe one sed process into another, let alone two others.
    – G-Man Says 'Reinstate Monica'
    Commented Jun 5, 2020 at 23:22
1

Without a loop:

sed 's/\(.*\)|/\1\
/; s/|/ /g; s/\
/|/'
  • Replace the last vertical bar in each line with a newline.  (This may be inserted into the pattern space, but is guaranteed not to be there initially.)
  • Replace all remaining vertical bars with spaces.
  • Replace the newline (inserted with the first command) with a vertical bar.
Improve this answer
1
  • That is exactly the last approach (the slowest) in my answer, sorry. :-)
    – user232326
    Commented May 19, 2020 at 1:47
1

If number of fields is always the same, cut +tr & paste with a shell that supports <():

paste -d\| <(cut -d\| -f1-8 file | tr \| ' ') <(cut -d\| -f9 file)

Or with GNU cut&paste:

paste -d\| <(cut -d\| --output-delimiter=' ' -f1-8 file) <(cut -d\| -f9 file)
Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .