10

I have some files in the format

Y15-SUB-B04-P17-BK_M02734_4_000000000-ANNUF_1_1111_24724_4878;size=1;
Y15-SUB-B05-P22-LM_M02734_4_000000000-ANNUF_1_1111_20624_14973;size=1;
Y15-SUB-B05-P22-LM_M02734_4_000000000-ANNUF_1_1103_11326_10379;size=1;

I wish to replace every occurrence of the underscore (_) with a colon (:) EXCEPT for the first one. I want an output like this:

Y15-SUB-B04-P17-BK_M02734:4:000000000-ANNUF:1:1111:24724:4878;size=1;
Y15-SUB-B05-P22-LM_M02734:4:000000000-ANNUF:1:1111:20624:14973;size=1;
Y15-SUB-B05-P22-LM_M02734:4:000000000-ANNUF:1:1103:11326:10379;size=1;

I know I can use sed -i '' 's/_/:/g' old_file to replace ALL (or sed 's/_/:/g' old_file > new_file), and that I could add numbers to replace only the 2nd, 4th or so occurrence:

sed 's/_/:/2' old_file > new_file

But how to replace every occurrence on each line BUT the first?

3
  • Maybe a two step process? Replace all underscores with colons, then replace the first colon with an underscore?
    – 0xSheepdog
    May 31, 2019 at 18:20
  • 1
    Sure, that would work, only my file is some 14+ gb, and each replacement process takes about 1 hour, so if there were just one parsing step, that would be preferable. Thank you though.
    – Christoffer Bugge Harder
    May 31, 2019 at 18:27
  • 4
    Ah, so. Good info to have in the question, then. Things like requirements and limitations help us consider the entire situation.
    – 0xSheepdog
    May 31, 2019 at 18:31

7 Answers 7

Reset to default
10

Using GNU sed (other versions may behave differently, thanks glenn jackman):

 sed -i'' 's/_/:/2g' file

This will change all _ to : skipping the first occurrence on each line.

2
  • 3
    This is GNU sed. Other sed implementations act differently when given both 2 and g (e.g. the BSD-derived sed on MacOS gives an error)
    – glenn jackman
    May 31, 2019 at 19:15
  • 1
    ... and the POSIX specification says “If both g and n are specified, the results are unspecified.”
    – G-Man Says 'Reinstate Monica'
    Jun 3, 2019 at 3:25
6

Using Posix-sed constructs only we do like as:

$ sed -e '
     y/_/\n/
     s/\n/_/
     y/\n/:/
' inp.file

Based on the suggestions by Stephane, some more methods follow here:

$ perl -pe 's/(^\G.*?_)?.*?\K_/:/g' inp.file 

$ perl -pe 'my $n; s/_/$n++?":":$&/ge' inp.file 

$ perl -pe 's/_\K(.*)/$1 =~ y|_|:|r/e' inp.file
3
  • Or perl -pe 's/(^\G.*?_)?.*?\K_/:/g' or perl -pe 'my $n; s/_/$n++?":":$&/ge' to do it in one s/// invocation only.
    – Stéphane Chazelas
    Jun 1, 2019 at 6:43
  • This changes the first _ if the line has only one. Also, why the \G placeholder? The ^ is sufficient or at least makes the \G redundant.
    – Rakesh Sharma
    Jun 5, 2019 at 19:09
  • Yes, it was silly.
    – Stéphane Chazelas
    Jun 5, 2019 at 19:50
5

Is awk okay? You could use _ as the field separator, and print out:

<field 1>_<field 2>:<field n>:<field n+1>:...

Like this:

awk -F_ '{ printf("%s_%s", $1, $2); for (x = 3; x <=NF; x++) { printf(":%s", $x); }; printf("\n"); }'

If the structure is the same for each line you could hard-code the number of fields to avoid the loop (runs in about 2/3 of the time according to a very rough preliminary trial):

awk -F_ '{printf("%s_%s:%s:%s:%s:%s:%s:%s\n", $1, $2, $3, $4, $5, $6, $7, $8);}'
3
  • Thank you so much - I am unfortunately almost helpless in AWK, so I would have to be spoon-fed the exact command to make it work. if the first file is file1, and the second the output file (file2), then what should I write exactly? Sorry for my ineptitude.
    – Christoffer Bugge Harder
    May 31, 2019 at 18:30
  • 1
    Add file1 > file2 to the end of the command, to have it read in file1 and write out to file2. Just like you would with sed -- both parse text a line at a time!
    – user4443
    May 31, 2019 at 18:35
  • 1
    Note the sed commands in the other answers seem to be faster than either of these commands.
    – user4443
    May 31, 2019 at 18:50
5

With perl, to match the character _ and replace from the first instance on-wards as below. 

perl -pe '{$n=0}s{_}{++$n > 1 ? ":" : $&;}ge' file

The part s{_} identifies the _ within the line and if its the 2nd occurrence replace with : or replace with the same character($&)

5

Here is another simple awk script, (standard Linux gawk), no loops:

cat script.awk
match($0,/^[^_]*_/,a){ # match current line to first _ (including) into a[0] variable
   sub(a[0],"");       # remove a[0] from current line
   gsub("_",":");      # replace all _ to : in current line
   print a[0]""$0;     # output a[0] and current line
}

run:

awk -f script.awk input.txt

or:

awk 'match($0,/^[^_]*_/,a){sub(a[0],"");gsub("_",":");print a[0]""$0;}' input.txt
1
  • (1) sub() is not a great way to perform surgery on strings, because its first argument is a regular expression.  If the text before the first _ looks like a regular expression, this is likely to fail.  (2) You know that print a[0]""$0 can be written print a[0] $0 or print a[0]$0, right?  (It’s a matter of style.  I don’t particularly like the a[0]""$0 form, but it’s not wrong.)
    – G-Man Says 'Reinstate Monica'
    Jun 3, 2019 at 3:32
2

Given the sample input has no colon characters, a portable sed approach is to convert all underscores to colons, and then convert the first colon back to an underscore.

sed 's/_/:/g;s/:/_/' filename

@0xSheepdog sort of suggests this in the comments, but failed to point out this could be done with one invocation of sed.

1
  • I’m sure you know this, but, for the benefit of readers who aren’t as smart as you:  Warning: This will fail if any line of input has a colon (:) before the first underscore (_).
    – G-Man Says 'Reinstate Monica'
    Jun 3, 2019 at 3:39
2

A simple sed command will work fine for this:

Command: sed "s/_/:/2g" filename

Output

Y15-SUB-B04-P17-BK_M02734:4:000000000-ANNUF:1:1111:24724:4878;size=1;
Y15-SUB-B05-P22-LM_M02734:4:000000000-ANNUF:1:1111:20624:14973;size=1;
Y15-SUB-B05-P22-LM_M02734:4:000000000-ANNUF:1:1103:11326:10379;size=1;

Note: Suppose if you want to replace in the same file, use the below command:

sed  -i "s/_/:/2g" filename
1
  • 2
    This gives me this error message: > sed: 1: "s/:/_/2g": more than one number or 'g' in substitute flags
    – Christoffer Bugge Harder
    May 31, 2019 at 20:17

Not the answer you're looking for? Browse other questions tagged .