0

I have a file like this:

$ cat file
    rep1 rep2
g1001_INpfu_DN44908_c3_g1 17.85 19.95
g10042/1330/2846_INpfu_DN43979_c0_g3 34.07 29.19
g1077/1457/278/278_INpfu_PRJNA287145_DN42983_c0_g1 20.69 21.64
g100/100_INpfu_DN43143_c0_g1 52.36 33.64
g79/79/79/79_INpfu_DN45068_c4_g1 58.83 74.58

The text before the first "_" is group number(s), like

  • gnumber_ ...
  • gnumber1/number2_ ...
  • gnumber1/number2/number3_ ...

etc.  (Spaces are added for illumination.)  Having a slash means multiple groups.  Sometimes there are duplicate group numbers within a line, and I want to remove them.

The expected results should be like:

    rep1 rep2
g1001_INpfu_DN44908_c3_g1 17.85 19.95
g10042/1330/2846_INpfu_DN43979_c0_g3 34.07 29.19
g1077/1457/278_INpfu_PRJNA287145_DN42983_c0_g1 20.69 21.64
g100_INpfu_DN43143_c0_g1 52.36 33.64
g79_INpfu_DN45068_c4_g1 58.83 74.58

In the last three lines, the same group numbers are removed, only keeping the unique group number(s).

0

You could try something like this, using sed:

$ sed -e :a -e 's:\([0-9][0-9]*\)/\1:\1:' -e ta file
rep1 rep2
g1001_INpfu_DN44908_c3_g1 17.85 19.95
g10042/1330/2846_INpfu_DN43979_c0_g3 34.07 29.19
g1077/1457/278_INpfu_PRJNA287145_DN42983_c0_g1 20.69 21.64
g100_INpfu_DN43143_c0_g1 52.36 33.64
g79_INpfu_DN45068_c4_g1 58.83 74.58

To handle partial matches such as g512/12/x or g512/5120/x without turning them into g512/x and g5120/x, you could add non-digit anchors either side:

sed -e :a -e 's:\([^0-9]\)\([0-9][0-9]*\)/\2\([^0-9]\):\1\2\3:' -e ta file

or slightly more readably using extended regular expressions

sed -E -e :a -e 's:([^0-9])([0-9]+)/\2([^0-9]):\1\2\3:' -e ta file

ex. given

$ cat file
    rep1 rep2
g1001_INpfu_DN44908_c3_g1 17.85 19.95
g10042/1330/2846_INpfu_DN43979_c0_g3 34.07 29.19
g1077/1457/278/278_INpfu_PRJNA287145_DN42983_c0_g1 20.69 21.64
g512/12_INpfu_DN43143_c0_g1 52.36 33.64
g100/100_INpfu_DN43143_c0_g1 52.36 33.64
g512/5120_INpfu_DN43143_c0_g1 52.36 33.64
g79/79/79/79_INpfu_DN45068_c4_g1 58.83 74.58

then

$ sed -E -e :a -e 's:([^0-9])([0-9]+)/\2([^0-9]):\1\2\3:' -e ta file
    rep1 rep2
g1001_INpfu_DN44908_c3_g1 17.85 19.95
g10042/1330/2846_INpfu_DN43979_c0_g3 34.07 29.19
g1077/1457/278_INpfu_PRJNA287145_DN42983_c0_g1 20.69 21.64
g512/12_INpfu_DN43143_c0_g1 52.36 33.64
g100_INpfu_DN43143_c0_g1 52.36 33.64
g512/5120_INpfu_DN43143_c0_g1 52.36 33.64
g79_INpfu_DN45068_c4_g1 58.83 74.58
6
  • Thank you so much! It works!!
    – ruby Sheng
    May 2 '20 at 3:09
  • That turns g512/12/x into g512/x though May 2 '20 at 7:42
  • @StéphaneChazelas thanks for catching that May 2 '20 at 10:49
  • That still turns g512/5120/x into g5120/x. May 3 '20 at 0:33
  • @G-ManSays'ReinstateMonica' yeah I'd like to delete this answer in light of the obviously better one but unfortunately that doesn't appear to be possible now that it's been accepted May 3 '20 at 0:39
-1

With perl:

perl -pe 's{^g(?:\d+/)*?(\d+)\K(?:/\1)+(?!\d)}{}' < your-file

It is strict in that it will remove the first sequence of repeating numbers only in the leading g/number/number[/number...]/number part and only if that part is not followed by a digit.

With sed, you could do the same with:

sed '
  \|^\(g\([[:digit:]]\{1,\}/\)*[[:digit:]]\{1,\}\).*| {
    h; # save a copy of original line
    s||\1:|; # remove all but the leading g/x/y/z
    s|\([g/]\)\([[:digit:]]\{1,\}\)\(/\2\)\{1,\}\([^[:digit:]]\)|\1\2\4|
    G; # append saved copy
    s|:\ng\([[:digit:]]\{1,\}/\)*[[:digit:]]\{1,\}||; # remove excess
  }' < your-file
2
  • @Isaac, yes the OP doesn't say only the consecutive repeats have to be trimmed, I possibly incorrectly derived that from his sample where it was the case. They don't say either that only one set of repeated numbers have to be trimmed. But the text of my answer does say what it does. May 2 '20 at 19:59
  • @Isaac. I don't find \# particularly less confusing than\|. I find a vertical bar works quite well as a delimiter, but feel free to use # if you prefer. I'll grant you that | doesn't work in vim's s command or could be confusing when using EREs. May 2 '20 at 20:00
-1

The replace section ends on the first _ The numbers to replace are separated by / There is a way to remove all repeated numbers (whether they are consecutive or not) like

  • last 278 in 278/278
  • last 278 in 278/1455/278
  • all 79 in 79/79/79/79 (all repetitions, the first is not a repetition).

With perl:

perl -pe 's//\1\2/ while /(?<!\d)(\d+)(?!\d)([^_]*)\/\1(?!\d)/ ' file
s//\1\2/     # substitute the regex by the captures of \1 and \2.
while        # for as long as there is a match in the regex.
/.../        # regex
(?<!\d)      # start on a non digit.
(\d+)        # contains several digits (at least one) (first capture: \1).
(?!\d)       # and ends in a non digit.
([^_]*)      # capture the characters between the repetitions (\2)
             # but do not extend the match after the `_`
\/\1         # match a `/` and what was captured by `\1`
             # that is: the repetition.
(?!\d)/      # followed by a non-digit.

For example, on this file:

rep1 rep2
g1001_INpfu_DN44908_c3_g1 17.85 19.95
g10042/1330/2846_INpfu_DN43979_c0_g3 34.07 29.19
g1077/1457/278/278_INpfu_PRJNA287145_DN42983_c0_g1 20.69 21.64
g1077/1457/278/1455/278_INpfu_PRJNA287145_DN42983_c0_g1 20.69 21.64
g1077/1457/278/1457/1088/278/1433_INpfu_PRJNA287145_DN42983_c0_g1 20.69 21.64
g100/100_INpfu_DN43143_c0_g1 52.36 33.64
g100/100/100_INpfu_DN43143_c0_g1 52.36 33.64
g2100/100_INpfu_DN43143_c0_g1 52.36 33.64
g79/79/79/79_INpfu_DN45068_c4_g1 58.83 74.58
g79/82/79/96/79/79_INpfu_DN45068_c4_g1 58.83 74.58

It will generate this output:

rep1 rep2
g1001_INpfu_DN44908_c3_g1 17.85 19.95
g10042/1330/2846_INpfu_DN43979_c0_g3 34.07 29.19
g1077/1457/278_INpfu_PRJNA287145_DN42983_c0_g1 20.69 21.64
g1077/1457/278/1455_INpfu_PRJNA287145_DN42983_c0_g1 20.69 21.64
g1077/1457/278/1088/1433_INpfu_PRJNA287145_DN42983_c0_g1 20.69 21.64
g100_INpfu_DN43143_c0_g1 52.36 33.64
g100_INpfu_DN43143_c0_g1 52.36 33.64
g2100/100_INpfu_DN43143_c0_g1 52.36 33.64
g79_INpfu_DN45068_c4_g1 58.83 74.58
g79/82/96_INpfu_DN45068_c4_g1 58.83 74.58

All repetitions have been erased.

Of course, it also works for your original file.

-1

The least amount of assumptions is to:

  • divide the input to fields separated by _ (as stated by the OP)
  • expect the the numbers are actually that, numbers, so a pattern split of

    patsplit( $1, v, /[0-9]{1,}/, s )
    

    where v is the array of values and s is the array of separators. Would extract all numbers and delimiters (the initial g and each /) of the first field (before the first _ as defined by the OP).

Then, to remove duplicated numbers (which might not be sorted) in the resulting v array, a double loop is required (if the numbers may be unsorted).

A double loop could be a problem for a big count of numbers (several thousands) since its worse case is n2. To improve such worse case to n*log(n) we would need to sort first.

This awk code does that:

awk 'BEGIN{OFS=FS="_"}
     NF>1{
         n=patsplit($1,v,/[0-9]{1,}/,s);            # count of numbers.
         for(i=1;i<=n;i++){                         # for each initial field and
             for(j=n;j>i;j--){                      # for each next field
                 if(v[j]!="" && v[i]==v[j]){        # check if they are equal
                     v[j]="";s[j-1]="";             # and remove the number and
                                                    # the preceding separator.
                 }
             }
         }
         str=""                                     # with a clean variable str
         for(i=1;i<=n;i++){                         # for all available numbers
             str=str "" sprintf("%s%s",s[i-1],v[i]) # rebuild the value of $1
         }
         $1=str;
     }
42                          # print. No, it is not the answer to
                            # Life, the Universe and everything
' file

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.