2

I have a list like this:

$$<002L_tbfl
Putative transcription factor 001R;
GO:0006355
GO:0046782
GO:0006351
IPR007031
$$<002L_FRG3G
Uncharacterized protein 002L;
GO:0033644
GO:0016021
IPR004251

I want every $$< to start a new line, with the following entries on the same line (tab-separated), until $$< appears again. Like this:

$$<002L_tbfl    Putative transcription factor 001R; GO:0006355  GO:0046782  GO:0006351  IPR007031
$$<002L_FRG3G   Uncharacterized protein 002L;   GO:0033644  GO:0016021  IPR004251

My method so far has been:

tr '\n' '\t'   < stage1 > stage2
sed 's/$$</\n/g' stage2 > stage3

The thing is, the above works totally fine on small files, but on my 4 GB file it seems to work, then returns a blank file within a fraction of the time with no errors or messages.
I've also tried tr '$$<' '\n' and that doesn't do it; it produces a weird file.

  • I edited your question; please verify that I didn't change the meaning (and fix it if I did).  P.S. The commands that you show don't produce the results that you show on the input data that you show — they delete the $$< strings. – Scott Sep 11 '16 at 3:34
  • Does tr '\n' '\t' < inputfile |sed 's/$$</\n/g'|tail -n +2 do what you want? – Julie Pelletier Sep 11 '16 at 3:39
1

Here's how to to it in sed:

sed -n '/$$</! H; /$$</{x; s/\n/\t/gp}; ${x; s/\n/\t/gp}' stage1 > stage3

In pieces:

  • sed -n means don't print default output (i.e., the massaged input); print only when there is a p command.
  • /$$</! H means when you see a line not containing $$<, append it to the "hold space" (i.e., a staging area).  The ! inverts the normal logic, meaning "do this for lines that don't satisfy this condition".  If you need to ignore $$< that occurs in the middle of a line, change this (and the next command) to use /^$$</.  (If you need to process $$< in the middle of a line some different way — e.g., insert a newline before it — edit your question to say so.)

    If you append a line to the hold space when there's already something there (in the hold space), sed inserts a newline between them, so this will build up text in the hold space that looks like this:

    $$<002L_tbflnewlinePutative transcription factor 001R;newlineGO:0006355…

    The hold space, like the "pattern space" (the ordinary, working line buffer) generally will not have an explicit newline at the end (it's implicit).  Of course it is possible to explicitly insert newlines into the spaces.

  • /$$</{…} means do the commands inside the braces on lines that contain $$<.

    • x means exchange the contents of the hold space and the pattern space.
    • s/\n/\t/gp means — well, it's obvious, isn't it? — it means replace newline with tab (in the pattern space) globally and print the result.

    When this command reads the first line of your input (which does contain $$<), the x moves that line ($$<002L_tbfl) from the pattern space into the hold space and moves the previous contents of the hold space into the pattern space.  But, since the initial content of the hold space is nothing, that means there is nothing for the s command to work on.  Subsequently, when you see $$< (e.g., on line 7), it brings the text with the embedded newlines (as illustrated above) into the pattern space and (as described) replaces all the newlines with tabs and prints the result.

  • ${…} means do the commands inside the braces when you get to the end of the input.  These are the same commands that we do when we see a $$<, to flush out the last line (i.e., the last bunch of lines) from the hold space.

Warning: this is not guaranteed to work on POSIX sed.  I have tested it on GNU sed.

1
$ cat ip.txt 
$$<002L_tbfl
Putative transcription factor 001R;
GO:0006355
GO:0046782
GO:0006351
IPR007031
$$<002L_FRG3G
Uncharacterized protein 002L;
GO:0033644
GO:0016021
IPR004251

$ perl -ne 'chomp if !eof; if($. > 1){print /\$\$</ ? "\n" : "\t"} print' ip.txt 
$$<002L_tbfl    Putative transcription factor 001R; GO:0006355  GO:0046782  GO:0006351  IPR007031
$$<002L_FRG3G   Uncharacterized protein 002L;   GO:0033644  GO:0016021  IPR004251
  • chomp if !eof remove newline from all input lines except last line of file
  • if($. > 1) input line number greater than 1
  • print /\$\$</ ? "\n" : "\t" add newline if line matches $$< else add tab
  • print prints the input line
0

Presumably there's a 32 bit limitation in effect, so only stream processing is possible. You can use awk, as in

awk 'NR==1 {printf "%s",$0; next;} $1~/^\$\$</ {printf "\n%s",$0; next;} {printf "\t%s",$0;}' < file

That will print all input lines in succession without newlines, except that lines (after the first) starting with $$< gain initial newlines.

Possibly you want a final newline, which would need and END stanza. See man awk for those variations.

0

You can simply save the 1st line that matches in the hold buffer and delete it; accumulate lines that don't match in the hold space and delete them if not the last line then on the remaining lines exchange buffers and translate all newlines to spaces:

sed '1{h;d;};/$$</!{H;$!d;};x;y/\n/ /' file

Note that the space used in the y command is a literal tab.

0

Works with Mawk and Gawk 3.x on Ubuntu 12, which support RS being a regex.

$ awk 'BEGIN { RS="\\$\\$<"; FS="\n"; OFS="\t" } NF && $1="$$<"$1' data

Output:

$$<002L_tbfl    Putative transcription factor 001R; GO:0006355  GO:0046782  GO:0006351  IPR007031   
$$<002L_FRG3G   Uncharacterized protein 002L;   GO:0033644  GO:0016021  IPR004251

We simply use $$< as the record separator and newline as the field separator.

This means that:

  • we get an empty record since the input begins with our record separator. We eliminate this by using NF as a condition: number of fields has to be nonzero.
  • $$< is removed from the input. We put it back into $1.

To print the fields with tabs in between, we set up the tab as the output field separator (OFS). { print } is the default action for a pattern, so we omit it.

The fact that we modify $1 also has the side effect of updating the record variable $0 by joining all the fields with OFS. Without this update, the original records would be printed verbatim, newlines and all.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.