4

I have a huge (ca. 20G) text file which contains millions of passages (a.k.a. paragraphs) with headers. Headers and paragraphs are always one line each, e.g.,

Sunshine
This is a sunny day.
Darkness
A cave is a dark place.

What I try to come up with is a terminal command which goes through the text and adds a '>' in front of every header, i.e., every odd-numbered line (lines 1, 3, 5, …), e.g.,

>Sunshine
This is a sunny day.
>Darkness
A cave is a dark place.

Any ideas?

If this is relevant: the above text was just an example. Most of the headers are MD5s, followed by a DNA sequence ('paragraph'), e.g.,

0002ebd9ca12d6b69dfc3066356fc299
CATTAACCATTGGATACCTTCGGGTATATCCCATCCGTGTCTACATACTCTTGTTGCTTTGGCAGGCCGTGGTCACACACTGTGGGCTATGCCTGCATGTGCCTGCCAGAGGACCA

… which I'm trying to convert to

>0002ebd9ca12d6b69dfc3066356fc299
CATTAACCATTGGATACCTTCGGGTATATCCCATCCGTGTCTACATACTCTTGTTGCTTTGGCAGGCCGTGGTCACACACTGTGGGCTATGCCTGCATGTGCCTGCCAGAGGACCA

  • 1
    Are blank lines in your input? – cuonglm Sep 7 '14 at 16:04
  • 1
    You say "in front", but the > is added as the 2nd character of your examples. Can you clarify where you want it and also, are the single quotes in your examples part of the text file or just a formatting slip-up in your post? – garethTheRed Sep 7 '14 at 16:14
  • sorry, for the confusion. I only want > in front of every second line beginning with the first. > was removed when I created this post and the text was put in the greyed out box. so I used ' to be able to show > – BSP Sep 7 '14 at 16:29
  • no blank lines in my file – BSP Sep 7 '14 at 16:44
4

To edit every other (a.k.a. every second) line, starting with the first, with GNU sed, do

sed '1~2s/^/>/' your_file

This will write the modified file to the standard output.  I.e., if you type just the above, the modified file will display on the screen.  You can put this into a new file by redirecting the output with a >; e.g.,

sed '1~2s/^/>/' your_file > your_new_file

or, if you want to modify your existing file, use -i:

sed -i '1~2s/^/>/' your_file
| improve this answer | |
  • 1
    sed -i on a 20g file is a really terrible idea. – mikeserv Jun 29 '15 at 9:49
3

POSIXly:

sed 's/^/>/;n' < file.in > file.out
| improve this answer | |
2

Another POSIX answer:

paste -d'>\n' /dev/null - - <infile

It gets:

>Sunshine
This is a sunny day.
>Darkness
A cave is a dark place.
| improve this answer | |
0
sed '1,${s/^/>/g;n;n;n}' filename

Testing

cat filename
'Sunshine

'This is a sunny day.

'Darkness

'A cave is a dark place

'Sunshine

'This is a sunny day.

'Darkness

'A cave is a dark place

'Sunshine

'This is a sunny day.

'Darkness

'A cave is a dark place

After I run the command, I get the output as,

sed '1,${s/^/>/g;n;n;n}' new
>'Sunshine

'This is a sunny day.

>'Darkness

'A cave is a dark place

>'Sunshine

'This is a sunny day.

>'Darkness

'A cave is a dark place

>'Sunshine

'This is a sunny day.

>'Darkness

'A cave is a dark place
| improve this answer | |
  • Sorry for the late response. I am on a Mac and had to change to a linux system to check your command. it seems to not work for me. a '>' was added to the first line and then after 7 lines or so – BSP Sep 7 '14 at 16:20
  • @Ramesh @BSP , please at first clean your data from blank lines via : egrep -v "^$" yoursourcedata > cleaneddata.txt – PersianGulf Sep 7 '14 at 18:31
0

With awk you could try something like,

awk 'NR % 4 == 1 {sub(/^/,">")} {print}' filename

References

https://stackoverflow.com/questions/2099471/add-a-prefix-string-to-beginning-of-each-line

| improve this answer | |
  • it seems to not work for me. a '>' was added to the first line and then after 7 lines or so. as i am new to unix commands i also dont see why.. – BSP Sep 7 '14 at 16:27
0

With no blank lines between each line and no ' character at the start:

$ awk '{print ((NR%2)? ">":"") $0}' passages.txt

gives:

>Sunshine
This is a sunny day.
>Darkness
A cave is a dark place.

Also, going by your responses to all the answers here, your input file isn't single lines with a Line Feed character at the end (\n). It might be worth checking it's source.

| improve this answer | |
  • Thanks for the effort. I tried both commands. My file is not double spaced. The first one adds the '>' only every second header. The second one also every second header but also adds a an additional line break – BSP Sep 7 '14 at 16:24
0

You can use Vim in Ex mode:

ex -sc '%s/\v(.*\n){2}/> &/|x' file
  1. % select all lines

  2. s substitute

  3. \v turn on magic

  4. x save and close

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.