4

I have a file containing line-separated text:

GCAACACGGTGGGAGCACGTCAACAAGGAGTAATTCTTCAAGACCGTTCCAAAAACAGCATGCAAGAGCG
GTCGAGCCTAGTCCATCAGCAAATGCCGTTTCCAGCAATGCAAAGAGAACGGGAAGGTATCAGTTCACCG
GTGACTGCCATTACTGTGGACAAAAAGGGCACATGAAGAGAGACTGTGACAAGCTAAAGGCAGATGTAGC

From this, I want to extract characters 10 to 80, so:

TGGGAGCACGTCAACAAGGAGTAATTCTTCAAGACCGTTCCAAAAACAGCATGCAAGAGCG
GTCGAGCCT

I have found how to count the characters in a file:

  wc -m file

and how to get a number of characters per line:

 awk '{print substr($0,2,6)}' file

but I cannot find a way to get the characters 10 to 80.

Newlines do not count as characters.

Any ideas?

Yes, this is DNA, from a full genome. I have extracted this bit of DNA from a fasta file containing different scaffolds (10 and 11 in this case) using

 awk '/scaffold_10\>/{p=1;next} /scaffold_11/{p=0;exit} p'

Ultimately, I would like to have a simple command to get characters 100 to 800 (or something like that) from that specified scaffold.

EDIT: Question continues here: use gff2fasta instead of a bash script to get parts of DNA sequences out of a full genome

  • 2
    Is this, a file about DNA information? – Spandan Apr 6 '17 at 8:36
  • 3
    dd if=file bs=1 count=71 skip=9 status=none – Satō Katsura Apr 6 '17 at 8:37
  • @Spandan yes, this is DNA (whole genome, need to extract certain bits of it...) – gugy Apr 6 '17 at 8:39
  • Sato's comment gives a neat solution if the file only contains DNA. If it's a fasta-formatted file with one or several headers, his solution sadly won't work. – Kusalananda Apr 6 '17 at 8:41
  • 1
    @gugy, just to clarify, your count only includes the characters GCAT, but the source file still has newlines which should not be counted on output? Also, what size can the source file be? – ilkkachu Apr 6 '17 at 9:26
6

I wonder how the line feed in the file should be handled. Does that count as a character or not?

If we just should take from byte 10 and print 71 bytes (A,C,T,G and linefeed) then Sato Katsura solution is the fastest (here assuming GNU dd or compatible for status=none, replace with 2> /dev/null (though that would also hide error messages if any) with other implementations):

 dd if=file bs=1 count=71 skip=9 status=none

If the line feed should be skipped then filter them out with tr -d '\n':

 tr -d '\n' < file | dd bs=1 count=70 skip=9 status=none

If the Fasta-header should be skipped it is:

 grep -v '^[;>]' file | tr -d '\n' | dd bs=1 count=70 skip=9 status=none

grep -v '^[;>]' file means skip all lines that start with ; or >.

  • I like this solution! can it be modified to only start counting after the occurence of >scaffold_2 ? the current answer ignores the lines starting with >, but counts from the beginning of the file, if I understand correctly? – gugy Apr 6 '17 at 13:19
  • It is best if you change the original question. Show exactly what the input and the output is and then someone would come with a solution. I'm not a sed expert but this one should do it: cat file | sed -n -e '/>scaffold_2/,$p' | grep -v '^>' | cut -zc 10-80 but I think grep could be avoided with the right sed option. (this cut-thing is good). – hschou Apr 6 '17 at 19:17
7
$ cat file1
GCAACACGGTGGGAGCACGTCAACAAGGAGTAATTCTTCAAGACCGTTCCAAAAACAGCATGCAAGAGCG
GTCGAGCCTAGTCCATCAGCAAATGCCGTTTCCAGCAATGCAAAGAGAACGGGAAGGTATCAGTTCACCG
GTGACTGCCATTACTGTGGACAAAAAGGGCACATGAAGAGAGACTGTGACAAGCTAAAGGCAGATGTAGC

check the length of each line

$ awk '{print length,$0}' file1
70 GCAACACGGTGGGAGCACGTCAACAAGGAGTAATTCTTCAAGACCGTTCCAAAAACAGCATGCAAGAGCG
70 GTCGAGCCTAGTCCATCAGCAAATGCCGTTTCCAGCAATGCAAAGAGAACGGGAAGGTATCAGTTCACCG
70 GTGACTGCCATTACTGTGGACAAAAAGGGCACATGAAGAGAGACTGTGACAAGCTAAAGGCAGATGTAGC

print the characters of 10-80

$ awk '{print substr($0,10,70)}' RS= file1
TGGGAGCACGTCAACAAGGAGTAATTCTTCAAGACCGTTCCAAAAACAGCATGCAAGAGCG
GTCGAGCC

That assumes the input contains no empty line (RS= enables the paragraph mode where every record is a paragraph (paragraphs being delimited by sequences of empty lines)) and it implies loading the whole file in memory.

  • alright, that works. thx! – gugy Apr 6 '17 at 9:13
  • 1
    I wonder... Does awk read the whole file into memory when doing it that way? – Kusalananda Apr 6 '17 at 9:17
  • yes.. that's correct. if the file size is very big.. then we encounter with slowness. – Kamaraj Apr 6 '17 at 9:21
  • 1
    Also, I think you need to count one character for the newline in the middle. (Now you're missing the final T from the example.) That gets more annoying if the part to be picked is longer, and especially if the lines are of different lengths for some reason. – ilkkachu Apr 6 '17 at 9:26
  • 2
    @Kusalananda, no, RS= is the paragraph mode. For slurp mode, see Slurp-mode in awk? – Stéphane Chazelas Apr 6 '17 at 14:40
5

For bytes (so would also work for single-byte characters like in your sample):

dd bs=1 skip=9 count=71 < file 2> /dev/null

Or more efficiently with GNU dd:

dd iflag=fullblock,skip_bytes,count_bytes skip=9 count=71 status=none < file

For characters, with zsh:

{
  IFS= read -ru0 -k9 discard &&
    IFS= read -ru0 -k71 text &&
    printf %s $text
} < file

(won't print anything if there are fewer than 80 characters in the file).

ksh93 and bash have a -N option similar to zsh's -k, but they don't support the NUL characters and the bash one is buggy.

With GNU awk:

awk -v RS='.{1}' -v ORS= 'NR>=10 {print RT}; NR == 80 {exit}'

We use .{1} as . being a single character would not be treated as a regexp.

Another option is to convert to a character encoding that has a fixed number of bytes per character (and has all possible characters) like UTF-32LE that has 4 bytes per character:

< file iconv -t UTF-32LE |
   dd bs=4 skip=9 count=71 2> /dev/null |
   iconv -f UTF-32LE
3

If you don't mind bringing the entire contents into memory, and having the line unwrapped, you can use command substitution to read it in (thanks to George Vasiliou for the tr improvement!)

data=$( tr -d '\n' < inputfile )

then print from (zero-based) 10, for a length of 70 bytes:

printf "%s\n" "${data:9:70}"
  • 1
    Why not data=$( tr -d '\n' < inputfile ) to avoid mem double loading? – George Vasiliou Apr 6 '17 at 13:06
  • Only because I didn't think of it, @GeorgeVasiliou! – Jeff Schaller Apr 6 '17 at 13:10
2
perl -l -0777pe '
   my($start, $stop) = qw/10 80/; $delta = $stop - $start--;
   (undef, $_, $a) = unpack "A${start}A${delta}A*";
   $_ .= $1 while length() - y/\n/\n/ < $delta and $a =~ /(.)/g;
'  scaffolded_file_10
2

Assuming the newline characters are not significant to the data, but just formatting in the file (code not tested):

BEGIN {
  linesize=70;
  start=10;
  end=80;
}
// {
  if ((NR>=int(start/linesize) && (NR<=int(end/linesize)) {
     from = NR==int(start/linesize) ? start % linesize : 0;
     to   = NR==int(end/linesize) ? (end % linesize)-from : linesize+1;
     print substr($0, from, to);
  }
  if (NR==int(end/linesize)) exit;
}

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.