unix: get characters 10 to 80 in a file

Question

I have a file containing line-separated text:

GCAACACGGTGGGAGCACGTCAACAAGGAGTAATTCTTCAAGACCGTTCCAAAAACAGCATGCAAGAGCG
GTCGAGCCTAGTCCATCAGCAAATGCCGTTTCCAGCAATGCAAAGAGAACGGGAAGGTATCAGTTCACCG
GTGACTGCCATTACTGTGGACAAAAAGGGCACATGAAGAGAGACTGTGACAAGCTAAAGGCAGATGTAGC

From this, I want to extract characters 10 to 80, so:

TGGGAGCACGTCAACAAGGAGTAATTCTTCAAGACCGTTCCAAAAACAGCATGCAAGAGCG
GTCGAGCCT

I have found how to count the characters in a file:

  wc -m file

and how to get a number of characters per line:

 awk '{print substr($0,2,6)}' file

but I cannot find a way to get the characters 10 to 80.

Newlines do not count as characters.

Any ideas?

Yes, this is DNA, from a full genome. I have extracted this bit of DNA from a fasta file containing different scaffolds (10 and 11 in this case) using

 awk '/scaffold_10\>/{p=1;next} /scaffold_11/{p=0;exit} p'

Ultimately, I would like to have a simple command to get characters 100 to 800 (or something like that) from that specified scaffold.

EDIT: Question continues here: use gff2fasta instead of a bash script to get parts of DNA sequences out of a full genome

Answer 1

$ cat file1
GCAACACGGTGGGAGCACGTCAACAAGGAGTAATTCTTCAAGACCGTTCCAAAAACAGCATGCAAGAGCG
GTCGAGCCTAGTCCATCAGCAAATGCCGTTTCCAGCAATGCAAAGAGAACGGGAAGGTATCAGTTCACCG
GTGACTGCCATTACTGTGGACAAAAAGGGCACATGAAGAGAGACTGTGACAAGCTAAAGGCAGATGTAGC

check the length of each line

$ awk '{print length,$0}' file1
70 GCAACACGGTGGGAGCACGTCAACAAGGAGTAATTCTTCAAGACCGTTCCAAAAACAGCATGCAAGAGCG
70 GTCGAGCCTAGTCCATCAGCAAATGCCGTTTCCAGCAATGCAAAGAGAACGGGAAGGTATCAGTTCACCG
70 GTGACTGCCATTACTGTGGACAAAAAGGGCACATGAAGAGAGACTGTGACAAGCTAAAGGCAGATGTAGC

print the characters of 10-80

$ awk '{print substr($0,10,70)}' RS= file1
TGGGAGCACGTCAACAAGGAGTAATTCTTCAAGACCGTTCCAAAAACAGCATGCAAGAGCG
GTCGAGCC

That assumes the input contains no empty line (RS= enables the paragraph mode where every record is a paragraph (paragraphs being delimited by sequences of empty lines)) and it implies loading the whole file in memory.

Answer 2

I wonder how the line feed in the file should be handled. Does that count as a character or not?

If we just should take from byte 10 and print 71 bytes (A,C,T,G and linefeed) then Sato Katsura solution is the fastest (here assuming GNU dd or compatible for status=none, replace with 2> /dev/null (though that would also hide error messages if any) with other implementations):

 dd if=file bs=1 count=71 skip=9 status=none

If the line feed should be skipped then filter them out with tr -d '\n':

 tr -d '\n' < file | dd bs=1 count=70 skip=9 status=none

If the Fasta-header should be skipped it is:

 grep -v '^[;>]' file | tr -d '\n' | dd bs=1 count=70 skip=9 status=none

grep -v '^[;>]' file means skip all lines that start with ; or >.

Answer 3

For bytes (so would also work for single-byte characters like in your sample):

dd bs=1 skip=9 count=71 < file 2> /dev/null

Or more efficiently with GNU dd:

dd iflag=fullblock,skip_bytes,count_bytes skip=9 count=71 status=none < file

For characters, with zsh:

{
  IFS= read -ru0 -k9 discard &&
    IFS= read -ru0 -k71 text &&
    printf %s $text
} < file

(won't print anything if there are fewer than 80 characters in the file).

ksh93 and bash have a -N option similar to zsh's -k, but they don't support the NUL characters and the bash one is buggy.

With GNU awk:

awk -v RS='.{1}' -v ORS= 'NR>=10 {print RT}; NR == 80 {exit}'

We use .{1} as . being a single character would not be treated as a regexp.

Another option is to convert to a character encoding that has a fixed number of bytes per character (and has all possible characters) like UTF-32LE that has 4 bytes per character:

< file iconv -t UTF-32LE |
   dd bs=4 skip=9 count=71 2> /dev/null |
   iconv -f UTF-32LE

Answer 4

If you don't mind bringing the entire contents into memory, and having the line unwrapped, you can use command substitution to read it in (thanks to George Vasiliou for the tr improvement!)

data=$( tr -d '\n' < inputfile )

then print from (zero-based) 10, for a length of 70 bytes:

printf "%s\n" "${data:9:70}"

Answer 5

perl -l -0777pe '
   my($start, $stop) = qw/10 80/; $delta = $stop - $start--;
   (undef, $_, $a) = unpack "A${start}A${delta}A*";
   $_ .= $1 while length() - y/\n/\n/ < $delta and $a =~ /(.)/g;
'  scaffolded_file_10

Answer 6

Assuming the newline characters are not significant to the data, but just formatting in the file (code not tested):

BEGIN {
  linesize=70;
  start=10;
  end=80;
}
// {
  if ((NR>=int(start/linesize) && (NR<=int(end/linesize)) {
     from = NR==int(start/linesize) ? start % linesize : 0;
     to   = NR==int(end/linesize) ? (end % linesize)-from : linesize+1;
     print substr($0, from, to);
  }
  if (NR==int(end/linesize)) exit;
}