I have a file containing line-separated text:


From this, I want to extract characters 10 to 80, so:


I have found how to count the characters in a file:

  wc -m file

and how to get a number of characters per line:

 awk '{print substr($0,2,6)}' file

but I cannot find a way to get the characters 10 to 80.

Newlines do not count as characters.

Any ideas?

Yes, this is DNA, from a full genome. I have extracted this bit of DNA from a fasta file containing different scaffolds (10 and 11 in this case) using

 awk '/scaffold_10\>/{p=1;next} /scaffold_11/{p=0;exit} p'

Ultimately, I would like to have a simple command to get characters 100 to 800 (or something like that) from that specified scaffold.

EDIT: Question continues here: use gff2fasta instead of a bash script to get parts of DNA sequences out of a full genome

  • 2
    Is this, a file about DNA information?
    – Spandan
    Commented Apr 6, 2017 at 8:36
  • 3
    dd if=file bs=1 count=71 skip=9 status=none Commented Apr 6, 2017 at 8:37
  • @Spandan yes, this is DNA (whole genome, need to extract certain bits of it...)
    – gugy
    Commented Apr 6, 2017 at 8:39
  • Sato's comment gives a neat solution if the file only contains DNA. If it's a fasta-formatted file with one or several headers, his solution sadly won't work.
    – Kusalananda
    Commented Apr 6, 2017 at 8:41
  • 1
    @gugy, just to clarify, your count only includes the characters GCAT, but the source file still has newlines which should not be counted on output? Also, what size can the source file be?
    – ilkkachu
    Commented Apr 6, 2017 at 9:26

6 Answers 6

$ cat file1

check the length of each line

$ awk '{print length,$0}' file1

print the characters of 10-80

$ awk '{print substr($0,10,70)}' RS= file1

That assumes the input contains no empty line (RS= enables the paragraph mode where every record is a paragraph (paragraphs being delimited by sequences of empty lines)) and it implies loading the whole file in memory.

  • alright, that works. thx!
    – gugy
    Commented Apr 6, 2017 at 9:13
  • 1
    I wonder... Does awk read the whole file into memory when doing it that way?
    – Kusalananda
    Commented Apr 6, 2017 at 9:17
  • yes.. that's correct. if the file size is very big.. then we encounter with slowness.
    – Kamaraj
    Commented Apr 6, 2017 at 9:21
  • 1
    Also, I think you need to count one character for the newline in the middle. (Now you're missing the final T from the example.) That gets more annoying if the part to be picked is longer, and especially if the lines are of different lengths for some reason.
    – ilkkachu
    Commented Apr 6, 2017 at 9:26
  • 2
    @Kusalananda, no, RS= is the paragraph mode. For slurp mode, see Slurp-mode in awk? Commented Apr 6, 2017 at 14:40

I wonder how the line feed in the file should be handled. Does that count as a character or not?

If we just should take from byte 10 and print 71 bytes (A,C,T,G and linefeed) then Sato Katsura solution is the fastest (here assuming GNU dd or compatible for status=none, replace with 2> /dev/null (though that would also hide error messages if any) with other implementations):

 dd if=file bs=1 count=71 skip=9 status=none

If the line feed should be skipped then filter them out with tr -d '\n':

 tr -d '\n' < file | dd bs=1 count=70 skip=9 status=none

If the Fasta-header should be skipped it is:

 grep -v '^[;>]' file | tr -d '\n' | dd bs=1 count=70 skip=9 status=none

grep -v '^[;>]' file means skip all lines that start with ; or >.

  • I like this solution! can it be modified to only start counting after the occurence of >scaffold_2 ? the current answer ignores the lines starting with >, but counts from the beginning of the file, if I understand correctly?
    – gugy
    Commented Apr 6, 2017 at 13:19
  • It is best if you change the original question. Show exactly what the input and the output is and then someone would come with a solution. I'm not a sed expert but this one should do it: cat file | sed -n -e '/>scaffold_2/,$p' | grep -v '^>' | cut -zc 10-80 but I think grep could be avoided with the right sed option. (this cut-thing is good).
    – hschou
    Commented Apr 6, 2017 at 19:17

For bytes (so would also work for single-byte characters like in your sample):

dd bs=1 skip=9 count=71 < file 2> /dev/null

Or more efficiently with GNU dd:

dd iflag=fullblock,skip_bytes,count_bytes skip=9 count=71 status=none < file

For characters, with zsh:

  IFS= read -ru0 -k9 discard &&
    IFS= read -ru0 -k71 text &&
    printf %s $text
} < file

(won't print anything if there are fewer than 80 characters in the file).

ksh93 and bash have a -N option similar to zsh's -k, but they don't support the NUL characters and the bash one is buggy.

With GNU awk:

awk -v RS='.{1}' -v ORS= 'NR>=10 {print RT}; NR == 80 {exit}'

We use .{1} as . being a single character would not be treated as a regexp.

Another option is to convert to a character encoding that has a fixed number of bytes per character (and has all possible characters) like UTF-32LE that has 4 bytes per character:

< file iconv -t UTF-32LE |
   dd bs=4 skip=9 count=71 2> /dev/null |
   iconv -f UTF-32LE

If you don't mind bringing the entire contents into memory, and having the line unwrapped, you can use command substitution to read it in (thanks to George Vasiliou for the tr improvement!)

data=$( tr -d '\n' < inputfile )

then print from (zero-based) 10, for a length of 70 bytes:

printf "%s\n" "${data:9:70}"
  • 1
    Why not data=$( tr -d '\n' < inputfile ) to avoid mem double loading? Commented Apr 6, 2017 at 13:06
  • Only because I didn't think of it, @GeorgeVasiliou!
    – Jeff Schaller
    Commented Apr 6, 2017 at 13:10
perl -l -0777pe '
   my($start, $stop) = qw/10 80/; $delta = $stop - $start--;
   (undef, $_, $a) = unpack "A${start}A${delta}A*";
   $_ .= $1 while length() - y/\n/\n/ < $delta and $a =~ /(.)/g;
'  scaffolded_file_10

Assuming the newline characters are not significant to the data, but just formatting in the file (code not tested):

// {
  if ((NR>=int(start/linesize) && (NR<=int(end/linesize)) {
     from = NR==int(start/linesize) ? start % linesize : 0;
     to   = NR==int(end/linesize) ? (end % linesize)-from : linesize+1;
     print substr($0, from, to);
  if (NR==int(end/linesize)) exit;

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .