Manual input of data makes a database polluted with records containing multiple newline characters. For good records delimited by double quotes in the beginning and at the end in a huge 60GB flat file with only a single column, they should always span only one line like this:

"Complete sequences of numerous mitochondrial, many prokaryotic, and several nuclear genomes are now available. "

For bad records, they span indefinite number of multiple lines like this:

"Current smoking was strongly and inversely associated with high-risk

patterns, after adjustment for concomitant risk factors. Relative to never

smokers, current smokers were significantly less likely to have a high-risk

pattern. "

These multi-line records prohibit downstream file splitting by the UNIX command split. split cannot intelligently recognize those multiple lines as a single record and this can lead to splitting a single record into separate files. The Perl below is too slow to merge those lines for the bad records first for this huge file before splitting, as $count cannot be printed after waiting for more than 2 hours.

open(INFO, $file) or die("Could not open $file.");
open(OUT, ">out") or die("Could not open $file.");

$mergedline = "";
foreach $line (<INFO>)  {
    print $count++;
    if ($line =~ /^".*"\n$/) {
                print OUT $line;
                $mergedline = "";
        } elsif ($line =~ /"\n$/) {
                print OUT $mergedline;
                $mergedline = "";
        } else {
                chomp $line;
                $mergedline .= $line;

Any handy UNIX command to solve this problem so that the output file is "clean" with only single-line records that can be processed by split?

sed appears to be an option, but neither of the following posts answer this question:




because their patterns of these posts are too regular and constant.

closed as unclear what you're asking by Satō Katsura, phk, Rui F Ribeiro, user34720, G-Man May 31 '17 at 20:18

Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • 1
    What is a "record"? What does the actual input file look like and what output do you expect? How can we identify a record? Is it any quoted string with no " characters inside it? Please edit your question and clarify. Also explain what Perl command you tried. – terdon May 31 '17 at 12:58
  • 1
    You have strings of text / sentences / paragraphs that are sometimes split in the middle with newlines? How come Perl is too slow to to merge them? It might be, if you try to deal with the full 60 GB twice, once to merge and once to split, but that should be doable in a pipeline. – ilkkachu May 31 '17 at 13:00
  • Can you modify the file in place first with something like sed and then split it? – cutrightjm May 31 '17 at 13:21
  • I feel like I've just read Chapter 2 of a three-chapter book. Please tell us the whole story. What is "this problem"? Presenting code you've tried is good — don't remove it from the question — but explain what you are doing in English also. – G-Man May 31 '17 at 20:18

Using sed to join the splitted lines only

sed ':a
N;s/\n/ /;ba' input >> output

takes 6 seconds for a 10 MB file on my system. This would be 10 hours for 60 GB.

bbe is a little faster

bbe -b '/"/:/"/' -o output -e 'y/\n/ /' input

but still takes 4 seconds.

I'm afraid those scripting languages are not the tool to perform good on extremly large files. How about writing a small program in C?


example using gawk:

awk 'BEGIN {RS = "\"\n"} {++n; print $0"\""> n".txt"}' input

This says split the file input on any sequence of " followed by a newline (\n). This will ignore newlines that don't immediately follow a quote mark, preserving multiline records. In this example, output is written to a text file, but if you removed the > n".txt" portion, you could send records to a pipeline instead.

  • This command splits records into independent files but they remain multiple lined. Yet, this is already good enough for post-processing and therefore should also be accepted as a solution. – hubx Jun 1 '17 at 6:57

Your Perl is slow because of the for loop being used to read in the file. You should really be using the while loop, since the for loop loads the whole file in memory at one go. That's why it's taking forever to print $count.

perl -ne '
   print,next if /^".*"$/m or /"$/m;
   chomp, $_ .= <>, redo unless eof;
' gene.data

Not the answer you're looking for? Browse other questions tagged or ask your own question.