1

I have a txt file which is formatted as 250,000 * 3600 (3600 rows and 250,000 column) I am going to split it into 3600 small txt which each file is 250,000 * 1.

I understand the split can basically split large file into small ones with certain size. If I followed the same technique, I would get the same size of files but each file can be either 249,999 * 1 or 250,001 *1.

Can anyone help me on this?

5
  • I don't understand what you want to do. Do you want to write each line into a separate file? Does “each file can be either 249,999 * 1 or 250,001 *1” mean that the lines have slightly different lengths, or that you want to cut at places other than line breaks?
    – Gilles 'SO- stop being evil'
    Nov 6, 2012 at 23:13
  • Yes, each line into a separate file. But I want each file is formatted exactly as 250,000 * 1, no more no less. If we only specify the bytes, say 2 kB per file, then we end up with getting 249,999 * 1 and 250,001 * 1 , which I don't want.
    – John_Amery
    Nov 7, 2012 at 0:31
  • I still don't get it. Are all your lines the same length or not? If they are, then split -b 250000 does put one line in each file.
    – Gilles 'SO- stop being evil'
    Nov 7, 2012 at 0:39
  • can you tell us how a file with 5 columns * 3 rows is split in the wrong way if you use split?
    – miracle173
    Nov 7, 2012 at 7:55
  • Thanks Gilles. Yes, each line is of the same length. But There are too many lines. It always shows 'too many files' and stops.
    – John_Amery
    Nov 7, 2012 at 17:42

2 Answers 2

Reset to default
2

I would suggest writing a small script:

NR_LINES=$(wc -l your-file | awk '{ print $1 }')
for line in {1..$NR_LINES}; do
  head -n $line your-file | tail -n 1 > your-file.part$line
done

This saves the number of lines of your file in the variable NR_LINES. Maybe your file consists of more lines than those you are interested in. In this case you should set the variable for yourself. The script iterates from 1 to the last line in your file. For each run it prints the first n lines ($line). This is piped to tail, which only prints the last line and saves the result in a file.

1
  • Thanks. I would like to cut myfile in every iterate and save every piece. I run this : NR_LINES=$(wc -l data.txt | awk '{ print $1 }') for line in {1..$NR_LINES}; do head -1 $line data.txt | tail -1 1 > data.txt.part$line done
    – John_Amery
    Nov 7, 2012 at 0:21
1

You can use csplit to split into each line by using a pattern /^/ matching start of line. The argument {*} repeats the pattern until the end of input.

csplit --elide-empty-files input.txt '/^/' '{*}'

Without --elide-empty-files (or -z), the file with index 0 wold be empty.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .