I have these three files:

file.txt.7z = 5.4GB
file-1.txt.7z = 251M
file-2.txt.7z = 7.7M

And they are the only files in a directory:

$ tree
├── file.txt.7z
├── file-1.txt.7z
└── file-2.txt.7z

I want to

  • unzip the files
  • combine them into one file
  • split that combined file into files of 500,000 lines
  • have as a result a number of files with a ".txt" extention

Right now I am achieving it this way:

p7zip -d "*.txt.7z"
cat file-1.txt >> file.txt
rm file-1.txt
cat file-2.txt >> file.txt
rm file-2.txt
split -l 500000 file.txt
for f in *; do mv "$f" "$f.txt"; done

How could I achieve this in a more elegant way?

  • 1
    Streamed archive format is more useful here
    – eri
    Oct 18 '17 at 20:13

7za + split solution (single pipeline):

7za e "*.7z" -so 2> /dev/null | split -l500000 --additional-suffix=".txt" --numeric-suffixes=1 - "file"

-- 7za options:

  • e - extract/decompress archive(s)

  • -so - write the content to STDOUT

-- split options:

  • --additional-suffix=".txt" - append suffix .txt to all resulting filenames

  • --numeric-suffixes=1 - use numeric suffixes starting at 1

  • - (hyphen) - read data from STDIN (standard input)

  • "file" - the common prefix for all resulting filenames

The above command will result to files with the following naming format: file01.txt, file02.txt etc.

  • It's a detail, but won't "*.7z" produce file-1.txt file-2.txt file.txt, i.e. the wrong order? Oct 19 '17 at 7:35

You could use pipes and the --filter option of split after decompression:

p7zip -d *.txt.7z
cat file.txt file-1.txt file-2.txt | split -l 500000 --filter='> $FILE.txt'
rm file*

Here is the documentation for the --filter option:

     With this option, rather than simply writing to each output file,
     write through a pipe to the specified shell COMMAND for each output
     file.  COMMAND should use the $FILE environment variable, which is
     set to a different output file name for each invocation of the
     command.  For example, imagine that you have a 1TiB compressed file
     that, if uncompressed, would be too large to reside on disk, yet
     you must split it into individually-compressed pieces of a more
     manageable size.  To do that, you might run this command:

          xz -dc BIG.xz | split -b200G --filter='xz > $FILE.xz' - big-

     Assuming a 10:1 compression ratio, that would create about fifty
     20GiB files with names ‘big-aa.xz’, ‘big-ab.xz’, ‘big-ac.xz’, etc.

If you need to keep a file with all the output, you could use tee, which copies standard input to standard output and to the file given as argument.

cat file.txt file-1.txt file-2.txt |
    tee all.txt |
    split -l 50000 --filter='> $FILE.txt'

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.