2

I have these three files:

file.txt.7z = 5.4GB
file-1.txt.7z = 251M
file-2.txt.7z = 7.7M

And they are the only files in a directory:

$ tree
.
├── file.txt.7z
├── file-1.txt.7z
└── file-2.txt.7z

I want to

  • unzip the files
  • combine them into one file
  • split that combined file into files of 500,000 lines
  • have as a result a number of files with a ".txt" extention

Right now I am achieving it this way:

p7zip -d "*.txt.7z"
cat file-1.txt >> file.txt
rm file-1.txt
cat file-2.txt >> file.txt
rm file-2.txt
split -l 500000 file.txt
for f in *; do mv "$f" "$f.txt"; done

How could I achieve this in a more elegant way?

1
  • 1
    Streamed archive format is more useful here
    – eri
    Oct 18, 2017 at 20:13

2 Answers 2

3

7za + split solution (single pipeline):

7za e "*.7z" -so 2> /dev/null | split -l500000 --additional-suffix=".txt" --numeric-suffixes=1 - "file"

-- 7za options:

  • e - extract/decompress archive(s)

  • -so - write the content to STDOUT


-- split options:

  • --additional-suffix=".txt" - append suffix .txt to all resulting filenames

  • --numeric-suffixes=1 - use numeric suffixes starting at 1

  • - (hyphen) - read data from STDIN (standard input)

  • "file" - the common prefix for all resulting filenames


The above command will result to files with the following naming format: file01.txt, file02.txt etc.

1
  • It's a detail, but won't "*.7z" produce file-1.txt file-2.txt file.txt, i.e. the wrong order? Oct 19, 2017 at 7:35
1

You could use pipes and the --filter option of split after decompression:

p7zip -d *.txt.7z
cat file.txt file-1.txt file-2.txt | split -l 500000 --filter='> $FILE.txt'
rm file*

Here is the documentation for the --filter option:

‘--filter=COMMAND’
     With this option, rather than simply writing to each output file,
     write through a pipe to the specified shell COMMAND for each output
     file.  COMMAND should use the $FILE environment variable, which is
     set to a different output file name for each invocation of the
     command.  For example, imagine that you have a 1TiB compressed file
     that, if uncompressed, would be too large to reside on disk, yet
     you must split it into individually-compressed pieces of a more
     manageable size.  To do that, you might run this command:

          xz -dc BIG.xz | split -b200G --filter='xz > $FILE.xz' - big-

     Assuming a 10:1 compression ratio, that would create about fifty
     20GiB files with names ‘big-aa.xz’, ‘big-ab.xz’, ‘big-ac.xz’, etc.

If you need to keep a file with all the output, you could use tee, which copies standard input to standard output and to the file given as argument.

cat file.txt file-1.txt file-2.txt |
    tee all.txt |
    split -l 50000 --filter='> $FILE.txt'
1

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .