0

I have this code to split large backup files into smaller chunks that can be saved on the FAT32 filesystem.

split -b $(echo "6*(2^30)-1" | bc) --verbose $1/jcubic.tar.gz $1/jcubic.tar.gz-

I have the chunks when I want to combine them into a bigger file I have this error:

gzip: stdin: not in gzip format
tar: Child returned status 1
tar: Error is not recoverable: exiting now

I use cat jcubic.tar.gz-* > jcubic.tar.gz my chunks have 29,7GB in total but the output file has 54.1GB. I have no idea why.

What is the proper way to split the file and combine it into one?

EDIT:

$ echo jcubic.tar.gz-*
jcubic.tar.gz-aa jcubic.tar.gz-ab jcubic.tar.gz-ac jcubic.tar.gz-ad 
jcubic.tar.gz-ae jcubic.tar.gz-af jcubic.tar.gz-ag jcubic.tar.gz-ah

EDIT2:

I've used Ext4 portion and created 6GB chunks:

$ split -b $(echo "6*(2^30)-1" | bc) --verbose jcubic.tar.gz jcubic.tar.gz-

Both the input and the chunks have 42.3GB

$ $ ls -lsh
 43G -rw-r--r--. 1 root root  43G 05-18 23:09 jcubic.tar.gz
6,1G -rw-r--r--. 1 kuba kuba 6,0G 05-21 19:19 jcubic.tar.gz-aa
6,1G -rw-r--r--. 1 kuba kuba 6,0G 05-21 19:20 jcubic.tar.gz-ab
6,1G -rw-r--r--. 1 kuba kuba 6,0G 05-21 19:21 jcubic.tar.gz-ac
6,1G -rw-r--r--. 1 kuba kuba 6,0G 05-21 19:22 jcubic.tar.gz-ad
6,1G -rw-r--r--. 1 kuba kuba 6,0G 05-21 19:23 jcubic.tar.gz-ae
6,1G -rw-r--r--. 1 kuba kuba 6,0G 05-21 19:24 jcubic.tar.gz-af
6,1G -rw-r--r--. 1 kuba kuba 6,0G 05-21 19:25 jcubic.tar.gz-ag
311M -rw-r--r--. 1 kuba kuba 311M 05-21 19:25 jcubic.tar.gz-ah

but when using cat:

cat jcubic.tar.gz-* ~/xxx/jcubic.tar.gz

I've got 77GB file that is not tar ball:

$ file jcubic.tar.gz
jcubic.tar.gz: data
$ ls -lh jcubic.tar.gz
-rw-r--r--. 1 kuba kuba 77G 05-21 20:15 jcubic.tar.gz
$ tar xzvf jcubic.tar.gz 
gzip: stdin: not in gzip format
tar: Child returned status 1
tar: Error is not recoverable: exiting now
10
  • Please add output of echo jcubic.tar.gz-* to your question (no comment).
    – Cyrus
    Commented May 21, 2023 at 15:34
  • @Cyrus added the output of that command.
    – jcubic
    Commented May 21, 2023 at 16:13
  • 1
    Please add output of file /path/to/jcubic.tar.gz and the command that issued the above error message (3 lines) to your question.
    – Cyrus
    Commented May 21, 2023 at 16:22
  • The thing is that the gzip error message you're getting is emitted when the file doesn't even start like a gzip file. Does gzip -d jcubic.tar.gz-aa > /dev/null suffer the same problem or just error out when it comes the unexpected end of stream? Commented May 21, 2023 at 16:37
  • Btw, since this sounds like you'd be a bit better off if you used a compressor with higher compression (so, instead of tar czf jcubic.tar.gz allthefiles, you could tar cf - allthefiles | zstd -12 | split -b $(( 2**31-1 )) jcubic.tar.zst-split Commented May 21, 2023 at 16:48

1 Answer 1

0

It was a silly mistake, my cat was an alias to bat. I rarely use it so I didn't notice.

Using:

env cat jcubic.tar.gz-* > jcubic.tar.gz

solved the issue.

A 6GB file on FAT32 is fine, I could extract the file from my USB hard drive that has FAT32 without issue.

3
  • 1
    If your filesystem supports files larger than (4G-1)B, it's definitely not FAT32! That's just a fact - the file size field in a FAT32 entry is 4 B long, so at most 2³²-1 bytes, which is 4 GB-1B. Maybe you have exfat instead of fat32? Commented May 21, 2023 at 18:58
  • @MarcusMüller maybe, it is the default filesystem that was included when I buy the USB hard drive. I just assume it's Fat32 because this is what I was using when sharing files between Windows and Linux. Just reading the Wikipedia entry en.wikipedia.org/wiki/ExFAT You're must be right. What is the max size of files on ExFat? I'm not able to find it on Wikipedia.
    – jcubic
    Commented May 21, 2023 at 19:23
  • On exfat, the maximum file size should be 128 Petabyte in theory. In practice, I'd expect half a Petabyte to work, because cluster sizes don't scale arbitrarily. Commented May 21, 2023 at 21:09

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .