0

My issue is that I am trying to copy files from my Ubuntu linux system which uses a ext4 file system to a Sandisk 1TB USB card, which uses an exFat system.

In the ext4 system, the entire directory structure occupies ~ 2.5Gb. When I copy it to the exFat system on the USB drive, it bloats to ~ 290Gb.

The files I am copying are images files (.jpg). As far as I can tell, the problem is that the originals are sparse files. By default, the ext4 system compresses them and the apparent size is much smaller (~10k for each file). However, as I found on the MSDN site, the exFat system is non-compressing and store sparse files in a non-sparse way. (I tried copying many times to force them to conform..through 'cp' or with 'tar gzip' and so forth...no luck. Its the same deal in Linux-->SanDisk (exFat) or MacOS-->SanDisk).

Example below, last 5 lines of a "du -all -h" on the SanDisk exFat structure:

du -all -h
.
.
1.0M    ./img_00000026.jpg
1.0M    ./img_00000089.jpg
1.0M    ./img_00000032.jpg
1.0M    ./img_00000081.jpg
1.0M    ./img_00000086.jpg
1.0M    ./img_00000049.jpg
1.0M    ./img_00000037.jpg
101M    .

When I use the "apparent-size" argument:

du -all --apparent-size -h
.
.
7.3K    ./img_00000026.jpg
7.8K    ./img_00000089.jpg
5.0K    ./img_00000032.jpg
7.5K    ./img_00000081.jpg
9.1K    ./img_00000086.jpg
8.8K    ./img_00000049.jpg
6.6K    ./img_00000037.jpg
1.9M    .

When I do the same on the Linux side (ext4 system), both listings are identical...it always shows the apparent size.

My question: is there anyway to force the exFat system to use the apparent size (or write them in a compressed way, but non-sparse) such that it doesn't hog up all my room?

I'm looking for a solution without reformatting the SanDisk USB to another file system.

  • 2
    exFAT doesn't support sparse files, so you shouldn't expect it to do so. – roaima Mar 16 at 22:45
  • I'm uncertain exFAT even supports sparse files. – Kusalananda Mar 16 at 22:45
  • 2
    I would be surprised by sparse JPEGs. I would also be surprised by any sparse file that didn't gzip well. It looks to me like the exFAT filesystem has a cluster size of 1MB, which is a formatting issue. However, you seem like you've looked into this all - what leads you to believe what you do about the systems and files involved? – Michael Homer Mar 16 at 22:46
  • I'm looking for the easy answer, of course ;-). I would suspect that the best solution is to reformat the SanDisk to a better file system. However, it's got my other project data on it as well. The files I originally got were for ML training, and were taken from an archive on Google drive created ~ 2016. I don't know why they were written as sparse files. – Rex Barker Mar 16 at 22:48
  • 1
    BTW, I did write a short python script that read the file using matplotlib, and then re-writing it as a different file...but the same result. The new file is still the same size as the original (showing 1Mb per file). So your suspicion that it was the formatted cluster size of 1MB could likely be correct. Good tip. – Rex Barker Mar 16 at 22:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.