cat ./* >> large.txt decreases in size

Question

I'm a novice in bash scripting. I wrote a script that concatenates many files from different directories into large files

#!/bin/bash
for f in foo/$1/*; do
    if [ -d "$f" ]; then
        for d in $f/*; do
                if [ -d "$d" ]; then
                file=$( echo ${d##*/} )
                mkdir -p bar/$1/"$file"/fastq/
                cat $d/*/fastq_pass/* >> bar/$1/"$file"/fastq/$file.fastq.gz
                        fi
                done
    fi
done

I've noticed that some of merged files lost 1-2 GB in size, which is weird because other files are ok. When I tried to do

cat foo/* >> bar/largefile.fastq.gz

there was no loss. So it's definitely because of my script. Where's error?

Answer 1

For efficiency, filesystems allocate disk space in clusters of blocks. Thus, a "1 byte" file would be allocated a whole cluster. Concatenating 100 "1 byte" files, producing 100 bytes of data would still take a single cluster.

Cluster and block sizes can vary, there's a subfield of Performance Analysis dedicated to optimizing for a particular application.