3

I need to paste a lot of .txt files together. I use this command:

cat *.txt > newfile.txt 

I noticed that some of these files are empty. How can I insert a control in the script to prevent that the cat from acting on these empty files?

Thank you.

8
  • 10
    If they're empty why does it matter they are included? Commented Feb 10, 2021 at 15:32
  • 5
    Also note that if newfile.txt already existed at the point the command was run, you could end up with a situation that cat included newfile.txt in its list from *.txt and kept reading from it and writing to it at the same time Commented Feb 10, 2021 at 15:33
  • 1
    @roaima, d'oh, yes of course. GNU cat actually complains about that, btw: cat: newfile.txt: input file is output file.
    – ilkkachu
    Commented Feb 10, 2021 at 15:36
  • 1
    @ilkkachu yes, but you can't guarantee GNU cat, hence the warning Commented Feb 10, 2021 at 15:39
  • 3
    Empty files have no content, so cat-ing them produces no output. So the result of concatenation of a set of files, including empty ones, should be exactly the same as the result of concateantion of the same set of files, excluding empty ones. So what's the reason to complicate?
    – raj
    Commented Feb 10, 2021 at 16:15

3 Answers 3

10

Not really necessary, but if you need to rule out empty files:

for i in *.txt; do [ "$i" != newfile.txt ] && [ -s "$i" ] && cat -- "$i"; done >newfile.txt

The -s test will be true if the given file exists and is not empty (this is a standard test; see man test). We also avoid processing the output file itself.

5
  • 1
    Note that this will append to the file if run multiple times (it's not idempotent); add a truncate -s0 newfile.txt; at the beginning if this is undesired.
    – Kyuuhachi
    Commented Feb 10, 2021 at 23:39
  • @Caagr98, it should be idempotent, as the first thing that's done there is the >newfile.txt redirection which truncates the file. However, newfile.txt will be included in the expansion of *.txt, and if it's not the first to be processed, you will either get an error from some cat implementations that the input is the same as the output, or you'll get an infinite loop filling up the disk (at least if the previous files were larger than a few kb). Commented Feb 14, 2021 at 16:18
  • Oh, I see your comment is probably referring to the version before @Kusalananda's edit. Before the edit, the problem would have happened after the second run indeed. The edit makes it occur every time, so I'd say it's not a valid edit. Commented Feb 14, 2021 at 16:23
  • @StéphaneChazelas Sorry for the botched edit. Will correct. (now fixed)
    – Kusalananda
    Commented Feb 14, 2021 at 16:55
  • This is a very, very good answer. It does not only provide the answer, but this method I didn't consider! +1. Commented Feb 14, 2021 at 18:01
4

In zsh, you'd use the L+0 glob qualifier:

(cat -- *.txt(n-.L+0)) > newfile.txt

(here, also limiting to regular files only (.) determined (as well as the size) after symlink resolution (-), and sorting the list of files numerically (n) so file10.txt comes after file9.txt for instance).

Normally, including empty files shouldn't make a difference, as cat on an empty file produces no output. There are however two things that make it a good idea:

  1. You don't want newfile.txt itself to be included in the list of files to pass to cat. Some cat implementations will complain if the input file is the same as where stdout go to. And with those that don't you could end up filling up your disk as cat would read the output file in a loop. Here, the glob is expanded after the redirection is performed. That redirection truncates the output file, so it will have a size of 0 at the time that glob is expanded.
  2. By removing unnecessary files from the list of arguments passed to cat, we make the command line shorter, so we make it less likely to reach the limit on its size. Though here, you could also work around the problem by using zsh's zargs.

Without zsh, but if you have GNU utilities, you could do something equivalent with:

 LC_ALL=C find -L . -maxdepth 1 -name '*.txt' ! -name '.*' \
                    ! -name newfile.txt -type f -size +0 -print0 |
   sort -V0 |
   xargs -r0 cat > newfile.txt

Here, we can't rely on -size +0, as if there are many many files in the current directory, xargs could end up running one cat with a first batch before find gets to check the size of newfile.txt (also here, xargs > newfile.txt and find are run concurrently, so the redirection may not have been performed by the time find starts and checks the size of newfile.txt if it existed beforehand).

We also need to exclude hidden files by hand as find doesn't do it by default. We also need LC_ALL=C to work around GNU find's issues with non-characters. In this case, xargs will work around the limit of the command line size.

The equivalences with the zsh approach are:

  • -L-
  • -type f.
  • -size +0L+0
  • sort -Vn
0

As in classic UNIX filesystems the overhead to read an empty file is just about the same as detecting that the file is empty, it makes little sense to treat empty files separate, significantly when cat-ing them.

Also you did not tell whether the files would have to be in order. If not (maybe find even preserves the order here), this may be more efficient than https://unix.stackexchange.com/a/633734/320598:

(find *.txt \! -size 0 | xargs cat) >newfile.txt

4
  • Using find . -prune -type f -name '*.txt' ! -name newfile.txt ! -size 0c -exec cat {} + >newfile.txt would be safer and standard. With GNU find: find . -maxdepth 1 -type f -name '*.txt' ! -name newfile.txt ! -empty -exec cat {} + >newfile.txt. Note that ! is not special when surrounded by spaces, at least not in a standard shell, not even in bash.
    – Kusalananda
    Commented Feb 14, 2021 at 15:54
  • @Kusalanda, true, but that wouldn't sort the list of file names like *.txt does Commented Feb 14, 2021 at 15:59
  • 1
    Among the problems: there's (1) problems with filenames starting with - causing find to treat them as option or predicate (2) if those *.txt files are or type directory, find will recurse into them (3) if some are symlinks, they will have a non-zero size even if the file they point to is empty (4) find's output format is not compatible with xargs' input format. Here it will choke on whitespace, quotes, backslashes in filenames (5) newfile.txt could end up in the list of filenames passed to cat causing disk fill up. Commented Feb 14, 2021 at 16:35
  • Is the answer really worse than unix.stackexchange.com/a/633734/320598? YOu are invited to edit or suggest edits.
    – U. Windl
    Commented Feb 15, 2021 at 7:30

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .