1

I have a folder with over 200k files in it. I want to archive only the ones beginning with a certain name(file names are super large). Example sudo find /path/to/_Logs -name "RATE*" -print0 | xargs -0 tar -czf logs_rate.tgz.

However, from time to time I receive the message : tar:Removing leading `/' from member names and the archive resets to 0 and starts over. At the end of the process I only have about 25 files out of a total of 22000. What am I missing or what am I doing wrong?

8
  • Is sudo find /path/to/_Logs -name "RATE*" -print0 | wc -c larger than getconf ARG_MAX? – Ljm Dullaart Oct 12 '20 at 9:39
  • @LjmDullaart it doesn't matter. xargs will run the tar command multiple times even if it's less than ARG_MAX . It's a common misconception that xargs will use as many arguments as possible; it will use a much lower value. Use xargs --show-limits to show which one (it's usually 128k). – user414777 Oct 12 '20 at 9:42
  • @Aiu That's because xargs does not run a single instance of tar will all the files, but will run in multiple times with batches of arguments. Useless to say, this will keep recreate the archive, and in the end it will only contain the last batch of files. You could try increasing the command line limit defaults of xargs, may it will be able to cope with all the files in a single run. E.g. with xargs -0 -s 2093104. – user414777 Oct 12 '20 at 9:43
  • @user414777: the point is, as you correctly stated, that xargs runs tar multiple times. If the limit would be getconf ARG_MAX, on my system, with 200k files, that would leave about 10 chars per filename. So it would hit that limit as well. – Ljm Dullaart Oct 12 '20 at 9:56
  • @user414777 what does the -s 2093104 do? What do these numbers stand for? – Aiurea Adica tot YO Oct 12 '20 at 10:00
0

As the comments already suggest, tar is run multiple times by xargs. The most obvious way to handle this would be to append the files to the archive instead of re-creating the archive from scratch.

 sudo find /path/to/_Logs -name "RATE*" -print0 | xargs -0 tar -rf logs_rate.tar

and compress afterwards.

The remark about the leading / is a warning only.

1
  • Commands like this are not recommended. There are better ways that are not based on side effects. – schily Oct 12 '20 at 10:11
0

I found a workaround. Split the command in 2 different things to bypass the long names and large content.

  1. sudo find /path/to/_Logs -name "RATE*" > test1.txt
  2. tar -czvT test1.txt -f logs.tgz
2
  • You seriously lost me with the "naming the program tar" . I used tar to create the archive as per my needs. – Aiurea Adica tot YO Oct 12 '20 at 10:35
  • 1
    @Aiu better: find ... -type f -print0 | tar --null -T- -zcf logs.tgz. (assuming GNU tar and find, as you already do). – user414777 Oct 12 '20 at 11:24
0

Do not use xargs with tar. There is no need to do so.

Gnu/tar (and I believe other tars) have an option -T (long form: --files-from) to read an unlimited number of filenames from a side file. (You can also send them through stdin using -T -.)

The --null option can be used so that the -T list can contain null-terminated strings, just like xargs (and for the same reason).

Multiple -T options can be used. Note that pathnames starting with - are still a problem: the -T option is merely an abbreviation for a list of names on the command line: it can include other options, such as -C.

2
  • "and I believe other tars) have an option -T" No they don't. bsdtar (which is not the default tar on all BSDs) does have it, but bsdtar kind of strives for GNU tar compatibility. – user414777 Oct 12 '20 at 11:31
  • I will hide behind the Centos tag on the question, then -- I did check the man page for that one. IIRC, my technique on SunOS 1.6 was to make a directory stuffed with symbolic links to all the files I wanted, and then use the tar -h option in that directory. – Paul_Pedant Oct 12 '20 at 12:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.