4

I have a folder containing a large amount of symlinked files. These files are each on the order of 10-11GB (fastq files to be specific). They come from a variety of source folders, but I made sure there's only one level of symlinks.

I'm trying to gzip them by simply doing:

gzip *.fastq

That results in a bunch of

too many levels of symbolic links

And thus fails.

However, when I do:

for i in `ls | egrep *.fastq$`; do gzip -c $i > $i.gz; done;

it does work. My question is simple. What is the difference between those? AFAIK, the only difference is that the second approach starts a new gzip process for each file, whereas the first one should do everything in one process. Can gzip only handle one symlinked file at a time? Doing the same on a test folder with normal files works both ways.

1
  • “Too many levels of symbolic links” is about one file that consists of a symlink to a symlink to a symlink to … (or variations on this theme). The fact that you're passing multiple files to gzip has no bearing on this. Your second command is weird and probably miscopied — egrep *.fastq$ should match nothing. What you probably meant to write is for i in *.fastq; do …. Do the paths recorded in the symlinks involve other symlinks? Which files cause “too many levels of symbolic links”? Did the command that worked end up missing some of the files? Jun 16 '14 at 23:20
4

A quick check of the gzip source (specifically, gzip 1.6 as included in Ubuntu 14.04), shows that the observed behavior comes from the function open_and_stat, beginning at line 1037 of gzip.c:

static int
open_and_stat (char *name, int flags, mode_t mode, struct stat *st)
{
  int fd;

  /* Refuse to follow symbolic links unless -c or -f.  */
  if (!to_stdout && !force)
    {
      if (HAVE_WORKING_O_NOFOLLOW)
        flags |= O_NOFOLLOW;
      else
        {
#if HAVE_LSTAT || defined lstat
          if (lstat (name, st) != 0)
            return -1;
          else if (S_ISLNK (st->st_mode))
            {
              errno = ELOOP;
              return -1;
            }
#endif
        }
    }

  fd = OPEN (name, flags, mode);
  if (0 <= fd && fstat (fd, st) != 0)
    {
      int e = errno;
      close (fd);
      errno = e;
      return -1;
    }
  return fd;
}

Note that the comment line states that gzip will not follow symlinks unless it is called with the -c or -f flags, and inside the #if ... #endif the errno variable is set to ELOOP (too many symbolic links encountered) if the file to be compressed is actually a symlink.

Now, from the gzip(1) man page, the -c and -f flags are:

   -c --stdout --to-stdout
         Write  output  on  standard output; keep original files unchanged.  If there are
         several input files, the output consists of a  sequence  of  independently  com‐
         pressed  members.  To  obtain  better  compression,  concatenate all input files
         before compressing them.


  -f --force
         Force compression or decompression even if the file has multiple  links  or  the
         corresponding  file  already  exists,  or if the compressed data is read from or
         written to a terminal. If the input data is not in a format recognized by  gzip,
         and  if the option --stdout is also given, copy the input data without change to
         the standard output: let zcat behave as cat.  If -f is not given, and  when  not
         running  in  the  background,  gzip  prompts  to verify whether an existing file
         should be overwritten.

Putting all together and going back to the original question:

  • The first example fails because it is trying to compress the actual symlink (even if it is not an actual link loop)
  • The second uses the -c flag, so it is reading the contents of the original file and then writing the compressed output to stdout, so it succeeds.
  • A third scenario is using -f instead of -c. In this case, gzip does not complain when tryng to compress a symlink, but upon decompression it becomes a regular file, as shown:
$ ls -l
total 4
-rw-rw-r-- 1 x86tux x86tux 13 Jun 16 13:10 realfile.txt
lrwxrwxrwx 1 x86tux x86tux 12 Jun 16 23:40 symlink.txt -> realfile.txt
$ gzip symlink.txt
gzip: symlink.txt: Too many levels of symbolic links
$ gzip -f symlink.txt
$ ls -l
total 8
-rw-rw-r-- 1 x86tux x86tux 13 Jun 16 13:10 realfile.txt
-rw-rw-r-- 1 x86tux x86tux 45 Jun 16 13:10 symlink.txt.gz
$ gunzip symlink.txt.gz
$ ls -l
total 8
-rw-rw-r-- 1 x86tux x86tux 13 Jun 16 13:10 realfile.txt
-rw-rw-r-- 1 x86tux x86tux 13 Jun 16 13:10 symlink.txt
$ md5sum *
618f486e0225d305d16d0648ed44b1eb  realfile.txt
618f486e0225d305d16d0648ed44b1eb  symlink.txt
1

The single process per file part might hurt some if there was any possibility of it bottle-necking your operation, but at 10-11 gigabytes it is very difficult to imagine any scenario in which it is the exec call on gzip that hampers progress.

In that same vein, if they were a bunch of small files then gzip would likely not be able to compress them as well having less data to compare per file, but, again, at 10-11 gigabytes per compression operation that will not be an issue.

Discovering the cause of the error would be interesting though, I think. I would suggest attempting to apply lsof to a backgrounded gzip pid and find out what's happening.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.