1

I have quite interesting observation which perhaps is a manifest of a bug.

My use case is to:

  • take a backup of a folder, having a set of files from this folder added to --exclude-from file. I'm working with incremental backup (-g option set)
  • on a new machine, restoring the tar to same location as on the original machine. Note, the new machine has already some files in the location; if anything exist, I want to replace / overwrite it.

Steps to reproduce

  1. Create files and folders structure as below:

    /tmp/src
    └── etc
        ├── A.cfg
        └── config
            ├── 1.cfg
            ├── 2.cfg
            └── 3.cfg
    
  2. Create exclude and include files

    mbp:tarTests jhartman$ cat /tmp/include.cfg
    /tmp/src/etc/
    mbp:tarTests jhartman$ cat /tmp/exclude.cfg
    /tmp/src/etc/config/*.cfg
    
  3. Create an archive

    $ rm -f /tmp/snapshot 
    $ tar -g /tmp/snapshot -cvz -f /tmp/test.tar.gz -X /tmp/exclude.cfg -T /tmp/include.cfg
    
    
    ./tar-1.27/src/tar: /tmp/src/etc: Directory is new
    ./tar-1.27/src/tar: /tmp/src/etc/config: Directory is new
    ./tar-1.27/src/tar: Removing leading `/' from member names
    /tmp/src/etc/
    /tmp/src/etc/config/
    /tmp/src/etc/A.cfg
    
  4. Un-archive to a new server. Before unarchiving, files in the new server:

    /tmp/src
    └── etc
        ├── A.cfg
        └── config
            ├── 3.cfg
            ├── 4.cfg
            └── 5.cfg
    
    • un-archive (on the target machine)

      $TAR_BASE/src/tar --overwrite --extract --force-local --listed-incremental=/dev/null --file /tmp/test.tar.gz -v -C /
      
      tmp/src/etc/
      tmp/src/etc/config/
      ./tar-1.30/src/tar: Deleting ‘tmp/src/etc/config/4.cfg’
      ./tar-1.30/src/tar: Deleting ‘tmp/src/etc/config/5.cfg’
      tmp/src/etc/A.cfg
      

So eventually, files in this folder:

/tmp/src
└── etc
    ├── A.cfg
    └── config
        └── 3.cfg

As you see, despite 4.cfg and 5.cfg were not int he src machine, they are gone. I'm guessing it's because of using /tmp/src/etc/config/*.cfg in the exclude file.

Worth to mention that this behaviour was changing across the versions: 1.30, 1.27, 1.27.1, 1.29 - NOK, 1.25, 1.26, 1.28 - OK (where NOK means - I could see Deleting message for the files in my concern).

What do you think?

Thank you, Jarek

Update, 2018-08-21, 20:51

Regardless of the version I run, output of the --list operation is same and shows D at the config folder:

tar --list --incremental --verbose --verbose --file /tmp/test.tar.gz

drwxr-xr-x jhartman/staff   16 2018-08-21 20:44 tmp/src/etc/
Y A.cfg
D config

drwxr-xr-x jhartman/staff   22 2018-08-21 20:44 tmp/src/etc/config/
N 1.cfg
N 2.cfg
N 3.cfg

-rw-r--r-- jhartman/staff   30 2018-08-21 20:44 tmp/src/etc/A.cfg

Update 21:03

For reference, scripts I've used for the tests are stored in https://github.com/jaroslawhartman/tar-tests repo.

See output.txt for detailed results of the tests.

  • Do you see differences in the output of tar --list --incremental --verbose --verbose --file /tmp/test.tar.gz when you use different versions of tar? (E.g. with 1.29 I see config being listed as deleted.) – Stephen Kitt Aug 21 '18 at 15:58
  • Hi, not really, no differences in this output. It's always D config, ----------------- TAR LIST drwxr-xr-x jhartman/staff 16 2018-08-21 20:44 tmp/src/etc/ Y A.cfg D config drwxr-xr-x jhartman/staff 22 2018-08-21 20:44 tmp/src/etc/config/ N 1.cfg N 2.cfg N 3.cfg -rw-r--r-- jhartman/staff 30 2018-08-21 20:44 tmp/src/etc/A.cfg ----------------- – Jarek Aug 21 '18 at 18:47
  • This is not about tar but about GNU tar. Note that GNU tar announced incremental backups in 1992 but incremental restores did never work for non-trivial cases even though there have been three related bug reports since 2004. You make things even more complex as you use exclude lists and as you in addition do not start with an empty target filesystem. Your expectations in this special case are subject of a discussion. I would be interested in such a discussion but this place does not seem to give the right base for such a discussion. – schily Aug 21 '18 at 19:09
  • Hi Schily, I would agree that what I've observed is a "feature" if that was consistent across the versions. If this comes and goes across the version I tend to call it a bug or recurring regression. I also cannot call it "non-trivial" case... But perhaps I'm just irritated... Thanks! – Jarek Aug 21 '18 at 19:57
  • This example unix.stackexchange.com/questions/411324/… is what I call non-trivial and this is still less complex than your problem. – schily Aug 21 '18 at 20:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.