0

This question is about compressing rather large data directories (around 22-23 GiB), specifically the integrity checks of such archives. I do these big backups about once a month, so the time it takes is not an issue here.

I am on Linux Mint 19.1 which is based on Ubuntu 18.04 and I want to use tar in combination with xz with the best checksum or verification available.

Software versions for future reference:

$ tar --version

tar (GNU tar) 1.29

and

$ xz --version

xz (XZ Utils) 5.2.2

Just to note:

  • I burn it on M-Disc media.

  • I don't mind if it takes long; I stress this.

  • If some settings depend on memory, then I have 16 GiB DDR4 in this laptop.


It appears to me, according to the xz man page, I see is no way of setting a solid compression that would maybe help with compression ratio.

Until this day, I just use the simplest way of doing my BluRay backup in one go:

$ tar -cJf ...

It was relatively fast, but almost no compression occurred when compressing some .iso files, which were already compressed inside, and there is still my question about increasing, ... well, a better word would be ensuring the integrity of my archives.

With the help of:

$ xz -l ...

I was able to somewhat analyze the output result + I found the CRC-64 is in place as the checksum method.


Now when writing this, I found one interesting way to specify the compression level, which is of course not set to maximum (9), nor extreme by default:

$ XZ_OPT=-9e tar ...

I also found that I could switch the xz checksum from the default CRC-64 to SHA-256:

$ XZ_OPT='-9 --extreme --check=sha256' tar ...

What I am wondering about now, is the --verify option of tar, from the man page:

attempt to verify the archive after writing it

Because if doing it in one go as seen above, adding this option results in:

tar: Cannot verify compressed archives

Instead, I would have to do something like:

$ tar --create --file="${backup_file}" "${dir_to_backup}" --verify

And archive it separately with xz afterwards; I am unsure how to make use of it directly?

^^^ Which is my Question ^^^


Example archive with little compression possible, as all of its contents have already been compressed and enclosed in .iso files for instance. Still, I made 1.8 GiB by compressing it, which is nice.

$ xz --list backup--2019-03-09.tar.xz

backup--2019-03-09.tar.xz (1/1)
  Streams:            1
  Blocks:             127
  Compressed size:    22.0 GiB (23,670,187,308 B)
  Uncompressed size:  23.8 GiB (25,546,864,640 B)
  Ratio:              0.927
  Check:              SHA-256
  Stream padding:     0 B
  Streams:
    Stream    Blocks      CompOffset    UncompOffset        CompSize      UncompSize  Ratio  Check      Padding
         1       127               0               0  23,670,187,308  25,546,864,640  0.927  SHA-256          0
  Blocks:
    Stream     Block      CompOffset    UncompOffset       TotalSize      UncompSize  Ratio  Check
         1         1              12               0      53,243,880     201,326,592  0.264  SHA-256
         1         2      53,243,892     201,326,592     196,992,880     201,326,592  0.978  SHA-256
         1         3     250,236,772     402,653,184     201,315,216     201,326,592  1.000  SHA-256
         1         4     451,551,988     603,979,776     201,335,864     201,326,592  1.000  SHA-256
         1         5     652,887,852     805,306,368     201,335,864     201,326,592  1.000  SHA-256
         1         6     854,223,716   1,006,632,960     201,334,900     201,326,592  1.000  SHA-256
         1         7   1,055,558,616   1,207,959,552     200,622,316     201,326,592  0.997  SHA-256
         1         8   1,256,180,932   1,409,286,144     201,335,832     201,326,592  1.000  SHA-256
         1         9   1,457,516,764   1,610,612,736     201,335,864     201,326,592  1.000  SHA-256
         1        10   1,658,852,628   1,811,939,328     201,330,732     201,326,592  1.000  SHA-256
         1        11   1,860,183,360   2,013,265,920     199,713,948     201,326,592  0.992  SHA-256
         1        12   2,059,897,308   2,214,592,512     175,761,000     201,326,592  0.873  SHA-256
         1        13   2,235,658,308   2,415,919,104     201,165,620     201,326,592  0.999  SHA-256
         1        14   2,436,823,928   2,617,245,696     201,321,784     201,326,592  1.000  SHA-256
         1        15   2,638,145,712   2,818,572,288     201,326,148     201,326,592  1.000  SHA-256
         1        16   2,839,471,860   3,019,898,880     200,038,544     201,326,592  0.994  SHA-256
         1        17   3,039,510,404   3,221,225,472     200,750,292     201,326,592  0.997  SHA-256
         1        18   3,240,260,696   3,422,552,064     201,310,636     201,326,592  1.000  SHA-256
         1        19   3,441,571,332   3,623,878,656     201,335,864     201,326,592  1.000  SHA-256
         1        20   3,642,907,196   3,825,205,248     201,335,864     201,326,592  1.000  SHA-256
         1        21   3,844,243,060   4,026,531,840     201,335,864     201,326,592  1.000  SHA-256
         1        22   4,045,578,924   4,227,858,432     201,335,864     201,326,592  1.000  SHA-256
         1        23   4,246,914,788   4,429,185,024     201,330,168     201,326,592  1.000  SHA-256
         1        24   4,448,244,956   4,630,511,616     199,487,940     201,326,592  0.991  SHA-256
         1        25   4,647,732,896   4,831,838,208     201,335,172     201,326,592  1.000  SHA-256
         1        26   4,849,068,068   5,033,164,800     201,334,468     201,326,592  1.000  SHA-256
         1        27   5,050,402,536   5,234,491,392     201,170,860     201,326,592  0.999  SHA-256
         1        28   5,251,573,396   5,435,817,984     201,200,520     201,326,592  0.999  SHA-256
         1        29   5,452,773,916   5,637,144,576     191,043,588     201,326,592  0.949  SHA-256
         1        30   5,643,817,504   5,838,471,168     196,939,368     201,326,592  0.978  SHA-256
         1        31   5,840,756,872   6,039,797,760     185,189,612     201,326,592  0.920  SHA-256
         1        32   6,025,946,484   6,241,124,352     201,298,652     201,326,592  1.000  SHA-256
         1        33   6,227,245,136   6,442,450,944     201,137,020     201,326,592  0.999  SHA-256
         1        34   6,428,382,156   6,643,777,536     201,166,576     201,326,592  0.999  SHA-256
         1        35   6,629,548,732   6,845,104,128     201,230,404     201,326,592  1.000  SHA-256
         1        36   6,830,779,136   7,046,430,720     201,265,840     201,326,592  1.000  SHA-256
         1        37   7,032,044,976   7,247,757,312     201,277,512     201,326,592  1.000  SHA-256
         1        38   7,233,322,488   7,449,083,904     201,252,812     201,326,592  1.000  SHA-256
         1        39   7,434,575,300   7,650,410,496     201,268,876     201,326,592  1.000  SHA-256
         1        40   7,635,844,176   7,851,737,088     201,220,980     201,326,592  0.999  SHA-256
         1        41   7,837,065,156   8,053,063,680     201,286,716     201,326,592  1.000  SHA-256
         1        42   8,038,351,872   8,254,390,272     201,162,416     201,326,592  0.999  SHA-256
         1        43   8,239,514,288   8,455,716,864     201,167,132     201,326,592  0.999  SHA-256
         1        44   8,440,681,420   8,657,043,456     201,191,296     201,326,592  0.999  SHA-256
         1        45   8,641,872,716   8,858,370,048     201,234,228     201,326,592  1.000  SHA-256
         1        46   8,843,106,944   9,059,696,640     201,300,992     201,326,592  1.000  SHA-256
         1        47   9,044,407,936   9,261,023,232     198,262,448     201,326,592  0.985  SHA-256
         1        48   9,242,670,384   9,462,349,824      91,113,148     201,326,592  0.453  SHA-256
         1        49   9,333,783,532   9,663,676,416     143,398,764     201,326,592  0.712  SHA-256
         1        50   9,477,182,296   9,865,003,008     178,570,780     201,326,592  0.887  SHA-256
         1        51   9,655,753,076  10,066,329,600     188,475,820     201,326,592  0.936  SHA-256
         1        52   9,844,228,896  10,267,656,192     193,534,136     201,326,592  0.961  SHA-256
         1        53  10,037,763,032  10,468,982,784     157,357,336     201,326,592  0.782  SHA-256
         1        54  10,195,120,368  10,670,309,376     111,495,992     201,326,592  0.554  SHA-256
         1        55  10,306,616,360  10,871,635,968      99,397,052     201,326,592  0.494  SHA-256
         1        56  10,406,013,412  11,072,962,560     192,862,268     201,326,592  0.958  SHA-256
         1        57  10,598,875,680  11,274,289,152     180,224,836     201,326,592  0.895  SHA-256
         1        58  10,779,100,516  11,475,615,744     165,539,672     201,326,592  0.822  SHA-256
         1        59  10,944,640,188  11,676,942,336     170,200,512     201,326,592  0.845  SHA-256
         1        60  11,114,840,700  11,878,268,928     130,293,232     201,326,592  0.647  SHA-256
         1        61  11,245,133,932  12,079,595,520     194,065,376     201,326,592  0.964  SHA-256
         1        62  11,439,199,308  12,280,922,112     187,087,700     201,326,592  0.929  SHA-256
         1        63  11,626,287,008  12,482,248,704     187,057,960     201,326,592  0.929  SHA-256
         1        64  11,813,344,968  12,683,575,296     187,377,228     201,326,592  0.931  SHA-256
         1        65  12,000,722,196  12,884,901,888     199,580,888     201,326,592  0.991  SHA-256
         1        66  12,200,303,084  13,086,228,480     152,912,664     201,326,592  0.760  SHA-256
         1        67  12,353,215,748  13,287,555,072     169,255,644     201,326,592  0.841  SHA-256
         1        68  12,522,471,392  13,488,881,664     187,663,644     201,326,592  0.932  SHA-256
         1        69  12,710,135,036  13,690,208,256     194,150,692     201,326,592  0.964  SHA-256
         1        70  12,904,285,728  13,891,534,848     193,988,696     201,326,592  0.964  SHA-256
         1        71  13,098,274,424  14,092,861,440     195,142,924     201,326,592  0.969  SHA-256
         1        72  13,293,417,348  14,294,188,032     200,672,944     201,326,592  0.997  SHA-256
         1        73  13,494,090,292  14,495,514,624     199,290,184     201,326,592  0.990  SHA-256
         1        74  13,693,380,476  14,696,841,216     199,263,024     201,326,592  0.990  SHA-256
         1        75  13,892,643,500  14,898,167,808     183,145,436     201,326,592  0.910  SHA-256
         1        76  14,075,788,936  15,099,494,400     167,990,704     201,326,592  0.834  SHA-256
         1        77  14,243,779,640  15,300,820,992     196,509,992     201,326,592  0.976  SHA-256
         1        78  14,440,289,632  15,502,147,584     147,559,752     201,326,592  0.733  SHA-256
         1        79  14,587,849,384  15,703,474,176     183,516,020     201,326,592  0.912  SHA-256
         1        80  14,771,365,404  15,904,800,768     135,885,500     201,326,592  0.675  SHA-256
         1        81  14,907,250,904  16,106,127,360     182,531,984     201,326,592  0.907  SHA-256
         1        82  15,089,782,888  16,307,453,952     165,100,712     201,326,592  0.820  SHA-256
         1        83  15,254,883,600  16,508,780,544     161,361,236     201,326,592  0.801  SHA-256
         1        84  15,416,244,836  16,710,107,136     201,335,864     201,326,592  1.000  SHA-256
         1        85  15,617,580,700  16,911,433,728     199,923,388     201,326,592  0.993  SHA-256
         1        86  15,817,504,088  17,112,760,320     193,446,284     201,326,592  0.961  SHA-256
         1        87  16,010,950,372  17,314,086,912     198,499,544     201,326,592  0.986  SHA-256
         1        88  16,209,449,916  17,515,413,504     201,335,864     201,326,592  1.000  SHA-256
         1        89  16,410,785,780  17,716,740,096     201,335,864     201,326,592  1.000  SHA-256
         1        90  16,612,121,644  17,918,066,688     201,335,864     201,326,592  1.000  SHA-256
         1        91  16,813,457,508  18,119,393,280     201,335,864     201,326,592  1.000  SHA-256
         1        92  17,014,793,372  18,320,719,872     201,335,864     201,326,592  1.000  SHA-256
         1        93  17,216,129,236  18,522,046,464     201,335,864     201,326,592  1.000  SHA-256
         1        94  17,417,465,100  18,723,373,056     201,330,604     201,326,592  1.000  SHA-256
         1        95  17,618,795,704  18,924,699,648     201,335,864     201,326,592  1.000  SHA-256
         1        96  17,820,131,568  19,126,026,240     183,578,156     201,326,592  0.912  SHA-256
         1        97  18,003,709,724  19,327,352,832     201,335,864     201,326,592  1.000  SHA-256
         1        98  18,205,045,588  19,528,679,424     201,162,480     201,326,592  0.999  SHA-256
         1        99  18,406,208,068  19,730,006,016     201,333,120     201,326,592  1.000  SHA-256
         1       100  18,607,541,188  19,931,332,608     199,120,720     201,326,592  0.989  SHA-256
         1       101  18,806,661,908  20,132,659,200     201,335,864     201,326,592  1.000  SHA-256
         1       102  19,007,997,772  20,333,985,792     201,335,864     201,326,592  1.000  SHA-256
         1       103  19,209,333,636  20,535,312,384     201,335,772     201,326,592  1.000  SHA-256
         1       104  19,410,669,408  20,736,638,976     201,335,864     201,326,592  1.000  SHA-256
         1       105  19,612,005,272  20,937,965,568     179,075,024     201,326,592  0.889  SHA-256
         1       106  19,791,080,296  21,139,292,160     128,422,864     201,326,592  0.638  SHA-256
         1       107  19,919,503,160  21,340,618,752     186,242,664     201,326,592  0.925  SHA-256
         1       108  20,105,745,824  21,541,945,344     199,270,940     201,326,592  0.990  SHA-256
         1       109  20,305,016,764  21,743,271,936     198,996,836     201,326,592  0.988  SHA-256
         1       110  20,504,013,600  21,944,598,528     175,004,932     201,326,592  0.869  SHA-256
         1       111  20,679,018,532  22,145,925,120     200,046,576     201,326,592  0.994  SHA-256
         1       112  20,879,065,108  22,347,251,712     178,979,024     201,326,592  0.889  SHA-256
         1       113  21,058,044,132  22,548,578,304     199,886,808     201,326,592  0.993  SHA-256
         1       114  21,257,930,940  22,749,904,896     177,371,480     201,326,592  0.881  SHA-256
         1       115  21,435,302,420  22,951,231,488     190,387,928     201,326,592  0.946  SHA-256
         1       116  21,625,690,348  23,152,558,080     194,894,972     201,326,592  0.968  SHA-256
         1       117  21,820,585,320  23,353,884,672     199,717,584     201,326,592  0.992  SHA-256
         1       118  22,020,302,904  23,555,211,264     152,814,704     201,326,592  0.759  SHA-256
         1       119  22,173,117,608  23,756,537,856     199,462,188     201,326,592  0.991  SHA-256
         1       120  22,372,579,796  23,957,864,448     175,908,224     201,326,592  0.874  SHA-256
         1       121  22,548,488,020  24,159,191,040     181,534,208     201,326,592  0.902  SHA-256
         1       122  22,730,022,228  24,360,517,632     195,606,504     201,326,592  0.972  SHA-256
         1       123  22,925,628,732  24,561,844,224     192,220,808     201,326,592  0.955  SHA-256
         1       124  23,117,849,540  24,763,170,816     188,762,236     201,326,592  0.938  SHA-256
         1       125  23,306,611,776  24,964,497,408     173,512,276     201,326,592  0.862  SHA-256
         1       126  23,480,124,052  25,165,824,000     106,397,932     201,326,592  0.528  SHA-256
         1       127  23,586,521,984  25,367,150,592      83,664,288     179,714,048  0.466  SHA-256
  • Regarding your solid compression comment, by compressing a tarball, you are getting solid compression — the compression is done across files in the tarball, not per file. – Stephen Kitt Mar 9 at 13:12
  • @StephenKitt But as far as the example in my question goes, there seem to be multiple Blocks / Streams, whatever you call it. – Vlastimil Mar 9 at 13:15
  • 1
    Yes, xz compresses blocks, but they don’t correspond to files. You might be able to increase the block size (in your case it’s currently 201,326,592 bytes). However you already are benefiting from solid compression (as mentioned in the Wikipedia page you linked to). – Stephen Kitt Mar 9 at 13:28
  • @StephenKitt Ok, didn't have idea. Thank you! – Vlastimil Mar 9 at 13:29
  • If your problem is verifying if the BluRay media is ok, do the checksum of the compressed file on the disk, and compare it after burning. No need to complicate stuff. Another option might be uncompressing it to /dev/null, if that is possible. – Rui F Ribeiro Mar 9 at 15:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.