13

I have six Linux logical volumes that together back a virtual machine. The VM is currently shutdown, so its easy to take consistent images of them.

I'd like to pack all six images together in an archive. Trivially, I could do something like this:

cp /dev/Zia/vm_lvraid_* /tmp/somedir
tar c /tmp/somedir | whatever

But that of course creates an extra copy. I'd like to avoid the extra copy.

The obvious approach:

tar c /dev/Zia/vm_lvraid_* | whatever

does not work, as tar recognizes the files a special (symlinks in this case) and basically stores the ln -s in the archive. Or, with --dereference or directly pointed at /dev/dm-X, it recognizes them as special (device files) and basically stores the mknod in the archive.

I've searched for command-line options to tar to override this behavior, and couldn't find any. I also tried cpio, same problem, and couldn't find any options to override it there, either. I also tried 7z (ditto). Same with pax. I even tried zip, which just got itself confused.

edit: Looking at the source code of GNU tar and GNU cpio, it appears neither of them can do this. At least, not without serious trickery (the special handling of device files can't be disabled). So, suggestions of serious trickery would be appreciated or alternate utilities.

TLDR: Is there some archiver that will pack multiple disk images together (taken from raw devices) and stream that output, without making extra on-disk copies? My preference would be output in a common format, like POSIX or GNU tar.

  • I convinced it. – mikeserv Sep 3 '14 at 8:30
11
+250

So recently I wanted to do this with tar. Some investigation indicated to me that it was more than a little nonsensical that I couldn't. I did come up with this weird split --filter="cat >file; tar -r ..." thing, but, well, it was terribly slow. And the more I read about tar the more nonsensical it seemed.

You see, tar is just a concatenated list of records. The constituent files are not altered in any way - they're whole within the archive. But they are blocked off on 512-byte block boundaries, and preceding every file there is a header. That's it. The header format is really, very simple as well.

So, I wrote my own tar. I call it... shitar.

z() (IFS=0; printf '%.s\\0' $(printf "%.$(($1-${#2}))d"))
chk() (IFS=${IFS#??}; set -f; set -- $(     
        printf "$(fmt)" "$n" "$@" '' "$un" "$gn"               
);  IFS=; a="$*"; printf %06o "$(($(
        while printf %d+ "'${a:?}"; do a=${a#?}; done 2>/dev/null
)0))")                                                                 
fmt() { printf '%s\\'"${1:-n}" %s "${1:+$(z 99 "$n")}%07d" \
    %07o %07o %011o %011o "%-${1:-7}s" ' 0' "${1:+$(z 99)}ustar  " %s \
    "${1:+$(z 31 "$un")}%s"
}

That's the meat and potatoes, really. It writes the headers and computes the chksum - which, relatively speaking, is the only hard part. It does the ustar header format... maybe. At least, it emulates what GNU tar seems to think is the ustar header format to the point that it does not complain. And there's more to it, it's just that I haven't really coagulated it yet. Here, I'll show you:

for f in 1 2; do echo hey > file$f; done
{ tar -cf - file[123]; echo .; } | tr \\0 \\n | grep -b .

0:file1                      #filename - first 100 bytes
100:0000644                  #octal mode - next 8
108:0001750                  #octal uid,
116:0001750                  #gid - next 16
124:00000000004              #octal filesize - next 12
136:12401536267              #octal epoch mod time - next 12
148:012235                   #chksum - more on this
155: 0                       #file type - gnu is weird here - so is shitar
257:ustar                    #magic string - header type
265:mikeserv                 #owner
297:mikeserv                 #group - link name... others shitar doesnt do
512:hey                      #512-bytes - start of file   
1024:file2                   #512 more - start of header 2
1124:0000644
1132:0001750
1140:0001750
1148:00000000004
1160:12401536267
1172:012236
1179: 0
1281:ustar  
1289:mikeserv
1321:mikeserv
1536:hey
10240:.                     #default blocking factor 20 * 512

That's tar. Everything's padded with \0nulls so I just turn em into \newlines for readability. And shitar:

#the rest, kind of, calls z(), fmt(), chk() + gets $mdata and blocks w/ dd
for n in file[123]
do d=$n; un=$USER; gn=$(id --group --name)
   set -- $(stat --printf "%a\n%u\n%g\n%s\n%Y" "$n")
   printf "$(fmt 0)" "$n" "$@" "$(chk "$@")" "$un" "$gn"
   printf "$(z $((512-298)) "$gn")"; cat "$d"  
   printf "$(x=$(($4%512));z $(($4>512?($x>0?$x:512):512-$4)))"
done |
{ dd iflag=fullblock conv=sync bs=10240 2>/dev/null; echo .; } |
tr \\0 \\n | grep -b .

OUTPUT

0:file1                 #it's the same. I shortened it.
100:0000644             #but the whole first file is here
108:0001750
116:0001750
124:00000000004
136:12401536267
148:012235              #including its checksum
155: 0
257:ustar  
265:mikeserv
297:mikeserv
512:hey
1024:file2
...
1172:012236             #and file2s checksum
...
1536:hey
10240:.

I say kind of up there because that isn't shitar's purpose - tar already does that beautifully. I just wanted to show how it works - which means I need to touch on the chksum. If it wasn't for that I would just be dding off the head of a tar file and done with it. That might even work sometimes, but it gets messy when there are multiple members in the archive. Still, the chksum is really easy.

First, make it 7 spaces - (which is a weird gnu thing, I think, as the spec says 8, but whatever - a hack is a hack). Then add up the octal values of every byte in the header. That's your chksum. So you need the file metadata before you do the header, or you don't have a chksum. And that's a ustar archive, mostly.

Ok. Now, what it is meant to do:

cd /tmp; mkdir -p mnt     
for d in 1 2 3                                                
do  fallocate -l $((1024*1024*500)) disk$d
    lp=$(sudo losetup -f --show disk$d)
    sync
    sudo mkfs.vfat -n disk$d "$lp"
    sudo mount  "$lp" mnt
    echo disk$d file$d | sudo tee mnt/file$d
    sudo umount mnt
    sudo losetup -d "$lp"
done

That makes three 500M disk images, formats and mounts each, and writes a file to each.

for n in disk[123]
do d=$(sudo losetup -f --show "$n")
   un=$USER; gn=$(id --group --name)
   set -- $(stat --printf "%a\n%u\n%g\n$(lsblk -bno SIZE "$d")\n%Y" "$n")
   printf "$(fmt 0)" "$n" "$@" "$(chk "$@")" "$un" "$gn"
   printf "$(z $((512-298)) "$gn")"
   sudo cat "$d"
   sudo losetup -d "$d"
done | 
dd iflag=fullblock conv=sync bs=10240 2>/dev/null |
xz >disks.tar.xz

Note - apparently block devices will just always block correctly. Pretty handy.

That tar's the contents of the disk device files in-stream and pipes the output to xz.

ls -l disk*
-rw-r--r-- 1 mikeserv mikeserv 524288000 Sep  3 01:01 disk1
-rw-r--r-- 1 mikeserv mikeserv 524288000 Sep  3 01:01 disk2
-rw-r--r-- 1 mikeserv mikeserv 524288000 Sep  3 01:01 disk3
-rw-r--r-- 1 mikeserv mikeserv    229796 Sep  3 01:05 disks.tar.xz

Now, the moment of truth...

 xz -d <./disks.tar.xz| tar -tvf -
-rw-r--r-- mikeserv/mikeserv 524288000 2014-09-03 01:01 disk1
-rw-r--r-- mikeserv/mikeserv 524288000 2014-09-03 01:01 disk2
-rw-r--r-- mikeserv/mikeserv 524288000 2014-09-03 01:01 disk3

Hooray! Extraction...

xz -d <./disks.tar.xz| tar -xf - --xform='s/[123]/1&/'  
ls -l disk*
-rw-r--r-- 1 mikeserv mikeserv 524288000 Sep  3 01:01 disk1
-rw-r--r-- 1 mikeserv mikeserv 524288000 Sep  3 01:01 disk11
-rw-r--r-- 1 mikeserv mikeserv 524288000 Sep  3 01:01 disk12
-rw-r--r-- 1 mikeserv mikeserv 524288000 Sep  3 01:01 disk13
-rw-r--r-- 1 mikeserv mikeserv 524288000 Sep  3 01:01 disk2
-rw-r--r-- 1 mikeserv mikeserv 524288000 Sep  3 01:01 disk3
-rw-r--r-- 1 mikeserv mikeserv    229796 Sep  3 01:05 disks.tar.xz

Comparison...

cmp disk1 disk11 && echo yay || echo shite
yay

And the mount...

sudo mount disk13 mnt
cat mnt/*
disk3 file3

And so, in this case, shitar performs ok, I guess. I'd rather not go into all of the things which it won't do well. But, I will say - don't do newlines in the filenames at the least.

You can also do - and maybe should, considering the alternatives I've offered -this with squashfs. Not only do you get the single archive built from the stream - but it's mountable and builtin to the kernel's vfs:

From pseudo-file.example:

# Copy 10K from the device /dev/sda1 into the file input.  Ordinarily
# Mksquashfs given a device, fifo, or named socket will place that special file
# within the Squashfs filesystem, this allows input from these special
# files to be captured and placed in the Squashfs filesystem.
input f 444 root root dd if=/dev/sda1 bs=1024 count=10

# Creating a block or character device examples

# Create a character device "chr_dev" with major:minor 100:1 and
# a block device "blk_dev" with major:minor 200:200, both with root
# uid/gid and a mode of rw-rw-rw.
chr_dev c 666 root root 100 1
blk_dev b 666 0 0 200 200

You might also use btrfs (send|receive) to stream out a subvolume into whatever stdin-capable compressor you liked. This subvolume need not exist before you decide to use it as compression container, of course.

Still, about squashfs...

I don't believe I'm doing this justice. Here's a very simple example:

 cd /tmp; mkdir ./emptydir
 mksquashfs ./emptydir /tmp/tmp.sfs -p \
    'file f 644 mikeserv mikeserv echo "this is the contents of file"'                             

Parallel mksquashfs: Using 6 processors
Creating 4.0 filesystem on /tmp/tmp.sfs, block size 131072.
[==================================================================================|] 1/1 100%
Exportable Squashfs 4.0 filesystem, gzip compressed, data block size 131072
        compressed data, compressed metadata, compressed fragments,... 
###...
###AND SO ON
###...

echo '/tmp/tmp.sfs /tmp/imgmnt squashfs loop,defaults,user 0 0'|
    sudo tee -a /etc/fstab >/dev/null

mount ./tmp.sfs     
cd ./imgmnt
ls

total 1
-rw-r--r-- 1 mikeserv mikeserv 29 Aug 20 11:34 file

cat file

this is the contents of file

cd ..
umount ./imgmnt

That's only the inline -p argument for mksquash. You can source a file with -pf containing as many of those as you like. The format is simple - you define a target file's name/path in the new archive's filesystem, you give it a mode and an owner, and then you tell it which process to execute and read stdout from. You can create as many as you like - and you can use LZMA, GZIP, LZ4, XZ... hmm there are more... compression formats as you like. And the end result is an archive into which you cd.

More on the format though:

This is, of course, not just an archive - it is a compressed, mountable Linux file-system image. Its format is the Linux kernel's - it is a vanilla kernel supported filesystem. In this way it is as common as the vanilla Linux kernel. So if you told me you were running a vanilla Linux system on which the tar program was not installed I would be dubious - but I would probably believe you. But if you told me you were running a vanilla Linux system on which the squashfs filesystem was not supported I would not believe you.

  • Mike, could we trouble you to create a small self-contained example so that people can experiment with it? It looks you might be doing at least part of that above, but I'm not sure. In input f 444 root root dd if=/dev/sda1 bs=1024 count=10 is f the file input? Perhaps it would be better to create a toy device, fill it with data, and write from it? And does all this require root? – Faheem Mitha Aug 20 '14 at 14:58
  • @FaheemMitha - yes I can do that, but I didn't do it here. The link is to the official documentation - it's taken straight from it. It would we better if I did a command example though. I've done it before - it's pretty cool. Anyway - the input file is a file in the squashfs archive - the filesystem image that results from running the command. When you do mksquash you can specify these pseudofile commands for commands that are run and from which the stdout is captured at compress time. – mikeserv Aug 20 '14 at 15:20
  • @FaheemMitha - oh, and it does not require root to do the compressing, though it may to do the mounting - it is a filesystem image that results. It's the same filesystem all of the Linux Live discs use. In fact - one very cool thing - is that you can create a root owned image using those pseudo-files without being root - like setting your device files and arbitrary MAJ:MIN numbers. – mikeserv Aug 20 '14 at 15:23
  • I guess it should be possible to create a device file, write to it, and then from it without ever mounting it, right? So, maybe it does not require root, which would obviously be preferable. – Faheem Mitha Aug 20 '14 at 15:26
  • Well, there is no btrfs involved here, so that won't work. But squashfs is crazy enough it might work. Though it has the downside of not being a common archive format. – derobert Aug 20 '14 at 17:46
4

You problem puzzled me for some time, and I think I have found a solution that would work.

I think you can achieve what you want with 7z using the -si{NAME} flag.

You will be able to adapt to your need.

7z a test.7z -siSDA2.txt < /dev/sda1
7z a test.7z -siSDA2.txt < /dev/sda2

7z l test.7z 

7-Zip [64] 9.20  Copyright (c) 1999-2010 Igor Pavlov  2010-11-18
p7zip Version 9.20 (locale=en_US.UTF-8,Utf16=on,HugeFiles=on,8 CPUs)

Listing archive: test.7z

--
Path = test.7z
Type = 7z
Method = LZMA
Solid = -
Blocks = 2
Physical Size = 1770
Headers Size = 162

   Date      Time    Attr         Size   Compressed  Name
------------------- ----- ------------ ------------  ------------------------
2014-08-19 22:01:08 .....         6314          804  SDA1.txt
2014-08-19 22:01:11 .....         6314          804  SDA2.txt
------------------- ----- ------------ ------------  ------------------------
                                 12628         1608  2 files, 0 folders

EDIT : Remove the Useless use of cat

  • It would be helpful to have a small example which people can try out. E.g. create a block device, write to it, then write out from it. Not requiring root would be a plus. – Faheem Mitha Aug 20 '14 at 16:53
  • In the example /dev/sda1 is a block device. The cat command has the purpose of dumping the content of the device to stdout.Then 7z create(or update) the archive, and store the data in the filename specified by -si parameter from stdin. The result within the archive is the content of each block device(s). The "cat" command needs root to read the data from device. – Tony Aug 20 '14 at 17:15
  • That is a Useless Use of Cat, but otherwise fits the bill pretty well. Oddly my 7z manpage doesn't mention -si can take a file name, but it works. It's not perfect (output can't be piped somewhere), but is definitely the best so far that outputs in a common format. – derobert Aug 20 '14 at 17:42
  • @FaheemMitha requiring root or not is going to depend on permission settings on your system, though only root can create new block devices. – derobert Aug 20 '14 at 17:43
  • @derobert Removed the cat :) – Tony Aug 20 '14 at 17:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.