293 reputation
18
bio website
location Zurich, Switzerland
age
visits member for 1 year, 10 months
seen yesterday

Apr
2
revised Virtual write-only file system for storing files in archive
added 121 characters in body
Mar
14
comment Virtual write-only file system for storing files in archive
Well, 16 MB is a large buffer to me. Anyway, thanks for the perl snipped, sed '$d' works too.
Mar
14
awarded  Scholar
Mar
14
comment Virtual write-only file system for storing files in archive
This looks like an option, but from your answer I fail to see how to create, say, a squashfs archive with a directory test and a file file in this directory. Could you please provide a brief example?
Mar
14
comment Virtual write-only file system for storing files in archive
I do get impressive compression ratios with a solid compressor for my use case, but, as you outlined, I assume the results would be worse if the files were larger than the dictionary size.
Mar
14
accepted Virtual write-only file system for storing files in archive
Mar
12
revised Virtual write-only file system for storing files in archive
added 541 characters in body
Mar
12
comment Virtual write-only file system for storing files in archive
I haven't found a way to suppress the final line in the output of cpio, other than grep -v 'blocks$'. (head -n -1 uses a very large buffer...) Makes this solution a bit of a hack, but never mind ;-)
Mar
12
comment Virtual write-only file system for storing files in archive
Will it compress across files?
Mar
12
comment Virtual write-only file system for storing files in archive
How would this work within the infrastructure I outlined?
Mar
12
comment Virtual write-only file system for storing files in archive
My files are about 100k each. Wouldn't be enough to allow the compressor to use a window of, say, 1M? xz seems to operate with a default dictionary size of 8M (at default compression level -6), which seems to be plenty for my use case. -- Diffs to a reference file are nice, but requires to construct a reference file first. Would a compressing file system detect files with near-identical content?
Mar
12
comment Virtual write-only file system for storing files in archive
I can confirm that tar reads the file list first, using the simple example I've added to my question. However, reading tar's source code again, it seems to me that it should read the list of files "on the fly" if not creating an incremental archive. Unfortunately, I have errors compiling tar from source... :-(
Mar
12
revised Virtual write-only file system for storing files in archive
added 131 characters in body
Mar
12
comment Virtual write-only file system for storing files in archive
Thanks. So, even RTFM is not enough ;-) I even looked into tar's code to see that there's a function that returns the next file name to be processed, which made me read the documentation again. -- So, stdout is directed to the gzip process via process substitution, and stderr is redirected to stdout which is processed by the next step in the pipe?
Mar
12
revised Virtual write-only file system for storing files in archive
added 44 characters in body
Mar
11
revised Virtual write-only file system for storing files in archive
added 2 characters in body
Mar
11
comment Virtual write-only file system for storing files in archive
@StephaneChazelas: Can this be implemented in userland?
Mar
11
comment Virtual write-only file system for storing files in archive
@casey: It's HTML, but I suppose I could use an HDF5 container.? Haven't considered this yet.
Mar
11
awarded  Self-Learner
Mar
11
revised Virtual write-only file system for storing files in archive
added 13 characters in body