3

We have a cpio archive that was created by generating a file that contains a list of absolute paths to be included in the archive. (one absolute path per line of a plain text file) The command to generate the archive is essentially:

cat list-of-files | cpio -ocvB > preserved.cpio

We later need to extract files from that archive. We again want to use a file that contains a list of files to be extracted (some subset of all of the files in the archive, again with the format of one absolute path per line of a plain text file).

cpio -icuBdmv `cat files-to-extract` < preserved.cpio

This works fine unless one of the paths contains a space. Generating the archive is fine, but when extracting the files, any file with a space in the name is silently skipped. All other paths in files-to-extract are successfully extracted.

I've been playing at the console trying to come up with some way to work around this, but to no avail. If I specify a single file with a space in the name and wrap it in quotes, the file is extracted successfully:

# This extracts the file successfully
cpio -icuBdmv "/foo/bar/some file.txt" < preserved.cpio

So I could read files-to-extract in a loop and extract each file one at a time, but these archives can be large (multiple GB), so that is dreadfully slow.

I tried a couple things to somehow try to either escape the spaces in file paths or quote each path value, but nothing I've tried has worked.

# Still skips the file with spaces:
cpio -icuBdmv `cat files-to-extract | sed 's/ /\\ /'` < preserved.cpio
# Extracts no files, even the ones without spaces:
cpio -icuBdmv `cat files-to-extract | sed 's/\(.*\)/"\1"/'` < preserved.cpio

It would really be nice to be able to do this extraction with a single run of cpio rather than having to loop and extract one file at a time. I'm sure this is just a problem with how I am providing those values to cpio, but for the life of me I cannot figure out why.

Probably not relevant, but just for completeness: this is on CentOS 7.2, using GNU cpio 2.11

Improve this question
3
  • Quoting in the result of $( ) substitution doesn't work, unless eval'ed or otherwise reparsed which gets very tricky. What you can do is set IFS='\n' before the command, preferably in a subshell so it automatically reverts: ( IFS='\n'; cpio -icuBdmv $(cat names) <archive ). But note if the substituted results are too big for your system's ARG_MAX this will fail, while xargs as in @Ian's answer will break it into groups small enough to run but still much larger and more efficient than one at a time.
    – dave_thompson_085
    May 27, 2016 at 6:41
  • cas' solution is the simple, obvious one that I failed to notice while looking at the man page, as they sorta implied :) I'm not a fan of info, but still, I should have seen the -E option where I was looking. Works like a charm, thanks!
    – Michael L.
    May 27, 2016 at 17:19
  • info docs are a lot nicer to read if you use pinfo rather than the FSF's own info. The pinfo UI is kind of like lynx, but for info files. I'd still prefer to have a proper man page (with at least a summary description of each option) as well as the .info reference manual.
    – cas
    May 28, 2016 at 7:31

2 Answers 2

Reset to default
3

cpio has a -E (--pattern-file) option, which allows you to read the list of filenames from a file instead of (or as well as) providing the filenames on the command line. For example:

cpio -icuBdmv -E files-to-extract < preserved.cpio

cpio also has -F to specify an archive name (instead of using stdin/stdout). -I and -O are similar but work instead of stdin or stdout respectively.

e.g. you can specify the archive with -F and provide the files list on stdin:

cpio -icuBdmv -F preserved.cpio < files-to-extract

or use both -E and -F

cpio -icuBdmv -E files-to-extract -F preserved.cpio

BTW, with many GNU programs (including cpio, the man pages are almost useless, but they are well-documented in .info files. On some Linux distributions, the info docs are often separate packages (e.g. on debian, cpio-doc), so you'll need to install them as well as an info reader (such as GNU info or pinfo)

anyway, here's some relevant extracts from the cpio info pages:

-E FILE, --pattern-file=FILE

Read additional patterns specifying filenames to extract or list from FILE. The lines of FILE are treated as if they had been non-option arguments to cpio. This option is used in copy-in mode,

and

-F ARCHIVE, --file=ARCHIVE

Archive filename to use instead of standard input or output. To use a tape drive on another machine as the archive, use a filename that starts with HOSTNAME:, where HOSTNAME is the name or IP address of the machine. The hostname can be preceded by a username and an @ to access the remote tape drive as that user, if you have permission to do so (typically an entry in that user's ~/.rhosts file).

Improve this answer
3

Spaces in file names can be tricky, but I think this might help.

Instead of cating your files-to-extract, try feeding it through xargs substituting the NULL character for EOL.

Something like

tr '\n' '\0' < list-of-files | xargs -0 -I{} sh -c 'cpio -icuBdmv "{}" < preserved.cpio'

should do the trick.

Improve this answer
6
  • given that the source file list-of-files didn't start off with NUL-separated records, tr won't and can't magically make it so. you may as well just just xargs -d'\n' .
    – cas
    May 27, 2016 at 11:48
  • @cas I don't think that's correct. tr is changing the carriage-returns into NULL characters, then xargs is using those as record separators to feed to cpio. However, it doesn't really matter, you provided a better and more thoroughly explained answer anyway :-)
    – Ian Petts
    May 29, 2016 at 9:20
  • 1
    yes, your tr command is changing newlines (NOT carriage returns) into NULs. However, if list-of-files contains any filenames with a newline in them (newline is an annoying but completely valid character in a filename), it will convert those newlines along with all the others. There is no way to tell the difference between a newline at the end of a filename and a newline embedded in the middle filename, therefore you can't convert newline-separated into NUL-separated. That's why I said that you may as well use xargs -d'\n'. It works exactly the same, without the pointless tr.
    – cas
    May 29, 2016 at 9:28
  • and without fooling yourself or others that you're properly handling ALL valid filenames. Which is where the problem lies. Using tr like this doesn't matter here (because cpio takes a \n-delimited list of names, anyway). The danger is in believing that the tr actually does something useful and worthwhile here (it doesn't), and then using it in a context where it does matter.
    – cas
    May 29, 2016 at 9:28
  • @cas That makes sense. Thank you for the clarification.
    – Ian Petts
    May 29, 2016 at 10:15

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .