How can I apply `cut` to several files and then `paste` the results?

Question

I often do operations like

paste <(cut -d, -f1 file1.csv) <(cut -d, -f1 file2.csv)

which is very tedious with more than a few files.

Can I automate this process, e.g. with globbing? I can save the cut results with

typeset -A cut_results
for f in file*.csv; do
    cut_results[$f]="$(cut -d, -f1 $f)"
done

but I'm not sure how to proceed from there.

Answer 1

You can automate this with globbing, specifically the e glob qualifier, plus eval, but it isn't pretty and the quoting is tricky:

eval paste *.csv(e\''REPLY="<(cut -d, -f1 $REPLY)"'\')

• The part between \'…\' is some code to execute for every match of the glob. It is executed with the variable REPLY set to the match, and can modify it.
• I put the code in single quotes so that it isn't expanded when the glob is parsed.
• The code REPLY="<(cut -d, -f1 $REPLY)" generates the string <(cut -d, -f1 file1.csv) if the match is file1.csv. The double quotes are necessary so that the part after the equal sign isn't expanded when the e code is executed apart from substituting the value of REPLY.
• Since each globbed file is replaced by a string,

It would be nicer to hide the complexity in a function. Minimally tested.

function map {
  emulate -LR zsh
  local cmd pre
  cmd=()
  while [[ $# -ne 0 && $1 != "--" ]]; do
    cmd+=($1)
    shift
  done
  if ((!$#)); then
    echo >&2 "Usage: $0: COMMAND [ARGS...] -- PREPROCESSOR [ARGS...] -- FILES..."
    return 125
  fi
  shift
  while [[ $# -ne 0 && $1 != "--" ]]; do
    pre+="${(q)1} "
    shift
  done
  if ((!$#)); then
    echo >&2 "Usage: $0: COMMAND [ARGS...] -- PREPROCESSOR [ARGS...] -- FILES..."
    return 125
  fi
  shift
  eval "${(@q)cmd}" "<($pre${(@q)^@})"
}

Sample usage (the syntax is reminiscent of zargs):

map paste -- cut -d, -f1 -- *.csv

Answer 2

I think your first line is about as good as it gets for a simple one-liner.

If there are a bunch of files with all different names, you could reduce the repetitive typing a bit with a simple history expansion "cheat":

First run <(cut -d, -f1

Note the trailing space. Also note that this command will give you a secondary prompt; just press Ctrl-C. The only point is to add it to the history.

Next run paste !!file1.csv) !!file2.csv)

The !! will expand to the full content of the previous command run, including the trailing space. Note that if you forget the trailing close parentheses you will get a secondary prompt; you can just type Ctrl-C and try again if this happens.

This is a bit hacky but good enough for a one-time use. If you're doing it a lot, you might write a bash function.

Answer 3

Try awk

awk '{L[FNR]=L[FNR] $1 "\t"}END{for(i=1;i<=FNR;i++)print L[i]}' *.csv

or paste with sed

paste *.csv | sed 's/ [^\t]*//g'

Answer 4

I'm studying bash scripting at the moment, and this seemed an excellent simple task to practice with, so I wrote the following. (My other answer gives the simple history expansion hack, but this is a full script and I deemed it worthy of making an additional answer.) I believe this is POSIX compatible and should work with #!/bin/sh, but not 100% sure. EDIT: Actually, the =~ is not POSIX compatible. You could take that check out and let cut return the error, though.

#!/bin/bash

fieldtocut=1
delimiter=','

usage () {
    cat << EOF
usage: $0 [-f FIELD] [-d DELIMITER] file1..
Cuts field FIELD from each file and pastes it.
Default field is 1, default delimiter is ','
EOF
    exit $1
}

while getopts ':f:d:' opt ; do
    case $opt in
        f)
            if [[ $OPTARG =~ ^[0-9]+$ ]] ; then
                fieldtocut="$OPTARG"
            else
                usage 1
            fi
            ;;
        d)
            delimiter=$OPTARG
            ;;
        *)
            usage 1
            ;;
    esac
done
shift $((OPTIND-1))

[ $# -eq 0 ] && usage 0

pasteargs=''

for file in "$@" ; do
    pasteargs=$(printf '%s' "$pasteargs" '<(cut -d$delimiter -f$fieldtocut ' "$file" ') ')
done

eval paste $pasteargs

Answer 5

Assuming your arguments are in "$@", I believe something like:

eval "paste $(printf "<( cut -d, -f1 %q ) " "$@")"

should do it.

Answer 6

Here's another way to do it that's very similar to the answer by Wildcard:

files=( file1.csv file2.csv)
eval paste "<( cut -d, -f1 ${^files[@]} )"

Instead of a for loop, this uses the ${^ ... } expansion which is Zsh-specific.

The reason files must be assigned first is that globbing is always done last, so if files needs to be generated automatically (as in files=( *.csv )) then something like ${^:-( *.csv )} would expand only after all the other expansions have occurred. We want it to expand first.

The ${^ ... } expansion causes the resulting array to act like the result of brace expansion. For example, assign x=(a b) and then compare echo ${x}y to echo ${^x}y.

The quoting is necessary to trick Zsh into treating the surrounding text like a literal string. Otherwise, it would split the command line at the spaces, so our ${^ ... } expansion would reduce to ""${^ ... }""; that is, each element would be surrounded only by an empty string. That is,

echo "<( cut -d, -f1 ${^files[@]} )"

and

echo "<( cut -d, -f1 "\
${^files[@]}\
" )"

are equivalent, but are not the same as

echo <( cut -d, -f1 ${^files[@]} )

But quoting introduces a new problem: the command line is parsed and split without regard to the expansion taking place. That is, even though we have effectively entered

paste <( cut -d, -f1 file1.csv ) <( cut -d, -f1 file2.csv )

as desired, this is in fact parsed as

paste '<( cut -d, -f1 file1.csv )' '<( cut -d, -f1 file2.csv )'

Therefore we need eval to re-parse the correctly formed expression. To see this in action, compare

setopt noxtrace
eval paste "<( cut -d, -f1 ${^files[@]} )" 1>/dev/null 2>&1

to

setopt xtrace
eval paste "<( cut -d, -f1 ${^files[@]} )" 1>/dev/null 2>&1

I hoped that some combination of nested expansions, the ${ ... :- ... } expansion, and the parameter expansion flags Q, z, and/or s would lead to re-evaluation without eval, but evidently that's not the case. I also wish there was a way to force globbing, but again that seems impossible.

Answer 7

You could get awk to loop through the files in lockstep and report the field of interest from each file. Put this code into a file, say cut_files.awk

NR == FNR{printf "%s%s",$1, FS;
for (k=2; k<ARGC; ++k)
    {getline < ARGV[k]; printf "%s%s", $1, k==ARGC-1?"\n":FS}; next};
NR != FNR{for (k=2; k<ARGC; ++k) close(ARGV[k]); exit}

And then call it like so

awk -F',' -f cut_files.awk file1 file2 file3 file4 ....