1

I have a series of actions I need to preform on pairs of files, on a fair number of files. For simplicity, I'll focus this down on just doing a simple cat on a pair, to discuss the pairing and how to write the most straight forward pairing part of shell script.

Say I have 4 files, A.txt, B.txt, C.txt, and D.txt, and I want to write a compact script that will basically do:

 cat A.txt B.txt > AB.txt
 cat A.txt C.txt > AC.txt
 cat A.txt D.txt > AD.txt
 cat B.txt C.txt > BC.txt
 cat B.txt D.txt > BD.txt
 cat C.txt D.txt > CD.txt

I want to have one output for each unique combination, and AD.txt and DA.txt are not "unique" by this criteria.

But I'd like to make it a bit easier than as a shell script, that I can do for different sets of files, and just run it in a directory, and have it find all matches recursively. I immediately seem to have went the wrong direction, and made a mess of things:

find "$PWD" -type f -iname "*.txt" -exec [[SOME MAGIC CODE CREATING PAIRS OF FILE NAMES]] {} \; 
 \ cat "$MAGICPAIRfile1".txt "$MAGICPAIRfile2".txt >  
 \ "$MAGICPAIRfile1"-"$MAGICPAIRfile2".txt 

was thinking of exec'ing a couple pieces in there, one that dumps file names to a text buffer (bad buffer type for file name character strings, so I didn't), and then pass that buffer to yet another exec {} \;.

But I thought someone else might have a good idea?

3
  • 1
    Re recursiveness: should ./A be paired with ./dir/B, or do you just need ./A+./B (in each dir)?
    – rowboat
    Dec 10 '20 at 7:11
  • also based on what conditions first file should be A.txt and second B.txt in cat A.txt B.txt > AB.txt, or being first and second files doesn't matter and any one of XY.txt or YX.txt is fine but not both for X.txt and Y.txt files? also what if you had 5files, I mean if last one file remain single? Dec 10 '20 at 7:21
  • My find criteria changes, so each value will have a full path as input value to the scripted loop, which won't always be 'cat' Dec 11 '20 at 6:48
1

Here's my suggestion.

#!/bin/bash
files="empty"
for i in A B C D ; do
    for j in B C D ; do
     fn="$i$j"
     nf="$( echo $fn | rev )"
     # if nn is 1 $nf wasn't found in $files
      nn=1
      for q in $files ; do
        if [[ "$q" == "$nf" ]] ; then
               nn=0
         fi
        done
        if  [[  $nn -eq 1 ]] && [[ "$fn" != "$nf" ]] 
        then
           echo "cat $i.txt $j.txt >$fn.txt"
        fi
        files="$fn $nf $files"
    done
done
1
  • This is getting there, it's the embedded loop solution I was thinking of, I might edit this? If I can make the A, B, C, D and input from find, I'm pretty sure it might do what I am trying. Dec 10 '20 at 19:30
1

You can save the file arguments of your find command into an array. Also you can sort them before saving. Here, null separation has been used (-d '' for mapfile (==readarray), -print0 for find and -z for sort) which requires GNU utilities.

And do a double loop for them, i is running the whole length and j from i+1 to the end, and create the combinations. You can process into there each combination of the file arguments.

#!/bin/bash
mapfile -d '' arr < <(find . -type f -name '*.txt' -print0 | sort -z)

for ((i=0; i<"${#arr[@]}"; i++)); do
    for ((j=i+1; j<"${#arr[@]}"; j++)); do
        printf "Processing files: %s %s\n" "${arr[i]}" "${arr[j]}"
    done
done
Processing files: ./A.txt ./B.txt
Processing files: ./A.txt ./C.txt
Processing files: ./A.txt ./D.txt
Processing files: ./B.txt ./C.txt
Processing files: ./B.txt ./D.txt
Processing files: ./C.txt ./D.txt

For your specific example, to cat the files and have the desired output filename (assuming they are all in the same directory level), you could use find ... -printf '%f\0', to print only the filenames, and substring removal with parameter expansion, to create the commands. Slightly modified version, using newline separator for the filenames:

#!/bin/bash
mapfile -t arr < <(find . -type f -name '*.txt' -printf "%f\n" | sort)

for ((i=0; i<"${#arr[@]}"; i++)); do
    for ((j=i+1; j<"${#arr[@]}"; j++)); do
        cat "${arr[i]}" "${arr[j]}" > "${arr[i]%.*}${arr[j]}"
    done
done
3
  • I think that might be the magic I need! ``` mapfile -d '' arr < <(find . -type f -name '*.txt' -print0 | sort -z) ``` However, I'm getting: ``` Syntax error: redirection unexpected ``` Dec 10 '20 at 19:22
  • Is ddrescueview required for "mapfile?" Dec 10 '20 at 19:28
  • Ah, yea, silly me, tested by doing "sh bashscript.sh" and wonders why the "bash" parts weren't working! Sorry. Yes, it works just as you said. Thank you! Dec 11 '20 at 6:46
0

If you can use perl and assuming your filenames are "well-behaved":

find ... |
perl -0777 -MMath::Combinatorics -anE \
  'BEGIN{$,=" "}; say sort(@$_) for (combine(2, @F))' |
sort

Output when A\nB\nC\nD\n is the input:

A B
A C
A D
B C
B D
C D

To recreate your example (GNU sed):

... |
sed -E 's/([^.]+).([^ ]+) ([^.]+).([^ ]+)/cat \1.\2 \3.\4 > \1\3.\2/'
cat A.txt B.txt > AB.txt
cat A.txt C.txt > AC.txt
cat A.txt D.txt > AD.txt
cat B.txt C.txt > BC.txt
cat B.txt D.txt > BD.txt
cat C.txt D.txt > CD.txt

Which can then be piped to a shell for execution or done with the /e flag in GNU sed.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.