2

I'm trying to concatenate files from three different subdirectories into one file. The names of the files in each subdirectory are exactly the same. I'd like to use a loop to iterate through the subdirectories, and then place an iteration number into the newly named concatenated file, in a new directory. For example a directory structure like:

Foo
|||
||Bar3
|Bar2
Bar1

Inside each Bar(?) folder are files named: File1, File2, File3

I'd like to concatenate the Files with the same names to a larger file with a new name including a number:

cat Foo/Bar1/File1 Foo/Bar2/File1 Foo/Bar3/File1 > /combined_files/all_file1

cat Foo/Bar1/File2 Foo/Bar2/File2 Foo/Bar3/File2 > /combined_files/all_file2

cat Foo/Bar1/File3 Foo/Bar2/File3 Foo/Bar3/File3 > /combined_files/all_file3

From the Foo directory I can use:

for number in {1..3}
    do
    cat Bar1/File$number\_* Bar2/File$number\_* Bar3/File$number\_* > combined_files/'all_files'$number
    done
exit

But I need to have a more universal script, for larger numbers of Bar directories, and Files. I want something like

files=`ls ./Run1/ | wc -l`   #to count the number of files and assign a number
For n in {1..$files}
    do
    cat Bar1/File$n\_* Bar2/File$n\_* Bar3/File$n\_* > combined_files/'all_files'$n
    done

But I'm stuck.

  • 1
    Do you want to use all of Barx? Will x always be a single digit? – drewbenn Dec 10 '18 at 21:00
  • Yes I want all the files in BarX, but no X will go from 1 to a large number. – hoytpr Dec 10 '18 at 21:29
  • Does order need to be preserved between each run? Eg should Bar1/File2 always appear in the concatenated output before Bar2/File1? This will affect whether it can be accomplished using a simple one-liner, or whether a more complicated (perhaps recursive) result will be required. – cryptarch Dec 10 '18 at 21:32
  • Unfortunately yes. The files are ordered so the concatenation needs to be ordered. – hoytpr Dec 10 '18 at 21:34
  • This is similar to a question asked a few months ago, but the answers didn't seem to work for me, or maybe I don't interpret them correctly – hoytpr Dec 10 '18 at 21:49
1
#!/bin/sh

for pathname in Foo/Bar1/File*; do
    filename=${pathname##*/}
    cat "$pathname" \
        "Foo/Bar2/$filename" \
        "Foo/Bar3/$filename" >"combined/all_$filename"
done

This would loop over all files whose names matches File* under Foo/Bar1 (we assume that the pattern matches exactly the names we are actually interested in).

For each such file, we extract the filename portion of the pathname, yielding $filename (this could also have been done with filename=$(basename "$pathname")). We then concatenate the original file with the corresponding files in the Foo/Bar2 and Foo/Bar3 directories, writing the result to a new all_$filename file in some other directory.


With a bit of error checking:

#!/bin/sh

for pathname in Foo/Bar1/File*; do
    if [ ! -f "$pathname" ]; then
        printf '% is not a regular file, skipping\n' "$pathname" >&2
        continue
    fi

    filename=${pathname##*/}

    if [ -f "Foo/Bar2/$filename" ] &&
       [ -f "Foo/Bar3/$filename" ]
    then
        cat "$pathname" \
            "Foo/Bar2/$filename" \
            "Foo/Bar3/$filename" >"combined/all_$filename"
    else
        printf 'Missing %s or %s\n' "Foo/Bar2/$filename" "Foo/Bar3/$filename" >&2
    fi
done

A variation that also allows for a varied number of BarN subdirectories. It is assumed that each BarN directory is numbered sequentially from 1 to some large number.

#!/bin/sh

# This is just used to count the number of BarN subdirectories.
# The number of these will be $#.
set -- Foo/Bar*/

for pathname in Foo/Bar1/File*; do
    filename=${pathname##*/}

    n=1
    while [ "$n" -le "$#" ]; do
        if [ ! -f "Foo/Bar$n/$filename" ]; then
            printf '%s missing, %s will be incomplete\n' \
                "Foo/Bar$n/$filename" "combined/all_$filename" >&2
            break
        fi

        cat "Foo/Bar$n/$filename"
        n=$(( n + 1 ))
    done >"combined/all_$filename"
done
  • Just to make shorter, in your last code what if write cat "Foo/Bar1/$filename" >"combined/all_$filename" above n=1 and then make n=2 instead of 1. So it will save one comparison. – Prvt_Yadv Dec 10 '18 at 23:07
  • @Debian_yadav Or just remove the first if statement (which I just did because it looks nicer). In either case, the timing difference will be minuscule. – Kusalananda Dec 10 '18 at 23:15
  • Thanks @Kusalananda very much for the great explanation and code. My directory structure is not as perfect as I described. The BarX directories will be sequentially numbered, but the Filenames can vary from my original post. Using File* doesn't always capture the filename correctly. Would it be possible to use something from the other post, like: ${FILE#*/} to capture the entire filename? I will take the code you have written and work with it tonight. Thank-you. – hoytpr Dec 10 '18 at 23:28
  • @hoytpr If you want to match all files in Bar1 (that are not hidden filenames), just use Foo/Bar1/* instead of Foo/Bar1/File*, or construct some other pattern that will match the names that you are interested in. – Kusalananda Dec 12 '18 at 22:35
  • @Kusalananda; yes, that's what I should have been doing, but was overthinking when it was the directory structure using Foo/BarX/Filex when I should have been using just BarX/Filex or as you say; BarX/* when running the script from inside the Foo directory. Thanks! – hoytpr Dec 13 '18 at 19:56
0

Thank you again @Kusalananda and @Debian_yadav. I was able to make your script work on my system. My actual directory names are now:

Joes ||| ||Run3 |Run2 Run1

Inside each RunX directory I created Files named the same but with different contents

Run1\File1
Run2\File1
Run3\File1

First I ran the simple script you showed, slightly modified to my directory structure:

ANSWER

#!/bin/bash
for pathname in Run1/File*; do
    filename=${pathname##*/}
    cat "$pathname" \
        "Run2/$filename" \
        "Run3/$filename" > "RunCat/all_$filename"
one

The script output was a file "allFile1" with contents: 123

Your longer (final) script (I named K2script.sh) also worked on my system.
The output exactly the same again after slightly modifying the directory structure:

ANSWER

#!/bin/sh
# This is just used to count the number of RunN subdirectories.
# The number of these will be $#.
set -- Joes/Run*/

for pathname in Run1/File*; do
    filename=${pathname##*/}

    n=1
    while [ "$n" -le "$#" ]; do
        if [ ! -f "Run$n/$filename" ]; then
            printf '%s missing, %s will be incomplete\n' \
                "Run$n/$filename" "RunCat/all_$filename" >&2
            break
        fi

        cat "Run$n/$filename"
        n=$(( n + 1 ))
    done >"RunCat/all_$filename"
done 

Also, using the script from
an other stackexchange discussion

ANSWER
Changing the folder names made it work on my system.

#!/bin/bash
for FILE in Run1/* ; do
    FILE2=Run2/${FILE#*/}
    FILE3=Run3/${FILE#*/}
    if [ -f $FILE2 ] ; then
        cat $FILE $FILE2 $FILE3 > RunCat/${FILE#*/}
    fi
done

I've learned a lot about while [ "$n" -le "$#" ]; do and how ${pathname##*/} works, but wasn't able to fully grasp why ${FILE#*/} worked when other wildcard symbols or regular expressions would not.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.