0

I'm new to unix/bash/code in general, and I'm trying to merge multiple sample files with the same name in different directories into a new file in its own directory (the outputs of multiple data collections).

To do this, I'm trying to create a script which uses two arrays - listA, the name of each sample file, and list B, the names for the newly merged sample files.

My code looks a little like this:

#!/bin/sh

listA=( old1 old2 old3 etc.)
listB=( new1 new2 new3 etc.)

i=0

for i in $listA $listB

do

cp ./folder1/$listA ./merged/$listB
cat ./folder2/$listA >> ./merged/$listB
cat ./folder3/$listA >> ./merged/$listB
cat ./folder4/$listA >> ./merged/$listB

((i=+1))

done
echo "Done stitching"

As is, it seems to merge the files for the first entry in listA into the first file in listB perfectly, but it won't repeat the process for the subsequent entries on the list.

Any advice to make this work as intended? Apologies for my ignorance, I'm very new to all of this and enjoying the learning process immensely - just a bit stumped.

  • Are the files with consecutive names? Like file1 file2 and so on. Is it only the base name a difference between source and target filenames? – Romeo Ninov Feb 27 '18 at 8:42
  • Do you have control over the format of listA and listB? At the moment, they are two independent arrays. It'd be simpler to use a dictionary, if you are happy to use Bash 4. – Sparhawk Feb 27 '18 at 8:50
  • @RomeoNinov unfortunately no, it would make my job easier if they were, but they all have nonconsecutive sample IDs – bellicosemagpie Feb 27 '18 at 11:56
  • @Sparhawk I do & I have Bash 4, I'll have to do some reading on dictionaries. – bellicosemagpie Feb 27 '18 at 12:04
  • Having said that, even though dictionaries are probably the simplest way to parse, it'd be fiddly to enter the lists. What about feeding the script a text file of tab-delimited pairs of files instead? Would that be easiest to create? How many files are we talking about? – Sparhawk Feb 27 '18 at 20:37
0

Short answer

while IFS=$'\t' read -r old new; do cat folder*/"$old" > merged/"$new"; done <list.tsv

This assumes that list.tsv contains a tab-delimited list of source (old) and destination (new) files.

Explanation

  • while IFS=$'\t' read -r old new; FOO; done <list.tsv: this is a common shell contruct to read each line of list.tsv, then do FOO on each line. In this case, the input file separator IFS is a tab $'\t', i.e. the first field of the line is saved to $old, and the second field (after the tab) is saved to $new.
  • cat folder*/"$old" > merged/"$new": concatenate the files in folder*/"$old", and write this to merged/"$new". This assumes that the source files are all in folder*, e.g. named folder1, folder2, etc.
  • As with any script, I would certainly test this before running it. You can do that by copying a subset of your files to another directory, then using a short version of your input list list.tsv. Check the results, and if okay, then run it for real (but keep a backup!).

Notes on your suggested solution

I just thought I'd write a few comments on your proposed solution, in order to clarify how the various commands work. Hopefully this is helpful to your understanding!

  • listA=( old1 old2 old3 etc.). This creates an array. The way to access parts of it is shown here. Using for i in $listA… will just access the first entry, as you have (kinda) observed.
  • for i in a b c. This iterates over a, b, and c, putting each into $i. For example, try running for i in a b c; do echo $i; done. N.B. it makes no sense to refer to $listA within the do…done block. Similarly, i=0 and ((i=+1)) are inconsistent with for usage.
  • ./folder1/$listA. ./ is unnecessary. . is the current folder, so it doesn't do anything here.
  • cat FOO >>. cat will concatenate by itself (hence the name). There is no need to write separate cat statements that append >> to a file. You can just list multiple files after cat, e.g. cat file1 file2 file3 > output_file.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.