2

*edit: I've decided to append all the outputs to a single file.

I have 40 csv files that I need to edit. 20 have matching format and the names only differ by one character e.g., docA.csv, docB.csv, etc. The other 20 also match and are named pair_docA.csv, pair_docB.csv, etc.

I have the code written to edit and combine docA.csv and pair_docA.csv, but I'm struggling writing a loop that calls both the above files, edits them, and combines them under the name combinedA.csv, then goes on the the next pair.

Can anyone help my rudimentary bash scripting? Here's what I have thus far. I've tried in a single for loop, and now I'm trying in 2 (probably 3) for loops. I'd prefer to keep it in a single loop.

set -x
DIR=/path/to/file/location

for file in `ls $DIR/doc?.csv`
do

#code to edit the doc*.csv files ie $file
#output is called temp_doc*.csv

done

for pairdoc in `ls $DIR/pair_doc?.csv`
do

#code to edit the piar_doc*.csv files ie $pairdoc
#output is called temp_pair*.csv

done

#still need to combine the files. I have the join written for a single iteration, 
#but how do I loop the code to save each join as a different file corresponding
#to combined*.csv
2
  • Are the doc in the same directory? – Jason Croyle Feb 28 at 0:44
  • Yes, all the forms are in the $DIR. I am running a few commands (not shared here) that run scripts in other directories though. – swgRrr Feb 28 at 0:52
4
set -x
DIR=/path/to/file/location
TMPDIR=$(mktemp -d)
rm -f -- "$DIR/combined.csv"

for file in "$DIR"/doc?.csv
do

doc=${file##*/}

#------------ doc processing
temp_doc="$TMPDIR/$doc"

your_code "$file"  > "$temp_doc"

#------------- pair doc processing
pair="$DIR/pair_$doc"
temp_pair="$TMPDIR/$pair"

your_code  "$pair" >  "$temp_pair"


#--------- combine doc+pair
combined="$DIR/combined${doc/doc/}"

your_code "$temp_doc"  "$temp_pair"  > "$combined"
 

cat "$combined" >> "$DIR/combined.csv"
done

Note: substitute your processing codes in the three steps above

  • doc=${file##*/} What the above construct does is strip upto the last / starring from left in the shell variable $file. Remember, it has a head portion of $DIR . We just want the plain basename for this and store the result in another shell variable $doc.

  • For the purpose of concatenating all into one file, I have added the last cat line.

3
  • You really laid it all out there. Thank you! Can you walk me through the ## and the / in doc=${file##*/} – swgRrr Feb 28 at 1:45
  • Also, I think I've decided it would be more prudent to append the outcomes into a single combined.csv file. Is that what your code is doing on the last line? – swgRrr Feb 28 at 1:57
  • Added an explanation plus added a cat command to get the combined file. Please check. – guest_7 Feb 28 at 3:45
1

DISCLAIMER: This is by no means the best way to do this or by any measure the best approach but it show how to think about these types of problems if your just doing a one off run something like this is fine but should never be used by someone other then the person writing the scripts ie. DO NOT put on a computer others use. The basis principal is think simple most jobs are when broken down to there smallest steps.

I have left all the iterations on this post so you can see how very simple changes can change the outcome so you can compare them. The difficulty comes when you have to start testing for the inputs none of that code is present that is the code that would make this something useful ie. adding input for different file names, adding multiple checks for input arguments, ect. Again I repeat myself it is ok to do things like this as a one off but should never be used as a complete answer for something that is of a recurring nature.

#!/bin/bash
    
     for n in $(ls -1 doc*.csv);
     do
     cat $n >> combinded$n
     cat pair_$n >> combinded$n
  
     mv combinded$n $(echo combinded$n | sed 's/doc//g')
     done

This will take files named docA.csv and pair_docA.csv and combinded them into one file called combinedA.csv leaving the originals untouched. It will do this for all files in the directory following the same naming pattern.

NOTE: This must be run in the directory the files are in and the files must be named as they appear here.

TESTING:

doc_A.csv doc_B.csv pair_docA.csv pair_docB.csv

OUTPUT:

combindedA.csv combindedB.csv

EDIT: To answer your question about outputting everything to one file this would do that into a file called combinded.csv

#!/bin/bash
for n in $(ls -1 doc*.csv);
do
cat $n >> combinded.csv
cat pair_$n >> combinded.csv
done

This will output into one file in a format like this:

  • docA.csv
  • piar_docA.csv
  • docB.csv
  • pair_docB.csv

ect. till the end of the file.

EDIT: To add a directory as input and output gets a little harder you have to test for the arguments this script takes two arguments the first is the directory that contains the files and the second argument is the location of the output file both arguments are MANDATORY.

  #!/bin/bash

  dir=$1
  out=$2
  cd $1
  for n in $(ls -1 doc*.csv);
  do
  cat $n >> combinded$n
  cat pair_$n >> combinded$n

  mv combinded$n $out/$(echo combinded$n | sed 's/doc//g')
  done

EXAMPLE: ./script.sh /path/to/files /path/to/output

BOTH ARGUMENTS ARE NEEDED

10
  • Thanks! Can you walk me through what the sed is doing? – swgRrr Feb 28 at 0:54
  • sure the sed is just removing the word doc from combinded if you remove the sed command you end up with a output file named combindeddocA.csv – Jason Croyle Feb 28 at 0:58
  • The sed command step by step mean "s" = search "doc" = string to search for and because there is nothing between the last two // replace "doc" with nothing and the "g" = global do this over the whole string. – Jason Croyle Feb 28 at 0:59
  • Thanks again! You say this has to be run the directory the files are in, but can't I just assign $DIR before the for loop and reference it in my for loop like I did originally? – swgRrr Feb 28 at 1:41
  • Sure I'll change the script when I get back home shouldn't be very long when I do I leave you another comment. – Jason Croyle Feb 28 at 1:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.