Apply same sed command on multiple text files

Question

I am doing some pre-processing on files. I have 2 text files which contains data in the following format.

Text File 1

"Name","Age","Class"
"Total Students:","247"
"John","14","8"
"Sara","13","8"

Text File 2

"Name","Age","Class"
"Total Students:","119"
"John","15","9"
"Sara","16","9"

What I am trying to do is I am removing top 2 rows from these files and quotes and then moving the files to the output directory by using the following commands.

sed '1d' "$file" >> temp.txt
sed -i '1d' temp.txt
sed -i 's/"//g' temp.txt

mv temp.txt output/$file

The problem I am facing is that, these commands only apply to a single file. The file names are Class_8.txt and Class_9.txt. Is there any solution I can apply same command to both files? I wanted to retain the orignal file and move the processed files to output folder.

Answer 1

You can't conveniently run sed on multiple files and get it to write to more than one file in one go (if the input and output need to be separate files). It's possible using non-standard extensions or by hard-coding the names of the output files in the sed expressions.

Your operations are so simple though that we may want to use tail and tr in a loop instead:

for file in Class_{8,9}.txt; do
    tail -n +3 "$file" | tr -d '"' >output/"$file"
done

Or, if you really want to use sed,

for file in Class_{8,9}.txt; do
    sed -e '1,2d' -e 's/"//g' "$file" >output/"$file"
done

You could also copy the files first, then run sed with in-place editing in one go on the copies. This delegates, in a sense, the loop to the inner workings of GNU sed.

cp Class_{8,9}.txt output
sed -i -e '1,2d' -e 's/"//g' output/Class_{8,9}.txt

Note that removing the double quotes would mean writing invalid CSV output if any fields contain embedded commas or newlines. To delete only the unneeded double quotes, use CSV parser such as csvformat from csvkit.

The above commands all assume that output is an existing directory that you are allowed to create files in.

Answer 2

To write to different files from the same sed command, you can either use the w command or the w flag of the s command.

In your case, it seems that the last field of a row is the class field that determines which file it should be written to, so we don't need to care which is the source file at all. We don't even need to care for the first two rows, because they can be ignored, because they don't have a valid class:

sed -ne 's/"//g;/,8$/woutput/Class_8.txt' -e '/,9$/woutput/Class_9.txt' file*

So for all lines you remove the double quotes with s/"//g, then /,8$/ selects all lines with 8 in the last field, and those should be written to output/Class_8.txt. Same for class 9. Default output is suppressed by the -n option.

Of course, this does not scale for a large number of files.

Answer 3

-i.suf in sed adds .suf to the old files and writes changes to the new ones. You could thus 1) use sed to modify the files and keep the old ones with a suffix, 2) move the new files, and 3) rename the old ones to the original name by removing the suffix:

sed -i.bup '1,2d;s/"//g' file1.txt file2.txt file3.txt
mv file{1..3}.txt output
rename 's/.bup$//' file{1..3}.txt.bup

(rename is sometimes also called prename or perl-rename depending on distribution)

Answer 4

Assuming you would like to edit both files and redirect the output to a output directory with the same file names but an edited version, then this awk may work.

$ awk 'FNR > 1 { gsub(/\"/,""); print > "output/"FILENAME ; next } FNR > 1 { gsub(/\"/,""); print > "output/"FILENAME }' Class_8.txt Class_9.txt

Or as a script

$ cat awk.script

FNR > 1 {                      #Remove line 1 from first file
    gsub(/\"/,"")              #Substitute all double quotes for nothing
    print > "output/"FILENAME  #Send the output to new directory with the same filename
    next                       #Move on to the next file
} FNR > 1 {                    #Remove line 1 from second file
    gsub(/\"/,"")              #Substitute all double quotes for nothing
    print > "output/"FILENAME  #Send the output to new directory with the same filename
}

To use the script with your files, you can run the command as

$ awk -f awk.script Class_8.txt Class_9.txt

The output will be as follows

$ cat output/Class_8.txt
Total Students:,247
John,14,8
Sara,13,8

$ cat output/Class_9.txt
Total Students:,119
John,15,9
Sara,16,9