generate dyanmic filename and perform some operations on it and write to some other directory

Question

I have a testing folder with file name as shown below:

100080_Accounting  Statistic-42005_04May2020_0000-04May2020_0100.csv  
110001_ABCD Traffic_04May2020_header_only.csv


cat '110001_ABCD Traffic_04May2020_header_only.csv'
ID,NAME,LOCATION
1,Vikrant,Gurgaon
2,Bharat,Noida
3,Raju,Hyderabad

cat '100080_Accounting Statistic-42005_04May2020_0000-04May2020_0100.csv'
ID,NAME,AGE,DEPARTMENT
4,Raju,22,Admin
5,Rajeev,23,Admin

I am trying to extract a filename and appending that filename to each record of file while reading that particular file from testing directory. I have added a new column name as "GroupName" to header before actual records being written to the newly created file and wish to write those file to some other directory.

Below is my shell script where logic is working fine as expected but I am not able to figure out the way to write it some other directory with the same file name.. also I am getting issue while adding column to header of second file.

#!/bin/bash

# Go to where the files are located
filedir=/home/vikrant_singh_rana/testing/*

#reading file from directory
for filename in $filedir; do
        #extracting modified file name from file
        b=$(basename "$filename" ".csv" | awk -F "_" '{print $2}' | awk -F "-" '{print $1}')
        #reading line by line from file
        first='yes'
        while read -r line;do
                #reading header from file
                header=$(head -n1 "$filename")
                if [[ $first == 'yes' ]]; then
                        #writing header with new column just once
                printf '%s,%s\n' "GroupName" "$header"
                first='no'
                else
                        #writing modified filename to each record of a file
                printf '%s,%s\n' "$b" "$line"
                fi
        done <"$filename"
        #echo $filename
        output_filename=$(basename "$filename")
        #echo $output_filename

#done
done > /home/vikrant_singh_rana/enrichment_files/enrichment_file.txt

I am getting an output as below:

cat /home/vikrant_singh_rana/enrichment_files/enrichment_file.txt

GroupName,ID,NAME,AGE,DEPARTMENT
Accounting Statistic,4,Raju,22,Admin
Accounting Statistic,5,Rajeev,23,Admin
GroupName,ID,NAME,LOCATION
ABCD Traffic,1,Vikrant,Gurgaon
ABCD Traffic,2,Bharat,Noida
ABCD Traffic,3,Raju,Hyderabad

what I was expecting like as shown below and file name has be same which I am reading and output should written to separate file.

file name as :110001_ABCD Traffic_04May2020_header_only.csv

    GroupName,ID,NAME,AGE,DEPARTMENT
    Accounting Statistic,4,Raju,22,Admin
    Accounting Statistic,5,Rajeev,23,Admin

file name as :100080_Accounting  Statistic-42005_04May2020_0000-04May2020_0100.csv 
    GroupName,ID,NAME,LOCATION
    ABCD Traffic,1,Vikrant,Gurgaon
    ABCD Traffic,2,Bharat,Noida
    ABCD Traffic,3,Raju,Hyderabad

Answer 1

I was able to write output to a file and have made changes as below: Correct me incase it required any changes.

#!/bin/bash

# Go to where the files are located
filedir=/home/vikrant_singh_rana/testing/*
#reading file from directory
for filename in $filedir; do
        #extracting modified file name from file
        b=$(basename "$filename" ".csv" | awk -F "_" '{print $2}' | awk -F "-" '{print $1}')
        #reading line by line from file
        first='yes'
        output_filename=$(basename "$filename")
        while read -r line;do
                #reading header from file
                header=$(head -n1 "$filename")
                if [[ $first == 'yes' ]]; then
                        #writing header with new column just once
                printf '%s,%s\n' "CounterGroupName" "$header" > /home/vikrant_singh_rana/enrichment_files/"$output_filename"
                first='no'
                else
                        #writing modified filename to each record of a file
                printf '%s,%s\n' "$b" "$line" >> /home/vikrant_singh_rana/enrichment_files/"$output_filename"
                fi
        done <"$filename"
done