I want to append data of approx 500 files into one file and each of these files are located in their respective dirs.

Also How i can verify the contents of merged file e.g How i verify that data merged have data from all files or does not miss any data from all these file or each file got processed well.


You can use cat with the >> operator. Depending upon the directory structure and files location, you can use ls or find to find the files and in a shell for loop iterate over those files while appending to a target file. For example:

for i in $(find /my/path -iname "*.txt")
    cat $i >> result.txt
    echo "====$i====" >> result.txt

In order to verify that you have data from all files, you may insert a temporary pattern at the end of each append to the result.txt file. Later you can grep the pattern and verify if all the files were indeed appended. As shown in the above loop, the file result.txt should now contain the line ====filename==== for each of the file found. Later you can remove those lines using sed as follows:

sed /'^===='/d result.txt > tmp && mv tmp result.txt

Make sure the pattern you chose is unique otherwise other lines might get deleted with the above sed command.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.