Concatenating txt files and keeping directory name

Question

I am concatenating thousands of .txt files together but I want to keep the path written above the text of each .txt file so I can refer back to the directory to review files if needed.

Example:

file1.txt has the following content:

id: 1234
info: 1234
id: abcd
info: abcd

The path to file1.txt is pathto/dir1/sub1/file1.txt

file2.txt has the following content:

id: 5678
info: 5678

The path to file2.txt is pathto/dir2/sub2/file2.txt

The .txt file after concatenating should be formatted like:

pathto/dir1/sub1/file1.txt

id: 1234
info: 1234
id: abcd
info: abcd

pathto/dir2/sub2/file2.txt

id: 5678
info: 5678

Concatenating alone does not print the path in between each concatenated .txt file which is what I am trying to accomplish.

I will also cd ~/ into pathto/ and would need to run a command from there that will recursively scan all subdirectories looking for "file.txt" files

Answer 1

This sounds like your shell can do it out of the box:

#!/bin/bash
shopt -s globstar nocaseglob nullglob

for file in **/file.txt **/files.txt; do
  printf '---------------------\n' >> allfiles.txt
  printf '%s' "${file}" >> allfiles.txt
  printf '---------------------\n' >> allfiles.txt
  cat "${file}" >> allfiles.txt
done

Answer 2

Loop across the to level of directories, loop across each set of subdirectories, loop across the files. Something like this

for topdir in */
do
    top=${topdir%/}
    top=${top##*/}

    for subdir in "$topdir"/*/
    do
        sub=${subdir%/}
        sub=${sub##*/}

        printf -- '-%s\n--%s\n\n' "$top" "$sub"

        for file in "$subdir"/*
        do
            [ -f "$file" ] && cat <"$file"
        done
    done
done >output_file

I've used an explicit loop across files to ensure I'm only trying to concatenate files. You could dispense with that if you know for sure there are only files in each subdirectory.

---

In response to your amended question I can offer this significantly shorter solution. This searches from the current directory for files called file.txt and concatenates them with their pathnames to the output_file.

find . -type f -name 'file.txt' -exec sh -c 'for f in "$@"; do printf "%s\n\n" "${f#./}"; cat "$f"; echo; done' _ {} + >output_file

Answer 3

By your question and comments it would seem you are entering the folder and then doing something like this:

cat filename* > concatenatedfile.txt

Basically just running cat on all files that start with filename.

You can use pwd command first, and then run cat to just put path to the top of the file, and then append concatenation to the file.

pwd > concatenatedfile.txt
cat filename* >> concatenatedfile

This way you will create concatenatedfile.txt that has the path of the folder you are in, and then with >> you just append the contents of all the files that start with filename to the same file.

> overwrites the content of the file

>> appends to the end of the file, what is at top of the file stays unchanged

Answer 4

Here is what worked in the end:

find . -type f -iname 'file.txt' -printf '\n%P\n'  -exec cat {} \; > catfile.txt

Finds all files titled file.txt regardless of case and prints the data to an output file called catfile.txt.

Thanks everyone for the help!