Output file location and contents based on list of files

Question

I have a file that contains locations of different files called locationfile.txt it contains:

/home/user/documents/file1
/home/user/file2
/home/user/file3

Each text files have about 10 lines and I want to output all files to another file called finalfile.

I figured out that if I do

cat locationfile.txt | while read line; do cat "$line"; done

This prints all files together so it becomes a ~30 line text file.

What my question is, how can I spit that file out put include the file locations inbetween the text like this:

/home/user/documents/file1
text
text
text
""
/home/user/file2
text
text
text
""
/home/user/file3
text
text
text
""

Kusalananda · Accepted Answer · 2017-01-18 17:19:11Z

Before the cat do an echo or printf of the filename:

while read line; do printf '%s\n' "$line"; cat "$line"; done <locationfile.txt >finalfile

Or, more readable:

while read line; do
  printf '%s\n' "$line"
  cat "$line"
done <locationfile.txt >finalfile

Note that this requires all paths in locationfile.txt to have no \ in them.

Although having \ in path names is highly unusual, it would be safer to do

while read -r line; do
  printf '%s\n' "$line"
  cat "$line"
done <locationfile.txt >finalfile

... which doesn't have the same restrictions.

Adding a check to make sure the file actually exist too, and outputting a warning if it doesn't:

while read -r line; do
  if [[ -f "$line" ]]; then
    printf '%s\n' "$line"
    cat "$line"
  else
    printf 'Warning: "%s" does not exist\n' "$line" >&2
  fi
done <locationfile.txt >finalfile

"Does not exist" here means "is at least not a regular file".

If you want the "" there as well, as a sort of end-of-file-contents-marker, just put that out after cat:

while read -r line; do
  if [[ -f "$line" ]]; then
    printf '%s\n' "$line"
    cat "$line"
    echo '""'
  else
    printf 'Warning: "%s" does not exist\n' "$line" >&2
  fi
done <locationfile.txt >finalfile

And finally, give things self-documenting names:

while read -r file_path; do
  if [[ -f "$file_path" ]]; then
    printf '%s\n' "$file_path"
    cat "$file_path"
    echo '""'
  else
    printf 'Warning: "%s" does not exist\n' "$file_path" >&2
  fi
done <file_paths.txt >files_concat_with_paths.txt

while read line works with spaces in file names. And since here is no splitting of the inputl, you don't need to set IFS. Or am I missing something? — xhienne, Jan 18 '17 at 16:55
@xhienne You're not missing anything, that is correct. Setting IFS to be empty also disables the removal of initial and trailing whitespace in the input, which I usually want to retain, but which may cause problems here. I will amend the answer. — Kusalananda♦, Jan 18 '17 at 17:17
@xhienne, pretty sure multiple consecutive spaces will still be mishandled, but perhaps that's only on initial and trailing whitespace. — Wildcard, Jan 19 '17 at 4:28
@Wildcard You are right, both of you, leading and trailing spaces are not stored in line. But multiple consecutive inner spaces are. And if you don't specify line, the whole line goes into $REPLY, including leading and trailing spaces. — xhienne, Jan 19 '17 at 13:16

DopeGhoti · Accepted Answer · 2017-01-18 16:24:52Z

3

while read file; do
    ( echo "$file"; cat "$file"; echo '"""' ) >> /path/to/outputfile
done < /path/to/filelist

edited Jan 18 '17 at 16:24

answered Jan 18 '17 at 16:01

DopeGhoti

59.2k6 gold badges79 silver badges119 bronze badges

add a comment |

steeldriver · Accepted Answer · 2017-01-18 16:56:35Z

Here's an option if you have GNU awk, that avoids shell loops altogether:

xargs -d '\n' -a locationfile.txt gawk 'BEGINFILE{print FILENAME} 1' > newfile

or with GNU sed:

xargs -d '\n' -a locationfile.txt sed -s '1F' > newfile

If you don't care about the exact format of the filenames, then you can use this trick with head:

xargs -d '\n' -a locationfile.txt head -vn -0 > newfile

(the -n -0 tells head to output all lines except the first 0 i.e. all lines).

terdon · Accepted Answer · 2017-01-18 18:03:08Z

Another Perl approach:

$ perl -le 'do{print $_,`cat $_`,"\"\"";} for <>' locationfile.txt
text
text
text
""
text
text
text
""
text
text
text
""

That's using a bit of Perl magic and might need an explanation. The -l removes trailing newlines from each input line and adds a newline to each print call. The -e is how you give a script to perl.

Now, <> is "the contents of the input file split into an array". So, do {foo} for <> will run foo on each line of the input file. Inside that for loop, the special variable $_ is assigned to each input line. The `command` syntax will run command in the default shell (a system call) and return its output. So, cat $_ will cat each filename in locationfile.txt.

Altogether, print $_,`cat $_`,"\"\"" will print the file name (currently in $_), the result of the cat (the file's contents) and then "" (but we need to escape those, so \"\").

In other words, that script means "read each input line, print it, cat the file it contains and print a "" after each file.

pbm · Accepted Answer · 2017-01-18 16:00:49Z

2

You can just add echo "$line" to your loop:

cat locationfile.txt | while read line; do echo "$line" && cat "$line"; done

answered Jan 18 '17 at 16:00

pbm

21.7k6 gold badges33 silver badges50 bronze badges

You could also add && echo '""' after cat to match user's desired output :) – Sergiy Kolodyazhnyy Jan 18 '17 at 18:09
This did exactly what I wanted!!! Now I will add in a check file as @Kusalananda said – user211145 Jan 18 '17 at 18:38

add a comment |

Sergiy Kolodyazhnyy · Accepted Answer · 2017-01-18 17:54:23Z

AWK approach via system() function. We make up the command string via sprintf and pass it on to system()

$ awk '{print $0; cmd=sprintf("cat \"%s\"",$0);system(cmd); print "\"\""}' filenames.txt                                 
/home/xieerqi/input1.txt
Hello, I am input file #1
And this is second line of the text
""
/home/xieerqi/input2.txt
And I am input file #2
Roses are red, violets are blue
AWK is awesome, Python is too
""
/etc/resolv.conf
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 127.0.1.1
""

And Perl approach , but with opening file within the code:

$ perl -ne 'chomp; open($fh,$_) or die;print "$_\n";while($line = <$fh>){ print $line; print "\"\"\n" if eof}' filenames>
/home/xieerqi/input1.txt
Hello, I am input file #1
And this is second line of the text
""
/home/xieerqi/input2.txt
And I am input file #2
Roses are red, violets are blue
AWK is awesome, Python is too
""
/etc/resolv.conf
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 127.0.1.1
""

Stack Exchange Network

current community

your communities

more stack exchange communities

Output file location and contents based on list of files

6 Answers 6

Your Answer

Not the answer you're looking for? Browse other questions tagged linux awk sed cat or ask your own question.

Hot Network Questions

Output file location and contents based on list of files

6 Answers 6

Your Answer

Sign up or log in

Post as a guest

Not the answer you're looking for? Browse other questions tagged linux awk sed cat or ask your own question.

Related

Hot Network Questions