How to get the results of ls piped to wc when running inside of a for loop?

Question

So I am trying to make this a single command instead of having to dump the data to a file and then cat the file and pipe the output to wc -l. The whole line of code looks like this:

for file in `grep Rawdata freeze2-1.out | awk '{print $6}'`
 do
  ls -1 /appdata/frozen_files/$file | wc -l
 done

But the output gives me a count for every line instead of a total for the entire output. Is there anyway this can be done where I get the output of a single number for all of the files or do I have to redirect to a file and make another step?

Answer 1

You are trying to count the total number of files in a directory named by field 6 in the freeze2-1.out file, but only where "Rawdata" appears somewhere on that line of that file.

The main error in your initial post was simply one of logic or placement; you were asking wc -l to count the individual lines inside the loop inside of asking it for the total number of lines that the overall loop provided. That's what most of the existing answers are correcting.

I made up a sample freeze2-1.out file so that I could "play along at home"; here's what I put in it:

Rawdata 2 3 4 5 file1
Rawdata 2 3 4 5 file2
Rawdata 2 3 4 5 file3

I also populated a relative directory named "appdata/frozen_files" with directory names based on field 6 above. Note that your actual path is an absolute path starting with /appdata/frozen_files. I've placed 1 file in file1, 2 files in file2, and 3 files in file3, for a total of 6 files.

The first improvement that I'll suggest is to combine the grep and the awk, since awk can do pattern matching. RomanPerekhrest did this in their answer as well:

awk '/Rawdata/ { print $6 }' freeze2-1.out

The sample output is then:

file1
file2
file3

I'll point out here that since awk splits fields based on spaces, tabs, and newlines, printing specifically field 6 implies that each line of output will have a word with no spaces in it.

There are a few ways to count the number of files in a directory. Using ls -1 | wc -l is a way, but it fails in a very particular case where a filename has an embedded newline; you could create such a file manually with touch $'file\nname'. Since awk is reading lines of input delimited by newlines, we won't hit that special case here, but I wanted to mention it. For example:

# create the file
$ touch $'file\nname'

# check the listing; looks okay so far
$ ls -1
appdata
file?name
freeze2-1.out

# count them up
$ ls -1 | wc -l
4 

# woops! should be 3!

For that reason, I'll demonstrate two other methods to count the files in a directory.

The first method uses the set builtin to take its arguments and convert them into positional arguments named by $1, $2, $3, etc.

$ sum=0
$ for file in $(awk '/Rawdata/ { print $6 }' freeze2-1.out)
  do
    set -- "appdata/frozen_files/${file}/"*
    sum=$((sum + $#))
  done
$ echo "$sum"
6

This initializes a counter sum to zero, then loops through the filenames from the awk output. It calls set with the quoted path to the directory, followed by the * shell glob, which expands to all of the (non-hidden) files in that directory. That matches the ls -1 behavior of excluding hidden files (those starting with a period). The number of parameters is given by the special variable $#, which we add within the loop to sum, then print it at the end.

A second method, for shells that support arrays, is to put the globbed filenames into an array, then count the elements of the array:

$ sum=0
$ for file in $(awk '/Rawdata/ { print $6 }' freeze2-1.out )
  do
    n=("appdata/frozen_files/${file}"/*)
    sum=$((sum + ${#n[@]}))
  done
$ echo "$sum"
6

Answer 2

Move the word count to the end;

for file in $(grep Rawdata freeze2-1.out | awk '{print $6}')
 do
  ls -1 /appdata/frozen_files/$file
 done | wc -l

or sum it;

C=0
for file in $(grep Rawdata freeze2-1.out | awk '{print $6}')
 do
  C=$[$C+$(ls -1 /appdata/frozen_files/$file | wc -l)]
 done
echo $C;

or maybe use find if your file structure can match a pattern (or regex)

find /appdata/frozen_files/ -iname '*.raw' |wc -l