You are trying to count the total number of files in a directory named by field 6 in the freeze2-1.out file, but only where "Rawdata" appears somewhere on that line of that file.
The main error in your initial post was simply one of logic or placement; you were asking wc -l
to count the individual lines inside the loop inside of asking it for the total number of lines that the overall loop provided. That's what most of the existing answers are correcting.
I made up a sample freeze2-1.out file so that I could "play along at home"; here's what I put in it:
Rawdata 2 3 4 5 file1
Rawdata 2 3 4 5 file2
Rawdata 2 3 4 5 file3
I also populated a relative directory named "appdata/frozen_files" with directory names based on field 6 above. Note that your actual path is an absolute path starting with /appdata/frozen_files. I've placed 1 file in file1, 2 files in file2, and 3 files in file3, for a total of 6 files.
The first improvement that I'll suggest is to combine the grep
and the awk
, since awk can do pattern matching. RomanPerekhrest did this in their answer as well:
awk '/Rawdata/ { print $6 }' freeze2-1.out
The sample output is then:
file1
file2
file3
I'll point out here that since awk splits fields based on spaces, tabs, and newlines, printing specifically field 6 implies that each line of output will have a word with no spaces in it.
There are a few ways to count the number of files in a directory. Using ls -1 | wc -l
is a way, but it fails in a very particular case where a filename has an embedded newline; you could create such a file manually with touch $'file\nname'
. Since awk is reading lines of input delimited by newlines, we won't hit that special case here, but I wanted to mention it. For example:
# create the file
$ touch $'file\nname'
# check the listing; looks okay so far
$ ls -1
appdata
file?name
freeze2-1.out
# count them up
$ ls -1 | wc -l
4
# woops! should be 3!
For that reason, I'll demonstrate two other methods to count the files in a directory.
The first method uses the set
builtin to take its arguments and convert them into positional arguments named by $1, $2, $3, etc.
$ sum=0
$ for file in $(awk '/Rawdata/ { print $6 }' freeze2-1.out)
do
set -- "appdata/frozen_files/${file}/"*
sum=$((sum + $#))
done
$ echo "$sum"
6
This initializes a counter sum
to zero, then loops through the filenames from the awk output. It calls set
with the quoted path to the directory, followed by the *
shell glob, which expands to all of the (non-hidden) files in that directory. That matches the ls -1
behavior of excluding hidden files (those starting with a period). The number of parameters is given by the special variable $#
, which we add within the loop to sum
, then print it at the end.
A second method, for shells that support arrays, is to put the globbed filenames into an array, then count the elements of the array:
$ sum=0
$ for file in $(awk '/Rawdata/ { print $6 }' freeze2-1.out )
do
n=("appdata/frozen_files/${file}"/*)
sum=$((sum + ${#n[@]}))
done
$ echo "$sum"
6