0

I am counting files between certain numbers by the following:

for i in $(eval echo {$start..$stop}); do
ls /home/me/*/file_$i.txt 2> /dev/null | wc -l
done

$(eval echo {$start..$stop}) is necessary as sequence expressions cannot take general input.

However, this does not do what I am looking for, as for each iteration of the loop, it counts if that individual file exists or not. Instead, I want it to iterate over all the files, but count the total number that exists between $start and $stop (keep in mind I am dealing with a large number of files and trying to avoid overflow). How could I modify my above scheme to count the total number of existing files, instead of checking whether each iterated file exists?

  • My previous question could not handle general values $start and $stop. Thus, the loop was necessary. @msp9011 – theta Sep 24 '19 at 12:42
  • I get the error "/usr/bin/ls: Argument list too long" if I place $(eval echo {$start..$stop}) directly in the file string, making the loop necessary. – theta Sep 24 '19 at 12:48
  • Add the output of wc -l to a variable which you initialize with 0. – Panki Sep 24 '19 at 12:50
  • Are $start and $stop integers or letters? – Kusalananda Sep 24 '19 at 14:36
  • You could simply move the | wc -l to after done, unless you also want to support filenames with embedded newlines (which your current code does not do). – Kusalananda Sep 24 '19 at 15:27
1

Try this,

Count=0
for i in {$start..$stop}; do
   Counter="$(ls /home/me/*/file_$i.txt 2> /dev/null | wc -l)"
   Count=$((Count+Counter))
done
echo "$Count"
| improve this answer | |
  • 2
    Why was this answer accepted? As OP already pointed out in their question {$start..$stop} does not expand to (for instance) 7 8 9 10 11 but to literally {7..11}. – Socowi Sep 24 '19 at 13:27
  • @Socowi, note that it's a limitation of bash only, The {x..y} operator comes from zsh and {$start..$stop} works fine there. It also works fine in ksh93 or yash. – Stéphane Chazelas Sep 25 '19 at 6:33
  • That's good to know, but doesn't make this answer correct. The question is tagged as [bash] and not as [zsh]. – Socowi Sep 25 '19 at 9:42
1

Don't use a loop at all. Instead, list all files and use a command like grep or akw to filter out the files between start and stop. printf won't have the problem Argument list too long as it is a bash built-in.

printf '%s\0' /home/me/*/file_*.txt |
grep -cEzf <(seq "$start" "$stop" | sed 's/.*/_&\.txt$/')

Here we assume you have the GNU versions of grep which can handle null bytes. This is important to process paths safely. The null byte is the only character which cannot be part of a path so we can use it to separate paths.
If you don't have the GNU version you can create something that delimits paths by newlines under the assumption that no path contains a newline character. Replace \0 with \n and remove the -z flag.

If you change …_*.txt to something else you may also have to update the sed command.

| improve this answer | |
1
  1. Don't use eval unless you have no other alternative. It is potentially dangerous, depending on the contents of the $start and $stop variables. It is not a good idea to get into the habit of using it where it isn't necessary.

  2. Don't use the output of ls for anything but viewing in a terminal. Attempting to parse ls, even in the simplest fashion, can and will fail in all sorts of ways. Use anything that can produce a NUL-separated list of filenames instead: find -print0 is often a good choice.

  3. The bash built-in {x..y} sequence expansion can't evaluate variables, but the standalone seq program can take variables as arguments. It also has a useful -s option to specify the separator.

The following requires a version of grep that supports the -z option for NUL-separated input records (e.g. GNU, FreeBSD, and most other modern versions of grep).

# build a regular expression matching the desired sequence
re="$(seq -s '|' "$start" "$stop")"

# use the RE with `find ... -print0` and `grep -E -z -c`
find /home/me/*/ -maxdepth 1 -type f -name 'file_*.txt' -print0 | 
  grep -Ezc "/file_($re)\.txt$"

For example, if start=3 and stop=7, then $re will be 3|4|5|6|7. The grep command would then expand to:

grep -Ezc "/file_(3|4|5|6|7)\.txt$"

BTW, the -name 'file_*.txt' arguments for the find command aren't required. They can be dropped, and the pipeline will still run without error. All they do is reduce the input data that needs to be processed by grep. A very minor optimisation at best.

e.g.

find /home/me/*/ -maxdepth 1 -type f -print0 | grep -Ezc "/file_($re)\.txt$"
| improve this answer | |
0

A bash solution:

(
shopt -s nullglob
start=1   # change as needed
stop=500  # change as needed
printf '%s\n' $(printf '/home/me/*/file_%s.txt\n' $(seq "$start" "$stop")) | wc -l
)

Enable nullglob to match only existing files and execute the script in a subshell (…), so we don't have to determine nullglob's previous state and reset its old state when done. The inner printf is used to generate all possible filename patterns and the the outer printf is needed to interpret the * as glob character. The printed output lines are counted with wc -l.

| improve this answer | |
  • Instead of printf '%s\n' | wc -l which would not work properly if file paths contained newline characters, you could do printf '%.0sx' | wc -c. Or define a count() { echo "$#"; } function that returns the number of arguments. – Stéphane Chazelas Sep 25 '19 at 6:38
0

If I understand you correctly, there's no need for a loop:

Suppose a range between 12 and 76:

ls -l | grep -e "file_1[2-9].txt" -e "file_[2-6][0-9].txt" -e "file_7[0-6].txt"

Files 12 to 19

"file_1[2-9].txt"

Files 20 to 69:

"file_[2-6][0-9].txt"

Files 70 to 76

"file_7[0-6].txt"
| improve this answer | |
  • I guess this only works when $start and $stop are < 10. For numbers with multiple digits you have to generate a regex or use another tool like awk. – Socowi Sep 24 '19 at 13:37
  • @Socowi Yes, this does not work for numbers >10. I need to use another method, as you mention. – theta Sep 24 '19 at 13:41
  • Edited the answer, I'm not completely sure, but I think it does it. Feedback welcome. – schrodigerscatcuriosity Sep 24 '19 at 14:58
0

You can use process substitution and grep -f to search only for files within a range.

E.g. To count all files named file_[0-9]+.txt between range of 10 to 20:

start=10
end=20
ls -f1 | grep -F -f <(printf "file_%s.txt\n" $(seq ${start} ${end}))|wc -l
| improve this answer | |
0

With zsh:

count=0 range="<$start-$stop>"
/home/me/*/file_$~range.txt(Ne['((!++count))'])

Would give you the count without storing the list of files in memory.

  • <1-100> is a zsh glob operator that matches decimal numbers in that range (note that it matches on 2 but also 000002 for instance)
  • $~var is like $var except that the content of $var is considered as a pattern as opposed to a literal value.
  • (Ne['code']) is a glob qualifier
  • N: enables nullglob for that pattern so it doesn't fail if the glob doesn't match any file (which is always the case here).
  • e['code'] selects files based on the evaluation of the code. Here the code (((!++count))) increments $count and always returns false so the files are not selected (but $count has been incremented which is all we care about here).
| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.