(For simplicity I'll assume the file to read is the first argument - $1.)

I can do what I want externally with:

awk '/^#/ {next}; NF == 0 {next}; {print}' "$1" > $tempfile
while read var1 var2 var3 var4 < $tempfile; do
  # stuff with var1, etc.

However, it seems absurd to need to call awk every time I parse the config file. Is there a way to make read ignore commented or whitespace-only lines in a file, without external binaries/potential performance issues?

Answers so far are quite helpful! To clarify, I don't want to use a temp file, but I do want to read the config from a file, not from standard in. I'm well aware that I can use an input redirection when I call the script, but for various reasons that won't work in my circumstance.

I want to softcode the input to read from, e.g.:

[ $# -gt 0 ] && [ -r "$1" ] && configfile="$1"

while read var1 var2 var3 var4 < "$configfile" ; do

But when I try this, it just reads the first line of configfile over and over until I kill the process.

Maybe this should be its own question...but it's probably a single line change from what I'm doing. Where's my error?

  • 1
    The error is that the script redirects to the while command list instead of redirecting into the body of the loop (see my answer below). I think your default myconfigfile.txt code will work, see my answer for a more concise (and maybe less obvious) solution to overriding the default via a reference to $1, however my answer does not check the file for readability. – RobertL Nov 20 '15 at 21:02
  • 1
    There are advantages for not silently reverting to the default if the filename in $1 is unreadable. The script would behave as if $1 was processed, while actually using the default. Depending on the context, you might want to use the $1 value and let the shell print the error when the file cannot be opened for the redirection. – RobertL Nov 20 '15 at 21:09

You don't need a tempfile to do this, and sed (or awk) are far more flexible in comment processing than a shell case statement.

For example:

[ $# -gt 0 ] && [ -r "$1" ] && configfile="$1"

sed -e 's/[[:space:]]*#.*// ; /^[[:space:]]*$/d' "$configfile" |
    while read var1 var2 var3 var4; do
      # stuff with var1, etc.

This strips comments (with or without leading whitespace) and deletes empty lines from the input before piping it into the while loop. It handles comments on lines by themselves and comments appended to the end of the line:

# full-line comment
# var1 var2 var3 var4
abc 123 xyz def # comment here

Calling sed or awk for tasks like this isn't "absurd", it's perfectly normal. That's what these tools are for. As for performance, I'd bet that in anything but very tiny input files, the sed version would be much faster. Piping to sed has some startup overhead but runs very fast, while shell is slow.

  • The script that handles the config file is going to be run automatically every few minutes. It seemed strange to me to be re-stripping out comments on the same file every few minutes. But with the pipe set up this way, it does handle the other part of my question also, so perhaps this way is best. Thanks. – Wildcard Nov 20 '15 at 20:55
  • 1
    any other program that reads in a config file is going to be processing comments on every run too. if it bothers you, then strip the comments and redirect to, say, "$configfile.nocomments" only if "$configfile" is newer than "$configfile.nocomments", and use "$configfile.nocomments" as input to the while loop. – cas Nov 20 '15 at 21:05

This works because read breaks everything on whitespace (IFS), so if var1 is empty or starts with '#', then skip it.

while read var1 var2 var3 var4; do
   case $var1 in
       ''|\#*) continue ;;         # skip blank lines and lines starting with #
   echo "var1: '$var1'"
   # stuff with var1, etc.
done < "${1:-default_config_file}"

Then input has to be redirected into the loop instead of to the while command list. The "${1:-default_config_file}" expands to the first command line parameter if not empty, otherwise expands to default_config_file you can also use variable expansion, etc in the default value string.

Because you're interested in minimizing pre-processing, I think this is equivalent, but also removes all comments:

while read line; do
    echo "${line%%#*}" | {
        read var1 var2 var3 var4
        [ -z "$var1" ] && continue
        # stuff with var1, etc.
        for i in 1 2 3 4; do eval echo "\"var$i: \$var$i\""; done  #debug only!
done < "${1:-default_config_file}"

This uses the shell parameter expansion substring processing feature. ${line%%#*} expands to the original value of line except the first # and everything after it are removed. Load that into var1-4 and continue as usual. The test to continue shortens because we now only need to check for empty string, not #.

  • This is perfect. No external dependencies (no additional processes created), just as requested. One more part to the question, though—the config I want to read is contained in a specific file. I want to softcode that into the script, allowing for overrides. I've clarified this part of the question above. – Wildcard Nov 20 '15 at 20:44
  • 1
    ${1:-default} is useful but @wildcard wanted to test if "$1" was readable ([ -r "$1" ]) before overriding the default. – cas Nov 20 '15 at 21:00
  • 1
    @cas Thank you. That's what I get for updating based on comments before reading the updated question... After reflection, I commented on the question about a problem when silently reverting to default. – RobertL Nov 20 '15 at 21:13
  • 1
    i was going to change it to that myself until i re-read the question :) silently reverting is probably bad, but the script could warn or exit if "$1" isn't readable. – cas Nov 20 '15 at 21:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.