9

(For simplicity I'll assume the file to read is the first argument - $1.)

I can do what I want externally with:

tempfile=$(mktemp)
awk '/^#/ {next}; NF == 0 {next}; {print}' "$1" > $tempfile
while read var1 var2 var3 var4 < $tempfile; do
  # stuff with var1, etc.
done

However, it seems absurd to need to call awk every time I parse the config file. Is there a way to make read ignore commented or whitespace-only lines in a file, without external binaries/potential performance issues?


Answers so far are quite helpful! To clarify, I don't want to use a temp file, but I do want to read the config from a file, not from standard in. I'm well aware that I can use an input redirection when I call the script, but for various reasons that won't work in my circumstance.

I want to softcode the input to read from, e.g.:

configfile="/opt/myconfigfile.txt"
[ $# -gt 0 ] && [ -r "$1" ] && configfile="$1"

while read var1 var2 var3 var4 < "$configfile" ; do
  ...

But when I try this, it just reads the first line of configfile over and over until I kill the process.

Maybe this should be its own question...but it's probably a single line change from what I'm doing. Where's my error?

2
  • 1
    The error is that the script redirects to the while command list instead of redirecting into the body of the loop (see my answer below). I think your default myconfigfile.txt code will work, see my answer for a more concise (and maybe less obvious) solution to overriding the default via a reference to $1, however my answer does not check the file for readability.
    – RobertL
    Nov 20, 2015 at 21:02
  • 1
    There are advantages for not silently reverting to the default if the filename in $1 is unreadable. The script would behave as if $1 was processed, while actually using the default. Depending on the context, you might want to use the $1 value and let the shell print the error when the file cannot be opened for the redirection.
    – RobertL
    Nov 20, 2015 at 21:09

3 Answers 3

6

You don't need a tempfile to do this, and sed (or awk) are far more flexible in comment processing than a shell case statement.

For example:

configfile='/opt/myconfigfile.txt'
[ $# -gt 0 ] && [ -r "$1" ] && configfile="$1"

sed -e 's/[[:space:]]*#.*// ; /^[[:space:]]*$/d' "$configfile" |
    while read var1 var2 var3 var4; do
      # stuff with var1, etc.
    done

# Note: var1 etc are not available to the script at this
# point. They are only available in the sub-shell running
# the while loop, and go away when that sub-shell ends.

This strips comments (with or without leading whitespace) and deletes empty lines from the input before piping it into the while loop. It handles comments on lines by themselves and comments appended to the end of the line:

# full-line comment
# var1 var2 var3 var4
abc 123 xyz def # comment here

Calling sed or awk for tasks like this isn't "absurd", it's perfectly normal. That's what these tools are for. As for performance, I'd bet that in anything but very tiny input files, the sed version would be much faster. Piping to sed has some startup overhead but runs very fast, while shell is slow.


Update 2022-05-03:

Note that the variables (var1, var2, var3, etc) which are set in the while read loop will "go out of scope" when the while loop ends. The can only be used inside that while loop. The while loop is being run in a sub-shell because the config file is being piped into it. When that sub-shell dies, its environment goes with it - and a child process can not change the environment of its parent process.

If you want the variables to retain their values after the while loop, you need to avoid using a pipe. For example, use input redirection (<) and process substitution (<(...)):

while read var1 var2 var3 var4; do
  # stuff with var1, etc.
done < <(sed -e 's/[[:space:]]*#.*// ; /^[[:space:]]*$/d' "$configfile")

# remainder of script can use var1 etc if and as needed.

With this process substitution version, the while loop runs in the parent shell and the sed script is run as a child process (with its output redirected into the while loop). sed and its environment goes away when it finished, while the shell running the while loop retains the variables created/changed by the loop.

2
  • The script that handles the config file is going to be run automatically every few minutes. It seemed strange to me to be re-stripping out comments on the same file every few minutes. But with the pipe set up this way, it does handle the other part of my question also, so perhaps this way is best. Thanks.
    – Wildcard
    Nov 20, 2015 at 20:55
  • 1
    any other program that reads in a config file is going to be processing comments on every run too. if it bothers you, then strip the comments and redirect to, say, "$configfile.nocomments" only if "$configfile" is newer than "$configfile.nocomments", and use "$configfile.nocomments" as input to the while loop.
    – cas
    Nov 20, 2015 at 21:05
4

This works because read breaks everything on whitespace (IFS), so if var1 is empty or starts with '#', then skip it.

while read var1 var2 var3 var4; do
   case $var1 in
       ''|\#*) continue ;;         # skip blank lines and lines starting with #
   esac
   echo "var1: '$var1'"
   # stuff with var1, etc.
done < "${1:-default_config_file}"

Then input has to be redirected into the loop instead of to the while command list. The "${1:-default_config_file}" expands to the first command line parameter if not empty, otherwise expands to default_config_file you can also use variable expansion, etc in the default value string.

Because you're interested in minimizing pre-processing, I think this is equivalent, but also removes all comments:

while read line; do
    echo "${line%%#*}" | {
        read var1 var2 var3 var4
        [ -z "$var1" ] && continue
        # stuff with var1, etc.
        for i in 1 2 3 4; do eval echo "\"var$i: \$var$i\""; done  #debug only!
    }
done < "${1:-default_config_file}"

This uses the shell parameter expansion substring processing feature. ${line%%#*} expands to the original value of line except the first # and everything after it are removed. Load that into var1-4 and continue as usual. The test to continue shortens because we now only need to check for empty string, not #.

4
  • This is perfect. No external dependencies (no additional processes created), just as requested. One more part to the question, though—the config I want to read is contained in a specific file. I want to softcode that into the script, allowing for overrides. I've clarified this part of the question above.
    – Wildcard
    Nov 20, 2015 at 20:44
  • 1
    ${1:-default} is useful but @wildcard wanted to test if "$1" was readable ([ -r "$1" ]) before overriding the default.
    – cas
    Nov 20, 2015 at 21:00
  • 1
    @cas Thank you. That's what I get for updating based on comments before reading the updated question... After reflection, I commented on the question about a problem when silently reverting to default.
    – RobertL
    Nov 20, 2015 at 21:13
  • 1
    i was going to change it to that myself until i re-read the question :) silently reverting is probably bad, but the script could warn or exit if "$1" isn't readable.
    – cas
    Nov 20, 2015 at 21:22
2

You can do that without create temp file. The grep command will filter empty and commented lines.

while read var1 var2 var3; do
    echo $var1
    echo $var2
    echo $var3
    echo "etc..."
done < <(grep -v "^#\|^$" /opt/myconfigfile.txt)
1
  • Somehow I like this way most :-)
    – Jarek
    Jun 16, 2021 at 18:25

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .