2

I would like to process a multiline string and iterate it line by line, in a POSIX shell (/bin/sh) on a BSD platform. Bash is not included in the base BSD-distribution and has a GPL license - so I am trying to make it universally work with /bin/sh instead.

I found a solution using a pipe, however in the regular /bin/sh shell, these a processed in a separate process, meaning the following does not work:

MULTILINE="`cat ${SOMEFILE}`"
SOMEVAR="original value"

echo "${MULTILINE}" | while IFS= read -r SINGLELINE
do
 SOMEVAR="updated value"
 echo "this is a single line: ${SINGLELINE}"
 echo "SOMEVAR is now: ${SOMEVAR}"
done

echo "Final SOMEVAR is unchanged: ${SOMEVAR}"

In the above example, it accomplishes what I want, except for the fact that changes to variables such as ${SOMEVAR} are not accessible outside the while loop.

My question: how can I accomplish something like the above without this restriction? Note that many solutions require Bash, whereas I am using the standard POSIX-shell /bin/sh.

  • Why can you not use bash? (add to your question) – ctrl-alt-delor Mar 31 at 10:54
  • I can not reproduce your issue using /bin/sh on a BSD system. There is nothing in this code that would require bash. – Kusalananda Mar 31 at 10:57
  • My apologies. In trying to create a minimized sample of the actual issue, i made some errors. I updated the code sample now, and it should reflect the problem correctly now. The example indeed does not require Bash. – Steiner Mar 31 at 11:09
  • @Steiner In fact, it would have the same issue in bash. – Kusalananda Mar 31 at 11:21
  • 3
    There is no Bourne shell on the BSDs. Nor is the Bourne shell standard. Do not conflate the Bourne shell with a POSIX-conformant sh. – JdeBP Mar 31 at 11:29
5

You could use a here document:

while IFS= read -r SINGLELINE
do
  SOMEVAR="updated value"
  printf '%s\n' "this is a single line: ${SINGLELINE}"
  printf '%s\n' "SOMEVAR is now: ${SOMEVAR}"
done << EOF
$MULTILINE
EOF
printf '%s\n' "Final SOMEVAL is still $SOMEVAR"

Depending on the sh implementation, here-documents are implemented either as a deleted temporary file where the shell has stored the expansion of the variable followed by newline beforehand, or a pipe to which the shell feeds the expansion of the variable followed by newline. But in either case, except in the original Bourne shell (a shell that is no longer in use these days and is not a POSIX compliant shell), the command being redirected is not run in a subshell (as POSIX requires).

or you could use split+glob:

IFS='
' # split on newline only
set -o noglob
for SINGLELINE in $MULTILINE
do
  SOMEVAR="updated value"
  printf '%s\n' "this is a single line: ${SINGLELINE}"
  printf '%s\n' "SOMEVAR is now: ${SOMEVAR}"
done
printf '%s\n' "Final SOMEVAL is still $SOMEVAR"

But beware it skips empty lines.

  • Both solutions worked well for me with /bin/sh. I think this solution is better than the one Kusalananda provided, though his answer is very informative too. I particularly like the for-loop solution since it probably is faster because it does not use read which is very slow. Thanks! – Steiner Mar 31 at 15:54
  • @Steiner (Just had to quickly double check so that I didn't have a for-loop solution, which I fortunately did not have) – Kusalananda Mar 31 at 18:41
3

You would read directly from the file without the pipeline. This avoids running the while loop in a subshell, which allows you to see the changed value of $SOMEVALUE after the loop.

SOMEVAR="original value"

while IFS= read -r SINGLELINE
do
    SOMEVAR="updated value"
    printf 'this is a single line: %s\n' "$SINGLELINE"
    printf 'SOMEVAR is now: %s\n' "$SOMEVAR"
done <"$SOMEFILE"

printf 'Final SOMEVAR is: %s\n' "$SOMEVAR"

If you insist on having your $MULTILINE variable, then write that to a file and read it from there:

tmpfile=$(mktemp)
printf '%s\n' "$MULTILINE" >"$tmpfile"

while ...; do
   ...
done <"$tmpfile"
rm "$tmpfile"

Also related:

An answer to the above linked question also suggests writing your program in such a way that all uses of $SOMEVAR occurs within the subshell at the end of the pipeline:

MULTILINE=$(cat "$SOMEFILE")
SOMEVAR="original value"

printf '%s\n' "$MULTILINE" | {
    while IFS= read -r SINGLELINE
    do
        SOMEVAR="updated value"
        printf 'this is a single line: %s\n' "$SINGLELINE"
        printf 'SOMEVAR is now: %s\n' "$SOMEVAR"
    done

    printf 'Final SOMEVAR is: %s\n' "$SOMEVAR"
}

Also possibly related:

Other questions that may be of interest:

  • Indeed this works, but requires a file to be fed to the while-loop. My actual code is more complex, and requires a modified multiline string to be fed to the while-loop. Your solution to write this multiline string to a temporary file works, but is there a solution that does not require writing a temporary file? – Steiner Mar 31 at 11:28
  • @Steiner See update answer. There's not much else you could do without using features of specific shells. – Kusalananda Mar 31 at 11:33
  • Thank you that answers my question. The suggestion at the end and the links provided are very helpful too! – Steiner Mar 31 at 11:43
  • @Steiner ... except for using a here-document. Well, there's a thing I didn't think of :-) – Kusalananda Mar 31 at 18:42
0

It works for me :

$ cat bin/test
#! /bin/sh
SOMEFILE=$1
MULTILINE="`cat ${SOMEFILE}`"
SOMEVAR="blah"

echo "${MULTILINE}" | while IFS= read -r SINGLELINE
do
 echo "this is a single line: ${SINGLELINE}"
 echo "but accessing this var fails: ${SOMEVAR}"
done

and

$ bin/test bin/test
this is a single line: #! /bin/sh
but accessing this var fails: blah
this is a single line: SOMEFILE=$1
but accessing this var fails: blah
this is a single line: MULTILINE="`cat ${SOMEFILE}`"
but accessing this var fails: blah
this is a single line: SOMEVAR="blah"
but accessing this var fails: blah
this is a single line:
but accessing this var fails: blah
this is a single line: echo "${MULTILINE}" | while IFS= read -r SINGLELINE
but accessing this var fails: blah
this is a single line: do
but accessing this var fails: blah
this is a single line:  echo "this is a single line: ${SINGLELINE}"
but accessing this var fails: blah
this is a single line:  echo "but accessing this var fails: ${SOMEVAR}"
but accessing this var fails: blah
this is a single line: done
but accessing this var fails: blah
  • My apologies. In trying to create a minimized sample of the actual issue, i made some errors. I updated the code sample now, and it should reflect the problem correctly now. – Steiner Mar 31 at 11:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.