Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have a text file with the following format:

keyword value
keyword value

Where keyword is a single word and value is everything else until the end of line. I want to read the file from a shell script, in a way that the values (but not the keywords) undergo shell expansion.

With sed it's easy to match the keywords and value parts

keyword value value
keyword "value  value"
keyword `uname`

echo "$input"|sed -e 's/^\([^[:space:]]*\)[[:space:]]\(.*\)$/k=<\1> v=<\2>/'

which produces

k=<keyword> v=<value value>
k=<keyword> v=<"value  value">
k=<keyword> v=<`uname`>

but then the question is how can I embed a shell command into the replacement part of the sed expression. In this case I would like the replacement to be \1 `echo \2`.

share|improve this question
Uhm... I am not SO sure to give it as an answer, but using DOUBLE quoted with sed should let you use shell $(command) or $variables inside the expression. – St0rM Dec 27 '13 at 15:24
up vote 11 down vote accepted

Standard sed can't call a shell (GNU sed has an extension to do it, if you only care about non-embedded Linux), so you'll have to do some of the processing outside sed. There are several solutions; all require careful quoting.

It's not clear exactly how you want the values to be expanded. For example, if a line is

foo hello; echo $(true)  3

which of the following should the output be?

k=<foo> value=<hello; echo   3>
k=<foo> value=<hello; echo   3>
k=<foo> value=<hello; echo 3>
k=<foo> value=<foo hello

I'll discuss several possibilities below.

pure shell

You can get the shell to read the input line by line and process it. This is the simplest solution, and also the fastest for short files. This is the closest thing to your requirement “echo \2”:

while read -r keyword value; do
  echo "k=<$keyword> v=<$(eval echo "$value")>"

read -r keyword value sets $keyword to the first whitespace-delimited word of the line, and $value to the rest of the line minus trailing whitespace.

If you want to expand variable references, but not execute commands outside command substitutions, put $value inside a here document. I suspect that this is what you were really looking for.

while read -r keyword value; do
  echo "k=<$keyword> v=<$(cat <<EOF

sed piped into a shell

You can transform the input into a shell script and evaluate that. Sed is up to the task, though it's not that easy. Going with your “echo \2” requirement (note that we need to escape single quotes in the keyword):

sed  -e 's/^ *//' -e 'h' \
     -e 's/[^ ]*  *//' -e 'x' \
     -e 's/ .*//' -e "s/'/'\\\\''/g" -e "s/^/echo 'k=</" \
     -e 'G' -e "s/\n/>' v=\\</" -e 's/$/\\>/' | sh

Going with a here document, we still need to escape the keyword (but differently).

  echo 'cat <<EOF'
  sed -e 's/^ */k=</' -e 'h' \
      -e 's/[^ ]*  *//' -e 'x' -e 's/ .*//' -e 's/[\$`]/\\&/g' \
      -e 'G' -e "s/\n/> v=</" -e 's/$/>/'
  echo 'EOF'
 } | sh

This is the fastest method if you have a lot of data: it doesn't start a separate process for each line.


The same techniques we used with sed work with awk. The resulting program is considerably more readable. Going with “echo \2”:

awk '
  1 {
      kw = $1;
      sub(/^ *[^ ]+ +/, "");
      gsub(/\047/, "\047\\\047\047", $1);
      print "echo \047k=<" kw ">\047 v=\\<" $0 "\\>";
  }' | sh

Using a here document:

awk '
  NR==1 { print "cat <<EOF" }
  1 {
      kw = $1;
      sub(/^ *[^ ]+ +/, "");
      gsub(/\\\$`/, "\\&", $1);
      print "k=<" kw "> v=<" $0 ">";
  END { print "EOF" }
' | sh
share|improve this answer
great answer. i'm going to use the pure shell solution, as the input file is indeed small and performance is not a concern, also it's clean and readable. – Ernest A C Sep 17 '12 at 10:53
a bit of a hack but neat enough. e.g. use sed to call out to xxd to decode long hex string . . . cat FtH.ch13 | sed -r 's/(.*text.*: [)([0-9a-fA-F]*)]/\1$(echo \2|xxd -r -p)]/;s/^(.*)$/echo "\1"/g' |bash >FtHtext.ch13 Where FtH.ch13 has lines like "foo bar hex text test: [666f6f0a62617200]" – gaoithe Jan 12 at 12:38

If a non-sed solution is acceptable, this PERL snippet will do the job:

$ echo "$input" | perl -ne 'chomp; /^\s*(.+?)\s+(.+)$/ && do { $v=`echo "$2"`; chomp($v); print "k=<$1> v=<$v>\n"}'
share|improve this answer
thanks but i'd rather avoid using another scripting language if i can and keep it to standard unix commands and bourne shell – Ernest A C Sep 16 '12 at 15:26

You could try this approach:

keyword value value
keyword "value  value"
keyword `uname`

process() {
  k=$1; shift; v="$*"
  printf '%s\n' "k=<$k> v=<$v>"

eval "$(printf '%s\n' "$input" | sed -n 's/./process &/p')"

(if I get your intention right). That is insert "process" at the beginning of each non-empty line to make it a script like:

process keyword value value
process keyword "value  value"
process keyword `uname`

to be evaluated (eval) where process is a function that prints the expected message.

share|improve this answer
this is exactly it, thanks – Ernest A C Sep 16 '12 at 16:00

Having GNU sed you can use the following command:

sed -nr 's/([^ ]+) (.*)/echo "\1" \2\n/ep' input

Which outputs:

keyword value value
keyword value  value
keyword Linux

with your input data.


The sed command suppresses regular output using the -n option. -r is passed to use extended regular expressions which saves us some escaping of special chars in the pattern but it is not required.

The s command is used to transfer the input line into the command:

echo "\1" \2

The keyword get's quoted the value not. I pass the option e - which is GNU specific - to the s command, which tells sed to execute the result of the substitution as a shell command and read it's results into the pattern buffer (Even multiple lines). Using the option p after(!) e makes sed printing the pattern buffer after the command has been executed.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.