Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I have a text file with the following format:

keyword value
keyword value
...

Where keyword is a single word and value is everything else until the end of line. I want to read the file from a shell script, in a way that the values (but not the keywords) undergo shell expansion.

With sed it's easy to match the keywords and value parts

input='
keyword value value
keyword "value  value"
keyword `uname`
'

echo "$input"|sed -e 's/^\([^[:space:]]*\)[[:space:]]\(.*\)$/k=<\1> v=<\2>/'

which produces

k=<keyword> v=<value value>
k=<keyword> v=<"value  value">
k=<keyword> v=<`uname`>

but then the question is how can I embed a shell command into the replacement part of the sed expression. In this case I would like the replacement to be \1 `echo \2`.

share|improve this question
    
Uhm... I am not SO sure to give it as an answer, but using DOUBLE quoted with sed should let you use shell $(command) or $variables inside the expression. –  St0rM Dec 27 '13 at 15:24

3 Answers 3

up vote 6 down vote accepted

Sed can't call a shell, so you'll have to do some of the processing outside sed. There are several solutions; all require careful quoting.

It's not clear exactly how you want the values to be expanded. For example, if a line is

foo hello; echo $(true)  3

which of the following should the output be?

k=<foo> value=<hello; echo   3>
k=<foo> value=<hello; echo   3>
k=<foo> value=<hello; echo 3>
k=<foo> value=<foo hello
  3>

I'll discuss several possibilities below.

pure shell

You can get the shell to read the input line by line and process it. This is the simplest solution, and also the fastest for short files. This is the closest thing to your requirement “echo \2”:

while read -r keyword value; do
  echo "k=<$keyword> v=<$(eval echo "$value")>"
done

read -r keyword value sets $keyword to the first whitespace-delimited word of the line, and $value to the rest of the line minus trailing whitespace.

If you want to expand variable references, but not execute commands outside command substitutions, put $value inside a here document. I suspect that this is what you were really looking for.

while read -r keyword value; do
  echo "k=<$keyword> v=<$(cat <<EOF
$value
EOF
)>"
done

sed piped into a shell

You can transform the input into a shell script and evaluate that. Sed is up to the task, though it's not that easy. Going with your “echo \2” requirement (note that we need to escape single quotes in the keyword):

sed  -e 's/^ *//' -e 'h' \
     -e 's/[^ ]*  *//' -e 'x' \
     -e 's/ .*//' -e "s/'/'\\\\''/g" -e "s/^/echo 'k=</" \
     -e 'G' -e "s/\n/>' v=\\</" -e 's/$/\\>/' | sh

Going with a here document, we still need to escape the keyword (but differently).

{
  echo 'cat <<EOF'
  sed -e 's/^ */k=</' -e 'h' \
      -e 's/[^ ]*  *//' -e 'x' -e 's/ .*//' -e 's/[\$`]/\\&/g' \
      -e 'G' -e "s/\n/> v=</" -e 's/$/>/'
  echo 'EOF'
 } | sh

This is the fastest method if you have a lot of data: it doesn't start a separate process for each line.

awk

The same techniques we used with sed work with awk. The resulting program is considerably more readable. Going with “echo \2”:

awk '
  1 {
      kw = $1;
      sub(/^ *[^ ]+ +/, "");
      gsub(/\047/, "\047\\\047\047", $1);
      print "echo \047k=<" kw ">\047 v=\\<" $0 "\\>";
  }' | sh

Using a here document:

awk '
  NR==1 { print "cat <<EOF" }
  1 {
      kw = $1;
      sub(/^ *[^ ]+ +/, "");
      gsub(/\\\$`/, "\\&", $1);
      print "k=<" kw "> v=<" $0 ">";
  }
  END { print "EOF" }
' | sh
share|improve this answer
    
great answer. i'm going to use the pure shell solution, as the input file is indeed small and performance is not a concern, also it's clean and readable. –  Ernest A C Sep 17 '12 at 10:53

If a non-sed solution is acceptable, this PERL snippet will do the job:

$ echo "$input" | perl -ne 'chomp; /^\s*(.+?)\s+(.+)$/ && do { $v=`echo "$2"`; chomp($v); print "k=<$1> v=<$v>\n"}'
share|improve this answer
    
thanks but i'd rather avoid using another scripting language if i can and keep it to standard unix commands and bourne shell –  Ernest A C Sep 16 '12 at 15:26

You could try this approach:

input='
keyword value value
keyword "value  value"
keyword `uname`
'

process() {
  k=$1; shift; v="$*"
  printf '%s\n' "k=<$k> v=<$v>"
}

eval "$(printf '%s\n' "$input" | sed -n 's/./process &/p')"

(if I get your intention right). That is insert "process" at the beginning of each non-empty line to make it a script like:

process keyword value value
process keyword "value  value"
process keyword `uname`

to be evaluated (eval) where process is a function that prints the expected message.

share|improve this answer
    
this is exactly it, thanks –  Ernest A C Sep 16 '12 at 16:00

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.