14

I have a text file with the following format:

keyword value
keyword value
...

Where keyword is a single word and value is everything else until the end of line. I want to read the file from a shell script, in a way that the values (but not the keywords) undergo shell expansion.

With sed it's easy to match the keywords and value parts

input='
keyword value value
keyword "value  value"
keyword `uname`
'

echo "$input"|sed -e 's/^\([^[:space:]]*\)[[:space:]]\(.*\)$/k=<\1> v=<\2>/'

which produces

k=<keyword> v=<value value>
k=<keyword> v=<"value  value">
k=<keyword> v=<`uname`>

but then the question is how can I embed a shell command into the replacement part of the sed expression. In this case I would like the replacement to be \1 `echo \2`.

  • Uhm... I am not SO sure to give it as an answer, but using DOUBLE quoted with sed should let you use shell $(command) or $variables inside the expression. – St0rM Dec 27 '13 at 15:24
17

Standard sed can't call a shell (GNU sed has an extension to do it, if you only care about non-embedded Linux), so you'll have to do some of the processing outside sed. There are several solutions; all require careful quoting.

It's not clear exactly how you want the values to be expanded. For example, if a line is

foo hello; echo $(true)  3

which of the following should the output be?

k=<foo> value=<hello; echo   3>
k=<foo> value=<hello; echo   3>
k=<foo> value=<hello; echo 3>
k=<foo> value=<foo hello
  3>

I'll discuss several possibilities below.

pure shell

You can get the shell to read the input line by line and process it. This is the simplest solution, and also the fastest for short files. This is the closest thing to your requirement “echo \2”:

while read -r keyword value; do
  echo "k=<$keyword> v=<$(eval echo "$value")>"
done

read -r keyword value sets $keyword to the first whitespace-delimited word of the line, and $value to the rest of the line minus trailing whitespace.

If you want to expand variable references, but not execute commands outside command substitutions, put $value inside a here document. I suspect that this is what you were really looking for.

while read -r keyword value; do
  echo "k=<$keyword> v=<$(cat <<EOF
$value
EOF
)>"
done

sed piped into a shell

You can transform the input into a shell script and evaluate that. Sed is up to the task, though it's not that easy. Going with your “echo \2” requirement (note that we need to escape single quotes in the keyword):

sed  -e 's/^ *//' -e 'h' \
     -e 's/[^ ]*  *//' -e 'x' \
     -e 's/ .*//' -e "s/'/'\\\\''/g" -e "s/^/echo 'k=</" \
     -e 'G' -e "s/\n/>' v=\\</" -e 's/$/\\>/' | sh

Going with a here document, we still need to escape the keyword (but differently).

{
  echo 'cat <<EOF'
  sed -e 's/^ */k=</' -e 'h' \
      -e 's/[^ ]*  *//' -e 'x' -e 's/ .*//' -e 's/[\$`]/\\&/g' \
      -e 'G' -e "s/\n/> v=</" -e 's/$/>/'
  echo 'EOF'
 } | sh

This is the fastest method if you have a lot of data: it doesn't start a separate process for each line.

awk

The same techniques we used with sed work with awk. The resulting program is considerably more readable. Going with “echo \2”:

awk '
  1 {
      kw = $1;
      sub(/^ *[^ ]+ +/, "");
      gsub(/\047/, "\047\\\047\047", $1);
      print "echo \047k=<" kw ">\047 v=\\<" $0 "\\>";
  }' | sh

Using a here document:

awk '
  NR==1 { print "cat <<EOF" }
  1 {
      kw = $1;
      sub(/^ *[^ ]+ +/, "");
      gsub(/\\\$`/, "\\&", $1);
      print "k=<" kw "> v=<" $0 ">";
  }
  END { print "EOF" }
' | sh
  • great answer. i'm going to use the pure shell solution, as the input file is indeed small and performance is not a concern, also it's clean and readable. – Ernest A C Sep 17 '12 at 10:53
  • a bit of a hack but neat enough. e.g. use sed to call out to xxd to decode long hex string . . . cat FtH.ch13 | sed -r 's/(.*text.*: [)([0-9a-fA-F]*)]/\1$(echo \2|xxd -r -p)]/;s/^(.*)$/echo "\1"/g' |bash >FtHtext.ch13 Where FtH.ch13 has lines like "foo bar hex text test: [666f6f0a62617200]" – gaoithe Jan 12 '16 at 12:38
13

Having GNU sed you can use the following command:

sed -nr 's/([^ ]+) (.*)/echo "\1" \2\n/ep' input

Which outputs:

keyword value value
keyword value  value
keyword Linux

with your input data.

Explanation:

The sed command suppresses regular output using the -n option. -r is passed to use extended regular expressions which saves us some escaping of special chars in the pattern but it is not required.

The s command is used to transfer the input line into the command:

echo "\1" \2

The keyword get's quoted the value not. I pass the option e - which is GNU specific - to the s command, which tells sed to execute the result of the substitution as a shell command and read it's results into the pattern buffer (Even multiple lines). Using the option p after(!) e makes sed printing the pattern buffer after the command has been executed.

  • You can do without both the -n and p options i.e. sed -r 's/([^ ]+) (.*)/echo "\1" \2\n/e' input. But thanks for this! I did not know about the e option. – Kaushal Modi Nov 29 '16 at 17:42
  • @KaushalModi Oh yes, you are right! I'm sitting on the fence when it comes to e option (introduced by GNU). Is that still sed? :) – hek2mgl Nov 29 '16 at 17:44
  • Well, it worked for me. It is GNU sed by default for me (GNU sed version 4.2.1) on RHEL distribution. – Kaushal Modi Nov 29 '16 at 17:54
4

You could try this approach:

input='
keyword value value
keyword "value  value"
keyword `uname`
'

process() {
  k=$1; shift; v="$*"
  printf '%s\n' "k=<$k> v=<$v>"
}

eval "$(printf '%s\n' "$input" | sed -n 's/./process &/p')"

(if I get your intention right). That is insert "process" at the beginning of each non-empty line to make it a script like:

process keyword value value
process keyword "value  value"
process keyword `uname`

to be evaluated (eval) where process is a function that prints the expected message.

1

If a non-sed solution is acceptable, this PERL snippet will do the job:

$ echo "$input" | perl -ne 'chomp; /^\s*(.+?)\s+(.+)$/ && do { $v=`echo "$2"`; chomp($v); print "k=<$1> v=<$v>\n"}'
  • 1
    thanks but i'd rather avoid using another scripting language if i can and keep it to standard unix commands and bourne shell – Ernest A C Sep 16 '12 at 15:26
0

ONLY KISS SHORT PURE SED

I will do this

echo "ls_me" | sed -e "s/\(ls\)_me/\1/e" -e "s/to be/continued/g;"

and it working.

  • Could you please explain how does it work? – elysch Aug 29 '18 at 21:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.