7

I'm trying to make a sed command so that numbers longer than 3 digits were converted to hex. I.e. a string like 124 3275 7535 should result in 124 0xccb 0x1d6f. Here's what I currently have:

sed 's@\([0-9]\{4,\}\)@sh -c "printf 0x%x \1"@ge'

But when the string doesn't match, it attempts to run the unchanged string as an external command, so for the example string above I get

sh: 1: 124: not found

How can I achieve what I'm trying to do (preferably still using sed)?

  • 7
    Why do you prefer sed. It sucks at such tasks. sed is very good at what it was intended to do, which is: text Stream EDiting. What you need is something which can do mathematical work on numbers. sed was not designed for that. But I'm sure it can be done by sed, but that would most likely not be the preferable solution... (ps. sed rocks! for many jobs, but this isn't one of those) – Peter.O Jun 25 '15 at 10:58
  • @Peter.O I prefer sed because this transformation is part of a large number of transformations (like `sed -e 'trasf1' -e 'trasnf2' ... -e 'transfN'), and it seems it'd be easier to continue using this approach. But of course I could just chain the result to some other processor, designed to handle this transformation of long numbers. – Ruslan Jun 25 '15 at 11:20
  • 2
    You could use perl e.g. perl -pe 's/\d{4,}/sprintf "%#x", $&/ge' which would allow you to maintain the other expressions more-or-less as-is – steeldriver Jun 25 '15 at 11:43
  • 1
    @steeldriver: That's great! ... make it an answer, otherwise the awsomeness may go unnoticed ... One day, when I grow up, I want to be able to juggle perl that well - it makes awk look like COBOL :) – Peter.O Jun 25 '15 at 11:52
  • 1
    Note that Perl is perfectly capable of doing all the other transformations that you're doing with sed too. So switching to Perl needn't be a big problem, provided you know/learn Perl. – Jonathan Leffler Jun 25 '15 at 12:39
6

Although it's not "with sed" as per your question title, if you switch from sed to perl you could use an equivalent expression such as

perl -p -e 's/\b\d{4,}\b/sprintf "%#x", $&/ge'

which should allow you to preserve other expressions in your chain more-or-less as is.

  • Great. Can this be modified to not transform things like sd2525, i.e. things which aren't really numbers? I've tried adding a ([ \t]|^) before \d and \1 before sprintf, but apparently it is treated as a command, not as it'd be in sed... – Ruslan Jun 25 '15 at 12:03
  • When you say ...preserve other expressions more-or-less, are you referring to delimiter spacings? (they seem to be ok) – Peter.O Jun 25 '15 at 12:07
  • I mean to preserve other expressions, only change numbers. Something like this perl -pe 's/(\s)(\d{4,})(\s)/sprintf "%s%#x%s", $1, $2, $3/ge'. Are there any caveats in this implementation? – Ruslan Jun 25 '15 at 12:11
  • 2
    @Ruslan thanks for the reminder: perhaps simpler, just add word boundary anchors i.e. \b\d{4,}\b. I will update my answer. – steeldriver Jun 25 '15 at 12:13
  • @Peter.O I just meant that perl should also be able to handle your other -e transf expressions, since it provides a superset of sed's functionality – steeldriver Jun 25 '15 at 12:17
7

The e flag of the s command of the GNU implementation of sed is to evaluate the content of the pattern space after the substitution has been applied (successfully), and replace the pattern space with its output, not to evaluate the substitution.

Here, for an input like:

foo 1234 123

You'd need the subsitution to result in the pattern space containing:

printf %s 'foo '
printf 0x%x 1234
printf %s ' 123'

For the e flag to convert that to foo 0x3d2 123 via the shell command. That's not impossible like with:

LC_ALL=C sed -E "
  /[0-9]{4}/!b # optimisation
  s/'/&\\\\&/g
  s/[0-9]{4,}/'\nprintf 0x%x &\nprintf %s '/g
  s/.*/printf %s '&'/e"

But that's pretty awkward and means running one shell per matching line of input. Without even using that GNUism, you could as well do:

LC_ALL=C sed "
  s/'/&\\\\&/g
  s/[0-9]\{4,\}/'\\
printf 0x%x &\\
printf %s '/g
  s/.*/printf %s '&\\
'/" | sh

Which would run one sh.

Also, evaluating arbitrary data as shell code like that tends to make me nervous. For instance, without the LC_ALL=C above, that would constitute an arbitrary command execution vulnerability. Try for instance on something like the output of:

printf '0000\200; echo GOTCHA>&2\n'

in a UTF-8 locale.

Here, you'd rather want to use something like perl:

perl -pe 's/\d{4,}/sprintf "0x%x", $&/ge'

perl's e flag is more inline with what you expect. It does evalute the subsitution as perl code (and is not launching a new perl interpreter every time like with GNU sed's e).

5

awk was designed for exactly this type of broad-spectrum text manipulation. Note, there is no need to pipe to any secondary tool(s).

awk '{ for( fn=1;fn<=NF;fn++ ){
           fmat=(length($fn)>3)?"0x%x":"%s"
           dlim=(fn==NF?"\n":" ")  
           printf( fmat dlim, $fn )}}' <<<'124 3275 7535' 

output, as per your sample:

124 0xccb 0x1d6f
3
echo 124 3275 7535 |
sed 's/.*/[&]p/;s/[0-9]\{4,\}/]P&p[/g;1s/^/16o /' |dc

It's not so bad at it. You've just gotta remember what it's for.

There are a couple drawbacks here, though. For example, if there are any square brackets in that input string, the whole thing goes sideways. I have a couple lines to handle that, but I don't know if I wanna dig it up right now.

The other thing is, as written, that will do the hex numbers each followed by a \newline. Portably it's the only way to go w/ dc. W/ GNU dc you can replace the &p w/ &n.

Anyway:

output

124 CCB
 1D6F

With GNU's n:

output

124 CCB 1D6F

I guess we could do the 0x thing, if you want:

echo 124 3275 7535|
sed -Ee1i16o -e's/.*/[&]p/;s/[0-9]{4,}/0x]P&n[/g'|dc

...which assumes GNU tools all round.

output

124 0xCCB 0x1D6F

Whatever can be said about the other tools in your box, sed was designed to turn a stream into something useful. It's not very good w/ numbers, and that's a fact, but it's pretty good at passing them off to those tools which are, such as calculators.

There are other options with GNU sed - though none come off as simply, and I expect you'll be hard-pressed to find a solution which does, for that matter. Something like your code in the question:

echo 124 3275 7535|
sed -E "h;s/[0-9]{4,}/%#x/g;s/[0-9]+/%d/g
        s/.*/printf '&' \\\\/;G;e"

output

124 0xccb 0x1d6f
  • Wow! Well done - with sed alone +1. Maybe that should be on the first lines of your answer (as it answered the main preference of his question), I almost missed it at the end... – Peter.O Jun 25 '15 at 12:48
  • @Peter.O - it isn't - it's essentially system() - the dc way is far preferable. That's a shell per line. It can be interesting to have some other process manipulate sed's outfiles in strange ways, but e otherwise rarely blips my radar. dc is especially good at exactly this kind of thing: try echo 2o 10000p|dc or echo 10000o 10p|dc. In fact, though, i didn't realize the code in the question was trying to e - i thought it was sed <in |sh - i guess i just need to look at these things sometimes. I would probably have mentioned it at the outset otherwise. – mikeserv Jun 25 '15 at 12:52
0

I second what Peter.O said in the comments: here's a bash way to do this (it needs a at the end of each number):

echo '124 3275 7535 ' | while read -d ' ' x; do [ ${#x} -ge 4 ] && printf "0x%x " $x || printf "%d " $x; done

If your input stream doesn't have a at the end of the line (as it looks like from your example), than sed comes in handy:

echo '124 3275 7535' | sed 's/$/ /' | while read -d ' ' x; do [ ${#x} -ge 4 ] && printf "0x%x " $x || printf "%d " $x; done

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.