Using bash shell function inside AWK

Question

Is it possible to use bash function inside AWK somehow?

Example file (string, int, int, int)

Mike 247808 247809 247810

Trying to convert values from decimal to hexadecimal.

Function defined either in .bashrc or in shell script.

$ awk '{print $1 ; d2h($2)}' file

awk: calling undefined function d2h
 input record number 1, file file
 source line number 1

Answer 1

Try to use system() function:

awk '{printf("%s ",$1); system("d2h " $2)}' file

In your case system will call d2h 247808 and then append output of this command to printf output:

Mike 3C800

EDIT:

As system uses sh instead of bash I can't find a way to access .bashrc. But you can still use functions from your current bash script:

#!/bin/bash
d2h() {
    # do some cool conversion here
    echo "$1" # or just output the first parameter
}
export -f d2h
awk '{printf("%s ",$1); system("bash -c '\''d2h "$2"'\''")}' file

EDIT 2:

I don't know why, but this is not working on my Ubuntu 16.04. This is strange, because it used to work on Ubuntu 14.04.

Answer 2

You can call bash from awk and use its output. That is obviously dangerous from a performance perspective if it happens too often. Quoting the man page:

command | getline [var]

Run command piping the output either into $0 or var,

command would be a bash script that contains the function definition and executes the function.

Answer 3

Converting from decimal to hexadecimal is something that awk can very well do itself. And you could define an awk function to do it:

function d2h(d) {
  return sprintf("%x", d)
}

Now to answer the question in the general case, for awk to run bash functions, you'd need awk to execute a bash shell, that bash to interpret the definition of that function, and call that function, with the value extracted by awk passed as arguments.

Not trivial.

bash supports exporting functions via the environment, so it's available in subsequent invocations of bash, so that's one way to pass the definition of the function to the bash invoked by awk:

export -f d2h

The only ways for awk to execute a command (bash here) are with its system("cmd"), or print... | "cmd" or "cmd" | getline. In all cases, awk runs a shell to interpret that cmd, but it will be sh, not bash. So you need to construct a command line for sh that is a bash invocation that interprets a bash command line to invoke the function, so you need to be careful with quoting:

export -f d2h
<file awk -v q="'" '
  function shquote(s) {
    gsub(q, q "\\" q q, s)
    return q s q
  }
  {print $1; system("exec bash -c '\''d2h \"$1\"'\'' bash " shquote($2))}'

If you want to get the output of the function back into awk, you'd need to transfer it back via a pipe. For that, you'd use cmd | getline instead of system(cmd) (which leaves cmd's stdout untouched).

cmd | getline line stores one line (strictly speaking one record, records being lines by default), so to get the whole output in the cases where it's made of several lines, you'd need a loop such as:

awk '...
  cmd = "exec bash -c '\''d2h \"$1\"'\'' bash " shquote($2)
  output = ""
  while ((cmd | getline line) > 0) {
    output = output line RS
  }
  sub(RS "$", "", output) # remove the last newline
  ...'

That does mean running one sh and one bash for each each invocation of the function, so is going to be quite inefficient. That would end up being even significantly more inefficient than having bash do the reading and splitting with a while read loop:

(unset -v IFS; while read -r a b rest; do
  printf '%s\n' "$a"
  d2h "$b"
 done < file)

Also note that since shellshock, bash now export functions in environment variables that are named like BASH_FUNC_d2h%%. Some sh implementations including mksh and newer versions of dash remove those environment variables from the environment:

$ env 'foo%%=bar' dash -c 'printenv foo%%'
$ env 'foo%%=bar' mksh -c 'printenv foo%%'
$ env 'foo%%=bar' zsh  -c 'printenv foo%%'
bar
$ env 'foo%%=bar' bash -c 'printenv foo%%'
bar

So instead of relying on the flimsy function export feature, you could pass the function definition some other way. It could be via an environment variable with a usual name:

BASH_FUNCTIONS=$(typeset -f d2h) awk '
   ...
   cmd = "exec bash -c '\''eval \"$BASH_FUNCTIONS\";" \
         "d2h \"$1\"'\'' bash " shquote($2)
   ...'

Answer 4

Try doing this :

awk '{print $1 ; printf "%x\n", $2}' file

AFAIK, you can't use a bash function in awk but only a script. You can use an awk function if needed.

Answer 5

Using a user defined bash function inside awk

Disclaimer: I realize this is not what the OP is trying to do, but Google will lead others like me to this answer.

Situation

You have a bash script that is organized with functions (because you do not hate yourself or [most] coworkers) and at least 1 of those functions needs to call another from within awk.

Solution

Script

#!/bin/env bash

# The main function - it's a sound pattern even in BASH
main(){
    # In the awk command I do some tricky things with single quotes. Count carefully...
    # The first $0 is outside the single quotes so it is the name of the current bash script.
    # The second $0 is inside the single quotes so it is awk's current line of input.
    awk '{printf("%s. ", ++c); system("'$0' --do"); print $0}'<<-PRETEND_THIS_IS_AN_INPUT_STREAM
        and
        and
        well
    PRETEND_THIS_IS_AN_INPUT_STREAM
}

# functionized to keep things DRY
doit(){
    echo -n "doin' it "
}


# check for a command switch and call different functionality if it is found
if [[ $# -eq 1 && $1 == "--do" ]];
then
        doit
else
        main
fi

Output

$ ./example.sh
1. doin' it and
2. doin' it and
3. doin' it well

Answer 6

Just a quick example to demonstrate @HaukeLaging's command|getline :

1. let input be:

Dear friends
my name is `id -nu` and
today is `date "+%Y-%m-%d"`.

where we are following shell syntax, in the input,

`command`

is used to denote inline commands to be replaced by the result of his execution.

2. We can expand inline shell command by:

#!/usr/bin/gawk -f

BEGIN  { FS ="`";        }
NF==3  { $2 | getline $2 }
       { print           }

3. Usage (after the usual chmod ):

$ expand-inline input
Dear friends
my name is jjoao and
today is 2018-01-15.

Answer 7

This will give you good chances.

cat ../logs/em2.log.1 |grep -i 192.168.21.15 |awk '{system("date"); print $1}'

system function enables you to parse bash command within awk stream.

Answer 8

This answer if for GNU awk, but other versions may implement the same trick. The idea is that as long as the right-hand side of the pipe is exactly the same and doesn't exit, awk doesn't close the pipe. But if it is different, it will open a new pipe. For instance, if you type:

awk 'BEGIN{ shell= "/bin/bash" } { print $0 | "/bin/bash" } END { print "echo $a" | " /bin/bash" }' 

you can type:

a=5
echo $a

and get the expected (?) response 5: no new shell was spawned. But in the END section, I typed an extra space, so the RHS of the pipe is different, and a new shell is spawned with no value for variable a (and thus will print a blank line on exit. I cannot find it in the documentation, but the recommanded way of using long term pipes is as I have: put the command in the BEGIN section in a variable, and reuse that variable.

Giving a user-defined function to a bash session and then reusing it is an exercice left to the reader :)