I'm trying to pipe the output of one command to two different awk commands. Following this post I am using tee and process substitution. However, I can't see the output for the substituted process.

nvidia-smi | tee >(awk '/ C / {print $6}') | awk '/ C / {print $3}' | xargs -r ps -o user

This is supposed to show the users and memory usage for all gpu processes. The memory usage and PID are extracted from nvidia-smi, respectively, by awk '/ C / {print $6}' and awk '/ C / {print $3}' with the latter then being piped to ps -o user. The output contains only the users though.

What I would like is

<memory-of-process1> <name-of-user-running-process1>
<memory-of-process2> <name-of-user-running-process2>
<memory-of-process3> <name-of-user-running-process3> 

and what I am getting is


I have tried adding fflush() or stdbuf -o0 to the first awk command, as suggested here.

  • There are a few issues with process substitution. It often produces output after the main pipeline terminates and a new shell prompt appears, which indicates it runs in background. I suspect its stdout is therefore disconnected from the terminal. Try redirecting that stream to a file to verify it runs. You might consider tailing that file, although it is not a great solution. – Paul_Pedant May 1 '20 at 12:00
  • Yes I can redirect to a file. nvidia-smi | tee >(awk '/ C / {print $6}' > test) | awk '/ C / {print $3}' | xargs -r ps -o user, and then cat test prints the memory usages. – ludog May 1 '20 at 12:08
  • To achieve the synchronisation at a line level: have awk open a pipe from a ps command, send the pids, and read back the user names internally. You probably want to store all the nvidia data and do this in an END action, rather than run a pipe for each line of input. – Paul_Pedant May 1 '20 at 12:09
  • I'm sorry I wouldn't know how to turn what you suggest into a command. Does synchronisation at a line level mean that the name and memory for each proc are printed on the same line? ie, a different issue to the current absence of memory being printed at all – ludog May 1 '20 at 12:17
  • By the time you get through two separate awks, and xargs, and ps, and pipe buffering, there is no way the awk outputs will arrive in pairs. As it is, each of $6 and $3 will have a newline anyway. I can fake up a tested command in a couple of hour's time, if nobody beats me to it. – Paul_Pedant May 1 '20 at 12:26

More compact solution using a shell script:

nvidia-smi | grep ' C ' | while read _ _ pid _ _ mem; do
    user="$( ps -o user "${pid}" | tail -n +2 )"
    printf '%8s  %6d  %s\n' "${mem}" "${pid}" "${user}"

Note, this runs three extra processes per pid.

Original solution using awk:

#! /bin/bash

function nvidia-smi { cat <<'[][]'
A C 937 D E 1.7MB
A C 1232 D E 0.25MB
A E 6112 D E 13MB
A C 2008 D E 437KB
A C 2024 D E 314157

/ C / { Pid[NR] = $3; xPid[$3] = NR; Mem[NR] = $6; }
    for (j in Pid) pp = pp "," Pid[j];
    cmd = "ps 2>&1 -o pid,user -p " substr (pp, 2);
    while (cmd | getline) 
        if ($1 in xPid) User[xPid[$1]] = $2;
    close (cmd);
    fmt = "%8s  %6d  %s\n";
    for (j = 1; j <= NR; ++j) 
        if (j in Mem)
            printf (fmt, Mem[j], Pid[j], User[j]);
    nvidia-smi | awk "${AWK}"

The function nvidia-smi just presents some test data -- discard that. You need the AWK variable (12 lines in the multi-line constant between single-quotes) and the brief pipeline below it.

A test. I included the pid that was using the memory:

paul $ ./nVid
   1.7MB     937  syslog
  0.25MB    1232  root
   437KB    2008  postfix
  314157    2024  paul
  • Thanks. I can't say I fully follow the awk code, but it seems this just does everything inside awk. I can also confirm it works with the 'real' nvidia-smi, except sometimes the user name is not printed. 282MiB 4611 282MiB 4613 282MiB 12090 282MiB 12091 282MiB 12093 282MiB 26005 282MiB 26009 1355MiB 7646 bob 1355MiB 7648 bob 347MiB 24436 1247MiB 4611 bob 1247MiB 4613 bob 1287MiB 12090 bob 1355MiB 24436 bob 1287MiB 12091 bob 1287MiB 12093 bob 1091MiB 26005 bob 1073MiB 26009 bob – ludog May 1 '20 at 14:52
  • sorry, there should be newlines in that but I can't format it in a comment – ludog May 1 '20 at 14:55
  • There is a race condition in there. If a process has just ended, it is possible it was present when nvidia-smi was run but has disappeared by the time ps gets run. I really don't know how transient GPU processes can be: for example, are they only connected while they are actually refreshing something. One fix might be to fetch the whole ps output before and after nvidia runs, and check both lists for the pid. – Paul_Pedant May 1 '20 at 15:28
  • Is that data list from one run, or two? Every pid that shows up without a username also shows up again with a user name. 7646 and 7648 only show up once. Possibly the pids are permanent threads, and only get allocated to a username while they are connected. – Paul_Pedant May 1 '20 at 15:40
  • That is from one run. Some processes are spread over multiple gpus. It looks like that is the cause, that the second gpu is then printed without a name. Normally, the gpu processes in my case are not transient, ie last several minutes at least. – ludog May 1 '20 at 16:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.