80

Given a command that changes its behaviour when its output is going to a terminal (e.g. produce coloured output), how can that output be redirected in a pipeline while preserving the changed behaviour? There must be a utility for that, which I am not aware of.

Some commands, like grep --color=always, have option flags to force the behaviour, but the question is how to work around programs that rely solely on testing their output file descriptor.

If it matters, my shell is bash on Linux.

3
  • You have to know how the command tests its output file descriptor and somehow return results consistent with the results that would be seen from an output terminal. Dec 16, 2015 at 15:11
  • Nice question. I think it could be useful to also mention the MySQL client that does not support an "enable interactive mode". So you cannot run "ssh example.com mysql" with interaction. Dec 1, 2022 at 16:20
  • 1
    @ValerioBozz: ssh has the -t option (force pseudo-terminal allocation) exactly for that use case scenario. It also has the -tt option to force tty allocation, even if ssh has no local tty (which means you never need unbuffer on the remote side).
    – Amir
    Dec 2, 2022 at 9:36

7 Answers 7

42

A history of toolsets

You are not the first person to want such a tool. People have been wanting such tools for 30 years. And they've existed for almost that long, too.

The earliest tool for this sort of thing was Daniel J. Bernstein's "pty" package, described by Rich Salz as a "Ginsu knife", which Bernstein wrote back at the turn of the 1990s in order to cheat at nethack (sic!). Version 4 of the "pty" package was published in 1992 to comp.sources.unix (volume 25 issues 127 to 135). It's still locatable on the World Wide Web. Paul Vixie described it at the time:

What can I say? It slices, it dices, it washes dishes, it walks the dog. It "just works", meaning that if you follow the directions you'll get a working package without any pulling of hair or gnashing of teeth or other standard porting activities.

Bernstein later updated this, somewhen on or before 1999-04-07, with a "ptyget" package, which he announced:

I've put together a new pseudo-tty allocator, ptyget. An alpha version is at ftp://koobera.math.uic.edu/pub/software/ptyget-0.50.tar.gz. There's a ptyget mailing list; to join, send an empty message to [email protected]. I designed ptyget's interface from scratch. It's much more modular than pty; the basic pty interface has now been split into three pieces:

  • ptyget: a tiny, low-level program — the only setuid program in the package — that allocates a new pseudo-tty and passes it to the program of your choice
  • ptyspawn: another small program that runs a child process under a pseudo-tty, waiting for it to exit and watching for stops
  • ptyio: another, only slightly larger, program that moves data back and forth

The old Ginsu knife pty is now spelled ptybandage, which is a synonym for ptyget ptyio -t ptyspawn; pty -d, for attaching network programs to pseudo-ttys, is now spelled ptyrun, which is a synonym for ptyget ptyio ptyspawn; and nobuf is a synonym for ptyget ptyio -r ptyspawn -23x. I've split off the session management features into a separate package.

That separate package was the "sess" package.

"ptyget" is, incidentally, notable for exemplifying a very early version of, and one of the few published instances of, Berstein's own never-published "redo" build system. dependon is a clear precursor to redo-ifchange.

Usage

ptybandage

ptybandage is what people usually want in a login session. Its primary use case is making programs that are sensitive to whether their standard inputs, outputs, or errors are connected to terminals operate that way even though they are in fact in shell pipelines, or have their standard file descriptors redirected to file.

It takes a command to run (which has to be a proper external command, of course) and runs it in such a way that it thinks that its standard input, output, and error are attached to a terminal, connecting those through to ptybandage's original standard input, output, and error.

It deals with the nuances of running under job control shells, ensuring that a terminal STOP character not only stops ptybandage but also stops the program running attached to the inner terminal.

ptyrun

ptyrun is what people usually want in TCP network servers. Its primary use case is remote execution environments that have not themselves set up terminals, running programs that don't operate as desired when there's no terminal.

It doesn't expect to be running under a job control shell, and if the command being run receives a stop signal it is simply restarted.

Available toolsets

Dru Nelson publishes both "pty" version 4 and "ptyget".

Paul Jarc publishes a fixed version of ptyget, that attempts to deal with the operating-system-specific pseudo-terminal device ioctls in the original that operating systems actually no longer provide.

The nosh source package comes with workalike ptybandange and ptyrun scripts, which use Laurent Bercot's execline tool and the nosh package's own pseudo-terminal management commands. As of nosh version 1.23 these are available pre-packaged in the nosh-terminal-extras package. (Earlier versions only supplied them to people who built from source.)

A few example uses

Jurjgen Oskam using ptybandage on AIX to feed input from a here document to a program that explicity opens and read its controlling terminal for a password prompt:

$ ptybandage dsmadmc <<EOF >uit.txt
joskam
password
query session
query process
quit
EOF

Andy Bradford using ptyrun on OpenBSD under daemontools and ucspi-tcp to make the bgplgsh interactive router control program accessible via the network whilst making it think that it is talking to a terminal:

#!/bin/sh
exec 2>&1
exec envuidgid rviews tcpserver -vDRHl0 0 23 ptyrun /usr/bin/bgplgsh

Further reading

6
  • 4
    Thanks for the lesson in history, but the tools you suggest are not available in a modern Linux distribution, hence not useful.
    – Amir
    Dec 17, 2015 at 8:53
  • 6
    There's a bunch of hyperlinks and an entire section of the answer devoted to where the tools most definitely are available.
    – JdeBP
    Dec 17, 2015 at 13:15
  • 2
    What's your opinion on expect? Jan 23, 2019 at 3:48
  • @CMCDragonkai, yes, expect seems a major omission from this answer. expect switched to using ptys in 1.0 released in February 1990, so would predate Daniel J. Bernstein's "pty" package Jul 16, 2022 at 13:12
  • Could not locate code for Debian 12/Bookworm. See socat answer below.
    – Paul
    Aug 16, 2023 at 2:47
36

You might get what you need by using unbuffer.

unbufferis a tcl / expect script. Look at the source if you want. Also note the CAVEATS section in man.

Also note that it does not execute aliases such as:

alias ls='ls --color=auto'

unless one add a trick as noted by Stéphane Chazelas:

If you do a alias unbuffer='unbuffer ' (note the trailing space), then aliases will be expanded after unbuffer.

7
  • Note on aliases acknowledged – it will also not work with any other shell constructs like builtins and functions.
    – Amir
    Dec 16, 2015 at 15:37
  • 7
    If you do a alias unbuffer='unbuffer ' (note the trailing space), then aliases will be expanded after unbuffer. May 18, 2016 at 16:52
  • 2
    unbuffer it is! sudo apt install expect -- That was unclear.
    – loxaxs
    Mar 17, 2018 at 11:35
  • Which cases does this help with? How?
    – Mikel
    Jan 31, 2020 at 7:13
  • Nice trick. Anyone have a good explanation about why this trailing-space/alias trick works? Even better if that comes with an explanation about why unbuffer could not see the alias to begin with. Also, is there any disadvantage to adding alias unbuffer='unbuffer ' to my ~/.bashrc to be used by default?
    – Cory Gross
    May 4, 2020 at 6:01
28

You can use socat to start your process with a pty connected, and get socat to connect the other end of the pty to a file. Which AFAIU is exactly what you asked:

socat EXEC:"my-command",pty GOPEN:mylog.log

This method will cause isatty called by my-command to return true and a process that relies only on that will be fooled to output control codes. Note that some processes (notably grep) also check the value of the TERM environment variable, so you might need to set that to something reasonable, like "xterm"

2
  • 1
    If you want the output still on stdout (to pipe to the next part of your command line), you can replace the second argument with a - or STDOUT. Adding a parameter of -t0 will avoid socat's timeout when one side closes.
    – keturn
    Dec 18, 2019 at 1:31
  • 1
    Adding -u makes sure socat only sends traffic one way. That's a non-issue as long as the destination is a plain file or stdout, but when getting more creative with the destination I've found one-way mode helps simplify things.
    – keturn
    Dec 18, 2019 at 1:37
22

There is also a nice solution posted here on Super User by KarlC:

Compile a small shared library:

echo "int isatty(int fd) { return 1; }" | gcc -O2 -fpic -shared -ldl -o isatty.so -xc -

Then tell your command to load this isatty(3) override dynamically:

LD_PRELOAD=./isatty.so mycommand

This might not work for every command out there, may even break some in unexpected ways, but would probably work in most cases.

6
  • 5
    For MacOS users, you can get the same behavior by using DYLD_INSERT_LIBRARIES=./isatty.so DYLD_FORCE_FLAT_NAMESPACE=y mycommand Mar 13, 2017 at 14:42
  • It works with the clang-tidy color output issue with cmake, thanks! Dec 2, 2020 at 4:36
  • Note for macOS: When using Apple made macOS system utilities they seem to ignore DYLD_INSERT_LIBRARIES. Probably for security reasons. Though for calling fs_usage, and enforcing full width when piping, just doing unbuffer fs_usage worked for me. (and NOT passing -w)
    – Chris
    Nov 17, 2021 at 23:57
  • Is there any significant reason why this would break anything in any way or was that just a guess? Is it safe or unsafe to preload this hack for every command?
    – Akito
    Dec 8, 2021 at 20:21
  • 1
    @Akito I have no reason to believe it might harm you or anything on your computer. It might lead to corner cases, e.g. when one part of a program uses this implementation of isatty and a different part of the program uses a statically linked implementation, leading to inconsistency.
    – Amir
    Dec 28, 2021 at 14:12
18

How about using script(1)?

For example:

script -q -c 'ls -G' out_file

Will save the ls output to out_file with the color codes preserved.

4
  • This does not work here. How colours are preserved? Is there a tool that I should use to output out_file with its colours?
    – Kira
    Dec 16, 2015 at 14:40
  • 3
    @Kira for viewing files with ANSI colour escape sequences, I use less -R. In this case, though, I wanted the output to continue in the pipeline, which did end up in my terminal eventually. Using cat for illustration, it was something like script -q -c 'ls -G' /dev/null | cat, which suppresses the typescript file entirely, leaving only the program’s output.
    – Amir
    Dec 16, 2015 at 15:05
  • To avoid creating a file, simply use a dash (-) as script output file, for exemple: script -q -c 'ls -G' - Nov 25, 2016 at 20:38
  • @FranklinPiat That created a file called - containing a copy of the output. (The output also went to the terminal anyway.)
    – Tanz87
    Mar 13, 2022 at 23:08
12

Unsatisfied with the solutions presented here so far, I released the python. She was effective. This solution doesn't require setuid permissions or any actually-insane monkey-patching with shared libraries and LD_LIBRARY_PATH. Save this script somewhere in your PATH. Don't forget to chmod +x it. Suppose you save this script as pty.

#!/usr/bin/env python

from sys import argv
import os
import signal

# I've had problems with python's File objects at this low a level, so
# we're going to use integers to specify all files in this script.
stdin = 0
stdout = 1
stderr = 2
# Include this if passing the command and arguments to fish to
# prevent fish from applying any expansions.
#import re
#def fish_escape(args):
#    def escape_one(arg):
#        return "'" + re.sub(r"('|\\)", r'\\\1', arg) + "'"
#    escaped_args = map(escape_one, args)
#    return ' '.join(escaped_args)

if len(argv) < 2:
    os.write(stderr,
b"""A tragically beautiful piece of hackery, made to fool programs like ls,
grep, rg, and fd into thinking they're actually connected to a terminal.
Its usage:

pty command [arg1 arg2 ...]

Examples:
pty ls --color -R | less -r
git log -p | pty rg <search terms> | less -r
""")
    exit(255)

# We do not use forkpty here because it would block ^Cs from reaching the
# child process. And we don't need that.
ptm, pts = os.openpty()
pid = os.fork()
if pid == 0:
    # The child runs this.
    # To get the behaviour we want, we only need to replace the process's
    # stdout with pts. Everything else should remain in place, so that things
    # like `ps -eF | pty rg python | less -r` will still work as intended.
    os.dup2(pts, stdout)
    # This is not like a subprocess.call(). It replaces the entire child
    # process with argv[1:], meaning execvp will not return! Web search
    # "fork exec" for more.
    os.execvp(argv[1], argv[1:])
    # Use this if calling fish.
    #os.execvp('fish', ['fish', '-c', fish_escape(argv[1:])])


# The parent runs this.

# If the parent doesn't close the slave end, the script won't be able to
# exit. The next read on ptm after the child process terminates would hang
# forever because pts would technically still be open.
os.close(pts)

# The whole process group gets SIGINT, including the child, so we don't need
# to react to it. We'll know when to leave, judging by what the child does.
signal.signal(signal.SIGINT, signal.SIG_IGN)

while True:
    try:
        chunk = os.read(ptm, 4096)
    except OSError:
        break
    try:
        os.write(stdout, chunk)
    except BrokenPipeError:
        # This happens when the parent is piping output to another process in a
        # pipeline, like in `pty ls --color -R | less -r`, and the receiving
        # process is terminated before the child has exited. If the receiving
        # process is less, this can happen very easily. It happens every time
        # the user decides to quit less before it has displayed all output. So,
        # we need to stop the child process now.
        os.kill(pid, signal.SIGTERM)
        # Also close the child's inputs and outputs, just in case it is
        # blocking on them and can't react to the SIGTERM as a result.
        os.close(ptm)
        break
wait_pid, status = os.waitpid(pid, 0)
exit(status >> 8)
11
  • Worked perfectly for me... I wanted to "massage" the output of 7zip while it was running, but it – seemingly – buffers its output if it is being piped. Using this put it back in "terminal mode" while still allowing the output to be piped.
    – TripeHound
    Apr 10, 2020 at 16:37
  • This is the only solution out of many I've tried that both (1) preserves colors (from ANSI escape codes) in piped output, and (2) passes SIGINT (Ctrl+C) to the child process. Thank you! I did have to change the script's shebang to specify python3 to get it working. On my Mac, the default Python installation is Python 2, which produces a "'BrokenPipeError' is not defined" error. Mar 19, 2021 at 3:24
  • Thank you, that seems to just work! (I use fishshell, so I tweaked it from os.execvp(argv[1], argv[1:]) to os.execvp('fish', ('fish',) + tuple([ '-c', ' '.join(argv[1:]) ])) so I can use my autoloading fish functions, and... that seems to work?) (And ubuntu 20.04 also needs a 3 added to the shebang.) (... and of course, you prolly need to mess around with like -r/-R flags when piping into less in order to get as close as possible to what you really want...)
    – Owen_AR
    Apr 3, 2021 at 12:53
  • If you're going to invoke a shell, you'll just have to remember to do extra escaping if one of the arguments going into python has spaces. Apr 5, 2021 at 5:11
  • @Owen_AR see my edit for specifics. Apr 6, 2021 at 20:47
0

Based on @Amir's answer, here is a script which generates and then includes the library at runtime:

#!/bin/bash
set -euo pipefail

function clean_up {
  trap - EXIT # Restore default handler to avoid recursion
  [[ -e "${isatty_so:-}" ]] && rm "$isatty_so"
}
# shellcheck disable=2154 ## err is referenced but not assigned
trap 'err=$?; clean_up; exit $err' EXIT HUP INT TERM

isatty_so=$(mktemp --tmpdir "$(basename "$0")".XXXXX.isatty.so)
echo "int isatty(int fd) { return 1; }" \
  | gcc -O2 -fpic -shared -ldl -o "$isatty_so" -xc -
# Allow user to SH=/bin/zsh faketty mycommand
"${SH:-$SHELL}" -c 'eval $@' - LD_PRELOAD="$isatty_so" "$@"
1
  • How is this script used and what exactly does it do? Jun 18, 2023 at 19:47

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .