How to remove line wrapping from DNF & YUM commands?

Question

When using dnf and yum on rpm based Linux distros (RHEL/Red Hat, Fedora, CentOS, etc) the utility will automatically wrap lines to make it more friendly for the user to read. This is problematic as it makes it extremely annoying to work with the data through pipelining.

For example:

$ dnf search jenkins-ssh-credentials-plugin-javadoc
Last metadata expiration check: 6 days, 15:30:08 ago on Thu Sep  1 21:09:10 2016.
============= N/S Matched: jenkins-ssh-credentials-plugin-javadoc =============
jenkins-ssh-credentials-plugin-javadoc.noarch : Javadoc for jenkins-ssh-credentials-plugin
$ dnf search jenkins-ssh-credentials-plugin-javadoc | grep ssh
====== N/S Matched: jenkins-ssh-credentials-plugin-javadoc =======
jenkins-ssh-credentials-plugin-javadoc.noarch : Javadoc for
                                              : jenkins-ssh-credentials-plugin

You can see that once the output for DNF is put through grep it decides to wrap the data in a completely different way then when normally displayed to the user.

Multiple issues have been filed about this behavior ( #584525, #986740 ) and consistently the issues are closed as CLOSED NOTABUG because "Yum is an interactive text-based ui which is not suited, nor intended for piping.". The solution as per the Red Hat developers is to "use a different tool for the job."

It seems unreasonable to have to do this, especially when the methods supplied (install repoquery for example) don't even exist within the dnf utilities and require installing a dozen more packages just to parse the output of this data.

Ideally a user would be able to just use the data in pipelining. In lieu of that, it would be nice to have a simple one-liner which could be used to make the data usable.

Answer 1

There is a great list of "example" commands/incantations for sed that have been curated for a very long time in via the name sed1line.txt (http://sed.sourceforge.net/sed1line.txt). In this file there is a great example which is helpful in this case:

# if a line begins with an equal sign, append it to the previous line
# and replace the "=" with a single space
sed -e :a -e '$!N;s/\n=/ /;ta' -e 'P;D'

In the case of our dnf/yum output we know that the wrapped lines will begin with one or more instances of white-space, followed by a colon (:), followed by more white-space. Using this knowledge we can modify our sed example to account for this and join the lines:

sed -r -e :a -e '$!N;s/\n[[:space:]]+://;ta' -e 'P;D'

In this case we have a decision to make, because the + is part of the "extended" regular expression set we need to either add the -r to use the + symbol, or fall back to using * in it's place (which will match zero or more instances of white-space). Either will work, the above example was provided to be pedantically correct.

The result is as follows:

$ dnf search jenkins-ssh-credentials-plugin-javadoc | sed -r -e :a -e '$!N;s/\n[[:space:]]+://;ta' -e 'P;D'  | grep ssh
============= N/S Matched: jenkins-ssh-credentials-plugin-javadoc ==============
jenkins-ssh-credentials-plugin-javadoc.noarch : Javadoc for jenkins-ssh-credentials-plugin

As you can see, now grep works as expected. In addition, this allows you to do more creative formatting of the output to make scanning through the list of packages easier to do:

$ dnf search ssh | sed -r -e :a -e '$!N;s/\n[[:space:]]+://;ta' -e 'P;D' | sort  | awk -F: '!($1~/^====/) {printf "%-40s : %s\n", $1, substr($0, index($0, $2))}'
ansible.noarch                           :  SSH-based configuration management, deployment, and task execution system
apache-sshd-javadoc.noarch               :  API documentation for apache-sshd
apache-sshd.noarch                       :  Apache SSHD
autossh.x86_64                           :  Utility to autorestart SSH tunnels
bareftp.x86_64                           :  File transfer client supporting the FTP, FTP over SSL/TLS (FTPS) and SSH
belier.noarch                            :  Generates scripts allowing you to chain many ssh connections
btrfs-sxbackup.noarch                    :  Incremental btrfs snapshot backups with push/pull support via SSH
...
trilead-ssh2.noarch                      :  SSH-2 protocol implementation in pure Java
WebShell.noarch                          :  SSL server for web-based SSH access from browsers and mobile devices
x11-ssh-askpass.x86_64                   :  A passphrase dialog for X and not only for OpenSSH

Answer 2

yum and dnf write progress messages, overstriking a line by using carriage returns. Those messages are not wrapped. grep is not designed for that, assuming lines do not have embedded control characters. grep does make some assumptions about terminals, but that's a different story.

I use a sed script for filtering the overstruck lines to a "final" line (deleting all of the overstruck part). In the example script2log.sed,

# $Id: script2log.sed,v 1.3 2015/02/04 23:50:12 tom Exp $
#
# Trim ordinary ANSI sequences, then OSC sequences, then backspace
# sequences, then trailing CR's and finally overstruck sections of
# lines.
#
# There are still several interesting cases which cannot be handled
# with a script of this sort.  For example:
#       CSI K (clear line)
#       cursor movement within the line
s/␛[[][<=>?]\{0,1\}[;0-9]*[@-~]//g
s/␛[]][^␛]*␇//g
s/␛[]][^␛]*␛\\//g
:loop
s/[^␈]␈\(.\)/\1/g
t loop
s/␍␍*$//g
s/^.*␍//g
s/␛[^[]//g

that is done with these two commands, which first removes any number of trailing carriage returns on a line, and then removes any section of the line including a carriage return. What is left is only the last copy of the line to be written (the one you want):

s/␍␍*$//g
s/^.*␍//g

(and yes, those are literal carriage return characters in the script).

When I capture output from yum (or dnf), I do not try piping to grep (that is guaranteed to produce poor results). Rather, I use script to capture the output, and post-process that using sed, e.g.,

script -c "yum upgrade"
sed -f script2log.sed typescript >upgrade.log