1

It's a small question about Bash and Linux. I have a file with special chars like \n, \r and so on. I'm trying to print the file without using those special chars. For example if the file content is hi \t hello then it will print it as is (and not hi hello). Using cat file does not do the work. Tried to look into the man of cat and I saw that I can use the -A option. It prints: abc^Mabc. But what is ^M? How can I make it print as of form \<char>?

6
  • think you are getting this wrong. there are no "special character". the character named carriage return is a character that instructs the printer to go back to the beginning of the line. it is typically written as \r, but that is just a convention to type a character that you cannot type otherwise.
    – umläute
    Nov 17 '20 at 22:11
  • also, if you don't want carriage-returns/line-feeds, how are you planning to print more than one line?
    – umläute
    Nov 17 '20 at 22:12
  • 1
    See man ascii. ^M means Ctrl-M: effectively, you look for M as a printable in the right-hand column, and follow that row to the left for the corresponding control character, which is CR '\r'. You might look at od -An -t c which does 16 chars per line. Nov 17 '20 at 22:15
  • I mention that there are 256 possible 8-bit characters, of which 93 are printable and 9 more have a well-known \r-type representation. There are 26 further control characters with names like DEL, ENQ and DC4, and 128 that are not ASCII at all. cat -vet or od at least show a consistent and defined view. Nov 17 '20 at 23:15
  • Everything makes a whole lot more sense after export LC_ALL=C, because UTF-8 treats chars >= 128 as multi-byte chars. Nov 17 '20 at 23:23
1

If your file has the characters literally escaped, say, if "hi\thello\n" has the inverted slash and the 't' letter indicating there should be a tab character, and the slash and 'n' where should be a newline, then you have to interpret this notation and convert them into the real characters.

[Previous solution using echo didn't work, as pointed by Paul Pedant]

After some testing, arrived at this simple method of unquoting literally escaped characters in Bash:

x="$(cat filename)"
echo -en "$x"

To verify its operation, I created this file (with cat > filename then typing the lines, then Ctrl-D Enter after five spaces in the last line and other Ctrl-D Enter to close the file):

    This line starts with 4 leading spaces, a tab here:\t. Has a real newline (RN) here:
Newline code here:\n, six spaces and a dot:      . RN:
Return code here:\r, because the return code the first three words are hidden in the terminal.RN:
  This line starts with two spaces, then has four and a tab here:    \tRN:
This final line ends with five spaces without an ending newline:     

Running the above command on this file shows some strange output in my terminal, as the return code causes overwriting of part of a line and doesn't show the real width of the tabs.

To check the actual characters in the output, passed it to the octal dumper od:

x="$(cat filename)"; echo -en "$x" | od -cx

where -c shows non printable as escaped chars and -x shows their hexadecimal code (TAB=09, RETURN=0D, NEWLINE=0A, SPACE=20, etc.), resulting in:

0000000                   T   h   i   s       l   i   n   e       s   t
           2020    2020    6854    7369    6c20    6e69    2065    7473
0000020   a   r   t   s       w   i   t   h       4       l   e   a   d
           7261    7374    7720    7469    2068    2034    656c    6461
0000040   i   n   g       s   p   a   c   e   s   ,       a       t   a
           6e69    2067    7073    6361    7365    202c    2061    6174
0000060   b       h   e   r   e   :  \t   .       H   a   s       a
           2062    6568    6572    093a    202e    6148    2073    2061
0000100   r   e   a   l       n   e   w   l   i   n   e       (   R   N
           6572    6c61    6e20    7765    696c    656e    2820    4e52
0000120   )       h   e   r   e   :  \n   N   e   w   l   i   n   e
           2029    6568    6572    0a3a    654e    6c77    6e69    2065
0000140   c   o   d   e       h   e   r   e   :  \n   ,       s   i   x
           6f63    6564    6820    7265    3a65    2c0a    7320    7869
0000160       s   p   a   c   e   s       a   n   d       a       d   o
           7320    6170    6563    2073    6e61    2064    2061    6f64
0000200   t   :                           .       R   N   :  \n   R   e
           3a74    2020    2020    2020    202e    4e52    0a3a    6552
0000220   t   u   r   n       c   o   d   e       h   e   r   e   :  \r
           7574    6e72    6320    646f    2065    6568    6572    0d3a
0000240   ,       b   e   c   a   u   s   e       t   h   e       r   e
           202c    6562    6163    7375    2065    6874    2065    6572
0000260   t   u   r   n       c   o   d   e       t   h   e       f   i
           7574    6e72    6320    646f    2065    6874    2065    6966
0000300   r   s   t       t   h   r   e   e       w   o   r   d   s
           7372    2074    6874    6572    2065    6f77    6472    2073
0000320   a   r   e       h   i   d   d   e   n       i   n       t   h
           7261    2065    6968    6464    6e65    6920    206e    6874
0000340   e       t   e   r   m   i   n   a   l   .   R   N   :  \n
           2065    6574    6d72    6e69    6c61    522e    3a4e    200a
0000360       T   h   i   s       l   i   n   e       s   t   a   r   t
           5420    6968    2073    696c    656e    7320    6174    7472
0000400   s       w   i   t   h       t   w   o       s   p   a   c   e
           2073    6977    6874    7420    6f77    7320    6170    6563
0000420   s   ,       t   h   e   n       h   a   s       f   o   u   r
           2c73    7420    6568    206e    6168    2073    6f66    7275
0000440       a   n   d       a       t   a   b       h   e   r   e   :
           6120    646e    6120    7420    6261    6820    7265    3a65
0000460                  \t   R   N   :  \n   T   h   i   s       f   i
           2020    2020    5209    3a4e    540a    6968    2073    6966
0000500   n   a   l       l   i   n   e       e   n   d   s       w   i
           616e    206c    696c    656e    6520    646e    2073    6977
0000520   t   h       f   i   v   e       s   p   a   c   e   s       w
           6874    6620    7669    2065    7073    6361    7365    7720
0000540   i   t   h   o   u   t       a   n       e   n   d   i   n   g
           7469    6f68    7475    6120    206e    6e65    6964    676e
0000560       n   e   w   l   i   n   e   :
           6e20    7765    696c    656e    203a    2020    2020
0000576

Nothing is lost and the codes are converted to the single desired chars.

PROBLEM: This simple operation can only convert small files, because Bash variables have a maximum length of 32k or so. To extend this, we have to make a loop reading convenient sized chunks of the file and converting them. For this partial block reading the utility of choice is dd. This is a full converter script using this method, with blocks of 16k (16384):

#!/bin/bash
# unescape.sh: converts escaped chars (\t) to actual chars (TAB)
# 2020.11.19 Fjor
#-------------------------------
[ -z "$2" ] \
    && echo "Use: $0 inputfile outputfile  to unescape chars (\t -> TAB)" \
    && exit
IFILE="$1"
OFILE="$2"

#-- exists input file?
[ ! -r "$IFILE" ] \
    && echo "$0: Can't read inputfile $IFILE" \
    && exit

#-- don't destroy existing output file
[ -f "$OFILE" ] \
    && read -p "$0: Output file $OFILE exists, overwrite(s/n)? " \
    && [ x"$REPLY" != xs ] \
    && exit \
    || rm -f $OFILE

let START=0
let STEP=16384
let SIZE="$(stat -c %s $IFILE)"
let NSTEPS=SIZE/STEP+1

echo -n "Converting..."
for ((n=0 ; n<$NSTEPS ; n++)) ; do
    echo -ne "\r$START bytes converted..."
    #-- ibs input block size, skip N blocks, copy count blocks
    x="$(dd if=$IFILE ibs=$STEP skip=$n count=1 2>/dev/null)"
    echo -ne "$x" >> $OFILE
    let START+=STEP
done 
echo -e "\rConversion complete ($SIZE bytes)." 

#-- end --#

Let's save as unescape.sh, chmod u+x unescape.sh and call it:

./unescape filename.txt outresult.txt

If the file has NULL characters, will appear warnings and the NULLs will be ommited.

4
  • Not convinced this is sufficiently robust. read and read -r remove newlines, and leading and training space, so echo never even sees those. read also returns bad status if there is no trailing newline, so a final unterminated line is discarded. And echo -e does not convert CR to a visible \r, it converts a visible \r to an actual CR. Nov 18 '20 at 15:10
  • You're right; my simple text processing needs weren't so picky, and the question asked for a Bash solution for converting escaped chars to chars, like \t to TAB. Of course it will be better to use a more powerful language, like Python, as shown in this answer: Process escape sequences in Python.
    – Fjor
    Nov 19 '20 at 2:58
  • Deleted previous solution that have errors, as Paul Pedant said. The new is better!
    – Fjor
    Nov 19 '20 at 3:58
  • Included file converter script in the modified (and working!) solution.
    – Fjor
    Nov 27 '20 at 2:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.