4

The script below currently removes the ^M character (Ctrl+V+M). I feel it's a bit long winded but I also need to add ^I and any other characters I might see in the future.

Is there an easier way to add ^I (Ctrl+V+I)? This is the first script I wrote for myself about 6 months ago after attending a 2 day shell programming class. I'm not sure if I made it longer than it needs to be, so any general tips would also be appreciated.

#!/bin/bash  

    echo "$# item(s) to review."
    question='Do you want to remove the ^M characters?'

    for file
    do
            if grep "^M" "$file" >> /dev/null 2> /dev/null
            then
                    echo "$file contains special characters"
                    echo $question
                    read answer
                            if    [[ "$answer" == [yY] ]]
                            then
                                    cat "$file" | sed "s/^M//" > "$file.safe"
                                    echo "Special characters have been removed and $file.safe has been created."
                            elif  [[ "$answer" == [yY][eE][sSaA]* ]]
                            then
                                    cat "$file" | sed "s/^M//" > "$file.safe"
                                    echo "Special characters have been removed and $file.safe has been created."
                            else
                                    echo "Special characters have NOT been removed."
                            fi
            elif [[ -d $file ]]
            then
                    echo "$file is a directory"
            else
                    echo "No special characters in $file"
            fi
    done
1
  • The tr command may be of interest.
    – thrig
    Nov 28, 2017 at 21:44

4 Answers 4

4

This is certainly much, much longer than it needs to be. All you need is the tr utility, plus a loop and redirections to act on the files that are passed as arguments to the script.

#!/bin/sh
for file do
  tr -d '\r\t' <"$file" >"$file.safe"
done

With the option -d, tr removes the specified characters. The characters to remove are passed together as the first non-option argument. You can use backslash escapes to represent special characters: \n for a newline (^J), \r for a carriage return (^M), \t for a tab (^I), etc.

I haven't reproduced the code for asking the user because it's pointless. Directories will cause an error with redirection anyway, and it's really the job of the caller not to request a nonsensical action such as treating a directory as a regular file, so I also skipped that part.

If you want to replace the original file, write to a temporary file then move the result in place.

#!/bin/sh
for file do
  tmp="$(TMPDIR=$(dirname -- "$file") mktemp)"
  tr -d '\r\t' <"$file" >"$tmp" && mv -f -- "$tmp" "$file"
done

The temporary file name is constructed using mktemp so that the script is robust. It will work as long as you have write permission to the directory containing the file, without risking overwriting an existing file. It's secure even if that directory is writable by other users who might try to inject other data (a potential problem in /tmp).

The mv command is only invoked if the call to tr succeeded, so there's no risk of losing data if tr fails, e.g. because the disk becomes full midway through.

If you want to avoid replacing the file by a new, identical file if it doesn't contain any special characters, there are two ways:

  • You can check for the special characters first. There are several ways to do it. One way is to remove everything except those special characters and count the number of resulting characters. As an optimization, pipe through head -c 1 so that you don't need to go through the whole file if a special character is found close to the top: that way the count is 0 if there's nothing to do and 1 otherwise.

    if [ "$(tr -dc '\r\t' <"$file" | head -c 1 | wc -c)" -ne 0 ]; then
      tr -d '\r\t' <"$file" >"$tmp" && mv -f -- "$tmp" "$file"
    fi
    
  • You can do the transformation, then check if it's identical to the original. This can be slower if the files are often already in the desired state. On the other hand, this technique generalizes to cases where it isn't easy to determine whether the file is in the desired state.

    tr -d '\r\t' <"$file" >"$tmp" &&
    if cmp -s "$tmp" "$file"; then
      rm -- "$tmp"
    else
      mv -f -- "$tmp" "$file"
    fi
    
1
  • 1
    The 2 reasons for asking are 1. I wanted to see if I could do it, and 2. When I use * as $1 to check all files within the directory and certain files, like this script, needs to keep the special characters. Also, I'm the only one who uses the script, and I use it to clean up files before I insert them into autosys or if I have copied them from a Windows environment.
    – Emile
    Nov 28, 2017 at 22:40
1

You can put a loop around your script. So:

 for c in "^I" "^M"; do
    for file; do
       if grep "$c" "$file"; then
          ...
          etc.
          ...
       fi
    done
 done
1

I prefer this perl one liner. The '\cM' is the control-M character. The original file(s) will be backed up with the extension '.bak' This extension can be your choice.

perl -i.bak -pe 's/\cM//g;'  file(s)

Example using a class of characters to remove. In the brackets perl will find control-I and control-M and remove them. I've not tested this exactly though.

perl -i.bak -pe 's/[\cM\cI]//g;' files(s)
0

Have you thought of using

 tr -d .....<characterlist>....

For example, get rid of any non-printable characters and put into another file:

 cat filename | tr -cd '[:print:]' >/tmp/x.out

Modify the characterlist to suit your application....see the tr man apage for more information.

Also it is nice because regex ranges are allowed:

 echo '\001\002\003\004' | tr -d '[\001-\003]' | od -c

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .