Remove leading whitespace before possible shebang

Question

I have a large collection of non-binary files in one place. Some of them have shebangs and of those some have (for some inexplicably reasons) whitespaces in front of the shebangs. This includes empty lines and lines with only whitespaces!

Example 1:

    #!/usr/bin/env foo bar

Example 2:

   #!/usr/bin/env foo bar

Example 3:

#! /bin/sh -e

Example 4:

    ______          
   / ____/___  ____ 
  / /_  / __ \/ __ \
 / __/ / /_/ / /_/ /
/_/    \____/\____/ 


This is Foo News #324 with the tip of the day:
Don't forget to put #!/bin/sh on top of your shell script files!

I would love for a solution for GNU-based (Linux) systems which would remove the leading white spaces of the file for example 1 and 2, while leaving 3 and especially 4 alone (even if it includes something like a shebang inside it).

Example 1 and 2 would become:

#!/usr/bin/env foo bar

What I unsuccessfully tried so far:

• As a first step trying to discern between examples 1-3 and 4:

  grep -Pzo '^[ \t\n]+#! ?[ \w/.-]+'
  

  Did not work because grep: unescaped ^ or $ not supported with -Pz.

• Using awk:

  awk 'BEGIN {ws_check=1} !/[ \t]+/ {ws_check=0}  /#! ?[ \w/.-]+/,0 && ws_check { print }'
  

  Would still a lot of work in order to detect example 4 but also to only print the parts of the left-trimmed line with the shebang but not trimming the rest.

Answer 1

I would use perl to slurp the file into memory and remove any leading whitespace if and only if the first non-whitespace character in the file is a shebang:

perl -i.bak -0pe 's/^\s+(?=#!)//' file

Or, for many files:

for f in ./*; do perl -i.bak -0pe 's/^\s+(?=#!)//' "$f"; done

The (?=#!) is a positive lookahead, so the substitution operator will only remove whitespace (including newlines and tabs) from the start of the file that are followed by a #!. The -i.bak ensures you keep backups of all modified files, just in case. If you're sure it works as expected, you can rm *.bak.

The perl options used here are:

• -0: This specifies the input record separator ($/) as an octal or hexadecimal number. Using an -0 by itself makes perl slurp the file and basically treat it as a single line. *-i.bak : edit the file inplace, and create a backup of the original with the .bak extension.
• -p : process an input file line-by-line and print each line after applying the script given by -e.
• -e : pass a script to be executed as a command line parameter.

Answer 2

perl -i -p -e 'if ($. == 1) {s/^\s+#!/#!/}' *

This will remove leading white space before #! ONLY on the first line ($. == 1) of every file. all other lines are passed through unmodified. perl will update the files whether anything has changed or not (i.e. they'll have a new inode and timestamps will be updated). See man perlrun and search for the second occurrence of -i\[ for details)

If you only want to modify (change the timestamp, inode etc) files which have the faulty whitespace-before-#!, try something like this:

awk '/^[[:blank:]]+#!/ && FNR==1 { printf "%s\0", FILENAME }; {nextfile}' * |
    xargs -0r perl -i -p -e 'if ($. == 1) {s/^\s+#!/#!/}'

awk outputs a list of matching files (first line has leading whitespace before #!), delimited by NULs. This is fed into xargs -0r to run the perl one-liner on them.

The nextfile function requires GNU awk. It can be omitted in other versions of awk but will run slower (as it has to read in every line of every file, rather than skip to the next file after examining the first line).

This could have been done entirely in perl but that would have required a lot more code than just piping awk's output into xargs perl