1

I have a large collection of non-binary files in one place. Some of them have shebangs and of those some have (for some inexplicably reasons) whitespaces in front of the shebangs. This includes empty lines and lines with only whitespaces!

Example 1:

    #!/usr/bin/env foo bar

Example 2:


   #!/usr/bin/env foo bar

Example 3:

#! /bin/sh -e

Example 4:


    ______          
   / ____/___  ____ 
  / /_  / __ \/ __ \
 / __/ / /_/ / /_/ /
/_/    \____/\____/ 


This is Foo News #324 with the tip of the day:
Don't forget to put #!/bin/sh on top of your shell script files!

I would love for a solution for GNU-based (Linux) systems which would remove the leading white spaces of the file for example 1 and 2, while leaving 3 and especially 4 alone (even if it includes something like a shebang inside it).

Example 1 and 2 would become:

#!/usr/bin/env foo bar

What I unsuccessfully tried so far:

  • As a first step trying to discern between examples 1-3 and 4:

    grep -Pzo '^[ \t\n]+#! ?[ \w/.-]+'
    

    Did not work because grep: unescaped ^ or $ not supported with -Pz.

  • Using awk:

    awk 'BEGIN {ws_check=1} !/[ \t]+/ {ws_check=0}  /#! ?[ \w/.-]+/,0 && ws_check { print }'
    

    Would still a lot of work in order to detect example 4 but also to only print the parts of the left-trimmed line with the shebang but not trimming the rest.

  • Is the blank line in example 2 intended? – cuonglm Jun 19 '16 at 16:47
  • While the question remains weird, since the shebang usually is the first line, you can try the following: sed '1s/^[ \t]*#//' filename – Valentin Bajrami Jun 19 '16 at 16:54
  • @val0x00ff that won't work for example2. – terdon Jun 19 '16 at 17:02
  • @terdon Why wouldn't it? The [ \t]* matches white spaces and tabs. I shouldn't have included the # there. So the sed '1s/^[ \t]*// should work as expected. – Valentin Bajrami Jun 19 '16 at 17:04
  • @val0x00ff there's an extra blank line that the OP also wants to remove. – terdon Jun 19 '16 at 17:05
4

I would use perl to slurp the file into memory and remove any leading whitespace if and only if the first non-whitespace character in the file is a shebang:

perl -i.bak -0pe 's/^\s+(?=#!)//' file

Or, for many files:

for f in ./*; do perl -i.bak -0pe 's/^\s+(?=#!)//' "$f"; done

The (?=#!) is a positive lookahead, so the substitution operator will only remove whitespace (including newlines and tabs) from the start of the file that are followed by a #!. The -i.bak ensures you keep backups of all modified files, just in case. If you're sure it works as expected, you can rm *.bak.

The perl options used here are:

  • -0: This specifies the input record separator ($/) as an octal or hexadecimal number. Using an -0 by itself makes perl slurp the file and basically treat it as a single line. *-i.bak : edit the file inplace, and create a backup of the original with the .bak extension.
  • -p : process an input file line-by-line and print each line after applying the script given by -e.
  • -e : pass a script to be executed as a command line parameter.
1
perl -i -p -e 'if ($. == 1) {s/^\s+#!/#!/}' *

This will remove leading white space before #! ONLY on the first line ($. == 1) of every file. all other lines are passed through unmodified. perl will update the files whether anything has changed or not (i.e. they'll have a new inode and timestamps will be updated). See man perlrun and search for the second occurrence of -i\[ for details)

If you only want to modify (change the timestamp, inode etc) files which have the faulty whitespace-before-#!, try something like this:

awk '/^[[:blank:]]+#!/ && FNR==1 { printf "%s\0", FILENAME }; {nextfile}' * |
    xargs -0r perl -i -p -e 'if ($. == 1) {s/^\s+#!/#!/}'

awk outputs a list of matching files (first line has leading whitespace before #!), delimited by NULs. This is fed into xargs -0r to run the perl one-liner on them.

The nextfile function requires GNU awk. It can be omitted in other versions of awk but will run slower (as it has to read in every line of every file, rather than skip to the next file after examining the first line).

This could have been done entirely in perl but that would have required a lot more code than just piping awk's output into xargs perl

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.