7

I want to change the first line of hundreds of files recursively in the most efficient way possible. An example of what I want to do is to change #!/bin/bash to #!/bin/sh, so I came up with this command:

find ./* -type f -exec sed -i '1s/^#!\/bin\/bash/#!\/bin\/sh/' {} \;

But, to my understanding, doing it this way sed has to read the whole file and replace the original. Is there a more efficient way to do this?

4
  • 4
    does it matter? How large are your files? Unless they're absolutely enormous, chances are that the sed command you have will deal with them faster than it takes to find and implement another solution. (And if you do have enormous shell scripts... well, it might be a good point to reconsider their structure.)
    – ilkkachu
    Commented Aug 13, 2020 at 20:00
  • you are right, I'm just curious since i'm new to this
    – Akuseru
    Commented Aug 13, 2020 at 20:05
  • 1
    ah, in that case...
    – ilkkachu
    Commented Aug 13, 2020 at 20:07
  • Maybe head avoids reading the whole file?
    – Harv
    Commented Aug 21, 2020 at 19:39

5 Answers 5

19

Yes, sed -i reads and rewrites the file in full, and since the line length changes, it has to, as it moves the positions of all other lines.

...but in this case, the line length doesn't actually need to change. We can replace the hashbang line with #!/bin/sh␣␣ instead, with two trailing spaces. The OS will remove those when parsing the hashbang line. (Alternatively, use two newlines, or a newline + hash sign, both of which create extra lines the shell will eventually ignore.)

All we need to do is to open the file for writing from the start, without truncating it. The usual redirections > and >> can't do that, but in Bash, the read-write redirection <> seems to work:

echo '#!/bin/sh  ' 1<> foo.sh

or using dd (these should be standard POSIX options):

echo '#!/bin/sh  ' | dd of=foo.sh conv=notrunc

Note that strictly speaking, both of those rewrite the newline at the end of the line too, but it doesn't matter.

Of course, the above overwrites the start of the given file unconditionally. Adding a check that the original file has the correct hashbang is left as an exercise... Regardless, I probably wouldn't do this in production, and obviously, this won't work if you need to change the line to a longer one.

5
  • 3
    In general, it would be good to extract the existing data up to the first newline, and make a new version of that (of the same length) to be overwritten. It is fine to insert two spaces before the shell, as in #! /bin/sh, to preserve any following options. In fact, with enough knowledge of the format, you could find enough non-significant space to insert some text and still update in situ. The system will read/alter/write at least one whole block anyway, which gives plenty of scope. Commented Aug 13, 2020 at 23:07
  • 1
    Even better would be echo '#!/bin/sh -' to fix those shebangs where the - is missing. (see Why the "-" in the "#! /bin/sh -" shebang?) Commented Aug 14, 2020 at 7:52
  • 1
    But, please, honestly, there's "could" and "should". Is it worth jumping through the hoops and testing this to make sure the spaces are right, etc. etc. than simply doing it brute force with sed. This requires a bunch of testing. It has ramifications if it goes wrong. It relies on specific white space (which historically has not be the best of ideas). When the testing takes longer than the brute force process, it's worth pausing to reflect on it all. Commented Aug 14, 2020 at 15:56
  • @WillHartung, yes, and that's why I started by asking under the question if it (the most efficient way) matters... and why I said above I wouldn't probably do this in production, etc. etc.
    – ilkkachu
    Commented Aug 14, 2020 at 16:06
  • Regarding the newline, one could use printf instead of echo, or add -n to echo, but like you said, none of this really matters.
    – JoL
    Commented Aug 14, 2020 at 16:10
6

An optimization would be to use {} + instead of {} \;.

find . -type f -exec sed -i '1s|^#!/bin/bash|#!/bin/sh|' {} +

Instead of invoking one sed process for each found file, you provide the files as arguments to a single sed process.

POSIX specification of find on {} + (my bold):

If the primary expression is punctuated by a <plus-sign>, the primary shall always evaluate as true, and the pathnames for which the primary is evaluated shall be aggregated into sets. The utility utility_name shall be invoked once for each set of aggregated pathnames.

2
  • 1
    i found a post that explains the difference of the 2 in more detail, in case anyone needs it stackoverflow.com/questions/6085156/…
    – Akuseru
    Commented Aug 13, 2020 at 20:43
  • 1
    I actually like this answer better than ikkachu's, but both are useful. This one, though, taught me something about find that I missed when I learned the command 26 years ago. Thanks!
    – ichabod
    Commented Aug 14, 2020 at 16:14
5

I'd do:

#! /bin/zsh -
LC_ALL=C # work with bytes instead of characters.
shebang_to_replace=$'#!/bin/bash\n'
       new_shebang=$'#!/bin/sh -\n'

length=$#shebang_to_replace

ret=0
for file in **/*(N.L+$((length - 1)));do
  if
    read -u0 -k $length shebang < $file &&
      [[ $shebang = $shebang_to_replace ]]
  then
    print -rn -- $new_shebang 1<> $file || ret=$?
  fi
done
exit $ret

Like @ilkkachu's approach, the file is overwritten in place with a string which is exactly the same size. The differences are:

  • we ignore hidden files and files in hidden dirs (think .git one for instance) as it's unlikely you want to consider those (you used find ./* which would have skipped the hidden files and dirs of the current directory, but not those of subdirs). Add the D glob qualifier if you do want them.
  • we don't bother looking into files that are not big enough to hold the original shebang to replace (we use . as the equivalent of -type f, so we're already retrieving the inode information from the file, so we might as well check the size there).
  • we're actually checking that the file starts with the right shebang to replace, reading as few bytes as necessary (here it has to be zsh as other shells can't deal with arbitrary byte values).
  • we're using #!/bin/sh - as replacement which is the correct shebang for /bin/sh scripts (#!/bin/bash - would be the correct shebang for /bin/bash scripts by the way). See Why the "-" in the "#! /bin/sh -" shebang? for details.

Errors in overwriting files are reported in the exit status, but not errors in traversing the directory tree, nor errors in reading the files, though that could be added.

In anycase, it only replaces the shebangs that are exactly #!/bin/bash, not other shebangs that use bash as interpreter like #! /bin/bash, #! /bin/bash -Oextglob, #! /usr/bin/env bash, #! /bin/bash -efu. For those, you'd need to decide what to do. -efu are sh options but -Oextglob has no sh equivalent for instance.

You could extend it to support the easiest cases like:

#! /bin/zsh -
LC_ALL=C # work with bytes instead of characters.
zmodload zsh/system || exit

minlength=11 # length of "#!/bin/bash"
maxlength=1024 # arbitrary here.

ret=0
for file in **/*(N.L+$minlength);do
  if
    sysread -s $maxlength buf < $file &&
      [[ $buf =~ $'(^#![\t ]*((/usr)?/bin/env[ \t]+bash|/bin/bash)([ \t]+-([aCefux]*))?[ \t]*)\n' ]]
  then
    shebang=$match[1] newshebang="#!/bin/sh -$match[5]"
    print -r -- ${(r[$#shebang])newshebang} 1<> $file || ret=$?
  fi
done
exit $ret

Here allowing a number of different shebangs with a number of supported options which are reproduced in the new /bin/sh shebang, right-padded (with the r[length] parameter expansion flag) to the same size as the original.

2

Hypothetically, if you need to make such a change on an enormous file, sed -i will in fact create a temporary file of the same size which might be a problem.

This problem can be solved with some trickery by using a tool like dd (or using mmap() or fseek() in C) to modify chunks in the file one at a time. For instance if you want to add 2 characters at the start of the file, you can work backwards from the end, shifting everything forward by 2 characters.

The lesson here is that in UNIX, files are often treated as "streams" that flow through a program, but they can also be treated as byte arrays which can be edited in-place.

2

Files are one long contiguous range of bytes. Your replacing of bash with sh will essentially need to remove the two bytes (assuming UTF-8 or similar) that make up ba. Files can't have holes in them, so everything starting from sh will have to be written two bytes earlier into the file.

This requires a rewrite of the entire file, or at least starting from the changed part.

There are ways to replace bytes in a file, for example with innocent spaces if the format allows that, without having to rewrite the entire file, see the accepted answer.

1
  • 2
    Some file systems allow or may allow collapsing a range of bytes in a file, but that's generally limited to full filesystem blocks so wouldn't apply for those 2 bytes here. See fallocate -c on Linux for instance. Commented Aug 14, 2020 at 12:41

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .