38

I'm writing shell scripts for my server, which is a shared hosting running FreeBSD. I also want to be able to test them locally, on my PC running Linux. Hence, I'm trying to write them in a portable way, but with sed I see no way to do that.

Part of my website uses generated static HTML files, and this sed line inserts correct DOCTYPE after each regeneration:

sed -i '1s/^/<!DOCTYPE html> \n/' ${file_name.html}

It works with GNU sed on Linux, but FreeBSD sed expects the first argument after -i option to be extension for backup copy. This is how it would look like:

sed -i '' '1s/^/<!DOCTYPE html> \n/' ${file_name.html}

However, GNU sed in turn expects the expression to follow immediately after -i. (It also requires fixes with newline handling, but that's already answered in here)

Of course I can include this change in my server copy of the script, but that would mess i.e. my use of VCS for versioning. Is there a way to achieve this with sed in a fully portable way?

  • The two sed snippets you provided are identical, are you sure there isn't a typo? Also, i am able to execute GNU sed supplying the backup extension right after -i – iruvar Sep 29 '13 at 23:34
  • Duh, thanks for spotting this. I've fixed my question. The second line results in error in my sed, it expects '1s/^/<!DOCTYPE html> \n/' to be a file and complains it can't find it. – Red Sep 29 '13 at 23:37
  • Cross reference: sed in-place flag that works both on Mac (BSD) and Linux on Stack Overflow. – MvG Jan 10 '17 at 8:50
36

GNU sed accepts an optional extension after -i. The extension must be in the same argument with no intervening space. This syntax also works on BSD sed.

sed -i.bak -e '…' SOMEFILE

Note that on BSD, -i also changes the behavior when there are multiple input files: they are processed independently (so e.g. $ matches the last line of each file). Also this won't work on BusyBox.

If you don't want to use backup files, you could check which version of sed is available.

case $(sed --help 2>&1) in
  *GNU*) set sed -i;;
  *) set sed -i '';;
esac
"$@" -e '…' "$file"

Or alternatively, to avoid clobbering the positional parameters, define a function.

case $(sed --help 2>&1) in
  *GNU*) sed_i () { sed -i "$@"; };;
  *) sed_i () { sed -i '' "$@"; };;
esac
sed_i -e '…' "$file"

If you don't want to bother, use Perl.

perl -i -pe '…' "$file"

If you want to write a portable script, don't use -i — it isn't in POSIX. Do manually what sed does under the hood — it's only one more line of code.

sed -e '…' "$file" >"$file.new"
mv -- "$file.new" "$file"
  • 2
    GNU sed -i also implies -s. And the easiest way to check for a GNU sed is with the sed v command which is a valid noop for GNU but fails everywhere else. – mikeserv Sep 4 '14 at 19:11
  • Inspired by the above tips, here's an single-line (if ugly) portable version for those who really want one, though it does spawn a subshell: sed -i$(sed v < /dev/null 2> /dev/null || echo -n " ''") -e '...' "$file" If it's not GNU sed, it inserts a space followed by two singe quotes after -i so that it works on BSD. GNU sed gets only -i. – Ivan X Apr 26 '15 at 21:42
  • 2
    @IvanX I'm wary of using the presence of the v command to test for GNU sed. What if FreeBSD decided to implement it? – Gilles Apr 26 '15 at 21:54
  • @Gilles Fair point, but the man page for GNU sed describes v as being exactly for that purpose (detecting that it's GNU sed and not something else), so one would hope that *BSD would honor that. I can't think of another test, offhand, that takes no action on GNU sed, while causing an error on BSD sed (or vice versa), other than using -i itself, but that would require creating a dummy file first. Your test for sed above is OK but unwieldy for inline. Avoiding -i entirely, as you suggest, certainly seems like the safest bet, but I'm ok with using sed v given that's its purpose for existing. – Ivan X Apr 26 '15 at 22:20
  • 1
    @lapo An alternative is to define a function, see my edit. – Gilles Mar 22 '16 at 12:04
10

If you don't find a trick to make sed play nice, you could try:

  1. Don't use -i :

    sed '1s/^/<!DOCTYPE html> \n/' "${file_name.html}" > "${file_name.html}.tmp" &&
      mv "${file_name.html}.tmp" "${file_name.html}"
    
  2. Use Perl

    perl -i -pe 'print "<!DOCTYPE html> \n" if $.==1;' "${file_name.html}"
    
8

ed

You can always use ed to prepend a line to an existing file.

$ printf '0a\n<!DOCTYPE html>\n.\nw\n' | ed my.html

Details

The bits around the <!DOCTYPE html> are commands to ed instructing it to add that line to the file my.html.

sed

I believe this command in sed can also be used:

$ sed -i '1i<!DOCTYPE html>\n` testfile.csv
  • I eventually resorted to Perl, but using ed is a good alternative that's not popular among Unix-like users as it should be. – Red Sep 30 '13 at 14:29
  • @Red - glad to hear you resolved you issue. Yeah I'd not seen that one before, googling turned it up and it actually seemed like the most portable, apt way to do this. – slm Sep 30 '13 at 14:30
7

You can also do manually what perl -i does under the hood:

{ rm -f file && { echo '<!DOCTYPE html>'; cat; } > file;} < file

Like perl -i, there's no backup, and like most solutions given here, beware it may affect the permissions, ownership of the file and may turn a symlink into a regular file.

With:

sed '1i\
<!DOCTYPE html>' file 1<> file

sed would overwrite the file over itself, so would not affect ownership and permissions or symlinks. It works with GNU sed because sed will typically have read a buffer full of data from file (4k in my case) before overwriting it with the i command. That wouldn't work if the file was more than 4k except for the fact that sed also buffers its output.

Basically sed works on blocks of 4k for reading and writing. If the line to insert is smaller than 4k, sed will never overwrite a block it has not read yet.

I wouldn't count on it though.

  • Should be echo '<!DOCTYPE html>' or escaped without "" quotes. – A.D. May 22 '15 at 12:36
  • @A.D. Good point. I tend to forget about that bug^Wfeature of interactive bash/zsh as I generally disable it for myself. – Stéphane Chazelas May 22 '15 at 12:41
  • This is one of the very few answers here that doesn't have a wide open security hole of redirecting to a "temp file" with a static, predictable name without checking if it already exists. I believe this should be the accepted answer. Very nice demonstration of the use of group commands, also. – Wildcard Mar 11 '16 at 1:59
3

FreeBSD sed, which is used on Mac OS X as well, needs the -e option after the -i switch to define & recognise the following (regex) command correctly & unambiguously.

In other words, sed -i -e ... should work with both FreeBSD & GNU sed.

More generally, omitting the backup extension after FreeBSD sed -i requires some explicit sed option or switch following the -i to avoid confusion on part of FreeBSD sed while parsing its command-line arguments.

(Note, however, that sed in-place file edits lead to file inode changes, see "In-place" editing of files).

(As a general hint, recent versions of FreeBSD sed have the -r switch to increase compatibility with GNU sed).

echo a > testfile.txt
ls -li testfile.txt
#gsed -i -e 's/a/A/' testfile.txt
#bsdsed -i 's/a/A/' testfile.txt  # does not work
bsdsed -i -e 's/a/A/' testfile.txt
ls -li testfile.txt
cat testfile.txt
  • 5
    No, bsdsed -i -e 's/a/A/' is not in-place editing, it's editing with saving the original with a "-e" suffix (testfile.txt-e). – Stéphane Chazelas May 22 '15 at 11:27
  • note that GNU sed also supports -E (in addition to -r) for compatibility with FreeBSD. -E is likely to be specified in the next POSIX version, so we should all be forgetting about that -r non-sense and pretend it never existed. – Stéphane Chazelas May 22 '15 at 11:30
2

You can use Vim in Ex mode:

ex -sc '1i|<!DOCTYPE html>' -cx file
  1. 1 select first line

  2. i insert text and newline

  3. x save and close

Or even, as in the comments, plain old standard ex:

printf '%s\n' 1i '<!DOCTYPE html>' . x | ex file 
  • There's nothing Vim-specific here. This is fully compliant with POSIX specifications for ex, except that implementations are not required to support multiple -c flags. For definite portability I would use printf '%s\n' 1i '<!DOCTYPE html>' . x | ex file – Wildcard Oct 16 '16 at 8:31
1

To emulate sed -i for a single file portably while avoiding race conditions as much as possible:

sed 'script' <<FILE >file
$(cat file)
FILE

By the way, this also handles the possible problem that sed -i introduces in that, depending on directory and file permissions, sed -i might enable a user to overwrite a file which that user does not have permissions to edit.

You might also do backups like:

sed 'script' <<FILE >file
$(tee file.old <file)
FILE

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.