16

A non-root linux user owns a text file that's located in /etc, where the non-root linux user does not have permission to create files. The non-root linux user can edit the file manually via vi without issue. Programmatic and manual attempts to replace a text string within the file using sed -i, perl -i, and other in-directory temp file methods are failing due to permissions issues when the non-root linux user runs the text-replacement script. I've found a solution in the Perl Cookbook (noted farther down) but it has concerning warnings, plus it looks a bit beyond me at present. Can anyone suggest a simpler option?

Server Info:

  • cat /etc/os-release: Oracle Linux Server 8.9

  • uname -a: Linux server01.domain.com 5.4.17-2136.322.6.4.el8uek.x86_64

  • perl: (v5.26.3) built for x86_64-linux-thread-multi

  • sed: (GNU sed) 4.5

  • awk: GNU Awk 4.2.1, API: 2.0 (GNU MPFR 3.1.6-p2, GNU MP 6.1.2)

  • vi: VIM - Vi IMproved 8.0 (2016 Sep 12, compiled Aug 5 2022 07:42:15)

  • Location of file to edit: /etc

  • Permissions on /etc:

    drwxr-xr-x  130 root   root        12288 Jun 21 11:50 etc
    
  • Text file to edit: /etc/targetfile

    -rw-rw-r-- 1 justauser group1 1864 Jun 19 10:52 targetfile
    
  • File length of /etc/targetfile can be as large as 200 lines that are about 50 characters each.

Example/abbreviated /etc/targetfile contents:

f1112:/dir1/dir2/59.35:N            # Comment
f3332:/dir1/dir2/59.35:N            # Comment
f4442:/dir1/dir2/59.35:N            # Comment
f555:/dir1/dir2/59.35:N             # Comment
f666s2:/dir1/dir2/59.35:N           # Comment
f777s2:/dir1/dir2/59.35:N           # Comment

Goal: Programmatically change /dir1/dir2/59.35 to /dir1/dir2/59.77 in targetfile located in /etc, where the non-root user running the script is "justauser" who lacks permissions to create files in /etc.

Constraints:

  • Permissions on /etc can't change.
  • Existing server programs/utilities can't be upgraded.
  • All programs/commands must be called from within a bash shell script.
  • The bash shell script must be executed as "justauser", not as root.
  • Not using sudo is strongly preferred.

Miscellaneous:

While logged in as the linux user "justauser", command line editing /etc/targetfile using vi works fine. Commands like sed -i and perl -i fail since "justauser" lacks write perms in /etc to create temp files.

perl -i -p -e 's/f4442:\/dir1\/dir2\/59.35:N/f4442:\/dir1\/dir2\/59.77:N/g' /etc/targetfile 
--Can't remove /etc/targetfile: Permission denied, skipping file.

sed -i 's/f4442:\/dir1\/dir2\/59.35:N/f4442:\/dir1\/dir2\/59.77:N/' /etc/targetfile
--sed:  sed: couldn't open temporary file /etc/sedO2SLSF: Permission denied
Rough Intended Usage Example (static values replacing variables to come):
#!/bin/bash
...

function _editConfig {
echo "Editing targetfile..."
perl -i -p -e 's/f4442:\/dir1\/dir2\/59.35:N/f4442:\/dir1\/dir2\/59.77:N/g' /etc/targetfile
if [ $? -ne 0 ]
        then
                echo "Error on _editConfig function.\n Terminating program"
                EXITCODE=1
fi
}
...

Options seen:

Perl Cookbook 7.10 Modifying a File in Place Without a Temporary File:

open(F, "+< $infile")       or die "can't read $infile: $!";
$out = '';
while (<F>) {
    s/DATE/localtime/eg;
    $out .= $_;
}
seek(F, 0, 0)               or die "can't seek to start of $infile: $!";
print F $out                or die "can't print to $infile: $!";
truncate(F, tell(F))        or die "can't truncate $infile: $!";
close(F)                    or die "can't close $infile: $!";
...
"This approach is for the truly determined. 
It's harder to write, takes more memory (potentially a lot more),
doesn't keep a backup file, 
and may confuse other processes trying 
to read from the file you're updating. 
For most purposes, therefore, we suggest it's probably not worth it."

Final Comments:

Is there an in-place edit option that doesn't use an in-directory temp file that is simpler than the Perl Cookbook example?

The Perl Cookbook example is a bit hard to follow for a Perl novice, and it also requires some kind of modification/conversion... I'd think...to work in the body of the bash script.

4
  • "Existing server programs/utilities can't be upgraded." ... But do you have sponge from moreutils already installed by any chance?
    – muru
    Commented Jun 22 at 2:01
  • 6
    Not in a position to write a tested answer (feel free for someone to adapt this into an answer), but shouldn't something like vi /etc/filename -c s/xxx/yyy/ -c :wq work?
    – TripeHound
    Commented Jun 22 at 6:34
  • 2
    The cause of your problem is that sed -i and perl -i do not edit a file; instead, they replace the file with a new one, which the user cannot create in that directory. Commented Jun 24 at 14:10
  • Unfortunately, sponge is not installed. Commented Jun 24 at 23:02

5 Answers 5

19

Use ed rather than sed. This is normally an interactive editor, but you can feed commands to its standard input using a here-doc.

ed /etc/filename <<EOF
1,$s/old/new/g
w
EOF
2
  • 4
    Good old ed ! Nice one :) (great also for huuuge files) Commented Jun 22 at 15:08
  • I was going to say 'Use ex' (since vi is known to work), which will look very much like this.
    – j4nd3r53n
    Commented Jun 24 at 11:55
17

The problem with perl -i and sed -i is that they try to create a backup/temporary file in /etc, and $USER doesn't have write permission on /etc. You could:

cp /etc/myfile  $HOME/myfile
sed -i ... $HOME/myfile
cp $HOME/myfile /etc/myfile

This will leave the backup file in $HOME. Backup files (old configs) don't belong in /etc.

5
  • 3
    Basically true for reason and solution, but not exact: perl -i (without any extension following -i) will rename the original file to some temporary file, process it into the new file with the original name and then unlink the the (original) temporary file. So no actual "backup" file. Commented Jun 21 at 22:39
  • But it tries to create the "some temporary file" in /etc, see above explanation.
    – waltinator
    Commented Jun 21 at 22:59
  • 4
    Like I said, your answer and reasoning is basically right (write permission to /etc is needed since new files are created there), with the only exception that this is not a backup file but a temporary file which gets deleted afterwards. Commented Jun 21 at 23:04
  • 3
    Or to save one step sed [...] /etc/myfile > "$HOME"/myfile; cp "$HOME"/myfile /etc/myfile (with added quoting)
    – kos
    Commented Jun 22 at 2:30
  • 4
    would suggest instead sed [...] /etc/myfile > "$HOME"/myfile && cp "$HOME"/myfile /etc/myfile so that it only replaces original if there are no errors.
    – user10489
    Commented Jun 22 at 10:45
11

This is not the safest way, as an error in the sed script will result in the truncation of /etc/targetfile.

To get around the inability to write temporary files to /etc, of course the naive way would be to try and sed [...] /etc/targetfile >/etc/targetfile, which would truncate /etc/targetfile before /etc/targetfile has had a chance to be read by sed; one way to get around this would be to use sponge, which won't overwrite the target file until STDIN has been fully read, effectively preventing the truncation of the source file before it has been fully processed, at the same time allowing to go around the limitation imposed by the permissions on /etc.

Per its man page:

sponge reads standard input and writes it out to the specified file. Unlike a shell redirect, sponge soaks up all its input before opening the output file. This allows constricting pipelines that read from and write to the same file.

sed [...] /etc/targetfile | sponge /etc/targetfile
2
  • 2
    And if there is a typo in the [...] part, sed will output nothing, and you thus will nothing into /etc/targetfile ... Commented Jun 22 at 15:07
  • 3
    @OlivierDulac Fair point, this is not the safest solution. I added a bold caveat at the start of the answer, thanks for pointing that out
    – kos
    Commented Jun 22 at 15:20
6

I'd just use awk and store everything in memory then write it out at the end (after awk has closed the input file), e.g. using any awk:

awk '
    { sub("^f4442:/dir1/dir2/59.35:N", "f4442:/dir1/dir2/59.77:N"); rec=rec $0 ORS }
    END { printf "%s", rec > FILENAME }
' file

With the posted sample input note the change in the 3rd line from the top:

$ cat file
f1112:/dir1/dir2/59.35:N            # Comment
f3332:/dir1/dir2/59.35:N            # Comment
f4442:/dir1/dir2/59.35:N            # Comment
f555:/dir1/dir2/59.35:N             # Comment
f666s2:/dir1/dir2/59.35:N           # Comment
f777s2:/dir1/dir2/59.35:N           # Comment

$ awk '
    { sub("^f4442:/dir1/dir2/59.35:N", "f4442:/dir1/dir2/59.77:N"); rec=rec $0 ORS }
    END { printf "%s", rec > FILENAME }
' file

$ cat file
f1112:/dir1/dir2/59.35:N            # Comment
f3332:/dir1/dir2/59.35:N            # Comment
f4442:/dir1/dir2/59.77:N            # Comment
f555:/dir1/dir2/59.35:N             # Comment
f666s2:/dir1/dir2/59.35:N           # Comment
f777s2:/dir1/dir2/59.35:N           # Comment

If the file is large, though, I'd suggest using a temp file as:

tmp=$(mktemp) &&
awk '{sub("^f4442:/dir1/dir2/59.35:N", "f4442:/dir1/dir2/59.77:N")} 1' file > "$tmp" &&
cat "$tmp" > file
rm -f "$tmp"

In that way you're again only changing the contents of file, not replacing it.

P.S. Personally, I wouldn't use a regexp search and replace for this as it introduces too many gotchas regarding what could be present in the input and/or search string, I'd use literal strings instead, e.g.:

awk -v old='f4442:/dir1/dir2/59.35:N' -v new='f4442:/dir1/dir2/59.77:N' '
    index($0,old) == 1 { $0 = new substr($0,length(old)+1) }
    { rec = rec $0 ORS }
    END { printf "%s", rec > FILENAME }
' file

That would still have to be tweaked if the directory path could contain backslashes but that's easy to handle if necessary, e.g. see ENVIRON[] at how-do-i-use-shell-variables-in-an-awk-script.

3
  • 1
    Good because you can do arbitrarily complex edits, deletions and insertions, based on row numbers, content, etc. For a large file, I would use a line array rather than keep extending a single variable rec. Commented Jun 22 at 15:19
  • 1
    @Paul_Pedant arrays have problems as input gets large too because they're allocated in exponentially larger chunks as the previous allocation is exceeded. I updated my answer with a suggestion of what to do if the input file is large.
    – Ed Morton
    Commented Jun 22 at 17:43
  • 2
    I normally test performance and volume against a million-line (125MB) text file from the Princeton cs algorithms (leipzig1M.txt) without issues on my 8GB Laptop. But it wouldn't be hard to switch between methods for Large and Huge data. Commented Jun 22 at 22:42
4

I did not mention this case in the Perl Cookbook because it is so rare, but because your replacement string is the same length exactly as your original string, it actually is possible to update the disk file in place — simply by resetting your seek pointer back to where it had been when that line had been read.

The following code works on your file, as tested on an Ubuntu system:

#!/usr/bin/env perl

use strict;
use warnings;

###my $filename   = "$ENV{HOME}/tmp/targetfile"; # for testing
my $filename   = "/etc/targetfile";
my $old_string = "f4442:/dir1/dir2/59.35:N";
my $new_string = "f4442:/dir1/dir2/59.77:N";

length($old_string) == length($new_string)
    || die "old and new string must be the same length";

open(my $fh, "+<", $filename) || die "can't open $filename for update: $!";
$fh->autoflush(1);

my $filepos = 0;

while (<$fh>) {
    next unless s/\Q$old_string\E/$new_string/g;
    warn "updating line of $filename at position $filepos"; # or comment out if you're confident :)
    seek($fh, $filepos, 0) || die "can't seek to pos $filepos in $filename: $!";
    print($fh $_)          || die "can't write new line to $filename as pos $filepos: $!";
}
continue {
    $filepos = tell($fh);
}

close($fh) || die "can't close $filename: $!";

The moral of the story is that, rather surprisingly, you really can use the same handle for both reading and writing, if you're very, very, very careful—and only if there is a seek on the stdio filehandle between changing I/O "directions" (going from read to write or vice versa), as there is here. That's needed to keep the stdio buffer in order, and while perl uses its own I/O buffering system, the same inherited POSIX rules apply.

This approach is especially appealing because it no longer takes memory proportionate to the size of the file the way the nasty "read it all into memory first" approach does. That means you can run this on a file with a hundred million lines if you care to and it will consume no more memory than it would for a tiny file.

It’s awfully courageous, though, given the lack of backup.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .