I have a file that looks like this:


I want to make a copy of the file, in which the first i in every line is replaced by an o. I then want to concatenate the edited file back to the original file (preferrably without needing to specify an output file).

So the output would look like this:


I know some one-liners that can do this. However, when using sed, there were problems with the encoding (the file contains some unusual characters). Using perl, I didn't have that issue, but was looking for a way to fit this into a perl script as "elegantly" as possible.

I'm using a Unix operating system.

  • Unix in itself doesn't mean anything. What Unix is it? HP/UX, Solaris, AIX? – Stéphane Chazelas Jan 11 '16 at 9:58

When you append data to the file you're reading, you're running the risk of entering an infinite loop and growing the file forever as you end-up processing the data you've written earlier.

You could guard against that with things like:

perl -pe '
  BEGIN{seek(STDOUT,0,2);$end = tell STDOUT}
  last if tell(ARGV) > $end;
  s/i/o/' < file >> file

Inside a perl script:

open OUT, ">>", "file" or die "open file: $!";
open IN, "<", "file" or die "open file: $!";
seek(OUT,0,2) or die "seek: $!";
$end = tell OUT;
while (tell IN < $end && <IN>) {
  print OUT $_;
close IN;
close OUT;
sed 's/^i/o/;H;1h;$!d;x;q' <infile >>infile

If the file is small enough to fit into memory, then the above should work. I can't think of any reason you might have an encoding problem unless your sed is buggy. A sane sed should handle any valid character encoding you might care to throw at it.

If it is not small enough to fit into memory, then on a system which understands the /dev/fd/[num] links (which is practically any Unix-like system), and given a shell which uses tmp files for here-documents and not pipes (which is most of them, to include the Bourne shell, bash, and zsh but not yash or ash variants such as BSD sh, dash or busybox sh which use pipes instead), and enough free ${TMPDIR:-/tmp} space to hold the buffer while it is being edited, then the following should work:

sed -nf- file <<"" >>file
w /dev/fd/0
$r /dev/fd/0

That will work because the shell will get a temp file and a file-descriptor for the here-document, write the sed script into it, unlink() the temp file (and so remove its one and only link in the file-system), then fork sed as a child to inherit it, and restore its own state to that it was in before calling sed - and so drop its own descriptor to the temp file. At that point the file exists only as sed's stdin descriptor, and the kernel is bound to maintain the file only so long as any handle to it exists, but as soon as all descriptors are released it will remove a file with 0 file-system links.

So sed will read its script in from the deleted temp -file, and then it will truncate it as its named write file - which is only a link to the deleted file from which it read its script - and before pulling each input line it will write a copy of its pattern space out there. sed will autoprint -nothing, but on its $ last input line it will read to its stdout the file to which it has been writing all the while - and that will be >> appended to its named edit file.

When sed is through and its process terminates the last remaining descriptor to the <<"" here-doc source will be closed, and the kernel will afterward cleanup the file. In the meanwhile, no other process will have any means of accessing the file, and so it is therefore immune to any possibility of another process somehow affecting sed's work buffer.

If -nf- doesn't work it is probably only because your sed doesn't interpret - to mean stdin (though most do) and you should instead use -nf/dev/fd/0.

  • Note that many non-GNU sed implementations have a very limited size for their pattern (and hold) space (POSIX only guarantees 10 full lines IIRC) – Stéphane Chazelas Jan 11 '16 at 9:53
  • The /dev/fd/0 approach will only work on Linux. On other systems, opening /dev/fd/0 is more like a dup(0). – Stéphane Chazelas Jan 11 '16 at 9:57
  • @StéphaneChazelas - it's 4k in pattern space and 4k in hold space minimum - or maybe 8k? It's one of those. I would think ful lines were LINEMAX dependent, but isnt it usually 512 bytes? Is that what you mean by full lines? – mikeserv Jan 11 '16 at 11:27
  • Yes, looks like it's 8k indeed so it's even less than 10 full lines (at least 20480 per POSIX). Not sure where I got the 10 lines from (10240 was also the number I had in mind, and I thought the minimum LINE_MAX was 1024). Memory's getting really bad... – Stéphane Chazelas Jan 11 '16 at 11:46
  • @StéphaneChazelas - well, with luck, you won't have to remember it. how often do those kinds of arbitrary restrictions really come up anymore anyway? i dont think there are any current baseline utilities that stick to arbitrary size restrictions like that. keeping stuff w/in a line (and i could swear its 512 - that's pr's -width in a stream, merge anyway) makes sense of course, but nothing implodes anymore given such a slight poke. and why should it? most of us can fit... ok i cant count that high number of lines in memory without noticing... im more interested in the fd link comment...? – mikeserv Jan 11 '16 at 12:27

You can use memory-mapping to "cheat" by creating a restricted memory map over the file limited to the initial size of the file. Separately, open another handle to this file and seek this handle to the end. Start iterating over the memory map, writing each line that is read to the other file handle positioned at the end of the file. Representative python code

import mmap
with open('file', 'r+') as f1, open('file', 'r+b')  as f2:
    mm = mmap.mmap(f2.fileno(), 0) #memory map restricted to current file length
    f1.seek(0, 2) #seek to end of file
    for line in mm:
            f1.write(line.replace('i', 'o', 1))

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.