62

I have a fairly large file (35Gb), and I would like to filter this file in situ (i.e. I don't have enough disk space for another file), specifically I want to grep and ignore some patterns — is there a way to do this without using another file?

Let's say I want to filter out all the lines containing foo: for example...

14
  • 3
    @Tshepang: I think he wants to write back to the same file. Apr 11, 2011 at 10:04
  • 5
    "in situ" is a Latin phrase meaning "in place". Literally, "in position". Apr 11, 2011 at 10:16
  • 3
    In that case, the question should be clearer, something like is there a way to modify a file in-place?
    – tshepang
    Apr 11, 2011 at 10:43
  • 6
    @Tshepang, "in situ" is a fairly common phrase used in English to describe exactly that - I thought the title was fairly self explanatory... @Gilles, I figured as much, easier to wait for more disk space! ;)
    – Nim
    Apr 11, 2011 at 11:48
  • 2
    @Nim: Well, I think in-place is more common than in situ.
    – tshepang
    Apr 12, 2011 at 7:07

10 Answers 10

42

At the system call level this should be possible. A program can open your target file for writing without truncating it and start writing what it reads from stdin. When reading EOF, the output file can be truncated.

Since you are filtering lines from the input, the output file write position should always be less than the read position. This means you should not corrupt your input with the new output.

However, finding a program that does this is the problem. dd(1) has the option conv=notrunc that does not truncate the output file on open, but it also does not truncate at the end, leaving the original file contents after the grep contents (with a command like grep pattern bigfile | dd of=bigfile conv=notrunc)

Since it is very simple from a system call perspective, I wrote a small program and tested it on a small (1MiB) full loopback filesystem. It did what you wanted, but you really want to test this with some other files first. It's always going to be risky overwriting a file.

overwrite.c

/* This code is placed in the public domain by camh */

#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <errno.h>

int main(int argc, char **argv)
{
        int outfd;
        char buf[1024];
        int nread;
        off_t file_length;

        if (argc != 2) {
                fprintf(stderr, "usage: %s <output_file>\n", argv[0]);
                exit(1);
        }
        if ((outfd = open(argv[1], O_WRONLY)) == -1) {
                perror("Could not open output file");
                exit(2);
        }
        while ((nread = read(0, buf, sizeof(buf))) > 0) {
                if (write(outfd, buf, nread) == -1) {
                        perror("Could not write to output file");
                        exit(4);
                }
        }
        if (nread == -1) {
                perror("Could not read from stdin");
                exit(3);
        }
        if ((file_length = lseek(outfd, 0, SEEK_CUR)) == (off_t)-1) {
                perror("Could not get file position");
                exit(5);
        }
        if (ftruncate(outfd, file_length) == -1) {
                perror("Could not truncate file");
                exit(6);
        }
        close(outfd);
        exit(0);
}

You would use it as:

grep pattern bigfile | overwrite bigfile

I'm mostly posting this for others to comment on before you try it. Perhaps someone else knows of a program that does something similar that is more tested.

7
  • I wanted to see if I could get away without writing something for it! :) I guess this will do the trick! Thanks!
    – Nim
    Apr 11, 2011 at 13:06
  • 2
    +1 for C; does seem to work, but I see a potential problem: the file is being read from the left side at the time as the right is writing to the same file and unless you coordinate the two processes, you would have overwrite problems potentially on the same blocks. It might be better for the file integrity to use smaller block size since most of the core tools will likely use 8192. This might slow down the program enough to avoid conflicts (but cannot guarantee). Maybe read larger portions into memory (not all) and write in smaller blocks. Could also add a nanosleep(2)/usleep(3).
    – Arcege
    Apr 11, 2011 at 13:15
  • 4
    @Arcege: Writing is not done in blocks. If your read process has read 2 bytes and your write process writes 1 byte, only the first byte will change and the read process can continue reading at byte 3 with the original contents at that point unchanged. Since grep will not output more data than it reads, the write position should always be behind the read position. Even if you are writing at the same rate as reading, it will still be ok. Try rot13 with this instead of grep, and then again. md5sum the before and after and you'll see its the same.
    – camh
    Apr 11, 2011 at 13:45
  • 6
    Nice. This may be a valuable addition to Joey Hess's moreutils. You can use dd, but it's cumbersome. Apr 11, 2011 at 21:24
  • 'grep pattern bigfile | overwrite bigfile' - I got this working without errors, but what I don't understand is - isn't the requirement to replace what's in the pattern with some other text? so shouldn't it be something like: 'grep pattern bigfile | overwrite /replace-text/ bigfile' Dec 25, 2016 at 21:14
26

With any Bourne-like shell:

{
  cat < bigfile | grep -v to-exclude
  perl -e 'truncate STDOUT, tell STDOUT'
} 1<> bigfile

For some reason, it seems people tend to forget about that 40 year old¹ and standard read+write redirection operator.

We open bigfile in read+write mode and (what matters most here) without truncation on stdout while bigfile is open (separately) on cat's stdin. After grep has terminated, and if it has removed some lines, stdout now points somewhere within bigfile, we need to get rid of what's beyond this point. Hence the perl command that truncates the file (truncate STDOUT) at the current position (as returned by tell STDOUT).

(the cat is for GNU grep that otherwise complains if stdin and stdout point to the same file).


¹ Well, while <> has been in the Bourne shell from the start in the late seventies, it was initially undocumented and not properly implemented. It was not in the original implementation of ash from 1989 and, while it is a POSIX sh redirection operator (since the early 90s as POSIX sh is based on ksh88 which always had it), it was not added to FreeBSD sh for instance until 2000, so portably 15 year old is probably more accurate. Also note that the default file descriptor when not specified is 0 in all shells, except that in ksh93 it changed from 0 to 1 in ksh93t+ in 2010 (breaking backward compatibility and POSIX compliance)

6
  • 2
    Can you explain the perl -e 'truncate STDOUT, tell STDOUT'? It works for me without including that. Any way to achieve the same thing without using Perl? Feb 1, 2016 at 21:57
  • 1
    @AaronBlenkush, see edit. Feb 1, 2016 at 22:12
  • 1
    Absolutely brilliant - thank you. I was there then, but don't remember this.... A reference for the "36 year old" standard would be fun, since it isn't mentioned at en.wikipedia.org/wiki/Bourne_shell. And what was it used for? I see a reference to a bug fix in SunOS 5.6: redirection "<>" fixed and documented (used in /etc/inittab f.i.). which is one hint.
    – nealmcb
    Jul 31, 2017 at 3:21
  • 2
    @nealmcb, see edit. Jul 31, 2017 at 7:05
  • 1
    @akhan, that other approach replaces the file with a new one, it doesn't overwrite the same file in place. In particular, permissions ownership, and other metadata of the original file will be lost. May 8, 2021 at 8:51
20

You can use sed to edit files in place (but this does create an intermediate temporary file):

To remove all lines containing foo:

sed -i '/foo/d' myfile

To keep all lines containing foo:

sed -i '/foo/!d' myfile
7
  • interesting, will this temp file need to be the same size as the original though?
    – Nim
    Apr 11, 2011 at 11:50
  • 3
    Yes, so that's probably no good.
    – pjc50
    Apr 11, 2011 at 13:00
  • 21
    This is not what the OP is asking for since it creates a second file.
    – Arcege
    Apr 11, 2011 at 13:15
  • 1
    This solution will fail on a read-only file system, where "read-only" means that your $HOME will be writable, but /tmp will be read-only (by default). For instance, if you have Ubuntu and you've booted into the Recovery Console, this is commonly the case. Also, the here-document operator <<< will not work there either, as it requires /tmp to be r/w because it will write a temporary file into there as well. (cf. this question incl. a strace'd output) Dec 3, 2014 at 14:36
  • yeah this won't work for me either, all the sed commands I have tried will replace the current file with a new a file (despite the --in-place flag). Dec 25, 2016 at 20:51
20

I'll assume that your filter command is what I'll call a prefix shrinking filter, which has the property that byte N in the output is never written before having read at least N bytes of input. grep has this property (as long as it's only filtering and not doing other things like adding line numbers for matches). With such a filter, you can overwrite the input as you go along. Of course, you need to be sure of not making any mistake, since the overwritten part at the beginning of the file will be lost forever.

Most unix tools only give a choice of appending to a file or truncating it, with no possibility of overwriting it. The one exception in the standard toolbox is dd, which can be told not to truncate its output file. So the plan is to filter the command into dd conv=notrunc. This doesn't change the size of the file, so we also grab the length of the new content and truncate the file to that length (again with dd). Note that this task is inherently non-robust — if an error occurs, you're on your own.

export LC_ALL=C
n=$({ grep -v foo <big_file |
      tee /dev/fd/3 |
      dd of=big_file conv=notrunc; } 3>&1 | wc -c)
dd if=/dev/null of=big_file bs=1 seek=$n

You can write rougly equivalent Perl. Here's a quick implementation that doesn't try to be efficient. Of course, you may want to do your initial filtering directly in that language as well.

grep -v foo <big_file | perl -e '
  close STDOUT;
  open STDOUT, "+<", $ARGV[0] or die;
  while (<STDIN>) {print}
  truncate STDOUT, tell STDOUT or die
' big_file
0
10

Even though this is an old question, it seems to me it's a perennial question, and a more general, clearer solution is available than has been suggested so far. Credit where credit is due: I'm not sure I would have come up with it without considering Stéphane Chazelas's mention of the <> update operator.

Opening a file for update in a Bourne shell is of limited utility. The shell gives you no way to seek on a file, and no way to set its new length (if shorter than the old one). But that's easily remedied, so easily I'm surprised it's not among the standard utilities in /usr/bin.

This works:

$ grep -n foo T
8:foo
$ (exec 4<>T; grep foo T >&4 && ftruncate 4) && nl T; 
     1  foo

As does this (hat tip to Stéphane):

$ { grep foo T && ftruncate; } 1<>T  && nl T; 
     1  foo

(I'm using GNU grep. Perhaps something's changed since he wrote his answer.)

Except, you have no /usr/bin/ftruncate. For a couple dozen lines of C, you can, see below. This ftruncate utility truncates an arbitrary file descriptor to an arbitrary length, defaulting to standard output and the current position.

The above command (1st example)

  • opens file descriptor 4 on T for update. Just as with open(2), opening the file this way positions the current offset at 0.
  • grep then processes T normally, and the shell redirects its output to T via descriptor 4.
  • ftruncate calls ftruncate(2) on descriptor 4, setting the length to the value of the current offset (exactly where grep left it).

The subshell then exits, closing descriptor 4. Here is ftruncate:

#include <err.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>

int
main( int argc, char *argv[] ) {
  off_t i, fd=1, len=0;
  off_t *addrs[2] = { &fd, &len };

  for( i=0; i < argc-1; i++ ) {
    if( sscanf(argv[i+1], "%lu", addrs[i]) < 1 ) {
      err(EXIT_FAILURE, "could not parse %s as number", argv[i+1]);
    }
  }

  if( argc < 3 && (len = lseek(fd, 0, SEEK_CUR)) == -1 ) {
    err(EXIT_FAILURE, "could not ftell fd %d as number", (int)fd);
  }


  if( 0 != ftruncate((int)fd, len) ) {
    err(EXIT_FAILURE, argc > 1? argv[1] : "stdout");
  }

  return EXIT_SUCCESS;
}

N.B., ftruncate(2) is nonportable when used in this way. For absolute generality, read the last written byte, reopen the file O_WRONLY, seek, write the byte, and close.

Given that the question is 5 years old, I'm going to say this solution is nonobvious. It takes advantage of exec to open a new descriptor, and the <> operator, both of which are arcane. I can't think of a standard utility that manipulates an inode by file descriptor. (The syntax could be ftruncate >&4, but I'm not sure that an improvement.) It's considerably shorter than camh's competent, exploratory answer. It's just a little clearer than Stéphane's, IMO, unless you like Perl more than I do. I hope someone finds it useful.

A different way to do the same thing would be an executable version of lseek(2) that reports the current offset; the output could be used for /usr/bin/truncate, which some Linuxi provide.

1
  • Fantastic answer. Thank you.
    – likebike
    Jul 13, 2021 at 1:23
5

ed is probably the right choice to edit a file in-place:

ed my_big_file << END_OF_ED_COMMANDS
g/foo:/d
w
q 
END_OF_ED_COMMANDS
4
  • I like the idea, but unless different ed versions behave differently..... this is from man ed (GNU Ed 1.4)... If invoked with a file argument, then a copy of file is read into the editor's buffer. Changes are made to this copy and not directly to file itself.
    – Peter.O
    Apr 14, 2011 at 17:34
  • @fred, if you're implying that saving the changes will not affect the named file, you're incorrect. I interpret that quote to say that your changes aren't reflected until you save them. I do concede that ed isn't a gool solution for editing 35GB files since the file is read into a buffer. Apr 14, 2011 at 19:07
  • 2
    I was thinking that it meant the full file would be loaded into the buffer.. but perhaps only the section(s) it neeeds are loaded into the buffer.. I've been curious about ed for a while... I thought it could do in-situ editing... I'll just have to try a big file... If it works it is a reasonable solution, but as I write, I'm starting to think that this may be what inspired sed (freed from working with large data chunks... I've noticed that 'ed' can actually accept streamed input from a script (prefixed with ! ), so it may have a few more interesting tricks up its sleeve.
    – Peter.O
    Apr 14, 2011 at 20:49
  • I am pretty sure the write operation in ed truncates the file and rewrites it. So this won't alter the data on disk in-place as the OP desires. Also, it can't work if the file is too big to be loaded in memory. Apr 14, 2017 at 17:08
5

You can use a bash read/write file descriptor to open your file (to overwrite it in-situ), then sed and truncate ... but of course, don't ever allow your changes to be larger than the amount of data read so far.

Here is the script (uses: bash variable $BASHPID )

# Create a test file
  echo "going abc"  >junk
  echo "going def" >>junk
  echo "# ORIGINAL file";cat junk |tee >( wc=($(wc)); echo "# ${wc[0]} lines, ${wc[2]} bytes" ;echo )
#
# Assign file to fd 3, and open it r/w
  exec 3<> junk  
#
# Choose a unique filename to hold the new file size  and the pid 
# of the semi-asynchrounous process to which 'tee' streams the new file..  
  [[ ! -d "/tmp/$USER" ]] && mkdir "/tmp/$USER" 
  f_pid_size="/tmp/$USER/pid_size.$(date '+%N')" # %N is a GNU extension: nanoseconds
  [[ -f "$f_pid_size" ]] && { echo "ERROR: Work file already exists: '$f_pid_size'" ;exit 1 ; }
#
# run 'sed' output to 'tee' ... 
#  to modify the file in-situ, and to count the bytes  
  <junk sed -e "s/going //" |tee >(echo -n "$BASHPID " >"$f_pid_size" ;wc -c >>"$f_pid_size") >&3
#
#@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
# The byte-counting process is not a child-process, 
# so 'wait' doesn't work... but wait we must...  
  pid_size=($(cat "$f_pid_size")) ;pid=${pid_size[0]}  
  # $f_pid_size may initially contain only the pid... 
  # get the size when pid termination is assured
  while [[ "$pid" != "" ]] ; do
    if ! kill -0 "$pid" 2>/dev/null; then
       pid=""  # pid has terminated. get the byte count
       pid_size=($(cat "$f_pid_size")) ;size=${pid_size[1]}
    fi
  done
  rm "$f_pid_size"
#@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
#
  exec 3>&- # close fd 3.
  newsize=$(cat newsize)
  echo "# MODIFIED file (before truncating)";cat junk |tee >( wc=($(wc)); echo "# ${wc[0]} lines, ${wc[2]} bytes" ;echo )  cat junk
#
 truncate -s $newsize junk
 echo "# NEW (truncated) file";cat junk |tee >( wc=($(wc)); echo "# ${wc[0]} lines, ${wc[2]} bytes" ;echo )  cat junk
#
exit

Here is the test output

# ORIGINAL file
going abc
going def
# 2 lines, 20 bytes

# MODIFIED file (before truncating)
abc
def
c
going def
# 4 lines, 20 bytes

# NEW (truncated) file
abc
def
# 2 lines, 8 bytes
5

For the benefit of anyone googling the question "how do I modify a file in-place?", the correct answer in the usual case is stop looking for obscure shell features that risk corrupting your file for negligible performance gain, and instead use some variation of this pattern:

grep "foo" file > file.new && mv file.new file

Only in the extremely uncommon situation that this is for some reason not feasible, should you seriously consider any of the other answers on this page (although they are certainly interesting to read). I will concede that the OP's conundrum of having no disk space to create a second file is exactly such a situation. Although even then, there are other options available, e.g. as provided by @Ed Randall and @Basile Starynkevitch.

4
  • 2
    I may missunderstand but has nothing to do with what the OP originaly asked. aka inline edit of bigfile without having enough diskspace for temporary file.
    – Kiwy
    Apr 6, 2018 at 14:42
  • @Kiwy It is an answer aimed at other viewers of this question (of which there have been almost 15,000 so far). The question "Is there a way to modify a file in-place?" has broader relevance than the OP's specific use case.
    – Todd Owen
    Apr 6, 2018 at 15:08
  • Not having space for a huge (35 GB) temp is what makes OP's Q interesting. This scenario may be inconvenient, however it's not unreasonable The Q does not involve a "negligible performance gain", it concern a resource limitation. OP is not asking for an "obscure shell feature", just for a solution. Aug 17, 2020 at 8:00
  • @ShpielMeister I've just edited my answer to hopefully clarify that I am providing this answer for the benefit of anyone who comes to this page looking for an answer to the simple version of the question "how do I modify a file in-place?". I know the OP's question is not that simple. I just want to help other readers. But if you still think the answer is unhelpful, please feel free to down-vote it.
    – Todd Owen
    Aug 17, 2020 at 13:52
3

I'd memory-map the file, do everything in-place using char* pointers to naked memory, then unmap the file and truncate it.

1
  • 4
    +1, but only because the widespread availability of 64-bit CPUs and OSes makes it possible to do that with a 35 GB file now. Those still on 32-bit systems (the vast majority even of this site's audience, I suspect) won't be able to use this solution. Apr 14, 2011 at 16:21
2

Not exactly in-situ but - this could be of use in similar circumstances.
If disk space is a problem, compress the file first (since it is text this will give a huge reduction) then use sed (or grep, or whatever) in the usual way in the middle of an uncompress/compress pipeline.

# Reduce size from ~35Gb to ~6Gb
$ gzip MyFile

# Edit file, creating another ~6Gb file
$ gzip -dc <MyFile.gz | sed -e '/foo/d' | gzip -c >MyEditedFile.gz
2
  • 2
    But surely gzip is writing the compressed version to the disk before replacing it with the compressed version, so you need at least that much extra space, unlike the other options. But it is safer, if you've got the space (which I don't....)
    – nealmcb
    Jul 31, 2017 at 2:23
  • This is a clever solution that can be further optimized to perform only one compression instead of two: sed -e '/foo/d' MyFile | gzip -c >MyEditedFile.gz && gzip -dc MyEditedFile.gz >MyFile
    – Todd Owen
    Apr 6, 2018 at 14:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.