Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I have a fairly large file (35Gb), and I would like to filter this file in situ (i.e. I don't have enough disk space for another file), specifically I want to grep and ignore some patterns — is there a way to do this without using another file?

Let's say I want to filter out all the lines containing foo: for example...

share|improve this question
    
I don't understand. Do you want the output to go to stdout? –  Tshepang Apr 11 '11 at 10:03
    
@Tshepang: I think he wants to write back to the same file. –  Faheem Mitha Apr 11 '11 at 10:04
    
@Faheem, that is correct. –  Nim Apr 11 '11 at 10:14
    
"in situ" is a Latin phrase meaning "in place". Literally, "in position". –  Faheem Mitha Apr 11 '11 at 10:16
2  
In that case, the question should be clearer, something like is there a way to modify a file in-place? –  Tshepang Apr 11 '11 at 10:43
show 5 more comments

7 Answers 7

up vote 24 down vote accepted

At the system call level this should be possible. A program can open your target file for writing without truncating it and start writing what it reads from stdin. When reading EOF, the output file can be truncated.

Since you are filtering lines from the input, the output file write position should always be less than the read position. This means you should not corrupt your input with the new output.

However, finding a program that does this is the problem. dd(1) has the option conv=notrunc that does not truncate the output file on open, but it also does not truncate at the end, leaving the original file contents after the grep contents (with a command like grep pattern bigfile | dd of=bigfile conv=notrunc)

Since it is very simple from a system call perspective, I wrote a small program and tested it on a small (1MiB) full loopback filesystem. It did what you wanted, but you really want to test this with some other files first. It's always going to be risky overwriting a file.

overwrite.c

/* This code is placed in the public domain by camh */

#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <errno.h>

int main(int argc, char **argv)
{
        int outfd;
        char buf[1024];
        int nread;
        off_t file_length;

        if (argc != 2) {
                fprintf(stderr, "usage: %s <output_file>\n", argv[0]);
                exit(1);
        }
        if ((outfd = open(argv[1], O_WRONLY)) == -1) {
                perror("Could not open output file");
                exit(2);
        }
        while ((nread = read(0, buf, sizeof(buf))) > 0) {
                if (write(outfd, buf, nread) == -1) {
                        perror("Could not write to output file");
                        exit(4);
                }
        }
        if (nread == -1) {
                perror("Could not read from stdin");
                exit(3);
        }
        if ((file_length = lseek(outfd, 0, SEEK_CUR)) == (off_t)-1) {
                perror("Could not get file position");
                exit(5);
        }
        if (ftruncate(outfd, file_length) == -1) {
                perror("Could not truncate file");
                exit(6);
        }
        close(outfd);
        exit(0);
}

You would use it as:

grep pattern bigfile | overwrite bigfile

I'm mostly posting this for others to comment on before you try it. Perhaps someone else knows of a program that does something similar that is more tested.

share|improve this answer
    
I wanted to see if I could get away without writing something for it! :) I guess this will do the trick! Thanks! –  Nim Apr 11 '11 at 13:06
    
+1 for C; does seem to work, but I see a potential problem: the file is being read from the left side at the time as the right is writing to the same file and unless you coordinate the two processes, you would have overwrite problems potentially on the same blocks. It might be better for the file integrity to use smaller block size since most of the core tools will likely use 8192. This might slow down the program enough to avoid conflicts (but cannot guarantee). Maybe read larger portions into memory (not all) and write in smaller blocks. Could also add a nanosleep(2)/usleep(3). –  Arcege Apr 11 '11 at 13:15
3  
@Arcege: Writing is not done in blocks. If your read process has read 2 bytes and your write process writes 1 byte, only the first byte will change and the read process can continue reading at byte 3 with the original contents at that point unchanged. Since grep will not output more data than it reads, the write position should always be behind the read position. Even if you are writing at the same rate as reading, it will still be ok. Try rot13 with this instead of grep, and then again. md5sum the before and after and you'll see its the same. –  camh Apr 11 '11 at 13:45
3  
Nice. This may be a valuable addition to Joey Hess's moreutils. You can use dd, but it's cumbersome. –  Gilles Apr 11 '11 at 21:24
add comment

echo -e "$(grep pattern bigfile)" >bigfile

share|improve this answer
    
This does not work if the file is big and the grepped data exceeds the length of what the commandline allows. it then corrupts the data –  Anthon Dec 15 '13 at 11:30
add comment

You can use a bash read/write file descriptor to open your file (to overwrite it in-situ), then sed and truncate ... but of course, don't ever allow your changes to be larger than the amount of data read so far.

Here is the script (uses: bash variable $BASHPID )

# Create a test file
  echo "going abc"  >junk
  echo "going def" >>junk
  echo "# ORIGINAL file";cat junk |tee >( wc=($(wc)); echo "# ${wc[0]} lines, ${wc[2]} bytes" ;echo )
#
# Assign file to fd 3, and open it r/w
  exec 3<> junk  
#
# Choose a unique filename to hold the new file size  and the pid 
# of the semi-asynchrounous process to which 'tee' streams the new file..  
  [[ ! -d "/tmp/$USER" ]] && mkdir "/tmp/$USER" 
  f_pid_size="/tmp/$USER/pid_size.$(date '+%N')" # %N is a GNU extension: nanoseconds
  [[ -f "$f_pid_size" ]] && { echo "ERROR: Work file already exists: '$f_pid_size'" ;exit 1 ; }
#
# run 'sed' output to 'tee' ... 
#  to modify the file in-situ, and to count the bytes  
  <junk sed -e "s/going //" |tee >(echo -n "$BASHPID " >"$f_pid_size" ;wc -c >>"$f_pid_size") >&3
#
#@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
# The byte-counting process is not a child-process, 
# so 'wait' doesn't work... but wait we must...  
  pid_size=($(cat "$f_pid_size")) ;pid=${pid_size[0]}  
  # $f_pid_size may initially contain only the pid... 
  # get the size when pid termination is assured
  while [[ "$pid" != "" ]] ; do
    if ! kill -0 "$pid" 2>/dev/null; then
       pid=""  # pid has terminated. get the byte count
       pid_size=($(cat "$f_pid_size")) ;size=${pid_size[1]}
    fi
  done
  rm "$f_pid_size"
#@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
#
  exec 3>&- # close fd 3.
  newsize=$(cat newsize)
  echo "# MODIFIED file (before truncating)";cat junk |tee >( wc=($(wc)); echo "# ${wc[0]} lines, ${wc[2]} bytes" ;echo )  cat junk
#
 truncate -s $newsize junk
 echo "# NEW (truncated) file";cat junk |tee >( wc=($(wc)); echo "# ${wc[0]} lines, ${wc[2]} bytes" ;echo )  cat junk
#
exit

Here is the test output

# ORIGINAL file
going abc
going def
# 2 lines, 20 bytes

# MODIFIED file (before truncating)
abc
def
c
going def
# 4 lines, 20 bytes

# NEW (truncated) file
abc
def
# 2 lines, 8 bytes
share|improve this answer
add comment

I'd memory-map the file, do everything in-place using char* pointers to naked memory, then unmap the file and truncate it.

share|improve this answer
    
+1, but only because the widespread availability of 64-bit CPUs and OSes makes it possible to do that with a 35 GB file now. Those still on 32-bit systems (the vast majority even of this site's audience, I suspect) won't be able to use this solution. –  Warren Young Apr 14 '11 at 16:21
add comment

I'll assume that your filter command is what I'll call a prefix shrinking filter, which has the property that byte N in the output is never written before having read at least N bytes of input. grep has this property (as long as it's only filtering and not doing other things like adding line numbers for matches). With such a filter, you can overwrite the input as you go along. Of course, you need to be sure of not making any mistake, since the overwritten part at the beginning of the file will be lost forever.

Most unix tools only give a choice of appending to a file or truncating it, with no possibility of overwriting it. The one exception in the standard toolbox is dd, which can be told not to truncate its output file. So the plan is to filter the command into dd conv=notrunc. This doesn't change the size of the file, so we also grab the length of the new content and truncate the file to that length (again with dd). Note that this task is inherently non-robust — if an error occurs, you're on your own.

export LC_ALL=C
n=$({ grep -v foo <big_file |
      tee /dev/fd/3 |
      dd of=big_file conv=notrunc; } 3>&1 | wc -c)
dd if=/dev/null of=big_file bs=1 seek=$n

You can write rougly equivalent Perl. Here's a quick implementation that doesn't try to be efficient. Of course, you may want to do your initial filtering directly in that language as well.

grep -v foo <big_file | perl -e '
  close STDOUT;
  open STDOUT, "+<", $ARGV[0] or die;
  while (<STDIN>) {print}
  truncate STDOUT, tell STDOUT or die
' big_file
share|improve this answer
    
+1 kudos for the neat and portable perl script –  kubanczyk Jun 17 at 9:53
add comment

ed is probably the right choice to edit a file in-place:

ed my_big_file << END_OF_ED_COMMANDS
g/foo:/d
w
q 
END_OF_ED_COMMANDS
share|improve this answer
    
I like the idea, but unless different ed versions behave differently..... this is from man ed (GNU Ed 1.4)... If invoked with a file argument, then a copy of file is read into the editor's buffer. Changes are made to this copy and not directly to file itself. –  Peter.O Apr 14 '11 at 17:34
    
@fred, if you're implying that saving the changes will not affect the named file, you're incorrect. I interpret that quote to say that your changes aren't reflected until you save them. I do concede that ed isn't a gool solution for editing 35GB files since the file is read into a buffer. –  glenn jackman Apr 14 '11 at 19:07
    
I was thinking that it meant the full file would be loaded into the buffer.. but perhaps only the section(s) it neeeds are loaded into the buffer.. I've been curious about ed for a while... I thought it could do in-situ editing... I'll just have to try a big file... If it works it is a reasonable solution, but as I write, I'm starting to think that this may be what inspired sed (freed from working with large data chunks... I've noticed that 'ed' can actually accept streamed input from a script (prefixed with ! ), so it may have a few more interesting tricks up its sleeve. –  Peter.O Apr 14 '11 at 20:49
add comment

You can use sed to edit files in place (but this does create an intermediate temporary file):

To remove all lines containing foo:

sed -i '/foo/d' myfile

To keep all lines containing foo:

sed -i '/foo/!d' myfile
share|improve this answer
    
interesting, will this temp file need to be the same size as the original though? –  Nim Apr 11 '11 at 11:50
1  
Yes, so that's probably no good. –  pjc50 Apr 11 '11 at 13:00
5  
This is not what the OP is asking for since it creates a second file. –  Arcege Apr 11 '11 at 13:15
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.