7

I have a gazillion files which need to be reduced in size. I found that most (not all) files have an end section which can be cut without losing information:

Data 1
Data 2
something_unimportant_here END DATA
Rubbish 1
Rubbish 2

How can I edit a file (end hence, all) by deleting the line including "END DATA" and all following, in-place, changing only those the files that contain the pattern, thereby minimising write access to the disk (many, many files and slow disk).

If possible, I would like to add a new last line to the file (my own end tag) so that the file's syntax stays correct -- again, only in those files containing the pattern.

I was thinking of using ed, like

echo ',s/END DATA/ ???? '\\n'q'\\n'wq' | ed "$file"

but cannot seem to manage the ???? part correcty.

Expected output:

Data 1
Data 2
NEW END
  • 1
    What is the expected output? – Nasir Riley Feb 22 at 13:36
6

It sounds like the sequence of commands you're looking for is

/END DATA/,$d
q
.a
NEW END
.
wq

or as a one-liner

printf '%s\n' '/END DATA/,$d' 'q' '.a' 'NEW END' '.' 'wq'

(You can replace wq with ,p for testing.)

Ex. given

$ cat file
Data 1
Data 2
something_unimportant_here END DATA
Rubbish 1
Rubbish 2

then

$ printf '%s\n' '/END DATA/,$d' 'q' '.a' 'NEW END' '.' 'wq' | ed -s file

gives

$ cat file
Data 1
Data 2
NEW END
  • Thanks, that worked. I needed a q\\n before .a not to change non-matching files and write them, though (ed won't exit upon q after changing something). – Ned64 Feb 22 at 15:08
  • (1) As far as I can tell, the . in .a is not necessary. (2) A slightly shorter alternative is printf '%s\n' '/END DATA/,$c' 'NEW END' '.' 'q' 'wq'.  Of course it has the disadvantage that it will try to execute NEW END and . as commands if END DATA is not found. – G-Man Says 'Reinstate Monica' Feb 23 at 6:35
  • @G-ManSays'ReinstateMonica' Thanks. The command needs to work both if the search pattern is absent and present. In the former case the file may not be written to. Will try leaving out . and edit as I find out. – Ned64 Feb 23 at 9:59
  • @Ned64 ed works by first reading the WHOLE file (either in memory or in a temporary file), modifying the data, and then writing back the whole onto the original (and replacing it non-atomically) even if you change a single bit in the file. I'm sure you're happy with this solution, but it does not match your "minimising write access to the disk" requirement. In general, it's not practical to use ed with large files. – mosvy Feb 23 at 11:58
  • @chepner No, I believe it needs a q before a. The file is not to be written to if the pattern is not found. – Ned64 Feb 24 at 0:12
8

You should be able to do it by just truncating the file in place without having to write a new copy of the file like sed -i/perl -i/ed/gawk -i inplace would do. With perl:

find . -name '*.txt' -type f -exec perl -ne '
  BEGIN{@ARGV=map{"+<$_"}@ARGV} # open files in read+write mode in the
                                # while(<>) loop implied by -n
  if (/END DATA/) {
    seek ARGV,-length,1; # back to beginning of matching line
    print ARGV "NEW END\n";
    truncate ARGV, tell ARGV;
    close ARGV; # skip to next file
  }' {} +

That minimises the I/O as perl stops reading as soon as it finds a match, and NEW END\n is the only thing it writes. It also writes in place, so the files metadata (ownership, permission, acls, sparseness...) are preserved and hard links are not broken.

With -exec {} + we also minimise the number of perl invocations.

|improve this answer|||||
  • Thanks for the perl version, will vote after testing later. – Ned64 Feb 23 at 9:54
6

With GNU grep and GNU sed

grep -lZ 'END DATA' *.txt | xargs -0 sed -i -e '/END DATA/,${//i foo' -e 'd}'

where *.txt assumes all your files are in current directory ending with .txt extension. If you need to recursively search for files, GNU grep also supports -r/-R options.

/END DATA/,$ range of lines to operate

//i foo here // will match the previously used regex, i.e. /END DATA/ and i command will add the new ending marker as needed

as i command has to be separated by newline, -e option is used to separate the d command to delete all lines matched by the range

as an alternate, you can also use this, but only one file will be passed at a time to sed:

grep -lZ 'END DATA' *.txt | xargs -0 -n1 sed -i -e '/END DATA/{i foo' -e 'Q}'
|improve this answer|||||
  • 1
    Thanks, this works. Upvoted. Slightly prefer the pure sed (no grep) answer, and it was the first, so accepting that. – Ned64 Feb 22 at 15:13
  • yeah no issues, I added grep because you mentioned minimizing writes – Sundeep Feb 22 at 15:34
  • Good point. The same can be achieved by the extra single q in ed/sed which exits without writing iff no change has occured. – Ned64 Feb 22 at 16:03
  • Does the Q version have any advantage over the d version?  As you point out, it will invoke sed many times (and therefore will do more I/O). – G-Man Says 'Reinstate Monica' Feb 23 at 6:36
  • @G-ManSays'ReinstateMonica' not sure, I added that version in case there are too many lines in each file after END DATA there by helping when quitting early – Sundeep Mar 1 at 14:00
3

Keep it simple and just use awk for the file manipulation part, e.g., with GNU find, awk, grep, and xargs:

find . -type f -exec grep -lZ 'END DATA' {} + |
    xargs -r0 awk -i inplace '/END DATA/{nextfile} 1'

or to print your own end tag at the end of every file:

... |
  xargs -r0 awk -i inplace '/END DATA/{print "NEW END"; nextfile} 1'
|improve this answer|||||
  • Yuck, that will copy the whole file instead of truncating it in place. – mosvy Feb 23 at 12:08
  • 2
    @mosvy "yuck" is a severe over-reaction. Every UNIX tool that does "inplace" editing (perl -i, sed -i, awk -i inplace, ruby -i, etc.) does that (except ed which makes a copy of the whole file in a buffer!) and it's what the standard advice for every other UNIX tool to modify the original file does (tool file > tmp && mv tmp file). The vast (and I mean vast) majority of people wanting to modify the original file are absolutely fine with it, it's a very, VERY rare occasion when someone actually needs the file truncated instead. – Ed Morton Feb 23 at 14:31
  • Except that you don't need any -i feature to truncate a file at some offset and write something there. The OP said "minimising write access to the disk", and your solution simply fails to do that. Whether it's really warranted to take the OP at their word is a completely different matter ;-) But people do have to deal with huge files, and simply copying them through each time they're modified is not practical, so this doesn't look like a made-up problem. – mosvy Feb 24 at 5:37
  • This is offtopic, but FYI some of the tools mentioned by you use different strategies to implement the -i feature, all bad, and not always similar to tool file > tmp && mv tmp file (with tmp on the same fs as file). For instance, ruby will open the file, unlink it and then open as output a new file with the same name. About perl see the links and notes here. – mosvy Feb 24 at 12:27
  • 1
    @mosvy I'm not saying it's a made up problem, just that it's an extremely rare problem that should be only be addressed when it proves to be a problem in a specific case, just like anything else performance related that requires complicating a solution that's extremely common, simple and perfectly fine in the vast majority of cases. – Ed Morton Feb 24 at 14:47
2

This python 3.8 solution is loosely based on Stephane's in-place truncate solution with a couple of differences 1. The code doesn't rely on external utilities for directory traversal 2. Files are memory-mapped for ease of locating the END DATA string

Place the code in a .py file and pass the directory name as a parameter

import mmap
import os
import sys
from contextlib import closing

def yield_all_files(dir_):
    for root, dir_, files in os.walk(dir_):
        yield from (os.path.join(root, file_) for file_ in files if file_.endswith('.txt'))

if __name__ == '__main__':
    old = b'END DATA'
    new = b'NEW END\n'
    dir_ = sys.argv[1]
    for file_ in yield_all_files(dir_):
        with open(file_, mode='r+b') as f:
            with closing(mmap.mmap(f.fileno(), length=0, access=mmap.ACCESS_WRITE)) as mm:
                if (loc := mm.find(old)) > -1:
                    mm.seek(loc)
                    mm.write(new)
                    mm.resize(mm.tell()) 
|improve this answer|||||
  • Does this process all files in the directory or all files in the directory tree (i.e., recursively)?  Either way, how would you change it to work the other way? – G-Man Says 'Reinstate Monica' Feb 23 at 6:42
  • @G-ManSays'ReinstateMonica', as it stands it would process all the files in the directory tree. To change this to process all files in the directory, I would change the body of yield_all_files function to yield from (f.path for f in os.scandir(dir_) if f.isfile()) – iruvar Feb 23 at 20:29
1

Combining Sundeep’s answer and Ed Morton’s answer, but without xargs:

find . -type f -name '*.txt' …(other criteria)… -exec grep -q 'END DATA' {} ';'  -a  \
                                -exec sed -i -e '/END DATA/,${//i NEW END' -e 'd}' {} +
  • find, of course, selects files.  By default, it searches the specified director(ies) recursively.  To search only the current directory, add -maxdepth 1 after the ..
  • grep -q quietly exits with a “success” exit status if a file contains the pattern being searched for (END DATA) and “false” otherwise.
  • -a means “AND”, like && on a shell command line.  It means “do the following thing if (only if) the previous thing succeeded.”  Actually, it’s the default conjunction operator between find predicates (tests/actions), so you can leave it out.  I included it only for clarity.
  • So the sed command, which is copied verbatim from Sundeep’s answer (but with foo changed to NEW END), is executed only on files that contain the END DATA string and satisfy the other find tests.
  • -exec … + causes sed to be invoked once with multiple files, as xargs does it.
    Note that we can’t use -exec … + with the grep command because it doesn’t allow testing exit status.
|improve this answer|||||
0

Using awk to find the offset of the pattern, and dd to truncate the file at that point and append the new trailer:

# usage truncatoo pattern new_end find_args ...
truncatoo(){
    pat=$1; shift; tail=$1; shift
    LC_CTYPE=C TAIL=$tail find "$@" -exec awk -v q="'" "$pat"'{
           gsub(q,q"\\"q q,FILENAME);
           system("printf \"$TAIL\" | dd bs="l" seek=1 of="q FILENAME q" 2>/dev/null");
           exit
       }
       {l+=length()+1}
    ' {} \;
}

truncatoo '/END DATA/' 'NEW END\n' file.txt
truncatoo '/END DATA/' 'NEW END\n' . -type f -name '*.txt'

With an awk implementation which supports nextfile (gawk, bwk, some versions of mawk [1]), that could be done more efficiently by passing batches of files to awk:

# usage truncatoo pattern new_end find_args ...
truncatoo(){
    pat=$1; shift; tail=$1; shift
    LC_CTYPE=C TAIL=$tail find "$@" -exec awk -v q="'" "$pat"'{
           gsub(q,q"\\"q q,FILENAME);
           system("printf \"$TAIL\" | dd bs="l" seek=1 of="q FILENAME q" 2>/dev/null");
           l=0; nextfile
       }
       {l+=length()+1}
    ' {} +
}
$ file="a'b\$q  * r"; seq 1 100 >"$file"
$ truncatoo /7/ 'CUT\n' "$file"; cat "$file"
1
2
3
4
5
6
CUT

Instead of the icky 2>/dev/null status=noxfer could be used with dd implementations which support it.

The quoting kludge and environment variables passing is a mess, it could use some improvement.

[1]: according to the GNU awk manual, it should be supported in mawk too. However, the older version of mawk from Debian 10 does not support it.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.