1

I have a very large file I need to remove a pattern from. The problem is, there are no newline separators in the file. The pattern to remove looks like this: ... 1666 more items where 1666 can be any number.

  • 1
    What happens with a simple sed 's/... [0-9]* more items//g'? Is there an error because the "line" is too long? – Philippos Sep 26 at 5:51
  • "very large file" How large? – John1024 Sep 26 at 6:28
1

I don't know if sed is the best tool for this. I personally go to perl for such things, and have come up with this:

perl -pe 'BEGIN{ $/ = " more items"; } s/\.\.\.\s\d+ more items$//;'

-e is the program text to execute.

-p means to do an implicit loop over the program text once per record (by default a record is a line, see below for modifying that).

This first sets the record separator $/ from the default newline (\n) to the string " more items"; the block indicated by BEGIN { } is done once at the beginning.

Now it will read the input one chunk at a time, up to (and including) the record separator. So every loop you have "whatever ... 1234 more items" in the input buffer.

The s/\.\.\.\s\d+ more items$// is a substitution which removes any string of three dots (escaped to remove the special meaning of "any character"), followed by a space (\s), followed by one or more digits (\d+) and the text " more items" at the end of the record ($). Matching the end of the record should not be necessary but might speed up the matching.

The result is printed by default due to the -p option.

1

If you are using GNU sed (sometimes called gsed), then long lines are fine up to the limit of memory.

If you are using other sed, such as on MacOS or BSD, there will be limits. The GNU sed manual explains:

For those who want to write portable sed scripts, be aware that some implementations have been known to limit line lengths (for the pattern and hold spaces) to be no more than 4000 bytes. The POSIX standard specifies that conforming sed implementations shall support at least 8192 byte line lengths. GNU sed has no built-in limit on line length; as long as it can malloc() more (virtual) memory, you can feed or construct lines as long as you like.

Thus, if you have GNU sed, you can use a sed solution such as suggested by @philippos:

sed 's/\.\.\. [0-9]* more items//g'

Using awk

As an alternative, if you have GNU awk (sometimes called gawk), try:

awk -v RS='\.\.\. [0-9]* more items' 1 ORS="" File

Thus uses the regex \.\.\. [0-9]* more items as the record separator on input and the empty string as the record separator on output. This has the effect of removing any match of the regex \.\.\. [0-9]* more items.

Because awk only reads in one record at a time, this will consume less memory than the sed solution.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.