I keep a logfile of a program's output. The trouble is sometimes errors occur dumping very long lines of 7bit ascii data(encoded binary) I don't care to keep. These lines can be 200KB+ long before reaching a newline.

What might be a short, pipeable way eg. with sed to change only lines exceeding 250 characters long, by keeping only the first 80 and last 40 characters of that line, possibly replacing the middle with only a _?


In sed, all commands can be prefixed by a condition that indicates what lines to apply the command to. A common kind of condition is a search pattern. The search pattern /.\{250\}/ matches lines with more than 250 characters. For such lines, match the first 80 characters and the last 40, and replace the whole line by the prefix, __ and the suffix.

sed -e '/.\{250\}/ s/^\(.\{80\}\).*\(.\{40\}\)$/\1__\2/'

You can even arrange for the pattern of the replacement command to match only sufficiently long lines.

sed -e 's/^\(.\{80\}\).\{130,\}\(.\{40\}\)$//'
  • I get your 1st example (though haven't tested if it still prints shorter lines unchanged). In your 2nd example, doesn't the middle search expression catch strings of exactly 130 length only (hence 250 char lines)?
    – Marcos
    Jun 19 '12 at 8:50
  • I think I see; what's needed is a variable-length regex between the 130 and 40 matchers. So this might be the final sed way: sed -i.bak -e 's/^\(.\{80\}\).\{130,\}.*\(.\{40\}\)$/\1 ... \2/' All those \ make it annoying to read, but I understand why they're necessary. The awk way is more human readable for someone not used to these regexes.
    – Marcos
    Jun 19 '12 at 9:00
  • 1
    @Marcos; If your situiation allows you to use sed's extended regex (option -r), then you can use the clearer ( ) and { } ... The \( , etc method is more portable.
    – Peter.O
    Jun 19 '12 at 12:57

I don't know sed well, so my solution is in awk:

awk 'length>250{len=length;$0=substr($0,1,80) "_" substr($0,len-40+1)};1' file
  • Nice and concise, thanks. (Too bad awk lacks a -i in-place file replace option like sed. I'll just awk ... file1 > file2 && mv -f file2 file1)
    – Marcos
    Jun 18 '12 at 18:42
  • Minor detail, I was able to make do without the extra len variable altogether
    – Marcos
    Jun 18 '12 at 20:04
  • 1
    @Marcos Be careful with sed -i - it's not portable and most versions I've seen don't handle special files (like symlinks) correctly (it'll delete the link and replace it with a copy of the file).
    – jw013
    Oct 3 '12 at 17:36

Here's an awk command that will do this:

awk 'len=length{if(len>250){print substr($0,0,80),"__",substr($0,len-40,len)}else{print $0}}' data.txt


If the line is longer than 250 characters long print the first 80 characters, followed by a string of ___ and then the last 40 characters.

If the line is shorter than 250 characters, simply print out the original line.


There is a pipeable way to extract the first 80, and last 40, characters of a line with sed, however sed does not have 'if statements', so there's no way directly in sed to test for string length and then perform an operation on it.

Capturing the first 80 and last 40 characters of all lines can be done with | sed -e 's/^\(.\{80\}\).*\(.\{40\}\)/\1\_\2/'.

sed -i -e 's/^\(.\{80\}\).*\(.\{40\}\)/\1\_\2/' logfile

will perform inplace modification of your file, but it's limited to operating on all lines.

  • Also useful esp. with -i, as I may be looking to keep closer to only the first 1024 and last 1024 chars, which shouldn't modify most lines, only ridiculously long ones. Still the awk method lets me replace the middle with a " ... " easily in such cases.
    – Marcos
    Jun 18 '12 at 20:11
  • 1
    change \1\_\2 to \1 ... \2 to do the same with sed. HTH. Jun 18 '12 at 21:07
  • 1
    Why use \{120\} if the OP asked for first 80 characters? Your command gets the first 120 and last 40 characters. There is no need to worry about the substitute changing short lines because short lines will never match the substitute pattern. If you really wanted to make sure short lines were untouched, you could specify the /.\{120,\}/ address for the substitute command.
    – jw013
    Jun 18 '12 at 21:38
  • ah, because i got confused and though he asked for the first 120. thanks for pointing that out. :) Jun 18 '12 at 21:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.