Tell me more ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.
up vote 2 down vote favorite
1

I keep a logfile of a program's output. The trouble is sometimes errors occur dumping very long lines of 7bit ascii data(encoded binary) I don't care to keep. These lines can be 200KB+ long before reaching a newline.

What might be a short, pipeable way eg. with sed to change only lines exceeding 250 characters long, by keeping only the first 80 and last 40 characters of that line, possibly replacing the middle with only a _?

share|improve this question

4 Answers

up vote 3 down vote accepted

I don't know sed well, so my solution is in awk:

awk 'length>250{len=length;$0=substr($0,1,80) "_" substr($0,len-40+1)};1' file
share|improve this answer
Nice and concise, thanks. (Too bad awk lacks a -i in-place file replace option like sed. I'll just awk ... file1 > file2 && mv -f file2 file1) – Marcos Jun 18 '12 at 18:42
Minor detail, I was able to make do without the extra len variable altogether – Marcos Jun 18 '12 at 20:04
1  
@Marcos Be careful with sed -i - it's not portable and most versions I've seen don't handle special files (like symlinks) correctly (it'll delete the link and replace it with a copy of the file). – jw013 Oct 3 '12 at 17:36
up vote 8 down vote

In sed, all commands can be prefixed by a condition that indicates what lines to apply the command to. A common kind of condition is a search pattern. The search pattern /.\{250\}/ matches lines with more than 250 characters. For such lines, match the first 80 characters and the last 40, and replace the whole line by the prefix, __ and the suffix.

sed -e '/.\{250\}/ s/^\(.\{80\}\).*\(.\{40\}\)$/\1__\2/'

You can even arrange for the pattern of the replacement command to match only sufficiently long lines.

sed -e 's/^\(.\{80\}\).\{130,\}\(.\{40\}\)$//'
share|improve this answer
I get your 1st example (though haven't tested if it still prints shorter lines unchanged). In your 2nd example, doesn't the middle search expression catch strings of exactly 130 length only (hence 250 char lines)? – Marcos Jun 19 '12 at 8:50
I think I see; what's needed is a variable-length regex between the 130 and 40 matchers. So this might be the final sed way: sed -i.bak -e 's/^\(.\{80\}\).\{130,\}.*\(.\{40\}\)$/\1 ... \2/' All those \ make it annoying to read, but I understand why they're necessary. The awk way is more human readable for someone not used to these regexes. – Marcos Jun 19 '12 at 9:00
1  
@Marcos; If your situiation allows you to use sed's extended regex (option -r), then you can use the clearer ( ) and { } ... The \( , etc method is more portable. – Peter.O Jun 19 '12 at 12:57
up vote 1 down vote

Here's an awk command that will do this:

awk 'len=length{if(len>250){print substr($0,0,80),"__",substr($0,len-40,len)}else{print $0}}' data.txt

Explanation:

If the line is longer than 250 characters long print the first 80 characters, followed by a string of ___ and then the last 40 characters.

If the line is shorter than 250 characters, simply print out the original line.

share|improve this answer
up vote 1 down vote

There is a pipeable way to extract the first 80, and last 40, characters of a line with sed, however sed does not have 'if statements', so there's no way directly in sed to test for string length and then perform an operation on it.

Capturing the first 80 and last 40 characters of all lines can be done with | sed -e 's/^\(.\{80\}\).*\(.\{40\}\)/\1\_\2/'.

sed -i -e 's/^\(.\{80\}\).*\(.\{40\}\)/\1\_\2/' logfile

will perform inplace modification of your file, but it's limited to operating on all lines.

share|improve this answer
Also useful esp. with -i, as I may be looking to keep closer to only the first 1024 and last 1024 chars, which shouldn't modify most lines, only ridiculously long ones. Still the awk method lets me replace the middle with a " ... " easily in such cases. – Marcos Jun 18 '12 at 20:11
1  
change \1\_\2 to \1 ... \2 to do the same with sed. HTH. – Tim Kennedy Jun 18 '12 at 21:07
1  
Why use \{120\} if the OP asked for first 80 characters? Your command gets the first 120 and last 40 characters. There is no need to worry about the substitute changing short lines because short lines will never match the substitute pattern. If you really wanted to make sure short lines were untouched, you could specify the /.\{120,\}/ address for the substitute command. – jw013 Jun 18 '12 at 21:38
ah, because i got confused and though he asked for the first 120. thanks for pointing that out. :) – Tim Kennedy Jun 18 '12 at 21:43

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.