2

So far I've been able to get around sed's more advanced features like look-ahead/look-behind across multiple lines but I'd like to understand how tasks like the following can be achieved with sed as I have the feeling that my approach of doing this e.g. within python is not necessary and can also be done within a filter pipeline on cmd.

a stripped example of the data coming in is like this:

1b41cf70 0
1cb8dd19 1
620f0b67 2
620f0b67 3
f35d35fe 4
3a6fb62a 5
620f0b67 6
620f0b67 7
620f0b67 8
b958a7ea 9
f35d35fe 10
f35d35fe 11
620f0b67 12

the first columns width is always identical (contains a shortened hash-value) and the second columns content is completely odered, numerical and without gaps (and therefore potentionally unnecessary besides providing orientation on lenghtier lists).

the desired output would be either like this (putting the index of the last consecutive occurance into an additional column):

1b41cf70 0
1cb8dd19 1
620f0b67 2 3
f35d35fe 4
3a6fb62a 5
620f0b67 6 8
b958a7ea 9
f35d35fe 10 11
620f0b67 12

or even better with aggregated numbers of repeating values (math expressions (adding) seem to be more easily done with awk but my skills with that are even worse, so this is just to illustrate what other outcomes would be desireable):

1b41cf70 0
1cb8dd19 1
620f0b67 2 +1
f35d35fe 4
3a6fb62a 5
620f0b67 6 +2
b958a7ea 9
f35d35fe 10 +1
620f0b67 12

I tried to follow several similar but yet different questions found across the SO-space but could not wrap my head around possibly simpler parts possibly leading to a solution like why sed '$!N;/^\([^\ ]\+\)\ [0-9]\+\n\1\ /{P;d}' sampledata will cut lines with indexes 3,7,11 but not 8.

My system has GNU sed version 4.8 and awk version 5.1.0 installed and I'd love to learn how I could use one of them to get this done. And no, this is not homework but lengthy lists of hashes with plenty of redundancy that need to be compacted and compared. ;)

1
3

Completely ignoring the original second column, we may use uniq -c to count the number of times a string occurs repeated on consecutive lines.

Taking the two-field output from uniq -c, we may create a third field whenever a string occurs repeated more than once (on the form +x where x is the number of times the field occurs, minus one). We then rearrange the two first fields and print.

cut -d ' ' -f 1 file |
uniq -c |
awk '$1 > 1 { $3 = "+" $1 - 1 } { nr += $1; $1 = $2; $2 = nr - 1 - $3; print }'

The nr variable represents the line number in the original file.

Output given the data in the question:

1b41cf70 0
1cb8dd19 1
620f0b67 2 +1
f35d35fe 4
3a6fb62a 5
620f0b67 6 +2
b958a7ea 9
f35d35fe 10 +1
620f0b67 12
3
  • Thanks for this elegant approach which seems almost readable to me as an awk-noob. However, the results I'm getting with your commands do slightly differ to the output you wrote as the lines containing the span-count (beginning with +) carry the upper instead of the lower line-number in $2 in my case, namely 620f0b67 3 +1 instead of 620f0b67 2 +1, 620f0b67 8 +2 instead of 620f0b67 6 +2 and f35d35fe 11 +1 instead of f35d35fe 10 +1. any idea, how to fix this? my guess is that var nr needs to be adjusted in the first set of curly braces but left intact within the second?
    – antiplex
    Aug 3 at 8:37
  • 1
    @antiplex Sorry, at some point the code in the text and the result in the text started to diverge. I've fixed it now by subtracting $3 from the value of $2 in the code. Thanks for the heads-up!
    – Kusalananda
    Aug 3 at 9:16
  • 2
    while its said "the second columns content is completely odered, numerical and without gaps" at least you need to know the first number value before. i, e, you need open the file and read the second column of the first line then adjust the codes accordingly. maybe that's not 0, it's start from 10, or -20 etc Aug 3 at 12:42
3

Using awk:

awk 'function prnt() { print buf, preV; preK=$1; preV=""; buf=$0 }
preK!=$1             { prnt(); next } { preV=$2 }
END                  { prnt() }' infile

Output:

1b41cf70 0
1cb8dd19 1
620f0b67 2 3
f35d35fe 4
3a6fb62a 5
620f0b67 6 8
b958a7ea 9
f35d35fe 10 11
620f0b67 12

awk 'function prnt() { print buf, (c?"+"c:""); preK=$1; c=0; buf=$0 }
preK!=$1             { prnt(); next } { c++ }
END                  { prnt() }' infile

Output:

1b41cf70 0
1cb8dd19 1
620f0b67 2 +1
f35d35fe 4
3a6fb62a 5
620f0b67 6 +2
b958a7ea 9
f35d35fe 10 +1
620f0b67 12
5
  • sweet, now I have an idea on how to use variables as buffers in awk and do some conditional branching! what I don't get how the counter is initialized (can one increment an empty string ("") in awk?) and what the c:"" part in (c?"+"c:"") exactly does.
    – antiplex
    Aug 3 at 9:09
  • 1
    @antiplex in no such way we can say you want increment a string,it has no meaning for me at all. this c?"+"c:"" is known as ternary condition written as condition?retun-this-if-true:return-this-if-false, so it checks if variable c is not zero then it return character + followed by its value "+"c, else it returns "" empty string, and then we print the result. Aug 3 at 10:10
  • indeed, incrementing a string is a nonsense term, sorry for being not precise here. what I meant is that I wonder where/how c is initialized and to what value. all related assignments I were able to identify were c="" (line1) and c++ (line2) and I was wondering what magic drives awk to seemingly turn an empty string into an int. thanks for clarifying the ternary condition-syntax, I was kind of close I guess but missed the details.
    – antiplex
    Aug 3 at 11:58
  • @antiplex ahhh, that should be c=0 while emptying variables like c="" awk do covert the value of that var to string but then when we do c++ it converts to number. and by default variables value is empty string and will be 0 if anywhere it used as number and no initialization is required (at least for your question) and of course for some other conditions we better do it. if you look into my answer revision history I removed NR==1 and left only preK!=$1, because you didn't have first column anywhere which it is 0 or empty otherwise for those lines condition preK!=$1 will be false... Aug 3 at 12:27
  • ... because preK has not initialized, so it will be empty string whenever if $1 was empty, or wil be 0 when $1 also was 0 (that is awk job to perform string or number comparison/conversion, but also we can force it to tell aak which to use), so then its followed block will not be run that's why I was used NR==1 to ensure it will be run on first line and then so followed block will be executed and then variables will be initialized from first line info. now I changed that c="" to c=0 to reduce beautiful girl (awk) job a bit for conversion : ) thanks for heads up! Aug 3 at 12:31
2

You asked for sed. Here are 2 versions close to your own attempt but using POSIX ERE extended regular expressions. Both keep a maximum of 2 lines in the pattern space.

sed -E '
    :Q
    $!N
    /^([^ ]+) ([0-9]+)( [0-9]+)?\n\1 ([0-9]+)$/{
        s//\1 \2 \4/
        bQ
    }
    P
    D
' -- file

where:

  • unless on last line ($!) appends a newline and next line to current (N)
  • the match expression /…/ captures fields 1 and 2 as \1 and \2, a possible last index as \3, and finally next line's index as \4
  • if field 1 repeats on next line the entire pattern space is replaced with field 1 (hash), field 2 (first index) and last index, and branches to start of script -- the empty regex in the s command reapplies the last regex used (in /…/)
  • otherwise prints and deletes first line (P;D;) and resumes loop

Output:

1b41cf70 0
1cb8dd19 1
620f0b67 2 3
f35d35fe 4
3a6fb62a 5
620f0b67 6 8
b958a7ea 9
f35d35fe 10 11
620f0b67 12

If instead:

/^([^ ]+) ([0-9]+)( ([+]+))?\n\1 [0-9]+$/{
    s//\1 \2 \4+/

output becomes:

1b41cf70 0
1cb8dd19 1
620f0b67 2 +
f35d35fe 4
3a6fb62a 5
620f0b67 6 ++
b958a7ea 9
f35d35fe 10 +
620f0b67 12

sed isn't keen on counting (but it can be done).


Lastly, a few comments on your sed script which uses POSIX BREs

  • don't escape characters inside []s except for the escape char., ], and possibly -
  • in BRE + isn't a quantifier but a plain plus sign
  • no need to escape the space character
  • for portability use a semicolon before } ending a list of editing commands
  • the d command deletes the entire pattern space not just up to the first newline
1

The following will work using any awk in any shell on every Unix box:

$ cat tst.awk
$1 != prev {
    if ( NR > 1 ) {
        prt()
    }
    prev = $1
    beg  = $2
}
{ end = $2 }
END { prt() }

function prt() {
    print prev, beg (beg == end ? "" : OFS "+" (end-beg))
}

$ awk -f tst.awk file
1b41cf70 0
1cb8dd19 1
620f0b67 2 +1
f35d35fe 4
3a6fb62a 5
620f0b67 6 +2
b958a7ea 9
f35d35fe 10 +1
620f0b67 12

or if you prefer:

$ cat tst.awk
$1 != prev {
    if ( NR > 1 ) {
        prt()
    }
    prev = $1
    beg  = $2
}
{ end = $2 }
END { prt() }

function prt() {
    print prev, beg (beg == end ? "" : OFS end)
}

$ awk -f tst.awk file
1b41cf70 0
1cb8dd19 1
620f0b67 2 3
f35d35fe 4
3a6fb62a 5
620f0b67 6 8
b958a7ea 9
f35d35fe 10 11
620f0b67 12
0

Just a quick obfuscating multiline find-replace solution (this time in Perl)

perl -0pe 's/(\w+) (\d+)(\n\1 (\d+))+/$1 $2 $4/g' file

The correspondent (gnu)sed version could be...

sed -rz 's/(\w+) ([0-9]+)(\n\1 ([0-9]+))+/\1 \2 \4/g' file

For the "+" output, we have to do some extra calcution:

perl -0pe 's/(\w+) (\d+)(\v\1 (\d+))+/"$1 $2 +" . ($4-$2)/ge' file
0

Using awk:

datamash groups the first field with unique values of second field. And awk prints the desired output.

$ datamash -W -g 1 unique 2 <input | 
awk '{gsub(/,[0-9]{1,},|,/, " ")}1'
1b41cf70 0
1cb8dd19 1
620f0b67 2 3
f35d35fe 4
3a6fb62a 5
620f0b67 6 8
b958a7ea 9
f35d35fe 10 11
620f0b67 12
$ datamash -W -g 1 unique 2 <input | 
awk '{s=gsub(/,[0-9]{1,}/, ""); 
print (s) ? $1 OFS $2 OFS "+" s : $1 OFS $2}'
1b41cf70 0
1cb8dd19 1
620f0b67 2 +1
f35d35fe 4
3a6fb62a 5
620f0b67 6 +2
b958a7ea 9
f35d35fe 10 +1
620f0b67 12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.