0

I'm used to grep | uniq -c | sort -rn things, but this case is a little bit more complicate. If you copy a whatsapp web chat you'll get something like

[3:14 pm, 25/09/2020] James Smith: Hello!
[6:42 pm, 25/09/2020] John Doe: hi
[6:43 pm, 25/09/2020] James Smith: I was wondering..
if blah blah
and also blah
[6:45 pm, 25/09/2020] James Smith: blah blah blah blah...

I'm trying to get some statistical info about each person on the conversation, starting for just the char count (wc -c).

 74  James Smith
  2  John Doe

How can that be done? I.e., text count for each participant on chat. In the above example Smith contribute with 74 chars and Doe with 2.

To test I copy several lines of a WhatsApp Web chat that meets the hypotheses. To easily paste them to a file: xsel -b > filet_to_test.txt.

Valid assumptions:

  • Messages could have newlines.
  • Couldn't contain strings like [6:45 pm, 25/09/2020] at beginning of a 2nd+ line.
  • New message/record will start with something of the form [6:45 pm, 25/09/2020] , probably ^\[\d{1,2}:\d{2}\s[ap]m,\s\d{2}/\d{2}/\d{4}\]\s.
  • After the timestamp the usernames doesn't contain colons.

Would be more desirable a solution that could be extendable (maybe usingmiller), to for e.g. get the char/word count between range or for each day/hour of the week.

4
  • @Quasímodo Thanks for the feedback. I edit my question to make more clear what I'm trying to achieve. BTW I wonder why the straightforward downvote.
    – Pablo A
    Sep 30, 2020 at 20:57
  • @fra-san Well noted. I have assumed no newlines. In such case, even if someone sends a tag, there is no ambiguity. If there may be newlines, it is impossible to distinguish a new message.
    – Quasímodo
    Sep 30, 2020 at 21:27
  • "Couldn't contain strings like [6:45 pm, 25/09/2020]." > That is an artificial restriction. Surely any user can send a message with that. With this in mind, I don't think there is any possible general answer.
    – Quasímodo
    Sep 30, 2020 at 22:46
  • @Quasímodo I add that assumption to make the problem more simple, since is very unlikely. Any assumption is artificial. On the other hand, sending messages with newlines is pretty common. Maybe not in just one command but I'm sure is possible considering the added regex as the begginng of each record/message.
    – Pablo A
    Sep 30, 2020 at 22:52

2 Answers 2

2

A similar approach as @dani-garcia. First civilise your file into a tab separated one:

cat file | 
  tr "\n" "\000" | 
  sed "s/\x0\[/\n/g" | 
  sed "1 s/\[//; s/\]/\t/1; s/:/\t/2; s/\x0/ /g" |
  sed -E "s/(^[^,]+), ([0-9]{1,2}).([0-9]{2}).([0-9]{4})/\4-\3-\2 \1/" 

2020-09-25 3:14 pm      James Smith     Hello!
2020-09-25 6:42 pm      John Doe        hi
2020-09-25 6:43 pm      James Smith     I was wondering.. if blah blah and also blah
2020-09-25 6:45 pm      James Smith     blah blah blah blah...

by translating all \newlines to null

cat file | tr "\n" "\000" | 

then with sed globally reinsert \newlines wherever you have the pattern null[

sed "s/\x0\[/\n/g" | 

finally tidy up the individual lines by losing the leading [ on line 1

sed "1 s/\[//; 

replacing the first ] with a \tab

    s/\]/\t/1;

replacing the second : with a \tab

    s/:/\t/2;

and finally replace any remaining null with to avoid joining words together where they were originally separated by a \newline

    s/\x0/ /g"

and sort out your dates so they sort well

    sed -E "s/(^[^,]+), ([0-9]{1,2}).([0-9]{2}).([0-9]{4})/\4-\3-\2 \1/" 

Now you have your fields separated you can sort, group, count or whatever at will..

| awk -F'\t' '{chats[$2]++; words[$2]+=split($3,tmp," "); chars[$2]+=length($3)}
   END{for (who in chats){
     S=(chats[who]==1)?"":"s";
     s=(words[who]==1)?"":"s";
     print who" sent "chats[who]" message"S" with "words[who]" word"s" and "chars[who]" characters"}}'

James Smith sent 3 messages with 14 words and 73 characters
John Doe sent 1 message with 1 word and 2 characters
0
0

You could separate each message into three parts (date, person, message) and then use awk arrays indexed by the condition to be met, and finally print all values in the array, for example:

awk '{printf "%s%s", (NR>1&&/^\[.*\]/?"\n":""),$0}END{print " "}' test.txt | sed 's/\(^\[.*\]\) \(.*\): \(.*\)/\1\t\2\t\3/g' | awk 'BEGIN{FS="\t"} {arr[$2] =arr[$2]$3} END{for (i in arr) print length(arr[i]),i}'

being test.txt your input file.

Explanation:

The first command (awk '{printf "%s%s", (NR>1&&/^\[.*\]/?"\n":""),$0}END{print " "}' test.txt ) removes the newlines if the line does not start with [blahblahblah], i.e. it is not a new message, so the whole message will be in the same line.

The second command (sed 's/\(^\[.*\]\) \(.*\): \(.*\)/\1\t\2\t\3/g') separates each line in three parts: the date (with pattern [.*]), the person (between the date and the colon) and the message. Then it outputs each line with each part separated by a tab.

Lastly, the third command (awk 'BEGIN{FS="\t"} {arr[$2] =arr[$2]$3} END{for (i in arr) print length(arr[i]),i}') use awk arrays indexed by the person, and it outputs the length of the concatenation of the messages for each person.

Assumptions

  • the date is delimited by [ and ] and does not contain these symbols in between.
  • The person name does not contain a colon
  • Messages could contain strings like [6:45 pm, 25/09/2020] if they are not at the start of a new line.

I'm not familiarized with miller, but probably you can do something similar to your extendable desired solution changing the last awk command.

Probably this is not the most efficient way, but it works.

1
  • I have to make a deeper analysis to find what is wrong, bug just copying several lines of a WhatsApp chat that meets the hypotheses doesn't give the expected output. You can just select, copy and xsel -b > filet_to_test.txt to test it too.
    – Pablo A
    Oct 21, 2020 at 23:50

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .