0

I already know what bits and pieces of this command do:

zgrep -v "org" /path/to/files/* | zgrep "FollowEvent" | zgrep -o 'login":"[^"]*"' | cut -d'"' -f3 | sort | uniq -c | sed '1i{
       s/\s*\([0-9]*\)\s*\(.*\)/"\2": \1,/;$a}' > usernames_followevents.txt

Piece by piece:

1) zgrep is used to grep (search) through .json.gz files

2) zgrep -v "org" /path/to/files/* means find entries in each file in /path/to/files/* that do not contain "org".

3) | is a pipe; it means "and then"

4) zgrep "FollowEvent" means find the string "FollowEvent" within the found results from the first zgrep.

5) | is a pipe; it means "and then"

6) zgrep -o 'login":"[^"]*"' means find nonempty matches for the string login":" and all text following the word `login' in the entry.

7) | cut -d'"' -f3 means "and then take only the third field from the resulting match", which is, in this case, a username.

8) | sort | uniq -c means "and then sort the usernames, and then count the number of unique instances of each username".

So far, we have:

zgrep -v "org" /path/to/files/* | zgrep "FollowEvent" | zgrep -o 'login":"[^"]*"' | cut -d'"' -f3 | sort | uniq -c

which is finding, in all entries in all files in /path/to/files/* that do not contain the string "org", but do contain the string "FollowEvent", all usernames (which is text in the third field following "login"), and then sorting these usernames and counting the number of times each username appears.

My problem is with this part:

sed '1i{ s/\s*\([0-9]*\)\s*\(.*\)/"\2": \1,/;$a}'

I know (or think I know) this much:

1) sed is a stream editor that allows manipulation of text.

2) sed '1i{ means "in the preceding line, insert {"

3) All together, this command returns {"username":count of that username} for all usernames in all files as discussed earlier. It then puts these in a file called usernames_followevents.txt.

4) The part "\2": means "put double quotes around the username, which is the second field (?) and then insert a :".

I would like to manipulate the sed command, but without understanding the rest of the details, I can't begin to make modifications.

Could anyone please explain what each part of the sed command is doing?

4
  • I'm fairly good at sed, but 1i{...} makes no sense to me, because i means just to insert the text after it before line 1 is printed. Did you paste it correctly? Edit: I think your missing newlines, white space is relevant in sed.
    – seshoumara
    Jun 2, 2019 at 18:54
  • I don't think sed 1i{ means "insert a {". The braces are used to allow multiple sed expressions for a single address. In this case the s/ indicates a substitution. The expression is designating the first group \1 as any size group of integers in the input file: [0-9]*, possibly surrounded by whitespace. Can you elaborate on what $a is set to, I don't know that part but here it is going to run as an additional sed expression after the substitution as far as I can tell.
    – Jeff
    Jun 2, 2019 at 19:15
  • @JeffH. Normally yes, but only if the brace is immediately after a line address. What you have here is 1i and then the text to insert. Please see my answer of what I think that sed code meant.
    – seshoumara
    Jun 2, 2019 at 19:53
  • The fact that it was written incorrectly to start led to my confusion, thanks for clarifying @seshoumara. That's why I didn't want to submit an answer to this one yet.
    – Jeff
    Jun 2, 2019 at 21:31

1 Answer 1

2

The way that sed command is written now is incorrect. It should be either a script like this:

1i{
s/\s*\([0-9]*\)\s*\(.*\)/"\2": \1,/
$a}

or in a single line like this:

sed -e '1i{' -e 's/\s*\([0-9]*\)\s*\(.*\)/"\2": \1,/' -e '$a}'

Literally everything you put after commands i and a until a newline or end of expression (with -e) is directly printed on the standard output.

As for what it does, let's break it down:

1i{

1 is a line address. It tells sed when to execute a command. When the content of the first line is read into the pattern space (without the trailing newline), it inserts '{' on the standard output on a separate line. Note that the pattern space is unaltered, it didn't add the '{' to it.

s is the search and replace command, the most versatile command in sed. \s matches a white space. \(regex\) is grouping the regex inside it like in math, but also stores what was matched by it in a numeric register based on the order of that group: \1 to \9.

The output of uniq -c is something like this:

    occurrences string
    3 user

Now the complicated part:

\s*\([0-9]*\)\s*\(.*\)

Still at line 1. The pattern space is a bunch of spaces, then '3 user'. To match this we search for white space zero or more times, then a digit many times, that is a number, (in my opinion it should have been + instead of *) that is stored in register \1, then space (* is not needed I think), then any character many times (again + would have been better) that is stored in register \2. So now, occurrence is in \1 and string/user in \2.

"\2": \1,

The whole line was matched and pieces were stored, now we replace what was matched with quote, then user, then quote, colon, space, occurrence and comma.

$a}

$ is also a line address. If the current line is the last one, at the moment not, call the a command to append '}' to the standard output on a separate line.

This is the end of code processing for this line, automatic printing of the pattern space such as it is after the manipulation is done, then the content of the second line is read and the whole cycle repeats.

Example output:

{
"user": 3,
}

That is basically a JSON file format, though improperly indented.

That's it. Sorry for writing novels :)

2
  • THANK YOU for writing novels! This was very helpful. I noticed you said at the beginning that the command was written incorrectly, but I literally copy/pasted the line from a working .sh file. Maybe spacing isn't so important? Jun 3, 2019 at 13:24
  • 1
    @StatsSorceress I think somehow there should be a newline in the .sh script after 1i{, otherwise it would not work. I'm glad it was helpful.
    – seshoumara
    Jun 3, 2019 at 22:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.