My end goal is to have a script that will count the instances of each username in all files.
A username is a string, in quotes, that follows the string 'login'. For example, in one file, I might have:
{"this":"is', {"a":"strange"}, "type":{"of":"object", "but":"please"},
"go":"withit", "login":"username1"}
{"this":"is', {"login":"username2"}, "type":{"of":"object", "but":"please"},
"go":"withit"}
And in another file, I might have:
{"this":"is', {"a":"strange"}, "type":{"of":"object", "but":"please"},
"go":"withit", "login":"username3"}
{"login":"username1", "please":"gowithit"}
In which case, I'd like to have a txt file that contains a dict object with the count of the number of times each username appears in the files:
{"username1": 2, "username2":1, "username3":1}
I've read a few things to get me started, but I can't seem to put this together. I've sort of pseudocoded it, but I can't progress from this point.
I think I need to do this in two stages.
1) Get a list of all the usernames
2) Count the number of times each username appears in all files.
For task 1):
grep 'login:' * | sed 's/^.*: //'
#Except I think this gets everything from the line after 'login', which isn't what I want.
For task 2):
for all_usernames_in_file:
stringval = username_read_from_saved_file
cat * | grep -c $stringval > output.txt
Can anyone take it from here?
EDIT:
Do you mean I should do this:
grep -o 'login":"[^"]*"' /path/to/dir/* | cut -d'"' -f3 | sort | uniq -c | sed '1i{ s/\s*\([0-9]*\)\s*\(.*\)/"\2": \1,/;$a}' > output.txt
EDIT 2: Still not working. I'm trying to diagnose by understanding what each command does.
Let's say I'm just looking at this part to start:
grep -o 'login":"[^"]*"' /path/to/dir/* | cut -d'"' -f3 | sort | uniq -c > myfile.txt
Right now, myfile.txt
is blank.
Here's what I think this command is doing:
grep -o
matches non-empty parts of a matching line.
'login":"[^"]*"'
is the string we want grep to match. In the middle, the [^"]
matches any character after login":"
not equal to "
, and the *
says we want any length of match - that is, the length of the username doesn't matter, we want everything between the quotes.
|
is a pipe. It means "and then"
cut -d '"' -f3
means slice up the returned line (all stuff after login":"
), using the delimiter "
, and take field 3 (that is, just the username).
|
is a pipe. It means "and then"
sort
the usernames
|
is a pipe. It means "and then"
Get the unique usernames and count the number of times each appears.
If I take that much, and put a > myfile.txt
at the end, then I should end up with a txt file that contains usernames and a count of the number of times each appears. It won't be well-formatted, but it will exist.
Why am I not getting such a file?
NOTE: does it matter that I'm searching through .json.gz
formatted files? I've gotten the script to work when searching through txt
, but not through the other format.
{"this":"is',
a typo?jq
) fails due to stray objects without keys.