Fetching, merging and exporting data from CURL

Question

Before anything else, english isn't my laguage. I'll try to be understandable :-) Second, I am not a regular Linux user, I'm just starting out.

I'm using curl and the API of a web chat platform to obtain the status of my team. Each member possess an ID. For each ID, I pipe the result in a text file agent-#.txt. So in my folder, if I have 5 members, I obtain 5 files: agent-1.txt, agent-2.txt and so on.

Each file contain this:

{
    "used": 0,
    "last": "2018-06-01 15:25:55",
    "presence": [],
    "away": false,
    "free": 3,
    "last_utc": "2018-06-01 13:25:55",
    "last_timezone": "Europe/Paris",
    "online": true,
    "offline": true
}

Next, using sed I delete the data I don't need:

sed -i -e '/{/d; /last/d; /presence/d; /last_utc/d; /last_timezone/d; /offline/d; /}/d; s/"//g; s/,//g; s/[ ]//g; /^\s*$/d' *

I obtain, for each file:

used:#
away:false
free:#
online:true

Where # is a number (total used and free slots for the agent).

Ok, but now I want to combine those data to obtain:

1. The total number of members whose status is online:true
2. The total number of free and used slot

Ideally I'm searching for a command that will output the total in another file. But I you have an idea (command, full script) to deal with the whole process, I'm also very interested!

Answer 1

Using your unmodified JSON files (before you run sed over them), and jq:

Number of people online:

jq -s 'map(select(.online))|length' agent-*.txt

Number of free slots:

jq -s '[.[].free]|add' agent-*.txt

Number of used slots:

jq -s '[.[].used]|add' agent-*.txt

All three as a single JSON document:

jq -s '{online: map(select(.online))|length, free: [.[].free]|add, used: [.[].used]|add}' agent-*.txt

Same as above, but just the numbers:

jq -s '(map(select(.online))|length), ([.[].free]|add), ([.[].used]|add)' agent-*.txt

Answer 2

You could concatenate the agent-*.txt files and process them by awk.

Note the -F : to use full column as field separator.

An incomplete example:

#!/bin/sh

awk -F : -f - agent-*.txt <<EOF
/^used:/{
    used+=\$2;
}
/^online:true/{
    online++;
}
END{
    printf("used: %d\n", used);
    printf("online: %d\n", online);
}
EOF

Probably, if you change carefully the matching expressions, you'll can skip the sed step.