1

We have this sample curl command that retrieves data as output. Aside from that, that output also contains another URL that can be used again to perform curl command to get the next data. Please help in creating a bash script that automatically uses the next URL in the response, in script to programmatically return events from next page.

Example:

curl -H 'Authorization: bearer ' -XGET 'https://.loggly.com/apiv2/events/iterate?q=*&from=-10m&until=now&size=1'

Output sample:

{
"events": [
   {
       "raw": "{\"function\": \"handle\", \"publish_success\": 83, \"timestamp\": \"17-02-01 00:58:04,852149\", \"start_time\": 1485910682.351175, \"args\": \"\", \"duration\": 2501, \"message\": \"end publish run\", \"publish_fail\": 0, \"level\": \"INFO\", \"source_duration\": {\"duration\": 1, \"end_time\": 1485910682.351801}, \"feature_duration\": {\"duration\": 1427, \"end_time\": 1485910683.786751}, \"pathname\": \"/opt/loggly/web/app/alert/management/commands/alerterd.py\", \"lineno\": 329, \"cache_duration\": {\"duration\": 1, \"end_time\": 1485910684.821093}, \"action\": \"alerterd\", \"update_duration\": {\"duration\": 8, \"end_time\": 1485910684.852121}, \"end_time\": 1485910684}",
       "logtypes": [
          "json",
          "syslog"
       ],
       "timestamp": 1485910684852,
       "unparsed": null,
       "logmsg": "{\"function\": \"handle\", \"publish_success\": 83, \"timestamp\": \"17-02-01 00:58:04,852149\", \"start_time\": 1485910682.351175, \"args\": \"\", \"duration\": 2501, \"message\": \"end publish run\", \"publish_fail\": 0, \"level\": \"INFO\", \"source_duration\": {\"duration\": 1, \"end_time\": 1485910682.351801}, \"feature_duration\": {\"duration\": 1427, \"end_time\": 1485910683.786751}, \"pathname\": \"/opt/loggly/web/app/alert/management/commands/alerterd.py\", \"lineno\": 329, \"cache_duration\": {\"duration\": 1, \"end_time\": 1485910684.821093}, \"action\": \"alerterd\", \"update_duration\": {\"duration\": 8, \"end_time\": 1485910684.852121}, \"end_time\": 1485910684}",
       "id": "7ce48bda-e819-11e6-808b-12a1c1f6d2c3",
       "tags": [],
       "event": {
           "syslog": {
               "severity": "Informational",
               "appName": "msg",
               "timestamp": "2017-02-01T00:58:04.852+00:00",
               "facility": "local use 1",
               "priority": "142",
               "host": "127.0.0.1"
           },
           "json": {
               "function": "handle",
               "publish_success": 83,
               "level": "INFO",
               "timestamp": "17-02-01 00:58:04,852149",
               "start_time": 1485910682.351175,
               "cache_duration": {
                   "duration": 1,
                   "end_time": 1485910684.821093
           },
           "update_duration": {
               "duration": 8,
               "end_time": 1485910684.852121
           },
           "duration": 2501,
           "pathname": "/opt/loggly/web/app/alert/management/commands/alerterd.py",
           "end_time": 1485910684,
           "source_duration": {
               "duration": 1,
               "end_time": 1485910682.351801
           },
           "action": "alerterd",
           "message": "end publish run",
           "publish_fail": 0,
           "feature_duration": {
               "duration": 1427,
               "end_time": 1485910683.786751
           },
           "lineno": 329
           }
       }
   }
],
"next": "https://<subdomain>.loggly.com/apiv2/events/iterate?next=eea25ee6-0e48-4428-a544-36d6441d132c"

}

so we're hoping to run another command again like this:

curl -H 'Authorization: bearer ' -XGET 'https://.loggly.com/apiv2/events/iterate?next=eea25ee6-0e48-4428-a544-36d6441d132c'

0

2 Answers 2

1

Have a look at the jq command. It is found in most repositories these days if it isn't already installed. It's for parsing JSON from the command line. You can read more in the man pages or here:

https://jqlang.github.io/jq

For example, if you pipe the output from your curl command to jq like this:

   curl .... | jq '.next' 

You will get just the URL value surrounded in quotes like this:

"https://<subdomain>.loggly.com/apiv2/events/iterate?next=eea25ee6-0e48-4428-a544-36d6441d132c"

You can test that value for validity or pass it to another curl or whatever you need.

1
  • 2
    You'll want jq -r '.next' to strip off the quotes. Aug 17, 2023 at 1:43
0

You should use something like jq, fx, or jshon to parse the .json and extract the .next portion.

jq: My go to JSON parser, simple and quick.

fx: JSON parser designed to be used with curl.

jshon: Really fast and light JSON parser.

Then your bash script (using jq) can be as simple as:

#!/bin/bash
JSONF="filewithurl.json"
curl  $(jq -r ".next" < ${JSONF}) -H 'Authorization: bearer '

Note this is untested because your url was not complete, I assume for security or code disclosure reasons.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .