1

I've got some JSON logs from AWS that I'm looking to clean up. They look like this...

    {
        "ingestionTime": 1568961184459, 
        "timestamp": 1568961184430, 
        "message": "START RequestId: 0304cf0d-da16-4d01-b4de-da8d528144ac Version: $LATEST\n", 
        "eventId": "34989003600358241981666605756070906630881684882471780352", 
        "logStreamName": "2019/09/20/[$LATEST]71cac888c4a54d11b7e2c97108ad8ba1"
    }, 
    {
        "ingestionTime": 1568961199507, 
        "timestamp": 1568961184432, 
        "message": "2019-09-20T06:33:04.432Z\t0304cf0d-da16-4d01-b4de-da8d528144ac\tAttempting to subscribe John Doe (johndoe@gmail.com) to newsletter...\n", 
        "eventId": "34989003600402843472063667020546010973018230982964936704", 
        "logStreamName": "2019/09/20/[$LATEST]71cac888c4a54d11b7e2c97108ad8ba1"
    }, 

Then I've got them up to this... by using...

{
  "ingestionTime": 1568961184459,
  "timestamp": 1568961184430,
  "message": "START RequestId: 0304cf0d-da16-4d01-b4de-da8d528144ac Version: $LATEST\n",
  "eventId": "34989003600358241981666605756070906630881684882471780352",
  "logStreamName": "2019/09/20/[$LATEST]71cac888c4a54d11b7e2c97108ad8ba1"
}
{
  "ingestionTime": 1568961199507,
  "timestamp": 1568961184432,
  "message": "2019-09-20T06:33:04.432Z\t0304cf0d-da16-4d01-b4de-da8d528144ac\tAttempting to subscribe John Doe (johndoe@gmail.com) to newsletter ...\n",
  "eventId": "34989003600402843472063667020546010973018230982964936704",
  "logStreamName": "2019/09/20/[$LATEST]71cac888c4a54d11b7e2c97108ad8ba1"
}

This command: aws logs filter-log-events --log-group-name /aws/lambda/$npm_package_name --region us-east-2 | jq '.events[]'

It would be nice if those \t's were actually tabs, and the \n at the end could just be removed. Or separating the message field into several new lines for each new tab.

How might I do this? I don't really know enough bash.

2
  • 1
    tr won't do it, because it's 2 characters not one. Have you tried sed?
    – Scottie H
    Sep 30 '19 at 22:18
  • 1
    I ended up using this combo of jq and tr aws logs filter-log-events --log-group-name /aws/lambda/$npm_package_name --region us-east-2 | jq -r '.events[].message' | tr '\t' '\n' and that actually worked quite nicely. Oct 1 '19 at 0:12
3

jq output JSON-encoded strings by default.

Would you want the raw message string, use -r or --raw-output:

jq -r .message file.json

(if file.json is the JSON document that you are showing). That would expand the tabs and newlines in that particular string.

5
  • I'm not trying to output to a file, just printing out on the terminal. Adding the -r flag to my current command doesn't change the output at all. Sep 30 '19 at 21:49
  • 1
    @Costa No it wouldn't, because, as I said, assumed you were running it on the data that you showed in the question. You never show the original data, only what was in the events array. You may want to try jq -r '.events[].message' on the output of your aws command (but I can for obvious reasons not test that).
    – Kusalananda
    Sep 30 '19 at 21:51
  • Let me post the original as well then. Sep 30 '19 at 21:55
  • 1
    @Costa Note that the JSON you added is a copy (albeit indented) of what was already in the question.
    – Kusalananda
    Oct 1 '19 at 8:00
  • That's just a sad reflection of my coding skills. I thought I had done something because jq had colorized the JSON output : ) Oct 1 '19 at 20:54
1

Parsing JSON in bash is never recommended, but ....

if you can massage the line:

    "message": "newline\n\nword\tword\tword", 

to become:

X_message="newline\n\nword\tword\tword"

then you can:

$ echo -e "$X_message"
newline

word    word    word

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.