3
{
  "auth": 'log',
  "appid": 21,
  "custid": "599c1f910f53ada8468b4567",
  "hwid": "59e719ba0f53adfd6a8b4597"
}

24/10/2017 12:44:24
--------------------------------------------------------------------------------
{
  "auth": 'log',
  "appid": 21,
  "custid": "599c1f910f53ada8468b4567",
  "hwid": "59e719ba0f53adfd6a8b4597"
}

I need to transform a list of jsons delimited by some log lines from a file into a single json . so far i've tried this:

tac tst.txt | sed '/---------/I,+2 d' | tac > out.json

and managed to delete the log lines, but I need to replace them with a comma and add them into a list. How can I do this ? or there is any alternative for this using python ?

  • 1
    Can you put an expected output? – tachomi Oct 24 '17 at 14:01
  • [{ "auth": 'log', "appid": 21, "custid": "599c1f910f53ada8468b4567", "hwid": "59e719ba0f53adfd6a8b4597" }, { "auth": 'log', "appid": 21, "custid": "599c1f910f53ada8468b4567", "hwid": "59e719ba0f53adfd6a8b4597" }] – xRow Oct 24 '17 at 14:05
  • but just the comma instead of the log lines for this command would be ok for me – xRow Oct 24 '17 at 14:06
  • 1
    Note, 'log' is NOT valid json value. An additional replacement to "log" should be considered – RomanPerekhrest Oct 24 '17 at 14:06
2

sed + jq solution:

sed -E "/^(---|[0-9][0-9])/d; s/'([^']+)'/\"\1\"/" tst.txt | jq -s '' out.json
  • /^(---|[0-9][0-9])/d - delete the unnecessary lines which start either with --- or 2 digits

  • s/'([^']+)'/\"\1\"/" - replace a value in single quotes to be a valid JSON value (enclosed in double quotes)

  • jq -s '' - instead of running the filter for each JSON object in the input, read the entire input stream into a large array


The out.json contents:

[
  {
    "auth": "log",
    "appid": 21,
    "custid": "599c1f910f53ada8468b4567",
    "hwid": "59e719ba0f53adfd6a8b4597"
  },
  {
    "auth": "log",
    "appid": 21,
    "custid": "599c1f910f53ada8468b4567",
    "hwid": "59e719ba0f53adfd6a8b4597"
  }
]
  • Very helpful, also thank you for introducing me to jq :) – xRow Oct 24 '17 at 14:36
  • sry, I accepted yesterday but my JS was disabled, I think now is ok – xRow Oct 25 '17 at 12:19
1

Assuming that the quotes around log are corrected to double quotes, using sed:

sed -e '1s/^/[/' \
    -e '$s/$/]/' \
    -e 's/^-.*$/,/' \
    -e '/^[0-9]/d' file | jq .

The sed expressions:

  1. Insert a [ in the beginning of the first line of input.
  2. Insert a ] in the end of the last line of input.
  3. Replace any line starting with - with a single comma.
  4. Delete lines starting with any digit.

The call to jq is optional (it only sorts out the formatting).

Output:

[
  {
    "auth": "log",
    "appid": 21,
    "custid": "599c1f910f53ada8468b4567",
    "hwid": "59e719ba0f53adfd6a8b4597"
  },
  {
    "auth": "log",
    "appid": 21,
    "custid": "599c1f910f53ada8468b4567",
    "hwid": "59e719ba0f53adfd6a8b4597"
  }
]

Short alternative using jq -s or jq --slurp:

sed -e '/^[-0-9]/d' file | jq -s

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.