I'd like to know if it's possible to convert the following format


and end up with;;;

The JSON file is 127 GB & 1,267,984,961 lines. The only thing that will change in each line is the IP & Hostname.

  • 1
    What have you tried, and how specifically was file size an issue? Feb 29 '20 at 22:18
  • 1
    Yes. Use a json parser. (I can't remember the name of any, at this time. However I know there is at least one in the Debian repos, and in python library, and ... Feb 29 '20 at 23:28

You could try jq - for example, using the \(...) string interpolation

jq -r '"\(.name)-\(.name)::\(.value);"' file.json

As far as I know, it doesn't attempt to slurp the whole file unless you explicitly tell it to (using the -s / --slurp option), so while it may be slow, it shouldn't choke however large the file.

  • Getting the following error c:\jq>jq -r '"(.name)-(.name)::(.value);"' file.json jq: error: syntax error, unexpected INVALID_CHARACTER, expecting $end (Windows cmd shell quoting issues?) at <top-level>, line 1: '(.name)-(.name)::(.value);' jq: 1 compile error I'm running jq in the same folder as the json file i want to convert.
    – jim456
    Mar 1 '20 at 16:42
  • @jim456 if your question is about jq on Windows, perhaps you should ask elsewhere? This is a Unix and Linux site. Mar 1 '20 at 17:08
  • I've got ubuntu running on a virtual machine now. How do i output the file with the new format once its finished.
    – jim456
    Mar 1 '20 at 19:25
  • @jim456 jq will write to the terminal by default - if you want to capture the output in a new file, you will need to re-direct it like jq -r '...' > newfile Mar 1 '20 at 19:56
  • i'm running in terminal jq -r '"\(.name)-\(.name)::\(.value);"' -r '/home/james/Desktop/json/xch.json' > '/home/james/Desktop/output.txt' would this be the correct way to output.
    – jim456
    Mar 1 '20 at 22:32

As alluded to by ctrl-alt-delor in the comment, python is an option

For example, with a relatively recent python3 version

import sys, json

for line in sys.stdin:
    doc = json.loads(line)

You would put this in a file (say parse_json.py) and run as follows:

python3 parse_json.py <json.file

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.