0

I'd like to know if it's possible to convert the following format

{"timestamp":"1579053433","name":"1.10.207.64","value":"node-fnk.pool-1-10.dynamic.totinternet.net","type":"ptr"}
{"timestamp":"1579067130","name":"1.10.207.65","value":"node-fnl.pool-1-10.dynamic.totinternet.net","type":"ptr"}
{"timestamp":"1579098581","name":"1.10.207.66","value":"node-fnm.pool-1-10.dynamic.totinternet.net","type":"ptr"}

and end up with

1.10.207.64-1.10.207.64::node-fnk.pool-1-10.dynamic.totinternet.net;
1.10.207.65-1.10.207.65::node-fnl.pool-1-10.dynamic.totinternet.net;
1.10.207.66-1.10.207.66::node-fnm.pool-1-10.dynamic.totinternet.net;

The JSON file is 127 GB & 1,267,984,961 lines. The only thing that will change in each line is the IP & Hostname.

Improve this question
2
  • 1
    What have you tried, and how specifically was file size an issue?
    – steeldriver
    Feb 29, 2020 at 22:18
  • 1
    Yes. Use a json parser. (I can't remember the name of any, at this time. However I know there is at least one in the Debian repos, and in python library, and ...
    – ctrl-alt-delor
    Feb 29, 2020 at 23:28

2 Answers 2

Reset to default
0

You could try jq - for example, using the \(...) string interpolation

jq -r '"\(.name)-\(.name)::\(.value);"' file.json

As far as I know, it doesn't attempt to slurp the whole file unless you explicitly tell it to (using the -s / --slurp option), so while it may be slow, it shouldn't choke however large the file.

Improve this answer
6
  • Getting the following error c:\jq>jq -r '"(.name)-(.name)::(.value);"' file.json jq: error: syntax error, unexpected INVALID_CHARACTER, expecting $end (Windows cmd shell quoting issues?) at <top-level>, line 1: '(.name)-(.name)::(.value);' jq: 1 compile error I'm running jq in the same folder as the json file i want to convert.
    – jim456
    Mar 1, 2020 at 16:42
  • @jim456 if your question is about jq on Windows, perhaps you should ask elsewhere? This is a Unix and Linux site.
    – steeldriver
    Mar 1, 2020 at 17:08
  • I've got ubuntu running on a virtual machine now. How do i output the file with the new format once its finished.
    – jim456
    Mar 1, 2020 at 19:25
  • @jim456 jq will write to the terminal by default - if you want to capture the output in a new file, you will need to re-direct it like jq -r '...' > newfile
    – steeldriver
    Mar 1, 2020 at 19:56
  • i'm running in terminal jq -r '"\(.name)-\(.name)::\(.value);"' -r '/home/james/Desktop/json/xch.json' > '/home/james/Desktop/output.txt' would this be the correct way to output.
    – jim456
    Mar 1, 2020 at 22:32
0

As alluded to by ctrl-alt-delor in the comment, python is an option

For example, with a relatively recent python3 version

import sys, json

for line in sys.stdin:
    doc = json.loads(line)
    print(f'{doc["name"]}-{doc["name"]}::{doc["value"]}')

You would put this in a file (say parse_json.py) and run as follows:

python3 parse_json.py <json.file
Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .