Take the subset of the file related to the time stamp

Question

I have some json file in dictionary format. The lines looks like here:

{"a":"1", "b":"2", "c":"3", "time":1334572551435}
{"a":"1", "b":"2", "c":"4", "time":1334575352456}
{"a":"2", "b":"2", "c":"7", "time":1334575335345}
...

time is in a UTC format. The whole file has about 300 milliones unique lines (anyway the same time could come twice). How I can choose a lines for some particular time, for example between 1334575352456 and 1334575353456?

I personally have the next idea from some tutorial:

awk ’$"time" == 1334575352456, $"time" == 1334575353456’ inputfile.json

anyway, I suppose this solution is for column "time", not for dictionary with the key "time" (Actually $"time" should be $4)

Answer 1

Try:

$ awk -F'[:}]' '$(NF-1) >= 1334575352456 && $(NF-1) <= 1334575353456' file
{"a":"1", "b":"2", "c":"4", "time":1334575352456}

-F'[:}]' set fields delimiter is : or }, so you can access the the time value by accessing second from last field, using $(NF-1).

To save ouput, use:

$ awk -F'[:}]' '$(NF-1) >= 1334575352456 && $(NF-1) <= 1334575353456' file > output.txt

If you have gawk 4.1.0 and above, you can use -i option to make edit inplace:

$ awk -i inplace -F'[:}]' '$(NF-1) >= 1334575352456 && $(NF-1) <= 1334575353456' file

Answer 2

Since the input is a set of JSON objects, it makes sense to use a JSON-aware tool.

Using the jq JSON processor to extract only the entries that have a time value strictly between some t0 and t1 time values (the -c option used here selects "compact" output):

jq -c --argjson t0 1334572551435 --argjson t1 1334575352456 \
    'select(.time > $t0 and .time < $t1)' file

The equivalent command with Miller (mlr), albeit without parametrising the start and end of the time range:

mlr --json filter '$time > 1334572551435 && $time < 1334575352456' file