I have a JSON log file test.log:

{"type":"1","fname":"AAA","lname":"bbb" }    
{"type":"2","state":"gujarat" }     
{"type":"3","city":"ahmedabad" }    
{"type":"3","city":"rajkot" }   
{"type":"1","fname":"ccc","lname":"ddd" }    

I run the following commands

cat test.log | jq -c "select(.type==\"1\")|{fname:.fname}" > type_1.txt     
cat test.log | jq -c "select(.type==\"3\")|{city:.city}" > type_3.txt   

Is it possible for a single jq command to generate multiple output files?

I want to run the jq command only one time on the test.log file and generate an output file for each item type.

  • 1
    As far as I know, jq can't even write one file by itself... With extra assumptions (e.g., the input is much larger than the output), one could arrange something to keep and tag both types you are interested it, and filter them subsequently into two files. – dhag Oct 27 '15 at 19:59

According to the IO section of the manual as of version 1.5 jq can only write to the standard output and print debug messages to stderr.

What you could do if you don't want to run jq on test.log multiple times, e.g, because it is a huge file, is

jq -c '(select(.type=="1") | {fname:.fname}), (select(.type=="3") | {city:.city})' test.log > temp
# The file 'temp' contains one entry per line.
grep '^{"fname"' temp > type_1.txt
grep '^{"city"' temp > type_3.txt   
rm temp

It may be easier to use a full-blown language here. Like:

perl -MJSON -ne '
  BEGIN{$j = JSON->new}
    for $i ($j->incr_parse) {
      $t = delete($i->{type});
      $out{$t} .= to_json($i) . "\n";
    for $t (keys %out) {
      open OUT, ">", "type_$t.json" or die "type_$t.json: $!\n";
      print OUT $out{$t};
  }' < input.json

jq only writes to stdout (or stderr), so you need another program for splitting the stream into different logfiles.
First you enrich the js command of @nwk with a logfile field.

jq -c '(select(.type=="1") | {"logfile":.type},{fname:.fname}),
       (select(.type=="3") | {"logfile":.type},{city:.city})' test.log

You can pipe this output to awk, that can pick up the filenames

jq -c '(select(.type=="1") | {"logfile":.type},{fname:.fname}),
       (select(.type=="3") | {"logfile":.type},{city:.city})' test.log |
       awk -F"\"" 'BEGIN {FPAT="[^{}:\"]+" } 
            $1=="logfile" {lf="type_"$2".txt"; next}
            {print >> lf}'

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.