1

I have thousands of JSON files that look like:

file1 (key1:value_list1)

{"2mac:acg":["1-248","3-245","3-246","4-245","4-246","5-245","5-246","6-243","6-245","6-246","6-247","6-296","7-245","7-295","7-296","8-236","8-239","8-240","8-294","8-295","8-296","9-235","9-236","9-239","9-294","10-293","10-294","10-295","11-15","11-16","11-293","11-294","12-16","12-290","12-291","12-292","12-293","12-294","13-25","13-26","13-27","13-28","13-290","13-292","13-293","14-24","14-25","14-26","14-27","14-290","15-24","15-25","16-24","16-25","16-233","16-234","16-235","17-22","17-23","17-24","17-25","17-59","17-233","17-234","17-235","18-22","18-23","18-24","18-25","18-43","18-213","18-214","18-215","18-229","18-230","18-232","18-233","18-234","19-42","19-43"]}

file2 (key2:value_list2)

{"4qld:aaa":["3-245","3-246","4-245","4-246","5-245","5-246","6-243","6-245","6-246","6-247","6-296","7-245","7-295","7-296","8-236","8-239","8-240","8-294","8-295","8-296","9-235","9-236","9-239","9-294","10-293","10-294","10-295","11-15","11-16","11-293","11-294","12-16","12-290","12-291","12-292","12-293","12-294","13-25","13-26","13-27","13-28","13-290","13-292","13-293","14-24","14-25","14-26","14-27","14-290","15-24","15-25","16-24","16-25","16-233","16-234","16-235","17-22","17-23","17-24","17-25","17-59","17-233","17-234","17-235","18-22","18-23","18-24","18-25","18-43","18-213","18-214","18-215","18-229","18-230","18-232","18-233","18-234","19-42","19-43","19-55"]}

file3 (key3:value_list3)

{"6k8h:c":["1-248","2-134","3-245","3-246","4-245","4-246","5-245","5-246","6-243","6-245","6-246","6-247","6-296","7-245","7-295","7-296","8-236","8-239","8-240","8-294","8-295","8-296","9-235","9-236","9-239","9-294","10-293","10-294","10-295","11-15","11-16","11-293","11-294","12-16","12-290","12-291","12-292","12-293","12-294","13-25","13-26","13-27","13-28","13-290","13-292","13-293","14-24","14-25","14-26","14-27","14-290","15-24","15-25","16-24","16-25","16-233","16-234","16-235","17-22","17-23","17-24","17-25","17-59","17-233","17-234","17-235","18-22","18-23","18-24","18-25","18-43","18-213","18-214","18-215","18-229","18-230","18-232","18-233","18-234","19-42","19-43"]}

I want to merge these files into one, which should look like this:

{"2mac:acg":["1-248","3-245","3-246","4-245","4-246","5-245","5-246","6-243","6-245","6-246","6-247","6-296","7-245","7-295","7-296","8-236","8-239","8-240","8-294","8-295","8-296","9-235","9-236","9-239","9-294","10-293","10-294","10-295","11-15","11-16","11-293","11-294","12-16","12-290","12-291","12-292","12-293","12-294","13-25","13-26","13-27","13-28","13-290","13-292","13-293","14-24","14-25","14-26","14-27","14-290","15-24","15-25","16-24","16-25","16-233","16-234","16-235","17-22","17-23","17-24","17-25","17-59","17-233","17-234","17-235","18-22","18-23","18-24","18-25","18-43","18-213","18-214","18-215","18-229","18-230","18-232","18-233","18-234","19-42","19-43"], "4qld:aaa":["3-245","3-246","4-245","4-246","5-245","5-246","6-243","6-245","6-246","6-247","6-296","7-245","7-295","7-296","8-236","8-239","8-240","8-294","8-295","8-296","9-235","9-236","9-239","9-294","10-293","10-294","10-295","11-15","11-16","11-293","11-294","12-16","12-290","12-291","12-292","12-293","12-294","13-25","13-26","13-27","13-28","13-290","13-292","13-293","14-24","14-25","14-26","14-27","14-290","15-24","15-25","16-24","16-25","16-233","16-234","16-235","17-22","17-23","17-24","17-25","17-59","17-233","17-234","17-235","18-22","18-23","18-24","18-25","18-43","18-213","18-214","18-215","18-229","18-230","18-232","18-233","18-234","19-42","19-43","19-55"], "6k8h:c":["1-248","2-134","3-245","3-246","4-245","4-246","5-245","5-246","6-243","6-245","6-246","6-247","6-296","7-245","7-295","7-296","8-236","8-239","8-240","8-294","8-295","8-296","9-235","9-236","9-239","9-294","10-293","10-294","10-295","11-15","11-16","11-293","11-294","12-16","12-290","12-291","12-292","12-293","12-294","13-25","13-26","13-27","13-28","13-290","13-292","13-293","14-24","14-25","14-26","14-27","14-290","15-24","15-25","16-24","16-25","16-233","16-234","16-235","17-22","17-23","17-24","17-25","17-59","17-233","17-234","17-235","18-22","18-23","18-24","18-25","18-43","18-213","18-214","18-215","18-229","18-230","18-232","18-233","18-234","19-42","19-43"]}

the concat model should be {key1:value_list_1, key2:value_list2, key3:value_list3,...,key_last:value_list_last}

Thanks to @thanasisp, I used jq to concat them by jq -s 'add' file1 file2 file3. It worked well when hundreds of files were concatenated. But if there are thousands of files, it didn't work and replied with an error message: Argument list too long! So I wondered how to figure it out and if there are other methods to deal with it. Thanks! PS: there is enough memory in the server.

2 Answers 2

2
jq -c -s add file*

This will read all files that match the pattern file* into an array in jq. The -s (--slurp) option causes the single array to be created from all input files. Each element of this large array is one object from one of the files. The array elements are combined with add to form one single object.

The -c option makes jq produce "compact" output.

If you have too many files, the shell will fail to execute the command due to overrunning the maximum allowed length of the command line.

If this happens, you may use find to create a stream of JSON objects for the jq command to process.

find . -name '*.json' -type f -exec cat {} + | jq -c -s add >final

This uses cat from find to create a stream of JSON objects from the input files (any regular file with a name ending in .json in or below the current directory). The jq command collects these into an array that is then combined as before into a single object. The final result is outputted to the file called final.

Note that if you have a collision between keys (the same key in two or more files), the last key found and its value will overwrite the previous key and its value.

4
  • thanks for your reply. but it still returns 'Argument list too long' when I run cat *.json'|jq -c -s add or jq -c -s add file[123].
    – Jiao
    Sep 23, 2022 at 11:10
  • @Jiao I'm not surprised. That's why I used find in my answer.
    – Kusalananda
    Sep 23, 2022 at 11:10
  • thanks for reminding me. I already have a file name list, because json files were stored in pretty different Dir, and not all json files need to be concated. that's why I cannot use find against my files.
    – Jiao
    Sep 23, 2022 at 11:16
  • @Jiao This does not seem to be part of the question. Note that I can't see your files and that I have no way of knowing how you have organised your data. For all I know, you have thousands of files called file1, file2, etc.
    – Kusalananda
    Sep 23, 2022 at 11:22
1

Sounds like you don't even need jq, just to replace the trailing } with , in all but the last file, and remove the leading { from all but the first file.

In zsh:

autoload zargs
files=( *.json(Nn) )            # here sorted numerically or:
files=( ${(f)"$(<file.list)"} ) # to read the list one per line from
                                # a file.list
case $#files in
  (1) cat -- $files;;
  (<2->)
    sed -- 's/}$/,/' $files[1]
    zargs -r -- $files[2,-2] -- sed -- 's/^{//; s/}$/,/'
    sed -- 's/^{//' $files[-1]
esac > result.json

(insert |paste -sd '\0' - before > if you want the result on one line).

Or concatenate them first and do the substitution on the result, here using syntax compatible with ksh (at least those variants that have printf builtin), zsh, yash or bash, but assuming GNU xargs or compatible:

printf '%s\0' "${files[@]}" |
  xargs -r0 cat -- |
  sed '1!s/^{//; $!s/}$/,/' > result.json

That assumes the input json files have properly delimited lines.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .