0

I'm adjusting a JSON file that holds a large number of objects. Here are the first three objects as an example:

{"id":"0704.0001","submitter":"Pavel Nadolsky","authors":"C. Bal\\'azs, E. L. Berger, P. M. Nadolsky, C.-P. Yuan","title":"Calculation of prompt diphoton production cross sections at Tevatron and\n  LHC energies","comments":"37 pages, 15 figures; published version","journal-ref":"Phys.Rev.D76:013009,2007","doi":"10.1103/PhysRevD.76.013009","report-no":"ANL-HEP-PR-07-12","categories":"hep-ph","license":null,"abstract":"  A fully differential calculation in perturbative quantum chromodynamics is\npresented for the production of massive photon pairs at hadron colliders. All\nnext-to-leading order perturbative contributions from quark-antiquark,\ngluon-(anti)quark, and gluon-gluon subprocesses are included, as well as\nall-orders resummation of initial-state gluon radiation valid at\nnext-to-next-to-leading logarithmic accuracy. The region of phase space is\nspecified in which the calculation is most reliable. Good agreement is\ndemonstrated with data from the Fermilab Tevatron, and predictions are made for\nmore detailed tests with CDF and DO data. Predictions are shown for\ndistributions of diphoton pairs produced at the energy of the Large Hadron\nCollider (LHC). Distributions of the diphoton pairs from the decay of a Higgs\nboson are contrasted with those produced from QCD processes at the LHC, showing\nthat enhanced sensitivity to the signal can be obtained with judicious\nselection of events.\n","versions":[{"version":"v1","created":"Mon, 2 Apr 2007 19:18:42 GMT"},{"version":"v2","created":"Tue, 24 Jul 2007 20:10:27 GMT"}],"update_date":"2008-11-26","authors_parsed":[["Bal\u00e1zs","C.",""],["Berger","E. L.",""],["Nadolsky","P. M.",""],["Yuan","C. -P.",""]]}
{"id":"0704.0002","submitter":"Louis Theran","authors":"Ileana Streinu and Louis Theran","title":"Sparsity-certifying Graph Decompositions","comments":"To appear in Graphs and Combinatorics","journal-ref":null,"doi":null,"report-no":null,"categories":"math.CO cs.CG","license":"http://arxiv.org/licenses/nonexclusive-distrib/1.0/","abstract":"  We describe a new algorithm, the $(k,\\ell)$-pebble game with colors, and use\nit obtain a characterization of the family of $(k,\\ell)$-sparse graphs and\nalgorithmic solutions to a family of problems concerning tree decompositions of\ngraphs. Special instances of sparse graphs appear in rigidity theory and have\nreceived increased attention in recent years. In particular, our colored\npebbles generalize and strengthen the previous results of Lee and Streinu and\ngive a new proof of the Tutte-Nash-Williams characterization of arboricity. We\nalso present a new decomposition that certifies sparsity based on the\n$(k,\\ell)$-pebble game with colors. Our work also exposes connections between\npebble game algorithms and previous sparse graph algorithms by Gabow, Gabow and\nWestermann and Hendrickson.\n","versions":[{"version":"v1","created":"Sat, 31 Mar 2007 02:26:18 GMT"},{"version":"v2","created":"Sat, 13 Dec 2008 17:26:00 GMT"}],"update_date":"2008-12-13","authors_parsed":[["Streinu","Ileana",""],["Theran","Louis",""]]}
{"id":"0704.0003","submitter":"Hongjun Pan","authors":"Hongjun Pan","title":"The evolution of the Earth-Moon system based on the dark matter field\n  fluid model","comments":"23 pages, 3 figures","journal-ref":null,"doi":null,"report-no":null,"categories":"physics.gen-ph","license":null,"abstract":"  The evolution of Earth-Moon system is described by the dark matter field\nfluid model proposed in the Meeting of Division of Particle and Field 2004,\nAmerican Physical Society. The current behavior of the Earth-Moon system agrees\nwith this model very well and the general pattern of the evolution of the\nMoon-Earth system described by this model agrees with geological and fossil\nevidence. The closest distance of the Moon to Earth was about 259000 km at 4.5\nbillion years ago, which is far beyond the Roche's limit. The result suggests\nthat the tidal friction may not be the primary cause for the evolution of the\nEarth-Moon system. The average dark matter field fluid constant derived from\nEarth-Moon system data is 4.39 x 10^(-22) s^(-1)m^(-1). This model predicts\nthat the Mars's rotation is also slowing with the angular acceleration rate\nabout -4.38 x 10^(-22) rad s^(-2).\n","versions":[{"version":"v1","created":"Sun, 1 Apr 2007 20:46:54 GMT"},{"version":"v2","created":"Sat, 8 Dec 2007 23:47:24 GMT"},{"version":"v3","created":"Sun, 13 Jan 2008 00:36:28 GMT"}],"update_date":"2008-01-13","authors_parsed":[["Pan","Hongjun",""]]}

I'm trying to add a new key called title_length to every object. This key's value should be the length of the value of the title key.

So far, I've tried

jq '.[] | .title_length = (.title | length)' file.json > new_file.json

But this gives me the following error:

jq: error (at file.json:74850): Cannot index string with string "title"

I don't understand the error, and I can't seem to bypass it.

2
  • How exactly does the file "hold" the objects? is there an outer structure such as an array, or are they simply concatenated as shown in your example? May 31, 2023 at 18:50
  • To my understanding, they are simply connected as in the example.
    – A.Co
    May 31, 2023 at 18:54

2 Answers 2

3

If each line of the file contains a JSON object (a jsonlines file), then the leading .[] is incorrect because . will be each line's object, not an array.

You want simply

jq -c '.title_length = (.title | length)'

or

jq -c '. + {title_length: .title | length}'

Note the -c option to keep the output condensed like the input is.

0

glenn has already diagnosed the issue (you have a set of objects, not an array) and have given you a corrected jq expression.

I'm just going to show an alternative way of doing the same thing using another tool, Miller (mlr), a general purpose utility for processing structured data.

mlr --jsonl put '$title_length = strlen($title)' file

This would read the file as a JSONL data set (a collection of JSON objects, one per line). For each such object, it creates a new field, title_length, with the value calculated as the string length of the object's title field. The output would be produced on standard output as JSONL.

This way of solving the tasks may feel more familiar to users of awk, or people unfamiliar with jq.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .