Questions tagged [jq]
Questions about the command line JSON processing tool jq.
188
questions
0
votes
1answer
36 views
How to extract fields from encoded JSON object
Hi i'm trying to extract the tokens last_name, first_name and phone from the following JSON data:
{"message":"{\"_\":\"user\",\"pFlags\":{\"contact\&...
1
vote
1answer
38 views
How can I extract two data fields (1 scalar & 1 array) per node out of a very large (> 100,000 lines) JSON file?
I've got a 139,000 line JSON file, whose structure basically looks like this (it's an extract from OpenStreetMap):
{
"type": "FeatureCollection",
"generator": "...
1
vote
1answer
50 views
How to sort a stream of json objects by field value using jq
I'm starting with json that looks like this:
{
"object": "list",
"data": [
{
"id": "in_1HW85aFGUwFHXzvl8wJbW7V7",
"object&...
5
votes
2answers
276 views
How to extract a value from a JSON file containing an encoded JSON object
The command aws s3api get-bucket-policy --bucket bucketname
outputs:
{
"Policy": "{\"Version\":\"2012-10-17\",\"Id\":\"S3SecureTransportPolicy\&...
1
vote
2answers
79 views
Convert huge old json into new json format through some sort of linux commands?
I have original json like this. In general, my json is very large. To make things simpler and easier to understand I have reduced the entries.
{
"clientSettings":[
{
"...
2
votes
1answer
91 views
Merge jq output into a comma separated string and new line
I'm trying to extract specific columns from this JSON file (which is probably incorrectly formatted).
Here is a data sample:
{ "_id" : { "$oid" : "4ddb1e4a9a0881572c000008&...
0
votes
1answer
28 views
using yq to base64 encode k8s secret values
I have a simple secret.yaml file:
env:
USERNAME: user
PASSWORD: pass
i'm trying to use yq / jq to encode the values for creating k8s secret, so my final result should be:
apiVersion: v1
kind: ...
3
votes
1answer
66 views
Get field and nested field at the same time using jq
I am trying to get two attributes I need from the JIRA API. Given the following input to JQ:
{
"expand": "names,schema",
"startAt": 0,
"maxResults": 50,
...
0
votes
1answer
48 views
How to merge arrays from multiple json files with JQ?
I have multiple JSON Files with the same structure , the list is the array.
$ jq 'keys' file_1.json
[
"itemsPerPage",
"links",
"list",
"startIndex"
]
...
1
vote
1answer
24 views
Using jq, how can I flatten to tsv and have nested array flattened as comma delimited value?
I have the following JSON format:
{
"page":1,
"total_results":1,
"total_pages":1,
"results":[
{
"id":6037,
"genre_ids":[35,80,...
1
vote
0answers
43 views
Retrieve nested element with jq
I have a list of JSON objects(one object is given below) where the parent element will dynamically be changed ("Oauth2", "Oauth2c"), so that, how can I retrieve a nested level ...
1
vote
1answer
80 views
Jq: Select Key and Value of One Prop
I have a json translation file that is structured like this.
{
"STRING_ID_1": {
"en": "Some englisch text.",
"de": "Some german text."
},
...
2
votes
1answer
137 views
Limit output depth of jq
I want to explore arbitrary document using jq. To that end, I would like to limit the depth to which jq descends into documents and only show me the first n, e.g. 3, levels.
Suppose I have the ...
1
vote
1answer
25 views
How to set Json Value with JQ as the result of command piped to JQ
I have a function in a bash script which gets an ID number from the JSON output of a command. I want to set this value in another JSON file. How can I do this?
build_vpc(){
aws ec2 create vpc --...
1
vote
2answers
489 views
How to sum the number of JSON elements in documents inside a directory using jq
I have around 3K JSON files inside a directory and each file contains approximately 250 (+/-) JSON elements inside it, I want to count the total number of JSON elements from those files as a sum.
...
1
vote
1answer
61 views
curl jq get value from field name
created=$(curl -i -X POST -H 'Content-Type:application/json' --data "$(payload)" https://myurl/resource)
The above return a json object
{
"revision": {
"clientId"...
1
vote
1answer
73 views
How does jq work with keys containing slash?
My test json file is as follows:
{
"server":"xxx",
"cert":"line1\nline2",
"security/path": "/var/log/systems.log"
}
I would like ...
1
vote
1answer
27 views
add values to json arrays
I have some JSON objects that I need to add a value to if it doesn't already exist.
Each object will be the following except the contact_group inside each array (1-5) will vary:
{
"contact_groups&...
4
votes
3answers
404 views
help to grep/print value of a key in json that is stored in a variable?
I have a variable with nested json,
a={"version":"3.0","user":"unknown_unknown","dateGenerated":"2020-07-08T11:53:23Z","status":&...
1
vote
1answer
82 views
Parsing .json in Bash
Have a json file I am trying to parse from podfox to allow me to rename the downloaded files in a "friendly" way. Here's a snippet of json I am working with:
{
"episodes": [
...
4
votes
3answers
261 views
within json data, how to transform integer to dollar amount using jq?
I have data like this
{
"dateCreated": "2020-06-12",
"status": "pending",
"depositDate": "2020-06-15",
"amount": 41237
}
{
...
1
vote
2answers
40 views
Change the value of a key in json while the key is mutable using jq
I have json-files which I wan't to modify. The problem is, that the target-key varies in the json-files. E.g. those two json:
{
"tasks": [{
"type": "type1",
...
0
votes
1answer
134 views
Grep a key value pair in JSON as CLI
I have a big Datalog with JSON. One JSON element looks like this:
{
"context": {
"callback": {
"Data": {
"LastSessionTime": "2020-06-07T09:16:19Z",
"TrackingEnabled": "1",
...
0
votes
2answers
144 views
How to properly encode string based on json file?
I have the following basic.json file:
{
"user": "user",
"pass": "password"
}
I'm trying to encode it in base64 like this:
"Basic dXNlcjpwYXNzd29yZA=="
I think I'm close:
echo Basic $(echo ...
3
votes
2answers
158 views
Convert Json table arrays to objects with jq
I am trying to convert a json structure that key1: array of values, key2: array of values ,.... to an array of objects. The size of arrays is same and each object is just aggregate of items at ...
-1
votes
1answer
248 views
Merging multiple JSON files removing duplicate objects by a field [closed]
I am trying to merging hundreds of thousand JSON files that have a single object. with the help of this answer I have successfully added all objects into a single object as an array. Now I want to ...
1
vote
1answer
318 views
Merging multiple json file into one object using JQ with argument
I have thousands of JSON file that I want to merge in one object. Those all are not in a similar format. Let me explain in detail. Here is the first sample of JSON
[
{
"value 1": 1,
...
0
votes
1answer
144 views
How should I replace a value in JSON file with its md5 value using jq command?
I have a big log file containing a line as below example :
{"data_1":210,"target_number":1096748811,"extra_data":66}
{"data_1":0,"target_number":7130881445,"extra_data":56}
{"data_1":1712,"...
0
votes
1answer
50 views
Merging json files using jq
I am trying to merge two json files using command "jq -s . file1 file2". But I want these file's content to be comma separated, neither it should start with [].
file 1:
{
"node1": {
"...
0
votes
1answer
55 views
How to print streaming JSON to file after processing with JQ and SED [duplicate]
I am using the following code to download streaming financial data
curl -s -H "Content-Type: application/json" -H "Authorization: Bearer XXX..." "https://...=EUR_USD" | jq --raw-output '[.time, .bids[...
0
votes
0answers
41 views
specific filtering of Azure ARM JSON templates using jq
Hi I am trying to filter this JSON AZURE ARM template in order just to return a specific array.
This jq is currently providing me with the arrays I need but I need to search through this and grab a ...
0
votes
1answer
166 views
Loop through json using jq to get multiple value and merge them in a single line
I am using jq to get some data in desired format, complete json data of a pod is pasted here https://pastebin.com/UMJA7xd5. This data is one of the pod in json format. I am going to run it on all pods,...
0
votes
1answer
98 views
JQ: convert value into nested key-value object
I am lost with an easy-looking JQ problem. Given a key-value object, I want to transform the value into a nested object with a "value"-key. I tried permutations with 'to_entries' and native output as ...
0
votes
1answer
350 views
Pass Bash associative arrays as argument to jq filter
I am working with some location data and am querying an API with U.S. postal codes and returns a result like:
{
"resultStatus":"SUCCESS",
"zip5":"30301",
"defaultCity":"ATLANTA",
"...
5
votes
1answer
242 views
how do I fix a jq expression returning Invalid path expression error?
I have a package.json file that looks like this:
{
"name": "service",
"version": "1.0.0",
"private": true,
"description": "my service",
"license": "none",
"scripts": {
"build": "tsc"
...
0
votes
1answer
67 views
Trying to convert json to csv using jq
I can't seem to figure out how to use jq to turn my data into a csv. I have the following json:
{
"id": 2,
"type": "message",
"date": "2010-01-11T01:19:45",
"edited": "1969-12-31T19:00:00"...
4
votes
2answers
1k views
Convert JSON array into CSV
I'm looking at solutions to convert JSON to a CSV. It seems most of the solutions expect the JSON to be a single object rather than an array of objects.
All the solutions I've tried from here seem to ...
1
vote
2answers
1k views
Add JSON objects to array using jq
My goal is to output a JSON object using jq on the output of a find command in bash. It could either be a one-line command or a bash script.
I have this command which creates JSON objects from each ...
0
votes
1answer
43 views
Select specific all key paths and values for a specific key name in deeply nested JSON
I have a very large JSON schema that is semi structured, the nested unit structures are tagged by a key: "ObjectType" to indicate some meaningful information.
I am trying to use jq to select paths of ...
1
vote
2answers
686 views
Convert JSON of arrays to CSV with headers using JQ
I am getting Json as column ID and arrays -
{
"ObjectID": [
0,
1,
2,
3,
4
],
"Name": [
"Apple",
"Orange",
"Grape",
...
1
vote
1answer
85 views
How do I filter the contents of a json array, but keep the parent with jq?
I have some JSON that looks like this:
{
"Vpcs": [
{
"VpcId": "vpc-1",
"OwnerId": "abc"
},
{
"VpcId": "vpc-2",
"OwnerId": "def"
}
]
}
I just want the ...
0
votes
0answers
84 views
Json to Csv with JQ
I want to convert a json file to csv file.
I have a script:
cat convert.sh
INPUTFILE="test.json"
OUTPUTFILE="outputfile-formatted.csv"
jq -r '(.[0] | keys_unsorted) as $keys | $keys, map([.[ $keys[]...
0
votes
1answer
54 views
jq: globbing possible?
I have the following json output:
{ x0: { scores: { small, human readable }, payload: { huge data } }
{ x1: ... }
{ x2: ... }
...
{ xn: ... }
What I would like to do is, when I finish generating ...
1
vote
1answer
3k views
Parsing JSON data from a curl API request
I'm using Shodan's API https://developer.shodan.io/api to get my current network alerts. I want to parse out the id for the alerts with jq.
The curl request is curl -X GET -i https://api.shodan.io/...
1
vote
1answer
268 views
JQ - add element to array and return full json
I'm trying to add an element to an array like this:
jq --arg root "arn:aws:iam::$ACCOUNT_ID:root" \
'.Statement[] | select(.Sid=="Allow use of the key") \
| .Principal.AWS ...
4
votes
1answer
347 views
How to print path and key values of JSON file
I would like to print each path and value of a json file with included key values line by line. Given the following json and jq, is it also possible to add the values for each line? If not using jq is ...
-1
votes
1answer
84 views
looping through JSON array in shell script using for loop
Below is the curl command output, need script or command to print "doc_count" less than 10 and save the "key" in output.txt file.
{
"took": 117,
"timed_out": false,
"_shards": {
"total": ...
0
votes
1answer
281 views
How to process two strings in one jq command?
I am trying to process two strings in one jq command by mimicking the answer to this SO question.
But I couldn't even get past simple tasks such as concatenate two strings (stored in bash variables ...
0
votes
1answer
1k views
jq append data and keep the rest
I'm trying to add a value to array with jq and keep the rest of the content:
"access":{
"allowed_users":[
"admin",
"test",
],
The above is what I want to append. If ...
2
votes
1answer
547 views
jq: how to print one line per sub-object values, repeating top level value with sub-objects value
I'm trying to print only certain values from a json (one line per sub-object) repeating one value from the top object on each line.
Sample input:
[
{
"name": "level1Name",
"lv1id"...
