2

I've got this JSON data that looks like this:

[
  {
    "loginId": "7638749",
    "customerprofileDetails": {
      "securityQuestions": [
        "What is your favorite sports team?",
        "What is your favorite song?",
        "Who is your favorite artist?"
      ]
    }
  }
]

I have a script where I pass in that file as the first argument. The script looks like this:

for json in `cat $PWD/$1 | jq -cr '.[]'` ; do
    echo "$json"
done

For some reason, when I run this script, it outputs this:

> fileFromJson.sh tmp2.json
{"loginId":"7638749","customerprofileDetails":{"securityQuestions":["What
is
your
favorite
sports
team?","What
is
your
favorite
song?","Who
is
your
favorite
artist?"]}}

Why are those newlines there and what's causing this problem? When I just run cat $PWD/tmp2.json | jq -cr '.[]', it prints out on one line. I'm using MacOS if that matters.

2
  • Background reading: unix.stackexchange.com/questions/131766/… . I'm not sure that this can be considered a duplicate: what were you trying to do with this for loop? The input is a single piece of JSON, what is there to iterate on? Jul 30, 2018 at 18:20
  • @Gilles the real input has multiple elements. I need to create a file with the loginId value and use the customerprofileDetails as its content. Jul 30, 2018 at 18:25

2 Answers 2

4

The unquoted command substitution `cat $PWD/$1 | jq -cr '.[]'` does the following:

  1. Run the command cat $PWD/$1 | jq -cr '.[]'. Which by the way is an overly complicated way of writing jq -cr '.[]' <"$1", except that this variant doesn't break if $1 contains certain special characters or is an absolute path.
  2. Break the resulting string into whitespace-separated pieces. Your question is not very clear, but you seem to expect newline-separated pieces — that's not what an unquoted substitution does.
  3. Treat each piece as a wildcard pattern and, if it matches one or more file name, replace that piece by the list of matching file names.

Steps 2 and 3 are colloquially known as the “split+glob operator” and this is almost never what you want. Always use double quotes around variable and command substitutions unless you know exactly why you need to leave out the quotes.

To process a file line by line in the shell, use a while read loop:

jq -cr '.[]' <"$1" |
  while IFS= read -r line; do
    printf '%s\n' "$line"
  done

But you may be better off piping into some other tool. Shells aren't good at processing large amounts of string data by themselves.

3
0

By default, echo always outputs a trailing newline.

You can pass the -n argument to echo to prevent it from doing so.

From man echo:

-n do not output the trailing newline

Edit:

The "issue" here is not in your cat $PWD/$1 | jq -cr '.[]' command, it's in the echo line itself, or more precisely, the for loop. With the for loop, you are splitting what you're passing to it (the output of the jq command) word by word.

So essentially, you're passing every single string that is separated by a whitespace one-by-one to the echo command, which will by default add a trailing newline everytime it is executed.

4
  • That worked, but I don't understand why. Doesn't jq -c convert the json to a single line already? Besides, there were never newlines between those words originally. Why are there trailing newlines between spaces? Jul 30, 2018 at 18:21
  • I'll edit my answer.
    – confetti
    Jul 30, 2018 at 18:21
  • True (for bash, not for all other shells), but not relevant. Changing echo "$json" to echo -n "$json" will remove the newlines but will not print the original. Changing to echo -n "$json "` will not print the original either. Consider the input ["hello * world"] Jul 30, 2018 at 18:22
  • I have edited my answer.
    – confetti
    Jul 30, 2018 at 18:24

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .