3

I am trying to come up with a tool what for sending data up to a github gist via the API. The problem is the github api takes the gist content as a single line with all escape sequences written out literally, like such:

{
  "test.txt": {
    "filename": "test.txt",
    "type": "text/plain",
    "language": "Shell",
    "raw_url": "https://gist.githubusercontent.com/jessebutryn/5c8b2a95b4b016e2fa33edee294c732b/raw/474f72ad32c843c18e9a61a228a31df6b85a8da1/test.txt",
    "size": 96,
    "truncated": false,
    "content": "#!/bin/sh\n\n# comment\nfunc () {\n\tfor ((i=1;i<10;i++)); do\n\t\t:\n\tdone\n}\n\nprintf '%s\\n' Foo bar baz\n"
  }
}

That content is displayed as follows:

#!/bin/sh

# comment
func () {
    for ((i=1;i<10;i++)); do
        :
    done
}

printf '%s\n' Foo bar baz

Which needs to be converted to:

#!/bin/sh\n\n# comment\nfunc () {\n\tfor ((i=1;i<10;i++)); do\n\t\t:\n\tdone\n}\n\nprintf '%s\\n' Foo bar baz\n

Are there any tools that do this in one action? If not does anyone know how it could be done with sed or any of the standard unix tools?


Note: Any literal escape sequences in the original text will need to be escaped to prevent github from interpreting them (however this would be a secondary issue that doesn't necessarily need to be solved in this question, but would be a nice to have):

ie:

printf '%s\n' Foo bar baz

becomes:

printf '%s\\n' Foo bar baz
2
  • @MichaelHomer: jq would be acceptable. I didn't realize this was a JSON thing (I mean obviously I know the data is in a JSON object but I just figured github wants it on one line for other reasons)
    – jesse_b
    Oct 26 '19 at 19:58
  • @mosvy: Pretty close
    – jesse_b
    Oct 26 '19 at 20:04
2
jq -R -s '.' < datafile

This reads in all of datafile as a string, and then has jq just print it out as a JSON string. It will give you a quoted string suitable for substituting into that template directly with the contents of datafile in it. The data will be correctly JSON-quoted with only the RFC 7159 escapes used, and will be in one big line because JSON doesn't allow string literals to span multiple lines.


You could also assemble the whole document in jq with a template JSON file and

jq --arg f "$(cat datafile)" '.["test.txt"].content = $f' < template.json

Very recent versions of jq have a --rawfile f datafile option that you can use to load a file into a string instead of the command substitution; you could also swap things around with -R --slurp --slurpfile t template.json datafile and t["test.txt"].content = ..

1
  • Perfect! Thanks!
    – jesse_b
    Oct 26 '19 at 20:12
1

I had to escape some JSON for inclusion in other JSON, and there's a really simple solution:

$ jq @json <<< '{"Hello": "World"}'
"{\"Hello\":\"World\"}"
0

If you have an existing JSON document in file.json and you just want to insert the content into the correct place, using jq,

jq --rawfile content test.txt '."test.txt".content |= $content' file.json

If the content should be base64-encoded:

base64 test.txt | jq --rawfile content /dev/stdin '."test.txt".content |= $content' file.json

Building the JSON from scratch, using jo (approximating the meta data from the output of stat and file):

#!/bin/sh

gist_url='some url'
filename=test.txt

jo -p "$filename"="$( jo \
        filename="$filename" \
        type="$( file -b --mime-type "$filename" )" \
        language="$( file -b "$filename" )" \
        raw_url="$gist_url" \
        size="$( stat -c %s "$filename" )" \
        truncated=false \
        content=@"$filename" )"

Adjust the stat and file utilities according to your Unix (the above uses Linux options).

Output:

{
   "test.txt": {
      "filename": "test.txt",
      "type": "text/x-shellscript",
      "language": "POSIX shell script, ASCII text executable",
      "raw_url": "some url",
      "size": 108,
      "truncated": false,
      "content": "#!/bin/sh\n\n# comment\nfunc () {\n    for ((i=1;i<10;i++)); do\n        :\n    done\n}\n\nprintf '%s\\n' Foo bar baz"
   }
}

Use %"$filename" in place of @"$filename" to get a base64-encoded value for the content key:

{
   "test.txt": {
      "filename": "test.txt",
      "type": "text/x-shellscript",
      "language": "POSIX shell script, ASCII text executable",
      "raw_url": "some url",
      "size": 108,
      "truncated": false,
      "content": "IyEvYmluL3NoCgojIGNvbW1lbnQKZnVuYyAoKSB7CiAgICBmb3IgKChpPTE7aTwxMDtpKyspKTsgZG8KICAgICAgICA6CiAgICBkb25lCn0KCnByaW50ZiAnJXNcbicgRm9vIGJhciBiYXoK"
   }
}

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.