I am trying to come up with a tool what for sending data up to a github gist via the API. The problem is the github api takes the gist content as a single line with all escape sequences written out literally, like such:

  "test.txt": {
    "filename": "test.txt",
    "type": "text/plain",
    "language": "Shell",
    "raw_url": "https://gist.githubusercontent.com/jessebutryn/5c8b2a95b4b016e2fa33edee294c732b/raw/474f72ad32c843c18e9a61a228a31df6b85a8da1/test.txt",
    "size": 96,
    "truncated": false,
    "content": "#!/bin/sh\n\n# comment\nfunc () {\n\tfor ((i=1;i<10;i++)); do\n\t\t:\n\tdone\n}\n\nprintf '%s\\n' Foo bar baz\n"

That content is displayed as follows:


# comment
func () {
    for ((i=1;i<10;i++)); do

printf '%s\n' Foo bar baz

Which needs to be converted to:

#!/bin/sh\n\n# comment\nfunc () {\n\tfor ((i=1;i<10;i++)); do\n\t\t:\n\tdone\n}\n\nprintf '%s\\n' Foo bar baz\n

Are there any tools that do this in one action? If not does anyone know how it could be done with sed or any of the standard unix tools?

Note: Any literal escape sequences in the original text will need to be escaped to prevent github from interpreting them (however this would be a secondary issue that doesn't necessarily need to be solved in this question, but would be a nice to have):


printf '%s\n' Foo bar baz


printf '%s\\n' Foo bar baz
  • @MichaelHomer: jq would be acceptable. I didn't realize this was a JSON thing (I mean obviously I know the data is in a JSON object but I just figured github wants it on one line for other reasons) – jesse_b Oct 26 '19 at 19:58
  • @mosvy: Pretty close – jesse_b Oct 26 '19 at 20:04
jq -R -s '.' < datafile

This reads in all of datafile as a string, and then has jq just print it out as a JSON string. It will give you a quoted string suitable for substituting into that template directly with the contents of datafile in it. The data will be correctly JSON-quoted with only the RFC 7159 escapes used, and will be in one big line because JSON doesn't allow string literals to span multiple lines.

You could also assemble the whole document in jq with a template JSON file and

jq --arg f "$(cat datafile)" '.["test.txt"].content = $f' < template.json

Very recent versions of jq have a --rawfile f datafile option that you can use to load a file into a string instead of the command substitution; you could also swap things around with -R --slurp --slurpfile t template.json datafile and t["test.txt"].content = ..

  • Perfect! Thanks! – jesse_b Oct 26 '19 at 20:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.