9

In a shell script...

How do I capture stdin to a variable without stripping any trailing newlines?

Right now I have tried:

var=`cat`
var=`tee`
var=$(tee)

In all cases $var will not have the trailing newline of the input stream. Thanks.

ALSO: If there is no trailing newline in the input, then the solution must not add one.

UPDATE IN LIGHT OF THE ACCEPTED ANSWER:

The final solution that I used in my code is as follows:

function filter() {
    #do lots of sed operations
    #see https://github.com/gistya/expandr for full code
}

GIT_INPUT=`cat; echo x`
FILTERED_OUTPUT=$(printf '%s' "$GIT_INPUT" | filter)
FILTERED_OUTPUT=${FILTERED_OUTPUT%x}
printf '%s' "$FILTERED_OUTPUT"

If you would like to see the full code, please see the github page for expandr, a little open-source git keyword-expansion filter shell script that I developed for information security purposes. According to rules set up in .gitattributes files (which can be branch-specific) and git config, git pipes each file through the expandr.sh shell script whenever checking it in or out of the repository. (That is why it was critical to preserve any trailing newlines, or lack thereof.) This lets you cleanse sensitive information, and swap in different sets of environment-specific values for test, staging, and live branches.

  • what you do here is not necessary. filter takes stdin - it runs sed. You catch stdin in $GIT_INPUT then print that back to stdout over a pipe to filter and catch its stdout in $FILTERED_OUTPUT and then print it back to stdout. All 4 lines at the bottom of your example above could be replaced with just this: filter. No offense meant here, it's just... you're working too hard. You don't need the shell variables most of the time - just direct the input to the right place and pass it on. – mikeserv Sep 10 '14 at 17:30
  • No, what I do here is necessary because if I just do filter, then it will add newline characters to the ends of any input streams that did not end in newlines initially. In fact I originally did just do filter but ran into that problem which led me to this solution because neither "always add newlines" nor "always strip newlines" are acceptable solutions. – CommaToast Sep 10 '14 at 17:52
  • sed probably will do the extra newline - but you should handle that in filter not with all the rest. And all of those functions that you have basically do the same thing - a sed s///. You're using the shell to pipe data it has saved in its memory to sed so that sed might replace that data with other data that the shell has stored in its memory so sed can pipe it back to the shell. Why not just [ "$var" = "$condition" ] && var=new_value? I also don't get the arrays - are you storing the array name in [0] then using sed to replace that with the value in [1]? Maybe chat? – mikeserv Sep 10 '14 at 18:27
  • @mikeserv - What would be the benefit of moving that code inside filter? It works perfectly as-is. Regarding how the code at my link works and why I set it up the way that I did, yeah, lets talk about it in a chat room. – CommaToast Sep 10 '14 at 20:52
7

The trailing newlines are stripped before the value is stored in the variable. You may want to do something like:

var=`cat; echo x`

and use ${var%x} instead of $var. For instance:

printf "%s" "${var%x}"

Note that this solves the trailing newlines issue, but not the null byte one (if standard input is not text), since according to POSIX command substitution:

If the output contains any null bytes, the behavior is unspecified.

But shell implementations may preserve null bytes.

  • Would text files typically contain null bytes? I can't see why they would. But the script that you just mentioned does not seem work. – CommaToast Sep 9 '14 at 0:51
  • @CommaToast Text files don't contain null bytes. But the question just says stdin / input stream, which may not be text in the most general case. – vinc17 Sep 9 '14 at 0:53
  • OK. Well I tried it from the command line and it didn't do anything, and from within my script itself, your suggestion fails because it adds "..." at the end of the file. Also if there was no newline there, then it still adds one. – CommaToast Sep 9 '14 at 0:59
  • @CommaToast The "..." was just an example. I've clarified my answer. No newline is added (see the text before the "..." in the example). – vinc17 Sep 9 '14 at 1:06
  • 1
    Well, shells shouldn't hide things, that is not cool. Those shells ought to be fired. I don't like it when my computer thinks it knows better than me. – CommaToast Sep 9 '14 at 2:14
4

You can use the read built-in to accomplish this:

$ IFS='' read -d '' -r foo < <(echo bar)

$ echo "<$foo>"
<bar
>

For a script to read STDIN, it'd simply be:

IFS='' read -d '' -r foo

 

I'm not sure what shells this will work in though. But works fine in both bash and zsh.

  • Neither -d nor the process substitution (<(...)) are portable; this code will not work in dash, for instance. – chepner Sep 9 '14 at 21:24
  • Well the process substitution isn't part of the answer, that was only part of the example showing that it works. As for -d, that's why I put the disclaimer at the bottom. The OP doesn't specify the shell. – Patrick Sep 9 '14 at 22:38
  • @chepner - while the style differs slightly, the concept certainly does work in dash. You just use <<HEREDOC\n$(gen input)\nHEREDOC\n - in dash - which uses pipes for heredocs the same way other shells use them for process substitution - it makes no difference. The read -d thing is just specifying a delimiter - you can do the same a dozen ways - just be sure about it. Though you will need some tail to gen input. – mikeserv Sep 10 '14 at 19:30
  • You set IFS='' so it doesn't put spaces in between the lines it reads in eh? Cool trick. – CommaToast Sep 11 '14 at 1:05
  • Actually in this case IFS='' probably isn't necessary. It's meant so that read won't collapse spaces. But when it's reading into a single variable, it has no effect (that I can recall). But I just feel safer leaving it on :-) – Patrick Sep 11 '14 at 2:20
2

You can do like:

input | { var=$(sed '$s/$/./'); var=${var%.}; }

Whatever you do $var disappears as soon as you step outside of that { current shell ; } grouping anyway. But it could also work like:

var=$(input | sed '$s/$/./'); var=${var%.}
  • 1
    It should be noted that with the first solution, i.e. having to use $var in the { ... } grouping, is not always possible. For instance if this command is run inside a loop and one needs $var outside the loop. – vinc17 Sep 9 '14 at 1:57
  • @vinc17 - if it is a loop I desired to use, then I would use it in place of the {} braces .It is true - and is explicitly noted in the answer - that the value for $var is very likely to disappear entirely when the { current shell; } grouping is closed. Is there some more explicit way to say it than, Whatever you do $var disappears...? – mikeserv Sep 9 '14 at 2:01
  • @vinc17 - probably the best way, though: input | sed "s/'"'/&"&"&/g;s/.*/process2 '"'-> &'/" | sh – mikeserv Sep 9 '14 at 2:25
  • 1
    There's also the _variables function of bash_completion, which stores the result of a command substitution in a global variable COMPREPLY. If a pipeline solution were used to keep newlines, the result would be lost. In your answer, one has the impression that both solutions are equally good. Moreover it should be noted that the pipeline solution behavior heavily depends on the shell: a user could test echo foo | { var=$(sed '$s/$/./'); var=${var%.}; } ; echo $var with ksh93 and zsh, and thinks that it is OK, while this code is buggy. – vinc17 Sep 10 '14 at 21:36
  • 1
    You did not say "it doesn't work". You just said "$var disappears" (which is actually not true since this depends on the shell — the behavior is unspecified by POSIX), which is a rather neutral sentence. The second solution is better because it doesn't suffer from this problem, and its behavior is consistent in all POSIX shells. – vinc17 Sep 10 '14 at 22:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.