2

If I have a script that sends different data types to STDOUT for each file it processes. How can I separate each data type so that another script that reads STDIN knows which is what?

For example. I have a script that produces two different (unknown) strings and two different (unknown) numbers for each file it processes. Then there is another script that reads from STDIN to process each of the given strings, numbers. How can I format the output of the first script, so that the second can be made to identify each type correctly?

I'm used to serializing data via JSON, for the web, but I wonder if there is a more lightweight or built-in solution for STDIN/STDOUT? Maybe some unique separator or something I'm missing?

  • 4
    The basic concept behind designing pipes (piping stdout to stdin) is to make all program print text, so it supports only unstructured, untyped data. Using some encoding for data and serialization formats is up to your shoulders. – myaut Sep 21 '17 at 18:41
  • @MichaelHomer: I believe you was wrong in removing the linux tag. It could matter in a subtle way (things could be different on AIX) – Basile Starynkevitch Sep 22 '17 at 7:41
  • 1
    It seems very unlikely, but the question is going to be closed as too broad anyway, so if it's relevant it can be re-added after it's edited and reopened. stdout is just a byte stream everywhere I can think of, and it'd be a user-space issue to layer something on. – Michael Homer Sep 22 '17 at 7:47
  • @MichaelHomer: I was implicitly thinking of /proc; it surely is different on other Unixes. – Basile Starynkevitch Sep 22 '17 at 8:03
  • 1
    @myaut Pipes have very little to do with text. A hypothetical example: gzip -c file | base64 > file.gz.base64. It's rather the idea of utilities as "filters" that implies unstructured data. – Kusalananda Sep 22 '17 at 9:29
7

You don't explain what is your stdout and where does it goes! (For example the stdout of a CGI application goes -indirectly- to some browser).

The Unix philosophy and Unix pipelines want stdin and stdout to be plain text, but this is just a convention. In some cases, you could have other conventions (for example lpr or lp often prefer PDF on stdin).

If you are coding some program, you might make it have some mode (e.g. specified by some program argument) to output more structured text, such as JSON or XML or CSV or YAML. Notice that jq can process JSON.

Some programs detect (using isatty and/or fstat) when their stdout (or their stdin) is a terminal and act accordingly (perhaps by using ncurses, termios, or ANSI escape codes).

Many textual formats (notably XML) have conventional starting characters or headers, so in some cases guessing them (i.e. finding out the file format) is possible. Look also into MIME and libmagic (and file(1)).

I have a script that produces two different (unknown) strings and two different (unknown) numbers for each file it processes. Then there is another script that reads from STDIN to process each of the given strings, numbers. How can I format the output of the first script, so that the second can be made to identify each type correctly?

If you know (and document) that the unknown strings are nice enough to not contain control characters such as newlines (so they each are a single line) you could decide for some format like

FIRSTSTRING: firststring

FIRSTNUMBER: firstnumber

for example

 FIRSTSTRING: foo bar is nice!
 FIRSTNUMBER: 42

then write and use some simple awk script (or perhaps use sed) for your second script

In other words, decide and document a simple ad-hoc format; in your case that might be simpler to handle than JSON. You can have your own ad-hoc conventions provided you document them (perhaps using some EBNF notations).

A lot of tools are doing that all ready; for example ps, ls, df, and ifconfig have ad-hoc but well documented output formats and conventions. And likewise for proc(5). Hence many scripts can parse such outputs.

However, JSON was designed to be simple, flexible, scalable, extensible, ... And it is able to represent arbitrary strings (even with control characters, many lines, etc...). If that matters to you, use it.

One could reinvent and reimplement all Unix utilities to output e.g. JSON or XML (but doing that is a lot of work). For example, some people reinvented proc(5) and made a kernel module to have an /xmlproc/ pseudo file system instead of the /proc/ one which outputted system kernel data in some XML form. But that was not successful! Social conventions matter a lot (that is why it is so important to document your output format, at the very least in a long comment).

(even if you use JSON or XML you need to document how you use them)

BTW, a lot of existing Unix tools may add subtleties in conventions. For example, a space or a tab or a return character in a file path is possible (see path_resolution(7)) but could be frowned upon (so I never do that). The $HOME directory of some user could in theory contain a return character or a colon, but if you did that most tools would suffer (and you are likely to break passwd(5)...). File paths starting with a dash are unfriendly, and so are very long paths (e.g. your $HOME might be 3000 characters long in theory but that would be really unwise).

  • Ok. I thought there was, maybe, some identifier or something, which could be used to separate the plaintext, so that you don't have to use JSON & co, when piping from one command line tool to the other. PS: Awesome answer! :) – Rotareti Sep 21 '17 at 18:59
7

You have two scripts, one produces two strings and two numbers. It is up to you to parse that in the second script, as two strings and two numbers.

Since both scripts are under your control, you are free to send the data from one to the other in any way that makes it convenient for the second script to read the data.

It's up to you to determine the format, order and interpretation of the data that you pipe between your two scripts. You may even send binary encoded data if that makes sense to you.

There are no data types that may be applied to the data, other than what you make up for yourself by deciding on a particular "contract" or "protocol" between your two scripts or programs.

Some standard Unix tools like sort and cut assumes input on a particular format but can be made to change their interpretation of the input data through the use of command line options.

Contrived example:

#!/bin/sh

echo 'first string'
echo 'second string'

echo '1.1'
echo '3.14'

The second script then reads this:

#!/bin/sh

IFS= read -r string1
IFS= read -r string2
read number1 number2

Or, with JSON:

#!/bin/sh

echo '{ "string1": "hello", "string2": "world", "numbers": [1.1,3.14] }'

Second script:

#!/bin/sh

jq -r '"A number: \(.numbers[])"'
  • 1
    @Rotareti Yes, it would break the format. In that case you could try using another string delimiter than newline, such as \0 (nul). This is what's done with -print0 in find and with -0 in xargs, for example (to handle filenames that have newline characters in their names). The \0 character is the only character that is not allowed in a pathname (and / is not allowed in a filename either). – Kusalananda Sep 28 '17 at 16:18
  • 1
    @Rotareti There is nothing stopping you from sending base64 encoded data between your shell scripts. You may also want to send special characters, like newlines, encoded in some form, possibly as \n. My whole point was to say that it's up to you to impose the restrictions on the input to your programs. – Kusalananda Sep 28 '17 at 16:21
  • 1
    @Rotareti Another example is the way that some database engines wants you to encode NULL in input data. If you load a text file inte MySQL, for example, NULL should be written as \N. It's an arbitrary convention. – Kusalananda Sep 28 '17 at 16:23
  • 1
    @Rotareti You can break any program by sending it junk. – Kusalananda Sep 28 '17 at 16:23
  • 1
    @Rotareti (unless it's a program that doesn't try to interpret the data, like cat or gzip) – Kusalananda Sep 28 '17 at 17:57
2

stdin and stdout are just the file descriptors 0 and 1 of a process and whatever open file description they point to, themselves resulting from, for instance the opening of a file with the open() system call, or creation of a pipe with pipe() or open() on a named pipe, or socket with connect()/accept()/socketpair(), etc.

At the very least, most of those would be able to read and write any sequence of bytes as a stream. For those that are typically used for inter-process communication like pipes and stream sockets, there is generally no preservation of message boundaries.

For instance, in a shell command line like:

writer | reader

where, you'll have the writer's stdout being one end of a pipe (or socketpair depending on the shell) and the reader's stdin the other end

if writer does a write(1, "foo", 3); write(1, "bar", 3), the reader won't be able to tell there are two messages coming in unless it happens to do its read() in between the writer's two writes.

There are some file types like datagram sockets (UDP, SCTP or unix domain at least), or SOCK_SEQPACKET on some systems that preserves message boundaries, but you'd need a different API than read() and write() if you want to allow empty messages, and the reader would have to know in advance the maximum possible size of those messages and allocate a buffer that big to receive it. You'll still need to use some form of encoding to specify the nature of the content of those messages.

Example:

$ strace -e write dd bs=2 count=3 if=/dev/zero status=none | strace -fe read cat
write(1, "\0\0", 2)                     = 2
write(1, "\0\0", 2)                     = 2
write(1, "\0\0", 2)                     = 2
read(0, "\0\0\0\0\0\0", 131072)         = 6
read(0, "", 131072)                     = 0

3 writes of size 2, 1 read of size 6, though depending on timing, you may see 3 reads of size 2, or one read of size 4 and one of size 2. That was with a pipe or SOCK_STREAM socketpair.

With a SOCK_SEQPACKET socketpair:

$ perl -MSocket -e '
   socketpair(my $rdr, my $wtr, AF_UNIX, SOCK_SEQPACKET, PF_UNSPEC);
   shutdown($rdr, 1); shutdown($wtr, 0);
   if (fork) {
     open STDIN, "<&", $rdr; close $wtr; close $rdr; sleep 1;
     exec qw(strace -e read cat)
   } else {
     open STDOUT, ">&", $wtr; close $rdr; close $wtr;
     exec qw(strace -e write dd count=3 bs=2 status=none if=/dev/zero)
   }'
write(1, "\0\0", 2)                     = 2
write(1, "\0\0", 2)                     = 2
write(1, "\0\0", 2)                     = 2
+++ exited with 0 +++
read(0, "\0\0", 131072)                 = 2
read(0, "\0\0", 131072)                 = 2
read(0, "\0\0", 131072)                 = 2
read(0, "", 131072)                     = 0
+++ exited with 0 +++

That's 3 reads for those 3 writes, even though we delayed the reads by one second, long after the writes were done.

So, in the end, best is to encode the type and length one way or other. A compact form and one that allows the reader to process the data as soon as it comes (but the writer to know the length of the message in advance) is to use TLV (type, length, value) encoding. You'll still need for the writer and reader to agree on the length and type of the "type" and "length" words (for instance, 32 bit little endian integer) and how to interpret the type values.

Or you can use any of the many serialisation formats there are out there that produce text that is also safer for interchange across systems with different endianness or channels that don't allow NUL bytes or do some line delimiter transcoding. Like json, XML, perl's Data::Dumper, php's serialize(), the output of some shell's typeset -p depending on the language you're using...

Not the answer you're looking for? Browse other questions tagged or ask your own question.