3

Suppose we have the following program, which calls read() twice:

#include <stdio.h>
#include <unistd.h>

#define SIZE 0x100

int main(void)
{
    char buffer1[SIZE];
    char buffer2[SIZE];

    printf("Enter first input: \n");
    fflush(stdout);
    read(STDIN_FILENO, buffer1, SIZE);

    printf("Enter second input: \n");
    fflush(stdout);
    read(STDIN_FILENO, buffer2, SIZE);

    printf("\nFirst input:\n%s", buffer1);
    printf("\nSecond input:\n%s", buffer2);

    return 0;
}

When we call it directly, we can enter 1 for the first input and 2 for the second input in order to have it print:

First input:
1

Second input:
2

How can this be achieved when using input redirection?

The following methods don't work, since the first read consumes both inputs:

Pipe redirection:

$ { echo "1"; echo "2"; } | ./main_read
Enter first input:
Enter second input:

First input:
1
2

Second input:

Heredoc redirection:

$ ./main_read << EOF
1
2
EOF
Enter first input:
Enter second input:

First input:
1
2

Second input:

The assumption is that the source code cannot be changed, and that the input must sometimes be shorter than SIZE.

Is there any way to signal the first read() to stop reading, in order for the second read() to consume the rest of the input?

6
  • Thanks, I've already visited this page, and I've reviewed it again now. If it contains an obvious answer to my question, I must be missing it. Can you please point me to the section I should be concentrating on?
    – Dvd848
    Commented May 21, 2021 at 20:05
  • What do you mean by "the input must sometimes be shorter than SIZE"? (Maybe it's just me, but I'm unable to parse "must sometimes" in this context).
    – fra-san
    Commented May 21, 2021 at 20:15
  • @Quasímodo - no problem, thanks anyway.
    – Dvd848
    Commented May 21, 2021 at 20:20
  • @fra-san - I mean, providing 0x100 bytes as the first input in order to max-out the first read() isn't an acceptable solution.
    – Dvd848
    Commented May 21, 2021 at 20:24
  • Not even padding the actual input with NUL bytes?
    – fra-san
    Commented May 21, 2021 at 20:25

3 Answers 3

3

This is likely not to provide an acceptable solution to you, but considering that:

  • the source code cannot be changed

  • the shell can not change where the open file descriptors of a running program point to, nor make a running program stop reading from a file descriptor

Some alternatives you are left with (short of exploiting a race condition) are:

  • Trying to make sure your program is always fed with SIZE bytes at a time:

    {
      echo foo | dd bs=256 conv=sync
      echo bar | dd bs=256 conv=sync
    } 2>/dev/null | ./main_read
    

    Output:

    Enter first input: 
    Enter second input: 
    
    First input:
    foo
    
    Second input:
    bar
    

    This assumes, as a minimum, that SIZE is smaller than the size of the pipe buffer.

  • Wrapping the invocation of your program in an expect (or equivalent) script:

    expect <<'EOT'
    spawn ./main_read
    expect "Enter first input:"
    send "foo\n"
    expect "Enter second input:"
    send "bar\n"
    expect eof
    EOT
    

    Or, in a way that allows you to pipe to it the output of other commands, read separately (assuming your operating system provides processes with the /dev/fd/n file descriptors):

    echo foo | {
      echo bar |
        expect 4<&0 <<'EOT'
        spawn ./main_read
        set chan [open "/dev/fd/3"]
        gets $chan line
        expect "Enter first input:"
        send "$line\r"
        close $chan
        set chan [open "/dev/fd/4"]
        gets $chan line
        expect "Enter second input:"
        send "$line\r"
        close $chan
        expect eof
    EOT
    } 3<&0
    

    In both cases, the output is:

    spawn ./main_read
    Enter first input: 
    foo
    Enter second input: 
    bar
    
    First input:
    foo
    
    Second input:
    bar
    
  • On systems (such as Linux) that allow for opening pipes in a non-blocking fashion, you may use FIFOs to make the shell read from and write to your program. For instance:

    makefifo fifo
    {
      exec 3<>fifo
      ./main_read 0<&3
    } |
    sh -c '
      # read a line from the pipe
      # read from a file or a different pipe, write to fifo
      # repeat ...
      # echo the output from the pipe
    ' mysh
    

    Though, if expect is available to you, I see no compelling reason to prefer reinventing it.

Note that, however, as others have pointed out, there is no guarantee that all of your program's reads will actually get SIZE bytes.

3
  • @Dvd848 I edited my answer substantially after initially posting it -- just a ping, I think question askers are not notified when answers are edited.
    – fra-san
    Commented May 22, 2021 at 12:11
  • Thanks! For what it's worth, I think the example you edited out was worth keeping too, it's not as flexible as the other examples but is probably what I'd go for in cases where the extra flexibility isn't needed.
    – Dvd848
    Commented May 22, 2021 at 16:12
  • @Dvd848 Thank you for your feedback, I added that example back.
    – fra-san
    Commented May 22, 2021 at 16:23
2

The assumption is that the source code cannot be changed

You should work primarily on changing that assumption.

There are in general no guarantees about how many bytes a read() call returns When reading from a regular file, it usually returns as many bytes as requested (up to how much are available), but that might not be the case for all types of file descriptors. Scheduling between the processes running on the system, and other such timing issues may also affect how much data is available with one call.

Doing a single read() call without checking the amount of data read is simply almost always wrong. Even cases like dd (which is explicitly supposed to expose the behaviour of the read() calls) and reading from datagram sockets (where a each individual read() gives a single message) will need the program to know how much data it got.

If the program is supposed to read lines, it should use fgets() or getline() and not raw read(). If it's supposed to read other sorts of blocks, it should implement some other way to tell the blocks apart. That could be either prepending them with a length, or using separate file descriptors, or using some delimiter (like newline, but it could be longer than just one byte).

That is, unless you arrange stdin to be connected to a datagram socket, but that would be a very unusual setup, and you couldn't really use regular input redirection to provide the data.

Also, before passing the buffers to printf("%s"), the program should make sure they contain the NUL bytes terminating the strings. As it is now, the program produces undefined behaviour if the data provided by either read() does not contain the NUL, including the case where the second read() returns no data at all.

2
  • Thanks, I was trying to create a simple reference code to demonstrate the problem and using strings seemed like the simplest way to go. This is also the reason for skipping the different mandatory checks. In retrospective I regret this, I should have probably provided the missing checks/error handling as well.
    – Dvd848
    Commented May 22, 2021 at 18:16
  • @Dvd848, yes, well. It's a bit unclear what you're actually aiming to do. I mean, do you have some actual existing application that does reads like that, or are you just asking out of academic interest? If it's an existing application, then it would help to know which one it is, since it would probably have some other way to do that. If you're asking just out of curiosity, then yeah, datagram sockets are one thing that definitely keep message boundaries intact. Most stuff just works like streams of bytes with the structure built on the application level.
    – ilkkachu
    Commented May 22, 2021 at 20:53
1

As suggested in the comments below you could do:

{ echo a & sleep 0.1; echo b; } | ./main

Notice that you may have to adjust amount of time to sleep. The point of this command is to make the first invocation of read() think that a is the entire input it got. The assumption is that the C program will reach the second read() after echo a & (note the & - it's sent to the background) has already finished and was processed by the first read(). But since Linux is a multi-user true multitasking OS which additionally performs lazy virtual memory allocation sleep 0.1 may not be enough in all cases for this assumption to work.

The reason it works and

{ echo a && echo b; } | ./main

doses not is that read() reads the entire available stdin up to SIZE characters the first time it's read, leaving no character to be read for the second time. If you checked the value read() returns:

#include <stdio.h>
#include <unistd.h>
#include <stdint.h>

#define SIZE 0x100

int main(void)
{
    char buffer1[SIZE];
    char buffer2[SIZE];

    printf("Enter first line of input: \n");

    ssize_t read_bytes = read(STDIN_FILENO, buffer1, SIZE);
    buffer1[read_bytes] = '\0';
    printf("First input - count of read bytes: %jd\n", (intmax_t) read_bytes);

    printf("Enter second line of input: \n");

    read_bytes = read(STDIN_FILENO, buffer2, SIZE);
    printf("Second input - count of read bytes: %jd\n", (intmax_t) read_bytes);
    buffer2[read_bytes] = '\0';

    printf("\nFirst input:\n%s", buffer1);
    printf("\nSecond input:\n%s", buffer2);

    return 0;
}

you'd see that it doesn't read any character the second time:

$ ./main << EOF
1
2
EOF
Enter first line of input:
First input - count of read bytes: 4

First input:
1
2
Enter second line of input:
Second input - count of read bytes: 0

Second input:

First input:
1
2

Second input:

To make { echo a && echo b; } | ./main work you'd have to either switch to getline() or save both inputs to a single buffer and use strtok() to parse the buffer by newlines. getline() version could look like this:

#include <stdio.h>
#include <unistd.h>
#include <stdlib.h>

int main(void)
{
    size_t size = 0x100;
    char *buffer1 = malloc(size);
    if (buffer1 == NULL)
        {
            perror("malloc");
        }


    char *buffer2 = malloc(size);
    if (buffer2 == NULL)
        {
            perror("malloc");
        }

    printf("Enter first line of input: \n");
    getline(&buffer1, &size, stdin);

    printf("Enter second line of input: \n");
    getline(&buffer2, &size, stdin);

    printf("\nFirst input:\n%s", buffer1);
    printf("\nSecond input:\n%s", buffer2);

    free(buffer1);
    free(buffer2);

    return 0;
}

Example:

$ ./main << EOF
1
2
EOF
Enter first line of input:
Enter second line of input:

First input:
1

Second input:
2

There are 3 more points I'd like to discuss here:

  • you don't need fflush(stdout); because stdout is always flushed after a newline

  • you don't need to lookup for man pages on the Internet because you have them locally - just type man 2 read in the terminal or open them inside your editor (Emacs for example can do that)

  • you have a bug in the code you posted in your question - read() does not add nul bytes automatically, you have to do it yourself to avoid UB. It should be:

      printf("Enter first line of input: \n");
      ssize_t read_bytes = read(STDIN_FILENO, buffer1, SIZE - 1);
      buffer1[read_bytes] = '\0';
    
7
  • @ArkadiuszDrabczyk - Thanks, I agree with your recommendation that using a different function instead of read would be better, but the assumption is that the code cannot be changed.
    – Dvd848
    Commented May 21, 2021 at 20:47
  • But you have to change it if you want it to behave differently. Commented May 21, 2021 at 20:48
  • @ArkadiuszDrabczyk - I tried to create a simplified reference code for the problem (also the reason for skipping error checking, NULL byte addition etc.), I don't control the actual program sources.
    – Dvd848
    Commented May 21, 2021 at 20:50
  • 1
    { echo a & sleep 0.1; echo b; } | ./main only works if ./main is scheduled between the two writes by the left-hand echo's. That's not guaranteed, and might not be the case on a heavily-loaded system.
    – ilkkachu
    Commented May 22, 2021 at 16:54
  • 1
    @ilkkachu: both problems fixed (I'd like everyone to read my posts so carefully!) Commented May 22, 2021 at 17:11

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .