5

I want to write a CGI, which must read a specified number of bytes from STDIN. My idea is to do it this way:

dd bs=$CONTENT_LENGTH count=1

But I was wondering, if the block size is limited by anything else but the RAM.

$ dd bs=1000000000000
dd: memory exhausted by input buffer of size 1000000000000 bytes (931 GiB)

The manual page of GNU's coreutils does not specify any limit.

  • 2
    It depends on RAM and swap and GNU is irrelevant here. – schily Jan 21 at 13:13
  • 3
    "I want to write a CGI [in shell, using dd]" very bad idea. Also, no matter how big your $CONTENT_LENGTH is, that dd command may read just a single byte. If successful. – mosvy Jan 22 at 0:24
  • 1
    there is no limit to bs – JoelFan Jan 22 at 16:08
  • 3
    @ceving perhaps JoelFan’s comment was a play on words... – Stephen Kitt Jan 22 at 16:30
  • 1
    @ceving no need to apologise, I’m sure your English is better than my German! See this wiktionary entry for a possible interpretation (used as a noun). – Stephen Kitt Jan 22 at 16:44
24

The POSIX specifications for dd don’t specify a maximum explicitly, but there are some limits:

On a 64-bit platform, size_t is 64 bits in length; in addition, it’s unsigned, so dd will fail when given values greater than 264 – 1:

$ dd if=/dev/zero of=/dev/null bs=18446744073709551616
dd: invalid number: ‘18446744073709551616’

On Linux on 64-bit x86, SSIZE_MAX is 0x7fffffffffffffffL (run echo SSIZE_MAX | gcc -include limits.h -E - to check), and that’s the input limit:

$ dd if=/dev/zero of=/dev/null bs=9223372036854775808
dd: invalid number: ‘9223372036854775808’: Value too large for defined data type

$ dd if=/dev/zero of=/dev/null bs=9223372036854775807
dd: memory exhausted by input buffer of size 9223372036854775807 bytes (8.0 EiB)

Once you find a value which is accepted, the next limit is the amount of memory which can be allocated, since dd needs to allocate a buffer before it can read into it.

Once you find a value which can be allocated, you’ll hit the read limit (on Linux and other systems with similar limits), unless you use GNU dd and specify iflag=fullblock:

$ dd if=/dev/zero of=ddtest bs=4294967296 count=1
0+1 records in
0+1 records out
2147479552 bytes (2.1 GB, 2.0 GiB) copied, 38.3037 s, 56.1 MB/s

(dd copied just under 231 bytes, i.e. the Linux limit mentioned above, not even half of what I asked for).

As explained in the Q&A linked above, you’ll need fullblock to reliably copy all the input data in any case, for any value of bs greater than 1.

  • 2147479552 bytes (2.1 GB, 2.0 GiB) copied That number is exactly as documented: "On Linux, write() (and similar system calls) will transfer at most 0x7ffff000 (2,147,479,552) bytes". See cannot write(2) file larger than 2GB (up to 2TB) – phuclv Jan 22 at 11:21
  • Yes, see the third bullet point. – Stephen Kitt Jan 22 at 12:01
  • The "fullblock" problem gets triggered in a CGI almost always, because the CGI reads from a pipe as in the following example: { printf a; sleep 1; printf b; } | dd bs=2 count=1 2>/dev/null | wc -c. In this example dd used without the fullblock option reads just one byte instead of two. – ceving Jan 22 at 16:16
6

Regardless of its maximum value, you have a bigger problem there; from the POSIX spec:

The dd utility shall copy the specified input file to the specified output file with possible conversions using specific input and output block sizes. It shall read the input one block at a time, using the specified input block size; it shall then process the block of data actually returned, which could be smaller than the requested block size.

(emphasis added)

As I wrote in the past, dd is an extremely stupid tool: in your case, it essentially boils down to

char *buf = malloc(bs);
for(int i = 0; i < count; ++i) {
    int len = read(STDIN_FILENO, buf, bs);
    if(len == 0) break;
    write(STDOUT_FILENO, buf, len);
}
free(buf);

bs is just the argument dd uses to perform the read(2) syscall, but read(2) is allowed to perform a "short read", i.e. to return less bytes than requested. Indeed, it's what it does if it has some bytes available right now, even if they aren't all you asked for; this is typical if the input file is a tty, a pipe or a socket (so you are particularly at risk with your CGI...). Just try:

$ dd bs=1000 count=1
asd
asd
0+1 records in
0+1 records out
4 bytes copied, 1.75356 s, 0.0 kB/s

Here I typed in asd and pressed enter; dd read it (performing a single read(STDIN_FILENO, buf, 1000) and wrote it out; it did one read as requested, so it exits. It doesn't look like it copied 1000 bytes.

Ultimately, plain "standard" dd is a way too stupid tool for most needs; you can wrangle it to do what you need by either:

  • by using bs=1 and using count for the number of bytes; this is guaranteed to copy the number of bytes you need (if available before EOF), but it's quite inefficient, as it performs one syscall per byte;
  • add the fullblock flag; this makes sure that dd accumulates a full input block before writing it out. Notice however that this is nonstandard (GNU dd has it, IDK about others).

Ultimately, if you are going for non-POSIX extension, my suggestion is to just use head -c: it will do the Right Thing with sensible buffering and no particular size limits, ensuring correctness and good performance.

  • It requires 5 lines of shell code to check the size of a file. It is ok for me to buffer in memory. My upload limit is 100 MB. – ceving Jan 22 at 6:25
  • 2
    @ceving: ... and what do you gain? more complicated code that is just as brittle as the original? Just use head -c, it's concise and works correctly in all the cases. – Matteo Italia Jan 22 at 8:57
  • 3
    Yes, it's the most important point here. You'd only use dd if you wanted to make sure the data is read in one read() system call which in the OP's CGI context doesn't make much sense. Might be worth pointing out that CGIs should never be written as shell scripts if they take any input from the client as it's very difficult to avoid vulnerabilities there. – Stéphane Chazelas Jan 22 at 11:12
  • 1
    @ceving, if it's just head -c 100M > /fixed/path/to/file it's fine (as long as the shell is not a version of bash vulnerable to shellshock), but if the rest of the script handles any CGI input (CGI parameters, REQUEST_URI, HTTP_HOST...), then you're probably in big trouble already. Over half the CGI shell scripts I've seen had vulnerabilities in them. – Stéphane Chazelas Jan 22 at 12:55
  • 1
    "But what can go wrong, when I write stdin with dd into a file? I would say this is the most secure way to handle uploads." - Shell is notoriously dangerous for things like this; it can often be influenced by out-of-band options, such as environment variables. Although this was a specific incident, Shellshock is a good example that reminded the world of this. – marcelm Jan 22 at 18:54
2

The maximum depends on the system (including its allocation policies) and the currently available memory.

Instead of trying to read everything at once (you could exhaust memory, slow things down because of swap, you would have to add checks to see if it actually worked ...) you could read reasonable sized blocks with dd.

Let's say you want to read those bytes and put them into a file. In bash you could run something like this (the total bytes are in $total):

block=65535
count=$(expr $total / $block)
rest=$(expr $total % $block)
(dd bs=$block count=$count;dd bs=$rest count=1) > filename

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.