Questions tagged [buffer]

This tag is for questions about kernel buffer caches, including pipe buffers. These are used to store the recently accessed files and/or frequently accessed files.

Filter by
Sorted by
Tagged with
396
votes
13answers
204k views

Turn off buffering in pipe

I have a script which calls two commands: long_running_command | print_progress The long_running_command prints a progress but I'm unhappy with it. I'm using print_progress to make it more nice (...
146
votes
6answers
142k views

How big is the pipe buffer?

As a comment in I'm confused as to why "| true" in a makefile has the same effect as "|| true" user cjm wrote: Another reason to avoid | true is that if the command produced ...
26
votes
3answers
5k views

Is there truth to the philosophy that you should sync; sync; sync; sync?

When I was first introduced to Linux, working at Cisco Systems in 2000, I was taught the merits of the sync command, used to flush buffers to disk to prevent filesystem corruption / data loss. I was ...
2
votes
2answers
758 views

Is there a command like “tee” that limits the file size and treats the output file like a fixed-size queue?

Does there exist a command where one can do something like: someprogram | tee --limit=1MB afile Whereby "afile" would contain the most recent 1 MB of output of "someprogram"?
17
votes
5answers
5k views

Is there a way to sync only one partition?

Is there a way to sync only one partition instead of all partitions? Something like "sync /dev/sdc1".
2
votes
1answer
2k views

Daemontools multilog loses log line time information. How to fix it?

I am using daemontools to monitor a process and its output log. I am using multilog to write the logs to disk. The run script for the log is: #!/bin/bash PATH=/usr/local/bin:/usr/bin:/bin cd /usr/...
2
votes
2answers
967 views

How to make any program work with the tee command?

I wrote a program (in Ruby) and it worked well. It prints out a few lines of text and then pause for a minute and then repeats the above. It works well (in Mac OS X's terminal), but I noted that if I ...
23
votes
1answer
14k views

What exactly is scrollback and scrollback buffer?

What are "scrollback" and "scrollback buffer" in programs like bash and screen, and how do they relate to the tty, the programs being run, and stdin/ stdout/ stderr? Here is the only definition of "...
13
votes
1answer
5k views

`unbuffer` or `stdbuf` for removing stdout buffering?

Is there a difference between unbuffer(1) and stdbuf(1)? From what I gather, unbuffer makes more than the "best effort" of calling the libc function set(X)buf at the beginning, and then letting things ...
6
votes
3answers
3k views

Are there any ways or tools to dump the memory cache and buffer?

I want to check out what files are loaded in buffer. Just so it's clear, the buffers & cache I'm referring to are what show up when you run the free -m command: $ free -m total ...
24
votes
5answers
41k views

Restrict size of buffer cache in Linux

Is there a way to tell the Linux kernel to only use a certain percentage of memory for the buffer cache? I know /proc/sys/vm/drop_caches can be used to clear the cache temporarily, but is there any ...
16
votes
1answer
19k views

Buffer size for capturing packets in kernel space?

Going through the man page of tcpdump, it seems kernel can drop the packets if the buffer is full. I was wondering if: that size is configurable and/or where can I see the size for my distro? From ...
12
votes
5answers
3k views

Utility to buffer an unbounded amount of data in a pipeline?

Is there a utility that I can stick in a pipeline to decouple read and write speeds? $ producer | buf | consumer Basically, I want a utility buf that reads its input as fast as possible, storing it ...
7
votes
1answer
3k views

How to reduce Linux' write buffer for removable devices?

When writing content to removable devices on Linux (USB sticks/HDDs, SD cards, etc), I often see incredible write speeds in the first few seconds, sometimes in the order of GB/s (filling of the write ...
11
votes
6answers
1k views

A program that could buffer stdin or file

Aside from using a temporary file to help, is there a way/program could buffer input from stdin but does not output the contents until getting the EOF. I don't want to use a shell variable either(e.g. ...
3
votes
3answers
3k views

piped command after grep not working

I want to monitor a file. So I have tailed a file using below command, to execute a script per line. tail -3f logfile.log | grep "action.*add" | sed -u -e "s/^/'/" -e "s/$/'/" | xargs -L 1 -P 5 bash ...
2
votes
1answer
309 views

Block device cache v.s. a filesystem

Block devices provide buffering. This means that write() on a block device can return success, before the kernel has written the data to the device. A program can wait for all the buffered writes by ...
6
votes
2answers
3k views

linux stdbuf - line-buffered stdin option does not exist

The man page for the stdbuf command states that line-buffered mode is invalid as a standard input buffering option. What is the reason for this? tail -f access.log | stdbuf -iL cut -d' ' -f1 | uniq ...
5
votes
2answers
2k views

How to control buffering of pipes in bash scripts?

I need my script to be executed with stdbuf -i0 -o0 -e0 otherwise user will wonder why there is no output (becuase of pipe buffer). How can I acheive it? #!/usr/bin/stdbuf -i0 -o0 -e0 /bin/bash ...
8
votes
3answers
2k views

Piping commands, modify stdin write to stdout

echo Hello World | nano - | less I want to modify Hello -> Goodbye while in the the text editor. It does not have to be with nano. I am stuck with nano reading stdin but requiring me to write to a ...
7
votes
1answer
6k views

Increase FIFO size limit

Thanks to the answers to my other question, I now understand that FIFO on Linux, i.e. /dev/xconsole has a buffer limit of 64 KB. How can I increase this limit to 128 KB? Apparently, I will need to ...
6
votes
1answer
1k views

Ksh loses data after piping 16K bytes

I recently found that ksh may lose some data after printing more than 16K bytes to the stdout if it is blocked for a couple of seconds. This test.sh script prints out 257*64 (16448) bytes: #!/usr/...
5
votes
1answer
2k views

Difference between “blockdev --flushbufs” and “sync” on Linux?

Is there a difference in practice between running blockdev --flushbufs and sync(1) on Linux? (apart from blockdev flushing for a specific device, and sync being system-wide). sync(1) manpage says it ...
7
votes
2answers
5k views

Buffering (named) pipe in GNU OS

In the GNU OS a process can only write data to a pipe if another process reads the same data (from the same pipe) at the same time. Is there something like a pipe which lets the 1st process write and ...
4
votes
2answers
758 views

stdbuf supposed behavior for subprocesses

I didn't exactly find something about the following in the man-page. How is the supposed behavior in subprocesses spawned by a process which was itself spawned by stdbuf? E.g.: stdbuf -oL myprog ...
4
votes
1answer
2k views

Unbuffered socat command to connect serial ports in remote machines and log the data

I'm currently using the following command. It reads from serial ttyUSB0 in local machine, and bidirectionally connects to ssh, through two tee commands for logging. On the remote end, socat connects ...
4
votes
3answers
1k views

How to append data to buffer in shell script?

I'd like to do following using shell script: (for simplicity, I use same data for INPUT. In real case, the data changes with loop label jj) #!/bin/sh for jj in `seq 100`; do cat INPUT.file >&...
4
votes
3answers
618 views

What to use instead of `buffer` for 1GB buffer? [duplicate]

buffer(1) seems to be old-ish and have hard-coded values preventing it to cache large amount of data. $ buffer -m 1G max_shmem 1 too low // it doesn't even understand gigabytes $ buffer -m 1000M ...
1
vote
2answers
301 views

Specifying height for vertical splits in vim

Usually I am editing two files, A and B, split horizontally. I set it up, so that one buffer would only occupy the first 2 lines at the top of the screen, and the rest is for the other buffer. ...
0
votes
1answer
463 views

What controls the buffering of stdout stderr?

What part of the system sets up the buffering of the three standard streams when a program is started? Is this part of linux, or glibc, or maybe bash? Does POSIX define the behaviour, or is it part ...