Questions tagged [buffer]

This tag is for questions about kernel buffer caches, including pipe buffers. These are used to store the recently accessed files and/or frequently accessed files.

Filter by
Sorted by
Tagged with
1
vote
1answer
945 views

What is the difference between buffer and cache?

I know that a cache is used to improve the speed of reading data from hard drive because the speed of reading data from hard drive and the speed of reading data from RAM are very different. So the ...
-1
votes
1answer
305 views

Problems with external hard disk

I can't overcome a problem I have with my hard disk! I don't know if it is the right section, but the problem I have started suddenly while I was copying files on Debian. The problem is that my ...
1
vote
0answers
229 views

Ubuntu crashes on startup with a call trace (release_bts_buffer)

A friend of mine is having problems with her Ubuntu laptop and I have trouble finding informations that could help me know what is going on. I think this is a bit beyond the scope of my abilities. ...
13
votes
1answer
5k views

`unbuffer` or `stdbuf` for removing stdout buffering?

Is there a difference between unbuffer(1) and stdbuf(1)? From what I gather, unbuffer makes more than the "best effort" of calling the libc function set(X)buf at the beginning, and then letting things ...
27
votes
3answers
6k views

Is there truth to the philosophy that you should sync; sync; sync; sync?

When I was first introduced to Linux, working at Cisco Systems in 2000, I was taught the merits of the sync command, used to flush buffers to disk to prevent filesystem corruption / data loss. I was ...
1
vote
1answer
142 views

Can't grep output of apt-get

If I run: apt-get install --dry-run 2klic-embedded-gui=*-master I get: Reading package lists... Done Building dependency tree Reading state information... Done 2klic-embedded-gui is already ...
1
vote
2answers
1k views

stdin unbuffered

I'm executing a program with an stdin redirection: $ prog < f In that case, the stdin is fully buffered. Is there a trick to make it unbuffered or line buffered? EDIT. Without modifying the ...
1
vote
1answer
1k views

Is it useful for a proxy to buffer less data than Linux RCVBUF can?

I am looking at how HTTP proxies and reverse proxies deal with slow client problems. The idea is that the upstream server only have a limited slots for clients and if the client is slow to receive ...
2
votes
2answers
1k views

How to make any program work with the tee command?

I wrote a program (in Ruby) and it worked well. It prints out a few lines of text and then pause for a minute and then repeats the above. It works well (in Mac OS X's terminal), but I noted that if I ...
17
votes
5answers
5k views

Is there a way to sync only one partition?

Is there a way to sync only one partition instead of all partitions? Something like "sync /dev/sdc1".
6
votes
3answers
2k views

How Do I Unbuffer The Output Passed From an Interactive Command Into a Pipeline Ending With `tee`?

     I'm troubleshooting an interactive command and would like to: see output printed to my screen with its original coloration, unbuffered or line-buffered (instead of ...
2
votes
0answers
100 views

linux: Discard dirty pages for block device (on usb disconnect)

When forcefully removing a USB device on linux while there are outstanding writes to the device the kernel log fills up with blk_update_request: I/O error, dev sdg, sector 50618368 Buffer I/O error ...
0
votes
2answers
520 views

Is a buffer always on the hash queue?

The author while talking about buffer cache in the book - The design of the UNIX operating system, says that a buffer is always on a hash queue, but it may or may not be on the free list. What ...
2
votes
1answer
290 views

How to correctly unbuffer a pipeline?

I'm trying to follow various answers in this, this and this threads, but my pipeline still appears buffered. Here's an example attempt: unbuffer sh -c 'echo -n test; sleep 1; echo; sleep 1; echo -n ...
2
votes
2answers
871 views

Is there a command like “tee” that limits the file size and treats the output file like a fixed-size queue?

Does there exist a command where one can do something like: someprogram | tee --limit=1MB afile Whereby "afile" would contain the most recent 1 MB of output of "someprogram"?
3
votes
0answers
139 views

Anyone know where to change this “bufsize” value? ( st / mt LTO tape drives )

I am trying to speed test some new LTO tape drives but cannot seem to send data to the tape via dd for any block size above 327,680 bytes. I must have a 1M blocksize for my application. [root@host]# ...
4
votes
3answers
1k views

Replacing shell script while running it

I have a board that is running a "patch" script. The patch script always runs in the background and it is a shell script that runs the following pseudo code: while true; do # checks if a patch ...
18
votes
3answers
6k views

Prevent unwanted buffers from opening

I'm using emacs for my daily javascript editing, to switch between buffers I use C-x LEFT and C-x RIGHT and I'm fine with this (even if I find it difficult to know the path of the file I'm modifying). ...
0
votes
1answer
999 views

Buffer memory increased

My buffer memory in server got increased after taking backups from multiple servers using rsync. This disk got almost full, removed the backups. As far as I understand buffers shoot up if we perform a ...
3
votes
2answers
11k views

Can I increase the system pipe buffer max?

I'd like to make a FIFO pipe with a buffer of ~5MB. I know the default FIFO pipe buffer max in linux is around 1MB. I see that it lives in /proc/sys/fs/pipe-max-size I tried to set it as follows: ...
0
votes
1answer
457 views

`tail -f` consumes last line partially, doesn't care about newlines or nul

I've got a temporary file (not a fifo/pipe) that needs to be watched by multiple reader scripts. Each script uses a background process to watch the temporary file, using this code: function ...
5
votes
1answer
904 views

Can DD be used to add a buffer to a pipe?

I'm reading and writing audio from a named pipe. I'm running into an issue where my read process lags, causing the pipe's buffer to fill up and drop frames. I'd like to insert a buffer of arbitrarily ...
1
vote
0answers
648 views

How does Linux kernel know when to use memory buffer/cache?

When we do a find in linux, I am guessing the kernel will store the result into buffer/cache. Let say after an hour, some changes of the folder and files happened, so my question is, when we do the ...
5
votes
2answers
814 views

Read everything in a pipe's buffer without waiting

Is there a way to read everything already in the buffer of a pipe, pipe them to another program, and exit immediately without waiting for any more input? It has to be binary safe. It's ok to require ...
12
votes
6answers
2k views

A program that could buffer stdin or file

Aside from using a temporary file to help, is there a way/program could buffer input from stdin but does not output the contents until getting the EOF. I don't want to use a shell variable either(e.g. ...
7
votes
1answer
6k views

Increase FIFO size limit

Thanks to the answers to my other question, I now understand that FIFO on Linux, i.e. /dev/xconsole has a buffer limit of 64 KB. How can I increase this limit to 128 KB? Apparently, I will need to ...
2
votes
1answer
30 views

Is it possible to give caching preferences to the filesystem?

I have a directory (/srv/data) with a large number (20,000,000) with a number of small (4-40KB) files. These files are all located in subdirectories of /srv/data, where the concatenation of the ...
4
votes
1answer
2k views

Unbuffered socat command to connect serial ports in remote machines and log the data

I'm currently using the following command. It reads from serial ttyUSB0 in local machine, and bidirectionally connects to ssh, through two tee commands for logging. On the remote end, socat connects ...
1
vote
2answers
151 views

locate(1) buffers its output [duplicate]

I'm using locate(1) from GNU findutils for a little task and it seems as if it buffers its output. I am piping the output of locate to another task that will process the lines as locate finds them. ...
2
votes
1answer
223 views

regular awk buffering issue when calling shell commands

Some precisions: regular awk, NOT gawk AIX 6.1, and old shell: GNU bash, version 2.05b.0(1) I am trying to display some things in the right order, hence I do pipe outputs through the shell's "sort" ...
1
vote
2answers
166 views

Misbehaving irssi Pagination in tmux

I'm experiencing some strange console pagination behaviour in tmux when SSH'd to a Ubuntu host (EC2 instance on AWS). I've demonstrated this behaviour by running irssi inside the tmux and hitting PgUp ...
5
votes
2answers
2k views

How to control buffering of pipes in bash scripts?

I need my script to be executed with stdbuf -i0 -o0 -e0 otherwise user will wonder why there is no output (becuase of pipe buffer). How can I acheive it? #!/usr/bin/stdbuf -i0 -o0 -e0 /bin/bash ...
7
votes
1answer
3k views

How to reduce Linux' write buffer for removable devices?

When writing content to removable devices on Linux (USB sticks/HDDs, SD cards, etc), I often see incredible write speeds in the first few seconds, sometimes in the order of GB/s (filling of the write ...
1
vote
1answer
1k views

Filter out duplicate lines of tail -f [duplicate]

I often need to tail -f apache access logs for websites to troubleshoot issues- one thing that makes it annoying is that anyone loading a page once may cause 12+ lines to get written to the log, and ...
14
votes
2answers
3k views

“tail -f | iconv -fsjis” does not output anything

I want to tail -f a file, but its content is in sjis encoding, so I need to have it converted to the native (utf-8) encoding of my terminal. When I do tail -f x | iconv -fsjis there will be no ...
24
votes
1answer
15k views

What exactly is scrollback and scrollback buffer?

What are "scrollback" and "scrollback buffer" in programs like bash and screen, and how do they relate to the tty, the programs being run, and stdin/ stdout/ stderr? Here is the only definition of "...
3
votes
1answer
846 views

non-blocking/multi-threaded cat

cat is blocking from what I see in the code, i.e. it uses blocking read() and then it uses blocking write(). I want to call some tool where I disable all stdout buffering by purpose (e.g. like ...
4
votes
2answers
823 views

stdbuf supposed behavior for subprocesses

I didn't exactly find something about the following in the man-page. How is the supposed behavior in subprocesses spawned by a process which was itself spawned by stdbuf? E.g.: stdbuf -oL myprog ...
3
votes
0answers
301 views

Does Linux perform buffering twice when hosting a VM?

I ran a KVM VM on my CentOS host and found that after performing a HDD-intensive operation, it used all of the host's memory I assigned to it. However, free on the guest suggests that the memory's ...
4
votes
3answers
1k views

Does sysctl kern.bufcachepercent not work in OpenBSD 5.2 above 1.7GB?

I run OpenBSD/amd64 5.2 on a system with 12GB of RAM, and I want to use about 6GB to 8GB for filesystem caching. By default, 5.2 amd64 comes with sysctl kern.bufcachepercent set to 20 (20%); I've ...
16
votes
1answer
20k views

Buffer size for capturing packets in kernel space?

Going through the man page of tcpdump, it seems kernel can drop the packets if the buffer is full. I was wondering if: that size is configurable and/or where can I see the size for my distro? From ...
28
votes
3answers
10k views

How to change the name of a vim buffer

Is it possible to change the name of a buffer in vim? Specifically, I'm using Conque Shell to open shells in vim (each shell is in a buffer) and with multiple shells, I see: 10: bash - 1 11: bash - ...
12
votes
5answers
3k views

Utility to buffer an unbounded amount of data in a pipeline?

Is there a utility that I can stick in a pipeline to decouple read and write speeds? $ producer | buf | consumer Basically, I want a utility buf that reads its input as fast as possible, storing it ...
4
votes
3answers
1k views

use line buffering when piping output to file

I have a perl program that prints to stdout, which I then pipe into a file, like this: ./skript.pl > file.txt 2>&1 This makes the output of that skript buffer. However, I would like to be ...
3
votes
1answer
6k views

How to identify cause of large buffer memory usage?

I have a production machine running a 2.6 kernel (Centos 6.7) with 64 GB of RAM of which 18 GB is apparently used for buffer memory. I am interested in identifying specifically what that 18 GB is ...
4
votes
2answers
1k views

Avoid output buffering for a command group (curly braces) in bash script

I have a bash script containing a group of commands in curly braces { ... }. This group contains some initial echo commands and then one loop. At each iteration the loop executes various slow commands ...
4
votes
0answers
705 views

Understanding inotifywait, pipes and buffers

I want to monitor every file change in a directory with inotifywait. inotifywait shall write to a FIFO buffer, which then can be read, leisurely. While experimenting with relatively huge amounts of ...
1
vote
2answers
718 views

awk output has extra carriage returns and is buffered

My ultimate goal is to get a zenity progress dialog to work while encoding with fdkaac. I first started with some code that is working when I encode "aa.wav" to "aa.mp3" with lame. This results in a ...
0
votes
1answer
236 views

Wanted: nonassociative array in AWK

I am looking at speeding up some code in AWK by serially processing an internal string representation instead of serially processing an array representation. (For Examples 1 and 2 below assume ...
4
votes
3answers
1k views

How to append data to buffer in shell script?

I'd like to do following using shell script: (for simplicity, I use same data for INPUT. In real case, the data changes with loop label jj) #!/bin/sh for jj in `seq 100`; do cat INPUT.file >&...