Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [buffer]

This tag is for questions about kernel buffer caches, including pipe buffers. These are used to store the recently accessed files and/or frequently accessed files.

-1
votes
1answer
46 views

What controls the buffering of stdout stderr?

What part of the system sets up the buffering of the three standard streams when a program is started? Is this part of linux, or glibc, or maybe bash? Does POSIX define the behaviour, or is it part ...
0
votes
0answers
10 views

Linux RedHat write buffer config

I have Linux RedHat 6.2. How do I configure files to buffer 4096 (or higher 2^N values) bytes before writes? Of course, when an EOF is received, write the file.
0
votes
0answers
28 views

UDP send buffer error in netstat

I we are running voice application which stream UDP (RTP) packets for voice, I am investigation some strange issue where my ifconfig ens4 | grep error counter going up specifically overrun If you see ...
3
votes
1answer
38 views

Using multiple copy buffers in Bash

Ctrlu cuts text from current location to the beginning of line. Now, I can paste this with Ctrly whenever I want until the next time I hit Ctrlu to copy another line in the buffer. Now the problem I ...
0
votes
0answers
105 views

flush tty device buffer from terminal

I have a serial device on /dev/ttyAMA0. I have program written in C that communicates with this device. But apart from the C program I sometimes need to input some configuration data from terminal, ...
0
votes
0answers
22 views

Sizing of Samba Linux Server

Do you know any performance of sizing recommendations about Samba 4.7.1 ? For few informations : 2vCPUs / 1 Core 8GB of RAM 140 simultaneous Windows users / 250 at most / 20 NFS volumes mounted via ...
4
votes
0answers
32 views

Tailing file using less +F/shift-F, turn off line buffering?

When using tail -f, it prints each character as it is written to the file, but less +F won't print until it sees a linefeed. And if the data being written to the file does not contain linefeeds, must ...
2
votes
0answers
40 views

fq_codel not working on saturated link?

I'm setting up a test to saturate a link between two machines with a gig router in between them, then run fq_codel on Debian Buster and see if my ping times can improve (what I think fq_codel is ...
0
votes
0answers
22 views

Lifespan of blocks in a UNIX block-cache

Recently, I've been getting into OS development and have been experimenting with writing my own small kernel. I've just begun to look understand and implement a block-cache, for caching data read from/...
0
votes
0answers
29 views

Finding usage of “buff/cache” (as mentioned in free)?

When I run free, I see free -h total used free shared buff/cache available Mem: 15Gi 1.7Gi 4.3Gi 149Mi 9.5Gi ...
0
votes
0answers
44 views

Buffering issue when looking at netcat output

I have problem I am trying to understand: I am sending some UDP data to local host. It is a simple UDP packet with a counter. It counts up to 16 (0xF) and stops. When I grep the received data on the ...
2
votes
1answer
273 views

Why do terminal emulators still have screen flicker?

Why are there still visual artifacts when terminal emulators draw text-based applications? This on recent computers that render 3D games and GUI windows including anti-aliased vector fonts with no ...
1
vote
0answers
17 views

How can I detect a buffer overflow for attached usb recording devices? [closed]

I am streaming data with the realsense d435 camera and recording it into a ros bag. Sometimes there are chunks of data missing and I suspect it is a buffer problem. Is there a way to monitor the size ...
1
vote
2answers
121 views

why /proc/net/udp is showing wrong rx_queue size in CentOS server?

I have a java server application. while(true) { serverSocket.receive(receivePacket); process(receivePacket); serverSocket.send(sendPacket); try { ...
3
votes
1answer
52 views

Page cache behavior guarantees and log files

Are there any guarantees as to when and in what order dirty pages are written out to disk? In the specific case of log structured files (files that are only ever appended to), is it possible for dirty ...
0
votes
0answers
14 views

How to get the network buffer usage of some process

I have an Ubuntu server, which contains some user-processes. It always crashes after starting-up several hours. "crashes" means that I cannot neither ssh to it nor ping it anymore. I'm doubting that ...
1
vote
1answer
57 views

How to pipe to less unbuffered?

I'm trying to display progress, and make it usable under less: while true; do echo -n .; sleep 1; done | less Is there a way to make it work?
1
vote
2answers
99 views

What are two caveats of line buffering in Unix?

A paragraph from APUE Chapter 5.4 about line buffer: Line buffered. In this case, the standard I/O library performs I/O when a newline character is encountered on input or output. This allows us ...
0
votes
1answer
29 views

Is buffer used only for writing or can it be used for both reading and writing?

I am reading about caching and buffering from Operating System Concepts, and here is a copy of it. Is it correct that cache is used only for reading not for writing? Is buffer used only for writing ...
2
votes
1answer
178 views

Block device cache v.s. a filesystem

Block devices provide buffering. This means that write() on a block device can return success, before the kernel has written the data to the device. A program can wait for all the buffered writes by ...
0
votes
0answers
130 views

Linux Interface RX error & rx_brb_discard counter higher

I am running CentOS 7.5 Linux and server is totally idle no in production but today i got alert stating interface eth error and when i check interface counter was very high but it stopped after that, ...
0
votes
0answers
44 views

Does the spool buffer work like a buffer in memory for the spool file on disk?

Operating System Concepts discusses spooling in the following. "A spool is a buffer". Is the spool buffer in memory? "Each application’s output is spooled to a separate disk file." Are spool files ...
1
vote
1answer
94 views

free -m: free size reduced because of cache/buff

I'm using the function rosbag of ROS. rosbag can record lots of data from other ROS nodes. In a word, it will generate a huge file, such as a file of 200MB. After generating such a file, I found that ...
0
votes
0answers
139 views

rx_drops_no_pbuf drops on Emulex Corporation OneConnect 10Gb NIC

I have noticed overrun on one of my HP server which has following nic [root@CentOS7 ~]# lspci | grep -i eth 03:00.0 Ethernet controller: Emulex Corporation OneConnect 10Gb NIC (be3) (rev 01) 03:00.1 ...
1
vote
1answer
45 views

Copy terminal scrollback buffer preserving ANSI escape sequences, but without pre-starting script/screen

I have a live running mate-terminal whose scrollback buffer already contains a lot of text with color/bolding, but (unfortunately) no screen management/capture program (like script/screen/tmux) was ...
1
vote
1answer
50 views

How does the kernel decides which buffer to use?

When the kernel accesses a disk block, it searches for a buffer with the appropriate device-block number combination. This line is from the book "The design of the unix operating system". So my ...
0
votes
1answer
21 views

How do I limit the maximum RAM for Linux VPS?

I rent a shared VPS that runs Debian 9. I am alloted 4GB RAM (4096MB), but sometimes my processes go over 4GB. This is largely due to the kernel caching a lot of my stuff, so I'd like to know if I ...
1
vote
0answers
62 views

hard disk high temperature and frequent access while idle

I have bought two WD Gold 4TB and installed debian sid with kde on a single RAID1 partition, md0: # cat /proc/mdstat Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [...
1
vote
1answer
555 views

What is the difference between buffer and cache?

I know that a cache is used to improve the speed of reading data from hard drive because the speed of reading data from hard drive and the speed of reading data from RAM are very different. So the ...
-1
votes
1answer
209 views

Problems with external hard disk

I can't overcome a problem I have with my hard disk! I don't know if it is the right section, but the problem I have started suddenly while I was copying files on Debian. The problem is that my ...
1
vote
0answers
156 views

Ubuntu crashes on startup with a call trace (release_bts_buffer)

A friend of mine is having problems with her Ubuntu laptop and I have trouble finding informations that could help me know what is going on. I think this is a bit beyond the scope of my abilities. ...
1
vote
1answer
97 views

Can't grep output of apt-get

If I run: apt-get install --dry-run 2klic-embedded-gui=*-master I get: Reading package lists... Done Building dependency tree Reading state information... Done 2klic-embedded-gui is already ...
-1
votes
2answers
622 views

stdin unbuffered

I'm executing a program with an stdin redirection: $ prog < f In that case, the stdin is fully buffered. Is there a trick to make it unbuffered or line buffered? EDIT. Without modifying the ...
2
votes
2answers
699 views

How to make any program work with the tee command?

I wrote a program (in Ruby) and it worked well. It prints out a few lines of text and then pause for a minute and then repeats the above. It works well (in Mac OS X's terminal), but I noted that if I ...
6
votes
3answers
1k views

How Do I Unbuffer The Output Passed From an Interactive Command Into a Pipeline Ending With `tee`?

     I'm troubleshooting an interactive command and would like to: see output printed to my screen with its original coloration, unbuffered or line-buffered (instead of ...
2
votes
0answers
73 views

linux: Discard dirty pages for block device (on usb disconnect)

When forcefully removing a USB device on linux while there are outstanding writes to the device the kernel log fills up with blk_update_request: I/O error, dev sdg, sector 50618368 Buffer I/O error ...
1
vote
1answer
237 views

How to correctly unbuffer a pipeline?

I'm trying to follow various answers in this, this and this threads, but my pipeline still appears buffered. Here's an example attempt: unbuffer sh -c 'echo -n test; sleep 1; echo; sleep 1; echo -n ...
3
votes
0answers
102 views

Anyone know where to change this “bufsize” value? ( st / mt LTO tape drives )

I am trying to speed test some new LTO tape drives but cannot seem to send data to the tape via dd for any block size above 327,680 bytes. I must have a 1M blocksize for my application. [root@host]# ...
0
votes
2answers
389 views

Is a buffer always on the hash queue?

The author while talking about buffer cache in the book - The design of the UNIX operating system, says that a buffer is always on a hash queue, but it may or may not be on the free list. What ...
4
votes
3answers
648 views

Replacing shell script while running it

I have a board that is running a "patch" script. The patch script always runs in the background and it is a shell script that runs the following pseudo code: while true; do # checks if a patch ...
0
votes
1answer
712 views

Buffer memory increased

My buffer memory in server got increased after taking backups from multiple servers using rsync. This disk got almost full, removed the backups. As far as I understand buffers shoot up if we perform a ...
3
votes
2answers
8k views

Can I increase the system pipe buffer max?

I'd like to make a FIFO pipe with a buffer of ~5MB. I know the default FIFO pipe buffer max in linux is around 1MB. I see that it lives in /proc/sys/fs/pipe-max-size I tried to set it as follows: ...
0
votes
1answer
275 views

`tail -f` consumes last line partially, doesn't care about newlines or nul

I've got a temporary file (not a fifo/pipe) that needs to be watched by multiple reader scripts. Each script uses a background process to watch the temporary file, using this code: function ...
5
votes
1answer
706 views

Can DD be used to add a buffer to a pipe?

I'm reading and writing audio from a named pipe. I'm running into an issue where my read process lags, causing the pipe's buffer to fill up and drop frames. I'd like to insert a buffer of arbitrarily ...
1
vote
0answers
557 views

How does Linux kernel know when to use memory buffer/cache?

When we do a find in linux, I am guessing the kernel will store the result into buffer/cache. Let say after an hour, some changes of the folder and files happened, so my question is, when we do the ...
2
votes
2answers
586 views

Is there a command like “tee” that limits the file size and treats the output file like a fixed-size queue?

Does there exist a command where one can do something like: someprogram | tee --limit=1MB afile Whereby "afile" would contain the most recent 1 MB of output of "someprogram"?
11
votes
6answers
1k views

A program that could buffer stdin or file

Aside from using a temporary file to help, is there a way/program could buffer input from stdin but does not output the contents until getting the EOF. I don't want to use a shell variable either(e.g. ...
5
votes
2answers
587 views

Read everything in a pipe's buffer without waiting

Is there a way to read everything already in the buffer of a pipe, pipe them to another program, and exit immediately without waiting for any more input? It has to be binary safe. It's ok to require ...
2
votes
1answer
28 views

Is it possible to give caching preferences to the filesystem?

I have a directory (/srv/data) with a large number (20,000,000) with a number of small (4-40KB) files. These files are all located in subdirectories of /srv/data, where the concatenation of the ...
4
votes
1answer
1k views

Unbuffered socat command to connect serial ports in remote machines and log the data

I'm currently using the following command. It reads from serial ttyUSB0 in local machine, and bidirectionally connects to ssh, through two tee commands for logging. On the remote end, socat connects ...