Skip to main content

Questions tagged [buffer]

This tag is for questions about kernel buffer caches, including pipe buffers. These are used to store the recently accessed files and/or frequently accessed files.

Filter by
Sorted by
Tagged with
0 votes
0 answers
22 views

How can I get the _real_ progress of dd, ignoring fast/staging buffers? [duplicate]

I'm using the dd command to write to a USB stick. My command is pretty straightforward: dd if=myimage.iso of=/dev/sdd bs=1M status=progress and indeed, I seem to be getting the progress reported: ...
einpoklum's user avatar
  • 9,653
0 votes
1 answer
44 views

Is my SSD dead? can't mount. I/O errors, 29k+ fsck errors

So I work with a couple of people and we ssh into a Linux server for work. There are several SSDs, and one of the non-boot Samsung 870 SSDs crashed the other day in the morning. This led to an ...
Derek Xiao's user avatar
0 votes
0 answers
46 views

Reading the output from arduino slower than it produces the output: what is the buffer size?

I have two different boards with Atmega16U2 (Uno) and ch341 (Nano). They are not genuine actually, but it doesn't matter here I think. I was asked to look at some code which didn't work as expected. ...
LRDPRDX's user avatar
  • 133
0 votes
2 answers
112 views

Terminating the whole pipeline at once when its last process ends

Given the pipeline printf '%s\n' 1 2 3 4 5 6 7 8 9 | while read -r num do echo "$num" > /dev/stderr echo "$num" done | while read -r num do echo $(( $num * 10 )) [ &...
fuumind's user avatar
  • 449
0 votes
1 answer
38 views

Is it possible to set larger buffers for file access on linux

I have a process thats reading the whole filesystem and hashing files. It slows down (by 4x or so) because the reads are causing a lot of seeking. Small chunks of each file are being read by each of 4 ...
Pete's user avatar
  • 133
0 votes
1 answer
32 views

How do Kernel use pagecache?

I have a problem with pagecache that I don't understand. As I understand, the pagecache will serve as disk cache for reading from and writing to disk. But I don't know how the kernel can map 10G ...
Tai Nguyen Huu's user avatar
0 votes
0 answers
50 views

Output of python script like if it's typed in interactive mode

I have a python file and I want to obtain the output just like I'm typing the content in interactive mode (like this question https://stackoverflow.com/q/59008423 but with a shell script) I try with ...
Pablo Cárdenas's user avatar
18 votes
1 answer
1k views

Why does unbuffer -p mangle its input?

$ seq 10 | unbuffer -p od -vtc 0000000 1 \n 2 \n 3 \n 4 \n 5 \n 6 \n 7 \n 8 \n Where did 9 and 10 go? $ printf '\r' | unbuffer -p od -An -w1 -vtc \n Why was \r changed to \...
Stéphane Chazelas's user avatar
0 votes
1 answer
522 views

Linux: Getting the kernel buffer size for a socket

I have a C application which receives a lot of data over a TCP socket. Is it somehow possible to get the kernel buffer size for that file descriptor / socket? I would like to know how much data is ...
Kevin Meier's user avatar
0 votes
1 answer
465 views

conditional flush tcp sockets on linux

whenever I write data to a tcp socket, it waits until its filled with enough data ex: 64k (max packet size) then it sends data to the wire. this misses the data at the right time for the other end. ...
user3840019's user avatar
1 vote
1 answer
209 views

Create a file for writing with controlled flushing to disk in large chunks

On Linux I have a process (ffmpeg) that writes very slowly (even slower than 1kb / s sometimes) to disk. Ffmpeg can buffer this to 256kb chunks that get written infrequently but ffmpeg hangs ...
Pete's user avatar
  • 133
0 votes
1 answer
427 views

How to clear the /dev/kmsg ring buffer?

Is there a way for me to force clear the /dev/kmsg kernel ring buffer?
Nimrodshn's user avatar
  • 101
7 votes
1 answer
654 views

Why does tar's handling of stdout and - differ?

After a comment from OP, I discovered that /dev/stdout gives blocks of 10 KiB even after disabling buffering, but - does not. Why is this? I could not find anything regarding this in man tar nor man ...
wjwrpoyob's user avatar
  • 438
0 votes
0 answers
2k views

linux drop_cache using "echo 3 > /proc/sys/vm/drop_caches" not working as expected

In our production environment we are running drop cache command echo 3 > /proc/sys/vm/drop_caches to free the RAM. But also what I found is dropping caches is not a good practice and also it won't ...
Swastik's user avatar
  • 103
0 votes
1 answer
1k views

Is rsync really a culprit for causing low memory on our VM?

In a VM of 16GB RAM, we are running rsync as a cron job(for every 10 minutes) on our production to sync GBs of folder from AWS EFS to local storage. After few days of running we found that the VM is ...
Swastik's user avatar
  • 103
2 votes
0 answers
190 views

How to find the memory address of the parameter(s) of a function

For context, I am attempting to exploit a buffer overflow as a practice exercise. I need to pass a parameter into a function via memory address. I have found the address for the function I would like ...
1Poseidon3's user avatar
0 votes
0 answers
81 views

Why piping cat into head -c 5 for a chardev results in many more calls to the driver's read than just calling head -c 5 on the chardev?

Taking inspiration from this blog post, I'm playing around with linux device drivers (which I'm studying from ). The read field of the file_operations associated with the driver is initialized to the ...
Enlico's user avatar
  • 1,677
0 votes
3 answers
453 views

How can I create an expanding buffer?

I want to be able to scroll back through all of the output for each session in terminal. When using cat on large files, for example, there could be hundreds, or even thousands, of lines (I write a lot ...
Lee's user avatar
  • 97
3 votes
1 answer
215 views

Why is the stdout line buffering for cut -c differ from cut -b?

Our RHEL 7 machines have great long Log files and I asked about buffering of cut in this question. That question remains but a bit of experimentation showed a different issue. I decided to try using ...
user1683793's user avatar
1 vote
1 answer
541 views

How is the default stdout buffering set?

Running on Red Hat EL7, we have these great long lines in our log files so I do a tail -f Log | cut -c1-$COLUMNS This works great on some systems but other--apparently identical--systems, the pipe ...
user1683793's user avatar
1 vote
1 answer
90 views

Piping line-by-line

I have a script that "listens" to amixer events (volume change and mute), piping the volume and mute status to xob. It works great, but the (ab)use of stdbuf -oL <command> looks wrong ...
fff's user avatar
  • 13
0 votes
1 answer
566 views

Writing to unix socket file as fast as possible

I'm trying to write with perl to a unix socket file as fast as possible but failing on full message buffer. I tried autoflush and manual flush without success. use IO::Select; use IO::Socket::UNIX; ...
user496909's user avatar
1 vote
1 answer
2k views

Is it safe to clear the buffers and cache on a linux server

After reading this link: How do you empty the buffers and cache on a Linux system?, I know that there are some commands that can help us to empty the buffers and cache of the OS. But I'm not sure if ...
Yves's user avatar
  • 3,331
0 votes
2 answers
735 views

Why is there less latency and jitter when there is additional high bandwidth traffic between two computers with Ubuntu 22.04 installed on them?

I have two computers that are connected via an Ethernet cable with Ubuntu 22.04 installed on them. I have a client on computer A) which is sending UDP packets to a server on computer B) and I am ...
Nandor Takacs's user avatar
2 votes
1 answer
312 views

pv gives 100% progress with small buffer

I have to do a long task (convert image format) for every file of a folder. I achieved to use pv to write the estimation of duration with (used sleep here to simulated processing time): pv -B 1 =(find ...
rafoo's user avatar
  • 121
0 votes
1 answer
126 views

How to trigger two commands when pressing a key bind

So I wanted to pass the text stored in my buffer through a socket to other machine, for that I manage to get a socket done, I just need to fire up the following command when in copy-mode-vi bind -T ...
freddie_ventura's user avatar
0 votes
0 answers
562 views

Why are packets being dropped (kfree_skb)?

I am currently sending ~9mpps, each packet is 72bytes large, and is a UDP DNS request. The packets are being sent to a Dell server I have using an intel i40e interface. The server is using all of its ...
Dave's user avatar
  • 621
0 votes
2 answers
4k views

Explanation about free, available and buff/cache and how to find what process is buff/cache saved for in a server in the command "free -mh "

I see explanation for free -mh as: free displays the total amount of free and used physical and swap memory in the system, as well as the buffers and caches used by the kernel. The information is ...
Tayto's user avatar
  • 1
1 vote
2 answers
749 views

why do we need to pass buffers to system calls in order to have information returned? Why can't system calls allocate the buffer internally?

I'm reading TLPI, and in section 4.4: System calls don’t allocate memory for buffers that are used to return information to the caller. Instead, we must pass a pointer to a previously allocated ...
sudeepdino008's user avatar
1 vote
1 answer
2k views

Clear socket buffer of CAN adapter

I have a USB/CAN adapter, from lsusb: OpenMoko, Inc. Geschwister Schneider CAN adapter it provides a socket interface, from ip addr: 5: can0: <NOARP,UP,LOWER_UP,ECHO> mtu 16 qdisc pfifo_fast ...
Mark's user avatar
  • 733
0 votes
1 answer
125 views

When is it beneficial to add -u to tr?

One of the arguments to the tr command is: -u Guarantee that any output is unbuffered. Under what circumstances is it a good idea to not buffer the output? e.g. should I use -u if I expect each ...
falsePockets's user avatar
0 votes
0 answers
181 views

Are direct writes to block devices buffered? How do I ensure it's written to disk before removing it?

Say I do something like dd if=file of=/dev/sdc (simplified for illustration), is this buffered? Can I sync it like I would any other filesystem write, or do I need to do something else to make sure ...
Daffy's user avatar
  • 425
5 votes
1 answer
639 views

Why is mawk's output (STDOUT) buffered even though it is the terminal?

I am aware that STDOUT is usually buffered by commands like mawk (but not gawk), grep, sed, and so on, unless used with the appropriate options (i.e. mawk --Winteractive, or grep --line-buffered, or ...
ChennyStar's user avatar
  • 1,795
1 vote
1 answer
9k views

Buffer I/O error on dev sda6, logical block 44, async page read

I don't know why this happens... I was just doing my work (development) and suddenly one of my drive start lagging then I just unmount it by right click on it. After it is still lagging, so I restart ...
Rohit Nishad's user avatar
3 votes
1 answer
822 views

configure vim to use both tmux and system clipboard

I'm fairly new to vim (I'm using nvim) so forgive my lack of knowledge, many questions are similar to mine, but not quite the same. I would like this: vim by default should cut/copy to a buffer, not ...
5c0tt_b0t's user avatar
1 vote
1 answer
2k views

How does buffering for TCP packets work?

If I write a few times in a quick succession on a socket (with the POSIX function write), usually all the data I wrote gets sent in a single TCP packets. Unless I write too much or unless I wait too ...
Blue Nebula's user avatar
1 vote
0 answers
41 views

linux handling of files

I want to perform updates on a file via multiple processes in parallel. These processes all open this file for write in parallel. Abbreviations used: f : file, p[i] : process i, b[i] : buffer for FD ...
uzumas's user avatar
  • 150
1 vote
1 answer
824 views

Pipe from xwd to ImageMagick's convert produces cropped image

I'm trying to take a screenshot of a remote host (using xwd) and convert it to png (using convert). I need it to be an unattended oneliner, so this is what I'm doing: sshpass -p THE_PASSWORD ssh user@...
José Tomás Tocino's user avatar
2 votes
0 answers
619 views

ping: sendmsg: No buffer space available

I am running a ping from one of the terminals: ping www.google.com This runs without any issues indefinitely. Now on another terminal, I run: for i in {1..5}; do speedtest; done Now after sometime on ...
Jean's user avatar
  • 211
1 vote
1 answer
1k views

Do sync and writing to /proc/sys/vm/drop_caches really empty all the disk cache/buffers?

When you want to empty disk caches/buffers, you usually issue the following command: # sync && echo 3 > /proc/sys/vm/drop_caches But after typing that command, free doesn't really report 0 ...
Mikhail Morfikov's user avatar
3 votes
2 answers
544 views

How do I feed data to a pipe until it's full, no more no less?

I want to use pipes on Linux as a synchronization primitive between a master process and a slave process. The classic way is to create two pipes, but I believe there's a way to use a single fd instead....
Alisa Sireneva's user avatar
1 vote
0 answers
190 views

Buffer Memory Drops to Zero

We have been having problems with a system that becomes inaccessible. Characteristics are 1) sudden spike in blocked and run queues and 2) buffer memory drops to zero. The symptoms are 1) existing ...
EricN's user avatar
  • 11
3 votes
1 answer
865 views

broken pipe when tar extracted from stdin buffer

I am restoring files from a tar archive on an LTO-7 tape to a locally mounted network share. If I directly restore to the share it runs very slowly (90 MB/s). When I use an additional buffer I get the ...
swami's user avatar
  • 205
0 votes
1 answer
196 views

Bright Color Escape Sequences not saved in Tmux capture-pane?

I'm trying to use tmux to create a pty for me that I can read the contents of, including color. I'm using tmux capture-pane -e -t paneName and tmux save-buffer tmuxbuffer.txt and I get normal colors, ...
Gregory's user avatar
  • 53
0 votes
0 answers
839 views

Effect of Buffer Size in File I/O in Unix

I am trying to understand the inner workings of Unix based OSs. I was reading on buffered I/O, and how the buffer size affects the number of system calls made, which in turn affects the total time ...
Prashant Pandey's user avatar
4 votes
2 answers
489 views

Buffering stdout separately from stderr

I am running a utility, which emits the following: Progress to its Standard Error Data/Yield/Output to its Standard Output I did not build the utility, nor can I easily modify it. I wish to do the ...
JamesTheAwesomeDude's user avatar
2 votes
1 answer
333 views

Does it have 3 levels of buffering, or 2 levels of buffering, when using stdio library?

From The Linux Programming Interface: (I've read the related sections. I pasted this sum-up diagram just for illustration.) I know that for direct IO which uses read and write system calls. There is ...
Rick's user avatar
  • 1,197
2 votes
2 answers
721 views

File order on FAT/FAT32/VFAT file systems

I have several audio devices (car radio, portable radio, MP3 player) that take SD cards and USB sticks with a FAT file system on it. Because these devices have limited intelligence they do not sort ...
Frank van Wensveen's user avatar
2 votes
2 answers
828 views

How to send output of `pv` to python script, and output progress one line at a time?

I use tar to quickly copy thousands of small files which I pipe to pv to show the progress and speed of the copying. $ sudo tar c --files-from /tmp/camfilenames | pv --width 40 | sudo tar x -C /...
marcinpl87's user avatar
1 vote
1 answer
785 views

Disable buffering for file descriptors in bash

I may be misunderstanding what's going on here, but I feel it's got to do with pipe buffering. I have a script which uses several file descriptors (# 3 and above) for various logging levels. Depending ...
Aayla Secura's user avatar