Questions tagged [buffer]
This tag is for questions about kernel buffer caches, including pipe buffers. These are used to store the recently accessed files and/or frequently accessed files.
173
questions
0
votes
1
answer
22
views
Linux: Getting the kernel buffer size for a socket
I have a C application which receives a lot of data over a TCP socket. Is it somehow possible to get the kernel buffer size for that file descriptor / socket? I would like to know how much data is ...
- 203
0
votes
1
answer
52
views
conditional flush tcp sockets on linux
whenever I write data to a tcp socket, it waits until its filled with enough data ex: 64k (max packet size) then it sends data to the wire.
this misses the data at the right time for the other end.
...
1
vote
1
answer
52
views
Create a file for writing with controlled flushing to disk in large chunks
On Linux I have a process (ffmpeg) that writes very slowly (even slower than 1kb / s sometimes) to disk. Ffmpeg can buffer this to 256kb chunks that get written infrequently but ffmpeg hangs ...
- 133
0
votes
1
answer
78
views
How to clear the /dev/kmsg ring buffer?
Is there a way for me to force clear the /dev/kmsg kernel ring buffer?
- 101
0
votes
0
answers
11
views
How to test Buffer Cache in Pintos (Pintos Project 4)
I'm working on Pintos Project 4, specifically the Buffer Cache part.
I have completed the code, but I am unsure how to test whether it is functioning correctly.
7
votes
1
answer
627
views
Why does tar's handling of stdout and - differ?
After a comment from OP, I discovered that /dev/stdout gives blocks of 10 KiB even after disabling buffering, but - does not. Why is this? I could not find anything regarding this in man tar nor man ...
- 376
0
votes
0
answers
45
views
How to allocate socket buffer memory from local memory of specific NUMA node
My computer has 2 CPUs, and each node has its local memory. They operate using NUMA architecture. The operating system (O/S) is Rocky Linux 9.1 (x86_64).
Additionally, each CPU is equipped with a ...
- 1
0
votes
0
answers
29
views
How to implement lossy buffering? [duplicate]
I want to capture log data and analyze it in real time. Specifically the output of mysql's slow query log. However this needs to be minimally invasive.
Pointing the source to a FIFO rather than a ...
- 5,110
0
votes
0
answers
110
views
Vim show all buffers at top of view, hot keys to move to different buffers?
I have recently realized I have been using tabs instead of buffers for editing multiple files. This appears to be a common mistake for a lot of people.
I want to shift to using buffers for opening ...
- 542
0
votes
0
answers
675
views
linux drop_cache using "echo 3 > /proc/sys/vm/drop_caches" not working as expected
In our production environment we are running drop cache command echo 3 > /proc/sys/vm/drop_caches to free the RAM. But also what I found is dropping caches is not a good practice and also it won't ...
- 3
0
votes
1
answer
434
views
Is rsync really a culprit for causing low memory on our VM?
In a VM of 16GB RAM, we are running rsync as a cron job(for every 10 minutes) on our production to sync GBs of folder from AWS EFS to local storage. After few days of running we found that the VM is ...
- 3
2
votes
0
answers
133
views
How to find the memory address of the parameter(s) of a function
For context, I am attempting to exploit a buffer overflow as a practice exercise. I need to pass a parameter into a function via memory address. I have found the address for the function I would like ...
- 21
0
votes
0
answers
59
views
Why piping cat into head -c 5 for a chardev results in many more calls to the driver's read than just calling head -c 5 on the chardev?
Taking inspiration from this blog post, I'm playing around with linux device drivers (which I'm studying from ).
The read field of the file_operations associated with the driver is initialized to the ...
- 1,471
0
votes
0
answers
50
views
Is there a way to determine the optimal input/output buffer sizes for a binary that reads from and writes to stdout?
I have a binary that reads from stdin output from bzcat, performs some transforms on the data, and then writes out to stdout to be compressed via gzip and finally written to disk.
Not actual code, but ...
- 63
0
votes
3
answers
111
views
How can I create an expanding buffer?
I want to be able to scroll back through all of the output for each session in terminal.
When using cat on large files, for example, there could be hundreds, or even thousands, of lines (I write a lot ...
- 85
3
votes
1
answer
153
views
Why is the stdout line buffering for cut -c differ from cut -b?
Our RHEL 7 machines have great long Log files and I asked about buffering of cut in this question. That question remains but a bit of experimentation showed a different issue.
I decided to try using ...
- 301
1
vote
1
answer
183
views
How is the default stdout buffering set?
Running on Red Hat EL7, we have these great long lines in our log files so I do a
tail -f Log | cut -c1-$COLUMNS
This works great on some systems but other--apparently identical--systems, the pipe ...
- 301
1
vote
1
answer
64
views
Piping line-by-line
I have a script that "listens" to amixer events (volume change and mute), piping the volume and mute status to xob. It works great, but the (ab)use of stdbuf -oL <command> looks wrong ...
- 13
0
votes
1
answer
417
views
Writing to unix socket file as fast as possible
I'm trying to write with perl to a unix socket file as fast as possible but failing on full message buffer. I tried autoflush and manual flush without success.
use IO::Select;
use IO::Socket::UNIX;
...
- 11
1
vote
1
answer
1k
views
Is it safe to clear the buffers and cache on a linux server
After reading this link: How do you empty the buffers and cache on a Linux system?, I know that there are some commands that can help us to empty the buffers and cache of the OS.
But I'm not sure if ...
- 3,181
0
votes
2
answers
506
views
Why is there less latency and jitter when there is additional high bandwidth traffic between two computers with Ubuntu 22.04 installed on them?
I have two computers that are connected via an Ethernet cable with Ubuntu 22.04 installed on them. I have a client on computer A) which is sending UDP packets to a server on computer B) and I am ...
2
votes
1
answer
170
views
pv gives 100% progress with small buffer
I have to do a long task (convert image format) for every file of a folder. I achieved to use pv to write the estimation of duration with (used sleep here to simulated processing time):
pv -B 1 =(find ...
- 121
0
votes
1
answer
76
views
How to trigger two commands when pressing a key bind
So I wanted to pass the text stored in my buffer through a socket to other machine, for that I manage to get a socket done, I just need to fire up the following command when in copy-mode-vi
bind -T ...
0
votes
0
answers
296
views
Why are packets being dropped (kfree_skb)?
I am currently sending ~9mpps, each packet is 72bytes large, and is a UDP DNS request. The packets are being sent to a Dell server I have using an intel i40e interface. The server is using all of its ...
- 542
0
votes
2
answers
3k
views
Explanation about free, available and buff/cache and how to find what process is buff/cache saved for in a server in the command "free -mh "
I see explanation for free -mh as:
free displays the total amount of free and used physical and swap
memory in the system, as well as the buffers and caches used by
the kernel. The information is ...
- 1
1
vote
2
answers
500
views
why do we need to pass buffers to system calls in order to have information returned? Why can't system calls allocate the buffer internally?
I'm reading TLPI, and in section 4.4:
System calls don’t allocate memory for buffers that are used to return information to the caller. Instead, we must pass a pointer to a previously allocated ...
- 161
1
vote
1
answer
1k
views
Clear socket buffer of CAN adapter
I have a USB/CAN adapter, from lsusb:
OpenMoko, Inc. Geschwister Schneider CAN adapter
it provides a socket interface, from ip addr:
5: can0: <NOARP,UP,LOWER_UP,ECHO> mtu 16 qdisc pfifo_fast ...
- 713
0
votes
1
answer
104
views
When is it beneficial to add -u to tr?
One of the arguments to the tr command is:
-u Guarantee that any output is unbuffered.
Under what circumstances is it a good idea to not buffer the output?
e.g. should I use -u if I expect each ...
- 758
0
votes
0
answers
165
views
Are direct writes to block devices buffered? How do I ensure it's written to disk before removing it?
Say I do something like dd if=file of=/dev/sdc (simplified for illustration), is this buffered? Can I sync it like I would any other filesystem write, or do I need to do something else to make sure ...
- 375
5
votes
1
answer
508
views
Why is mawk's output (STDOUT) buffered even though it is the terminal?
I am aware that STDOUT is usually buffered by commands like mawk (but not gawk), grep, sed, and so on, unless used with the appropriate options (i.e. mawk --Winteractive, or grep --line-buffered, or ...
- 1,351
1
vote
1
answer
7k
views
Buffer I/O error on dev sda6, logical block 44, async page read
I don't know why this happens... I was just doing my work (development) and suddenly one of my drive start lagging then I just unmount it by right click on it.
After it is still lagging, so I restart ...
- 111
2
votes
1
answer
703
views
configure vim to use both tmux and system clipboard
I'm fairly new to vim (I'm using nvim) so forgive my lack of knowledge, many questions are similar to mine, but not quite the same.
I would like this:
vim by default should cut/copy to a buffer, not ...
- 71
1
vote
1
answer
807
views
How does buffering for TCP packets work?
If I write a few times in a quick succession on a socket (with the POSIX function write), usually all the data I wrote gets sent in a single TCP packets. Unless I write too much or unless I wait too ...
- 125
1
vote
0
answers
41
views
linux handling of files
I want to perform updates on a file via multiple processes in parallel. These processes all open this file for write in parallel.
Abbreviations used:
f : file,
p[i] : process i,
b[i] : buffer for FD ...
- 150
1
vote
1
answer
545
views
Pipe from xwd to ImageMagick's convert produces cropped image
I'm trying to take a screenshot of a remote host (using xwd) and convert it to png (using convert). I need it to be an unattended oneliner, so this is what I'm doing:
sshpass -p THE_PASSWORD ssh user@...
2
votes
0
answers
524
views
ping: sendmsg: No buffer space available
I am running a ping from one of the terminals:
ping www.google.com
This runs without any issues indefinitely.
Now on another terminal, I run:
for i in {1..5}; do speedtest; done
Now after sometime on ...
- 211
1
vote
1
answer
1k
views
Do sync and writing to /proc/sys/vm/drop_caches really empty all the disk cache/buffers?
When you want to empty disk caches/buffers, you usually issue the following command:
# sync && echo 3 > /proc/sys/vm/drop_caches
But after typing that command, free doesn't really report 0 ...
- 10.3k
3
votes
2
answers
451
views
How do I feed data to a pipe until it's full, no more no less?
I want to use pipes on Linux as a synchronization primitive between a master process and a slave process. The classic way is to create two pipes, but I believe there's a way to use a single fd instead....
- 41
1
vote
0
answers
165
views
Buffer Memory Drops to Zero
We have been having problems with a system that becomes inaccessible. Characteristics are 1) sudden spike in blocked and run queues and 2) buffer memory drops to zero.
The symptoms are 1) existing ...
- 11
3
votes
1
answer
684
views
broken pipe when tar extracted from stdin buffer
I am restoring files from a tar archive on an LTO-7 tape to a locally mounted network share. If I directly restore to the share it runs very slowly (90 MB/s). When I use an additional buffer I get the ...
- 207
0
votes
1
answer
148
views
Bright Color Escape Sequences not saved in Tmux capture-pane?
I'm trying to use tmux to create a pty for me that I can read the contents of, including color. I'm using tmux capture-pane -e -t paneName and tmux save-buffer tmuxbuffer.txt and I get normal colors, ...
- 53
0
votes
0
answers
664
views
Effect of Buffer Size in File I/O in Unix
I am trying to understand the inner workings of Unix based OSs. I was reading on buffered I/O, and how the buffer size affects the number of system calls made, which in turn affects the total time ...
- 101
4
votes
2
answers
428
views
Buffering stdout separately from stderr
I am running a utility, which emits the following:
Progress to its Standard Error
Data/Yield/Output to its Standard Output
I did not build the utility, nor can I easily modify it.
I wish to do the ...
2
votes
1
answer
265
views
Does it have 3 levels of buffering, or 2 levels of buffering, when using stdio library?
From The Linux Programming Interface:
(I've read the related sections. I pasted this sum-up diagram just for illustration.)
I know that for direct IO which uses read and write system calls. There is ...
- 1,127
2
votes
2
answers
512
views
File order on FAT/FAT32/VFAT file systems
I have several audio devices (car radio, portable radio, MP3 player) that take SD cards and USB sticks with a FAT file system on it. Because these devices have limited intelligence they do not sort ...
2
votes
2
answers
734
views
How to send output of `pv` to python script, and output progress one line at a time?
I use tar to quickly copy thousands of small files which I pipe to pv to show the progress and speed of the copying.
$ sudo tar c --files-from /tmp/camfilenames |
pv --width 40 |
sudo tar x -C /...
- 161
1
vote
1
answer
693
views
Disable buffering for file descriptors in bash
I may be misunderstanding what's going on here, but I feel it's got to do with pipe buffering. I have a script which uses several file descriptors (# 3 and above) for various logging levels. Depending ...
- 282
-1
votes
1
answer
174
views
Which hardware is used by buffer
I know that the cache is used to store data so that future requests for that data can be served faster. Normally we have three level cache: L1, L2 and L3 in the current computer architecture. We can ...
- 3,181
0
votes
1
answer
807
views
Linux FIFO buffering depends on invocation order of readers and writers
I'm trying to write a simple python script that reads from a fifo and then writes to another fifo.
I have created two FIFOs using the following command:
$ mkfifo input
$ mkfifo output
I invoke the ...
- 150
9
votes
1
answer
3k
views
Do I need to take action regarding my Microarchitectural Data Sampling (MDS) status?
My dmesg output contains the following line:
[ 0.265021] MDS CPU bug present and SMT on, data leak possible. See https://www.kernel.org/doc/html/latest/admin-guide/hw-vuln/mds.html for more ...
- 313