Questions tagged [buffer]

This tag is for questions about kernel buffer caches, including pipe buffers. These are used to store the recently accessed files and/or frequently accessed files.

Filter by
Sorted by
Tagged with
0
votes
0answers
13 views

What does 'Not Supported' mean in the msecli reports?

A Micron MTFDHAL800MCE SSD is installed on my computer system and the OS is Linux Mint18. Micron has provided msecli for the management and configuration of the SSDs using specific commands on the ...
94
votes
13answers
45k views

How can I do a 'change word' in Vim using the current paste buffer?

I have some text in my paste buffer, e.g. I did a yw (yank word) and now I have 'foo' in my buffer. I now go to the word 'bar', and I want to replace it with my paste buffer. To replace the text ...
409
votes
13answers
215k views

Turn off buffering in pipe

I have a script which calls two commands: long_running_command | print_progress The long_running_command prints progress but I'm unhappy with it. I'm using print_progress to make it nicer (namely, I ...
3
votes
3answers
4k views

piped command after grep not working

I want to monitor a file. So I have tailed a file using below command, to execute a script per line. tail -3f logfile.log | grep "action.*add" | sed -u -e "s/^/'/" -e "s/$/'/" | xargs -L 1 -P 5 bash ...
0
votes
0answers
19 views

mount cifs async cause temporary freeze of the directory tree

I have an issue with high buffering and high workload while copying large files to my samba mount. I have NAS with samba share and some Linux and Windows machine. My Windows machine works well - avg ...
-2
votes
1answer
257 views

Why does readdir only read 32K of directory entries at a time?

readdir() only reads 32K of directory entries at a time. Why does it read only 32K entries in one shot? Is it dependent on buffer or any other parameters? Can I change that value so that I can read ...
2
votes
2answers
69 views

How to send output of `pv` to python script, and output progress one line at a time?

I use tar to quickly copy thousands of small files which I pipe to pv to show the progress and speed of the copying. $ sudo tar c --files-from /tmp/camfilenames | pv --width 40 | sudo tar x -C /...
0
votes
1answer
56 views

Disable buffering for file descriptors in bash

I may be misunderstanding what's going on here, but I feel it's got to do with pipe buffering. I have a script which uses several file descriptors (# 3 and above) for various logging levels. Depending ...
-1
votes
1answer
52 views

Which hardware is used by buffer

I know that the cache is used to store data so that future requests for that data can be served faster. Normally we have three level cache: L1, L2 and L3 in the current computer architecture. We can ...
0
votes
1answer
84 views

Linux FIFO buffering depends on invocation order of readers and writers

I'm trying to write a simple python script that reads from a fifo and then writes to another fifo. I have created two FIFOs using the following command: $ mkfifo input $ mkfifo output I invoke the ...
148
votes
6answers
148k views

How big is the pipe buffer?

As a comment in I'm confused as to why "| true" in a makefile has the same effect as "|| true" user cjm wrote: Another reason to avoid | true is that if the command produced ...
2
votes
1answer
602 views

Do I need to take action regarding my Microarchitectural Data Sampling (MDS) status?

My dmesg output contains the following line: [ 0.265021] MDS CPU bug present and SMT on, data leak possible. See https://www.kernel.org/doc/html/latest/admin-guide/hw-vuln/mds.html for more ...
1
vote
1answer
47 views

avoid bash buffering problem in log processing

I have bash buffering a problem similar to what can be found here: Turn off buffering in pipe The socat solution in the above question is quite interesting as I have access to this command in my ...
14
votes
3answers
1k views

What happens when a file that is 100% paged in to the page cache gets modified by another process

I know that when a page cache page is modified, it is marked dirty and requires a writeback, but what happens when: Scenario: The file /apps/EXE, which is an executable file, is paged in to the page ...
0
votes
0answers
55 views

background process with “&” and problem with buffering

I've written this script in the ~/.bashrc file and I've sourced the file, but it's not working properly. It's purpose is to open the link provided by the jupyter notebook in the Chrome browser in a ...
1
vote
0answers
21 views

Why doesn't sync guarantee that buffers are flushed?

As far as I can see, the only use of sync is to ensure that there's no incomplete information, which is supposed to be safely written on disk but may actually be still only in a volatile buffer. ...
0
votes
1answer
284 views

Disable buff/cache linux for fixing ubuntu hangs on full ram

I have 8 gb of ram, ssd and very greedy for ram Android studio/gradle. and sometimes when gradle builds project(eats ram) it hangs all ubuntu(and xubuntu). ui becomes so laggish - mouse cursor moves 1 ...
0
votes
0answers
35 views

Displaying a changing buffer in realtime in linux

Sometimes when I am searching for a song on my machine, I use the following: locate *artist*.mp3 | sort | less The thing about this is that the sort will buffer until all files have been located, ...
2
votes
2answers
295 views

Flush the pipe/printf buffers externally for already running process with known PID

I am writing a data logging app, all programs are started like: ./program > out.bin The data collector periodically pools the stdout output files and reads the data. The issue is that the IO ...
1
vote
0answers
59 views

Unbuffer converting all characters to bell?

When I pipe a large-ish output through unbuffer, all I end up getting is an endless stream of the bell character (u0007). See here, a 1631-character string of arbitrary content piped through unbuffer ...
0
votes
0answers
63 views

PuTTY (or other terminal) with output buffering for slow connection

I'm looking for terminal that can buffer output (keystrokes) before sending them. I need this, because I have some devices on very slow and not reliable connections (ping 400-600ms, 20% packet loss) ...
2
votes
1answer
421 views

Block device cache v.s. a filesystem

Block devices provide buffering. This means that write() on a block device can return success, before the kernel has written the data to the device. A program can wait for all the buffered writes by ...
4
votes
3answers
671 views

What to use instead of `buffer` for 1GB buffer? [duplicate]

buffer(1) seems to be old-ish and have hard-coded values preventing it to cache large amount of data. $ buffer -m 1G max_shmem 1 too low // it doesn't even understand gigabytes $ buffer -m 1000M ...
1
vote
1answer
842 views

What controls the buffering of stdout stderr?

What part of the system sets up the buffering of the three standard streams when a program is started? Is this part of linux, or glibc, or maybe bash? Does POSIX define the behaviour, or is it part ...
0
votes
0answers
21 views

Linux RedHat write buffer config

I have Linux RedHat 6.2. How do I configure files to buffer 4096 (or higher 2^N values) bytes before writes? Of course, when an EOF is received, write the file.
1
vote
1answer
555 views

Avoiding Buffering With Output Redirection

I'm trying to write some functions I can utilize in my scripts to essentially tag all output as Error vs Standard output, throw date/time stamps on it, and also include the name of the function that ...
0
votes
0answers
582 views

UDP send buffer error in netstat

I we are running voice application which stream UDP (RTP) packets for voice, I am investigation some strange issue where my ifconfig ens4 | grep error counter going up specifically overrun If you see ...
3
votes
1answer
198 views

Using multiple copy buffers in Bash

Ctrlu cuts text from current location to the beginning of line. Now, I can paste this with Ctrly whenever I want until the next time I hit Ctrlu to copy another line in the buffer. Now the problem I ...
4
votes
2answers
861 views

Does “mount -o remount,ro” flush filesystem buffers?

On Linux, does mount -o remount,ro flush filesystem buffers/cache, or should I also run sync to achieve that?
1
vote
2answers
190 views

What are two caveats of line buffering in Unix?

A paragraph from APUE Chapter 5.4 about line buffer: Line buffered. In this case, the standard I/O library performs I/O when a newline character is encountered on input or output. This allows us ...
0
votes
0answers
817 views

flush tty device buffer from terminal

I have a serial device on /dev/ttyAMA0. I have program written in C that communicates with this device. But apart from the C program I sometimes need to input some configuration data from terminal, ...
1
vote
2answers
288 views

why /proc/net/udp is showing wrong rx_queue size in CentOS server?

I have a java server application. while(true) { serverSocket.receive(receivePacket); process(receivePacket); serverSocket.send(sendPacket); try { ...
0
votes
0answers
41 views

Sizing of Samba Linux Server

Do you know any performance of sizing recommendations about Samba 4.7.1 ? For few informations : 2vCPUs / 1 Core 8GB of RAM 140 simultaneous Windows users / 250 at most / 20 NFS volumes mounted via ...
4
votes
0answers
99 views

Tailing file using less +F/shift-F, turn off line buffering?

When using tail -f, it prints each character as it is written to the file, but less +F won't print until it sees a linefeed. And if the data being written to the file does not contain linefeeds, must ...
2
votes
0answers
82 views

fq_codel not working on saturated link?

I'm setting up a test to saturate a link between two machines with a gig router in between them, then run fq_codel on Debian Buster and see if my ping times can improve (what I think fq_codel is ...
6
votes
3answers
3k views

Are there any ways or tools to dump the memory cache and buffer?

I want to check out what files are loaded in buffer. Just so it's clear, the buffers & cache I'm referring to are what show up when you run the free -m command: $ free -m total ...
11
votes
1answer
971 views

“Leaky” pipes in linux

Let's assume you have a pipeline like the following: $ a | b If b stops processing stdin, after a while the pipe fills up, and writes, from a to its stdout, will block (until either b starts ...
2
votes
1answer
710 views

Why do terminal emulators still have screen flicker?

Why are there still visual artifacts when terminal emulators draw text-based applications? This on recent computers that render 3D games and GUI windows including anti-aliased vector fonts with no ...
3
votes
1answer
101 views

Page cache behavior guarantees and log files

Are there any guarantees as to when and in what order dirty pages are written out to disk? In the specific case of log structured files (files that are only ever appended to), is it possible for dirty ...
0
votes
0answers
55 views

How to get the network buffer usage of some process

I have an Ubuntu server, which contains some user-processes. It always crashes after starting-up several hours. "crashes" means that I cannot neither ssh to it nor ping it anymore. I'm doubting that ...
1
vote
1answer
174 views

How to pipe to less unbuffered?

I'm trying to display progress, and make it usable under less: while true; do echo -n .; sleep 1; done | less Is there a way to make it work?
0
votes
1answer
34 views

Is buffer used only for writing or can it be used for both reading and writing?

I am reading about caching and buffering from Operating System Concepts, and here is a copy of it. Is it correct that cache is used only for reading not for writing? Is buffer used only for writing ...
2
votes
1answer
3k views

buffer (not cache) usage growing, find the process using it

I've a system where the buffer usage is growing over the time. I trace it while monitoring the memory usage with dstat ex: # dstat -t -m ----system---- ------memory-usage----- time | used ...
1
vote
1answer
314 views

free -m: free size reduced because of cache/buff

I'm using the function rosbag of ROS. rosbag can record lots of data from other ROS nodes. In a word, it will generate a huge file, such as a file of 200MB. After generating such a file, I found that ...
1
vote
1answer
104 views

Copy terminal scrollback buffer preserving ANSI escape sequences, but without pre-starting script/screen

I have a live running mate-terminal whose scrollback buffer already contains a lot of text with color/bolding, but (unfortunately) no screen management/capture program (like script/screen/tmux) was ...
1
vote
1answer
147 views

How does the kernel decides which buffer to use?

When the kernel accesses a disk block, it searches for a buffer with the appropriate device-block number combination. This line is from the book "The design of the unix operating system". So my ...
0
votes
1answer
36 views

How do I limit the maximum RAM for Linux VPS?

I rent a shared VPS that runs Debian 9. I am alloted 4GB RAM (4096MB), but sometimes my processes go over 4GB. This is largely due to the kernel caching a lot of my stuff, so I'd like to know if I ...
1
vote
0answers
101 views

hard disk high temperature and frequent access while idle

I have bought two WD Gold 4TB and installed debian sid with kde on a single RAID1 partition, md0: # cat /proc/mdstat Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [...
5
votes
2answers
33k views

How to reduce buffers\cache

A monitoring system keeps alerting that my machine is reaching\breaking through it's RAM utilization threshold which is 15 GBs. I've done some reading and understood that the apparent RAM utilization ...
9
votes
4answers
7k views

Completely buffer command output before piping to another command?

Is there a way to only execute a command after another is done without a temp file? I have one longer running command and another command that formats the output and sends it to a HTTP server using ...