Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [io]

I/O is an acronym for input/output, which refers to the mechanism or process for the transfer of information between one system or component and another.

1
vote
0answers
13 views

Run Command When IO Load Average is Low?

I have a bash script which calls an external program I cannot modify. The first stage of this program uses high IO and the second stage uses almost no IO, instead it uses a lot of CPU, it does not ...
2
votes
0answers
30 views

A linux “cp” command equivalent that does read/write threading and buffer sharing for increased throughput?

First, let me address your concerns. I'm copying from one slow drive, to another (different) slow drive. Blocking for reads on one drive until I write to another leaves a lot of room for improvement. ...
3
votes
1answer
23 views

How to execute local script by passing arguments on remote via ssh as a different user?

I am trying to execute a local script on remote server via ssh using keytab authentication. I am able to connect to remote via ssh as sachin(user who gets the ticket) and then I am spawning a bash ...
23
votes
1answer
3k views

Why don't reads from /dev/zero count as I/O?

I am emptying out a hard drive on some Linux 4.x OS using this command: sudo sh -c 'pv -pterb /dev/zero > /dev/sda' And I opened another tty and started sudo htop and noticed this: PID USER ...
0
votes
0answers
5 views

CFQ - “In case of asynchronous requests, all the requests from all the processes are batched together according to their process's I/O priority”

I think the following statements about CFQ contradict each other: "In case of asynchronous requests, all the requests from all the processes are batched together according to their process's I/O ...
0
votes
1answer
28 views

Why am I seeing a ton of SyS_poll?

I have two different Linux desktops connected to a Synology NAS like so: 10.0.0.20:/volume1/chronicle /chronicle nfs rw,hard,intr 0 0 and I am frequently seeing my computer locking up ...
1
vote
0answers
30 views

Weird I/O stalls affecting a whole desktop

After a recent hardware migration I started experiencing weird I/O stalls affecting my desktop Debian Stretch system. Typical symptoms, all happening during each stall: I stop being able to interact ...
1
vote
1answer
47 views

Splitting file by 1st column: Too many open files

This is an extension of the question: Splitting a file into multiple files based on 1st column value. Since I just joined, I do not have enough rep points to add a comment to the above question. ...
0
votes
1answer
21 views

How should we interpret the phrase “asynchronous requests” in cfq-iosched.txt?

The old cfq-iosched.txt mentions "asynchronous" or "async" requests in several places. CFQ maintains the per process queue for the processes which request I/O operation(synchronous requests). In ...
-3
votes
1answer
112 views

What is the equivalent (programmatically) of cat /dev/something > file.txt [closed]

I'm copying a video from a camera stream using a command cat /dev/video0 > file.mpeg This works perfectly, however I want too break up the file into smaller chunks... So I've been waiting pressing ...
2
votes
0answers
15 views

Find the size of all relevant caches when doing I/O

So, I'm looking at the stats from iostat, which updates every few seconds, and I see that I/O to device happens in bulks, say, around 40M/s, once every half a minute or so. I'm debugging an ...
-1
votes
1answer
58 views

tar with ssh: what happens if creation speed is higher than network speed?

I'm concerned what will happen if speed of source read + stream create operations gets higher than underlying network protocol speed? Is tar going to create and cache data locally before it is ...
-1
votes
3answers
44 views

Oneliner for outputting to file only if string is matched (bash)?

I am making a script where I curl a URL and output to a file like this: curl http://example.com/$1 > $1 Is there any way to make it dismiss anything that doesn't include, say <head>? I ...
2
votes
0answers
62 views

ext4 jbd2 and I/O priority

I want to delete a huge directory with over 6 million files across over 4000 directories, totaling about 26GB of data. As I knew this was going to take a while I used ionice to set the I/O priority ...
1
vote
0answers
25 views

Performance issues on internal SATA device causing high load

I have a problem with the performance of my laptop. In fact I needed to install quite some things recently and the whole machine froze from time to time completely. First some general information: I ...
1
vote
0answers
61 views

How to exclude a directory or partition on pmlogger scan?

As on the details below it look like that the cause of the problem is the pmlogger is spawning the find command. And its takes a long time for the find to scan all the files in one of my partition. ...
4
votes
0answers
52 views

Limit tar read speed

I'm using tar do make backups of a machine. But, it is using a lot of I/O and slows down the whole machine. So, is there a way to limit the read speed of tar? I know about pv, but it limit only the ...
0
votes
0answers
15 views

ionice vs nice for mysqlimport

I am trying to import a huge .sql file into a mysql server. What (if any) is a best option to speed up the process (all other parameters being equal) reducing the process niceness to -20? setting ...
0
votes
0answers
29 views

Cache behavior in dd command

I am performing some dd outputs and running in parallel vmstat: With no direct write (i.e. via cache) $ dd if=/dev/urandom of=somefile.txt bs=1M count=200 200+0 records in 200+0 records out ...
1
vote
1answer
89 views

USB hard drive I/O slow speeds

I have several Mediasonic ProBox HF2-SU3S2 with 4 drives - all with the same symptoms - extremely slow I/O (at best, about 10MB/s write). Each unit is configured with a software RAID 5 using mdadm ...
1
vote
1answer
50 views

Detecting misbehaving USB flash memory devices, that block the rest's IO

I'm using f3 and a custom Bash script to test USB flash memory in big amounts. A common problem I encounter is that some faulty drives would starve all the healthy ones for IO, effectively stalling ...
0
votes
1answer
27 views

bfq_async_charge_factor says async writes are given 10x less than “fair” share of device throughput. How can I observe this?

I've been reading about CONFIG_WBT, and also BFQ. I tried to compare WBT v.s. CFQ in my hard drive. I learned CFQ tries to control Linux' massive async writeback, but its success is limited because ...
2
votes
2answers
338 views

What does Fedora Workstation 29 use as the default I/O scheduler?

If it depends on the exact type of block device, then what is the default I/O scheduler for each type of device? Background information Fedora 29 includes a Linux kernel from the 4.19 series. (...
0
votes
1answer
63 views

Does linux systems flush dirty pages to disk one block by one block?

As far as I know, in linux systems, when modifying the content of a file, the pages containing the content of the file in page cache will be marked dirty, and will be flushed to disk eventually. What ...
1
vote
0answers
95 views

Determine the specific benefit of Writeback Throttling (CONFIG_WBT)

[PATCH 0/8] Throttled background buffered writeback v7 Since the dawn of time, our background buffered writeback has sucked. When we do background buffered writeback, it should have little ...
2
votes
2answers
67 views

Bash: turn stderr back on after suppressing

I've made this script which automates installing/uninstalling git. In my function for testing if git is installed, I use the git --version command and test the return code. I dislike the stderr ...
1
vote
1answer
160 views

FIO reporting slow write speeds while DD reports fast ones

According to most disk benchmarks, sequential write speeds are typically very close to the read speeds. Mine are in the ~500MB/s range. $ dd if=/dev/zero of=tempfile bs=1M count=5120 conv=notrunc ...
1
vote
0answers
57 views

atop shows queue depth is higher than maximum

In my atop logs, I see avq ("average queue depth") over 600. This is on an LVM Logical Volume (LV). Both the LV and the physical volume have nr_requests (maximum queue depth) of 128. (Note, to see ...
1
vote
0answers
28 views

Cache Memory Leak (or looks like one) slows down a program

I have a strange problem that looks like a Memory Cache Leak (not a memory leak) leading to a slow down of a program. Let me set the stage first. Reproducibly (using ganglia to monitor), on a cluster ...
0
votes
1answer
45 views

Disk I/O Information

I want to get the read and write, service time, queue length and wait time of my disks. the OS is CentOS 6. I use iostat. when I run this command: iostat -x -d /dev/sda the output is: ...
1
vote
2answers
865 views

Is memory mapped I/O only used internally by OS, not exposed to and used by programmers on top of Linux?

In Operating System Concepts, memory mapped files and memory mapped I/O are two different things. See below about memory mapped I/O. To use memory mapped files, we have mmap(). To use memory ...
0
votes
1answer
60 views

is `write()` asynchronous write?

From Operating System Concepts Another issue that can affect the performance of I/O is whether writes to the file system occur synchronously or asynchronously. Synchronous writes occur in ...
3
votes
2answers
109 views

Disk Performance Monitoring [closed]

I have a physical server with 2 disks, sda and sdb. I want to monitor their I/O and performance. The monitoring element has 3 types: ops, sps, and bps. What are these and which one is better to ...
1
vote
0answers
13 views

Is there any way to check the most requested(in terms of IO) areas of the file-system inside a docker container?

I've recently read this interesting suggestion under "Performance best practices" subtitle here: https://docs.docker.com/storage/storagedriver/aufs-driver/ Use volumes for write-heavy workloads: ...
3
votes
2answers
598 views

How does memory mapping a file have significant performance increases over the standard I/O system calls?

Operating System Concepts says Consider a sequential read of a file on disk using the standard system calls open(), read(), and write(). Each file access requires a system call and disk access. ...
-1
votes
1answer
46 views

What are two caveats of line buffering in Unix?

A paragraph from APUE Chapter 5.4 about line buffer: Line buffered. In this case, the standard I/O library performs I/O when a newline character is encountered on input or output. This allows us ...
5
votes
1answer
94 views

System lags when doing large R/W operations on external disks

I am having some issues with system-wide latency/lagging when doing large disk imaging operations on an Ubuntu 18.04 system. Here's the system specs: Processor: Intel Core i7 (never near capacity on ...
1
vote
2answers
69 views

Does raw I/O bypass some or most services provided by the driver?

Operating System Concepts by Abraham Silberschatz 9th 2012 says Raw I/O bypasses all the filesystem services, such as file I/O demand paging, file locking, prefetching, space allocation, file names, ...
0
votes
1answer
41 views

How is a sequentially accessible device made randomly accessible?

If a device offers only sequential access not random access, can a device driver or a file system make the device look like randomly accessible to users? If yes, is this done by providing buffer/...
0
votes
1answer
32 views

Is a hard drive under direct I/O still randomly accessible?

A hard drive is randomly accessible. On Linux, when performing direct I/O i.e. raw I/O on a block device file (such as for hard drive) or regular file by passing O_DIRECT flag to open(), is it only ...
3
votes
1answer
207 views

Is a device being a block or character device determined purely by hardware?

From https://unix.stackexchange.com/a/472920/674 A "character device" and "block device" are abstractions, usually used in Unix-style systems in classifying various devices. Is a device being a ...
0
votes
1answer
80 views

Do both synchronous and asynchronous data transfer imply sequential access, not random access?

http://www.engr.iupui.edu/~skoskie/ECE362/lecture_notes/LNB25_html/text12.html says There are many serial data transfer protocols. The protocols for serial data transfer can be grouped into two ...
0
votes
0answers
37 views

Does the spool buffer work like a buffer in memory for the spool file on disk?

Operating System Concepts discusses spooling in the following. "A spool is a buffer". Is the spool buffer in memory? "Each application’s output is spooled to a separate disk file." Are spool files ...
-4
votes
1answer
52 views

Are there implications between block/character device and random/sequential access? [closed]

Operating System Concepts says: Devices vary on many dimensions, as illustrated in Figure 13.7. • Character-stream or block. A character-stream device transfers bytes one by one, whereas a ...
0
votes
2answers
42 views

Are interrupts used to signify that output is ready or input has completed?

Operating System Concepts says: During I/O, the various device controllers raise interrupts when they are ready for service. These interrupts signify that output has completed, or ...
3
votes
2answers
52 views

How does `cat`'s I/O model differ from that of other utilities?

From the output of stdbuf --help (GNU): Usage: stdbuf OPTION... COMMAND Run COMMAND, with modified buffering operations for its standard streams. ... NOTE: If COMMAND adjusts the buffering of its ...
1
vote
1answer
119 views

How to specify multiple schedulers on the kernel boot command line?

We have systems with both spinning mechanical disks, and NVME storage. We want to reduce the CPU overhead for IO by taking any IO scheduler out of the way. We want to specify this on the Linux boot ...
2
votes
0answers
51 views

non-root access to physical blocks occupied by a file

ioctl() and FIBMAP can be used to determine the physical blocks occupied by a file, given logical blocks obtainable via stat(); however, root access is required. Why is root access necessary for ...
0
votes
1answer
10 views

How the IOs are throttled from IO scheduler to scsi low level drivers

IO scheduler has a queue depth of 128 by default. And block device has a queue depth of 32 by default. Where does the IOs are buffered. Does IO scheduler send all IOs to scsi low level drivers and ...
0
votes
1answer
26 views

Understanding workload sharing of readwrite in fio

I am exploring fio tools with different options. I want to share the read and write workload, so supplied below jobfile to fio. [read-4K-write-64K] blocksize=4K,64K, rw=readwrite rwmixread=50 size=...