Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange
The results are in! See what nearly 90,000 developers picked as their most loved, dreaded, and desired coding languages and more in the 2019 Developer Survey.

Questions tagged [io]

I/O is an acronym for input/output, which refers to the mechanism or process for the transfer of information between one system or component and another.

0
votes
0answers
19 views

Who is causing extremely high CPU usage when writing to SSD?

When writing to SSD (regardless if over file system or directly into /dev/sda) I observe extremely high CPU usage, with top showing CPU: 0% usr 53% sys 0% nic 9% idle 27% io 0% irq 9% ...
0
votes
0answers
29 views

How to create a new character device under /proc/device file?

I am trying to create a node with mknod /tmp/ttyACM0 c 166 0 command but then No such device or address error returns when I try to open serial port with ttyACM0 because there is no device in proc/...
0
votes
0answers
32 views

How shell delivers user's input to program and shows program's output? [duplicate]

Since shell (for example, Bash) is just another program with its own stdin/stdout/stderr streams, how does it deliver user's input to running program? Here's simple program, that reads value from ...
1
vote
1answer
22 views

Enabling Asynchronous IO at HP-UX Operating System

We are using HP-UX Itanium version B.11.31 and this server is used for running Sybase database version ASE15.7. I would like to know how to enable Asynchronous IO at Operating System level. I went ...
0
votes
2answers
40 views

Can one processes have a descriptor that is non-blocking while another process have a descriptor referencing the same file/socket that is blocking?

I am more interested in sockets than regular files, but basically I want to know whether one process can "see" a socket as blocking where another process can see it as non-blocking. I am guessing yes,...
1
vote
0answers
15 views

Using Systemtap probing Getting the virtual address location of bytes for every read write operation in Linux

Using Systemtap probing tool I got the way to get total amount of bytes read by a file by probing syscall.read and syscall.read.return function. syscall.read.return has returnval() which gives the ...
1
vote
0answers
16 views

Record filesystem activity

I would like to record all filesystem activity so that I can later replay it at will (for benchmarking purposes). What's the best way of doing that? I considered strace, but rejected that because it ...
1
vote
0answers
68 views

ls command is too slow for some directories

When I run ls command for some directories, it may take about 1.5 seconds to complete even if directory has just a few files, but for most of directories it's very fast. It seems that it's slow for ...
0
votes
0answers
10 views

Readonly flag after error: why does Kernel detect an error?

I'm debugging a C++ application running on Linux Embedded: uname -r: 4.14.40 The application is downloading a file on an external flash SD card: mount: sd-path type vfat (rw,relatime,fmask=0022,...
0
votes
0answers
21 views

USB disk disconnects when exchange quite high data

I have an embedded PC (based on Atom Z3735F with a 32 GB eMMC for o.s.) that I use as NAS with OpenMediaVault 4.1.2-1. On an USB port I connected a 1 TB Hard-disk for storage. Whenever I do something ...
3
votes
2answers
69 views

Reading and writing data temporarily without using the filesystem

I'm using ansible in a vagrant environment managing a debian box. Due multiple ansible provisioners with vault encrypted data (e. g. database root password) I have to enter the vault password once by ...
1
vote
0answers
43 views

Why can't I interrupt IO operations on linux?

I tried to abort mkfs.ntfs applied on USB drive (forgot the Quck format flag) and then another big disk IO operation (using dd). They both took more than several minutes to abort, so I just rebooted, ...
0
votes
0answers
48 views

How does 200 million function call of fgets translate into only 25,224 system calls?

A I/O efficiency test from APUE: Test file is said to be "98.5 MB with 3 million lines." Code used in "Figure 3.6": #include "apue.h" #define BUFFSIZE 4096 int main(void) { int n; ...
0
votes
0answers
28 views

Is there a file that can be used to monitor `/dev/sda`s only IO activity in Linux?

I am familiar with iotop and those tools. But for some strange reason I have to write a Ruby program on my own to monitor only read and write activity of all the partitions on /dev/sda). I want to ...
2
votes
0answers
28 views

Run Command When IO Load Average is Low?

I have a bash script which calls an external program I cannot modify. The first stage of this program uses high IO and the second stage uses almost no IO, instead it uses a lot of CPU, it does not ...
2
votes
0answers
42 views

A linux “cp” command equivalent that does read/write threading and buffer sharing for increased throughput?

First, let me address your concerns. I'm copying from one slow drive, to another (different) slow drive. Blocking for reads on one drive until I write to another leaves a lot of room for improvement. ...
3
votes
1answer
83 views

How to execute local script by passing arguments on remote via ssh as a different user?

I am trying to execute a local script on remote server via ssh using keytab authentication. I am able to connect to remote via ssh as sachin(user who gets the ticket) and then I am spawning a bash ...
23
votes
1answer
3k views

Why don't reads from /dev/zero count as IO_RBYTES?

I am emptying out a hard drive on some Linux 4.x OS using this command: sudo sh -c 'pv -pterb /dev/zero > /dev/sda' And I opened another tty and started sudo htop and noticed this: PID USER ...
0
votes
1answer
15 views

CFQ - “In case of asynchronous requests, all the requests from all the processes are batched together according to their process's I/O priority”

I think the following statements about CFQ contradict each other: "In case of asynchronous requests, all the requests from all the processes are batched together according to their process's I/O ...
0
votes
1answer
34 views

Why am I seeing a ton of SyS_poll?

I have two different Linux desktops connected to a Synology NAS like so: 10.0.0.20:/volume1/chronicle /chronicle nfs rw,hard,intr 0 0 and I am frequently seeing my computer locking up ...
1
vote
0answers
44 views

Weird I/O stalls affecting a whole desktop

After a recent hardware migration I started experiencing weird I/O stalls affecting my desktop Debian Stretch system. Typical symptoms, all happening during each stall: I stop being able to interact ...
1
vote
1answer
58 views

Splitting file by 1st column: Too many open files

This is an extension of the question: Splitting a file into multiple files based on 1st column value. Since I just joined, I do not have enough rep points to add a comment to the above question. ...
0
votes
1answer
30 views

How should we interpret the phrase “asynchronous requests” in cfq-iosched.txt?

The old cfq-iosched.txt mentions "asynchronous" or "async" requests in several places. CFQ maintains the per process queue for the processes which request I/O operation(synchronous requests). In ...
-3
votes
1answer
123 views

What is the equivalent (programmatically) of cat /dev/something > file.txt [closed]

I'm copying a video from a camera stream using a command cat /dev/video0 > file.mpeg This works perfectly, however I want too break up the file into smaller chunks... So I've been waiting pressing ...
2
votes
0answers
15 views

Find the size of all relevant caches when doing I/O

So, I'm looking at the stats from iostat, which updates every few seconds, and I see that I/O to device happens in bulks, say, around 40M/s, once every half a minute or so. I'm debugging an ...
-1
votes
1answer
61 views

tar with ssh: what happens if creation speed is higher than network speed?

I'm concerned what will happen if speed of source read + stream create operations gets higher than underlying network protocol speed? Is tar going to create and cache data locally before it is ...
-1
votes
3answers
46 views

Oneliner for outputting to file only if string is matched (bash)?

I am making a script where I curl a URL and output to a file like this: curl http://example.com/$1 > $1 Is there any way to make it dismiss anything that doesn't include, say <head>? I ...
2
votes
0answers
165 views

ext4 jbd2 and I/O priority

I want to delete a huge directory with over 6 million files across over 4000 directories, totaling about 26GB of data. As I knew this was going to take a while I used ionice to set the I/O priority ...
1
vote
0answers
29 views

Performance issues on internal SATA device causing high load

I have a problem with the performance of my laptop. In fact I needed to install quite some things recently and the whole machine froze from time to time completely. First some general information: I ...
1
vote
0answers
80 views

How to exclude a directory or partition on pmlogger scan?

As on the details below it look like that the cause of the problem is the pmlogger is spawning the find command. And its takes a long time for the find to scan all the files in one of my partition. ...
5
votes
0answers
67 views

Limit tar read speed

I'm using tar do make backups of a machine. But, it is using a lot of I/O and slows down the whole machine. So, is there a way to limit the read speed of tar? I know about pv, but it limit only the ...
0
votes
0answers
24 views

ionice vs nice for mysqlimport

I am trying to import a huge .sql file into a mysql server. What (if any) is a best option to speed up the process (all other parameters being equal) reducing the process niceness to -20? setting ...
0
votes
0answers
39 views

Cache behavior in dd command

I am performing some dd outputs and running in parallel vmstat: With no direct write (i.e. via cache) $ dd if=/dev/urandom of=somefile.txt bs=1M count=200 200+0 records in 200+0 records out ...
1
vote
2answers
108 views

USB hard drive I/O slow speeds

I have several Mediasonic ProBox HF2-SU3S2 with 4 drives - all with the same symptoms - extremely slow I/O (at best, about 10MB/s write). Each unit is configured with a software RAID 5 using mdadm ...
1
vote
1answer
54 views

Detecting misbehaving USB flash memory devices, that block the rest's IO

I'm using f3 and a custom Bash script to test USB flash memory in big amounts. A common problem I encounter is that some faulty drives would starve all the healthy ones for IO, effectively stalling ...
0
votes
1answer
28 views

bfq_async_charge_factor says async writes are given 10x less than “fair” share of device throughput. How can I observe this?

I've been reading about CONFIG_WBT, and also BFQ. I tried to compare WBT v.s. CFQ in my hard drive. I learned CFQ tries to control Linux' massive async writeback, but its success is limited because ...
2
votes
2answers
500 views

What does Fedora Workstation 29 use as the default I/O scheduler?

If it depends on the exact type of block device, then what is the default I/O scheduler for each type of device? Background information Fedora 29 includes a Linux kernel from the 4.19 series. (...
0
votes
1answer
122 views

Does linux systems flush dirty pages to disk one block by one block?

As far as I know, in linux systems, when modifying the content of a file, the pages containing the content of the file in page cache will be marked dirty, and will be flushed to disk eventually. What ...
1
vote
0answers
116 views

Determine the specific benefit of Writeback Throttling (CONFIG_WBT)

[PATCH 0/8] Throttled background buffered writeback v7 Since the dawn of time, our background buffered writeback has sucked. When we do background buffered writeback, it should have little ...
2
votes
2answers
75 views

Bash: turn stderr back on after suppressing

I've made this script which automates installing/uninstalling git. In my function for testing if git is installed, I use the git --version command and test the return code. I dislike the stderr ...
1
vote
1answer
319 views

FIO reporting slow write speeds while DD reports fast ones

According to most disk benchmarks, sequential write speeds are typically very close to the read speeds. Mine are in the ~500MB/s range. $ dd if=/dev/zero of=tempfile bs=1M count=5120 conv=notrunc ...
1
vote
0answers
80 views

atop shows queue depth is higher than maximum

In my atop logs, I see avq ("average queue depth") over 600. This is on an LVM Logical Volume (LV). Both the LV and the physical volume have nr_requests (maximum queue depth) of 128. (Note, to see ...
1
vote
0answers
32 views

Cache Memory Leak (or looks like one) slows down a program

I have a strange problem that looks like a Memory Cache Leak (not a memory leak) leading to a slow down of a program. Let me set the stage first. Reproducibly (using ganglia to monitor), on a cluster ...
0
votes
1answer
50 views

Disk I/O Information

I want to get the read and write, service time, queue length and wait time of my disks. the OS is CentOS 6. I use iostat. when I run this command: iostat -x -d /dev/sda the output is: ...
1
vote
2answers
1k views

Is memory mapped I/O only used internally by OS, not exposed to and used by programmers on top of Linux?

In Operating System Concepts, memory mapped files and memory mapped I/O are two different things. See below about memory mapped I/O. To use memory mapped files, we have mmap(). To use memory ...
0
votes
1answer
90 views

is `write()` asynchronous write?

From Operating System Concepts Another issue that can affect the performance of I/O is whether writes to the file system occur synchronously or asynchronously. Synchronous writes occur in ...
3
votes
2answers
266 views

Disk Performance Monitoring [closed]

I have a physical server with 2 disks, sda and sdb. I want to monitor their I/O and performance. The monitoring element has 3 types: ops, sps, and bps. What are these and which one is better to ...
1
vote
0answers
14 views

Is there any way to check the most requested(in terms of IO) areas of the file-system inside a docker container?

I've recently read this interesting suggestion under "Performance best practices" subtitle here: https://docs.docker.com/storage/storagedriver/aufs-driver/ Use volumes for write-heavy workloads: ...
3
votes
2answers
1k views

How does memory mapping a file have significant performance increases over the standard I/O system calls?

Operating System Concepts says Consider a sequential read of a file on disk using the standard system calls open(), read(), and write(). Each file access requires a system call and disk access. ...
1
vote
2answers
98 views

What are two caveats of line buffering in Unix?

A paragraph from APUE Chapter 5.4 about line buffer: Line buffered. In this case, the standard I/O library performs I/O when a newline character is encountered on input or output. This allows us ...