Questions tagged [io]

I/O is an acronym for input/output, which refers to the mechanism or process for the transfer of information between one system or component and another.

1
vote
1answer
55 views

Could AIO fsync improve dpkg performance?

Could dpkg, the Debian package manager, gain a noticeable performance improvement by using one of the AIO fsync() operations, instead of sync_file_range() + fsync()? The [proposed] fsync2() API is ...
0
votes
1answer
26 views

Slow IO on OracleVM [closed]

We have a slow disk speed problem on the oracle vm's. Actually we have a same config two diffrent OVM(rac1, rac2), they works like active passive.they are oracle DB servers. One of them(rac1) no disk ...
1
vote
0answers
20 views

cgroup IO control v.s. (the new types of) Linux AIO

In Linux v5.0, do some types of Asynchronous IO bypass the cgroup IO controllers? EDIT: buffered write() and fsync() might be relatively complex to analyse. I found an interesting post about ...
0
votes
1answer
43 views

Is there a kind of '/dev/null' to fake output file path?

Let says that I have a CLI which transforms a file_a into a file_b like so: transform --input /path/to/file_a --ouptut /path/to/file_b I have developed a program that use this CLI and need to unit ...
31
votes
5answers
2k views

How to redirect stdout to a file, and stdout+stderr to another one?

How can I achieve cmd >> file1 2>&1 1>>file2 That is, the stdout and stderr should redirect to one file (file1) and only stdout (file2) should redirect to another (both in append ...
2
votes
2answers
99 views

NVMe disk shows 80% io utilization, partitions show 0% io utilization

I have a CentOS 7 server (kernel 3.10.0-957.12.1.el7.x86_64) with 2 NVMe disks with the following setup: # lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT nvme0n1 259:0 0 477G 0 ...
5
votes
2answers
118 views

Why does hexdump try to read through EOF?

When run from a terminal, hexdump does not react to a single ^D at the beginning of the line, as cat, od, bc, etc. do, unless there was no input yet: prompt% hexdump -C <control-D> prompt% ...
1
vote
1answer
88 views

Understanding File Streams and File Descriptors

For extra context, I asked this question earlier, and thought I understood things, but not anymore. I know that echo ignores stdin. I know that stdin, stderr and stdout exist and, in this context, I ...
0
votes
0answers
60 views

ZFS Disk Image Performance

I am trying to move from LVM to ZFS as a backing store for raw virtual machine / disk images on a new server. I am using Ubuntu 18.04 and the SSDs are all Intel D3-S4510. The problem I am ...
0
votes
0answers
39 views

How to find max queue length for device?

Recently I ran into some I/O overload problrms on some of our machines and I have been searching to find thresholds for some metrics like: avgqu-sz await svctm Then I saw this post on Stack ...
0
votes
0answers
82 views

Who is causing extremely high CPU usage when writing to SSD?

When writing to SSD (regardless if over file system or directly into /dev/sda) I observe extremely high CPU usage, with top showing CPU: 0% usr 53% sys 0% nic 9% idle 27% io 0% irq 9% ...
0
votes
0answers
41 views

How to create a new character device under /proc/device file?

I am trying to create a node with mknod /tmp/ttyACM0 c 166 0 command but then No such device or address error returns when I try to open serial port with ttyACM0 because there is no device in proc/...
0
votes
0answers
34 views

How shell delivers user's input to program and shows program's output? [duplicate]

Since shell (for example, Bash) is just another program with its own stdin/stdout/stderr streams, how does it deliver user's input to running program? Here's simple program, that reads value from ...
1
vote
1answer
35 views

Enabling Asynchronous IO at HP-UX Operating System

We are using HP-UX Itanium version B.11.31 and this server is used for running Sybase database version ASE15.7. I would like to know how to enable Asynchronous IO at Operating System level. I went ...
0
votes
2answers
46 views

Can one processes have a descriptor that is non-blocking while another process have a descriptor referencing the same file/socket that is blocking?

I am more interested in sockets than regular files, but basically I want to know whether one process can "see" a socket as blocking where another process can see it as non-blocking. I am guessing yes,...
1
vote
0answers
15 views

Using Systemtap probing Getting the virtual address location of bytes for every read write operation in Linux

Using Systemtap probing tool I got the way to get total amount of bytes read by a file by probing syscall.read and syscall.read.return function. syscall.read.return has returnval() which gives the ...
2
votes
0answers
24 views

Record filesystem activity

I would like to record all filesystem activity so that I can later replay it at will (for benchmarking purposes). What's the best way of doing that? I considered strace, but rejected that because it ...
1
vote
0answers
106 views

ls command is too slow for some directories

When I run ls command for some directories, it may take about 1.5 seconds to complete even if directory has just a few files, but for most of directories it's very fast. It seems that it's slow for ...
0
votes
0answers
12 views

Readonly flag after error: why does Kernel detect an error?

I'm debugging a C++ application running on Linux Embedded: uname -r: 4.14.40 The application is downloading a file on an external flash SD card: mount: sd-path type vfat (rw,relatime,fmask=0022,...
0
votes
0answers
51 views

USB disk disconnects when exchange quite high data

I have an embedded PC (based on Atom Z3735F with a 32 GB eMMC for o.s.) that I use as NAS with OpenMediaVault 4.1.2-1. On an USB port I connected a 1 TB Hard-disk for storage. Whenever I do something ...
3
votes
2answers
72 views

Reading and writing data temporarily without using the filesystem

I'm using ansible in a vagrant environment managing a debian box. Due multiple ansible provisioners with vault encrypted data (e. g. database root password) I have to enter the vault password once by ...
1
vote
0answers
57 views

Why can't I interrupt IO operations on linux?

I tried to abort mkfs.ntfs applied on USB drive (forgot the Quck format flag) and then another big disk IO operation (using dd). They both took more than several minutes to abort, so I just rebooted, ...
0
votes
0answers
52 views

How does 200 million function call of fgets translate into only 25,224 system calls?

A I/O efficiency test from APUE: Test file is said to be "98.5 MB with 3 million lines." Code used in "Figure 3.6": #include "apue.h" #define BUFFSIZE 4096 int main(void) { int n; ...
0
votes
0answers
30 views

Is there a file that can be used to monitor `/dev/sda`s only IO activity in Linux?

I am familiar with iotop and those tools. But for some strange reason I have to write a Ruby program on my own to monitor only read and write activity of all the partitions on /dev/sda). I want to ...
2
votes
0answers
35 views

Run Command When IO Load Average is Low?

I have a bash script which calls an external program I cannot modify. The first stage of this program uses high IO and the second stage uses almost no IO, instead it uses a lot of CPU, it does not ...
2
votes
0answers
52 views

A linux “cp” command equivalent that does read/write threading and buffer sharing for increased throughput?

First, let me address your concerns. I'm copying from one slow drive, to another (different) slow drive. Blocking for reads on one drive until I write to another leaves a lot of room for improvement. ...
3
votes
1answer
316 views

How to execute local script by passing arguments on remote via ssh as a different user?

I am trying to execute a local script on remote server via ssh using keytab authentication. I am able to connect to remote via ssh as sachin(user who gets the ticket) and then I am spawning a bash ...
23
votes
1answer
3k views

Why don't reads from /dev/zero count as IO_RBYTES?

I am emptying out a hard drive on some Linux 4.x OS using this command: sudo sh -c 'pv -pterb /dev/zero > /dev/sda' And I opened another tty and started sudo htop and noticed this: PID USER ...
0
votes
1answer
17 views

CFQ - “In case of asynchronous requests, all the requests from all the processes are batched together according to their process's I/O priority”

I think the following statements about CFQ contradict each other: "In case of asynchronous requests, all the requests from all the processes are batched together according to their process's I/O ...
0
votes
1answer
39 views

Why am I seeing a ton of SyS_poll?

I have two different Linux desktops connected to a Synology NAS like so: 10.0.0.20:/volume1/chronicle /chronicle nfs rw,hard,intr 0 0 and I am frequently seeing my computer locking up ...
1
vote
0answers
70 views

Weird I/O stalls affecting a whole desktop

After a recent hardware migration I started experiencing weird I/O stalls affecting my desktop Debian Stretch system. Typical symptoms, all happening during each stall: I stop being able to interact ...
1
vote
1answer
115 views

Splitting file by 1st column: Too many open files

This is an extension of the question: Splitting a file into multiple files based on 1st column value. Since I just joined, I do not have enough rep points to add a comment to the above question. ...
0
votes
1answer
36 views

How should we interpret the phrase “asynchronous requests” in cfq-iosched.txt?

The old cfq-iosched.txt mentions "asynchronous" or "async" requests in several places. CFQ maintains the per process queue for the processes which request I/O operation(synchronous requests). In ...
-3
votes
1answer
137 views

What is the equivalent (programmatically) of cat /dev/something > file.txt [closed]

I'm copying a video from a camera stream using a command cat /dev/video0 > file.mpeg This works perfectly, however I want too break up the file into smaller chunks... So I've been waiting pressing ...
2
votes
0answers
17 views

Find the size of all relevant caches when doing I/O

So, I'm looking at the stats from iostat, which updates every few seconds, and I see that I/O to device happens in bulks, say, around 40M/s, once every half a minute or so. I'm debugging an ...
-1
votes
1answer
63 views

tar with ssh: what happens if creation speed is higher than network speed?

I'm concerned what will happen if speed of source read + stream create operations gets higher than underlying network protocol speed? Is tar going to create and cache data locally before it is ...
-1
votes
3answers
47 views

Oneliner for outputting to file only if string is matched (bash)?

I am making a script where I curl a URL and output to a file like this: curl http://example.com/$1 > $1 Is there any way to make it dismiss anything that doesn't include, say <head>? I ...
3
votes
0answers
290 views

ext4 jbd2 and I/O priority

I want to delete a huge directory with over 6 million files across over 4000 directories, totaling about 26GB of data. As I knew this was going to take a while I used ionice to set the I/O priority ...
1
vote
0answers
29 views

Performance issues on internal SATA device causing high load

I have a problem with the performance of my laptop. In fact I needed to install quite some things recently and the whole machine froze from time to time completely. First some general information: I ...
1
vote
0answers
107 views

How to exclude a directory or partition on pmlogger scan?

As on the details below it look like that the cause of the problem is the pmlogger is spawning the find command. And its takes a long time for the find to scan all the files in one of my partition. ...
5
votes
2answers
95 views

Limit tar read speed

I'm using tar do make backups of a machine. But, it is using a lot of I/O and slows down the whole machine. So, is there a way to limit the read speed of tar? I know about pv, but it limit only the ...
0
votes
0answers
28 views

ionice vs nice for mysqlimport

I am trying to import a huge .sql file into a mysql server. What (if any) is a best option to speed up the process (all other parameters being equal) reducing the process niceness to -20? setting ...
0
votes
0answers
58 views

Cache behavior in dd command

I am performing some dd outputs and running in parallel vmstat: With no direct write (i.e. via cache) $ dd if=/dev/urandom of=somefile.txt bs=1M count=200 200+0 records in 200+0 records out ...
1
vote
2answers
160 views

USB hard drive I/O slow speeds

I have several Mediasonic ProBox HF2-SU3S2 with 4 drives - all with the same symptoms - extremely slow I/O (at best, about 10MB/s write). Each unit is configured with a software RAID 5 using mdadm ...
1
vote
1answer
60 views

Detecting misbehaving USB flash memory devices, that block the rest's IO

I'm using f3 and a custom Bash script to test USB flash memory in big amounts. A common problem I encounter is that some faulty drives would starve all the healthy ones for IO, effectively stalling ...
0
votes
1answer
33 views

bfq_async_charge_factor says async writes are given 10x less than “fair” share of device throughput. How can I observe this?

I've been reading about CONFIG_WBT, and also BFQ. I tried to compare WBT v.s. CFQ in my hard drive. I learned CFQ tries to control Linux' massive async writeback, but its success is limited because ...
2
votes
2answers
613 views

What does Fedora Workstation 29 use as the default I/O scheduler?

If it depends on the exact type of block device, then what is the default I/O scheduler for each type of device? Background information Fedora 29 includes a Linux kernel from the 4.19 series. (...
0
votes
1answer
173 views

Does linux systems flush dirty pages to disk one block by one block?

As far as I know, in linux systems, when modifying the content of a file, the pages containing the content of the file in page cache will be marked dirty, and will be flushed to disk eventually. What ...
1
vote
0answers
148 views

Determine the specific benefit of Writeback Throttling (CONFIG_WBT)

[PATCH 0/8] Throttled background buffered writeback v7 Since the dawn of time, our background buffered writeback has sucked. When we do background buffered writeback, it should have little ...
2
votes
2answers
85 views

Bash: turn stderr back on after suppressing

I've made this script which automates installing/uninstalling git. In my function for testing if git is installed, I use the git --version command and test the return code. I dislike the stderr ...