Questions tagged [pv]
Pipe Viewer, or pv, is a Unix tool for monitoring pipelines and the progress of data through them.
62
questions
0
votes
0
answers
16
views
Replicating SD card: hangs at 99%
The goal is to replicate an SD card.
Despite successfully replicating the SD card a few years ago, a recent attempt hangs at 99%, despite attempts from 2 different PCs.
user@JUPITER Desktop$ date; ...
- 1,196
2
votes
1
answer
122
views
pv gives 100% progress with small buffer
I have to do a long task (convert image format) for every file of a folder. I achieved to use pv to write the estimation of duration with (used sleep here to simulated processing time):
pv -B 1 =(find ...
- 121
1
vote
1
answer
103
views
How to avoid hardware damage with dd
I wanted to create an encrypted USB-stick. In the tutorial I used it said something like 'to avoid pattern based attacks dump random input to the drive'. So I did by dding /dev/urandom to the drive. ...
- 52
1
vote
1
answer
203
views
Cut off pipe after N bytes
I am piping information to a file using myTool > file.txt 2>&1, but the tool might generate GigaBytes worth of data - I need to cut off after the first N bytes, let say 2MB.
It seems pv can ...
- 111
0
votes
0
answers
96
views
did `dd` buffer entire ~700MB iso image?
I read that to transfer an iso image to a pendrive and print progress I should execute pipeline below.
$ dd if=$IMG bs=4M | pv -s 668M | sudo dd of=/dev/sdc bs=4M
pv should print and update a ...
- 164
2
votes
2
answers
1k
views
How to format hexdump as xxd, possible for xxd -revert?
I wish to dump the raw content of a SD card into a file for inspection. Most parts of it are zeroes. Learnt from this SuperUser answer I could use pv to show the progress of od and hexdump. Both ETA ...
- 325
0
votes
1
answer
562
views
Get hash from file while you copy or move it
Say I copy a file with pv, is there any way to also get any hash, like md5, sha1, etc without having to read origin twice? It has to work with big files or block devices.
Example command which does ...
- 185
10
votes
5
answers
20k
views
pvcreate: Can't use /dev/sda: device is partitioned
I'm currently installing arch linux and when I try to create a physical volume it gives me this error.
Can not use /dev/sda: device is partitioned
What is this error and how can I get rid of it?
PS: ...
- 101
1
vote
2
answers
230
views
Execute single command with write caching disabled
Whenever I write a large file to a USB drive, write caching makes it very difficult to track process. I can disable it in various ways like echo 1000000 > /proc/sys/vm/dirty_bytes which solves the ...
- 815
0
votes
1
answer
327
views
1TB drive compressed shows only 3.8GB, what did I do wrong?
On Linux Mint 20.2 Cinnamon I would like to create a disk image of my secondary disk drive (SATA) containing Windows 10, not that it matters now, directly gzip'ed using the Parallel gzip = pigz onto ...
- 26.8k
3
votes
1
answer
406
views
pv not printing to a pipe
Executing this command displays the output on console. But when output is piped to another command it does not work. See below.
(pv -F $'%t %r %e\n' /dev/nvme0n1p1 | gzip -c >/run/test.img )
0:00:...
- 37
0
votes
1
answer
892
views
PV command to show progress in Dialog with transfer rate in Mbits
I am using PV -n command to read partitions and using gzip with pipe to compress the read data and storing the file. While data is read and written I am using a while loop to show progress using linux ...
- 37
0
votes
1
answer
200
views
How does redirection to pv actually work?
I am trying to understand how the redirection is exactly working in this command
# tar -czf - ./Downloads/ | (pv -p --timer --rate --bytes > backup.tgz)
What is the english translation ?
All the ...
- 1,075
4
votes
1
answer
277
views
Multiple pv order
I have a script on a Linux machine with a fancy pv piped to a second pv that count a subset of the outputted lines.
Here's the script:
max=1000
for (( i=0; i<max; i++ )); do
[[ $(shuf -i 1-100 -n ...
- 198
3
votes
1
answer
811
views
pipe and redirection speed, `pv` and UUOC
I was testing different methods to produce random garbage and comparing their speed by piping output to pv, as in:
$ cmd | pv -s "$size" -S > /dev/null
I also wanted a "baseline ...
- 1,310
1
vote
2
answers
841
views
restoring hdd image using gzip throws error no space left on device
I created an image from a 256GB HDD using following command:
dd if=/dev/sda bs=4M | pv -s 256G | gzip > /mnt/mydrive/img.gz
later I tried to restore the image to another 512GB HDD on another ...
- 113
1
vote
0
answers
284
views
Invoke pv to monitor files when moving directories
I'm trying to move files (movies) between directories. I have a script which renames my movies and then a separate script which I use to open a new terminal and monitor the movies being moved from one ...
- 11
0
votes
2
answers
384
views
Continue copying file
I was copying a very big file and I accidentally stopped it. Can I resume copying data without need to delete copy and copy data again?
Command I used:
pv original.data > copy.data
- 962
0
votes
1
answer
388
views
How can I estimate the whole time of a write process including sync?
Progress and estimated time to write without and with sync
I have found no tool (or straightforward method) that will include flushing the buffers when showing the progress and estimating the ...
- 6,001
2
votes
2
answers
687
views
How to send output of `pv` to python script, and output progress one line at a time?
I use tar to quickly copy thousands of small files which I pipe to pv to show the progress and speed of the copying.
$ sudo tar c --files-from /tmp/camfilenames |
pv --width 40 |
sudo tar x -C /...
- 161
3
votes
1
answer
12k
views
Multithreaded xz, with gzip, pv, and pipes - is this the most efficient I can get?
I'm excited to learn that xz now supports multithreading:
xz --threads=0
But now I want to utilise this as much as possible. For example, to recompress gzips as xz:
gzip -d -k -c myfile.gz | pv | ...
1
vote
1
answer
179
views
Why should I set the source device with < > when using pv?
I have a question out of curiosity ;-)
I find that I could accomplish cloning a disk with the command pv instead of dd too.
For that I used what I found in the internet
pv < /dev/sda1 > /dev/...
- 11
1
vote
3
answers
229
views
Writing to 2 SD cards at the same time
I would like to write an image at two SD cards at the same time. I envision at least two concurrent writing scenarios:
Is it possible to write the image with a tee or similar bifurcating mechanism?
...
- 1,196
0
votes
1
answer
169
views
Pipe viewer logging
Using the pipe viewer (pv) utility, I can visually monitor the data flow going through a pipe.
I would however like to be able to log the data flow in a file instead of dynamically viewing it in a ...
- 171
0
votes
1
answer
764
views
Total Progress over entire bash shell script
EDIT FOR CLARITY :
say I have the following script (let's posit pv and curl are already installed) :
(that currently runs under ubuntu but that I plan to make POSIX-compliant so that it may run on ...
- 296
0
votes
2
answers
2k
views
df -h results different than VGdisplay / LVdisplay
Im trying to remove the added disk to the VM, as the disk space is not needed anymore.
However, it looks to be that the VM has in the VG almost all the data used what is available there.
However, i'...
6
votes
5
answers
2k
views
Using pv with md5sum
I used md5sum with pv to check 4 GiB of files that are in the same directory:
md5sum dir/* | pv -s 4g | sort
The command completes successfully in about 28 seconds, but pv's output is all wrong. ...
- 3,913
2
votes
1
answer
867
views
Use pv with find -exec
I use the following command to verify ~700 GiB of backed-up files:
$ find -type f -exec md5sum {} + | sort > ~/checksums
This takes many hours, so I would like to integrate pv into the command to ...
- 3,913
4
votes
2
answers
2k
views
How can I run multiple pv commands in parallel?
I want to run a sequence of command pipelines with pv on each one. Here's an example:
for p in 1 2 3
do
cat /dev/zero | pv -N $p | dd of=/dev/null &
done
The actual commands in the pipe don't ...
- 8,241
6
votes
3
answers
3k
views
With the command pv it is possible to clone a drive, how do I mount it? [duplicate]
I was able to backup a drive using the following command.
pv -EE /dev/sda > disk-image.img
This is all well and good, but now I have no way of seeing the files unless I use this command
pv disk-...
- 195
1
vote
1
answer
456
views
watch netcat transfer dump from android to pc
how can i monitor netcat transferring from android to my linux machine
i used this command on android device ( sender ) to make a full dump for my device :
dd if=/dev/block/mmcblk0 | busybox nc -...
- 123
3
votes
3
answers
752
views
Pipe-Viewer problem with changing Rate-Limit
I'm using pv for sending files via ssh.
I can change "active pv" the limit at under 100M without any problem.
When i set active pv process to 100M or 1G or higher I cant change rate anymore...
BUT! ...
- 409
0
votes
1
answer
853
views
How to get PID when start bash script
I'm running bash files for zfs send jobs and this my bash file example:
zfs send -Rc tank/test@snap | pv -fs datasize -F "%p***%t***%e***%r***%b" |
mbuffer -q -s 128k -m 1G -O ip:port
When I start ...
- 409
3
votes
1
answer
2k
views
What is the best way pause "zfs send via PV" and resume
I'm using PV for my ZFS send-recv replication.
I use ZFS resume token too but i want to pause and resume like sigstop, sigcontinue.
Because using resume token means sending same thing again.
So how ...
- 409
4
votes
2
answers
2k
views
How to use pv to show progress of openssl encryption / decryption?
I need to encrypt and be able to decrypt files with openssl, currently I do this simply with:
openssl enc -aes-256-cbc -salt -in "$input_filename" -out "$output_filename"
and the decryption with:
...
- 26.8k
1
vote
2
answers
2k
views
lvm and a partitioning question
So when I do (as a root)
fdisk -l
I see /dev/sda1 and /dev/sda2
Now I am practicing creating logical volumes, when I tried partitioning
/dev/sda2
I got two new partition /dev/sda2p1 and /dev/...
- 1,409
0
votes
1
answer
868
views
Bash redirect 'interactive' output to special file
I want to redirect e.g. the output i.e. the progress bar of pv to /dev/tty1.
... | pv -perts 100G | ... 2> /dev/tty1
But what I get is whenever the progress bar updates it does so in an new ...
- 1,532
6
votes
1
answer
4k
views
Pipe a command to pv but hide all the original command's output
I'm trying to use pv, but I want to hide the command I piped in's output while still be able to see pv's output. Using command &> /dev/null | pv doesn't work (as in, pv doesn't receive any data)...
- 95
3
votes
0
answers
225
views
How can I buffer a pipe with minimal added latency?
I am connecting to a debian system over a serial connection and streaming real time data from a pressure sensor. Originally I used:
python3.6 pressure_streamer.py
The data is being produced at 32....
1
vote
0
answers
132
views
What's wrong with the way I'm using PV?
I'm using PV to insert a buffer of size 50M between my video recording command and my video streaming command:
/usr/bin/raspivid -b 8000000 -w 1280 -h 720 -rot 270 -v -fps 40 -o - -t 0 | pv -B 50M |
...
- 143
1
vote
1
answer
2k
views
How does pv work?
How does the following command work?
pv file.tar.gz | tar -xz
From my understanding the pipe operator | creates a pipe and stdout of pv is mapped to the O_WRONLY end of the pipe and tar's stdin is ...
1
vote
0
answers
77
views
pv working with existing PID doesn't work with -s option
I would like to both use -d and -s option while calling pv. I tried as below:
$ cat -n /tmp/x.py
1 import time
2 import threading
3 import sys
4
5 def w():
6 ...
- 158
0
votes
0
answers
840
views
Pipe viewer (PV) not working?
I'm trying to use Pipe Viewer on a video pipe that I'm both writing and reading from.
The below command works perfectly when I pass temp_video.h264 as an -i input to ffmpeg. However, when I attempt ...
- 9
3
votes
2
answers
160
views
Which command produces more data per second?
Which command produces more data per second? This could be useful to quickly fill a file with garbage data or to test data transfer rates. So far, I found that "/dev/zero" is the quickest one.
$ cat /...
- 141
5
votes
1
answer
3k
views
How track progress of a command in a pipe if only the size of its input is known in advance?
I would like to track progress of a slow operation using pv. The size of the input of this operation is known in advance, but the size of its output is not. This forced me to put pv to the left of the ...
- 446
16
votes
1
answer
1k
views
How to get an average pipe flow speed
If myfile is increasing over time, I can get the number of line per second using
tail -f | pv -lr > /dev/null
It gives instantaneous speed, not average.
How can I get the average speed (i.e the ...
- 4,698
12
votes
2
answers
4k
views
How to show progress with GPG for large files?
I need to encrypt a large file using gpg. Is it possible to show a progress bar like when using the pv command?
- 1,001
1
vote
1
answer
646
views
Can the unit displayed in the transfer rate meter in pv be changed?
In pv, the rate meter is displayed as
47.5MiB 0:00:00 [ 165MiB/s] [================================>] 100%
where the unit used for the transfer stats is MiB (1024 bytes). Is it possible to change ...
0
votes
0
answers
489
views
LVM Logical Volume (linear) best performance
As a system administrator, one thing I do very often is increase LV's size. In most cases I have to extend first the VG, adding a new disk (we are talking about virtual machines). I have got a lot of ...
- 495
0
votes
1
answer
155
views
What on earth is pv -bartpes
Many pages advise using a switch for pv called -bartpes. The following command from this page is an example:
# openssl enc -aes-256-ctr -pass pass:"$(dd if=/dev/urandom bs=128 count=1 2>/dev/null |...
- 3,913