Questions tagged [pv]

Pipe Viewer, or pv, is a Unix tool for monitoring pipelines and the progress of data through them.

Filter by
Sorted by
Tagged with
0 votes
0 answers
32 views

How to avoid hardware damage with dd

I wanted to create an encrypted USB-stick. In the tutorial I used it said something like 'to avoid pattern based attacks dump random input to the drive'. So I did by dding /dev/urandom to the drive. ...
user avatar
0 votes
0 answers
34 views

Cut off pipe after N bytes

I am piping information to a file using myTool > file.txt 2>&1, but the tool might generate GigaBytes worth of data - I need to cut off after the first N bytes, let say 2MB. It seems pv can ...
user avatar
0 votes
0 answers
46 views

did `dd` buffer entire ~700MB iso image?

I read that to transfer an iso image to a pendrive and print progress I should execute pipeline below. $ dd if=$IMG bs=4M | pv -s 668M | sudo dd of=/dev/sdc bs=4M pv should print and update a ...
user avatar
1 vote
1 answer
482 views

How to format hexdump as xxd, possible for xxd -revert?

I wish to dump the raw content of a SD card into a file for inspection. Most parts of it are zeroes. Learnt from this SuperUser answer I could use pv to show the progress of od and hexdump. Both ETA ...
user avatar
  • 233
0 votes
1 answer
79 views

Get hash from file while you copy or move it

Say I copy a file with pv, is there any way to also get any hash, like md5, sha1, etc without having to read origin twice? It has to work with big files or block devices. Example command which does ...
user avatar
1 vote
4 answers
3k views

pvcreate: Can't use /dev/sda: device is partitioned

I'm currently installing arch linux and when I try to create a physical volume it gives me this error. Can not use /dev/sda: device is partitioned What is this error and how can I get rid of it? PS: ...
user avatar
  • 11
1 vote
2 answers
56 views

Execute single command with write caching disabled

Whenever I write a large file to a USB drive, write caching makes it very difficult to track process. I can disable it in various ways like echo 1000000 > /proc/sys/vm/dirty_bytes which solves the ...
user avatar
  • 674
0 votes
1 answer
106 views

1TB drive compressed shows only 3.8GB, what did I do wrong?

On Linux Mint 20.2 Cinnamon I would like to create a disk image of my secondary disk drive (SATA) containing Windows 10, not that it matters now, directly gzip'ed using the Parallel gzip = pigz onto ...
user avatar
3 votes
1 answer
98 views

pv not printing to a pipe

Executing this command displays the output on console. But when output is piped to another command it does not work. See below. (pv -F $'%t %r %e\n' /dev/nvme0n1p1 | gzip -c >/run/test.img ) 0:00:...
user avatar
0 votes
1 answer
245 views

PV command to show progress in Dialog with transfer rate in Mbits

I am using PV -n command to read partitions and using gzip with pipe to compress the read data and storing the file. While data is read and written I am using a while loop to show progress using linux ...
user avatar
0 votes
1 answer
88 views

How does redirection to pv actually work?

I am trying to understand how the redirection is exactly working in this command # tar -czf - ./Downloads/ | (pv -p --timer --rate --bytes > backup.tgz) What is the english translation ? All the ...
user avatar
  • 883
4 votes
1 answer
122 views

Multiple pv order

I have a script on a Linux machine with a fancy pv piped to a second pv that count a subset of the outputted lines. Here's the script: max=1000 for (( i=0; i<max; i++ )); do [[ $(shuf -i 1-100 -n ...
user avatar
  • 198
3 votes
1 answer
315 views

pipe and redirection speed, `pv` and UUOC

I was testing different methods to produce random garbage and comparing their speed by piping output to pv, as in: $ cmd | pv -s "$size" -S > /dev/null I also wanted a "baseline ...
user avatar
  • 1,110
1 vote
2 answers
456 views

restoring hdd image using gzip throws error no space left on device

I created an image from a 256GB HDD using following command: dd if=/dev/sda bs=4M | pv -s 256G | gzip > /mnt/mydrive/img.gz later I tried to restore the image to another 512GB HDD on another ...
user avatar
  • 113
1 vote
0 answers
199 views

Invoke pv to monitor files when moving directories

I'm trying to move files (movies) between directories. I have a script which renames my movies and then a separate script which I use to open a new terminal and monitor the movies being moved from one ...
user avatar
  • 11
0 votes
2 answers
129 views

Continue copying file

I was copying a very big file and I accidentally stopped it. Can I resume copying data without need to delete copy and copy data again? Command I used: pv original.data > copy.data
user avatar
  • 772
0 votes
1 answer
163 views

How can I estimate the whole time of a write process including sync?

Progress and estimated time to write without and with sync I have found no tool (or straightforward method) that will include flushing the buffers when showing the progress and estimating the ...
user avatar
  • 4,587
2 votes
2 answers
485 views

How to send output of `pv` to python script, and output progress one line at a time?

I use tar to quickly copy thousands of small files which I pipe to pv to show the progress and speed of the copying. $ sudo tar c --files-from /tmp/camfilenames | pv --width 40 | sudo tar x -C /...
user avatar
3 votes
1 answer
7k views

Multithreaded xz, with gzip, pv, and pipes - is this the most efficient I can get?

I'm excited to learn that xz now supports multithreading: xz --threads=0 But now I want to utilise this as much as possible. For example, to recompress gzips as xz: gzip -d -k -c myfile.gz | pv | ...
user avatar
1 vote
1 answer
151 views

Why should I set the source device with < > when using pv?

I have a question out of curiosity ;-) I find that I could accomplish cloning a disk with the command pv instead of dd too. For that I used what I found in the internet pv < /dev/sda1 > /dev/...
user avatar
1 vote
3 answers
170 views

Writing to 2 SD cards at the same time

I would like to write an image at two SD cards at the same time. I envision at least two concurrent writing scenarios: Is it possible to write the image with a tee or similar bifurcating mechanism? ...
user avatar
  • 1,104
0 votes
1 answer
102 views

Pipe viewer logging

Using the pipe viewer (pv) utility, I can visually monitor the data flow going through a pipe. I would however like to be able to log the data flow in a file instead of dynamically viewing it in a ...
user avatar
0 votes
1 answer
559 views

Total Progress over entire bash shell script

EDIT FOR CLARITY : say I have the following script (let's posit pv and curl are already installed) : (that currently runs under ubuntu but that I plan to make POSIX-compliant so that it may run on ...
user avatar
  • 296
0 votes
2 answers
2k views

df -h results different than VGdisplay / LVdisplay

Im trying to remove the added disk to the VM, as the disk space is not needed anymore. However, it looks to be that the VM has in the VG almost all the data used what is available there. However, i'...
user avatar
6 votes
5 answers
2k views

Using pv with md5sum

I used md5sum with pv to check 4 GiB of files that are in the same directory: md5sum dir/* | pv -s 4g | sort The command completes successfully in about 28 seconds, but pv's output is all wrong. ...
user avatar
  • 3,311
2 votes
1 answer
658 views

Use pv with find -exec

I use the following command to verify ~700 GiB of backed-up files: $ find -type f -exec md5sum {} + | sort > ~/checksums This takes many hours, so I would like to integrate pv into the command to ...
user avatar
  • 3,311
3 votes
2 answers
1k views

How can I run multiple pv commands in parallel?

I want to run a sequence of command pipelines with pv on each one. Here's an example: for p in 1 2 3 do cat /dev/zero | pv -N $p | dd of=/dev/null & done The actual commands in the pipe don't ...
user avatar
  • 7,670
6 votes
3 answers
2k views

With the command pv it is possible to clone a drive, how do I mount it? [duplicate]

I was able to backup a drive using the following command. pv -EE /dev/sda > disk-image.img This is all well and good, but now I have no way of seeing the files unless I use this command pv disk-...
user avatar
1 vote
1 answer
400 views

watch netcat transfer dump from android to pc

how can i monitor netcat transferring from android to my linux machine i used this command on android device ( sender ) to make a full dump for my device : dd if=/dev/block/mmcblk0 | busybox nc -...
user avatar
3 votes
3 answers
576 views

Pipe-Viewer problem with changing Rate-Limit

I'm using pv for sending files via ssh. I can change "active pv" the limit at under 100M without any problem. When i set active pv process to 100M or 1G or higher I cant change rate anymore... BUT! ...
user avatar
  • 389
0 votes
1 answer
547 views

How to get PID when start bash script

I'm running bash files for zfs send jobs and this my bash file example: zfs send -Rc tank/test@snap | pv -fs datasize -F "%p***%t***%e***%r***%b" | mbuffer -q -s 128k -m 1G -O ip:port When I start ...
user avatar
  • 389
3 votes
1 answer
1k views

What is the best way pause "zfs send via PV" and resume

I'm using PV for my ZFS send-recv replication. I use ZFS resume token too but i want to pause and resume like sigstop, sigcontinue. Because using resume token means sending same thing again. So how ...
user avatar
  • 389
4 votes
2 answers
2k views

How to use pv to show progress of openssl encryption / decryption?

I need to encrypt and be able to decrypt files with openssl, currently I do this simply with: openssl enc -aes-256-cbc -salt -in "$input_filename" -out "$output_filename" and the decryption with: ...
user avatar
1 vote
2 answers
2k views

lvm and a partitioning question

So when I do (as a root) fdisk -l I see /dev/sda1 and /dev/sda2 Now I am practicing creating logical volumes, when I tried partitioning /dev/sda2 I got two new partition /dev/sda2p1 and /dev/...
user avatar
  • 1,349
0 votes
1 answer
705 views

Bash redirect 'interactive' output to special file

I want to redirect e.g. the output i.e. the progress bar of pv to /dev/tty1. ... | pv -perts 100G | ... 2> /dev/tty1 But what I get is whenever the progress bar updates it does so in an new ...
user avatar
  • 1,572
6 votes
1 answer
3k views

Pipe a command to pv but hide all the original command's output

I'm trying to use pv, but I want to hide the command I piped in's output while still be able to see pv's output. Using command &> /dev/null | pv doesn't work (as in, pv doesn't receive any data)...
user avatar
2 votes
0 answers
177 views

How can I buffer a pipe with minimal added latency?

I am connecting to a debian system over a serial connection and streaming real time data from a pressure sensor. Originally I used: python3.6 pressure_streamer.py The data is being produced at 32....
user avatar
1 vote
0 answers
122 views

What's wrong with the way I'm using PV?

I'm using PV to insert a buffer of size 50M between my video recording command and my video streaming command: /usr/bin/raspivid -b 8000000 -w 1280 -h 720 -rot 270 -v -fps 40 -o - -t 0 | pv -B 50M | ...
user avatar
1 vote
1 answer
2k views

How does pv work?

How does the following command work? pv file.tar.gz | tar -xz From my understanding the pipe operator | creates a pipe and stdout of pv is mapped to the O_WRONLY end of the pipe and tar's stdin is ...
user avatar
1 vote
0 answers
75 views

pv working with existing PID doesn't work with -s option

I would like to both use -d and -s option while calling pv. I tried as below: $ cat -n /tmp/x.py 1 import time 2 import threading 3 import sys 4 5 def w(): 6 ...
user avatar
  • 158
0 votes
0 answers
719 views

Pipe viewer (PV) not working?

I'm trying to use Pipe Viewer on a video pipe that I'm both writing and reading from. The below command works perfectly when I pass temp_video.h264 as an -i input to ffmpeg. However, when I attempt ...
user avatar
3 votes
2 answers
157 views

Which command produces more data per second?

Which command produces more data per second? This could be useful to quickly fill a file with garbage data or to test data transfer rates. So far, I found that "/dev/zero" is the quickest one. $ cat /...
user avatar
  • 141
5 votes
1 answer
2k views

How track progress of a command in a pipe if only the size of its input is known in advance?

I would like to track progress of a slow operation using pv. The size of the input of this operation is known in advance, but the size of its output is not. This forced me to put pv to the left of the ...
user avatar
  • 430
16 votes
1 answer
1k views

How to get an average pipe flow speed

If myfile is increasing over time, I can get the number of line per second using tail -f | pv -lr > /dev/null It gives instantaneous speed, not average. How can I get the average speed (i.e the ...
user avatar
  • 4,248
12 votes
2 answers
3k views

How to show progress with GPG for large files?

I need to encrypt a large file using gpg. Is it possible to show a progress bar like when using the pv command?
user avatar
  • 941
1 vote
1 answer
549 views

Can the unit displayed in the transfer rate meter in pv be changed?

In pv, the rate meter is displayed as 47.5MiB 0:00:00 [ 165MiB/s] [================================>] 100% where the unit used for the transfer stats is MiB (1024 bytes). Is it possible to change ...
user avatar
0 votes
0 answers
448 views

LVM Logical Volume (linear) best performance

As a system administrator, one thing I do very often is increase LV's size. In most cases I have to extend first the VG, adding a new disk (we are talking about virtual machines). I have got a lot of ...
user avatar
  • 455
0 votes
1 answer
145 views

What on earth is pv -bartpes

Many pages advise using a switch for pv called -bartpes. The following command from this page is an example: # openssl enc -aes-256-ctr -pass pass:"$(dd if=/dev/urandom bs=128 count=1 2>/dev/null |...
user avatar
  • 3,311
1 vote
2 answers
3k views

How to redirect dd to pv? [duplicate]

This is my dd command which I need to modify: dd if=/tmp/nfs/image.dd of=/dev/sda bs=16k Now I would like to use pv to limit the speed of copying from NFS server. How can I achieve that? I know that ...
user avatar
  • 245
56 votes
3 answers
44k views

Is it better to use cat, dd, pv or another procedure to copy a CD/DVD?

Background I'm copying some data CDs/DVDs to ISO files to use them later without the need of them in the drive. I'm looking on the Net for procedures and I found a lot: Use of cat to copy a medium: ...
user avatar