Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [compression]

Shrinking (compressing) and restoring (decompressing) of data.

0
votes
0answers
21 views

pxz (parallel xz) output meaning

I am running pxz with -v option, to "show progress" according to the man page. I am getting as output a sequence of seemingly random numbers, like: 0 1 7 3 2 5 8 4 9 6 6 9 5 0 1 2 3 8 7 4 ... It ...
0
votes
1answer
22 views

Is there a way to create a (compressed) squashfs that contains an uncompressed file

We have a large file that needs to be uncompressed inside a compressed squashfs. Is this possible without patching mksquashfs? Something like this would be nice: $ mksquashfs existing_squash.sqsh ...
4
votes
1answer
36 views

Time-adaptive compression tool

I'm asking about a scenario of copying a big file to a remote server. A simplest case is: tar c myfile | ssh myserver tar x If network connectivity is fast then all is fine. On a slower network I ...
0
votes
0answers
27 views

Unzip files in loop

I have the following code which extracts numbers from netcdf files and every loop puts them in another netcdf file. mapfile Latitude < <(tr -s ' ' '\n' < final_ADCP_Saved.matLatitude.txt) #...
1
vote
1answer
42 views

How do I reduze the size of a video to a target size?

I want to reduce the size of a video to be able to send it via email and such. I looked at this question where I got good advice on how to reduce it. Problem is that I need to manually calculate the ...
0
votes
1answer
42 views

How can I unload a filesystem module in a RHEL 7 server in order to harden it?

I'm trying to harden my RHEL Linux server 7 according to the CIS guidelines, so I want to unload the cramfs. I typed the following command: modprobe -n -v cramfs and I got the following answer: ...
0
votes
0answers
34 views

Why I cannot tar gzip all the contents of a specific folder

I have a folder containing several folders. I want to tar and gzip the parent folder with all the contents of the subfolders. However, when I use tar -cvzf folder.tar.gz Folder/ only a few ...
4
votes
1answer
362 views

Zip/Tar file compressed to larger size? [duplicate]

I have 4 files that I want to include them in a .zip file. drwxr-xr-x 7 skemelio themelis 4,0K Οκτ 18 00:21 CSharp drwxr-xr-x 11 skemelio themelis 4,0K Οκτ 18 00:17 gnome-games drwxr-xr-x 8 ...
0
votes
1answer
93 views

tar + compress folder to tar.gz and removed the folder

by the following way we tar the folder presto-server-0.210 to file tar -czvf presto-server-archive.tar.gz presto-server-0.210 so after we tar the folder , folder still exists is it possible to use ...
2
votes
0answers
46 views

Compress VirtualBox Linux Virtual Machine

I am trying to create a VirtualBox linux virtual machine on which I want to mainly run Wine to install a windows application. I managed to install Lubuntu 18.04.2, since it is the lightest Ubuntu ...
0
votes
0answers
73 views

How can I copy sparse files from a ext4 file system to an exfat file system and keep the apparent size?

My issue is that I am trying to copy files from my Ubuntu linux system which uses a ext4 file system to a Sandisk 1TB USB card, which uses an exFat system. In the ext4 system, the entire directory ...
1
vote
1answer
24 views

Using terminal to zip unknown amount of packages with dynamic names

I am trying to figure out a solution for unix based systems (maxOS, Linux) to zip unknown amount of packages, ideally without requiring users to install any additional third party software. I have ...
2
votes
1answer
106 views

Best compression for operating system image

I have an Operating system image of size 2.5G. I have a device with a limited size. Thus I was looking for the best possible solution for providing the compression. Below are the commands and ...
0
votes
0answers
35 views

Compare file compression methods by sending to /dev/null

I'd like to compare the time to completion for compression algorithms and parameters. I'm confident that the inputs will be set up properly. In order to save space and avoid some variability my ...
0
votes
0answers
39 views

How to make use of tar --verify directly?

This question is about compressing rather large data directories (around 22-23 GiB), specifically the integrity checks of such archives. I do these big backups about once a month, so the time it takes ...
2
votes
0answers
27 views

Resumable compression

I'm in the process of compressing a very large file (several TBs) with the following command, and it's been running for several days. tar -I pigz -cf /path/to/compressed.tgz /dataset/to/compress I'm ...
3
votes
0answers
44 views

How to unpack a tag.gz file without having 2x free space?

I have about 1 million small files, that take up about 20 GB. I need to upload them to a host that only has 23 GB free space. I would prefer compressing the files locally to increase the transmission ...
0
votes
0answers
7 views

Least resource heavy file compression method on hosting server

There are many compression formats like tar, zip, bzip etc. On a shared hosting (Linux), if I were to archive a huge folder for migration, which format would be best (least resource heavy)? Storage ...
0
votes
0answers
29 views

Is it possible to dynamically add to a tar.lrz file?

A bit of a basic problem, I'm slightly embarrased I'm running into it, however man lrztar didn't show me any commands for this. Basically, lets say I've got an archiving station which handles ...
1
vote
1answer
50 views

Stream text into a file while compressing it

I have an Arduino Nano connected to a Raspberry Pi which streams text data (the output of a GY-91 sensor) to a listener on the Raspberry Pi. P: 974.044 | T: 22.65 | AG: -0.17 -0.10 1.01 | GD: 1....
2
votes
1answer
39 views

Improving performance of Script Using find and exec [closed]

I have a script that iterates through a given directory and automatically compresses those that do not contain at least one file used in less than 30 days. Now, I am wondering if I could improve ...
2
votes
2answers
297 views

compressing and decompressing dd image - zstd instead of gzip

Earlier I was using fsarchiver to create compressed partition image. Due to some weird behavior I am choosing to replace it with dd. However, I like how fsarchiver compressed with zstd. So, I ...
0
votes
1answer
46 views

fsarchiver taking ages to backup and restore

Using Linux Mint 19.1 bootable USB Ext4 filesystem Processor 8600k Ram 16gb By default fsarchiver uses zstd compression Earlier when I used the following two commands, each command individually took ...
2
votes
1answer
143 views

how to set NOCOMPRESS flag on a file in btrfs?

according to https://btrfs.wiki.kernel.org/index.php/Compression#How_do_I_enable_compression.3F there exist a "NOCOMPRESS" flag, how do i set the NOCOMPRESS flag on a file?
2
votes
1answer
165 views

Easy way to change compression level of a zip file [duplicate]

On linux, is there an easy way to recompress a zip file (changing compression-level) with preserving as most meta-data as possible (timestamps etc.)?
1
vote
1answer
236 views

tar: compress text stdin and stdout [duplicate]

Using tar, is it possible to compress stdin text and have the compressed data appear in stdout? Example: $ echo "test" | tar c - <compressed blob here> I'm using macOS Mojave.
0
votes
3answers
451 views

Linux determine ZIP size without compression

I would like to make a ZIP file using zip command. Zip file would contain large number of directories containing large numbers of subdirectories, files etc. Is there any way to calculate final size ...
9
votes
1answer
774 views

Uncompressed file estimation wrong?

I had a large (~60G) compressed file (tar.gz). I used split to break it into 4 parts and then cat to join them back together. However, now, when I am trying to estimate the size of the uncompressed ...
0
votes
1answer
78 views

Is there an efficient command for listing the header of a particular file in tar.bz2 without uncompressing/extracting?

I was working with a data.tar.bz2, which contains lots of other big files of the scale of a few hundred Gigabytes. I just wanted to see the header (first line) of a particular file file-of-interest, ...
0
votes
2answers
1k views

Compression Utility Max Files Size Limit | Unix/Linux

In Unix/Linux is there any max files size limit that a compression utility ( gzip/compress) can compress. I remember years ago it was mentioned in the gzip page that it can compress files up to 4 gb. ...
0
votes
2answers
313 views

Does tar.gz do effective file compression of large files?

I wanted to compress a ~29GB file significantly, and used the tar command in the terminal like this : tar cvzf file.tar.gz /path/to/directory and it compressed it to ~26GB, so I looked up maximum ...
-1
votes
1answer
307 views

How to split a compressed file?

I have a 44GB 7z compressed file that I compressed with lzma2 and it took around 11 hours (original file is a text file of 285GB). 7z a -t7z Files.7z -m0=lzma2 -mx=9 -aoa So, what I want is to do is ...
-2
votes
1answer
171 views

compress specific folder(directory) from folder with tar [closed]

I am trying to use tar to compress the subdirectories /root/folder/folder{1..5} into a single compressed archive test1.tar.bz2. I'm using the following command: tar -cvjf test1.tar.bz2 root/folder/...
0
votes
1answer
26 views

Arguments for, and the applicability of, using multiple forms of compression on one or more files?

I was wondering whether there are any scenarios which require file(s) to be compressed multiple times with the same or a different algorithm. As far as i can tell the file size can be reduced further ...
3
votes
1answer
880 views

cat on big files does not work

I'm trying to concatenate four big files in two. The files *_1P.gz contain the same amount of lines as the corrisponding *_2P.gz. The files A_1P.gz and A_2P.gz both contain 1104507560 lines. The ...
7
votes
1answer
583 views

Permission changes after extracting tgz file

While extracting a tgz file, I have noticed that the permission changes to something weird! The tgz file belongs to root:root however, the folder belongs to 502:games [root@rocks7 common]# ls -l -rw-...
3
votes
1answer
51 views

Create separate 7z archives for each directory in the current directory and additionally parallelize through GNU Parallel

I have about two hundred sub-directories located within a directory of interest: $ ls backup 201302 201607 201608 201609 201610 201701 201702 201705 201801 201802 I want to create a 7z archive xyz....
2
votes
1answer
81 views

Bash script for uncompressing multiple files at a once

I have a set of files: lium3@l128-37:/DATA/fr94/01 $ ls -l total 7784 -rw-r--r-- 1 lium3 mcm1 395575 May 2 1996 fr940104.0z -rw-r--r-- 1 lium3 mcm1 113815 May 2 1996 fr940104.1z -rw-r--r-- 1 ...
4
votes
4answers
2k views

Best compression tool for full disk imaging that can saturate a USB 3 connection on low-powered machines?

I often work on full disk images, I often need to read and write full disk images on not so capable laptops using a USB-3 disk as my temporary medium. Duplicating a raw image is probably the fastest ...
1
vote
1answer
4k views

Unzipping a .gz file to get a text file but getting a binary file? [closed]

I have a .gz file called document.txt.gz file. It is supposed to be a text file which got compressed to a gzip file. When I uncompress, I am getting a corrupt file. I ran the following commands: $ ...
2
votes
1answer
303 views

kernel module: set executable bit? compress with xz?

I wrote/tweaked a custom kernel module and installed it. It works as expected, but I've noticed that other kernel modules on my system are compressed with xz and have 0444 permissions, whereas I did ...
1
vote
2answers
226 views

Is it possible to change the zlib-flate compression level?

I use the zlib-flate compression utility. Is it possible to compress at different levels. The man page information is pretty vague. If not is there an alternative utility that allows you to compress ...
3
votes
1answer
156 views

How do I extract an “X1 archive”?

I have a strange file without a file extension that's supposedly an archive of the contents of a git repository. According to file (1), it's an "X1 archive": $ file my_git_repo my_git_repo: X1 ...
1
vote
2answers
751 views

compress file using /bin/gzip with particular patterns

I want to compress files individually with the particular pattern. For example, I have directory as below : /log/log1 -/1/log2/file.1 -/1/log2/file123 -/1/log2/file2.1 /log/log2 -/2/...
0
votes
2answers
320 views

Is there a fp8 (Fast PAQ) compression utility for linux?

I'm trying to compress a large archive of similar JPG images. The best compression algorithm I've found so far is fp8. In this question I show my tests and ask for better options. But fp8 is the best ...
0
votes
0answers
41 views

How to backup and compress quick rotating files to a remote server

I have a server application creating log files and rotating these files based on size. The files are time stamped at creation. The application then deletes the oldest file when X number of files in ...
0
votes
1answer
108 views

Recovering file from compressed raw data on btrfs

I was recently acquiring data from a networked instrument, saving results as text files. Unfortunately, I forgot to change the filename during one of my runs, and I managed to overwrite an important ...
4
votes
1answer
292 views

compression with btrfs-send / btrfs-receive

When I send a compressed subvol (i.e., one part of a filesystem mounted with the compress mount option) with btrfs-send, does btrfs-send send it compressed, or does it decompress it first? Similarly, ...
1
vote
1answer
610 views

Zlib Compression

I want to compress a file using Zlib (Best compression) i have googled a bit around and i have found a way to compress, but it isn't at the best compression level, I'd love to get some help OS ...
0
votes
1answer
43 views

compress a single file … zram ? cloop?

We have a legacy centos 5.11 server; filesystems are ext3, we cannot change. Is it possible to have realtime compression on a single file (a giant log file) ? zram : I cannot find it in the repos... ...