Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [compression]

Shrinking (compressing) and restoring (decompressing) of data.

0
votes
0answers
31 views

Why I cannot tar gzip all the contents of a specific folder

I have a folder containing several folders. I want to tar and gzip the parent folder with all the contents of the subfolders. However, when I use tar -cvzf folder.tar.gz Folder/ only a few ...
4
votes
1answer
354 views

Zip/Tar file compressed to larger size? [duplicate]

I have 4 files that I want to include them in a .zip file. drwxr-xr-x 7 skemelio themelis 4,0K Οκτ 18 00:21 CSharp drwxr-xr-x 11 skemelio themelis 4,0K Οκτ 18 00:17 gnome-games drwxr-xr-x 8 ...
0
votes
1answer
53 views

tar + compress folder to tar.gz and removed the folder

by the following way we tar the folder presto-server-0.210 to file tar -czvf presto-server-archive.tar.gz presto-server-0.210 so after we tar the folder , folder still exists is it possible to use ...
2
votes
0answers
39 views

Compress VirtualBox Linux Virtual Machine

I am trying to create a VirtualBox linux virtual machine on which I want to mainly run Wine to install a windows application. I managed to install Lubuntu 18.04.2, since it is the lightest Ubuntu ...
0
votes
0answers
40 views

How can I copy sparse files from a ext4 file system to an exfat file system and keep the apparent size?

My issue is that I am trying to copy files from my Ubuntu linux system which uses a ext4 file system to a Sandisk 1TB USB card, which uses an exFat system. In the ext4 system, the entire directory ...
1
vote
1answer
22 views

Using terminal to zip unknown amount of packages with dynamic names

I am trying to figure out a solution for unix based systems (maxOS, Linux) to zip unknown amount of packages, ideally without requiring users to install any additional third party software. I have ...
2
votes
1answer
86 views

Best compression for operating system image

I have an Operating system image of size 2.5G. I have a device with a limited size. Thus I was looking for the best possible solution for providing the compression. Below are the commands and ...
0
votes
0answers
31 views

Compare file compression methods by sending to /dev/null

I'd like to compare the time to completion for compression algorithms and parameters. I'm confident that the inputs will be set up properly. In order to save space and avoid some variability my ...
0
votes
0answers
27 views

How to make use of tar --verify directly?

This question is about compressing rather large data directories (around 22-23 GiB), specifically the integrity checks of such archives. I do these big backups about once a month, so the time it takes ...
2
votes
0answers
26 views

Resumable compression

I'm in the process of compressing a very large file (several TBs) with the following command, and it's been running for several days. tar -I pigz -cf /path/to/compressed.tgz /dataset/to/compress I'm ...
3
votes
0answers
33 views

How to unpack a tag.gz file without having 2x free space?

I have about 1 million small files, that take up about 20 GB. I need to upload them to a host that only has 23 GB free space. I would prefer compressing the files locally to increase the transmission ...
0
votes
0answers
6 views

Least resource heavy file compression method on hosting server

There are many compression formats like tar, zip, bzip etc. On a shared hosting (Linux), if I were to archive a huge folder for migration, which format would be best (least resource heavy)? Storage ...
0
votes
0answers
26 views

Is it possible to dynamically add to a tar.lrz file?

A bit of a basic problem, I'm slightly embarrased I'm running into it, however man lrztar didn't show me any commands for this. Basically, lets say I've got an archiving station which handles ...
1
vote
1answer
46 views

Stream text into a file while compressing it

I have an Arduino Nano connected to a Raspberry Pi which streams text data (the output of a GY-91 sensor) to a listener on the Raspberry Pi. P: 974.044 | T: 22.65 | AG: -0.17 -0.10 1.01 | GD: 1....
2
votes
1answer
38 views

Improving performance of Script Using find and exec [closed]

I have a script that iterates through a given directory and automatically compresses those that do not contain at least one file used in less than 30 days. Now, I am wondering if I could improve ...
2
votes
2answers
174 views

compressing and decompressing dd image - zstd instead of gzip

Earlier I was using fsarchiver to create compressed partition image. Due to some weird behavior I am choosing to replace it with dd. However, I like how fsarchiver compressed with zstd. So, I ...
0
votes
1answer
37 views

fsarchiver taking ages to backup and restore

Using Linux Mint 19.1 bootable USB Ext4 filesystem Processor 8600k Ram 16gb By default fsarchiver uses zstd compression Earlier when I used the following two commands, each command individually took ...
2
votes
1answer
99 views

how to set NOCOMPRESS flag on a file in btrfs?

according to https://btrfs.wiki.kernel.org/index.php/Compression#How_do_I_enable_compression.3F there exist a "NOCOMPRESS" flag, how do i set the NOCOMPRESS flag on a file?
2
votes
1answer
110 views

Easy way to change compression level of a zip file [duplicate]

On linux, is there an easy way to recompress a zip file (changing compression-level) with preserving as most meta-data as possible (timestamps etc.)?
1
vote
1answer
166 views

tar: compress text stdin and stdout [duplicate]

Using tar, is it possible to compress stdin text and have the compressed data appear in stdout? Example: $ echo "test" | tar c - <compressed blob here> I'm using macOS Mojave.
0
votes
3answers
315 views

Linux determine ZIP size without compression

I would like to make a ZIP file using zip command. Zip file would contain large number of directories containing large numbers of subdirectories, files etc. Is there any way to calculate final size ...
9
votes
1answer
722 views

Uncompressed file estimation wrong?

I had a large (~60G) compressed file (tar.gz). I used split to break it into 4 parts and then cat to join them back together. However, now, when I am trying to estimate the size of the uncompressed ...
0
votes
1answer
60 views

Is there an efficient command for listing the header of a particular file in tar.bz2 without uncompressing/extracting?

I was working with a data.tar.bz2, which contains lots of other big files of the scale of a few hundred Gigabytes. I just wanted to see the header (first line) of a particular file file-of-interest, ...
0
votes
2answers
750 views

Compression Utility Max Files Size Limit | Unix/Linux

In Unix/Linux is there any max files size limit that a compression utility ( gzip/compress) can compress. I remember years ago it was mentioned in the gzip page that it can compress files up to 4 gb. ...
0
votes
2answers
228 views

Does tar.gz do effective file compression of large files?

I wanted to compress a ~29GB file significantly, and used the tar command in the terminal like this : tar cvzf file.tar.gz /path/to/directory and it compressed it to ~26GB, so I looked up maximum ...
-1
votes
1answer
216 views

How to split a compressed file?

I have a 44GB 7z compressed file that I compressed with lzma2 and it took around 11 hours (original file is a text file of 285GB). 7z a -t7z Files.7z -m0=lzma2 -mx=9 -aoa So, what I want is to do is ...
-2
votes
1answer
138 views

compress specific folder(directory) from folder with tar [closed]

I am trying to use tar to compress the subdirectories /root/folder/folder{1..5} into a single compressed archive test1.tar.bz2. I'm using the following command: tar -cvjf test1.tar.bz2 root/folder/...
0
votes
1answer
25 views

Arguments for, and the applicability of, using multiple forms of compression on one or more files?

I was wondering whether there are any scenarios which require file(s) to be compressed multiple times with the same or a different algorithm. As far as i can tell the file size can be reduced further ...
3
votes
1answer
827 views

cat on big files does not work

I'm trying to concatenate four big files in two. The files *_1P.gz contain the same amount of lines as the corrisponding *_2P.gz. The files A_1P.gz and A_2P.gz both contain 1104507560 lines. The ...
7
votes
1answer
555 views

Permission changes after extracting tgz file

While extracting a tgz file, I have noticed that the permission changes to something weird! The tgz file belongs to root:root however, the folder belongs to 502:games [root@rocks7 common]# ls -l -rw-...
3
votes
1answer
39 views

Create separate 7z archives for each directory in the current directory and additionally parallelize through GNU Parallel

I have about two hundred sub-directories located within a directory of interest: $ ls backup 201302 201607 201608 201609 201610 201701 201702 201705 201801 201802 I want to create a 7z archive xyz....
2
votes
1answer
67 views

Bash script for uncompressing multiple files at a once

I have a set of files: lium3@l128-37:/DATA/fr94/01 $ ls -l total 7784 -rw-r--r-- 1 lium3 mcm1 395575 May 2 1996 fr940104.0z -rw-r--r-- 1 lium3 mcm1 113815 May 2 1996 fr940104.1z -rw-r--r-- 1 ...
4
votes
4answers
1k views

Best compression tool for full disk imaging that can saturate a USB 3 connection on low-powered machines?

I often work on full disk images, I often need to read and write full disk images on not so capable laptops using a USB-3 disk as my temporary medium. Duplicating a raw image is probably the fastest ...
1
vote
1answer
4k views

Unzipping a .gz file to get a text file but getting a binary file? [closed]

I have a .gz file called document.txt.gz file. It is supposed to be a text file which got compressed to a gzip file. When I uncompress, I am getting a corrupt file. I ran the following commands: $ ...
2
votes
1answer
248 views

kernel module: set executable bit? compress with xz?

I wrote/tweaked a custom kernel module and installed it. It works as expected, but I've noticed that other kernel modules on my system are compressed with xz and have 0444 permissions, whereas I did ...
1
vote
2answers
183 views

Is it possible to change the zlib-flate compression level?

I use the zlib-flate compression utility. Is it possible to compress at different levels. The man page information is pretty vague. If not is there an alternative utility that allows you to compress ...
0
votes
0answers
147 views

Video compressions kill metadata / Exiftool's “-tagsfromfile” does not help

This is a long standing practical problem (more than one year). I always want to finish it, but every time I tried I failed. I have consulted several threads for it. I have a bunch of daily-life ...
0
votes
0answers
103 views

Cannot zip a mutli-part archive, zip -FF command failing with out of memory error

Command zip -FF xxx.zip --out final_xxx.zip Fails with zip error out of memory or zip error : cannot split, read, write, file is too big.may be result of bad comrpression , try -fz I do not how try ...
3
votes
1answer
141 views

How do I extract an “X1 archive”?

I have a strange file without a file extension that's supposedly an archive of the contents of a git repository. According to file (1), it's an "X1 archive": $ file my_git_repo my_git_repo: X1 ...
1
vote
2answers
614 views

compress file using /bin/gzip with particular patterns

I want to compress files individually with the particular pattern. For example, I have directory as below : /log/log1 -/1/log2/file.1 -/1/log2/file123 -/1/log2/file2.1 /log/log2 -/2/...
0
votes
2answers
262 views

Is there a fp8 (Fast PAQ) compression utility for linux?

I'm trying to compress a large archive of similar JPG images. The best compression algorithm I've found so far is fp8. In this question I show my tests and ask for better options. But fp8 is the best ...
0
votes
0answers
30 views

How to backup and compress quick rotating files to a remote server

I have a server application creating log files and rotating these files based on size. The files are time stamped at creation. The application then deletes the oldest file when X number of files in ...
0
votes
1answer
103 views

Recovering file from compressed raw data on btrfs

I was recently acquiring data from a networked instrument, saving results as text files. Unfortunately, I forgot to change the filename during one of my runs, and I managed to overwrite an important ...
3
votes
1answer
264 views

compression with btrfs-send / btrfs-receive

When I send a compressed subvol (i.e., one part of a filesystem mounted with the compress mount option) with btrfs-send, does btrfs-send send it compressed, or does it decompress it first? Similarly, ...
1
vote
1answer
523 views

Zlib Compression

I want to compress a file using Zlib (Best compression) i have googled a bit around and i have found a way to compress, but it isn't at the best compression level, I'd love to get some help OS ...
0
votes
1answer
41 views

compress a single file … zram ? cloop?

We have a legacy centos 5.11 server; filesystems are ext3, we cannot change. Is it possible to have realtime compression on a single file (a giant log file) ? zram : I cannot find it in the repos... ...
1
vote
1answer
504 views

How to create a tar.xz archive containing a directory and a hidden file in its root?

I've got a directory that contains a directory (with subdirectories and files) and a hidden (with a dot in front of its name) file. e.g: dir1\ -- dir2\... -- .hiddenFile How do I create a tar.xz ...
4
votes
1answer
955 views

How to enable btrfs zstd compression

I saw mention of zstd in the documentation (4.14 and later), so I figured to give it a try. Starting from lubuntu 17.10, I upgraded my kernel to 4.15: # uname -sr Linux 4.15.0-041500-generic ...
1
vote
0answers
101 views

Can gzip change original file size?

/dev/sdf is a 32G SD. As far as I can tell, I had performed the following, but upon attempting to perform the final step and write to a SD, my file image appears to be one byte to large for the SD. ...
2
votes
0answers
1k views

btfs: Zstd vs lzo [closed]

Linux 4.14 added support for Zstandard (or Zstd) compression. Here is a summary of the algorithm taken from Facebook's project website: Zstandard is a real-time compression algorithm, providing ...