Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [compression]

Shrinking (compressing) and restoring (decompressing) of data.

2
votes
1answer
18 views

how to set NOCOMPRESS flag on a file in btrfs?

according to https://btrfs.wiki.kernel.org/index.php/Compression#How_do_I_enable_compression.3F there exist a "NOCOMPRESS" flag, how do i set the NOCOMPRESS flag on a file?
2
votes
1answer
29 views

Easy way to change compression level of a zip file [duplicate]

On linux, is there an easy way to recompress a zip file (changing compression-level) with preserving as most meta-data as possible (timestamps etc.)?
1
vote
1answer
36 views

tar: compress text stdin and stdout [duplicate]

Using tar, is it possible to compress stdin text and have the compressed data appear in stdout? Example: $ echo "test" | tar c - <compressed blob here> I'm using macOS Mojave.
0
votes
0answers
33 views

Compress a pdf using ghostscript but without compressing some selected pages and preserving bookmarks

I have a pdf file, made with Latex, which I want to compress. I usually do that with $ gs -sDEVICE=pdfwrite -dCompatibilityLevel=1.4 -dEmbedAllFonts=true -dSubsetFonts=true -dPDFSETTINGS=/printer -...
0
votes
3answers
72 views

Linux determine ZIP size without compression

I would like to make a ZIP file using zip command. Zip file would contain large number of directories containing large numbers of subdirectories, files etc. Is there any way to calculate final size ...
9
votes
1answer
646 views

Uncompressed file estimation wrong?

I had a large (~60G) compressed file (tar.gz). I used split to break it into 4 parts and then cat to join them back together. However, now, when I am trying to estimate the size of the uncompressed ...
0
votes
1answer
35 views

Is there an efficient command for listing the header of a particular file in tar.bz2 without uncompressing/extracting?

I was working with a data.tar.bz2, which contains lots of other big files of the scale of a few hundred Gigabytes. I just wanted to see the header (first line) of a particular file file-of-interest, ...
0
votes
2answers
123 views

Compression Utility Max Files Size Limit | Unix/Linux

In Unix/Linux is there any max files size limit that a compression utility ( gzip/compress) can compress. I remember years ago it was mentioned in the gzip page that it can compress files up to 4 gb. ...
0
votes
2answers
61 views

Does tar.gz do effective file compression of large files?

I wanted to compress a ~29GB file significantly, and used the tar command in the terminal like this : tar cvzf file.tar.gz /path/to/directory and it compressed it to ~26GB, so I looked up maximum ...
-1
votes
1answer
82 views

How to split a compressed file?

I have a 44GB 7z compressed file that I compressed with lzma2 and it took around 11 hours (original file is a text file of 285GB). 7z a -t7z Files.7z -m0=lzma2 -mx=9 -aoa So, what I want is to do is ...
-2
votes
1answer
40 views

compress specific folder(directory) from folder with tar [closed]

I am trying to use tar to compress the subdirectories /root/folder/folder{1..5} into a single compressed archive test1.tar.bz2. I'm using the following command: tar -cvjf test1.tar.bz2 root/folder/...
0
votes
1answer
24 views

Arguments for, and the applicability of, using multiple forms of compression on one or more files?

I was wondering whether there are any scenarios which require file(s) to be compressed multiple times with the same or a different algorithm. As far as i can tell the file size can be reduced further ...
3
votes
1answer
684 views

cat on big files does not work

I'm trying to concatenate four big files in two. The files *_1P.gz contain the same amount of lines as the corrisponding *_2P.gz. The files A_1P.gz and A_2P.gz both contain 1104507560 lines. The ...
7
votes
1answer
450 views

Permission changes after extracting tgz file

While extracting a tgz file, I have noticed that the permission changes to something weird! The tgz file belongs to root:root however, the folder belongs to 502:games [root@rocks7 common]# ls -l -rw-...
3
votes
1answer
23 views

Create separate 7z archives for each directory in the current directory and additionally parallelize through GNU Parallel

I have about two hundred sub-directories located within a directory of interest: $ ls backup 201302 201607 201608 201609 201610 201701 201702 201705 201801 201802 I want to create a 7z archive xyz....
2
votes
1answer
50 views

Bash script for uncompressing multiple files at a once

I have a set of files: lium3@l128-37:/DATA/fr94/01 $ ls -l total 7784 -rw-r--r-- 1 lium3 mcm1 395575 May 2 1996 fr940104.0z -rw-r--r-- 1 lium3 mcm1 113815 May 2 1996 fr940104.1z -rw-r--r-- 1 ...
3
votes
4answers
471 views

Best compression tool for full disk imaging that can saturate a USB 3 connection on low-powered machines?

I often work on full disk images, I often need to read and write full disk images on not so capable laptops using a USB-3 disk as my temporary medium. Duplicating a raw image is probably the fastest ...
0
votes
1answer
1k views

Unzipping a .gz file to get a text file but getting a binary file?

I have a .gz file called document.txt.gz file. It is supposed to be a text file which got compressed to a gzip file. When I uncompress, I am getting a corrupt file. I ran the following commands: $ ...
2
votes
1answer
127 views

kernel module: set executable bit? compress with xz?

I wrote/tweaked a custom kernel module and installed it. It works as expected, but I've noticed that other kernel modules on my system are compressed with xz and have 0444 permissions, whereas I did ...
1
vote
2answers
108 views

Is it possible to change the zlib-flate compression level?

I use the zlib-flate compression utility. Is it possible to compress at different levels. The man page information is pretty vague. If not is there an alternative utility that allows you to compress ...
0
votes
0answers
80 views

Video compressions kill metadata / Exiftool's “-tagsfromfile” does not help

This is a long standing practical problem (more than one year). I always want to finish it, but every time I tried I failed. I have consulted several threads for it. I have a bunch of daily-life ...
0
votes
0answers
65 views

Cannot zip a mutli-part archive, zip -FF command failing with out of memory error

Command zip -FF xxx.zip --out final_xxx.zip Fails with zip error out of memory or zip error : cannot split, read, write, file is too big.may be result of bad comrpression , try -fz I do not how try ...
3
votes
1answer
100 views

How do I extract an “X1 archive”?

I have a strange file without a file extension that's supposedly an archive of the contents of a git repository. According to file (1), it's an "X1 archive": $ file my_git_repo my_git_repo: X1 ...
1
vote
2answers
286 views

compress file using /bin/gzip with particular patterns

I want to compress files individually with the particular pattern. For example, I have directory as below : /log/log1 -/1/log2/file.1 -/1/log2/file123 -/1/log2/file2.1 /log/log2 -/2/...
0
votes
0answers
63 views

Inotifywait not detecting Wordpress upload but does if sftp upload instead

I have inotifywait successfully running as -m started from my user directory using a bash script. My user is a member of www-data. Here is the script: #!/bin/bash inotifywait -m -r --exclude '.*\.tmp'...
0
votes
2answers
144 views

Is there a fp8 (Fast PAQ) compression utility for linux?

I'm trying to compress a large archive of similar JPG images. The best compression algorithm I've found so far is fp8. In this question I show my tests and ask for better options. But fp8 is the best ...
0
votes
0answers
24 views

How to backup and compress quick rotating files to a remote server

I have a server application creating log files and rotating these files based on size. The files are time stamped at creation. The application then deletes the oldest file when X number of files in ...
0
votes
1answer
83 views

Recovering file from compressed raw data on btrfs

I was recently acquiring data from a networked instrument, saving results as text files. Unfortunately, I forgot to change the filename during one of my runs, and I managed to overwrite an important ...
2
votes
1answer
152 views

compression with btrfs-send / btrfs-receive

When I send a compressed subvol (i.e., one part of a filesystem mounted with the compress mount option) with btrfs-send, does btrfs-send send it compressed, or does it decompress it first? Similarly, ...
1
vote
1answer
302 views

Zlib Compression

I want to compress a file using Zlib (Best compression) i have googled a bit around and i have found a way to compress, but it isn't at the best compression level, I'd love to get some help OS ...
0
votes
1answer
30 views

compress a single file … zram ? cloop?

We have a legacy centos 5.11 server; filesystems are ext3, we cannot change. Is it possible to have realtime compression on a single file (a giant log file) ? zram : I cannot find it in the repos... ...
1
vote
1answer
326 views

How to create a tar.xz archive containing a directory and a hidden file in its root?

I've got a directory that contains a directory (with subdirectories and files) and a hidden (with a dot in front of its name) file. e.g: dir1\ -- dir2\... -- .hiddenFile How do I create a tar.xz ...
4
votes
1answer
742 views

How to enable btrfs zstd compression

I saw mention of zstd in the documentation (4.14 and later), so I figured to give it a try. Starting from lubuntu 17.10, I upgraded my kernel to 4.15: # uname -sr Linux 4.15.0-041500-generic ...
1
vote
0answers
88 views

Can gzip change original file size?

/dev/sdf is a 32G SD. As far as I can tell, I had performed the following, but upon attempting to perform the final step and write to a SD, my file image appears to be one byte to large for the SD. ...
1
vote
0answers
1k views

btfs: Zstd vs lzo [closed]

Linux 4.14 added support for Zstandard (or Zstd) compression. Here is a summary of the algorithm taken from Facebook's project website: Zstandard is a real-time compression algorithm, providing ...
1
vote
2answers
2k views

how to reduce the size of a pdf file from command line

I know this question has been answered in other threads, but none of the solutions worked for me. I have a pdf file which contains 4 plots with millions of colourful dots (Manhattan plots). The pdf ...
0
votes
1answer
298 views

How to compress all files from all subfolders if there is no `Archive.zip` in subfolder?

Rawinput: ➜ datatest tree . ├── a │   ├── README.md │   ├── code │   └── data │   └── apple.csv ├── archive.sh ├── f │   ├── README.md │   ├── code │   └── data │   ├── Archive.zip │   ├─...
5
votes
3answers
408 views

How to compress all files from several subfolders?

Raw input: ➜ test tree . ├── f1.md ├── f2.md ├── f3.md ├── f4.txt ├── f5.csv ├── f6.doc ├── s1 │   ├── code │   └── data │   ├── f1.md │   ...
0
votes
1answer
780 views

What invocation will use LZMA to compress tar output on my Mac (OSX 10.12.x)?

I've tried several invocations of the OEM tar to create LZMA-compressed tarballs. More specifically, I tried: tar -c -f --lzma Windows\ 7.vmwarevm.tar.lzma Windows\ 7.vmwarevm My efforts created an ...
1
vote
1answer
552 views

Selectively extract files from a .zip archive from command line

Suppose I have a large zip file (>50GB) and I want to extract some files off of it from the command line. To get the files I run the command: unzip -l myfile.zip | grep "foo" which gives me a list ...
2
votes
2answers
412 views

Best way to compress a folder of small pre-compressed files (images)?

I have a folder containing approximate 50GB of images and video files (~35k files) that I'd like to compress into a single file for storage along side a database. Are there any compression options (or ...
1
vote
1answer
74 views

Weird compression question

I'm trying to compress some raw sensor data from older recordings that I need and don't want to remove from my Centos server. The data recorded is in a proprietary format, but for all intents and ...
21
votes
5answers
3k views

On-the-fly stream compression that doesn't spill over into hardware resources?

I have 200 GB free disk space, 16 GB of RAM (of which ~1 GB is occupied by the desktop and kernel) and 6 GB of swap. I have a 240 GB external SSD, with 70 GB used1 and the rest free, which I need to ...
1
vote
0answers
353 views

Keep metadata or edit metadata with tar

I have a tarball. This tarball contains—of course—various files and directories. When I extract the tar, some of these files already exist, while others don't. What I would like to know is this: how ...
0
votes
2answers
632 views

Tar --absolute-names flag does not work

I would like to put everything into an archive from a given directory, while also saving the whole dierctory layout, so superior directory names would be required. I run the following command: tar --...
3
votes
2answers
418 views

How can I repack zip to 7zip?

How can I safely and quickly repack a zip archive to a 7zip archive? I checked options of 7zip program but I don't see any option to repack. I know I can extract to directory and then pack this ...
1
vote
1answer
190 views

Compressing Directory tree into multiple tar archives into a single xz

I'm trying to compress a large rootdir which contains many subdir_i the folder tree looks like: ./rootdir ./rootdir/subdir_1 ./rootdir/subdir_2 ... I am looking to output this to a single compressed ...
3
votes
1answer
92 views

Compression program showing live compression ratio

Some compression programs can show information (like compression ratio or time and size totals) while performing the task, like xz -v: --- % 2,580.2 KiB / 6,552.0 KiB = 0.394 1.2 MiB/s 0:05 ...
0
votes
2answers
102 views

What does TGZ mean?

I can recall my professor saying it means "Tar-able Gzip." I'm searching it in Google and couldn't find it. I know what Tar and Gzip are. I also know what a TGZ is, but I want to know the meaning of ...
1
vote
1answer
63 views

How can I compress files with the output name same as parent folder?

How can I compress files with the output name same as parent folder? For example, I have a the folders: ubup@neter:/NeuroNER-master/output$ tree . ├── en_2017-04-28_20-26-59-439552 │   ├── 000_test....