Questions tagged [compression]

Shrinking (compressing) and restoring (decompressing) of data.

Filter by
Sorted by
Tagged with
1
vote
1answer
298 views

bzip2 decompression, changes and compression of a text file through the pipeline

I am searching for a way to decompress a bzip2 compressed text file, change the content (adding new content, sorting etc.) and compress it again through pipelines. I already found a way to do this ...
22
votes
4answers
9k views

How to decompress jsonlz4 files (Firefox bookmark backups) using the command line?

There seems to be various JavaScript+browser specific ways of decompressing this, but isn't there some way to transform jsonlz4 files to something unlz4 will read?
1
vote
0answers
35 views

xzcmp does not work for some files. (Internal pipe error 141?)

I am using xzcmp to compare several XZ files. But I have a strange problem: For some files (*), xzcmp doesn't work and returns exit code 2, without error message. zxcmp version1.sql.xz version2.sql....
7
votes
2answers
11k views

7zip, xz, gzip, tar, etc — what should I use? [closed]

What would you recommend me to use: 7zip, xz, gzip, tar, etc which is more or less available on the most linux distributives and what does compression the best?
3
votes
1answer
2k views

Is there any way to check the compression ratio of a xz compressed file?

In windows by default one can see the compression ratio of zip files and after installing WinRAR it also supports this feature but currently I am using a linux and have compressed a file using lzma2 ...
4
votes
1answer
2k views

Is there any way to verify the integrity of a xz archive without decompressing the entire archive?

Can I check the integrity of the archive without decompressing it as the archive is 64GB. It is a backup of my old laptop and before i format my old laptop I wanted to verify the backup but it will ...
2
votes
1answer
2k views

A generic command to extract archive files [duplicate]

There are many archive formats: tar, tar.gz, tar.bz2, zip, rar, tar.xz and more. I would like to have a single command that would determine the type of archive I give to it as input and extract the ...
1
vote
1answer
789 views

Is safe to use rsync -z option for database sync (Postgresql binary replication)?

I see that this option is good for slow connections, because it enables the compression. But what this really means? Will this option compress the data only during the transfer? Or the files in the ...
0
votes
2answers
2k views

Compress ls result using tar

I want to compress log files by specific months. Listing all files with following commands ll | grep 'Sep' | grep 'oim_server' Then I need to compress with tar result of it How can I do it?
20
votes
2answers
3k views

Zipped hard drive image very big

I just made an image of a freshly installed dual boot (Ubuntu and Windows) using this command (which I've been using for a while for smaller images): dd if=/dev/sda | gzip > /mnt/drive.img.gz On ...
3
votes
3answers
3k views

Tool to compress/decompress STDIN

Is there a tool that compresses STDIN, and outputs to STDOUT? This tool (or its counterpart) should be able to decompress as well. Something simple is fine, but it must be lossless.
4
votes
1answer
5k views

How to load compressed kernel modules in Ubuntu?

I would like to use LZMA-compressed kernel modules on my system. Unfortunately Canoncial leaves that feature disabled both in kernel and user-space tools. Here's what I did so far: Compile and ...
2
votes
1answer
103 views

Putting m files in a directory into n tar files by size?

Does anyone know if it's possible to create n tar files (of roughly equal size) out of a larger collection of files in a directory, in such a way that they can be extracted individually? I was ...
1
vote
1answer
124 views

Get compressed size of compressed file with redirect to /dev/null?

I have a compressed file, and on compression or decompression, I want to get it's final compressed size (for compression) or original compressed size (for decompression). I have used wc -c before: $...
1
vote
0answers
137 views

Enable cuda support in libbsc

I am interested in using libbsc in CUDA support. libbsc is basically used for compression in GPU (as well as CPU). As far as I tested, it is running in CPU (openmp) not in NVIDIA GPU (CUDA driver ...
0
votes
1answer
1k views

Does Redhat and CentOS come with zlib preinstalled? [closed]

Does Redhat and CentOS come with zlib preinstalled? If yes, which versions have it? // need to know if I can expect users to have it on their Servers, since I can't install it there, and can't ask ...
9
votes
3answers
8k views

why should I use tar.xz instead of tar.gz? xz is a lossless data compression program and file format [duplicate]

Today first time in my life I saw tar.xz download. I searched the internet and found Wikipedia articles (xz and XZ Utils) Interesting quote about the users of xz xz has gained notability for ...
2
votes
1answer
8k views

Proper way to create a compressed, rsyncable mirror of a directory?

I would like to create a compressed mirror of some directories in such a way that it can be rsynced again, both from the original destination and to a remote destination. I can do it inefficiently in ...
0
votes
1answer
742 views

Reduce a size of a file(.jar) in unix

I want to reduce the big file (*.jar) in unix. i have tried with tar,bzip,gzip and zip commands. but all are same result not compressed much. Could any one help on this ???
0
votes
0answers
359 views

How to deal with corrupted .tar.lrz file

I have a file file.tar.lrz and tried to unzip it via lrzuntar file.tar.lrz but it gives an error: "Not an lrzip file" with lrunzip. I also tried to open it with file roller. file file.tar.lrz says ...
5
votes
2answers
5k views

How can I decompress and print the last few lines of a compressed text file?

I have 6 gzipped text files, each of which is ~17G when compressed. I need to see the last few lines (decompressed) of each file to check whether a particular problem is there. The obvious approach is ...
1
vote
1answer
408 views

How to compress all files in folder and erase the untared versions via command line?

I have a backup disk containing uncompressed versions of my files that I would like to compress to save space on this disk. The disk (containing files to compress) is full (about 3TB with 17GB free). ...
2
votes
3answers
830 views

Work with compressed dd images

How can I compress dd images and work with them at the same time? On Windows, I can mark it as compressed and work with it like a normal file. I can't re-partition, but I can mount some drives (For ...
1
vote
1answer
1k views

On the fly compression of read- and writeable folder (like Windows EFS)?

Is there a possibility to compress the contents of a directory in Unix (especially AIX) on the fly, while still being able to read and write in the directory? Like the Windows EFS encryption does?
0
votes
1answer
614 views

Generating word list base on possible numbers/characters via Crunch in Kali Linux

I'm new to crunch. I'm trying to generate a list containing 8 character long words. I've tried crunch 8 8 > pw.txt It freezes my computer because it doesn't have enough space for that. Is ...
9
votes
2answers
4k views

Print archive file list instantly (without decompressing entire archive)

A problem with .tar.gz archives is that, when I try to just list an archive's content, the computer actually decompresses it, which would take a very long time if the file is large. Other file ...
2
votes
1answer
544 views

Compressing a folder but do not compress specific file types but include them in the gz file

I am using tar with pigz to compress a folder and save a backup. Size of this folder ~250 GB or more. This folder has variety of content including numerous text and log files, ISOs and zip files in ...
2
votes
2answers
493 views

tar while avoiding compression of certain file types

I'm trying to create a very large backup of an entire hard disk, using tar. However, there are many files such as CR2 (Canon RAW 2), JPG and a few other files which are already compressed. I have ...
3
votes
1answer
697 views

Corrupted gz file

I have a MySQL database backup in a gz file. When trying to uncompress it I get the following: gzip: db_stepup.sql.gz: not in gzip format I read that sometimes is just a matter of removing the gz ...
0
votes
2answers
337 views

NTFS compression and RSync

I currently have a hard drive that I backup using RSync. I do weekly; monthly snapshots. I would like to compress some of my hard drive using NTFS compression. How would that affect my future ...
1
vote
0answers
41 views

Where is my 7za data being written to

I have a weird question about 7za on Centos 7. I am adding a file to a 7za archive using the following command (the total size of the directory is around 40GB) /usr/bin/7za a -ppassword -t7z -mmt=off ...
5
votes
4answers
6k views

Compress and remove original directory with limited storage

On my 3 TB external hard drive, I have a 2.7 TB directory containing relatively small files. I would like to compress this 2.7 TB directory and remove it to keep only the compressed version. The issue ...
0
votes
1answer
410 views

how to compress all folders in a directory

I have a directory full of folders and I was wondering if there is any command to compress all the folders in a directory X with date in the name of the file and not update old ones in case they ...
101
votes
8answers
18k views

Why would I tar a single file?

At my company, we download a local development database snapshot as a db.dump.tar.gz file. The compression makes sense, but the tarball only contains a single file (db.dump). Is there any point to ...
1
vote
1answer
1k views

gzip - is there a risk to lose files during compression if process dead?

Is there a risk of losing files during compression if process dies? I'm gzipping big files using wildcard, but accidentally running it without screen or nohup. Is there a risk to lose source files if ...
3
votes
1answer
709 views

GNU sort --compress-program compressing only first temporary

I am sorting big files (>100Go), and to reduce time spent on disk writes, I am trying to use GNU sort's --compress-program parameter. (Related: How to sort big files?) However, it appears in certain ...
10
votes
3answers
12k views

How do I reduce the size of a pdf file that contains images?

I have a pdf file that contains images and I want to reduce its size in order to upload it to a site with a size limit. So, how can I reduce the size of a pdf file from the command-line?
0
votes
1answer
1k views

Linux Gzip increasing size

I try to gzip a file abc.log which has size of 111 bytes, but after gzip, the size of the file increased to 125 bytes, why is that? Is it when i perform gzip, it will create header and trailer that ...
6
votes
3answers
10k views

Linux + how to compress the old log files automatically

Under /var/log/httpd folder, files are created with huge capacity!!! What solution can I use in order to compress the old files automatically (can logrotate handle this)? If yes, how to configure ...
5
votes
3answers
11k views

Unable to use tar to run an archive through bzip2 compression

I am having difficulty compression an archive with bzip using tar. The archive I have is user-logs.tar. I am attempting to run this: tar -jf user-logs.tar I get this error: tar: You must ...
2
votes
1answer
283 views

linux unbuffered compression options

I'm working on a embedded system and due to persistent memory restrictions, I need to compress a log file "on the fly". My objective is to have a one liner inside a script, which would look like this:...
2
votes
2answers
756 views

Squashfs minimal compression?

I'd like to create a squashfs image of a directory, but it's a huge one and even minimal compression would be a big win. Is there a way to do something like mksquashfs -1?
4
votes
1answer
836 views

Is there a compression method that supports solid compression and also adding data to the compressed file?

I have a large compressed .tar.xz file containing log files. The compression ratio is very good - but it takes a long time to compress, and if I want to add additional log files to it, I have to ...
1
vote
0answers
745 views

On-the-fly decompression

I have 44 TB worth of gzipped files that are 1.5 GB each when compressed or 2.0 GB when uncompressed. I have a tool that can read only uncompressed files. I would like to avoid the overhead of ...
1
vote
1answer
1k views

log compression using logrotate

I was trying to set compression for jboss log files. The log files I want to compress are console.log and server.log. Compression for console.log is working fine, but server.log I am seeing issues. I ...
0
votes
2answers
560 views

Tar big files in different directories

I have ~20 files, each about 10 GB in size, in different folders that I need to compress into a single tar. It is not practical to copy them all to one location. More precisely, the files are ...
0
votes
1answer
4k views

Compressing files using p7zip from the command-line

I would like to compress four ISO files using 7z into a new archive called ISOs.7z from the command-line on my Sabayon machine. These are the commands I have tried so far (I know none of these specify ...
0
votes
1answer
69 views

Verify files AFTER extracting gzip

I gzip a folder, transfer it to a remote location and then extract it to a folder. The problem is that this machine is a little unreliable and sometimes writes don't succeed and there can be zero-byte ...
3
votes
2answers
578 views

Extract compressed file by it's header as it is piped from stdout (locally or from remote location)?

I am sending compressed file with piping either local or from a network location. And on the receiving end, I would like to detect the type of compression and use the appropriate decompression utility ...
1
vote
1answer
2k views

How to convert tar.bz2 to .bz2? [closed]

I have a file with compression tar.bz2 , I would like to convert it to .bz2 . Is there a command that does this in one shot? I tried the below but no joy ! tar -xvjf US.tar.bz2 US bzip2: ...