Questions tagged [compression]

Shrinking (compressing) and restoring (decompressing) of data.

Filter by
Sorted by
Tagged with
0
votes
1answer
43 views

compress a single file … zram ? cloop?

We have a legacy centos 5.11 server; filesystems are ext3, we cannot change. Is it possible to have realtime compression on a single file (a giant log file) ? zram : I cannot find it in the repos... ...
1
vote
1answer
551 views

How to create a tar.xz archive containing a directory and a hidden file in its root?

I've got a directory that contains a directory (with subdirectories and files) and a hidden (with a dot in front of its name) file. e.g: dir1\ -- dir2\... -- .hiddenFile How do I create a tar.xz ...
3
votes
1answer
1k views

How to enable btrfs zstd compression

I saw mention of zstd in the documentation (4.14 and later), so I figured to give it a try. Starting from lubuntu 17.10, I upgraded my kernel to 4.15: # uname -sr Linux 4.15.0-041500-generic ...
1
vote
0answers
104 views

Can gzip change original file size?

/dev/sdf is a 32G SD. As far as I can tell, I had performed the following, but upon attempting to perform the final step and write to a SD, my file image appears to be one byte to large for the SD. ...
2
votes
0answers
2k views

btfs: Zstd vs lzo [closed]

Linux 4.14 added support for Zstandard (or Zstd) compression. Here is a summary of the algorithm taken from Facebook's project website: Zstandard is a real-time compression algorithm, providing ...
1
vote
2answers
3k views

how to reduce the size of a pdf file from command line

I know this question has been answered in other threads, but none of the solutions worked for me. I have a pdf file which contains 4 plots with millions of colourful dots (Manhattan plots). The pdf ...
0
votes
1answer
358 views

How to compress all files from all subfolders if there is no `Archive.zip` in subfolder?

Rawinput: ➜ datatest tree . ├── a │   ├── README.md │   ├── code │   └── data │   └── apple.csv ├── archive.sh ├── f │   ├── README.md │   ├── code │   └── data │   ├── Archive.zip │   ├─...
5
votes
3answers
545 views

How to compress all files from several subfolders?

Raw input: ➜ test tree . ├── f1.md ├── f2.md ├── f3.md ├── f4.txt ├── f5.csv ├── f6.doc ├── s1 │   ├── code │   └── data │   ├── f1.md │   ...
1
vote
2answers
2k views

What invocation will use LZMA to compress tar output on my Mac (OSX 10.12.x)?

I've tried several invocations of the OEM tar to create LZMA-compressed tarballs. More specifically, I tried: tar -c -f --lzma Windows\ 7.vmwarevm.tar.lzma Windows\ 7.vmwarevm My efforts created an ...
1
vote
1answer
1k views

Selectively extract files from a .zip archive from command line

Suppose I have a large zip file (>50GB) and I want to extract some files off of it from the command line. To get the files I run the command: unzip -l myfile.zip | grep "foo" which gives me a list ...
2
votes
2answers
562 views

Best way to compress a folder of small pre-compressed files (images)?

I have a folder containing approximate 50GB of images and video files (~35k files) that I'd like to compress into a single file for storage along side a database. Are there any compression options (or ...
1
vote
1answer
76 views

Weird compression question

I'm trying to compress some raw sensor data from older recordings that I need and don't want to remove from my Centos server. The data recorded is in a proprietary format, but for all intents and ...
23
votes
5answers
4k views

On-the-fly stream compression that doesn't spill over into hardware resources?

I have 200 GB free disk space, 16 GB of RAM (of which ~1 GB is occupied by the desktop and kernel) and 6 GB of swap. I have a 240 GB external SSD, with 70 GB used1 and the rest free, which I need to ...
1
vote
0answers
530 views

Keep metadata or edit metadata with tar

I have a tarball. This tarball contains—of course—various files and directories. When I extract the tar, some of these files already exist, while others don't. What I would like to know is this: how ...
0
votes
2answers
1k views

Tar --absolute-names flag does not work

I would like to put everything into an archive from a given directory, while also saving the whole dierctory layout, so superior directory names would be required. I run the following command: tar --...
3
votes
2answers
802 views

How can I repack zip to 7zip?

How can I safely and quickly repack a zip archive to a 7zip archive? I checked options of 7zip program but I don't see any option to repack. I know I can extract to directory and then pack this ...
1
vote
1answer
237 views

Compressing Directory tree into multiple tar archives into a single xz

I'm trying to compress a large rootdir which contains many subdir_i the folder tree looks like: ./rootdir ./rootdir/subdir_1 ./rootdir/subdir_2 ... I am looking to output this to a single compressed ...
3
votes
1answer
132 views

Compression program showing live compression ratio

Some compression programs can show information (like compression ratio or time and size totals) while performing the task, like xz -v: --- % 2,580.2 KiB / 6,552.0 KiB = 0.394 1.2 MiB/s 0:05 ...
0
votes
2answers
155 views

What does TGZ mean?

I can recall my professor saying it means "Tar-able Gzip." I'm searching it in Google and couldn't find it. I know what Tar and Gzip are. I also know what a TGZ is, but I want to know the meaning of ...
1
vote
1answer
82 views

How can I compress files with the output name same as parent folder?

How can I compress files with the output name same as parent folder? For example, I have a the folders: ubup@neter:/NeuroNER-master/output$ tree . ├── en_2017-04-28_20-26-59-439552 │   ├── 000_test....
1
vote
1answer
202 views

Can I recreate a gzip file exactly, given the original uncompressed file?

I have a series of gzip files which I wish to store more efficiently using xz, without losing traceability to a set of checksums of the gzip files. I believe this amounts to being able to recreate ...
1
vote
2answers
975 views

Is it possible to create a zip file without using the zip binary?

Basically I am working on a shell script in Cygwin which requires me to zip a couple of files (it must be a zip can't be gzip). The server I'm working on does not have the zip binary installed and I ...
3
votes
1answer
411 views

Can ncompress compress a file to 99.99% rate?

I'm using the old ncompress to create a backup file. And what I see is amazing: [root@centos6 home]# ll -l -b mytest* -rw-r--r--. 1 root root 1073741824 Mar 8 13:41 mytest.iso -rw-r--r--. 1 root ...
0
votes
1answer
485 views

Adding compression to .tar file?

I'm doing a backup scheme wherein we create an uncompressed .tar file, add any new or changed files to it for a week or two. Now, at the end of this, is there an easy way to add compression to the ....
1
vote
1answer
773 views

compressed swapfile (without zram or zswap)

Is it possible to have a compressed swap file without using zram or zswap? Actually, I've successfully mounted a swap file over sftp with sshfs with these commands and I'd like to use compression ...
0
votes
3answers
375 views

how come my backup script fails to build the tar.gz file and send it over scp?

I'm trying to create a script to automate MySQL backups and transfer the generated file to another server. I managed to create the dump file, but haven't made much progress from that point. The ...
2
votes
1answer
326 views

Configuring Apache to read svgz file type

I recently migrated a site to a new server. I'm now getting this error from Firefox when attempting to load an .svgz file: XML Parsing Error: not well-formed The svg rendered fine on the old server, ...
0
votes
2answers
658 views

Is it safe to do recursive compression with tar, gzip and pigz?

If I make a .tar.gz via tar czvf - ./myfiles/ | pigz -9 -p 16 > ./mybackup.tar.gz, Can I safely unzip an already gzip'd file ./myfiles/an_old_backup.tar.gz within the ./myfiles directory via ...
0
votes
2answers
2k views

Script to Tar file in a specific Directory

I'm trying to write a Unix script that will go to a specific directory and tar.gz all the log files that are older than 2 days. So far I have this but it doesn't work. Any help would be appreciated. ...
0
votes
0answers
371 views

Why is copying many files with Caja much slower than compressing+copying+uncompressing?

I am currently working with HASYv2 dataset which contains the hasy-data directory with 168.233 images of size 32px x 32px. I just wanted to copy that directory with Caja, but I skipped this after ...
2
votes
1answer
406 views

Is there a universal command-line archive extraction utility for Linux? [duplicate]

Extracting the contents of a (possibly compressed) archive file is a task I need to perform very often. Depending on what method of compression and archive format the author of an archive file chose, ...
1
vote
1answer
81 views

How can I create a compressed archive consisting of the largest files from multiple directories?

I want to write a script which will create a single compressed-archive file consisting of the largest files above a certain threshold from multiple directories. For example, I would like to know how ...
2
votes
0answers
161 views

How to extract and stitch together a multipart tar arcvhive without the initial file

I have a large folder with multi part tar files each 2gb in size, file.partaa through file.partgy, however I am missing the initial tar file. Is there any way that I can extract each part ...
8
votes
4answers
2k views

How to add a huge file to an archive and delete it in parallel

Say I have a file of 80GB /root/bigfile on a 100GB system and want to put this file in a archive /root/bigarchive.tar I obviously need to delete this file at the same time that it is added in the ...
1
vote
0answers
122 views

password protection for logrotate compression

Is password protection possible with logrotate while compressing in Linux? If yes would someone let me know how? /root/NewLog { missingok notifempty compress size 20k daily ...
1
vote
0answers
28 views

Unexpected less gzip compression with -9 [duplicate]

I do have a USB mass storage device of 4GB, in which I had created two partition, both are of ext4. First partition is 1.5G and another one is 2.5G First partition is almost full while second one is ...
7
votes
3answers
1k views

mass decompress gzip files without gz extension

I have a massive number of files with extensions like .0_1234 .0_4213 and .0_4132 etc. Some of these are gzip compressed and some are raw email. I need to determine which are compressed files, ...
0
votes
1answer
55 views

Logging to a new file for a continues running script

I have a script providing login continuously( like 2-3 pages in a second). I was using below one liner. sh myscript.sh > /home/user/mylog I am running another script to zip this file when it ...
1
vote
1answer
233 views

bzip2 decompression, changes and compression of a text file through the pipeline

I am searching for a way to decompress a bzip2 compressed text file, change the content (adding new content, sorting etc.) and compress it again through pipelines. I already found a way to do this ...
19
votes
4answers
8k views

How to decompress jsonlz4 files (Firefox bookmark backups) using the command line?

There seems to be various JavaScript+browser specific ways of decompressing this, but isn't there some way to transform jsonlz4 files to something unlz4 will read?
1
vote
0answers
33 views

xzcmp does not work for some files. (Internal pipe error 141?)

I am using xzcmp to compare several XZ files. But I have a strange problem: For some files (*), xzcmp doesn't work and returns exit code 2, without error message. zxcmp version1.sql.xz version2.sql....
4
votes
2answers
8k views

7zip, xz, gzip, tar, etc — what should I use? [closed]

What would you recommend me to use: 7zip, xz, gzip, tar, etc which is more or less available on the most linux distributives and what does compression the best?
3
votes
1answer
2k views

Is there any way to check the compression ratio of a xz compressed file?

In windows by default one can see the compression ratio of zip files and after installing WinRAR it also supports this feature but currently I am using a linux and have compressed a file using lzma2 ...
4
votes
1answer
1k views

Is there any way to verify the integrity of a xz archive without decompressing the entire archive?

Can I check the integrity of the archive without decompressing it as the archive is 64GB. It is a backup of my old laptop and before i format my old laptop I wanted to verify the backup but it will ...
1
vote
1answer
2k views

A generic command to extract archive files [duplicate]

There are many archive formats: tar, tar.gz, tar.bz2, zip, rar, tar.xz and more. I would like to have a single command that would determine the type of archive I give to it as input and extract the ...
1
vote
1answer
673 views

Is safe to use rsync -z option for database sync (Postgresql binary replication)?

I see that this option is good for slow connections, because it enables the compression. But what this really means? Will this option compress the data only during the transfer? Or the files in the ...
0
votes
2answers
2k views

Compress ls result using tar

I want to compress log files by specific months. Listing all files with following commands ll | grep 'Sep' | grep 'oim_server' Then I need to compress with tar result of it How can I do it?
20
votes
2answers
3k views

Zipped hard drive image very big

I just made an image of a freshly installed dual boot (Ubuntu and Windows) using this command (which I've been using for a while for smaller images): dd if=/dev/sda | gzip > /mnt/drive.img.gz On ...
3
votes
3answers
3k views

Tool to compress/decompress STDIN

Is there a tool that compresses STDIN, and outputs to STDOUT? This tool (or its counterpart) should be able to decompress as well. Something simple is fine, but it must be lossless.
4
votes
1answer
4k views

How to load compressed kernel modules in Ubuntu?

I would like to use LZMA-compressed kernel modules on my system. Unfortunately Canoncial leaves that feature disabled both in kernel and user-space tools. Here's what I did so far: Compile and ...