Shrinking (compressing) and restoring (decompressing) of data.

learn more… | top users | synonyms

0
votes
0answers
16 views

Where is my 7za data being written to

I have a weird question about 7za on Centos 7. I am adding a file to a 7za archive using the following command (the total size of the directory is around 40GB) /usr/bin/7za a -ppassword -t7z -mmt=off ...
0
votes
3answers
33 views

Compress and remove original directory with limited storage

On my 3 TB external hard drive, I have a 2.7 TB directory containing relatively small files. I would like to compress this 2.7 TB directory and remove it to keep only the compressed version. The issue ...
0
votes
1answer
35 views

how to compress all folders in a directory

I have a directory full of folders and I was wondering if there is any command to compress all the folders in a directory X with date in the name of the file and not update old ones in case they ...
90
votes
8answers
9k views

Why would I tar a single file?

At my company, we download a local development database snapshot as a db.dump.tar.gz file. The compression makes sense, but the tarball only contains a single file (db.dump). Is there any point to ...
0
votes
1answer
30 views

gzip - is there a risk to lose files during compression if process dead?

Is there a risk of losing files during compression if process dies? I'm gzipping big files using wildcard, but accidentally running it without screen or nohup. Is there a risk to lose source files if ...
3
votes
1answer
52 views

GNU sort --compress-program compressing only first temporary

I am sorting big files (>100Go), and to reduce time spent on disk writes, I am trying to use GNU sort's --compress-program parameter. (Related: How to sort big files?) However, it appears in certain ...
1
vote
1answer
38 views

How do I reduce the size of a pdf file that contains images?

I have a pdf file that contains images and I want to reduce its size in order to upload it to a site with a size limit. So, how can I reduce the size of a pdf file from the command-line?
0
votes
1answer
43 views

Linux Gzip increasing size

I try to gzip a file abc.log which has size of 111 bytes, but after gzip, the size of the file increased to 125 bytes, why is that? Is it when i perform gzip, it will create header and trailer that ...
2
votes
2answers
66 views

Linux + how to compress the old log files automatically

Under /var/log/httpd folder, files are created with huge capacity!!! What solution can I use in order to compress the old files automatically (can logrotate handle this)? If yes, how to configure ...
2
votes
3answers
56 views

Unable to use tar to run an archive through bzip2 compression

I am having difficulty compression an archive with bzip using tar. The archive I have is user-logs.tar. I am attempting to run this: tar -jf user-logs.tar I get this error: tar: You must ...
1
vote
1answer
62 views

linux unbuffered compression options

I'm working on a embedded system and due to persistent memory restrictions, I need to compress a log file "on the fly". My objective is to have a one liner inside a script, which would look like ...
1
vote
2answers
37 views

Squashfs minimal compression?

I'd like to create a squashfs image of a directory, but it's a huge one and even minimal compression would be a big win. Is there a way to do something like mksquashfs -1?
2
votes
1answer
33 views

Is there a compression method that supports solid compression and also adding data to the compressed file?

I have a large compressed .tar.xz file containing log files. The compression ratio is very good - but it takes a long time to compress, and if I want to add additional log files to it, I have to ...
1
vote
0answers
24 views

On-the-fly decompression

I have 44 TB worth of gzipped files that are 1.5 GB each when compressed or 2.0 GB when uncompressed. I have a tool that can read only uncompressed files. I would like to avoid the overhead of ...
1
vote
1answer
112 views

log compression using logrotate

I was trying to set compression for jboss log files. The log files I want to compress are console.log and server.log. Compression for console.log is working fine, but server.log I am seeing issues. I ...
0
votes
2answers
76 views

Tar big files in different directories

I have ~20 files, each about 10 GB in size, in different folders that I need to compress into a single tar. It is not practical to copy them all to one location. More precisely, the files are ...
0
votes
1answer
90 views

Compressing files using p7zip from the command-line

I would like to compress four ISO files using 7z into a new archive called ISOs.7z from the command-line on my Sabayon machine. These are the commands I have tried so far (I know none of these specify ...
0
votes
1answer
32 views

Verify files AFTER extracting gzip

I gzip a folder, transfer it to a remote location and then extract it to a folder. The problem is that this machine is a little unreliable and sometimes writes don't succeed and there can be zero-byte ...
3
votes
2answers
123 views

Extract compressed file by it's header as it is piped from stdout (locally or from remote location)?

I am sending compressed file with piping either local or from a network location. And on the receiving end, I would like to detect the type of compression and use the appropriate decompression utility ...
1
vote
1answer
163 views

How to convert tar.bz2 to .bz2?

I have a file with compression tar.bz2 , I would like to convert it to .bz2 . Is there a command that does this in one shot? I tried the below but no joy ! tar -xvjf US.tar.bz2 US bzip2: ...
2
votes
3answers
98 views

How can I keep rsync from using encryption and compression for the file transfers after the connection is authorized?

I understand that a secure connection (i.e. ssh) is needed to authorize a connection to the remote server. But after that is authorized can the data be transmitted without encryption and compression ...
5
votes
1answer
173 views

Get compressed size of piped output with gzip?

To get uncompressed size of already compressed file, I can use -l option in gzip utility: gzip -l compressedfile.gz However is there a way to get size of compressed file if I am piping the output? ...
22
votes
4answers
2k views

How to know how much space an uncompressed zip will take

Given a (really long) list of zip files, how can you tell the size of them once uncompressed?
0
votes
1answer
74 views

Browser (HTTP client) with support of bzip2 or lzma2 compression?

What browsers (HTTP clients) support HTTP compression using bzip2 or lzma2? (HTTP compression for page requests) I have only heard of Firefox patch, but I am not sure if it actually supports ...
0
votes
1answer
24 views

Generating files of varying variability?

I have used /dev/zero and /dev/random to generate files of certain size. However, is there a way to control variability of generated files? I would like to create few files that would have same ...
4
votes
3answers
240 views

How can I change the first line of a big gzip file without decompressing all of it?

I currently have a compressed file, A.gz which contains lots of tabulated data, including a header in the first line. I want to create another file, B.gz, which has the same data as the previous file, ...
0
votes
1answer
145 views

What is the highest compression ratio compressor for pdf, djvu, and images

Comments so far have not been specific with the type of files they needed to compress and didn't reach a definitive answer I have a 1 TB hard drive full of pdf,djvu mostly and the rest are images and ...
0
votes
0answers
38 views

How do I create a deflate file suitable for HTTP?

Say I have index.html. How do I obtain index.html.deflate appropriate for serving through HTTP with the "deflate" encoding? I am coding a simple web server and I don't want to implement streaming ...
4
votes
3answers
498 views

Compression / archiving that keeps permissions and file owner?

I know that similar questions have been asked about permissions, but which form of compression or archiving keeps the permissions and file owners of each file and directory? I was considering the ...
11
votes
3answers
706 views

What does the `-C` flag exactly do in `scp`?

I always use either rsync or scp in order to copy file from/to a remote machine. Recently, I discovered in the manual of scp (man scp) the flag -C -C Compression enable. Passes the -C flag to ...
0
votes
0answers
236 views

where to compress=lzo, btrfs with virtual disk images, with their vm's internally using btrfs

I've been using compress=lzo on my btrfs volumes, because I read this fast compression method actually increases drive performance. (Haven't tested myself.) I'm starting to use ...
3
votes
4answers
2k views

Is it possible to make a .tar.gz file directly from stdin? Or, I need to tar together already gzipped files

I'm going to tell you exactly what I need in order to clarify the cryptic question in the title. I'm currently making scheduled MySQL backups of all my databases with something like: mysqldump ... | ...
2
votes
1answer
236 views

How to reduce size of a FLAC file without damaging its quality?

I got collection of vinyls which I paid for them to be digitized in FLAC. However my Hard Drive is getting full (91% full as of today) and the smallest of these vinyls is 2.3 GB in size! I would ...
0
votes
1answer
108 views

Can I use lzma compression, without using “tar”, on OSX 10.10.3?

I kind of think of OSX as Unix by another name. But, if I need to move this to the Apple forum, please tell me! I can create tar files compressed using lzma as such: $ ls ./DbDumps alpha.dmp ...
6
votes
3answers
399 views

Does gz compression ratio improve over time?

I have some process that creates a stream of millions of highly similar lines. I'm piping this to gz. Does the compression ratio improve over time in such a setup? I.e. is the compression ratio better ...
0
votes
2answers
39 views

How to open compressed file in less mode [duplicate]

How do I open a compressed file in less mode. I am able to grep using zgrep and using zcatI can open, but the file is big I want to open in less mode.
8
votes
3answers
531 views

Can tarring files improve compression?

Can tarring a bunch of files together improve compression with the standard tools, e.g. gzip, bzip2, xz? I've long thought this to be the case but never tested it out. If we have 2 copies of the same ...
0
votes
1answer
632 views

Compress multiple files and keep their name?

I have hundreds of .csv files that I need to compress into one file each. It should look like this originalname.csv.gz
1
vote
0answers
26 views

Pipe gzip output to Bluetooth transfer?

How do I pipe gzip output to Bluetooth transfer? I am able to send send file over Bluetooth using this command: bluetooth-sendto --device=12:23:35:96:45:45 file.tar However, piping gzip didn't ...
13
votes
9answers
3k views

Memory problems when compressing and transferring a large number of small files (1TB in total)

I have 5 million files which take up about 1TB of storage space. I need to transfer these files to a third party. What's the best way to do this? I have tried reducing the size using .tar.gz, but ...
2
votes
1answer
57 views

lrzip compression does not shrink random data

I'm trying to shrink down some big files around 3.0 GB each and I decided to use lrzip after seeing a few benchmarks and such however is it something I'm doing wrong or something as whenever I run ...
8
votes
2answers
556 views

Create a gzip file of a predetermined size

I want to test an upload of files of content type application/x-gzip of various sizes ranging from 100 MB to 999 MB. How can go I about creating .gz files of these predetermined sizes? If I do dd ...
11
votes
3answers
7k views

How do I compress files in-place?

I have a machine with 90% hard-disk usage. I want to compress its 500+ log files into a smaller new file. However, the hard disk is too small to keep both the original files and the compressed ones. ...
1
vote
1answer
758 views

unpigz (and untar) to a specific directory

I know how to gunzip a file to a selected location. But when it comes to utilizing all CPU power, many consider pigz instead of gzip. So, the question is how do I unpigz (and untar) a *.tar.gz file ...
7
votes
2answers
1k views

How to store a large folder in a single file without compression

Sorry if this is a stupid question, but I want to take a 78gb folder and store it in a single file (for upload into a cloud service), as if I am compressing it in an archive, but I don't want any ...
-1
votes
1answer
78 views

Erratic behavior with gzip on unix

I am compressing log files daily and very recently I have been seeing some crazy results on the file type of the file after compression. Normally you would expect file filename.gz returning gzip ...
35
votes
8answers
51k views

How do I grep recursively through .gz files?

I am using a script to regularly download my gmail messages that compresses the raw .eml into .gz files. The script creates a folder for each day, and then compresses every message into its own file. ...
0
votes
2answers
446 views

Gzip file but excluding some directories|files and also append current date

I'm looking for a way to create a .tar.gz file from a directory but I have some doubts. I'm know that the command for create the file is: tar -zcvf /path/to/compressed/file.tar.gz /path/to/compress ...
1
vote
1answer
261 views

Handle compress command error messages on AIX

I am working on production issue and unable to reproduce error message for compress command. Current Solution We have case-esac block in shell wrapper that handles messages as follows; case "$MSG" ...
2
votes
3answers
986 views

How to compress the directory and its contents

How do I compress a directory and its contents in ksh on AIX for a file I do compress filename