Questions tagged [compression]

Shrinking (compressing) and restoring (decompressing) of data.

44 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
7
votes
2answers
589 views

How do I layer compression on top of encryption on top of BtrFS, per-file?

BtrFS should support this really, but it doesn't. So ECryptFS looks like it will fill the gap. The only thing is, how would I layer compression on top of this? Reasons for compression on top of ...
3
votes
0answers
47 views

How to unpack a tag.gz file without having 2x free space?

I have about 1 million small files, that take up about 20 GB. I need to upload them to a host that only has 23 GB free space. I would prefer compressing the files locally to increase the transmission ...
2
votes
0answers
54 views

Compress VirtualBox Linux Virtual Machine

I am trying to create a VirtualBox linux virtual machine on which I want to mainly run Wine to install a windows application. I managed to install Lubuntu 18.04.2, since it is the lightest Ubuntu ...
2
votes
0answers
27 views

Resumable compression

I'm in the process of compressing a very large file (several TBs) with the following command, and it's been running for several days. tar -I pigz -cf /path/to/compressed.tgz /dataset/to/compress I'm ...
2
votes
1answer
83 views

Bash script for uncompressing multiple files at a once

I have a set of files: lium3@l128-37:/DATA/fr94/01 $ ls -l total 7784 -rw-r--r-- 1 lium3 mcm1 395575 May 2 1996 fr940104.0z -rw-r--r-- 1 lium3 mcm1 113815 May 2 1996 fr940104.1z -rw-r--r-- 1 ...
2
votes
2answers
563 views

Best way to compress a folder of small pre-compressed files (images)?

I have a folder containing approximate 50GB of images and video files (~35k files) that I'd like to compress into a single file for storage along side a database. Are there any compression options (or ...
2
votes
0answers
161 views

How to extract and stitch together a multipart tar arcvhive without the initial file

I have a large folder with multi part tar files each 2gb in size, file.partaa through file.partgy, however I am missing the initial tar file. Is there any way that I can extract each part ...
2
votes
1answer
252 views

linux unbuffered compression options

I'm working on a embedded system and due to persistent memory restrictions, I need to compress a log file "on the fly". My objective is to have a one liner inside a script, which would look like this:...
2
votes
0answers
232 views

How do I create a deflate file suitable for HTTP?

Say I have index.html. How do I obtain index.html.deflate appropriate for serving through HTTP with the "deflate" encoding? I am coding a simple web server and I don't want to implement streaming ...
2
votes
0answers
88 views

Image set compression

Is there a good open source (not exclusive to Linux) for compression of a set of similar images? I have thousands of documents scanned but I do not access them often. The documents are forms, with ...
2
votes
1answer
326 views

Configuring Apache to read svgz file type

I recently migrated a site to a new server. I'm now getting this error from Firefox when attempting to load an .svgz file: XML Parsing Error: not well-formed The svg rendered fine on the old server, ...
1
vote
0answers
104 views

Can gzip change original file size?

/dev/sdf is a 32G SD. As far as I can tell, I had performed the following, but upon attempting to perform the final step and write to a SD, my file image appears to be one byte to large for the SD. ...
1
vote
0answers
532 views

Keep metadata or edit metadata with tar

I have a tarball. This tarball contains—of course—various files and directories. When I extract the tar, some of these files already exist, while others don't. What I would like to know is this: how ...
1
vote
1answer
238 views

Compressing Directory tree into multiple tar archives into a single xz

I'm trying to compress a large rootdir which contains many subdir_i the folder tree looks like: ./rootdir ./rootdir/subdir_1 ./rootdir/subdir_2 ... I am looking to output this to a single compressed ...
1
vote
1answer
81 views

How can I create a compressed archive consisting of the largest files from multiple directories?

I want to write a script which will create a single compressed-archive file consisting of the largest files above a certain threshold from multiple directories. For example, I would like to know how ...
1
vote
0answers
123 views

password protection for logrotate compression

Is password protection possible with logrotate while compressing in Linux? If yes would someone let me know how? /root/NewLog { missingok notifempty compress size 20k daily ...
1
vote
0answers
33 views

xzcmp does not work for some files. (Internal pipe error 141?)

I am using xzcmp to compare several XZ files. But I have a strange problem: For some files (*), xzcmp doesn't work and returns exit code 2, without error message. zxcmp version1.sql.xz version2.sql....
1
vote
0answers
126 views

Enable cuda support in libbsc

I am interested in using libbsc in CUDA support. libbsc is basically used for compression in GPU (as well as CPU). As far as I tested, it is running in CPU (openmp) not in NVIDIA GPU (CUDA driver ...
1
vote
0answers
41 views

Where is my 7za data being written to

I have a weird question about 7za on Centos 7. I am adding a file to a 7za archive using the following command (the total size of the directory is around 40GB) /usr/bin/7za a -ppassword -t7z -mmt=off ...
1
vote
0answers
659 views

On-the-fly decompression

I have 44 TB worth of gzipped files that are 1.5 GB each when compressed or 2.0 GB when uncompressed. I have a tool that can read only uncompressed files. I would like to avoid the overhead of ...
1
vote
1answer
2k views

How to convert tar.bz2 to .bz2?

I have a file with compression tar.bz2 , I would like to convert it to .bz2 . Is there a command that does this in one shot? I tried the below but no joy ! tar -xvjf US.tar.bz2 US bzip2: ...
1
vote
0answers
63 views

Pipe gzip output to Bluetooth transfer?

How do I pipe gzip output to Bluetooth transfer? I am able to send send file over Bluetooth using this command: bluetooth-sendto --device=12:23:35:96:45:45 file.tar However, piping gzip didn't work ...
1
vote
1answer
761 views

Handle compress command error messages on AIX

I am working on production issue and unable to reproduce error message for compress command. Current Solution We have case-esac block in shell wrapper that handles messages as follows; case "$MSG" ...
1
vote
2answers
600 views

Debian: Create compressed Harddrive for copy

On my Debian system I have a box with certain configuration settings that I like but I've had to change them continuously. There have been times that I've gotten into a some-what irreversible position....
1
vote
0answers
150 views

problem with /compress

I have script that was provided with some SW. The script output is something like that (in debug mode). + changeDirectory + '[' -d /cdr/work/proc_raw/proc ']' + cd /cdr/work/proc_raw/proc + echo '...
1
vote
0answers
48 views

Is it possible to get specific information about how a file is compressed?

File compression works on the basic idea of assigning commonly recurring chains of strings to a common index of a table where the full string is stored. For example: Input file: foo bar bash baz foo ...
0
votes
0answers
27 views

pxz (parallel xz) output meaning

I am running pxz with -v option, to "show progress" according to the man page. I am getting as output a sequence of seemingly random numbers, like: 0 1 7 3 2 5 8 4 9 6 6 9 5 0 1 2 3 8 7 4 ... It ...
0
votes
1answer
22 views

Is there a way to create a (compressed) squashfs that contains an uncompressed file

We have a large file that needs to be uncompressed inside a compressed squashfs. Is this possible without patching mksquashfs? Something like this would be nice: $ mksquashfs existing_squash.sqsh ...
0
votes
0answers
30 views

Unzip files in loop

I have the following code which extracts numbers from netcdf files and every loop puts them in another netcdf file. mapfile Latitude < <(tr -s ' ' '\n' < final_ADCP_Saved.matLatitude.txt) #...
0
votes
0answers
35 views

Why I cannot tar gzip all the contents of a specific folder

I have a folder containing several folders. I want to tar and gzip the parent folder with all the contents of the subfolders. However, when I use tar -cvzf folder.tar.gz Folder/ only a few ...
0
votes
1answer
134 views

tar + compress folder to tar.gz and removed the folder

by the following way we tar the folder presto-server-0.210 to file tar -czvf presto-server-archive.tar.gz presto-server-0.210 so after we tar the folder , folder still exists is it possible to use ...
0
votes
0answers
86 views

How can I copy sparse files from a ext4 file system to an exfat file system and keep the apparent size?

My issue is that I am trying to copy files from my Ubuntu linux system which uses a ext4 file system to a Sandisk 1TB USB card, which uses an exFat system. In the ext4 system, the entire directory ...
0
votes
0answers
36 views

Compare file compression methods by sending to /dev/null

I'd like to compare the time to completion for compression algorithms and parameters. I'm confident that the inputs will be set up properly. In order to save space and avoid some variability my ...
0
votes
0answers
47 views

How to make use of tar --verify directly?

This question is about compressing rather large data directories (around 22-23 GiB), specifically the integrity checks of such archives. I do these big backups about once a month, so the time it takes ...
0
votes
0answers
7 views

Least resource heavy file compression method on hosting server

There are many compression formats like tar, zip, bzip etc. On a shared hosting (Linux), if I were to archive a huge folder for migration, which format would be best (least resource heavy)? Storage ...
0
votes
0answers
29 views

Is it possible to dynamically add to a tar.lrz file?

A bit of a basic problem, I'm slightly embarrased I'm running into it, however man lrztar didn't show me any commands for this. Basically, lets say I've got an archiving station which handles ...
0
votes
1answer
49 views

fsarchiver taking ages to backup and restore

Using Linux Mint 19.1 bootable USB Ext4 filesystem Processor 8600k Ram 16gb By default fsarchiver uses zstd compression Earlier when I used the following two commands, each command individually took ...
0
votes
0answers
45 views

How to backup and compress quick rotating files to a remote server

I have a server application creating log files and rotating these files based on size. The files are time stamped at creation. The application then deletes the oldest file when X number of files in ...
0
votes
0answers
371 views

Why is copying many files with Caja much slower than compressing+copying+uncompressing?

I am currently working with HASYv2 dataset which contains the hasy-data directory with 168.233 images of size 32px x 32px. I just wanted to copy that directory with Caja, but I skipped this after ...
0
votes
1answer
55 views

Logging to a new file for a continues running script

I have a script providing login continuously( like 2-3 pages in a second). I was using below one liner. sh myscript.sh > /home/user/mylog I am running another script to zip this file when it ...
0
votes
0answers
323 views

How to deal with corrupted .tar.lrz file

I have a file file.tar.lrz and tried to unzip it via lrzuntar file.tar.lrz but it gives an error: "Not an lrzip file" with lrunzip. I also tried to open it with file roller. file file.tar.lrz says ...
0
votes
1answer
62 views

Verify files AFTER extracting gzip

I gzip a folder, transfer it to a remote location and then extract it to a folder. The problem is that this machine is a little unreliable and sometimes writes don't succeed and there can be zero-byte ...
0
votes
2answers
847 views

How to zip folder content on Ubuntu without keep the folder structure(path)

How can I compress a directory content without compress the full path folder structure too. I am using this command to zip a folder content under var/www/ directory, but when I unzip the application....
0
votes
2answers
365 views

Does tar.gz do effective file compression of large files?

I wanted to compress a ~29GB file significantly, and used the tar command in the terminal like this : tar cvzf file.tar.gz /path/to/directory and it compressed it to ~26GB, so I looked up maximum ...