Alternative of too much slow gzip -d command

Question

I'm using an production server for loading large data set to Hadoop to access from Hive table.

We are loading subscribers web browsing data of Telecom Sector. We've large number of .csv.gz file (File sizes around 300-500MB) which is compressed using gzip. Suppose a file is as below:

Filename: dna_file_01_21090702.csv.gz

Contents:

A,B,C,2

D,E,F,3

We unzip 50 or so files and concatenate to one file. For troubleshooting purposes, we append the file name as first column of every row.
So concatenet data file would be:

dna_file_01_21090702.csv.gz,A,B,C,2

dna_file_01_21090702.csv.gz,D,E,F,33

For that purposed written below bash script:

#!/bin/bash
func_gen_new_file_list()
{
        > ${raw_file_list}
        ls -U raw_directory| head -50 >> ${raw_file_list}
}
func_create_dat_file()
{
        cd raw_directory
        gzip -d `cat ${raw_file_list}`
        awk '{printf "%s|%s\n",FILENAME,$0}' `cat ${raw_file_list}|awk -F".gz" '{print $1}'` >> ${data_file}
}
func_check_dir_n_load_data()
{
        ## Code to copy data file to HDFS file system 
}
##___________________________ Main Function _____________________________
        ##__Variable            
        data_load_log_dir=directory
        raw_file_list=${data_load_log_dir}/raw_file_list_name
        data_file_name=dna_data_file_`date "+%Y%m%d%H%M%S"`.dat
        data_file=${data_load_log_dir}/${data_file_name}

        ##__Function Calls
        func_gen_new_file_list
        func_create_dat_file
        func_check_dir_n_load_data

Now the problem is gzip -d command performing extremely slow. I mean really really slow. If it unzip 50 files and make the concatenated data file the size would be around 20-25GB.

To unzip 50 files and concatenate it to one takes almost 1 hour which is huge. In this rate, its impossible to process all the data generated in a single day.

My production server(VM) is pretty powerful. Total core is 44 and RAM is 256GB. Also HARD Disk is very good and high performing. IOwait is around 0-5.

How can I faster this process? What is the alternatives of gzip -d. Is there any other way to make the concatenated data file more efficiently. Please note that we need to keep the file name in records for trouble shooting purpose.

Otherwise we could have just use zcat and append to a data file without unzipping at all.

Answer 1

There is a lot of disk I/O that could be replaced by pipes. The func_create_dat_file takes a list of 50 compressed files, reads each of them and writes the uncompressed data. It then reads each of the 50 uncompressed data files, and writes it out again with the filename prepended. All of this work is done sequentially so can not take any advantage of your multiple cpus.

I suggest you try

func_create_dat_file()
{
    cd raw_directory
    while IFS="" read -r f
    do
        zcat -- "$f" | sed "s/^/${f%.gz}|/"
    done < "${raw_file_list}" >> "${data_file}"
}

Here the compressed data is read once from disk. The uncompressed data is written once to a pipe, read once from the pipe and then written once to the disk. The transformation of the data happens in parallel with the reading and so can use 2 cpus.

[Edit] A comment asked to explain the sed "s/^/${f%.gz}|/" part. This is the code to put the filename as a new field at the start of each line. $f is the filename. ${f%.gz} removes .gz from the end of the string. There is nothing special about the | in this context, so ${f%.gz}| is the filename with a trailing .gz removed followed by a |. In sed s/old/new/ is the substitute (replace) command, it takes a regular expression for the old part. ^ as a regular expression matches the start of line, so putting this together it say change the beginning of line to be the filename without a trailing .gz and a |. The | was added to match the OP's program rather than the OP's description. If it really was a CSV (comma separated variable) file, then this should be a comma rather than a vertical bar.

Answer 2

So what about this hive and hadoop? When you give gzip -d a "raw file list", maybe that takes a unwanted detour through that distributed file system.

Something on that production server is definitely not working. One hour to decompress 20 GB (I leave out the details). I decompressed 100 MB chopped up in 11300 files in 0.8 seconds. This is roughly 20x faster. Using the naiv huge file list gzip call, and a ram disk. I also installed and tried parallel, as proposed. It is 10% faster: 0.7 seconds. So that is not the problem.

(I only have Mini-PC i5 with 8 GB Memory)

I have a Wattmeter running. During the slow 14 seconds loop, 6 W were used.
During my questionable loop-with-ampersand it was around 17 W, for 6 seconds. (Prompt, also X server, is 3.5 W, susp-to-ram is 1.1, off is...0.7 Watt)

14 s loop vs. 0.8 s huge arg list in my test is the same ratio as your overall 25 GB/h vs. my 100 MB/0.8 s.: TWENTYFOLD...as if you were using the slow for loop. I think hadoop and hive is disturbing gzip and bash with the huge arg list.