6

I have a huge log file compressed in .gz format and I want to just read the first line of it without uncompressing it to just check the date of the oldest log in the file.

The logs are of the form:

YYYY-MM-DD Log content asnsenfvwen eaifnesinrng
YYYY-MM-DD Log content asnsenfvwen eaifnesinrng
YYYY-MM-DD Log content asnsenfvwen eaifnesinrng

I just want to read the date in the first line which I would do like this for an uncompressed file:

read logdate otherstuff < logfile.gz
echo $logdate

Using zcat is taking too long.

  • I guess the method for getting the first line and reading line by line would be similar right? @JeffSchaller – Pratik Mayekar Jun 26 '18 at 12:55
  • Alright I fixed the question. I really just want to read the first line though @JeffSchaller – Pratik Mayekar Jun 26 '18 at 12:58
  • If any of the answers solves your issue, please consider "accepting" the answer. This is the best way to show gratitude on this site. Accepting an answer not only marks the question as resolved, but also signals to future readers that the accepted answer actually solved the issue. More information about this is available here: unix.stackexchange.com/help/someone-answers – Kusalananda May 18 at 12:21
14

Piping zcat’s output to head -n 1 will decompress a small amount of data, guaranteed to be enough to show the first line, but typically no more than a few buffer-fulls (96 KiB in my experiments):

zcat logfile.gz | head -n 1

Once head has finished reading one line, it closes its input, which closes the pipe, and zcat stops after receiving a SIGPIPE (which happens when it next tries to write into the closed pipe). You can see this by running

(zcat logfile.gz; echo $? >&2) | head -n 1

This will show that zcat exits with code 141, which indicates it stopped because of a SIGPIPE (13 + 128).

You can add more post-processing, e.g. with AWK, to only extract the date:

zcat logfile.gz | awk '{ print $1; exit }'
4

You could limit the amount of data you feed to zcat (or gzip -dc), then ask for the first line:

head -c 1000 logfile.gz | zcat 2>/dev/null | head -1 | read logdate otherstuff

Adjust the 1000 if that doesn't capture enough data to get the entire first line.

  • Shouldn't zcat+head be enough? The head process would exit when done, causing zcat to receive a PIPE signal and quit... – Kusalananda Jun 26 '18 at 13:02
  • Sure; it may fill the pipe, so we've limited the damage in either case, but I liked the idea of throttling it "manually", in response to limiting the amount gzip deals with. I do like the idea of asking head to get one line from the pipe, though -- removes the guessing about how many bytes to ask head for. – Jeff Schaller Jun 26 '18 at 13:05
  • I just need the first few characters before a space. So why would it be necessary to adjust the number of characters to get the entire line? @JeffSchaller – Pratik Mayekar Jun 26 '18 at 13:06
  • @PratikMayekar, if you chose too small a value, you may not get the entire first line. Since you just want the first 10-ish bytes, I chose something arbitrarily larger than 10. – Jeff Schaller Jun 26 '18 at 13:07
  • head -c 1000 logfile.gz | zcat 2>/dev/null | head -1 worked fine head -c 50 logfile.gz | zcat 2>/dev/null | head -1 gave no output. Why is that @JeffSchaller – Pratik Mayekar Jun 26 '18 at 13:13
3

To just match a date from the 1st line of a zipped file - zgrep solution:

zgrep -m1 -o '^[^[:space:]]*' logfile.gz

This will output the first YYYY-MM-DD for you.

1

If you just want the first line without decompressing the file:

gunzip -c logfile.gz | awk 'NR==1 {print; exit}'

That will send the compressed data to standard output without decompressing it and awk will print only the first line.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.