I have a 4TB big text file Exported from Teradata records, and I want to know how many records there are in that file.

How may I do this quickly and efficiently?

  • Is each line a record? If yes, you can just use wc -l – Panki Mar 7 '19 at 10:56
  • 1
    This doesn’t answer the stated question, but the fastest way would be to ask your Teradata system. – Stephen Kitt Mar 7 '19 at 11:08
  • If the export happened to put a comment at the top, that'd make it pretty fast to find. – Jeff Schaller Mar 7 '19 at 11:21
  • I tried Using vim -R filename it took around 1.5 Hrs – Santosh Garole Mar 8 '19 at 7:45

If this information is not already present as meta data in a separate file (or embedded in the data, or available through a query to the system that you exported the data from) and if there is no index file of some description available, then the quickest way to count the number of lines is by using wc -l on the file.

You can not really do it quicker.

To count the number of records in the file, you will have to know what record separator is in used and use something like awk to count these. Again, that is if this information is not already stored elsewhere as meta data and if it's not available through a query to the originating system, and if the records themselves are not already enumerated and sorted within the file.

  • Thanks @Kusalananda for such great explainanation. – Santosh Garole Apr 17 '20 at 9:12

You should not use line based utilities such as awk and sed. These utilities will issue a read() system call for every line in the input file (see that answer on why this is so). If you have lots of lines, this will be a huge performance loss.

Since your file is 4TB in size, I guess that there are a lot of lines. So even wc -l will produce a lot of read() system calls, since it reads only 16384 bytes per call (on my system). Anyway this would be an improvement over awk and sed. The best method - unless you write your own program - might be just

cat file | wc -l

This is no useless use of cat, because cat reads chunks of 131072 bytes per read() system call (on my system) and wc -l will issue more, but not on the file directly, instead on the pipe. But however, cat tries to read as much as possible per system call.

  • Won't an io redirect be faster than cat and pipe ? – pLumo Mar 7 '19 at 12:22
  • @RoVo Could be, have you tried it? – chaos Mar 7 '19 at 12:25
  • 1
    Short test with 10 iterations of wc -l with a 701MB file: wc -l file 1.7s ;; wc -l < file 1.7s ;; cat file | wc -l 2.6s. – pLumo Mar 7 '19 at 12:28
  • "These utilities will issue a read() system call for every line in the input file" -- That can't be true. read() only reads a bunch of bytes, it doesn't know how to read a line. The utilities might differ in the size of a buffer they use for read(), but that's not the same. It's likely that most utilities will read at least a couple of kB in one go, and that's usually enough for a few lines at minimum. – ilkkachu Nov 29 '19 at 9:41

Below is what worked for me, tail -5 the file and then grep the text in your last line with -n option in grep...

tail -5 "filename"

LC_ALL=C fgrep -n "text in yourlast line" "filename"
  • 2
    How does help speed up counting the number of lines in a huge file? You’re reading the file twice. – Stephen Kitt Nov 29 '19 at 8:53
  • @StephenKitt, I'm not so sure about that. tail might well be smart enough to start reading from the end of the file. Of course that doesn't make it any more useful to do all that unnecessary work with the grep, or help with the fact that the text in the last line might also appear elsewhere in the file. – ilkkachu Nov 29 '19 at 9:43
  • @ilkkachu ah yes, tail can indeed work backwards (and the GNU version does). If the text appears multiple times, fgrep will show multiple matches, but will still show the last line. The results might not be accurate if the last line’s contents aren’t obvious from tail’s output (e.g. an empty line or a line containing only whitespace). – Stephen Kitt Nov 29 '19 at 9:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.