I have a 4TB big text file Exported from Teradata records, and I want to know how many records there are in that file.
How may I do this quickly and efficiently?
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. It only takes a minute to sign up.Sign up to join this community
If this information is not already present as meta data in a separate file (or embedded in the data, or available through a query to the system that you exported the data from) and if there is no index file of some description available, then the quickest way to count the number of lines is by using
wc -l on the file.
You can not really do it quicker.
To count the number of records in the file, you will have to know what record separator is in used and use something like
awk to count these. Again, that is if this information is not already stored elsewhere as meta data and if it's not available through a query to the originating system, and if the records themselves are not already enumerated and sorted within the file.
You should not use line based utilities such as
sed. These utilities will issue a
read() system call for every line in the input file (see that answer on why this is so). If you have lots of lines, this will be a huge performance loss.
Since your file is 4TB in size, I guess that there are a lot of lines. So even
wc -l will produce a lot of
read() system calls, since it reads only
16384 bytes per call (on my system). Anyway this would be an improvement over
sed. The best method - unless you write your own program - might be just
cat file | wc -l
This is no useless use of cat, because
cat reads chunks of
131072 bytes per
read() system call (on my system) and
wc -l will issue more, but not on the file directly, instead on the pipe. But however,
cat tries to read as much as possible per system call.