Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I need to grab the first lines of a long text file for some bugfixing on a smaller file (a Python script does not digest the large text file as intended). However, for the bugfixing to make any sense, I really need the lines to be perfect copies, basically byte-by-byte, and pick up any potential problems with character encoding, end-of-line characters, invisible characters or what not in the original txt. Will the following simple solution accomplish that or I'd lose something using the output of head?

head infile.txt > output.txt

A more general question on the binary copy with head, dd, or else is now posted here.

share|improve this question
    
head only get 10 lines by default, you missed the rest of file. – cuonglm Mar 3 at 10:10
1  
The question mentions grabbing the first few lines. Also, if I needed the whole file, I always had the original. Or could make a copy more easily. – László Mar 3 at 10:16
up vote 10 down vote accepted

POSIX says that the input to head is a text file, and defines a text file:

3.397 Text File

A file that contains characters organized into zero or more lines. The lines do not contain NUL characters and none can exceed {LINE_MAX} bytes in length, including the <newline> character. Although POSIX.1-2008 does not distinguish between text files and binary files (see the ISO C standard), many utilities only produce predictable or meaningful output when operating on text files. The standard utilities that have such restrictions always specify "text files" in their STDIN or INPUT FILES sections.

So there is a possibility of losing information.

share|improve this answer
    
This is great, thanks. Can you think of a superior tool for slicing a text file, then? (I am happy to edit or repost the question for that if you have an answer.) – László Mar 3 at 10:23
1  
Perhaps a custom tool (or a version of head which uses a less-strict definition of "text files"). If I had to rely on it, I'd write a Perl script. – Thomas Dickey Mar 3 at 10:27
1  
you could use something like dd if you wanted, say, the first 10Kbytes of the file, for example: dd if=input of=output bs=1024 count=10. it's not a line-based copy, but will give you a usable smaller subset of the original file. – cas Mar 3 at 10:33
1  
You'd be better off raising another question about a specific tool, given the limitations in head (unless there's already one, which searching might reveal) rather than solve it in comments, which prevents other people from easily finding it later. – EightBitTony Mar 3 at 11:08
    
See the edit with the link to the new question. – László Mar 3 at 11:54

Apparently, head does indeed garble its input when it isn't a text file:

$ wc /bin/ls
   431   3454 126496 /bin/ls
$ head -n 431 /bin/ls > a
 wc a
   431   3447 125378 a
$ diff a /bin/ls
Binary files a and /bin/ls differ
$ md5sum /bin/ls a
42846aa64774a99f0f1feba36ec2e099  /bin/ls
de032f5aa5ef356fb7d5ab4dc622df2e  a
$ wc -c /bin/ls a
126496 /bin/ls
125378 a

Stéphane Chazelas makes a good point in the comments:

wc -l reports the number of newlines. /bin/ls likely has more bytes after the last occurrence of 0xa, which head -n 431 won't output. GNU tools (which you seem to be using) generally can deal with binary data (NUL bytes and arbitrarily long lines).

So, the reason the output is wrong when using head -n is that there is extra data after the last \n character. Looking at the source code for GNU head, I can confirm that both the function that reads by lines and the one that reads by bytes, use the same safe_open() call, so there really shouldn't be any difference between what they are capable of returning. This would suggest that using the GNU implementation of head (as typically found in most non-embedded OSes using Linux) is quite safe.

However, it looks like it works correctly if you tell it to work on bytes instead of lines (from man head):

  -c, --bytes=[-]NUM
          print  the  first  NUM bytes of each file; with the leading '-',
          print all but the last NUM bytes of each file

With the -c option, it seems to create identical files:

$ wc -c /bin/ls
126496 /bin/ls
$ head -c 126496 /bin/ls > a
$
$ md5sum /bin/ls a
42846aa64774a99f0f1feba36ec2e099  /bin/ls
42846aa64774a99f0f1feba36ec2e099  a

This is also the same result as what is obtained by dd:

$ dd if=/bin/ls of=a bs=126496 count=1
1+0 records in
1+0 records out
126496 bytes (126 kB, 124 KiB) copied, 0.000469919 s, 269 MB/s
$ diff a /bin/ls
$

I can't point to any official documentation that specifies that when using the -c flag, it will always produce correct binary output, but it seems like a reasonable assumption to make.

share|improve this answer
    
Ah, good and bad news for me. So I can try the -c option, but then need to figure how many bytes are in the first n lines exactly. Or if I take a round number of bytes, would the result still work as a text file with only the last line garbled or dropped? – László Mar 3 at 11:06
    
@László I would expect so, but I won't say I know for sure. That said, if your file does actually contain \n and is line based, I'm pretty sure head will act as promised. A simple test would be to i) head -n N file > foo to get the first N lines, then wc -c foo to get the bytes and head -c $bytes file >b to get the same number of bytes from the original file. Then, diff a b to see if the two methods produced identical files. – terdon Mar 3 at 11:09
2  
wc -l reports the number of newlines. /bin/ls likely has more bytes after the last occurrence of 0xa, which head -n 431 won't output. GNU tools (which you seem to be using) generally can deal with binary data (NUL bytes and arbitrarily long lines). – Stéphane Chazelas Mar 3 at 14:15
1  
@StéphaneChazelas ah, I see. That makes sense. I was looking at the source for head and the function for -c and the one that reads lines both use safe_read so I couldn't see why one would fail and the other not. Having data after the last newline would explain it. Thanks. – terdon Mar 3 at 14:24
1  
@László pelase see update and the comment above. You are almost certainly fine using head. – terdon Mar 3 at 14:27

You can use split(1). This will create a number of files that each correspond to a byte-correct slice of your file.

Example: FILE=test ; split -b 1000 $FILE $FILE.split. will create 1000-byte files test.split.aa and test.split.ab and so on, and cat $FILE.split.* > $FILE.recompose will yield a $FILE.recompose identical to the original $FILE. If your file is bigger than 1000*26*26 then you need to increase suffix-length (see man split)

With split -l 100 it will put 100 lines per file. I am fairly confident this will be byte-correct.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.