29

I am trying to print every Nth line out of a file with more than 300,000 records into a new file. This has to happen every Nth record until it reaches the end of the file.

Improve this question
  • see also: unix.stackexchange.com/q/214445/117549 – Jeff Schaller♦ Jun 4 '17 at 19:54
  • Looking in your comments, we cant understand what you need. Provide sample input and sample output. Do you need a range ? From Nth line up to EOF? – George Vasiliou Jun 4 '17 at 20:25
  • thanks, I have 355,000 records which is sorted but I need to get a sample of the data (1/3 which is about 100,000) so I thought if I retrieve the 300th of the sorted file from 1 to EOF, I should be able to get a fair sample. – Terisa Jun 4 '17 at 20:50
  • What the word "records" means to you? Do you refer to number of lines in a file or you refer to a number of files? Better describe your problem with terms like files and lines. Avoid the word record. Tell us how many lines has your file or how many files you need to parse. – George Vasiliou Jun 4 '17 at 21:00
  • 2
    Please explain your requirements more clearly. Against my answer you wrote. "For example for an input file with 300000 I should get 100000 records in the output." That sentence doesn't make any sense, unless if you mentioned that n=3 and you wanted the 3rd, 6th, 9th line. Or perhaps, you wanted the 1st, 4th, 7th line. There are multiple different solutions because the way you're asking the question is not clear. – Stephen Quan Jun 5 '17 at 2:27
49
awk 'NR % 5 == 0' input > output

This prints every fifth line.

To use an environment variable:

NUM=5
awk -v NUM=$NUM 'NR % NUM == 0' input > output
Improve this answer
  • I ran this command and got only 1166 in the output. I expected 100,000. – Terisa Jun 4 '17 at 21:09
  • awk 'NR % 300 ==0' 350000.records > 100000-records – Terisa Jun 4 '17 at 21:10
  • 3
    As commented in your "answer" below, pleas accept this answer as the solution. Thank you. – Deathgrip Jun 4 '17 at 21:30
  • 1
    or every 5th line starting at the 1st using NR % 5 == 1 or every 5th line starting at the 4th using NR % 5 == 4 – northern-bradley Oct 24 '18 at 20:23
  • 1
    ffmpeg and other programs expect the data of a file or files piped in. Your solution helped me list all the JPGs in a dir and feed ever 5th filename to cat to read the data to pipe to ffmpeg. Couldn't have done it without you! (I looked all over and tried dozens of possible solutions). @northern-bradley your ==1 helps too if there's only 1 file in the directory. cat $(ls *.jpg | awk 'NR % 5 == 1' -) | ffmpeg -r 15 -f image2pipe -vcodec mjpeg -i - -r 30 test.mp4 – Able Mac Jun 15 '19 at 4:15
19

To print every N  th line, use

sed -n '0~Np'
For example, to copy every 5th line of oldfile to newfile, do

sed -n '0~5p' oldfile > newfile

This uses sed’s first ~step address form, which means “match every step’th line starting with line first.”  In theory, this would print lines 0, 5, 10, 15, 20, 25, …, up to the end of the file.  Of course there is no line 0, so it just prints lines 5, 10, 20, 25, …;  0~5 is just a convenient alternative way of saying 5~5 (which prints every 5th line starting with line 5; i.e., lines 5, 10, 15, 20, 25, …).

For another example of this sed capability (which does not answer the question),

sed -n '2~5p' oldfile

would print lines 2, 7, 12, 17, 22, 27, …, up to the end of the file.

Improve this answer
  • 1
    Not sure what sed you are using, but sed -n '0,5p' filename (filename is a file with 533 lines) yields no output. sed -n '1,5p' filename for the same file prints the first 5 lines. – Deathgrip Jun 5 '17 at 1:38
  • My fingers are so stupid!  I have to watch them like a hawk. I've fixed their error. Thanks for alerting me to it. – G-Man Says 'Reinstate Monica' Jun 5 '17 at 1:42
  • 1
    Yes you meant sed -n '0~5p' filename. As for me, I went to double check and realized that the sed step-match functionality isn't even available on OSX. "D'oh!" – Deathgrip Jun 5 '17 at 1:47
  • You need GNU sed, N~M addresses are a GNU extension. – Satō Katsura Jun 5 '17 at 7:52
  • 1
    i like that this uses sed which is what i originally searched for but to my brain @deathgrip's use of awk is clearer – northern-bradley Oct 24 '18 at 20:17
1

Similarly to sed, we have also awk:

$ seq 1000000000 |awk 'NR==500000{print;exit}'
500000

NR=Number of line you want to print (and then exit to avoid waiting the file to finish). In your case 

awk 'NR==Nth{print;exit}' inputfile >outputfile

Where Nth is the Nth line number you need to print.

Improve this answer
  • Looks like the question was initially worded badly, and this answers the wrong question. – rjmunro Oct 9 '20 at 14:55
1
sed -n -e '5{p;q}' yourfile > newfile

Where N = 5, as an example for your case.

The sed command breaks down as: -n means "don't print lines by default"; then, on line 5, run a set { ... } of commands; those commands are: print the line, then quit.

Improve this answer
  • Thanks very much Jeff. Will try it now and let you know. – Terisa Jun 4 '17 at 20:00
  • This only does one record. How should I do it to do it for all the records (400,000) – Terisa Jun 4 '17 at 20:03
  • 1
    you asked for exactly the Nth line; that's what this does – Jeff Schaller♦ Jun 4 '17 at 20:03
  • I just ran it but still gives one one record only: sed -n -e '300{p;q}' orig_list_sorted.txt > 100000-records cat 100000-records 0105000168535V003004 – Terisa Jun 4 '17 at 20:15
  • 1
    It sounds like you should open a new question that spells out your different requirements. Is it "every N records"? Or "from N to the end of the file"? Search through the similar and linked questions to see other Q's and A's that might already solve your problem. – Jeff Schaller♦ Jun 4 '17 at 20:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.