I have a data file of numbers seperated by tabs, like this

1 2 3 4
2 4 6 8

My real file is 50000 columns wide and I only need every 100th column (column 100, 200, 300, 400, ...). Now I would like to remove all the other columns.

How can I do that?


That's what awk is for:

awk '{for(i=100;i<=NF;i+=100){printf "%s ",$i;} print ""}' file > output

Or, if you can have spaces inside your fields, specify tab as the field separator:

awk -F'\t' '{for(i=100;i<=NF;i+=100){printf "%s ",$i;} print ""}' file > output

Alternatively, you could use Perl:

perl -ane 'for($i=99;$i<=$#F;$i+=100){print "$F[$i] "}' file > output

To do this for multiple files, you can use a shell loop (assuming you want to run this on all files in the current directory):

for f in *; do
  awk '{for(i=100;i<=NF;i+=100){printf "%s ",$i;} print ""}' "$f" > "$f".new;
  • I tried the first solution and it worked. Is there a way to automate this if I want to cut the columns from multiple files, which are called xy_1.txt, xy_2.txt, xy_3.txt, ...? – Blub Bla Sep 16 '15 at 14:02
  • @BlubBla see update. – terdon Sep 16 '15 at 14:05

Although I don't know if it's appropriated with big files, you can do this with cut:

cut -d " " -f -100 < [your file]
  • This will only print field 100, not every 100th field as the OP requested. – terdon Sep 16 '15 at 14:06
  • No it won't, unless you forget the dash before 100. – pedroapero Sep 16 '15 at 14:59
  • Ah, yes, my bad. This will print fields 1 to 100 though. Still not what the OP wants. – terdon Sep 16 '15 at 15:01
  • With cut you could use cut -d' ' -f$(seq -s, 0 100 N) where N is the total number of columns but as you said, this won't work with big numbers (you'll most likely get an argument list too long). – don_crissti Sep 16 '15 at 16:18
  • I just realize what "every 100th column" means. Indeed, the awk method is the one, especially with the NR operator. Cheers! – pedroapero Sep 16 '15 at 16:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.