6

If I have a delimited file with many lines and columns (data.txt):

346 dfd asw  34
565 sd  wdew 34
667 ffg wew  23
473 sa  as   21
533 jhf qwe  54

and another file with line numbers that I want to extract (positions.txt)

3
5
8

How do I use the positions.txt file to extract those positions from data.txt? This is the result I would expect for this example:

667 ffg wew  23
533 jhf qwe  54
7

Simply with awk:

awk 'NR==FNR{ pos[$1]; next }FNR in pos' positions.txt data.txt
  • NR==FNR{ ... } - processing the 1st input file (i.e. positions.txt):
    • pos[$1] - accumulating positions(record numbers) set as pos array keys
    • next - jump to next record
  • FNR in pos - while processing the 2nd input file data.txt(FNR indicates how many records have been read from the current input file). Print record only if current record number FNR is in array of positions pos (search on keys)

Sample output:

667 ffg wew  23
533 jhf qwe  54
...
  • This seems to be much faster than the sed version below. – rmf Jan 19 '18 at 18:32
5

First create a sed script from the positions.txt file:

sed 's/$/p/' positions.txt

This will output

3p
5p
8p

This simple script will just print the indicated lines.

Then apply this to the data.txt file. If you're using bash (or any shell that understands process substitutions with <( ... )):

sed -n -f <( sed 's/$/p/' positions.txt ) data.txt

The -n stops sed from outputting anything other than what's explicitly printed by the given sed script.

With the examples given, this will yield

667 ffg wew  23
533 jhf qwe  54

If not using bash, then

sed 's/$/p/' positions.txt >filter.sed
sed -n -f filter.sed data.txt
rm -f filter.sed

... will do the same thing.

3

If positions.txt is sorted, it's also possible to do this in a single pass through both files, and without storing positions.txt in full. Simply read the next line off positions.txt when the previous matching line is met:

$ awk -vpos=positions.txt 'function get() { getline num < pos } 
     BEGIN { get() } NR==num { print; get() }' data.txt                 
667 ffg wew  23
533 jhf qwe  54

In practice, this is only useful if both files are really huge or you're really, really low on memory.

1
join <(cat -n data.txt) positions.txt | cut -d ' ' -f2-
  • Won't work for files with more than ten lines – iruvar Jan 20 '18 at 1:40
0

we can do this by simple for loop

Method1 using sed and for loop

for i in `cat positions.txt`; do sed -n ""$i"p" data.txt ; done

output

667 ffg wew  23
533 jhf qwe  54

Method2 using awk and for loop

for i in `cat positions.txt`; do awk -v i="$i" 'NR==i {print $0}' data.txt ;done

output

667 ffg wew  23
533 jhf qwe  54
  • 2
    These do have the downside that you'll read all of data.txt as many times as there are lines to pick out, which may be an issue if the file is very large. Then again, if the list of lines to pick is very large, that could also be an issue. Also, you do know that ""$i"p" leaves the $i unquoted, and that the "" at the start is a no-op? That might be better written as "${i}p". – ilkkachu Jan 19 '18 at 19:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.