find and print particular line from list of filename and line number

Question

I have a file input.txt which contains multiple filename in the below format. FILENAME_DATE_LINENUMBER , the input.txt contains many such filenames. The filename itself has precisely 5 underscore.

FILE_NAME_1.DAT_20180123_4
FILE_NAME_2.DAT_20180123_5
FILE_NAME_3.DAT_20180123_6
FILE_NAME_4.DAT_20180123_7

All files are present in sub directory as input.txt. I want to parse input.txt , iterate through each filename and print FILENAME and the specified line number ( from the FILENAME ) to output.txt

I understand that sed or awk will be used , and below command can do the job.

awk 'FNR==LINENUMBER {print FILENAME, $0}' *.txt >output.txt

But how can i iterate through the file input.txt and find the FILENAME and extract LINENUMBER from FILENAME to output.txt

The FILENAME specified in input.txt can in one of the sub directories where input.txt is located. There can be only one file with FILENAME in input.txt inside one of the sub directory ( one level ) form the input.txt location.

DIR
├── input.txt
│   ├── DIR1
│   │   ├── FILE_NAME_1.DAT
│   ├── DIR2
│   │   ├── FILE_NAME_2.DAT
│   ├── DIR3
│   │   ├── FILE_NAME_3.DAT

In output.txt it should be printed as

FILENAME
LINE( Extracted from FILENAME present in input.txt )

Answer 1

#!/bin/bash                                                                                   

do_one() {
    # two args: $1=filename_no_dir $2=line_number                                             
    # Find the single filename                                                                
    eval file=*"/$1"
    echo $1
    # $. == line number                                                                       
    perl -ne 'chomp; $.=='"$2"' and print "LINE($_)\n"' $file
}
export -f do_one

# Generate som test data                                                                      
parallel 'mkdir DIR{}; seq 100 110 >DIR{}/FILE_NAME_{}.DAT' ::: {1..4}

# Test input.txt                                                                              
cat <<EOF |                                                                                   
FILE_NAME_1.DAT_20180123_4                                                                    
FILE_NAME_2.DAT_20180123_5                                                                    
FILE_NAME_3.DAT_20180123_6                                                                    
FILE_NAME_4.DAT_20180123_7                                                                    
EOF                                                                                           
  # Remove _YYYYMMDD.* to get filename, and .*_ to get line number                            
  parallel do_one '{= s/_201\d\d\d\d\d.*// =}' '{= s/.*_// =}'

Output:

FILE_NAME_1.DAT
LINE(103)
FILE_NAME_2.DAT
LINE(104)
FILE_NAME_3.DAT
LINE(105)
FILE_NAME_4.DAT
LINE(106)

Answer 2

:> awk -F_ '{ print $1; print $3; }' inputfile
FILE1.DAT
4
FILE2.DAT
5
FILE3.DAT
6
FILE4.DAT
7

Answer 3

If I'm understanding you correctly,

while IFS=_ read -r filename unuseddate linenum
do
  printf "%s\n" "$filename"
  sed -n "${linenum}{p;q}" */"$filename"
done < input.txt > output.txt

This reads one line at a time from input.txt, splitting the line into 3 parts based on the underscore. It prints the filename then fires off a sed command that (by default prints nothing) and then on the specified line number, prints the line and quits that invocation of sed. The location of the filename should be in one of the immediate subdirectories of the current directory.

All of the output is then redirected to output.txt.

Answer 4

Complex solution with GNU parallel + find + awk:

Let's say each input file has a content similar to the following:

cat DIR1/FILE1.DAT_20180123_4
FILE1 a 
FILE1 b 
FILE1 c 
FILE1 d 
FILE1 e 
FILE1 f 
FILE1 g

---

So, by the above scheme, the 2nd line in file FILE2.DAT_20180123_5 would be FILE2 b and the 7th line in file FILE4.DAT_20180123_7 - FILE4 g. Assume input.txt file is the same as in the question.

The job:

find . -type f -regextype posix-egrep -regex ".*/($(paste -s -d'|' input.txt))" \
| parallel -j0 "awk -v n="{=s/.*_//=}" -v fn="{/}" \
               'NR==n{ print fn,\$0; exit }' {}" > output.txt

The final output.txt contents:

$ cat output.txt 
FILE4.DAT_20180123_7 FILE4 g
FILE3.DAT_20180123_6 FILE3 f 
FILE1.DAT_20180123_4 FILE1 d 
FILE2.DAT_20180123_5 FILE2 e