I have a file input.txt which contains multiple filename in the below format. FILENAME_DATE_LINENUMBER , the input.txt contains many such filenames. The filename itself has precisely 5 underscore.


All files are present in sub directory as input.txt. I want to parse input.txt , iterate through each filename and print FILENAME and the specified line number ( from the FILENAME ) to output.txt

I understand that sed or awk will be used , and below command can do the job.

awk 'FNR==LINENUMBER {print FILENAME, $0}' *.txt >output.txt

But how can i iterate through the file input.txt and find the FILENAME and extract LINENUMBER from FILENAME to output.txt

The FILENAME specified in input.txt can in one of the sub directories where input.txt is located. There can be only one file with FILENAME in input.txt inside one of the sub directory ( one level ) form the input.txt location.

├── input.txt
│   ├── DIR1
│   │   ├── FILE_NAME_1.DAT
│   ├── DIR2
│   │   ├── FILE_NAME_2.DAT
│   ├── DIR3
│   │   ├── FILE_NAME_3.DAT

In output.txt it should be printed as

LINE( Extracted from FILENAME present in input.txt )
  • 1
    With such questions you should always give example output. You want FILE1.DAT_20180123_4 transformed into FILE1.DAT 4? Jan 23, 2018 at 17:43
  • @shubhamdeodia, your output should be FILENAME LINE(under number) - update your question Jan 23, 2018 at 18:13
  • @RomanPerekhrest , thank you. I have updated. Jan 23, 2018 at 18:19
  • @shubhamdeodia, ok, can we assume that all files FILE*.DAT... are in the same directory as input.txt ? Jan 23, 2018 at 18:35
  • @RomanPerekhrest , no they are in the sub directories ( one level down ) from where input.txt is and we don't even know the name of sub directory. Jan 23, 2018 at 18:40

4 Answers 4


do_one() {
    # two args: $1=filename_no_dir $2=line_number                                             
    # Find the single filename                                                                
    eval file=*"/$1"
    echo $1
    # $. == line number                                                                       
    perl -ne 'chomp; $.=='"$2"' and print "LINE($_)\n"' $file
export -f do_one

# Generate som test data                                                                      
parallel 'mkdir DIR{}; seq 100 110 >DIR{}/FILE_NAME_{}.DAT' ::: {1..4}

# Test input.txt                                                                              
cat <<EOF |                                                                                   
  # Remove _YYYYMMDD.* to get filename, and .*_ to get line number                            
  parallel do_one '{= s/_201\d\d\d\d\d.*// =}' '{= s/.*_// =}'


:> awk -F_ '{ print $1; print $3; }' inputfile
  • what if FILE2.DAT is in nested directory from where inputfile.txt is. I mean we don't know the relative path for FILE2.DAT from inputfile and we don't even have the exact name of subdirectory. Jan 23, 2018 at 17:57
  • @shubhamdeodia Are you complaining to me about your input data...? Jan 23, 2018 at 18:02
  • I am sorry , my question was a little confusing, I have corrected it. I want to find the FILENAME in input.txt and then find line number in FILENAME and print it to putput.txt Jan 23, 2018 at 18:08

If I'm understanding you correctly,

while IFS=_ read -r filename unuseddate linenum
  printf "%s\n" "$filename"
  sed -n "${linenum}{p;q}" */"$filename"
done < input.txt > output.txt

This reads one line at a time from input.txt, splitting the line into 3 parts based on the underscore. It prints the filename then fires off a sed command that (by default prints nothing) and then on the specified line number, prints the line and quits that invocation of sed. The location of the filename should be in one of the immediate subdirectories of the current directory.

All of the output is then redirected to output.txt.

  • Hi , Is there a way we can tell the script to select last occurrence of underscore for line Number instead of 3rd occurrence because filename may contain more than 3 underscore. Jan 23, 2018 at 19:09
  • Not this script, because I based it on your sample input. You should really post representative data in your question in order to get usable answers.
    – Jeff Schaller
    Jan 23, 2018 at 19:15
  • Yes, you are right, this script works perfectly with 3 underscore. Thank you. Jan 23, 2018 at 19:18

Complex solution with GNU parallel + find + awk:

Let's say each input file has a content similar to the following:

cat DIR1/FILE1.DAT_20180123_4
FILE1 a 
FILE1 b 
FILE1 c 
FILE1 d 
FILE1 e 
FILE1 f 

So, by the above scheme, the 2nd line in file FILE2.DAT_20180123_5 would be FILE2 b and the 7th line in file FILE4.DAT_20180123_7 - FILE4 g. Assume input.txt file is the same as in the question.

The job:

find . -type f -regextype posix-egrep -regex ".*/($(paste -s -d'|' input.txt))" \
| parallel -j0 "awk -v n="{=s/.*_//=}" -v fn="{/}" \
               'NR==n{ print fn,\$0; exit }' {}" > output.txt

The final output.txt contents:

$ cat output.txt 
FILE4.DAT_20180123_7 FILE4 g
FILE3.DAT_20180123_6 FILE3 f 
FILE1.DAT_20180123_4 FILE1 d 
FILE2.DAT_20180123_5 FILE2 e
  • Hi , but the fileName itself has some underscore , to be specific 5 underscore. Jan 23, 2018 at 19:35
  • @shubhamdeodia, elaborate your file name convention: should all the files be ended with .DAT only? Jan 23, 2018 at 19:40
  • Yes, they all end with .DAT , but I can't install GNU parallel in my Linux distro. Don't have the privileges. Jan 23, 2018 at 19:41
  • @shubhamdeodia, the main problem is in your input.txt - it should have include full filenames not basenames. So reconsider your input.txt scheme and your life will be easier Jan 23, 2018 at 19:49
  • @shubhamdeodia Are the reasons covered on oletange.blogspot.dk/2013/04/why-not-install-gnu-parallel.html If not, please elaborate.
    – Ole Tange
    Jan 31, 2018 at 15:39

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .