1

I have a file input.txt which contains multiple filename in the below format. FILENAME_DATE_LINENUMBER , the input.txt contains many such filenames. The filename itself has precisely 5 underscore.

FILE_NAME_1.DAT_20180123_4
FILE_NAME_2.DAT_20180123_5
FILE_NAME_3.DAT_20180123_6
FILE_NAME_4.DAT_20180123_7

All files are present in sub directory as input.txt. I want to parse input.txt , iterate through each filename and print FILENAME and the specified line number ( from the FILENAME ) to output.txt

I understand that sed or awk will be used , and below command can do the job.

awk 'FNR==LINENUMBER {print FILENAME, $0}' *.txt >output.txt

But how can i iterate through the file input.txt and find the FILENAME and extract LINENUMBER from FILENAME to output.txt

The FILENAME specified in input.txt can in one of the sub directories where input.txt is located. There can be only one file with FILENAME in input.txt inside one of the sub directory ( one level ) form the input.txt location.

DIR
├── input.txt
│   ├── DIR1
│   │   ├── FILE_NAME_1.DAT
│   ├── DIR2
│   │   ├── FILE_NAME_2.DAT
│   ├── DIR3
│   │   ├── FILE_NAME_3.DAT

In output.txt it should be printed as

FILENAME
LINE( Extracted from FILENAME present in input.txt )
  • 1
    With such questions you should always give example output. You want FILE1.DAT_20180123_4 transformed into FILE1.DAT 4? – Hauke Laging Jan 23 '18 at 17:43
  • @shubhamdeodia, your output should be FILENAME LINE(under number) - update your question – RomanPerekhrest Jan 23 '18 at 18:13
  • @RomanPerekhrest , thank you. I have updated. – shubham deodia Jan 23 '18 at 18:19
  • @shubhamdeodia, ok, can we assume that all files FILE*.DAT... are in the same directory as input.txt ? – RomanPerekhrest Jan 23 '18 at 18:35
  • @RomanPerekhrest , no they are in the sub directories ( one level down ) from where input.txt is and we don't even know the name of sub directory. – shubham deodia Jan 23 '18 at 18:40
0
#!/bin/bash                                                                                   

do_one() {
    # two args: $1=filename_no_dir $2=line_number                                             
    # Find the single filename                                                                
    eval file=*"/$1"
    echo $1
    # $. == line number                                                                       
    perl -ne 'chomp; $.=='"$2"' and print "LINE($_)\n"' $file
}
export -f do_one

# Generate som test data                                                                      
parallel 'mkdir DIR{}; seq 100 110 >DIR{}/FILE_NAME_{}.DAT' ::: {1..4}

# Test input.txt                                                                              
cat <<EOF |                                                                                   
FILE_NAME_1.DAT_20180123_4                                                                    
FILE_NAME_2.DAT_20180123_5                                                                    
FILE_NAME_3.DAT_20180123_6                                                                    
FILE_NAME_4.DAT_20180123_7                                                                    
EOF                                                                                           
  # Remove _YYYYMMDD.* to get filename, and .*_ to get line number                            
  parallel do_one '{= s/_201\d\d\d\d\d.*// =}' '{= s/.*_// =}'

Output:

FILE_NAME_1.DAT
LINE(103)
FILE_NAME_2.DAT
LINE(104)
FILE_NAME_3.DAT
LINE(105)
FILE_NAME_4.DAT
LINE(106)
0
:> awk -F_ '{ print $1; print $3; }' inputfile
FILE1.DAT
4
FILE2.DAT
5
FILE3.DAT
6
FILE4.DAT
7
  • what if FILE2.DAT is in nested directory from where inputfile.txt is. I mean we don't know the relative path for FILE2.DAT from inputfile and we don't even have the exact name of subdirectory. – shubham deodia Jan 23 '18 at 17:57
  • @shubhamdeodia Are you complaining to me about your input data...? – Hauke Laging Jan 23 '18 at 18:02
  • I am sorry , my question was a little confusing, I have corrected it. I want to find the FILENAME in input.txt and then find line number in FILENAME and print it to putput.txt – shubham deodia Jan 23 '18 at 18:08
0

If I'm understanding you correctly,

while IFS=_ read -r filename unuseddate linenum
do
  printf "%s\n" "$filename"
  sed -n "${linenum}{p;q}" */"$filename"
done < input.txt > output.txt

This reads one line at a time from input.txt, splitting the line into 3 parts based on the underscore. It prints the filename then fires off a sed command that (by default prints nothing) and then on the specified line number, prints the line and quits that invocation of sed. The location of the filename should be in one of the immediate subdirectories of the current directory.

All of the output is then redirected to output.txt.

  • Hi , Is there a way we can tell the script to select last occurrence of underscore for line Number instead of 3rd occurrence because filename may contain more than 3 underscore. – shubham deodia Jan 23 '18 at 19:09
  • Not this script, because I based it on your sample input. You should really post representative data in your question in order to get usable answers. – Jeff Schaller Jan 23 '18 at 19:15
  • Yes, you are right, this script works perfectly with 3 underscore. Thank you. – shubham deodia Jan 23 '18 at 19:18
0

Complex solution with GNU parallel + find + awk:

Let's say each input file has a content similar to the following:

cat DIR1/FILE1.DAT_20180123_4
FILE1 a 
FILE1 b 
FILE1 c 
FILE1 d 
FILE1 e 
FILE1 f 
FILE1 g

So, by the above scheme, the 2nd line in file FILE2.DAT_20180123_5 would be FILE2 b and the 7th line in file FILE4.DAT_20180123_7 - FILE4 g. Assume input.txt file is the same as in the question.

The job:

find . -type f -regextype posix-egrep -regex ".*/($(paste -s -d'|' input.txt))" \
| parallel -j0 "awk -v n="{=s/.*_//=}" -v fn="{/}" \
               'NR==n{ print fn,\$0; exit }' {}" > output.txt

The final output.txt contents:

$ cat output.txt 
FILE4.DAT_20180123_7 FILE4 g
FILE3.DAT_20180123_6 FILE3 f 
FILE1.DAT_20180123_4 FILE1 d 
FILE2.DAT_20180123_5 FILE2 e
  • Hi , but the fileName itself has some underscore , to be specific 5 underscore. – shubham deodia Jan 23 '18 at 19:35
  • @shubhamdeodia, elaborate your file name convention: should all the files be ended with .DAT only? – RomanPerekhrest Jan 23 '18 at 19:40
  • Yes, they all end with .DAT , but I can't install GNU parallel in my Linux distro. Don't have the privileges. – shubham deodia Jan 23 '18 at 19:41
  • @shubhamdeodia, the main problem is in your input.txt - it should have include full filenames not basenames. So reconsider your input.txt scheme and your life will be easier – RomanPerekhrest Jan 23 '18 at 19:49
  • @shubhamdeodia Are the reasons covered on oletange.blogspot.dk/2013/04/why-not-install-gnu-parallel.html If not, please elaborate. – Ole Tange Jan 31 '18 at 15:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.