4

I am writing a shell script to parse a text file that is extracted from a standardization pdf file. I would like for each test group (identified by Group 0, Group 1... to get the list of test numbers, such as 101, 102, 412... for group 0. I have tried sed, awk, but I do not have sufficient skills to succeed. Ideally I would like to get the output translated into LaTeX code, ie each output item surrounded by a suitable string, such as

\section{Group0}
\Testdetails{101}
\Testdetails{102}
...............
\section{Group1}
\Testdetails{305}
................

This is the source file.

                                                Table 6

                       Tests                     EN 2591-                   Remarks

                                                            All models
 Group 0
 Visual examination                                101
 Examination of dimensions and mass                102      To be performed on one pair per layout, in
                                                            sealed and un-sealed versions
 Contact insertion and extraction forces           412      To be performed on one pair per layout, in
                                                            sealed and un-sealed versions
 Measurement of insulation resistance              206      Only specimens of group 6
 Voltage proof test                                207      Only specimens of group 6
 Contact resistance - Low level                    201
 Contact resistance at rated current               202
 Mating and unmating forces                        408      On specimens of groups 2, 4 and 6
 Visual examination                                101
 Group 1
 Rapid change of temperature                       305
 Visual examination                                101
 Interfacial sealing                               324
 Measurement of insulation resistance              206      Immersed connectors
 Voltage proof test                                207      Immersed connectors
 Insert retention in housing (axial)               410
 Contact retention in insert                       409
 Mechanical strength of rear accessories           420
 Contact retention system effectiveness            426
 (removable contact walkout)
 Visual examination                                101
 Group 2
 Contact retention in insert                       409
 Rapid change of temperature                       305
4
awk '
    $1 == "Group" {printf("\\section{%s%d}\n", $1, $2); next}
    {for (i=1; i<=NF; i++) 
        if ($i ~ /^[0-9][0-9][0-9]$/) {
            printf("\\Testdetails{%d}\n", $i)
            break
        }
    }
' 

Update based on comment:

awk '
    $1 == "Group" {printf("\\section{%s %d}\n", $1, $2); next}
    {
      title = sep = ""
      for (i=1; i<=NF; i++) 
        if ($i ~ /^[0-9][0-9][0-9]$/) {
          printf("\\subsection{%s} \\Testdetails{%d}\n", title, $i)
          break
        }
        else {
          title = title sep $i
          sep = FS
        }
    }
' 
5
  • I generate the source file by "/usr/local/bin/pdftotext -layout -f 54 -l 60 <path-to-pdf-file> - | iconv -f L1 -t UTF-8" . How can I best link the two commands to create the LaTeX code in a single step if possible? – Yves May 8 '12 at 15:46
  • @Yves, just extend your existing pipeline: pdftotext ... | iconv ... | awk ... – glenn jackman May 8 '12 at 16:07
  • Btw, is there a way to maintain the space between"Group" and the group number in the target file, ie \Section{Group 0} and not \Section{Group0}. I feel I would now need also to add the test title in the target code, such as \subsection{Examination of dimensions and mass} \TestDetails{102} This would mean a way to use for the title a many necessary fields as needed, since printf("\\subsection{"$1" "$2"} \\TestDetails{%d}\n", $i) returns obviously a truncated label, such as \subsection{Examination of} \TestDetails{102} – Yves May 10 '12 at 4:35
  • @Yves, see update. – glenn jackman May 10 '12 at 12:44
  • Fantastic. The Latex code generated provides exactly the desired result, and will enable to automate the process almost entirelY...Many thanks – Yves May 12 '12 at 10:50
2

One way with perl using regexp and assuming infile has the content you posted in the question.

Content of script.pl:

use warnings;
use strict;

while ( <> ) { 
    chomp;
    if ( m/\A\s*(Group)\s*(\d+)/ ) { 
        printf qq[\\Section{%s}\n], $1 . $2; 
        next;
    }   

    if ( m/\s(\d{3})(?:\s|$)/ ) { 
        printf qq[\\Testdetails{%s}\n], $1; 
    }   
}

Run it like:

perl script.pl infile

With following output:

\Section{Group0}                                      
\Testdetails{101}                                      
\Testdetails{102}                                      
\Testdetails{412}                                      
\Testdetails{206}                                      
\Testdetails{207}                                      
\Testdetails{201}                                      
\Testdetails{202}                                     
\Testdetails{408}                                      
\Testdetails{101}                                      
\Section{Group1}                                      
\Testdetails{305}                                     
\Testdetails{101}                                     
\Testdetails{324}                                     
\Testdetails{206}                                      
\Testdetails{207}                                        
\Testdetails{410}
\Testdetails{409}
\Testdetails{420}
\Testdetails{426}
\Testdetails{101}
\Section{Group2}
\Testdetails{409}
\Testdetails{305}
3
  • Thanks a lot. This is quite fascinating. I will need time to explore the code.. I have just realized that in some cases, the standards are bilingual, therefore my source file contains duplicated lines, ie contains two tables 6 such as above (one in english, one in a second language). Is there an easy way to alter the code to remove the duplicated groups and associated tests numbers? – Yves May 8 '12 at 16:49
  • @Yves: Do you mean not to process repeated groups? Only first group-0, only first group-1, and so on? – Birei May 8 '12 at 19:13
  • Yes. As a matter of fact, I am trying another approach, trying to get rid of the duplicated lines while extracting the text file from the original pdf document. I have noticed the pages in english are on even pages while the ones I want to discard are odd. I have not yet found a clean way to do this, since the routine I use can set a range of pages, (see below from 50 to 56) but not the even ones only:/usr/local/bin/pdftotext -layout -f 50 -l 56 <path to pdf file> - | iconv -f L1 -t UTF-8 | – Yves May 9 '12 at 17:13
2

For completeness here is a sed version:

sed -n -e 's#^ *Group \([0-9]\+\).*#\\Section{Group\1}#p' \
       -e 's#.*\b\([0-9][0-9][0-9]\)\b.*#\\Testdetails{\1}#p'

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.