1

I'd like to process many *.txt files in a directory - they have a common structure (; separated, common headers) but vary in the number of lines each is comprised of i.e. some are just a single line, others up to 8 lines long..

I would like to delete any lines other than the first and the last line for each file. Any helpful pointers?

Update: I've included some test data files as requested:

stat87.txt

Stations_id; Stationshoehe; Geogr.Breite; Geogr.Laenge; von_datum; bis_datum; Stationsname;                                                                                                             
 87;     ; 46.1123;  8.5440;19010101;19661229;Dres

stat01.txt

Stations_id; Stationshoehe; Geogr.Breite; Geogr.Laenge; von_datum; bis_datum; Stationsname;                                                                                                             
 1;     ; 47.8400;  8.8500;18910101;19580228;Aach                                                                                                                                                   
 1;  478; 47.8413;  8.8493;19580301;19860630;Aach

and stat56.txt for example.

Stations_id; Stationshoehe; Geogr.Breite; Geogr.Laenge; von_datum; bis_datum; Stationsname;                                                                                                             
 56;     ; 46.4580;  7.6320;18980101;19450321;Hamb
 56;     ; 46.4580;  7.6320;19450321;19880511;Hamb                                                                                                                                                   
 56;  103; 46.4411;  7.6345;19880601;19990630;Hamb

In this case I'd be especially looking to keep the first line of columns 5 and the last of column 6, so that the time-span of the station is captured.

Results:

find . -type f -name \*.txt -printf "%f\0" | xargs -0 -I xxxx sed -ni '
2 {
  $ {
    s/^[^;]*;[^;]*;[^;]*;[^;]*;\([^;]*\);\([^;]*\).*$/\1;\2/
    p
    q
  }
  s/^[^;]*;[^;]*;[^;]*;[^;]*;\([^;]*\).*$/\1/
  p
}
$ {
  s/^[^;]*;[^;]*;[^;]*;[^;]*;[^;]*;\([^;]*\).*$/\1/
  p
}' xxxx

generates....

19010101;19661229

18910101
19860630

18980101
19990630

I then use a simple sed loop to clean up the final files, by adding the ;

'for file in *.txt; do
  sed 'N;s/\n/;/' "$file" > "cleaned$file"
 done'

19010101;19661229

18910101;19860630

18980101;19990630
  • 1
    OK, the title says "delete first and last", the text says "delete any lines other than the first and last". What's true? – choroba Mar 30 '16 at 21:48
  • @choroba sorry, after midnight this part of the world.. – danny_C_O_T_W Mar 30 '16 at 22:00
  • What should happen to single line and two line files? – choroba Mar 30 '16 at 22:14
  • @choroba stay as is if possible. The meta-step I would like to complete is keep only column 2 of line 1, and column 3 of the last line for example.. These mark the start and endpoints of the variable I need. – danny_C_O_T_W Mar 30 '16 at 22:17
  • 3
    @kleanBean You should provide 2 or 3 data files content example and also an example of the expected result. – Jay jargot Mar 30 '16 at 23:36
2

Perfect, there is below the awk only version:

find . -type f -name \*.txt -printf "%f\0" | xargs -0 -I xxxx sed -ni '
 2 {
   $ {
     s/^[^;]*;[^;]*;[^;]*;[^;]*;\([^;]*\);\([^;]*\).*$/\1;\2/
     p
     q
   }
   s/^[^;]*;[^;]*;[^;]*;[^;]*;\([^;]*\).*$/\1/
   h
 }
 $ {
   s/^[^;]*;[^;]*;[^;]*;[^;]*;[^;]*;\([^;]*\).*$/\1/
   H
   x
   s/\n/;/
   p
 }' xxxx

Thank you to the very famous Sed - An Introduction and Tutorial by Bruce Barnett

Results:

$ cat stat01.txt
18910101;19860630
$ cat stat56.txt
18980101;19990630
$ cat stat87.txt
19010101;19661229

----


first version for reference

Based on your input, I invented data file format and a sed script to work them.

Give this a try:

$ find . -type f -name \*.txt -printf "%f\0" | xargs -0 -I xxxx sed -ni '
 2 {
   $ {
     s/^[^;]*;\([^;]*\);\([^;]*\).*$/\1;\2/
     p
     q
   }
   s/^[^;]*;\([^;]*\).*$/\1/
   p
 }
 $ {
   s/^[^;]*;[^;]*;\([^;]*\).*$/\1/
   p
 }' xxxx

It removes the first line which contains the headers.

It keeps only column 2 of the first data line encountered and the column 3 of the last data line of the file.

If a file contains only one data line, it keeps on one line column 2 and column 3.

hehehe this is odd, but I had a lot of fun !!!

The data files in the current directory:

$ cat test01.txt
Name;Price;Amount;Description
Bread;2.1;3;healthy one
$ cat test02.txt
Name;Price;Amount;Description
Water;0.0;100;For life
Wine;10.3;1;Less than half a glass a day
$ cat test03.txt
Name;Price;Amount;Description
House;1000.0;1;home
Car;500.5;0;no need
Bike;10.3;5;Good for the planet and for me

Results:

$ cat test01.txt
2.1;3
$ cat test02.txt
0.0
1
$ cat test03.txt
1000.0
5

Please provide 2 short data files content and the expected result, and I will modify this answer.

  • thanks!! works perfectly. added my code for the above example files.. – danny_C_O_T_W Mar 31 '16 at 17:12
0

You will need a loop over the files for that:

for file in *.txt; do
  lines=$(wc -l < "$file")
  if [ "$lines" -lt 3 ]; then
    echo "$file is short enough, not touching it."
  else
    # for testing, you can also use the -i option
    sed -n '1p;$p' "$file" > "$file.new"
  fi
done

The loop is necessary if you have files that are only one line long. With the command given by thrig they will turn up twice (try echo 1|sed -n '1p;$p').

  • sed -n '${p;q;};1p' avoids that problem. Making the whole loop reducible to for file in *.txt; do sed -i -n '${p;q;};1p' "$file"; done – Wildcard Mar 31 '16 at 19:17
0

Gawk is a much better tool for this task than sed. Repurposing the original approach's find-xargs pipeline and using the same output nomenclature:

find . -type f -name \*.txt -printf "%f\0" | xargs -0 gawk -F\; '
    FNR==2  { von = $5 }
    ENDFILE { print von FS $6 > "cleaned" FILENAME }
'

The code is made much simpler, much clearer, and eminently more maintainable.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.